SplitStore
Version 1.19.0 introduces SplitStore, an actively scalable blockstore for the Filecoin chain which reduces the performance impact of large blockstores.
SplitStore is a freestanding compacting blockstore that allows you to keep a small 60 GiB to 275 GiB working set in a hot blockstore and reliably archive out-of-scope objects in a coldstore. The coldstore can also be a discard store, whereby out-of-scope objects are discarded, a universal store, which will store all chain data or a messages store which will only store on-chain messages. The messages badger blockstore is the default storage type.
Preparing for SplitStore
/.lotus/datastore
folder! Always enable or manually prune your SplitStore on a fully prepared /.lotus/datastore
folder!- Manually delete the contents of your
/.lotus/datastore/chain
folder. - If you are already running a SplitStore enabed node, you will also need to clear your existing SplitStore folders. You can do this by either running
./lotus-shed spltstore clear
or by manually deleting the contents of your/.lotus/datastore/splitstore
folder.
Enabling SplitStore
To enable the splitstore, edit .lotus/config.toml
and add the following:
[Chainstore]
# type: bool
# env var: LOTUS_CHAINSTORE_ENABLESPLITSTORE
EnableSplitstore = true
If you intend to use the discard-store you also need to add the following:
[Chainstore.Splitstore]
# ColdStoreType specifies the type of the coldstore.
# It can be "messages" (default) to store only messages, "universal" to store all chain state or "discard" for discarding cold blocks.
#
# type: string
# env var: LOTUS_CHAINSTORE_SPLITSTORE_COLDSTORETYPE
ColdStoreType = "discard"
Configuration Options
[Chainstore.Splitstore]
# ColdStoreType specifies the type of the coldstore.
# It can be "messages" (default) to store only messages, "universal" to store all chain state or "discard" for discarding cold blocks.
#
# type: string
# env var: LOTUS_CHAINSTORE_SPLITSTORE_COLDSTORETYPE
ColdStoreType = "messages"
# HotStoreType specifies the type of the hotstore.
# Only currently supported value is "badger".
#
# type: string
# env var: LOTUS_CHAINSTORE_SPLITSTORE_HOTSTORETYPE
HotStoreType = "badger"
# MarkSetType specifies the type of the markset.
# It can be "map" for in memory marking or "badger" (default) for on-disk marking.
#
# type: string
# env var: LOTUS_CHAINSTORE_SPLITSTORE_MARKSETTYPE
MarkSetType = "badger"
# HotStoreMessageRetention specifies the retention policy for messages, in finalities beyond
# the compaction boundary; default is 0.
#
# type: uint64
# env var: LOTUS_CHAINSTORE_SPLITSTORE_HOTSTOREMESSAGERETENTION
HotStoreMessageRetention = 0
# HotStoreFullGCFrequency specifies how often to perform a full (moving) GC on the hotstore.
# A value of 0 disables, while a value 1 will do full GC in every compaction.
# Default is 20 (about once a week).
#
# type: uint64
# env var: LOTUS_CHAINSTORE_SPLITSTORE_HOTSTOREFULLGCFREQUENCY
HotStoreFullGCFrequency = 20
Operation
When the splitstore is first enabled, the existing blockstore becomes the coldstore and a fresh hotstore is initialized.
The hotstore is warmed up on the first startup to load all chain headers and state roots in the current head. This process allows us to immediately gain the performance benefits of a smaller blockstore, which can be substantial for full archival nodes.
All new writes are directed to the hotstore, while reads first hit the hotstore with fallback to the coldstore.
Once five finalities have elapsed and every subsequent finality, the blockstore compacts. Compaction is the process of moving all unreachable objects within the last four finalities from the hotstore to the coldstore. These objects are discarded if the system is configured with a discard coldstore. Chain headers are considered reachable all the way to the genesis block. Stateroots and messages are considered reachable only within the last four finalities unless there is a live reference to them.
Compaction
Compaction works transactionally with the following algorithm:
- We prepare a transaction whereby all i/o referenced objects through the API are tracked.
- We walk the chain and mark reachable objects, keeping four finalities of state roots and messages and all headers all the way to genesis.
- Once the chain walk is complete, we begin full transaction protection with concurrent marking; we walk and mark all references created during the chain walk. At the same time, all I/O through the API concurrently marks objects as live references.
- We collect cold objects by iterating through the hotstore and checking the mark set; if an object is not marked, then it is a candidate for purge.
- When running with a coldstore, we next copy all cold objects to the coldstore.
- At this point, we are ready to begin purging
- We then end the transaction and compact/garbage collect the hotstore.
Cold Store Garbage Collection
Garbage collection can be performed manually by running the lotus chain prune <flags>
command.
Relocating the Coldstore
Following successful configuration and activation of the SplitStore it is now also possible to further optimise daemon chain storage by relocating the coldstore data to slower and potentially less critical standard spinning disks. This can be accomplished by simply symlinking the current /<lotus-repo>/datastore/chain
folder to a new folder located in your standard storage path.
mkdir /<standard-storage-path>/chain
ln -s /<lotus-repo>/datastore/chain /<standard-storage-path>/chain
Utilities
lotus-shed
has a splitstore
command which provides some utilities:
rollback
– rolls back a splitstore installation. This command copies the hotstore on top of the coldstore, and then deletes the splitstore directory and associated metadata keys. It can also optionally compact/gc the coldstore after the copy (with the--gc-coldstore
flag) and automatically rewrite the lotus config to disable splitstore (with the--rewrite-config
flag). The node must be stopped before running this command.clear
– clears a splitstore installation for restart from snapshot.check
– asynchronously runs a basic healthcheck on the splitstore. The results are appended to<lotus-repo>/datastore/splitstore/check.txt
.info
– prints some basic information about the splitstore.