Lotus performs mining operations, such as sealing files into sectors, calculating proofs over those files and submitting the proofs on chain. Lotus also performs markets operations, providing storage and serving retrievals to clients.
It is now possible to run mining and markets subsystems in separate processes. Service providers can accept storage and retrieval deals without impacting ongoing mining operations. The markets process communicates with the mining process over JSON-RPC.
It is highly recommended to run the mining and markets processes on separate physical or virtual machines so that:
- the machine hardware can be targeted according to the typical workload of the process
- only the machine running the markets process exposes public ports
However it is still advantageous to run the processes separately on the same machine to isolate them - for example the storage service provider can stop and restart the markets process without affecting an ongoing Winning PoSt Window PoSt on the storage provider.
The steps below will guide you through the procedure to backup your mining node, create an initial configuration for your brand new markets node, disable markets functionality on the mining node, and bring both the mining and markets nodes online.
The following diagram summarizes the architecture of a split miner/markets deployment:
As you can see, only the markets and the fullnode/daemon run a libp2p host and expose public ports to the Internet. The mining/sealing/storage node stays entirely private. This confers protection and reduces operational risk.
The markets node speaks to the mining/sealing/storage node through the sealing and storage APIs, mainly to hand off pieces for sealing coming from storage deals, as well as to fetch pieces to serve retrievals.
Both the markets and the mining/sealing/storage nodes need access to the JSON-RPC endpoint of the daemon in order to query the chain and the state tree, as well as to push messages to the network via the mpool.
As you will see below, the
lotus-miner CLI commands need access to
all JSON-RPC endpoints. Remember that JSON-RPC endpoints are private and are not
exposed publicly (unless you operate a public fullnode or a gateway node!)
Another aspect worth mentioning is that both the markets and the mining/sealing/storage nodes can be configured to share the storage system (disk array, or filesystem), either locally when they’re running on the same machine, or as network mounts when running on separate machines. This setup enables more efficient access and reduces network IO loads.
lotus-miner monolith into subsystems
This document explains how to split an existing monolith
lotus-miner node into multiple processes to take advantage of this new architecture.
Lotus v1.11.1 introduced the notion of
subsystems in the
Currently there are 4 subsystems that can be enabled or disabled in the
[Subsystems] # EnableMining = true # EnableSealing = true # EnableSectorStorage = true # EnableMarkets = true
These options are set to
true by default and you are running a
lotus-miner monolith under this setting. Until now, the monolith
has been responsible for all functionality.
subsystems can be enabled and disabled individually per process within each node’s own config file. There are only two combinations that are supported right now:
marketsnode - a
lotus-minerprocess responsible for handling the storage market subsystem, and all functionality related to serve storage and retrieval deals;
[Subsystems] EnableMining = false EnableSealing = false EnableSectorStorage = false EnableMarkets = true
mining/sealing/provingnode - a
lotus-minerprocess responsible for Filecoin mining, sector storage, sealing and proving;
[Subsystems] EnableMining = true EnableSealing = true EnableSectorStorage = true EnableMarkets = false
lotus-miner instance is configured as a
markets node, it exposes
a libp2p interface. The libp2p ports should be publically available so that
the node can be dialed by clients that wish to make storage and retrieval deals.
This guide won’t go into details on how to make the endpoint connectable, as this ultimately depends on your network infrastructure and equipment, refer to the lotus-miner connectivity guide for more details. Generally speaking, configuring static port mappings on your external-facing router, and ensuring your markets node has an internal static IP address and port should be sufficient, in addition to adjusting any firewalls that may be present.
lotus-minerinstance is configured as a
mining/sealing/provingnode, it does not receive requests and should not be publicly exposed on the Internet. The
marketsnode communicates with the
mining/sealing/provingnode via its JSON RPC interface. Just to reiterate, the
mining/sealing/provingno longer runs a libp2p interface, and the
mining/sealing/provingdo not communicate over libp2p, but through HTTP and/or Websockets (JSON-RPC interface).
Split the Market Process
Preparation and Backup
Before splitting the markets service process from the monolith storage provider process,
we need to backup the storage provider’s metadata repository. Stop the
lotus daemon and
restart both of them with the
LOTUS_BACKUP_BASE_PATH environment variable:
/tmpfolder as it is a temporary file that is just used to initialize the markets node, it’s not used afterwards.
in the machine running the full node:
lotus daemon stop export LOTUS_BACKUP_BASE_PATH=/tmp
in the machine running the storage provider node:
lotus-miner stop export LOTUS_BACKUP_BASE_PATH=/tmp
Restart both daemon and lotus-miner
Create the backup on the machine running the storage provider node
lotus-miner backup /tmp/backup.cbor
Split the Market Subsystem
Step 1. Create a seed config.toml for the markets service
To initialize the markets service we need to create a seed
the markets node. Note: It is a temporary file and only used for initialize the market node, full configuration file will be loaded automatically upon initialization without overiding the sections set in this step.
First create a
config.toml file in a location of your choice(your market node needs to have access to read it), in this tutorial we will place it in the
/tmp directory on your market machine as it is a temporary file)
Enable the market subsystem and add the sessions that are related to the market process in lotus storage provider configuration in to the
If you intend to run the
mining/sealing/proving node on the same machine as the
markets node, make sure that their
[API] listener addresses** do not clash**.
By default the
lotus-miner API server listens to port 2345, so in the example
configuration above for the
markets process, we change the configuration for the
markets node API server to listen to
Make sure you adjust the
[Libp2p] section on the
markets node accordingly - it
needs to be publicly accessible so that clients can make storage and retrieval
deals with your system.
~/.lotusminer/config.tomlto preserve your configuration! If you want to initialize the market nodes with the default dealmaking configuration, you can skip this section.
[Subsystems] EnableMining = false EnableSealing = false EnableSectorStorage = false EnableMarkets = true [API] # Endpoint where the markets process will serve JSON-RPC requests ListenAddress = "/ip4/127.0.0.1/tcp/8787/http" RemoteListenAddress = "127.0.0.1:8787" [Libp2p] # Endpoint for libp2p requests (public) ListenAddresses = ["/ip4/0.0.0.0/tcp/24001", "/ip6/::/tcp/24001"] AnnounceAddresses = ["/ip4/18.104.22.168/tcp/24001"] [Dealmaking] # When enabled, the miner can accept online deals ConsiderOnlineStorageDeals = true # When enabled, the miner can accept offline deals ConsiderOfflineStorageDeals = true # When enabled, the miner can accept retrieval deals ConsiderOnlineRetrievalDeals = true # When enabled, the miner can accept offline retrieval deals ConsiderOfflineRetrievalDeals = true # When enabled, the miner can accept verified deals ConsiderVerifiedStorageDeals = true # When enabled, the miner can accept unverified deals ConsiderUnverifiedStorageDeals = true # A list made of Data CIDs to reject when making deals PieceCidBlocklist =  # Maximum expected amount of time getting the deal into a sealed sector will take # This includes the time the deal will need to get transferred and published # before being assigned to a sector # for more info, see below. ExpectedSealDuration = "24h0m0s" # When a deal is ready to publish, the amount of time to wait for more # deals to be ready to publish before publishing them all as a batch PublishMsgPeriod = "1h0m0s" # The maximum number of deals to include in a single publish deals message MaxDealsPerPublishMsg = 8 # A command used for fine-grained evaluation of storage deals (see below) Filter = "/absolute/path/to/storage_filter_program" # A command used for fine-grained evaluation of retrieval deals (see below) RetrievalFilter = "/absolute/path/to/retrieval_filter_program" [Fees] MaxPreCommitGasFee = "0.025 FIL" MaxCommitGasFee = "0.05 FIL" MaxTerminateGasFee = "0.5 FIL" MaxWindowPoStGasFee = "5 FIL" MaxPublishDealsFee = "0.05 FIL" MaxMarketBalanceAddFee = "0.007 FIL" # Fees needs to be present, or it will operate with the default values.
[Libp2p]section of the
mining/sealing/provingnode can be removed because it will no longer be running a Libp2p node, as we explained above.
Step 2. Disable the Market Subsystem on the
mining/sealing/proving storage provider process
- On your
mining/sealing/provingmachine, update your
~/.lotusminer/config.tomlto set the
[Subsystems] EnableMining = true EnableSealing = true EnableSectorStorage = true EnableMarkets = false
Step 3. Initialize a
markets service repository
- Create authentication tokens for the markets nodes and they will be used by the
marketsnode to make JSON-RPC calls to the
export APISEALER=`lotus-miner auth api-info --perm=admin` export APISECTORINDEX=`lotus-miner auth api-info --perm=admin`
- Initialize the
marketsnode. This performs a one-time setup of the markets node. Part of that setup includes updating the
peer idin the storage provider actor by submitting a message on chain. This is necessary so that storage and retrieval clients know that this storage provider’s deal-making endpoint is now publicly dialable/reachable on a new address (the new
For this tutorial, we will us ~/.lotusmarkets` as the markets repository paths. You do not need to manually create the folder in advance.
lotus-miner --markets-repo=~/.lotusmarkets init service --type=markets \ --api-sealer=$APISEALER \ --api-sector-index=$APISECTORINDEX \ --config=/tmp/config.toml \ /tmp/backup.cbor
- If during this procedure, you ended up changing the
marketsprocess to a different host and/or port, you must update storage provider’s
multiaddron-chain. Otherwise, clients wishing to make deals with you will no longer be able to connect to your node.
multiaddronchain should be the same as the
ListenAddressesyou set in the
Libp2psection of the config.toml on your market node.
lotus-miner actor set-addrs <NEW_MULTIADDR>
- After you have finished initializing the
marketsnode repository, you must restart the
mining/sealing/provingnode for the configuration changes to take effect.
Step 4. Consider configuring storage.json on the markets node
If you had customized storage locations for your monolith storage provider, consider either:
- copying the
~/.lotusminer/storage.jsonfile to the
~/.lotusmarketdirectory, if both storage provider and markets are running on the same machine and have access to the same paths.
~/.lotusmarket/storage.jsonto point to network mounts shared with the storage provider, if both processes are running on separate machines.
This will enable the markets process to access unsealed pieces directly from those locations, instead of streaming them to/through the storage provider node over the network, thus reducing network IO loads.
Step 5. Optional: Move the DAG store directory to the markets node repository
If you have initialized the dagstore, you can optionally move the DAG store directory to the lotus markets repository, to carry over the indices you have already built up.
mv ~/.lotusminer/dagstore ~/.lotusmarkets/
Step 6. Start the
markets storage provider process with the markets subsystem
markets node with the environment variable
LOTUS_MINER_PATH pointing to the
markets node repo.
lotus-miner run interacts with either the the
depending on the
LOTUS_MINER_PATH environment variable!
LOTUS_MINER_PATH=~/.lotusmarkets lotus-miner run
And that’s it, you have successfully set up your market subsystem!
lotus-miner CLI commands have been refactored to target the correct subsystem node, depending on issued command. For example:
- If you call
lotus-miner storage-deals list,
lotus-minerknows to target the
- If you call
lotus-miner sectors list,
lotus-minerknows to target the
In addition to the existing environment variables for
LOTUS_MARKETS_PATH were introduced for the market subsystem:
LOTUS_MARKETS_PATHis the location of your market repository.
MARKETS_API_INFOis the location of the market node.
When splitting storage provider and markets subsystems, we recommend adding
.bashrc (or equivalent) with the API address / path for your markets node. You can mix and match, for example, the configuration inside the machine running the markets node:
FULLNODE_API_INFO=... MINER_API_INFO=... LOTUS_MARKETS_PATH=...
A client configuration inside the machine running the
lotus-worker node that wants to interact with the market node and the full node:
FULLNODE_API_INFO=... MINER_API_INFO=... MARKETS_API_INFO=... LOTUS_MINER_PATH=... LOTUS_MARKETS_PATH=...
A client configuration that speaks to all nodes remotely:
FULLNODE_API_INFO=... MINER_API_INFO=... MARKETS_API_INFO=...
Interact with the Market Node from Other Process
For commands that are present in all
lotus-miner processes, by default the CLI targets the
mining/sealing/proving process, but you can explicitly target the
markets process with the
--call-on-markets flag. This is the case for
auth commands, for example.
In order to take advantage of this functionality, you should configure the following environment variable for both
mining/sealing/proving storage provider (or lotus-worker) and the
markets storage provider, in your run-commands file (
export LOTUS_MARKETS_PATH=~/.lotusmarkets export LOTUS_MINER_PATH=~/.lotusminer
If one of these nodes is on a remote machine, you should set the relevant API_INFO environment variables like below:
# <market_api_token> is located in $LOTUS_MINER_PATH/token export MARKETS_API_INFO=<market_api_token>/ip4/<lotus_miner_market_node_ip>/tcp/<lotus_miner_market_node_port>/http # <miner_api_token> is located in $LOTUS_MINER_PATH/token export MINER_API_INFO=<miner_api_token>:/ip4/<lotus_miner_node_ip>/tcp/<lotus_miner_node_port>/http
Now that you have both a
markets storage provider process and a
mining/sealing/proving storage provider process running, you can confirm that you can interact with each respective process with the following:
Get storage provider info
Get the peer identity of the
marketsstorage provider process
lotus-miner --call-on-markets net id
Get a list of storage deals from the
marketsstorage provider process
lotus-miner storage-deals list
Get a list of sectors from the
mining/sealing/provingstorage provider process
lotus-miner sectors list
lotus-miner monolith process
If you want to revert the changes listed above and go back to running
lotus-miner as a single process, run the following:
Make sure that the
mining/sealing/provingnode is publicly exposed, as we will be enabling the markets subsystem on it. Set up the addresses in the
Libp2psection of the
mining/sealing/provingrepository, update the
[Subsystems] EnableMining = true EnableSealing = true EnableSectorStorage = true EnableMarkets = true
Optional: Move back the DAG store directory to the monolith storage provider node repository if you have initialized dagstore before:
mv ~/.lotusmarkets/dagStore ~/$LOTUS_MINER_PATH/
Backup and restore the metadata related to storage deals from the
marketsinstance back to the monolith storage provider instance. Given that storage deals metadata would have changed on the
marketsinstance in case you accepted storage deals while running multi-services architecture, we have to copy it back to the monolith storage provider instance:
a. Build the
lotus-shedbinary if you haven’t done that before:
```shell make lotus-shed ```
```shell lotus-shed market export-datastore --repo ~/.lotusmarkets --backup-dir /tmp/deals-backup lotus-shed market import-datastore --repo ~/$LOTUS_MINER_PATH --backup-path /tmp/deals-backup/markets.datastore.backup ```
mining/sealing/provingnode (with the default LOTUS_MINER_PATH, which should point to your
mining/sealing/provingnode repo). Note that
lotus-minerinteracts with a given repository depending on the
Fetch the node identity. This is necessary as you have to update your storage provider’s peer identity on-chain, as it was changed to the identity of the markets node during the initialising of the markets service repository.
lotus-miner net id
Update the storage provider’s peer id and multiaddr on-chain with the result from the previous step.
lotus-miner actor set-peer-id 12D3XXXXX
As soon as the message is confirmed, clients will know to look for the node identity of your
mining/sealing/proving node, which now also runs the
markets subsystem, i.e. currently all Lotus subsystems.
multiaddr onchain should be the same as the
ListenAddresses you set in the
Libp2p section of the config.toml on your
mining/sealing/proving node now. If it’s changed, update it by running:
```shell lotus-miner actor set-addrs <NEW_MULTIADDR> ```
Feessection of the markets
config.tomlto be the same as same the mining/sealing/proving node.
[Fees] MaxPreCommitGasFee = "0.025 FIL" MaxCommitGasFee = "0.05 FIL" MaxTerminateGasFee = "0.5 FIL" MaxWindowPoStGasFee = "5 FIL" MaxPublishDealsFee = "0.05 FIL" MaxMarketBalanceAddFee = "0.007 FIL" [Addresses] PreCommitControl = ["f3uqltf1"] CommitControl = ["f3uqltf2"] TerminateControl = ["f3uqltf3"] DealPublishControl = ["f3uqltf4"] DisableOwnerFallback = false DisableWorkerFallback = false
Typically, if you initialize the markets node from the
config.toml used in the mining/sealing/proving node, the settings in this section will be the same as in the mining/sealing/proving node. If you are already running a subsystem and have changed the configuration in this area, copy the changes to the markets node. If you do not copy the changes, the
DealPublishControl configuration will not affect, and
DealPublish costs will be deducted from the worker wallet.