PoSt workers

PoSt workers are specialized instances of a lotus-worker. It enables you to offload windowPoSt and winningPoSt to separate workers.

While the Lotus Miner runs windowPoSt and winningPoSt by default, you can use specialized lotus workers to create a cluster of windowPoSt and winningPoSt workers that can handle multiple Proof-of-Spacetime partitions simultaneously.

A lotus worker instance can only be one of the following:

  • WindowPoSt worker
  • WinningPoSt worker
  • Worker for sealing tasks

Launching PoSt workers

Before launching your PoSt worker, you need to ensure that your worker meets the minimal requirements for the job it will perform. You should also consider how the PoSt workers can access the sealed sectors and set the correct environment variables before running the workers.

Minminal requirements

These are the minimal requirements for running each of the PoSt tasks:

TaskGPUVRAMRAM
winningPoStHighly recommended6 GiB*64 GiB*
windowPoStHighly recommended8.5 GiB128 GiB

*These are conservative numbers. We are currently exploring the lower boundaries for winningPoSt.

Remote storage access

The windowPoSt process requires reading random leaves of all the sealed sectors in a proving deadline. When setting up windowPoSt workers, one needs to consider how the workers can access those files. The PoSt workers can ask any other worker to read challenges for them, including the lotus-miner process, but it will prefer reading it from local paths.

Environment variables

Remember to have the appropriate nvidia-drivers and nvidia-opencl-icd installed if running OpenCL on your worker. If using CUDA, install the CUDA-toolkit and build lotus binaries with FFI_USE_CUDA=1

The following environment variables are required to be set before starting the worker:

export MINER_API_INFO:<TOKEN>:/ip4/<miner_api_address>/tcp/<port>/http
export RUST_GPU_TOOLS_CUSTOM_GPU="MODEL-NAME:CORES" # If you´re using a custom GPU
export FIL_PROOFS_PARAMETER_CACHE=/fast/disk/folder # > 100GiB!

When the PoSt worker starts, it needs to read and verify the Filecoin proof parameters. We recommend copying them over from your lotus-miner machine. Otherwise, they will be downloaded locally on first run.

The PoSt workers will fail to start if the file descriptor limit is not set high enough. You can raise this limit temporarily before starting the worker by running the command ulimit -n 1048576. Although, we recommend setting it permanently by following the Permanently Setting Your ULIMIT System Value guide.

Run the PoSt worker

Use the run command to start the worker:

lotus-worker run <flags>

You´ll need to specify which PoSt operation you want the worker to perform with one of the following flags set to true:

--winningpost                 enable winning post (default: false)
--windowpost                  enable window post (default: false)

A PoSt worker instance can only be either a winningPoSt worker, or a windowPoSt worker. Enabling a PoSt tasks will automatically disable all other tasks.

When a winningPoSt or windowPoSt worker connects to the lotus-miner, it will delegate all winningPoSt or windowPoSt tasks to that worker. If both tasks are delegated to separate PoSt workers, the lotus-miner will execute no PoSt tasks locally on the miner instance. If a worker is stopped, the lotus-miner instance automatically switches back to local PoSt.

You can verify that PoSt workers are connected to the lotus-miner with lotus-miner proving workers:

lotus-miner proving workers
> Worker 0, host windowPoSt
>         CPU:  [                                                                ] 0/48 core(s) in use
>         RAM:  [|                                                               ] 1% 4.005 GiB/119.2 GiB
>         VMEM: [|                                                               ] 1% 4.005 GiB/119.2 GiB
>         GPU:  [                                                                ] 0% 0.00/1 gpu(s) in use
>         GPU: GeForce RTX 2080 Ti, not used
> 
> Worker 1, host winningPoSt
>         CPU:  [                                                                ] 0/16 core(s) in use
>         RAM:  [|                                                               ] 1% 4.005 GiB/119.2 GiB
>         VMEM: [|                                                               ] 1% 4.005 GiB/119.2 GiB
>         GPU:  [                                                                ] 0% 0.00/1 gpu(s) in use
>         GPU: GeForce RTX 2080 Ti, not used

You can also verify that PoSt workers are connected in the workers section of lotus-miner info.

lotus-miner info

> Workers:
> 
>     Seal: 15
>     WdPoSt: 4
>     WinPoSt: 3

Stop the PoSt worker

Use the stop command to stop the worker:

lotus-worker stop <flags>

Multiple partitions

If you have multiple partitions in a single proving deadline and multiple windowPoSt workers, each partition will run on separate workers in parallel, up to the number of partitions.

Consider this proving deadline with four full partitions:

lotus-miner proving deadlines
Miner: f011235
deadline  partitions  sectors (faults)  proven partitions
0         4           9396 (0)          0

If the storage provider has four windowPoSt workers connected, the lotus-miner will compute each of the partitions on each of the workers in parallel. If one windowPoSt worker gets disconnected, leaving you with only three windowPoSt workers, the first three partitions will be computed in parallel. The first windowPoSt worker will pick up the last partition to finish its computation.

Depending on the scale of your operation it may be necessary to fine-tune the proving processes of your storage provider. This is particularly true in the case of DeclareFaultsRecovered messages across multiple partitions. This is done in the Proving section of the Lotus Miner’s config.toml:

  # Extract taken from version 1.17.1

[Proving]
  # Maximum number of sector checks to run in parallel. (0 = unlimited)
  # 
  # WARNING: Setting this value too high may make the node crash by running out of stack
  # WARNING: Setting this value too low may make sector challenge reading much slower, resulting in failed PoSt due
  # to late submission.
  # 
  # After changing this option, confirm that the new value works in your setup by invoking
  # 'lotus-miner proving compute window-post 0'
  #
  # type: int
  # env var: LOTUS_PROVING_PARALLELCHECKLIMIT
  #ParallelCheckLimit = 128

  # Maximum number of partitions to prove in a single SubmitWindowPoSt messace. 0 = network limit (10 in nv16)
  # 
  # A single partition may contain up to 2349 32GiB sectors, or 2300 64GiB sectors.
  # 
  # The maximum number of sectors which can be proven in a single PoSt message is 25000 in network version 16, which
  # means that a single message can prove at most 10 partinions
  # 
  # In some cases when submitting PoSt messages which are recovering sectors, the default network limit may still be
  # too high to fit in the block gas limit; In those cases it may be necessary to set this value to something lower
  # than 10; Note that setting this value lower may result in less efficient gas use - more messages will be sent,
  # to prove each deadline, resulting in more total gas use (but each message will have lower gas limit)
  # 
  # Setting this value above the network limit has no effect
  #
  # type: int
  # env var: LOTUS_PROVING_MAXPARTITIONSPERPOSTMESSAGE
  #MaxPartitionsPerPoStMessage = 0

  # In some cases when submitting DeclareFaultsRecovered messages,
  # there may be too many recoveries to fit in a BlockGasLimit.
  # In those cases it may be necessary to set this value to something low (eg 1);
  # Note that setting this value lower may result in less efficient gas use - more messages will be sent than needed,
  # resulting in more total gas use (but each message will have lower gas limit)
  #
  # type: int
  # env var: LOTUS_PROVING_MAXPARTITIONSPERRECOVERYMESSAGE
  #MaxPartitionsPerRecoveryMessage = 0

Version 1.17.1 also introduces the LOTUS_RECOVERING_SECTOR_LIMIT environment variable which allows you to limit the number of sectors declared in each DeclareFaultsRecovered message.

LOTUS_RECOVERING_SECTOR_LIMIT=<number-of-sectors>

Testing the setup

When making changes to your PoSt setup it is useful to verify that the changes works as intended without testing it on a real proving period and risk failing windowPoSt.

After you have set up your windowPoSt workers you can manually trigger a windowPoSt with lotus-miner proving compute window-post [deadline index]. It will not send any messages to the chain.

Edit this page on GitHub