The process of storing and retrieving data using the Filecoin network is slightly different from how most storage platforms work. This tutorial walks you through the whole end-to-end process of keeping your data and then getting it back when you need it! This tutorial should take you about an hour to complete.
Start storing your data on the Filecoin network. This section covers packaging your data, importing it into your local Lotus lite-node, finding a storage provider through the Filecoin Plus miner registry, creating a storage deal, and then waiting for the deal to complete. There’s a lot to do, so let’s dive in!
Things to note
As you’re going through this section, make a note of the following variables:
|Data CID||The content identifier (CID) of the data that you want to store using Filecoin.|
|Storage Provider ID #1||The unique identifier for each storage provider. You need to have two storage provider IDs for this tutorial.|
|Storage Provider ID #2||The unique identifier for each storage provider. You need to have two storage provider IDs for this tutorial.|
|Deal CID||The content identifier (CID) for a deal made with a storage provider.|
Prepare your data
For this tutorial, we’re going to create a dummy 5GB file full of random data and store it on the Filecoin network.
Move into your home folder:
Create a 5GB block of random data to serve as our payload:
MacOS users must run:
dd if=/dev/urandom of=5gb-filecoin-payload.bin bs=1m count=5200
Linux users should run:
dd if=/dev/urandom of=5gb-filecoin-payload.bin bs=1M count=5200
This process will take about 60 seconds to create a dummy file.
We now have our payload file ready to be stored using the Filecoin network.
Add data to Lotus
We need to tell our Lotus lite-node which file we want to store using Filecoin.
Import the payload into the
lotus daemonusing the
lotus client import 5gb-filecoin-payload.bin
Lotus creates a directed acyclic graph (DAG) based off the payload. This process takes a few minutes. Once it’s complete, Lotus will output the payload CID.
Import 3, Root bafykb...
This process takes about 60 seconds.
Make a note of the CID
bafykb.... This is your Data CID. We’ll use it in an upcoming section.
Now that Lotus knows which file we want to use, we can create a deal with a Filecoin storage provider to store our data!
Importing custom DAGs
Advance IPLD users may want to import custom DAGs into Lotus (you may skip this section if that is not you).
The CAR file format allows to serialize any IPLD-DAG (i.e. a IPLD-CBOR). Custom IPLD-DAGs should be encoded in a well-known format (like CBOR) as otherwise Lotus will not know how to interpret them.
If you built your own CAR file, make sure to import it directly with the
Files bigger than a sector
If your file is larger than a sector for the Filecoin network in use, you will need to split your file into multiple parts first.
Storage miners will specify which size(s) they’re offering so you can select the best option for you. Smaller sectors are faster, while larger sectors are more cost-effective.
Find a storage provider via Filecoin Plus
Storage providers get paid either by receiving FIL directly from users for storing their data, winning block rewards from the network, or both!
Getting paid from users is straightforward. If Laika wants to store some data, and Albert is a storage provider, the two of them can create a deal to store Laika’s data for
X amount of time for
Block rewards are randomly given to a storage provider every 30 seconds. The more data that a storage provider is storing, the higher their chances of winning the block reward. So if a storage provider accepts a deal from a user to store 5 GB of data, they have 5 chances to win the block reward for each 30 second round.
DataCap acts as a kind of multiplier for block rewards. If a storage provider accepts a deal from a user with DataCap attached, also known as a verified deal , then the Filecoin network treats that deal as though it’s 10x bigger. So a 5 GB deal gives the storage provider 50 chances to win the block reward instead of the usual 5 chances. Some storage providers find DataCap so valuable that they’re willing to make verified deals without charging any FIL! You can find a list of these storage providers using the Filecoin Plus Registry.
Signing up to Filecoin Plus is easy and free!
- Go to plus.fil.org.
- Under For Clients, click Proceed.
- Under Get Verified, click Get DataCap.
- Click Automatic Verification.
- Click Start next to the GitHub logo.
- In the
Requestfield, enter the public address you got from running
lotus wallet list. This step may take a few minutes to complete.
We need to find suitable storage providers before we can store our data. The Filecoin network allows storage providers to compete by offering different terms for pricing, acceptable data sizes, and other important deal parameters. It’s also important to consider the storage provider’s location; the closer the storage provider is to you, the faster the storage and retrieval process will be.
We’re going to use the Filecoin Plus Registry to find a couple of storage providers and check their information through the reputation system.
Filecoin Plus Registry
The Filecoin Plus Registry is a collection of geographically diverse storage providers that are willing to accept low-cost or free storage deals from users. The more storage providers that offer storage in different parts of the world, the faster we can work toward Filecoin’s underlying mission to store humanity’s most important information. It can help you compare storage providers based on their location, pricing and data size limitations, and also their reputation based on their historical performance.
Let’s find a couple of storage providers to store our data.
Using the table, find a couple of storage providers that suit your needs. Try to find storage providers that are geographically close to you, minimum file size is lower than 5 GiB, and charge 0 FIL for verified deals.
Once you find suitable storage providers, you can check more detail info about it by clicking the Arrow next to its reputation score.
Make sure to write down the IDs of the storage providers you want to use. We’ll be referring to these IDs in the next section.
Filecoin Plus Registry only represents a small portion of the entire Filecoin mining community, you can also use other Filecoin reputation systems like FilRep to check more storage provider metrics, like storage power in the network, reachability and overall success rate.
Now that you’ve found your storage providers, you can move onto creating a storage deal!
Create a deal
To complete this section, you need the Data CID you received after running
lotus client import and the IDs of the storage providers you want to use.
Start the interactive deal process:
lotus client deal
The interactive deal assistant will now ask you some questions.
Specify the CID of the payload you want to backup on Filecoin. This is the CID that you got from running
lotus client import ~/5gb-filecoin-payload.bin:
Data CID (from lotus client import): bafykbz...
Wait for Lotus to finish calculating the size of your payload. Lotus calculates this size by counting the individual bits in your payload to ensure that the size is accurate.
.. calculating data size
The duration of this process depends on the size of your file and the specification of your Lotus node. In tests, Lotus took around 20 minutes file of a ~7.5GB file with a 4-core CPU and 8GB RAM. These specifications are common for most end-user laptops.
Enter the number of days you want to keep this file on Filecoin. The minimum is 180 days:
Deal duration (days): 180
Tell Lotus whether or not this is a Filecoin Plus deal. Since you signed up to Filecoin Plus in an earlier step, select
Make this a verified deal? (yes/no): yes
Enter the miner IDs from the previous section with an empty space separating the two IDs:
Miner Addresses (f0.. f0..), none to find: f01000 f01001
Confirm your transaction by entering
----- Proposing from f136b5uqa73jni2rr745d3nek4uw6qiy6b6zmmvcq Balance: 2 FIL Piece size: 8GiB (Payload size: 7.445GiB) Duration: 7200h0m0s Total price: ~0 FIL (0 FIL per epoch) Verified: true Accept (yes/no): yes
Lotus will returns two Deal CIDs:
.. executing Deal (f01000) CID: bafyreict2zhkbwy2arri3jgthk2jyznck47umvpqis3hc5oclvskwpteau Deal (f01001) CID: bafeauyreict2zhkbwy2arri3jgthk2jyznck47umvpqis3hc5oclvskwpt
Take a note of the deal CIDs
Securing a deal
Given the network’s current speed and stability, users may find that individual deals with miners fail unexpectedly. For this reason, we suggest making up to 10 deals for each CAR file you want to store. While this may seem a bit over-kill, it’s a simple way to increase the chances of a successful deal and your data being stored. This work-around will become less and less necessary as the network matures.
Check the deal status
Once the data has been sent to the storage clients, the storage deals can take up to 24 hours to complete. You can check the progress of your deals.
List successful and pending deals by using the
lotus client list-dealscommand:
lotus client list-deals --show-failed
DO NOT TURN OFF YOUR LOTUS NODE! Your Lotus lite-node needs to remain online until the deal state has reached
StorageDealActive. See the Processing states table below to find out which states happen and when.
You can check the progress of any data transfers by running
lotus client list-transfers:
lotus client list-transfers
This command will output something like:
Sending Channels ID Status Sending To Root Cid Initiated? Transferred Voucher 1620782601911586915 Ongoing ...KPFTTwY7 ...zyd3kapm Y 224.1MiB ...bqhcidjmajbelhlxfqry3d7qlu3tvar45a"}} Receiving Channels ...
If the output of
lotus client list-transfersis empty, then your transfer has finished:
lotus client list-transfers
Sending Channels Receiving Channels
Because of the complex nature of Lotus and the Filecoin network, deals can be in one of many different states.
The following table is the list of states that a deal should enter, assuming there are no errors. This list is in chronological order, from when the deal is first created to when it has completed successfully:
|StorageDealUnknown||The current status of a deal is undefined or unknown. This could be because your full-node is out of sync.|
|StorageDealReserveClientFunds||The client is checking that it has enough FIL for the deal.|
|StorageDealClientFunding||The client has deposited funds into the StorageMarketActor and is waiting for the funds to appear.|
|StorageDealFundsReserved||Your FIL has been deposited into escrow and is ready to be used to pay for the deal.|
|StorageDealStartDataTransfer||The storage provider is ready to accept data from the client Lotus node.|
|StorageDealTransferring||The data is being transferred from the client Lotus node to the storage provider.|
|StorageDealCheckForAcceptance||The client is waiting for a storage provider to seal and publish a deal.|
|StorageDealProposalAccepted||The storage provider intends to accept a storage deal proposal; however, the storage provider has not made any commitment to do so at this point.|
|StorageDealAwaitingPreCommit||A deal is ready and must be pre-committed.|
|StorageDealSealing||The storage provider, is sealing data into a sector. The larger your data payload, the longer this will take.|
|StorageDealActive||The data is in a sealed sector, and the storage provider can provide the data back to you.|
|StorageDealExpired||A deal has passed its final epoch. The storage provider could still have the data available but is under no obligation to provide it to anyone.|
The following deal states mean there was a failure somewhere along the line, in alphabetical order:
|StorageDealError||There has been an unforeseen error. No further updates will occur.|
|StorageDealFailing||Something has gone wrong in a deal. Once data is cleaned up, the deal will finalize.|
|StorageDealProposalNotFound||Your full-node cannot find the deal you are looking for. This could be because it doesn’t exist, or your full-node is out of sync.|
|StorageDealProposalRejected||The storage provider, has chosen not to accept this deal. The storage provider may have provided a reason alongside this status message, but not always.|
|StorageDealRejecting||The storage provider has rejected the deal. This comes immediately before StorageDealProposalRejected.|
|StorageDealSlashed||The data was in a sector, and the storage provider got slashed for failing to prove that the data was available.|
The following deal states are informational, and do not mean that a deal has failed. This list is in alphabetical order:
|StorageDealAcceptWait||The storage provider is running custom decision logic to decide whether or not to accept the deal. The deal will have this status until the custom logic comes to a decision.|
|StorageDealClientTransferRestart||A storage deal data transfer from a client to a storage provider has restarted after a pause, likely caused by StorageDealProviderTransferAwaitRestart.|
|StorageDealFinalizing||All the data is within the sector, and the storage provider is performing the final checks to make sure that all the data is correct.|
|StorageDealProviderFunding||The storage provider has deposited funds into StorageMarketActor and is waiting for the funds to appear.|
|StorageDealProviderTransferAwaitRestart||The storage provider restarted while data was being transferred from the client to the storage provider. Once the storage provider is back online, it will wait for the client to resume the transfer.|
|StorageDealPublish||The deal is ready to be published on-chain.|
|StorageDealPublishing||The deal has been published but is yet to appear on-chain.|
|StorageDealReserveProviderFunds||The storage provider is checking that it has enough FIL for the deal.|
|StorageDealStaged||The deal has been published, and data is ready to be put into a sector. At this point, the storage provider has fully committed to storing your data.|
|StorageDealValidating||The storage provider is validating that the deal parameters are good for a proposal.|
|StorageDealVerifyData||All the data has been transferred, and the storage provider is now attempting to verify it against the PieceCID.|
|StorageDealWaitingForData||Either a manual transfer is occurring, or the storage provider has not received a data-transfer request from the client.|
These states come from the Lotus project GitHub repository.