LevelDB metadata corruption recovery

Problem: ##

Corruption in the LevelDB metadata might happen due to a bad RAM-stick. In such cases you will encounter issues restarting the lotus-miner process again. The error log will look something similar to this:

  - leveldb/table: corruption on data-block (pos=1352397): checksum mismatch, want=0x972e02b got=0x43ac9dfa [file=xxxx.ldb]

Environment: ##

• Mainnet
• Calibnet

Resolution: ##

First start by copying the corrupted metadata-folder to another location:

rclone copy -P .lotusminer/datastore/metadata ./oldmeta

Transferred:      218.913k / 218.913 kBytes, 100%, 80.787 MBytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            7 / 7, 100%
Elapsed time:          0s

Then install plyvel, a Python interface to LevelDB.

pip install plyvel

Collecting plyvel
  Downloading plyvel-1.5.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (766 kB)
     |████████████████████████████████| 766 kB 27.2 MB/s 
Installing collected packages: plyvel
Successfully installed plyvel-1.5.0

Then we can write our Python program to recover the metadata. Type python3 and write the program in the prompts:

python3

Python 3.8.10 (default, Jun 22 2022, 20:18:18) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import plyvel
>>> db = plyvel.DB('oldmeta')
>>> db2 = plyvel.DB('newmeta', create_if_missing=True)
>>> for key, v in db:
...     db2.put(key, v)
... 
>>> db2.close()
>>> [exit]

You should now have a new folder called newmeta with the recovered metadata. Move the original and corrupted metadata to a backup location.

rclone move -P .lotusminer/datastore/metadata ./backup-meta

We can now move and rename the recovered metadata in the newmeta folder into the .lotusminer/datastore/ folder.

rclone move -P ./newmeta .lotusminer/datastore/metadata

You can now restart the lotus-miner process and see that it has been recovered.

Extras: ##

These steps are only needed in cases where no proper metadata backup has been generated. It is advised to setup proper automated backups of the metadata