Damocles

# Import existing sector data

When one would like to migrate the sector storage directory sealed by other sealer solution to damocles, it needs to use damocles-manager to import and update the corresponding configuration file.

# Import and verify

Note: Both import and validation need to be done without damocles-manager daemon being started.

# import

damocles-manager provides an import tool called storage attach, which can be used as following:

damocles-manager util storage attach --verbose --name={storage name} <path>
  • name is an optional parameter;
  • <path> is the storage path, which is converted to an absolute path during import.

For details of name and <path>, please refer to [Common.PersistStores](./04.damocles-manager-config.md #commonpersiststores).

For example, by using...

damocles-manager util storage attach --verbose --name=a ./mock-tmp/remote

logs similar to the following would likely be produced:

2022-03-11T16:03:52.492+0800 DEBUG policy policy/const.go:18 NETWORK SETUP {"name": "mainnet"}
2022-03-11T16:03:52.493+0800 INFO cmd internal/util_storage.go:104 use match pattern "/home/dtynn/proj/github.com/ipfs-force-community/venus-cluster/mock-tmp/remote /sealed/*" {"name": "a", "strict": false, "read-only": false}
2022-03-11T16:03:52.493+0800 INFO cmd internal/util_storage.go:121 path "s-t010000-16" matched=true {"name": "a", "strict": false, "read-only ": false}
2022-03-11T16:03:52.494+0800 INFO cmd internal/util_storage.go:121 path "s-t010000-17" matched=true {"name": "a", "strict": false, "read-only ": false}
2022-03-11T16:03:52.494+0800 INFO cmd internal/util_storage.go:121 path "s-t010000-18" matched=true {"name": "a", "strict": false, "read-only ": false}
2022-03-11T16:03:52.508+0800 INFO cmd internal/util_storage.go:148 sector indexer updated for s-t010000-16 {"name": "a", "strict": false, "read-only": false}
2022-03-11T16:03:52.509+0800 INFO cmd internal/util_storage.go:148 sector indexer updated for s-t010000-17 {"name": "a", "strict": false, "read-only": false}
2022-03-11T16:03:52.509+0800 INFO cmd internal/util_storage.go:148 sector indexer updated for s-t010000-18 {"name": "a", "strict": false, "read-only": false}
2022-03-11T16:03:52.509+0800 INFO cmd internal/util_storage.go:152 3 sectors out of 3 files have been updated {"name": "a", "strict": false, "read-only": false}
2022-03-11T16:03:52.509+0800 WARN cmd internal/util_storage.go:153 add the section below into the config file: {"name": "a", "strict": false, "read-only": false}

[[Common.PersistStores]]
Name = "a"
Path = "/home/dtynn/proj/github.com/ipfs-force-community/damocles/mock-tmp/remote"
Strict = false
ReadOnly = false

At this point, the directory import has been completed, and the location information of all sectors has also been recorded. We can then complete the import by copying configuration in the output into the configuration file of damocles-manager.

# reimport

If we find that the information supplied during import is wrong, such as --name is misspelled, we can re-import with correct information to complete the import process. The location information of the sector will be overwritten and updated.

# separation of sealed_file and cache_dir

Some sealer components allow sealed_file and cache_dir to be on different storage instances, in which case regular import may fail to locate sector files. In this case, you can enable the split scan mode by adding the command line parameter --allow-splitted. In this mode, paths in the sealed folder and the cache folder that match the sector naming rules will be scanned separately.

At this point, the log will look similar to:

2022-04-19T19:11:55.137+0800 DEBUG policy policy/const.go:18 NETWORK SETUP {"name": "mainnet"}
2022-04-19T19:11:55.154+0800 INFO cmd internal/util_storage.go:120 scan for sectors(upgrade=false) {"name": "p3", "strict": false, "read-only": false , "splitted": true}
2022-04-19T19:11:55.154+0800 INFO cmd internal/util_storage.go:211 0 sectors out of 0 files have been found {"name": "p3", "strict": false, "read-only": false, "splitted": true}
2022-04-19T19:11:55.154+0800 INFO cmd internal/util_storage.go:145 scan for splitted cache dirs(upgrade=false) {"name": "p3", "strict": false, "read-only" : false, "splitted": true}
2022-04-19T19:11:55.155+0800 INFO cmd internal/util_storage.go:211 3 sectors out of 3 files have been found {"name": "p3", "strict": false, "read-only": false, "splitted": true}
2022-04-19T19:11:55.156+0800 INFO cmd internal/util_storage.go:120 scan for sectors(upgrade=true) {"name": "p3", "strict": false, "read-only": false , "splitted": true}
2022-04-19T19:11:55.156+0800 INFO cmd internal/util_storage.go:211 0 sectors out of 0 files have been found {"name": "p3", "strict": false, "read-only": false, "splitted": true}
2022-04-19T19:11:55.156+0800 INFO cmd internal/util_storage.go:145 scan for splitted cache dirs(upgrade=true) {"name": "p3", "strict": false, "read-only" : false, "splitted": true}
2022-04-19T19:11:55.156+0800 INFO cmd internal/util_storage.go:211 0 sectors out of 0 files have been found {"name": "p3", "strict": false, "read-only": false, "splitted": true}
2022-04-19T19:11:55.156+0800 WARN cmd internal/util_storage.go:166 add the section below into the config file: {"name": "p3", "strict": false, "read-only": false, "splitted": true}

[[Common.PersistStores]]
Name = "p3"
Path = "/home/dtynn/proj/github.com/ipfs-force-community/damocles/
mock-tmp/pstore3"
Strict = false
ReadOnly = false

Notes for using this mode:

  • There are no duplicate stored files due to storage exceptions in the target directory
  • Sectors with only cache_dir and no corresponding sealed_file won't be located

# Validation

The storage find tool provided by damocles-manager can be used to check whether the result of sector import is correct. It can be used as following:

damocles-manager util storage find <miner actor id> <sector number>

Continuing with an example of the validation command demonstrated above, we want to verify that the sector s-t010000-17 has been recorded correctly, we can use:

damocles-manager util storage find 10000 17

Usually a log similar to the following would be produced:

2022-04-19T19:13:15.235+0800 DEBUG policy policy/const.go:18 NETWORK SETUP {"name": "mainnet"}
2022-04-19T19:13:15.249+0800 INFO cmd internal/util_storage.go:279 sector s-t010000-17 located, sealed file in "a", cache dir in "a"
2022-04-19T19:13:15.249+0800 INFO cmd internal/util_storage.go:285 store instance exists {"instance": "a"}
2022-04-19T19:13:15.249+0800 INFO cmd internal/util_storage.go:285 store instance exists {"instance": "a"}

This means that our import and configuration work is complete.

# Validation exception: sector information was not recorded successfully

If you encounter something like the following during the verification process...

2022-03-11T16:45:59.120+0800 WARN cmd internal/util_storage.go:214 s-t010000-17 not found

Such log indicates that the specified sector was not imported successfully, and we need to recheck the import process.

# Validation exception: Storage configuration not updated

If you encounter something like the following during the verification process...

2022-03-11T16:22:34.044+0800 DEBUG policy policy/const.go:18 NETWORK SETUP {"name": "mainnet"}
2022-03-11T16:22:34.059+0800 INFO cmd internal/util_storage.go:218 found s-t010000-17 in "a"
2022-03-11T16:22:34.059+0800 WARN cmd internal/util_storage.go:227 store instance not found, check your config file

Such a log indicates that the configuration file of damocles-manager has not been successfully updated, and we need to update the configuration according to the method mentioned above.

←