Damocles

# SnapDeal Support

# SnapDeal Overview

SnapDeal is a sector upgrade proposal introduced in FIP-19 (opens new window) and merged in nv15. Compared to previous upgrade proposals that required a complete re-seal process, SnapDeal is relatively lightweight. Its cost is approximately:

  • Completing an add piece task
  • Completing a tree d task
  • Completing a snap_encode task, which costs about the same as a pc2 task
  • Completing a snap_prove task, which costs about the same as a c1 + c2 task

Therefore, SnapDeal is attractive for both first-time data onboarding and upgrading of existing CC sectors.

# Damocles SnapDeal Support

Damocles is designed to provide a way to mass produce storage power. For this purpose, we have introduced a SnapDeal production scheme that requires minimal human intervention, which we call SnapUp. The steps involved are as follows:

  1. Import existing CC sectors in batches as candidate sectors.
  2. Configure damocles-manager to enable SnapUp mode for specified SPs.
  3. Configure sealing_thread of damocles-worker to SnapUp production plans or add new sealing_thread for SnapUp.

Throughout this process, users only need to focus on importing new candidate sectors and monitoring the remaining of candidate sectors, as all other processes will be automated.

# Example

Let's use an example of a production cluster on the butterfly network to demonstrate how to configure the SnapUp production scheme.

# Importing Candidate Sectors

Using the newly added util sealer snap fetch tool, you can import CC sectors that meet the SnapUp condition (remaining lifetime greater than 180 days and satisfying the minimum lifetime requirement for storage deals) as candidate sectors per their deadlines.

./dist/bin/damocles-manager util sealer snap fetch 1153 3
2022-04-15T04:28:03.380Z        DEBUG   policy  policy/const.go:18      NETWORK SETUP   {"name": "butterfly"}
2022-04-15T04:28:03.401Z        INFO    cmd     internal/util_sealer_snap.go:53 candidate sectors fetched        {"available-in-deadline": 2, "added": 2}

# Monitoring Remaining Candidate Sector

./dist/bin/damocles-manager util sealer snap candidates 1153
2022-04-15T04:28:13.955Z        DEBUG   policy  policy/const.go:18      NETWORK SETUP   {"name": "butterfly"}
deadline  count
3         2

In the above example, there are currently 2 CC sectors available in the #3 deadline as candidates for upgrade.

# Configuring damocles-worker

In damocles-worker, the main configurations required related to SnapUp job are the resource allocation for snap_encode and snap_prove tasks in sealing_thread.

Here is an example configuration:

[[sealing_thread]]
location = "/data/local-snap-1"
plan = "snapup"

[[sealing_thread]]
location = "/data/local-snap-2"
plan = "snapup"

[[sealing_thread]]
location = "/data/local-snap-3"
plan = "snapup"

[[sealing_thread]]
location = "/data/local-snap-4"
plan = "snapup"

[[sealing_thread]]
location = "/data/local-snap-5"
plan = "snapup"

[[sealing_thread]]
location = "/data/local-snap-6"
plan = "snapup"

[[sealing_thread]]
location = "/data/local-snap-6"
plan = "snapup"

[processors.limitation.concurrent]
# ...
tree_d = 1
snap_encode = 5
snap_prove = 2

[[processors.snap_encode]]
envs = { FIL_PROOFS_USE_GPU_COLUMN_BUILDER = "1", FIL_PROOFS_USE_GPU_TREE_BUILDER = "1", CUDA_VISIBLE_DEVICES = "0", TMPDIR="/var/tmp/worker0" }

[[processors.snap_prove]]
envs = { CUDA_VISIBLE_DEVICES = "0" , TMPDIR="/var/tmp/worker0" }

[[processors.snap_encode]]
envs = { FIL_PROOFS_USE_GPU_COLUMN_BUILDER = "1", FIL_PROOFS_USE_GPU_TREE_BUILDER = "1", CUDA_VISIBLE_DEVICES = "0", TMPDIR="/var/tmp/worker1" }

[[processors.snap_prove]]
envs = { CUDA_VISIBLE_DEVICES = "1", TMPDIR="/var/tmp/worker1" }

The resource allocation for snap_encode can be similar to pc2, and for snap_prove, it can be similar to c2.

# Configuring damocles-manager

In damocles-manager, the required configuration mainly involves enabling SnapUp for specific SPs. Here is an example:

[[Miners]]
Actor = 1153
[Miners.Sector]
InitNumber = 0
MaxNumber = 10000
Enabled = true
EnableDeals = false

[Miners.SnapUp]
Enabled = true
Sender = "t1abjxfbp274xpdqcpuaykwkfb43omjotacm2p3za"

In this configuration:

  • The settings in [Miners.Sector] do not affect SnapUp production.
  • Under this example configuration, you would get...
    • continuous production of CC Sector
    • SnapUp continues production while there are available candidate sectors

# Tips

  • Considering the computing resources required by snap_encode and snap_prove, if you enable regular sector sealing and SnapUp at the same time in the same damocles-worker instance, you may encounter race condition of hardware resource. You can refer to 07.damocles-worker external executor configuration example (En doc to be updated).

  • Considering the deployment of sector persistent storage, the damocles-worker used for SnapUp needs to be able to have both read and write access to all persistent storage spaces (persist stores), and ensure that their names are consistent with those defined in damocles-manager.

  • Based on the above two points, we recommend using a separate box specifically for SnapUp production to avoid the configuration, operation and maintenance complexity caused by mixing regular sectors and SnapUp sealing.

# Continuous Improvement

The improvement and optimization of the SnapUp solution are still in progress. Currently we are mainly focusing on:

  • Convert semi-automatic candidate sector import to automatic mode, or provide equivalent operation and maintenance tools
  • More candidate sector import rules, such as import by storage configuration
  • Aggregation of on-chain messages to reduce costs
  • Other optimizations and tooling that can simplify operation & maintenance, reduce costs, and improve efficiency

← Worker config guide →