Skip to content

Latest commit

 

History

History
41 lines (33 loc) · 1.84 KB

README.md

File metadata and controls

41 lines (33 loc) · 1.84 KB
Raw

Spader

Spader is a simply utility to allow Filecoin SPs to manage the offline deal data download. This utility is stateful and offers a resumption on restart.

key Features

  1. Submits the downloaded deal to Boost
  2. Creates a list of completed downloads based on deal UUID. Makes easy to import in Boost
  3. Uses aria2 for download
  4. Spawns and control its own aria2 daemon
  5. Resumes download after restart
  6. Takes disk space into account (calculated based on piece size)
  7. User can define how many deals they want to process in parallel. It will keep working at full capacity till the storage is full

Pre-requisites

  1. User must have aria2 CLI installed on the server where they wish to run Spader.

  2. Install aria2p python package

    pip install aria2p

  3. Export LOTUS_FULLNODE_INFO and BOOST_API_INFO variables

    lotus auth api-info --perm admin
boostd auth api-info --perm admin

  4. Provide values to the empty variables defined in ##### VARIABLES ##### section of spade.py file

Importing deals

Once the download is completed, the program will write deal UUID and the corresponding .car file name to <download directory>/completed

File structure inside download directory

Name Description
aria2c.log aria2c logs specific downloads controlled by spader
aria2c.session aria2c session information. Used to resume downloads
completed Contains all the deals finished downloading
download Directory containing the downloaded files
failed Contains all the deals for which download errored out
spader.log Spader logs