IPFS_Model_Manager_Py

The IPFS model manager is supposed to manage the files that are being stored on the local node, as well as indices for all models in the libp2p peers and its own local node. It then decides to cache these files depending on how "hot" they are in terms of being requested from its own queue, and the pub sub broadcasts that have been sent requesting datasets, models, or inference. The model manager then uses local disk cache, the local IPFS node, S3, libp2p, IPFS_cluster, Huggingface Hub, storacha, and filecoin lassie for retrieval in order the "hotness" the files in question and the storage quotas. During retrieval, it runs a test at downloading readme.md, and whichever source is the fastest will be the one which it tries to retrieve the rest of the data.

Each model's metadata is in the form of a "manifest" which contains important information such as the search embedding for the manifest, the license, the entire file tree and IPFS hashes for all the files in the file tree, and hardware resource usage profiling and compatibility data that is used by the MLOPS code to schedule jobs based on hardware compatibility and estimated time to be completed.