Welcome to the Synthia subnet, a bleeding-edge initiative to accelerate the open-source AI space. Our mission is to harness the power of Commune's decentralized incentive markets to produce a continuous stream of synthetic training data with verified quality at scale.
- Synthia
Synthia is utilizing the state-of-the-art Anthropic Claude3 API to generate open-ended high-quality and diverse synthetic in-depth explanations of subjects picked from the Opus latent space based on varying esotericity, with varying target-audience, level of detail and abstraction at scale.
While any model or API can theoretically mine in the subnet, the validation is designed to target Claude3-level quality, due to its substantially superior ability to generate the desired synthetic data. Hence, we advise mining with the Claude3 API, although support for OpenAI's API is available.
In the rapidly evolving world of artificial intelligence, synthetic data has emerged as a crucial component in the training of advanced models. By utilizing the state-of-the-art Anthropic Claude3 API, we can generate open-ended subject-unconstrained high-quality and diverse synthetic in-depth explanations.
Major AI labs have already recognized the potential of synthetic data and are actively utilizing it to enhance their models. However, access to such data remains limited for the broader open-source community. The Synthia subnet aims to change that.
By harnessing the power of Commune's decentralized crypto-economic incentives, we aim to create the largest reliably high-quality synthetic intelligence dataset in the world that will serve as a catalyst for innovation in the Open-Source AI space.
Join us on this important journey as we distill the Closed-Source intelligence right into the hands of the Open-Source Community!
- You can check the HuggingFace leaderboard here!
- You can see the real-time dataset produced by Synthia here!
Make sure you are on the latest CommuneX version.
pip install communex --upgrade- Install Docker
- Run
docker pull ghcr.io/agicommies/synthia:9d23f1f - Run
docker run -v ~/.commune:/root/.commune -it [-p <port>:<port>] ghcr.io/agicommies/synthia:9d23f1f - Run
poetry shellto enter the enviroment
- You can quit docker with ctrl+d
- You can dettach from your session with ctrl+p followed by ctrl+q
- You can attach back to your session by running
docker attach <id> - You can list the ids of your containers with
docker ps - Note that you should pass the ports you're going to use to the container (with
-p <port>:<port>) to bind them to your host machine. - You can pass enviroments variables to docker with
-e <VARIABLE>=<value>. e.gdocker run -e ANTHROPIC_API_KEY=<your-anthropic-api-key> -v ~/.commune:/root/.commune -it ghcr.io/agicommies/synthia:9d23f1f
- Install Python 3
sudo apt install python3
- Install Poetry
- Install the Python dependencies with
poetry install - ! IMPORTANT Enter the Python environment with
poetry shell
- Install Nix with install.determinate.systems
- You can enter the nix shell environment with with
nix developor setup direnv to automatically load the environment when you enter the directory. - Install the Python dependencies with
poetry install - Get into the Python environment:
- If you are using
direnv, just re-entering the directory will do the trick.- Tip: you can force-reload with
direnv reload
- Tip: you can force-reload with
- If not, you can run
poetry shellto enter the Python environment.
- If you are using
-
Get an API key from Anthropic.
-
Create a file named
config.envin theenv/folder with the following contents (you can also see theenv/config.env.sampleas an example):ANTHROPIC_API_KEY="<your-anthropic-api-key>" OPENROUTER_API_KEY="<your-openrouter-api-key>" ANTHROPIC_MODEL=claude-3-opus-20240229 ANTHROPIC_MAX_TOKENS=1000 ANTHROPIC_TEMPERATURE=0.5
Alternatively, you can set up those values as enviroment variables. Note that you just need to provide the key to the provider that you're going to use
-
Serve the miner:
Make sure to be located in the root of synthia repository
cd synthiaProceed with running the miner:
python3 -m synthia.miner.cli <your_commune_key> --ip 0.0.0.0 [--port]
Alternatively, if you want to run a openrouter miner:
python3 -m synthia.miner.cli <your_commune_key> --ip 0.0.0.0 [--port] --provider openrouter
The ip is passed as 0.0.0.0 to accept outside connections, since the default, 127.0.0.1 accepts only local connections. Synthia has the netuid 3. Key is a name of your commune wallet/key. If you don't have a wallet, generate one by running
comx key create <name>
Note: you need to keep this process alive, running in the background. Some options are tmux, pm2 or nohup.
Example using pm2
pm2 start "comx module serve synthia.miner.anthropic.AnthropicModule <key> --subnets-whitelist <synthia netuid> --ip 0.0.0.0" --name <name>
-
Finally register the module on the Synthia subnet:
comx module register <name> <your_commune_key> --ip <your-ip-address> --port <port> --netuid <synthia netuid>
-
Make sure to serve and register the module using the same key.
-
If you are not sure about your
public ipaddress:curl -4 https://ipinfo.io/ip
-
Current
<synthia netuid>is 3. If you want to check for yourself, you can run:comx subnet list
And look for the name
synthia
-
Get an API key from Anthropic.
-
Gen an API key for embeddings from OpenAi
-
Create a file named
config.envin theenv/folder with the following contents (you can also see theenv/config.env.sampleas an example):ANTHROPIC_API_KEY="<your-anthropic-claude-api-key>" OPENROUTER_API_KEY="<your-openrouter-api-key>" ANTHROPIC_MODEL=claude-3-opus-20240229 ANTHROPIC_MAX_TOKENS=1000 ANTHROPIC_TEMPERATURE=0.5 OPENAI_API_KEY="<your-openai-api-key>"
Alternatively, you can set up those values as enviroment variables.
-
Register the validator
Note that you are required to register the validator first, this is because the validator has to be on the network in order to set weights. You can do this by running the following command:
comx module register <name> <your_commune_key> --netuid <synthia netuid>
The current synthia netuid is 3.
-
Serve the validator
python3 -m synthia.cli <your_commune_key> [--call-timeout <seconds>] [--provider <provider_name>]
The default value of the
--call-timeoutparameter is 65 seconds. You can pass --provider openrouter to run using openrouter providerNote: you need to keep this process alive, running in the background. Some options are tmux, pm2 or nohup.
- CPU: Quad-core Intel i3 or equivalent AMD processor, 2.5 GHz
- RAM: 2 GB
- Storage: 500mb + of free space
- GPU: Not needed
- Network: Broadband internet connection for online data syncing
If you want to run up to ~10+ miners / validators
- CPU: 4-core Intel i5 or equivalent AMD processor, 2.5 GHz-3.5 GHz
- RAM: 4 GB or more
- Storage: 128 GB SSD
- GPU: Not needed
- Network: Gigabit Ethernet or better
Eden has provided a new bash script that will walk you through the process of launching a validator or miner for a simpler and streamlined process.
Allow commands to be executed by the script:
chmod +x scripts/launch.sh
Run the launcher:
bash scripts/launch.sh
Just follow the prompts after that.
The launch script will prompt you step by step through the process of launching a validator or miner or both and execute the required commands without having to know details about the CLI.
Be aware that the launcher does execute commands that make changes on the block chain including balance transfers and module registration. Be sure you know what you'd like to do before using this tool as some actions cannot be undone. This tool is provided free of charge as is and with no warranty or guarantee. Use at your own risk.