Warning
The AI network is in it's Beta phase and although it is ready for production it is still under development. Please report any issues you encounter to the Livepeer Discord.
This repository hosts the AI runner for processing AI inference jobs on the Livepeer network.
The AI runner is a containerized Python application which processes inference requests on Livepeer AI's Pipelines and models. It loads models into GPU memory and exposes a REST API other programs like the Livepeer node AI worker can use to request AI inference requests. The AI runner code sits in the runner directory.
To build the AI runner locally and run examples, follow these steps:
- Follow the instructions in this document to download model checkpoints and build the runner image.
- Generate Go bindings for the runner OpenAPI spec with
make codegen. - Run any examples in the
cmd/examplesdirectory, e.g.,go run cmd/examples/text-to-image/main.go <RUNS> <PROMPT>.
A high level sketch of how the runner is used:
The AI runner, found in the app directory, consists of:
- Routes: FastAPI routes in app/routes that handle requests and delegate them to the appropriate pipeline.
- Pipelines: Modules in app/pipelines that manage model loading, request processing, and response generation for specific AI tasks.
It also includes utility scripts:
- bench.py: Benchmarks the runner's performance.
- gen_openapi.py: Generates the OpenAPI specification for the runner's API endpoints.
- dl_checkpoints.sh: Downloads model checkpoints from Hugging Face.
- modal_app.py: Deploys the runner on Modal, a serverless GPU platform.
Regenerate the OpenAPI specification for the AI runner's API endpoints with:
python gen_openapi.pyTo correspondingly generate the Go client bindings in the go-livepeer repository,
you should clone livepeer/go-livepeer and run:
# in the go-livepeer repo
make ai_worker_codegenAlternatively, if you want to test the client from a development version of
ai-runner, you can specify a commit hash or branch to generate from. e.g.:
# for commit `aa7ab76`
make ai_worker_codegen REF=aa7ab76
# for branch `vg/test`
make ai_worker_codegen REF=refs/heads/vg/testFor more on developing and debugging the AI runner, see the development documentation.
Based off of this repo.