Skip to content

Commit

Permalink
feat: Add tio your friendly cmd line uncle to run triton-llm servic…
Browse files Browse the repository at this point in the history
…es (#174)

This provides a simple example of how to write a triton-llm engine, and how to connect it to the OpenAI HTTP server.

This is the tool previously called `nio` and `llmctl`.

- **Inputs**: Text and HTTP.
- **Engines**: Echo, which streams your prompt back with a slight delay.

Build: `cargo build`

Pre-requisites: `nats-server` and `etcd` must be running locally, even though they are not yet used by `tio`.

Run with text input:
```
./target/debug/tio in=text out=echo_full --model-name test
```

Run with the triton-llm HTTP server:
```
./target/debug/tio in=http out=echo_full --http-port 8080 --model-name Echo-0B
```

List models:
```
curl localhost:8080/v1/models | jq
```

Will output
```
{
  "object": "list",
  "data": [
    {
      "id": "Echo-0B",
      "object": "object",
      "created": 1739400430,
      "owned_by": "nvidia"
    }
  ]
}
```

#### What's next

As triton-distributed gains features `tio` will be able to grow:
- When we get the pre-processor we can have token-in token-out engines. 
- When we get a pull-router we can have `in=nats` and `out=nats`.
- When we get discovery we can have dynamic engines.
  • Loading branch information
grahamking authored Feb 13, 2025
1 parent a36f530 commit 2abdbd1
Show file tree
Hide file tree
Showing 12 changed files with 4,340 additions and 2 deletions.
10 changes: 8 additions & 2 deletions .github/workflows/pre-merge-rust.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,15 +51,21 @@ jobs:
key: ${{ runner.os }}-cargo-tools-${{ github.head_ref || github.ref_name }}-${{ hashFiles('**/Cargo.lock') }}
- name: Set up Rust Toolchain Components
run: rustup component add rustfmt clippy
- name: Run Cargo Check
- name: Run Cargo Check on runtime
working-directory: runtime/rust
run: cargo check --locked
- name: Run Cargo Check on tio
working-directory: apps/tio
run: cargo check --locked
- name: Verify Code Formatting
working-directory: runtime/rust
run: cargo fmt -- --check
- name: Run Clippy Checks
- name: Run Clippy Checks on runtime
working-directory: runtime/rust
run: cargo clippy --no-deps --all-targets -- -D warnings
- name: Run Clippy Checks on tio
working-directory: apps/tio
run: cargo clippy --no-deps --all-targets -- -D warnings
- name: Install and Run cargo-deny
working-directory: runtime/rust
run: |
Expand Down
Loading

0 comments on commit 2abdbd1

Please sign in to comment.