Thabit

Open source platform to evaluate multiple LLM models with custom datasets.

Demo

Installation

pip3 install thabit

Test

pytest tests

Build

pip3 install -e .

Docs

Visit https://docs.thabit.ai

TODO:

Validate the input dataset.
UI for adding/editing config.
Visulaise Output (using UI).
Run eval per dataset (add folders for dataset and for evals). This is to simplify visualising results later using the UI.
```
root
├── datasets
│ └── a
└── evals
  └── a
```