Skip to content

Commit

Permalink
Add ADR for using AI Verify for running technical tests
Browse files Browse the repository at this point in the history
  • Loading branch information
uittenbroekrobbert committed Jun 19, 2024
1 parent 25b5b7a commit 7724299
Showing 1 changed file with 39 additions and 0 deletions.
39 changes: 39 additions & 0 deletions docs/projects/tad/adrs/0005-ai-verify-technical-tests.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# TAD-0005 Use AI Verify for technical tests

## Context

The AI Verify project is set up in a modular way, and the technical tests are one of the modules. The AI Verify
team is developing a feature which makes it possible to run the technical tests using an API, by using
a python file calling a method to run a test and providing the required configuration, like which
model and dataset to use and some test specific requirements. The result of the test is written to a file in a JSON format.

### Pros

* We have several technical tests we can use of the shelf.
* Tests are set up in a generic way using interfaces which allows others, like ourselves, to create their own plugins.
* Loading models, pipelines and data is done through the AI Verify toolkit, which does validation
and support checks.

### Cons

* The result of a test is dependent on how AI Verify has implemented it.
* Adding our own plugins may require adding AI Verify front end blocks we don't use ourselves.
* We are dependent on the AI Verify ecosystem for supported models and data formats.

## Assumptions

* We can wrap the API and other AI Verify requirements in a docker image.
* We can run the docker image independently where we only have to provide the model, datasets and other
required configuration to run a test.

## Decision

We will use the technical tests of AI Verify for the TAD tool.

## Risks

We are tied to the AI Verify ecosystem.

## Consequences

We have a set of technical tests we can integrate in the TAD tool.

0 comments on commit 7724299

Please sign in to comment.