-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add ADR for using AI Verify for running technical tests
- Loading branch information
1 parent
25b5b7a
commit 7724299
Showing
1 changed file
with
39 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
# TAD-0005 Use AI Verify for technical tests | ||
|
||
## Context | ||
|
||
The AI Verify project is set up in a modular way, and the technical tests are one of the modules. The AI Verify | ||
team is developing a feature which makes it possible to run the technical tests using an API, by using | ||
a python file calling a method to run a test and providing the required configuration, like which | ||
model and dataset to use and some test specific requirements. The result of the test is written to a file in a JSON format. | ||
|
||
### Pros | ||
|
||
* We have several technical tests we can use of the shelf. | ||
* Tests are set up in a generic way using interfaces which allows others, like ourselves, to create their own plugins. | ||
* Loading models, pipelines and data is done through the AI Verify toolkit, which does validation | ||
and support checks. | ||
|
||
### Cons | ||
|
||
* The result of a test is dependent on how AI Verify has implemented it. | ||
* Adding our own plugins may require adding AI Verify front end blocks we don't use ourselves. | ||
* We are dependent on the AI Verify ecosystem for supported models and data formats. | ||
|
||
## Assumptions | ||
|
||
* We can wrap the API and other AI Verify requirements in a docker image. | ||
* We can run the docker image independently where we only have to provide the model, datasets and other | ||
required configuration to run a test. | ||
|
||
## Decision | ||
|
||
We will use the technical tests of AI Verify for the TAD tool. | ||
|
||
## Risks | ||
|
||
We are tied to the AI Verify ecosystem. | ||
|
||
## Consequences | ||
|
||
We have a set of technical tests we can integrate in the TAD tool. |