Add tests for TFLite models #5
Comments
|
We can also test across different |
Progress on #6. A sample test report HTML file is available here: https://scotttodd.github.io/iree-test-suites/onnx_models/report_2024_09_17.html These new tests * Download models from https://github.com/onnx/models * Extract metadata from the models to determine which functions to call with random data * Run the models through [ONNX Runtime](https://onnxruntime.ai/) as a reference implementation * Import the models using `iree-import-onnx` (until we have a better API: iree-org/iree#18289) * Compile the models using `iree-compile` (currently just for `llvm-cpu` but this could be parameterized later) * Run the models using `iree-run-module`, checking outputs using `--expected_output` and the reference data Tests are written in Python using a set of pytest helper functions. As the tests run, they can log details about what commands they are running. When run locally, the `artifacts/` directory will contain all the relevant files. More can be done in follow-up PRs to improve the ergonomics there (like generating flagfiles). Each test case can use XFAIL like `@pytest.mark.xfail(raises=IreeRunException)`. As we test across multiple backends or want to configure the test suite from another repo (e.g. [iree-org/iree](https://github.com/iree-org/iree)), we can explore more expressive marks. Note that unlike the ONNX _operator_ tests, these tests use `onnxruntime` and `iree-import-onnx` at test time. The operator tests handle that as an infrequently ran offline step. We could do something similar here, but the test inputs and outputs can be rather large for real models and that gets into Git LFS or cloud storage territory. If this test authoring model works well enough, we can do something similar for other ML frameworks like TFLite (#5).
|
May start on this soon, given some recent regressions in tflite/tosa program compilation. |
|
https://pypi.org/project/ai-edge-litert/ no wheels published for Windows... same for the original https://pypi.org/project/tflite-runtime/. Well, that limits testing options. Might be able to test compilation without execution, or generate test golden inputs/outputs on Linux and check those files in. |
Ah! https://github.com/iree-org/iree/blob/main/integrations/tensorflow/test/python/iree_tfl_tests/test_util.py this code works on Windows still import tensorflow.compat.v2 as tf
self.tflite_interpreter = tf.lite.Interpreter(model_path=self.tflite_file) |
Progress on #5. This contains two simple test cases for demonstration purposes, one of which is currently failing due to a regression: iree-org/iree#19402. The test suite follows the same structure as the onnx_models test suite in this repository. Some cleanup and refactoring will be more evident as this grows. We could for example share the `compile_mlir_with_iree` helper function between both test suites.
|
Landed a test suite with two tests so far, running nightly: https://github.com/iree-org/iree-test-suites/actions/workflows/test_litert_models.yml?query=branch%3Amain The new tests also show how |
See nod-ai/SHARK-TestSuite#291
Search around for upstream test suites (corpus of .tflite files)
Could also test TOSA operators maybe using https://git.mlplatform.org/tosa/conformance_tests.git/ (see https://www.mlplatform.org/tosa/software.html)
The text was updated successfully, but these errors were encountered: