Begin testing models from the ONNX Model Zoo.

.github/workflows/test_onnx_models.yml

@@ -0,0 +1,61 @@
+# Copyright 2024 The IREE Authors
+#
+# Licensed under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+name: Test ONNX Models
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - ".github/workflows/test_onnx_models.yml"
+      - "onnx_models/**"
+  pull_request:
+    paths:
+      - ".github/workflows/test_onnx_models.yml"
+      - "onnx_models/**"
+  workflow_dispatch:
+  schedule:
+    # Runs at 3:00 PM UTC, which is 8:00 AM PST
+    - cron: "0 15 * * *"
+
+concurrency:
+  # A PR number if a pull request and otherwise the commit hash. This cancels
+  # queued and in-progress runs for the same PR (presubmit) or commit
+  # (postsubmit). The workflow name is prepended to avoid conflicts between
+  # different workflows.
+  group: ${{ github.workflow }}-${{ github.event.number || github.sha }}
+  cancel-in-progress: true
+
+jobs:
+  test-onnx-models:
+    runs-on: ubuntu-24.04
+    env:
+      VENV_DIR: ${{ github.workspace }}/.venv
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      # Install Python packages.
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - name: Setup Python venv
+        run: python3 -m venv ${VENV_DIR}
+      - name: Install IREE nightly release Python packages
+        run: |
+          source ${VENV_DIR}/bin/activate
+          python3 -m pip install -r onnx_models/requirements-iree.txt
+
+      # Run tests.
+      - name: Run ONNX models test suite
+        run: |
+          source ${VENV_DIR}/bin/activate
+          pytest onnx_models/ \
+            -rA \
+            --log-cli-level=info \
+            --timeout=120 \
+            --durations=0

README.md

@@ -18,11 +18,19 @@ See https://groups.google.com/g/iree-discuss/c/GIWyj8hmP0k/ for context.
 * Built with [cmake](https://cmake.org/) and run via
   [ctest](https://cmake.org/cmake/help/latest/manual/ctest.1.html) (for now?).
 
+### [onnx_models/](onnx_models/) : Open Neural Network Exchange models
+
+[![Test ONNX Models](https://github.com/iree-org/iree-test-suites/actions/workflows/test_onnx_models.yml/badge.svg?branch=main)](https://github.com/iree-org/iree-test-suites/actions/workflows/test_onnx_models.yml?query=branch%3Amain)
+
+* Tests that import, compile, and run ONNX models through IREE then compare
+  the outputs against a reference (ONNX Runtime).
+* Runnable via [pytest](https://docs.pytest.org/).
+
 ### [onnx_ops/](onnx_ops/) : Open Neural Network Exchange operations
 
 [![Test ONNX Ops](https://github.com/iree-org/iree-test-suites/actions/workflows/test_onnx_ops.yml/badge.svg?branch=main)](https://github.com/iree-org/iree-test-suites/actions/workflows/test_onnx_ops.yml?query=branch%3Amain)
 
 * 1250+ tests for [ONNX](https://onnx.ai/) framework
   [operators](https://onnx.ai/onnx/operators/).
-* Runnable via [pytest](https://docs.pytest.org/en/stable/) using a
+* Runnable via [pytest](https://docs.pytest.org/) using a
   configurable set of flags to `iree-compile` and `iree-run-module`.

onnx_models/.gitignore

@@ -0,0 +1 @@
+artifacts/*

onnx_models/README.md

@@ -0,0 +1,162 @@
+# ONNX Model Tests
+
+This test suite exercises ONNX (Open Neural Network Exchange: https://onnx.ai/)
+models. Most pretrained models are sourced from https://github.com/onnx/models.
+
+Testing follows several stages:
+
+```mermaid
+graph LR
+  Model --> ImportMLIR["Import into MLIR"]
+  ImportMLIR --> CompileIREE["Compile with IREE"]
+  CompileIREE --> RunIREE["Run with IREE"]
+  RunIREE --> Check
+
+  Model --> LoadONNX["Load into ORT"]
+  LoadONNX --> RunONNX["Run with ORT"]
+  RunONNX --> Check
+
+  Check["Compare results"]
+```
+
+## Quickstart
+
+1. Set up your virtual environment and install requirements:
+
+    ```bash
+    python -m venv .venv
+    source .venv/bin/activate
+    python -m pip install -r requirements.txt
+    ```
+
+    * To use `iree-compile` and `iree-run-module` from Python packages:
+
+        ```bash
+        python -m pip install -r requirements-iree.txt
+        ```
+
+    * To use local versions of `iree-compile` and `iree-run-module`, put them on
+      your `$PATH` ahead of your `.venv/Scripts` directory:
+
+        ```bash
+        export PATH=path/to/iree-build:$PATH
+        ```
+
+2. Run pytest using typical flags:
+
+    ```bash
+    pytest \
+      -rA \
+      --log-cli-level=info
+      --durations=0 \
+    ```
+
+    See https://docs.pytest.org/en/stable/how-to/usage.html for other options.
+
+## Advanced pytest usage
+
+* The `log-cli-level` level can also be set to `debug`, `warning`, or `error`.
+  See https://docs.pytest.org/en/stable/how-to/logging.html.
+* Run only tests matching a name pattern:
+
+    ```bash
+    pytest -k resnet
+    ```
+
+* Skip "medium" sized tests using custom markers
+  (https://docs.pytest.org/en/stable/example/markers.html):
+
+    ```bash
+    pytest -m "not size_medium"
+    ```
+
+* Ignore xfail marks
+  (https://docs.pytest.org/en/stable/how-to/skipping.html#ignoring-xfail):
+
+    ```bash
+    pytest --runxfail
+    ```
+
+* Run tests in parallel using https://pytest-xdist.readthedocs.io/
+  (note that this swallows some logging):
+
+    ```bash
+    # Run with an automatic number of threads (usually one per CPU core).
+    pytest -n auto
+
+    # Run on an explicit number of threads.
+    pytest -n 4
+    ```
+
+## Debugging tests outside of pytest
+
+Each test generates some files as it runs:
+
+```text
+├── artifacts
+│   └── vision
+│       └── classification
+│           ├── mnist-12_version17_cpu.vmfb      (Program compiled using IREE's llvm-cpu target)
+│           ├── mnist-12_version17_input_0.bin   (Random input generated using numpy)
+│           ├── mnist-12_version17_output_0.bin  (Reference output from onnxruntime)
+│           ├── mnist-12_version17.mlir          (The model imported to MLIR)
+│           ├── mnist-12_version17.onnx          (The model upgraded to a minimum supported version)
+│           └── mnist-12.onnx                    (The downloaded ONNX model)
+```
+
+Running a test with logging enabled will show what the test is doing:
+
+```console
+pytest --log-cli-level=debug -k mnist
+
+======================================= test session starts =======================================
+platform win32 -- Python 3.11.2, pytest-8.3.3, pluggy-1.5.0
+rootdir: D:\dev\projects\iree-test-suites\onnx_models
+configfile: pytest.ini
+plugins: reportlog-0.4.0, timeout-2.3.1, xdist-3.6.1
+collected 17 items / 16 deselected / 1 selected
+
+tests/vision/classification_models_test.py::test_mnist
+------------------------------------------ live log call ------------------------------------------
+INFO     onnx_models.utils:utils.py:125 Upgrading 'artifacts\vision\classification\mnist-12.onnx' to 'artifacts\vision\classification\mnist-12_version17.onnx'
+DEBUG    onnx_models.conftest:conftest.py:90 Session input [0]
+DEBUG    onnx_models.conftest:conftest.py:91   name: 'Input3'
+DEBUG    onnx_models.conftest:conftest.py:94   shape: [1, 1, 28, 28]
+DEBUG    onnx_models.conftest:conftest.py:95   numpy shape: (1, 1, 28, 28)
+DEBUG    onnx_models.conftest:conftest.py:96   type: 'tensor(float)'
+DEBUG    onnx_models.conftest:conftest.py:97   iree parameter: 1x1x28x28xf32
+DEBUG    onnx_models.conftest:conftest.py:129 Session output [0]
+DEBUG    onnx_models.conftest:conftest.py:130   name: 'Plus214_Output_0'
+DEBUG    onnx_models.conftest:conftest.py:131   shape (actual): (1, 10)
+DEBUG    onnx_models.conftest:conftest.py:132   type (numpy): 'float32'
+DEBUG    onnx_models.conftest:conftest.py:133   iree parameter: 1x10xf32
+DEBUG    onnx_models.conftest:conftest.py:217 OnnxModelMetadata(inputs=[IreeModelParameterMetadata(name='Input3', type='1x1x28x28xf32', data_file=WindowsPath('D:/dev/projects/iree-test-suites/onnx_models/artifacts/vision/classification/mnist-12_version17_input_0.bin'))], outputs=[IreeModelParameterMetadata(name='Plus214_Output_0', type='1x10xf32', data_file=WindowsPath('D:/dev/projects/iree-test-suites/onnx_models/artifacts/vision/classification/mnist-12_version17_output_0.bin'))])
+INFO     onnx_models.utils:utils.py:135 Importing 'artifacts\vision\classification\mnist-12_version17.onnx' to 'artifacts\vision\classification\mnist-12_version17.mlir'
+INFO     onnx_models.conftest:conftest.py:160 Launching compile command:
+  cd D:\dev\projects\iree-test-suites\onnx_models && iree-compile artifacts\vision\classification\mnist-12_version17.mlir --iree-hal-target-backends=llvm-cpu -o artifacts\vision\classification\mnist-12_version17_cpu.vmfb
+INFO     onnx_models.conftest:conftest.py:180 Launching run command:
+  cd D:\dev\projects\iree-test-suites\onnx_models && iree-run-module --module=artifacts\vision\classification\mnist-12_version17_cpu.vmfb --device=local-task --input=1x1x28x28xf32=@artifacts\vision\classification\mnist-12_version17_input_0.bin --expected_output=1x10xf32=@artifacts\vision\classification\mnist-12_version17_output_0.bin
+PASSED                                                                                       [100%]
+
+================================ 1 passed, 16 deselected in 1.81s =================================
+```
+
+For this test case there is one input with shape/type `1x1x28x28xf32` stored at
+`artifacts/vision/classification/mnist-12_version17_input_0.bin` and one output
+with shape/type `1x10xf32` stored at
+`artifacts/vision/classification/mnist-12_version17_output_0.bin`.
+
+We can reproduce the compile and run commands with:
+
+```bash
+iree-compile \
+  artifacts/vision/classification/mnist-12_version17.mlir \
+  --iree-hal-target-backends=llvm-cpu \
+  -o artifacts/vision/classification/mnist-12_version17_cpu.vmfb
+
+iree-run-module \
+  --module=artifacts/vision/classification/mnist-12_version17_cpu.vmfb \
+  --device=local-task \
+  --input=1x1x28x28xf32=@artifacts/vision/classification/mnist-12_version17_input_0.bin \
+  --expected_output=1x10xf32=@artifacts/vision/classification/mnist-12_version17_output_0.bin
+```