Update RVC2 and RVC4 benchmark scripts to work with the dai Benchmark Nodes #64

ptoupas · 2025-01-09T14:07:16Z

Updated the RVC2 benchmark to work with dai V3 using the newly added Benchmark Nodes.
- The new script accepts .blob, .tar.xz, and model slugs from HubAI.
Added a second benchmark option to RVC4 to use dai V3 as the way to bechmark the models.
- The new script accepts .tar.xz, and model slugs from HubAI. It does not accept .dlc files.

Using the dai Benchmark Nodes the latency measurements are not valid in most cases, so this is omitted from the Benchmark reports in the case where the dai benchmark is being used.

klemen1999

LGTM thanks!

modelconverter/utils/hubai_utils.py

…PI calls.

…XModifier class

… slugs from HubAI

… input when generating the parameters for the dai or the snpe

…pts-with-daiv3

…[ci skip]

… [ci skip]

…pts-with-daiv3

klemen1999

LGTM!

ptoupas added 3 commits January 8, 2025 21:21

Update benchmark scirpt for RVC2 using daiv3

c424740

Add dai based benchmark execution for RVC4 device

6cb5a2c

Ignore latency measurements on dai based benchmark reports

c4b1a5d

ptoupas added the fix Fixes a bug label Jan 9, 2025

ptoupas self-assigned this Jan 9, 2025

ptoupas requested a review from a team as a code owner January 9, 2025 14:07

ptoupas requested review from kozlov721, klemen1999, tersekmatija and conorsim and removed request for a team January 9, 2025 14:07

klemen1999 approved these changes Jan 9, 2025

View reviewed changes

modelconverter/utils/hubai_utils.py Outdated Show resolved Hide resolved

Update is_hubai_available to work with hubAI API calls

e97a453

klemen1999 reviewed Jan 10, 2025

View reviewed changes

modelconverter/utils/hubai_utils.py Outdated Show resolved Hide resolved

ptoupas added 11 commits January 10, 2025 12:02

Update is_hubai_available to work with various teams from HubAI

7295d96

Remove removeprefix to work with python version 3.8 [skip ci]

82a7044

Fix test_modifier test error with EfficientVIT model and change the A…

a34b9ed

…PI calls.

Update .pre-commit-config.yaml

44a097b

Fix model path and HubAI model slug parsing [ci skip]

7d4d223

Add HUBAI_API_KEY to getModelFromZoo calls [ci skip]

57b8982

Update Benchmarking Section of README file [ci skip]

d6e5da1

Update .pre-commit-config.yaml [ci skip]

4d3bc5b

Fix dlc parsing on Benchmark __init__

e8bc974

Update the way modify_onnx optimisation runs are conducted in the ONN…

e2a7ed7

…XModifier class

Fix SNPE benchmark on RVC4 and added support for benchmark over model…

cd2b088

… slugs from HubAI

kozlov721 approved these changes Jan 15, 2025

View reviewed changes

kozlov721 and others added 4 commits January 16, 2025 11:24

Updated ONNX version (#56)

addc5f1

Update the RVC4 benchmark to take into account the data type for each…

f0149cd

… input when generating the parameters for the dai or the snpe

Merge remote-tracking branch 'origin' into fix/update-benchmarks-scri…

2753987

…pts-with-daiv3

Update .pre-commit-config.yaml [ci skip]

8dfdb84

ptoupas added 12 commits January 27, 2025 10:23

Fix issue when extracting the model from NNArchive in snpe benchmark …

b58782c

…[ci skip]

Add bool tensor type during evaluation of onnx models on ONNXModifier…

9b2a602

… [ci skip]

Add a try except block on onnx optimisation and validation.

e081181

Merge remote-tracking branch 'origin' into fix/update-benchmarks-scri…

9cd7158

…pts-with-daiv3

add disable_onnx_optimisation flag on the example defaults.yaml file

565ae6e

Update dai requirement to version 3.0.0a12 [ci skip]

d37ec5e

Add botocore requirement

0548541

Remove the extra-index-url from the requirements-bench.txt file

c2f91f2

Update the README file regarding the depthai v3 installation.

8fd09f1

Update .pre-commit-config.yaml [ci skip]

d409c6c

Update README.md [ci skip]

6f3e950

Merge branch 'main' into fix/update-benchmarks-scripts-with-daiv3

1d08ae7

klemen1999 approved these changes Jan 31, 2025

View reviewed changes

ptoupas merged commit 00017f2 into main Jan 31, 2025
9 of 11 checks passed

ptoupas deleted the fix/update-benchmarks-scripts-with-daiv3 branch January 31, 2025 10:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update RVC2 and RVC4 benchmark scripts to work with the dai Benchmark Nodes #64

Update RVC2 and RVC4 benchmark scripts to work with the dai Benchmark Nodes #64

ptoupas commented Jan 9, 2025

klemen1999 left a comment

klemen1999 left a comment

Update RVC2 and RVC4 benchmark scripts to work with the dai Benchmark Nodes #64

Update RVC2 and RVC4 benchmark scripts to work with the dai Benchmark Nodes #64

Conversation

ptoupas commented Jan 9, 2025

klemen1999 left a comment

Choose a reason for hiding this comment

klemen1999 left a comment

Choose a reason for hiding this comment