Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Makefile: Minor cleanup and introduction of level-specific targets #599

Merged
merged 7 commits into from
Jan 2, 2025

Conversation

hanno-becker
Copy link
Contributor

@hanno-becker hanno-becker commented Dec 31, 2024

Based on #602

  • Use {func, kat, nistkat, acvp, ...} to build test binaries.
  • Use run_{func, kat, nistkat, acvp, bench, bench_components} to run and check test binaries.
  • Introduce level-specific targets
  • Other minor cleanups

@hanno-becker hanno-becker force-pushed the makefile_ext branch 4 times, most recently from b4aee3c to f3e1239 Compare December 31, 2024 18:55
@hanno-becker hanno-becker marked this pull request as ready for review December 31, 2024 19:32
@hanno-becker hanno-becker requested a review from a team as a code owner December 31, 2024 19:32
@hanno-becker hanno-becker force-pushed the makefile_ext branch 3 times, most recently from 2001594 to 8bf3698 Compare January 1, 2025 05:12
@hanno-becker hanno-becker added the benchmark this PR should be benchmarked in CI label Jan 1, 2025
Copy link

@oqs-bot oqs-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arm Cortex-A76 (Raspberry Pi 5) benchmarks

Benchmark suite Current: 8bf3698 Previous: 169a6c3 Ratio
ML-KEM-512 keypair 29036 cycles 29020 cycles 1.00
ML-KEM-512 encaps 35344 cycles 35398 cycles 1.00
ML-KEM-512 decaps 45883 cycles 45886 cycles 1.00
ML-KEM-768 keypair 49401 cycles 49348 cycles 1.00
ML-KEM-768 encaps 55601 cycles 55571 cycles 1.00
ML-KEM-768 decaps 70348 cycles 70329 cycles 1.00
ML-KEM-1024 keypair 72071 cycles 72075 cycles 1.00
ML-KEM-1024 encaps 80892 cycles 80847 cycles 1.00
ML-KEM-1024 decaps 100731 cycles 100681 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

Copy link

@oqs-bot oqs-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intel Xeon 4th gen (c7i)

Benchmark suite Current: 8bf3698 Previous: 169a6c3 Ratio
ML-KEM-512 keypair 13510 cycles 13540 cycles 1.00
ML-KEM-512 encaps 17261 cycles 17373 cycles 0.99
ML-KEM-512 decaps 22722 cycles 22944 cycles 0.99
ML-KEM-768 keypair 22487 cycles 22553 cycles 1.00
ML-KEM-768 encaps 24481 cycles 24621 cycles 0.99
ML-KEM-768 decaps 32437 cycles 32674 cycles 0.99
ML-KEM-1024 keypair 31449 cycles 31432 cycles 1.00
ML-KEM-1024 encaps 35115 cycles 34946 cycles 1.00
ML-KEM-1024 decaps 45887 cycles 45851 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

Copy link

@oqs-bot oqs-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intel Xeon 3rd gen (c6i)

Benchmark suite Current: 8bf3698 Previous: 169a6c3 Ratio
ML-KEM-512 keypair 20345 cycles 20332 cycles 1.00
ML-KEM-512 encaps 27012 cycles 27009 cycles 1.00
ML-KEM-512 decaps 35816 cycles 35816 cycles 1
ML-KEM-768 keypair 34891 cycles 34890 cycles 1.00
ML-KEM-768 encaps 38130 cycles 38186 cycles 1.00
ML-KEM-768 decaps 50936 cycles 50935 cycles 1.00
ML-KEM-1024 keypair 47984 cycles 47973 cycles 1.00
ML-KEM-1024 encaps 54144 cycles 54150 cycles 1.00
ML-KEM-1024 decaps 71586 cycles 71716 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

Copy link

@oqs-bot oqs-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AMD EPYC 3rd gen (c6a)

Benchmark suite Current: 8bf3698 Previous: 169a6c3 Ratio
ML-KEM-512 keypair 18143 cycles 18127 cycles 1.00
ML-KEM-512 encaps 23188 cycles 23185 cycles 1.00
ML-KEM-512 decaps 30612 cycles 30507 cycles 1.00
ML-KEM-768 keypair 31072 cycles 31071 cycles 1.00
ML-KEM-768 encaps 34171 cycles 34169 cycles 1.00
ML-KEM-768 decaps 44758 cycles 44724 cycles 1.00
ML-KEM-1024 keypair 44618 cycles 44607 cycles 1.00
ML-KEM-1024 encaps 49871 cycles 49891 cycles 1.00
ML-KEM-1024 decaps 64347 cycles 64348 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

Copy link

@oqs-bot oqs-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AMD EPYC 4th gen (c7a)

Benchmark suite Current: 8bf3698 Previous: 169a6c3 Ratio
ML-KEM-512 keypair 15096 cycles 15077 cycles 1.00
ML-KEM-512 encaps 19680 cycles 19664 cycles 1.00
ML-KEM-512 decaps 26303 cycles 26317 cycles 1.00
ML-KEM-768 keypair 25607 cycles 25547 cycles 1.00
ML-KEM-768 encaps 28177 cycles 28119 cycles 1.00
ML-KEM-768 decaps 37867 cycles 37839 cycles 1.00
ML-KEM-1024 keypair 35550 cycles 35583 cycles 1.00
ML-KEM-1024 encaps 40956 cycles 40970 cycles 1.00
ML-KEM-1024 decaps 53728 cycles 54481 cycles 0.99

This comment was automatically generated by workflow using github-action-benchmark.

Copy link

@oqs-bot oqs-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intel Xeon 4th gen (c7i) (no-opt)

Benchmark suite Current: 8bf3698 Previous: 169a6c3 Ratio
ML-KEM-512 keypair 34864 cycles 34805 cycles 1.00
ML-KEM-512 encaps 45010 cycles 44977 cycles 1.00
ML-KEM-512 decaps 58962 cycles 58905 cycles 1.00
ML-KEM-768 keypair 59130 cycles 59222 cycles 1.00
ML-KEM-768 encaps 71757 cycles 71873 cycles 1.00
ML-KEM-768 decaps 89238 cycles 89330 cycles 1.00
ML-KEM-1024 keypair 87591 cycles 87365 cycles 1.00
ML-KEM-1024 encaps 104638 cycles 104579 cycles 1.00
ML-KEM-1024 decaps 127572 cycles 127566 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

Copy link

@oqs-bot oqs-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arm Cortex-A55 (Snapdragon 888) benchmarks

Benchmark suite Current: 8bf3698 Previous: 169a6c3 Ratio
ML-KEM-512 keypair 58341 cycles 58368 cycles 1.00
ML-KEM-512 encaps 65750 cycles 65797 cycles 1.00
ML-KEM-512 decaps 84488 cycles 84531 cycles 1.00
ML-KEM-768 keypair 98984 cycles 98990 cycles 1.00
ML-KEM-768 encaps 110337 cycles 110464 cycles 1.00
ML-KEM-768 decaps 136838 cycles 136532 cycles 1.00
ML-KEM-1024 keypair 150233 cycles 150179 cycles 1.00
ML-KEM-1024 encaps 166349 cycles 166330 cycles 1.00
ML-KEM-1024 decaps 202290 cycles 202544 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

Copy link

@oqs-bot oqs-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graviton3

Benchmark suite Current: 8bf3698 Previous: 169a6c3 Ratio
ML-KEM-512 keypair 18960 cycles 18965 cycles 1.00
ML-KEM-512 encaps 23561 cycles 23553 cycles 1.00
ML-KEM-512 decaps 30674 cycles 30665 cycles 1.00
ML-KEM-768 keypair 32304 cycles 32294 cycles 1.00
ML-KEM-768 encaps 35880 cycles 35891 cycles 1.00
ML-KEM-768 decaps 46028 cycles 46046 cycles 1.00
ML-KEM-1024 keypair 46623 cycles 46629 cycles 1.00
ML-KEM-1024 encaps 52456 cycles 52444 cycles 1.00
ML-KEM-1024 decaps 66289 cycles 66247 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

Copy link

@oqs-bot oqs-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graviton2

Benchmark suite Current: 8bf3698 Previous: 169a6c3 Ratio
ML-KEM-512 keypair 29047 cycles 29021 cycles 1.00
ML-KEM-512 encaps 35361 cycles 35399 cycles 1.00
ML-KEM-512 decaps 45888 cycles 45885 cycles 1.00
ML-KEM-768 keypair 49397 cycles 49361 cycles 1.00
ML-KEM-768 encaps 55610 cycles 55579 cycles 1.00
ML-KEM-768 decaps 70375 cycles 70297 cycles 1.00
ML-KEM-1024 keypair 72120 cycles 72009 cycles 1.00
ML-KEM-1024 encaps 80925 cycles 80789 cycles 1.00
ML-KEM-1024 decaps 100794 cycles 100661 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

Copy link

@oqs-bot oqs-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AMD EPYC 4th gen (c7a) (no-opt)

Benchmark suite Current: 8bf3698 Previous: 169a6c3 Ratio
ML-KEM-512 keypair 45762 cycles 45731 cycles 1.00
ML-KEM-512 encaps 56885 cycles 56878 cycles 1.00
ML-KEM-512 decaps 76207 cycles 76270 cycles 1.00
ML-KEM-768 keypair 74520 cycles 74568 cycles 1.00
ML-KEM-768 encaps 88558 cycles 88637 cycles 1.00
ML-KEM-768 decaps 114472 cycles 114477 cycles 1.00
ML-KEM-1024 keypair 109353 cycles 109427 cycles 1.00
ML-KEM-1024 encaps 127268 cycles 127513 cycles 1.00
ML-KEM-1024 decaps 159963 cycles 160121 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

Copy link

@oqs-bot oqs-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intel Xeon 3rd gen (c6i) (no-opt)

Benchmark suite Current: 8bf3698 Previous: 169a6c3 Ratio
ML-KEM-512 keypair 56604 cycles 56624 cycles 1.00
ML-KEM-512 encaps 69425 cycles 69431 cycles 1.00
ML-KEM-512 decaps 91433 cycles 91368 cycles 1.00
ML-KEM-768 keypair 91902 cycles 91844 cycles 1.00
ML-KEM-768 encaps 107805 cycles 107860 cycles 1.00
ML-KEM-768 decaps 136372 cycles 136350 cycles 1.00
ML-KEM-1024 keypair 134763 cycles 134716 cycles 1.00
ML-KEM-1024 encaps 155314 cycles 155141 cycles 1.00
ML-KEM-1024 decaps 191683 cycles 191643 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

Copy link

@oqs-bot oqs-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AMD EPYC 3rd gen (c6a) (no-opt)

Benchmark suite Current: 8bf3698 Previous: 169a6c3 Ratio
ML-KEM-512 keypair 52401 cycles 52178 cycles 1.00
ML-KEM-512 encaps 65435 cycles 65771 cycles 0.99
ML-KEM-512 decaps 88510 cycles 88404 cycles 1.00
ML-KEM-768 keypair 84319 cycles 84745 cycles 0.99
ML-KEM-768 encaps 102031 cycles 101806 cycles 1.00
ML-KEM-768 decaps 131306 cycles 132081 cycles 0.99
ML-KEM-1024 keypair 124609 cycles 124039 cycles 1.00
ML-KEM-1024 encaps 145096 cycles 145723 cycles 1.00
ML-KEM-1024 decaps 182803 cycles 183725 cycles 0.99

This comment was automatically generated by workflow using github-action-benchmark.

Copy link

@oqs-bot oqs-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graviton3 (no-opt)

Benchmark suite Current: 8bf3698 Previous: 169a6c3 Ratio
ML-KEM-512 keypair 45397 cycles 45388 cycles 1.00
ML-KEM-512 encaps 54220 cycles 54215 cycles 1.00
ML-KEM-512 decaps 71155 cycles 71153 cycles 1.00
ML-KEM-768 keypair 74829 cycles 74828 cycles 1.00
ML-KEM-768 encaps 86075 cycles 86062 cycles 1.00
ML-KEM-768 decaps 108673 cycles 108808 cycles 1.00
ML-KEM-1024 keypair 111111 cycles 111124 cycles 1.00
ML-KEM-1024 encaps 125928 cycles 125924 cycles 1.00
ML-KEM-1024 decaps 154585 cycles 154614 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

Copy link

@oqs-bot oqs-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graviton2 (no-opt)

Benchmark suite Current: 8bf3698 Previous: 169a6c3 Ratio
ML-KEM-512 keypair 71315 cycles 71258 cycles 1.00
ML-KEM-512 encaps 85169 cycles 85064 cycles 1.00
ML-KEM-512 decaps 112672 cycles 112745 cycles 1.00
ML-KEM-768 keypair 117663 cycles 117665 cycles 1.00
ML-KEM-768 encaps 135356 cycles 135333 cycles 1.00
ML-KEM-768 decaps 172107 cycles 171948 cycles 1.00
ML-KEM-1024 keypair 174689 cycles 175207 cycles 1.00
ML-KEM-1024 encaps 196941 cycles 197231 cycles 1.00
ML-KEM-1024 decaps 243019 cycles 243461 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

Copy link

@oqs-bot oqs-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graviton4

Benchmark suite Current: 8bf3698 Previous: 169a6c3 Ratio
ML-KEM-512 keypair 18132 cycles 18141 cycles 1.00
ML-KEM-512 encaps 22146 cycles 22155 cycles 1.00
ML-KEM-512 decaps 28773 cycles 28789 cycles 1.00
ML-KEM-768 keypair 30566 cycles 30569 cycles 1.00
ML-KEM-768 encaps 33652 cycles 33637 cycles 1.00
ML-KEM-768 decaps 43172 cycles 43176 cycles 1.00
ML-KEM-1024 keypair 44218 cycles 44211 cycles 1.00
ML-KEM-1024 encaps 49662 cycles 49661 cycles 1.00
ML-KEM-1024 decaps 62640 cycles 62655 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

Copy link

@oqs-bot oqs-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graviton4 (no-opt)

Benchmark suite Current: 8bf3698 Previous: 169a6c3 Ratio
ML-KEM-512 keypair 42034 cycles 41978 cycles 1.00
ML-KEM-512 encaps 50091 cycles 50165 cycles 1.00
ML-KEM-512 decaps 66107 cycles 66057 cycles 1.00
ML-KEM-768 keypair 69151 cycles 69055 cycles 1.00
ML-KEM-768 encaps 79855 cycles 79759 cycles 1.00
ML-KEM-768 decaps 101120 cycles 101017 cycles 1.00
ML-KEM-1024 keypair 102207 cycles 102205 cycles 1.00
ML-KEM-1024 encaps 117201 cycles 117195 cycles 1.00
ML-KEM-1024 decaps 143472 cycles 143699 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

Copy link

@oqs-bot oqs-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bananapi bpi-f3 benchmarks

Benchmark suite Current: 8bf3698 Previous: 169a6c3 Ratio
ML-KEM-512 keypair 335030 cycles 335043 cycles 1.00
ML-KEM-512 encaps 445667 cycles 445693 cycles 1.00
ML-KEM-512 decaps 593678 cycles 593768 cycles 1.00
ML-KEM-768 keypair 556111 cycles 556100 cycles 1.00
ML-KEM-768 encaps 697923 cycles 697883 cycles 1.00
ML-KEM-768 decaps 889662 cycles 889605 cycles 1.00
ML-KEM-1024 keypair 821395 cycles 821199 cycles 1.00
ML-KEM-1024 encaps 998247 cycles 998090 cycles 1.00
ML-KEM-1024 decaps 1230467 cycles 1230182 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

Copy link

@oqs-bot oqs-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arm Cortex-A72 (Raspberry Pi 4) benchmarks

Benchmark suite Current: 8bf3698 Previous: 169a6c3 Ratio
ML-KEM-512 keypair 51475 cycles 52125 cycles 0.99
ML-KEM-512 encaps 58080 cycles 58732 cycles 0.99
ML-KEM-512 decaps 74564 cycles 74611 cycles 1.00
ML-KEM-768 keypair 88352 cycles 87684 cycles 1.01
ML-KEM-768 encaps 96502 cycles 96005 cycles 1.01
ML-KEM-768 decaps 120120 cycles 120180 cycles 1.00
ML-KEM-1024 keypair 131979 cycles 132237 cycles 1.00
ML-KEM-1024 encaps 144500 cycles 145035 cycles 1.00
ML-KEM-1024 decaps 175703 cycles 176330 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

@hanno-becker hanno-becker force-pushed the makefile_ext branch 2 times, most recently from ad654d9 to 74ae5bf Compare January 2, 2025 10:50
Copy link
Contributor

@mkannwischer mkannwischer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please rename run_defined back to check_defined.
Rest looks good for me. Thank you!

- Use `{func, kat, nistkat, ...}` to build test binaries.
- Use `run_{func, kat, nistkat, acvp}` to run and check test binaries.

No make targets have yet been introduced to run benchmarks.

Signed-off-by: Hanno Becker <[email protected]>
Previously, the `Makefile` exposed goals for building and running
tests for all security levels at once. For example, `make func`
would build functionality tests for ML-KEM-{512, 768, 1014}, and
`make run_func` would run them all.

This commit introduces level-specific phony targets
`make {,run_}xxx_{512,768,1024}` to target specific security
levels.

Signed-off-by: Hanno Becker <[email protected]>
Previously, the make targets for {func, kat, nistkat} were unsuitable
for running emulated tests since they would attempt to run the binary
natively, even when CROSS_PREFIX was passed for cross-compilation.

This commit allows emulated execution of cross-built binaries through
the environment variable EXEC_WRAPPER.

For readability, within the Makefile, the EXEC_WRAPPER variable is
abbreviated by W.

Signed-off-by: Hanno Becker <[email protected]>
This commit adds make targets `run_bench_{512,768,1024}` to run the
benchmark binaries; similar for bench_components. For the all-level
targets `run_bench` and `run_bench_components`, the `.WAIT` primitive
is used to ensure that benchmarks are not run in parallel.

Signed-off-by: Hanno Becker <[email protected]>
@hanno-becker
Copy link
Contributor Author

Please rename run_defined back to check_defined.
Rest looks good for me. Thank you!

done

Copy link
Contributor

@mkannwischer mkannwischer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@hanno-becker hanno-becker merged commit 67c264e into main Jan 2, 2025
104 checks passed
@hanno-becker hanno-becker deleted the makefile_ext branch January 2, 2025 11:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
benchmark this PR should be benchmarked in CI
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants