Find a way to make it meaningful per release

[helio-frota]

Similar to cargo llvm-cov test that shows coverage, it would be good to have a result for each release.

not exactly this but something like:

v0.1.0-alpha.11 - slower
v0.1.0-alpha.10 - faster
v0.1.0-alpha.9   - same as previous 

or

v0.1.0-alpha.11 - 9k requests
v0.1.0-alpha.10 - 12k requests
v0.1.0-alpha.9   - 10k requests

[helio-frota]

I just noticed hyperfoil already have a compare command, we don't need to bother with cat/cut/awk to compare the columns 🎉

[hyperfoil@in-vm]$ compare 0000 0001
Comparing runs 0000 and 0001
PHASE   METRIC                REQUESTS     MEAN                 p50                  p90                  p99                  p99.9                p99.99
basic   fetchSboms             +0(+0.00%)   -4.59 ms(-532.78%)   -4.61 ms(-533.18%)   -4.61 ms(-533.18%)   -4.61 ms(-533.18%)   -4.61 ms(-533.18%)   -4.61 ms(-533.18%)
rampUp  fetchAdvisories       +75(+2.50%)  -292.93 μs(-23.17%)  -212.99 μs(-15.57%)  -532.48 μs(-32.02%)  -950.27 μs(-47.93%)    -1.74 ms(-68.83%)   -8.70 ms(-293.37%)
rampUp  fetchImporters        +75(+2.50%)  -265.24 μs(-21.10%)  -196.61 μs(-14.37%)  -409.60 μs(-24.88%)  -835.58 μs(-42.86%)    -1.46 ms(-58.17%)    -1.39 ms(-44.97%)
rampUp  fetchSboms            +75(+2.50%)  -228.83 μs(-15.66%)  -172.03 μs(-10.99%)  -294.91 μs(-15.79%)  -770.05 μs(-34.81%)   -4.24 ms(-125.12%)   -8.19 ms(-211.86%)
rampUp  fetchVulnerabilities  +75(+2.50%)  -262.34 μs(-20.16%)  -196.61 μs(-14.12%)  -491.52 μs(-29.13%)  -876.54 μs(-42.29%)  -786.43 μs(-24.74%)    +4.62 ms(+89.81%)
[hyperfoil@in-vm]$ compare 0001 0002
Comparing runs 0001 and 0002
PHASE   METRIC                REQUESTS     MEAN                 p50                  p90                  p99                  p99.9                p99.99
basic   fetchSboms             +0(+0.00%)  +403.46 μs(+46.79%)  +405.50 μs(+46.92%)  +405.50 μs(+46.92%)  +405.50 μs(+46.92%)  +405.50 μs(+46.92%)  +405.50 μs(+46.92%)
rampUp  fetchAdvisories       -33(-1.09%)    +65.24 μs(+5.16%)    +16.38 μs(+1.20%)     -8.19 μs(-0.50%)    -24.58 μs(-1.26%)   +180.22 μs(+7.14%)  +573.44 μs(+19.34%)
rampUp  fetchImporters        -33(-1.09%)    +58.15 μs(+4.62%)        +0 ns(+0.00%)     -8.19 μs(-0.50%)    +16.38 μs(+0.84%)   +114.69 μs(+4.58%)  -393.22 μs(-14.55%)
rampUp  fetchSboms            -33(-1.09%)    +56.61 μs(+3.87%)    +32.77 μs(+2.09%)    -16.38 μs(-0.88%)        +0 ns(+0.00%)    -98.30 μs(-2.99%)   +7.67 ms(+198.31%)
rampUp  fetchVulnerabilities  -33(-1.09%)    +62.15 μs(+4.78%)    +24.58 μs(+1.76%)     +8.19 μs(+0.49%)    -32.77 μs(-1.61%)   -212.99 μs(-7.18%)   -5.51 ms(-129.23%)

[helio-frota]

thanks to hyperfoil compare and the podman version : )
https://github.com/trustification/scale-testing/tree/main/benchmarks#comparing-trustify-pm-mode-releases

[helio-frota]

#24 is related