Can engineers force memory reporting into one consistent scale unit (K or M)?

[vanvoorden]

Hi! I just started using the package and I like it a lot! I am running two parallel benchmarks with grouping metric to compare the memory usage between the two. I am seeing results like this:

Memory (resident peak)
╒════════════════════════════════════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Test                                                   │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞════════════════════════════════════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ First (K)                                              │    8913 │   10273 │   10273 │   10273 │   10273 │   10273 │   10273 │     100 │
├────────────────────────────────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Second (M)                                             │      10 │      10 │      10 │      10 │      10 │      10 │      10 │     100 │
╘════════════════════════════════════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

It's a little weird because one metric is reporting as 10273K and the other metric is reporting as 10M… do I have any option to pass a flag (similar to timeUnits) that can force that reporting to one unit so I can compare them a little easier? Thanks!

[hassila]

Hmm, I seem to remember we wanted to force the units to be the same and actually had an option for that / but couldn't see it now.

Would make sense to have the option - in the meantime you can try exploring some of the other output options here: https://swiftpackageindex.com/ordo-one/package-benchmark/1.22.4/documentation/benchmark/exportingbenchmarks

I seem to remember that you could compare two runs in jmh?

Away for a couple of weeks now and don't have access to a machine at the moment - so apologies for a bit fuzzy answers.

[vanvoorden]

@hassila Thanks!

[dabrahams]

Do this for all the measurements by default please in the textual format. Otherwise it's far too easy to misinterpret the results by 3 or more orders of magnitude! I have literally spent hours optimizing the wrong things because I overlooked a difference in units.

If you are not inclined to make it the default everywhere, please consider doing it when grouping by metric.

Thanks

[hassila]

I would be happy to change the default when grouping per metric, that makes sense (as that is typically for comparisons anyway).

[dabrahams]

That's wonderful and I don't want to push my luck, but… wouldn't consistency across the formats be better? People are still liable to compare regardless of grouping. How much value is there in using different units in the other format?

[hassila]

That's wonderful and I don't want to push my luck, but… wouldn't consistency across the formats be better? People are still liable to compare regardless of grouping. How much value is there in using different units in the other format?

Just to clarify, are you talking about memory metrics only?

[dabrahams]

All metrics. My particular interest wasn't in the memory use, in fact I was comparing time.

[hassila]

Ok, then it sounds like you just measure variations of the same thing in your benchmarks in the same suite, which is a valid use case and then it would make sense.

But we extensively have also different things being measured in the same suite (related to a specific subsystem), e.g.:

• Benchmark for measuring cost of opening a database
• Benchmark for traversing a specific dataset with a bulk cursor
• Benchmark for writing a specific dataset
• Benchmark for reading a single item with direct indexing
• Benchmark for extract database metadata and statistics

These have wildly different values for their metrics, are not comparable and makes no sense to compare between them. Forcing everything to the same metric would make measurements unusable for the comparison.

This is one reason why we have grouping per benchmark (where comparisons makes no sense), and grouping per metric (when the benchmarks should be compared).

It seems it make much sense to enforce the same units for the second use case (grouping per metric), as that is typically used for benchmarks which are variations of the same work that you want to compare.

It makes less sense for the per-benchmark grouping of metrics, as it would remove all useful information for the faster running benchmarks and all values would revert to the worse performing benchmarks magnitude.

I would suggest the following changes:

• Normalize the display unit when grouping by metric to make comparisons robust for the use case when you compare benchmarks that are variation on the same workload
• Keep the output adopted to the magnitude of the output when grouping per benchmark (which would be for different workload)
• Add an option to override and choose a specific magnitude for displaying memory usage across all benchmarks (basically disabling the automatic choice)
• Add an option to override and choose a specific magnitude for displaying time usage
• and finally an option for overriding magnitude also for counts (ARC, custom metrics, ...)

Do you agree and see the rationale?

[dabrahams]

SGTM, and thank you for your patient explanation!

[hassila]

Slightly different solution, but you can now override all units per metric individually as part of:
https://github.com/ordo-one/package-benchmark/releases/tag/1.28.0

it is also possible to force time units from command line specifically.

Please give it a try and open new issues if needed.