Standardized benchmark platforms

[stevefan1999-personal]

I saw that the benchmarks in dotnet blog are using arbitrary processors that are largely biased towards laptop CPUs and x86. Because we are moving towards cross-platform support, gains in one specific platform may not reflect well on another (for example, it maybe a significant gain on x86 while negative on arm64), especially when you consider the difference in micro-architectures, like big.little, SMT, cache setup (especially with AMD CPU who has massive L3 cache on some of the consumer CPUs and server CPUs)

In order to have a fair benchmark, we benchmark the following stable targets:

1. mainstream iOS device targets. This may mean up to 2 to 3 generations of iPhone and iPad devices.
2. Apple Silicons, which means M1 and M2 CPUs. iPad also overlapped on this category, for example, my iPad Pro uses an M1 processor
3. Snapdragon compute platform like 8cx and Microsoft SQ2. This is essentially benchmarking for Project Volterra/Windows Dev Kit and Surface Pro X devices
4. mainstream x86 category, like Intel Xeon/Intel Core/AMD EPYC/AMD Threadripper CPUs up to 2 to 3 generations. We have to disable hyperthreading and better to run under bare-metal as well.

The matrix can further subcategorize into operating systems like Windows, MacOS, FreeBSD and Linux.

Anyway, all I want to say is using i9-10885H, a laptop CPU with a specific focus on battery life, is not suitable as a reference is not well rounded either.

Having a "note" here is not enough. We should test more architectures in order to announce a real gain (or at the very least clarify with a caveat notice in the blog post, that this performance gains are available on a specific platform with a specific setup and need to take it as a grain of salt).

[huoyaoyuan]

Anyway, all I want to say is using i9-10885H, a laptop CPU with a specific focus on battery life

Note that 45W laptop processors are more optimized for limited power and thermal environment. It is able to consistently run one core at maximum performance.

Having a "note" here is not enough. We should test more architectures in order to announce a real gain

The performance improvements generally contain two categories: cross-platform and platform-specific. The cross-platform improvements are typically done in managed code and shared JIT transform. The hardware really doesn't matter for these types of improvements. The precise number may vary among platforms, but the blogs are meant to briefly introduce the improvements.

[huoyaoyuan]

There are monitoring for performance on a large variety of hardwares, but it would be too verbose to include all the numbers. There are also blog posts about platform-specific improvements like ARM64.

[En3Tho]

Technically if you ensure frequencies are locked (for example you can disable turbo boost and just run something like 2.7 stable) then your benchmark will be quite stable.

Whether this should be done for a small showcase on a blog post - I'm not sure because overall benchmark numbers will be quite high and throw a "slowness" shadow.

As for repo itself - it is already using a plethora of different cpus and architectures for benchmarking coming from the cloud environment. I'm pretty sure those are tuned to be stable for these purposes.