Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add benchmarks #169

Merged
merged 11 commits into from
Oct 8, 2024
Merged

Add benchmarks #169

merged 11 commits into from
Oct 8, 2024

Conversation

mingxwa
Copy link
Collaborator

@mingxwa mingxwa commented Oct 2, 2024

Changes

  • Added another folder benchmarks to benchmark the library in various scenarios.
  • Added benchmark for invocation via proxy and virtual functions on small/large objects.
  • Added benchmark for basic lifetime management for proxy, std::unique_ptr, std::shared_ptr and std::any for small/large objects.
  • Changed the build flavor on the pipeline from "Debug" to "Release" to facilitate benchmarking.
  • Due to Warnings in release build #170, removed "treat warning as error" compiler flags.

Observed from the CI build:

MSVC on Windows Server 2022 (x64) GCC on Ubuntu 24.04 (x64) Clang on Ubuntu 24.04 (x64) Apple Clang on macOS 14 (ARM64)
Indirect invocation on small objects via proxy vs. virtual functions 🟩 proxy is about 254.1% faster 🟩 proxy is about 31.8% faster 🟩 proxy is about 38.0% faster 🟩 proxy is about 3.5% faster
Indirect invocation on large objects via proxy vs. virtual functions 🟩 proxy is about 269.9% faster 🟩 proxy is about 28.3% faster 🟩 proxy is about 17.4% faster 🟩 proxy is about 2.5% faster
Basic lifetime management for small objects with proxy vs. std::unique_ptr 🟩 proxy is about 375.4% faster 🟩 proxy is about 369.8% faster 🟩 proxy is about 347.1% faster 🟩 proxy is about 254.6% faster
Basic lifetime management for small objects with proxy vs. std::shared_ptr (without memory pool) 🟩 proxy is about 494.4% faster 🟩 proxy is about 894.4% faster 🟩 proxy is about 847.2% faster 🟩 proxy is about 355.6% faster
Basic lifetime management for small objects with proxy vs. std::shared_ptr (with memory pool) 🟩 proxy is about 246.5% faster 🟩 proxy is about 589.5% faster 🟩 proxy is about 566.0% faster 🟩 proxy is about 157.3% faster
Basic lifetime management for small objects with proxy vs. std::any 🟩 proxy is about 61.6% faster 🟩 proxy is about 235.7% faster 🟩 proxy is about 222.5% faster 🟩 proxy is about 20.0% faster
Basic lifetime management for large objects with proxy (without memory pool) vs. std::unique_ptr 🟥 proxy is about 0.5% slower 🟥 proxy is about 2.7% slower 🟩 proxy is about 7.4% faster 🟥 proxy is about 8.0% slower
Basic lifetime management for large objects with proxy (with memory pool) vs. std::unique_ptr 🟩 proxy is about 117.4% faster 🟩 proxy is about 113.6% faster 🟩 proxy is about 161.1% faster 🟩 proxy is about 88.5% faster
Basic lifetime management for large objects with proxy vs. std::shared_ptr (both without memory pool) 🟩 proxy is about 12.1% faster 🟩 proxy is about 5.9% faster 🟩 proxy is about 16.0% faster 🟩 proxy is about 11.5% faster
Basic lifetime management for large objects with proxy vs. std::shared_ptr (both with memory pool) 🟩 proxy is about 8.7% faster 🟩 proxy is about 16.4% faster 🟩 proxy is about 8.6% faster 🟩 proxy is about 46.8% faster
Basic lifetime management for large objects with proxy (without memory pool) vs. std::any 🟩 proxy is about 33.6% faster 🟥 proxy is about 6.2% slower 🟩 proxy is about 5.5% faster 🟩 proxy is about 3.7% faster
Basic lifetime management for large objects with proxy (with memory pool) vs. std::any 🟩 proxy is about 192.1% faster 🟩 proxy is about 105.9% faster 🟩 proxy is about 156.4% faster 🟩 proxy is about 112.5% faster

Notes

  • Invocation benchmarks are performed against 1,000,000 objects of 100 types. Specifically,
    • A small object in the invocation benchmarks contains a single int field.
    • A large object in the invocation benchmarks contains 15 pointers (void*) and a single int field.
  • Lifetime management benchmarks are performed against 120,000 objects. Specifically,
    • 3 types of small objects are used for testing: int, std::shared_ptr<int> and std::unique_lock<std::mutex>.
    • std::array<char, 100>, std::array<std::string, 3> and std::unique_lock<std::mutex> padded with void*[15] are used as large object.
    • For benchmarks involving memory pool, std::pmr::unsynchronized_pool_resource is used.

@mingxwa mingxwa force-pushed the user/mingxwa/benchmarks branch 2 times, most recently from 5100105 to 5de03cf Compare October 2, 2024 07:40
@mingxwa mingxwa requested review from tian-lt and guominrui October 2, 2024 08:41
@mingxwa mingxwa marked this pull request as ready for review October 2, 2024 08:42
@mingxwa mingxwa force-pushed the user/mingxwa/benchmarks branch from 5de03cf to 767a58e Compare October 4, 2024 05:46
Copy link
Collaborator

@tian-lt tian-lt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mingxwa mingxwa merged commit 0e64a55 into microsoft:main Oct 8, 2024
5 checks passed
@mingxwa mingxwa deleted the user/mingxwa/benchmarks branch October 8, 2024 07:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants