Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

setup benchmark with BenchmarkTools #123

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

setup benchmark with BenchmarkTools #123

wants to merge 2 commits into from

Conversation

johnnychen94
Copy link
Member

@johnnychen94 johnnychen94 commented Mar 8, 2020 •
edited
Loading

This PR is to set up an extensible benchmark project for ImageCore. So except for the existing channelview and colorview benchmarks, I didn't add more trails here.

I'm not sure what the best practice wrt benchmark purpose is, so I'll play with it for quite a long time until I find myself satisfied.

Todo:

  • simplify codes
  • generate nice-looking report
  • tune benchmark parameters
  • set up CI

I would actually pause this a while, just open it in case some of you might want to build on top of it.

@codecov
Copy link

codecov bot commented Mar 8, 2020 •
edited
Loading

Codecov Report

Merging #123 into master will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master     #123   +/-   ##
=======================================
  Coverage   74.70%   74.70%           
=======================================
  Files           9        9           
  Lines         427      427           
=======================================
  Hits          319      319           
  Misses        108      108

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5e7c3eb...1c42d0e. Read the comment docs.

@johnnychen94 johnnychen94 mentioned this pull request Mar 10, 2020
Open
@johnnychen94
Copy link
Member Author

johnnychen94 commented Mar 10, 2020 •
edited
Loading

I don't have much time to organize and analyze the benchmark result this week, so I'll upload the raw data if any of you feel interested in it.

Benchmark results 
Julia Version 1.4.0-rc2.0
Commit b99ed72c95 (2020-02-24 16:51 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin19.0.0)
  CPU: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, skylake)

imagecore/baseline ratio (median)

>100% means channelview is slower than a plain array

2-element BenchmarkTools.BenchmarkGroup:
  tags: []
  "channelview" => 2-element BenchmarkTools.BenchmarkGroup:
	  tags: ["views"]
	  "getindex" => 4-element BenchmarkTools.BenchmarkGroup:
		  tags: ["index"]
		  "elt_inbounds" => 5-element BenchmarkTools.BenchmarkGroup:
			  tags: []
			  Gray{Normed{UInt8,8}} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(1731.72%)
				  (1024, 1024) => TrialRatio(2162.58%)
			  Gray{Float32} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(1853.36%)
				  (1024, 1024) => TrialRatio(2507.17%)
			  RGB{Normed{UInt8,8}} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["RGB"]
				  (128, 128) => TrialRatio(2505.88%)
				  (1024, 1024) => TrialRatio(3137.31%)
			  RGB{Float32} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["RGB"]
				  (128, 128) => TrialRatio(2199.22%)
				  (1024, 1024) => TrialRatio(3141.82%)
			  Gray{Bool} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(363.31%)
				  (1024, 1024) => TrialRatio(365.20%)
		  "index_inbounds_simd" => 5-element BenchmarkTools.BenchmarkGroup:
			  tags: []
			  Gray{Normed{UInt8,8}} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(4288.34%)
				  (1024, 1024) => TrialRatio(4517.23%)
			  Gray{Float32} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(1723.06%)
				  (1024, 1024) => TrialRatio(1828.97%)
			  RGB{Normed{UInt8,8}} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["RGB"]
				  (128, 128) => TrialRatio(4622.86%)
				  (1024, 1024) => TrialRatio(6737.18%)
			  RGB{Float32} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["RGB"]
				  (128, 128) => TrialRatio(2018.99%)
				  (1024, 1024) => TrialRatio(2667.40%)
			  Gray{Bool} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(359.07%)
				  (1024, 1024) => TrialRatio(377.37%)
		  "elt_boundscheck" => 5-element BenchmarkTools.BenchmarkGroup:
			  tags: []
			  Gray{Normed{UInt8,8}} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(2260.17%)
				  (1024, 1024) => TrialRatio(2474.02%)
			  Gray{Float32} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(2040.78%)
				  (1024, 1024) => TrialRatio(2771.95%)
			  RGB{Normed{UInt8,8}} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["RGB"]
				  (128, 128) => TrialRatio(2239.88%)
				  (1024, 1024) => TrialRatio(3107.33%)
			  RGB{Float32} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["RGB"]
				  (128, 128) => TrialRatio(2142.89%)
				  (1024, 1024) => TrialRatio(3295.88%)
			  Gray{Bool} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(390.33%)
				  (1024, 1024) => TrialRatio(384.83%)
		  "index_boundscheck" => 5-element BenchmarkTools.BenchmarkGroup:
			  tags: []
			  Gray{Normed{UInt8,8}} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(3214.27%)
				  (1024, 1024) => TrialRatio(4718.90%)
			  Gray{Float32} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(2310.17%)
				  (1024, 1024) => TrialRatio(2458.24%)
			  RGB{Normed{UInt8,8}} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["RGB"]
				  (128, 128) => TrialRatio(3144.02%)
				  (1024, 1024) => TrialRatio(6199.72%)
			  RGB{Float32} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["RGB"]
				  (128, 128) => TrialRatio(2689.97%)
				  (1024, 1024) => TrialRatio(3266.19%)
			  Gray{Bool} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(363.28%)
				  (1024, 1024) => TrialRatio(375.66%)
	  "setindex" => 2-element BenchmarkTools.BenchmarkGroup:
		  tags: ["index"]
		  "index_inbounds_simd" => 5-element BenchmarkTools.BenchmarkGroup:
			  tags: []
			  Gray{Normed{UInt8,8}} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(4242.09%)
				  (1024, 1024) => TrialRatio(4556.47%)
			  Gray{Float32} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(3605.06%)
				  (1024, 1024) => TrialRatio(6118.86%)
			  RGB{Normed{UInt8,8}} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["RGB"]
				  (128, 128) => TrialRatio(3681.32%)
				  (1024, 1024) => TrialRatio(5944.58%)
			  RGB{Float32} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["RGB"]
				  (128, 128) => TrialRatio(3953.12%)
				  (1024, 1024) => TrialRatio(8742.37%)
			  Gray{Bool} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(5758.51%)
				  (1024, 1024) => TrialRatio(5799.99%)
		  "index_boundscheck" => 5-element BenchmarkTools.BenchmarkGroup:
			  tags: []
			  Gray{Normed{UInt8,8}} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(3415.77%)
				  (1024, 1024) => TrialRatio(4324.57%)
			  Gray{Float32} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(4648.61%)
				  (1024, 1024) => TrialRatio(5060.75%)
			  RGB{Normed{UInt8,8}} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["RGB"]
				  (128, 128) => TrialRatio(3554.73%)
				  (1024, 1024) => TrialRatio(5907.18%)
			  RGB{Float32} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["RGB"]
				  (128, 128) => TrialRatio(8423.11%)
				  (1024, 1024) => TrialRatio(10012.53%)
			  Gray{Bool} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(3328.97%)
				  (1024, 1024) => TrialRatio(3469.25%)
  "colorview" => 2-element BenchmarkTools.BenchmarkGroup:
	  tags: ["views"]
	  "setindex" => 2-element BenchmarkTools.BenchmarkGroup:
		  tags: ["index"]
		  "index_inbounds_simd" => 5-element BenchmarkTools.BenchmarkGroup:
			  tags: []
			  Gray{Normed{UInt8,8}} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(4138.62%)
				  (1024, 1024) => TrialRatio(4258.32%)
			  Gray{Float32} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(2056.51%)
				  (1024, 1024) => TrialRatio(2346.55%)
			  RGB{Normed{UInt8,8}} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["RGB"]
				  (128, 128) => TrialRatio(1678.30%)
				  (1024, 1024) => TrialRatio(1945.87%)
			  RGB{Float32} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["RGB"]
				  (128, 128) => TrialRatio(2079.43%)
				  (1024, 1024) => TrialRatio(1994.03%)
			  Gray{Bool} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(3339.60%)
				  (1024, 1024) => TrialRatio(4109.08%)
		  "index_boundscheck" => 5-element BenchmarkTools.BenchmarkGroup:
			  tags: []
			  Gray{Normed{UInt8,8}} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(3468.02%)
				  (1024, 1024) => TrialRatio(4346.89%)
			  Gray{Float32} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(4527.61%)
				  (1024, 1024) => TrialRatio(6239.39%)
			  RGB{Normed{UInt8,8}} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["RGB"]
				  (128, 128) => TrialRatio(1889.68%)
				  (1024, 1024) => TrialRatio(1921.28%)
			  RGB{Float32} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["RGB"]
				  (128, 128) => TrialRatio(2240.99%)
				  (1024, 1024) => TrialRatio(2476.97%)
			  Gray{Bool} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(4187.13%)
				  (1024, 1024) => TrialRatio(4412.78%)
	  "getindex" => 4-element BenchmarkTools.BenchmarkGroup:
		  tags: ["index"]
		  "elt_inbounds" => 5-element BenchmarkTools.BenchmarkGroup:
			  tags: []
			  Gray{Normed{UInt8,8}} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(2073.77%)
				  (1024, 1024) => TrialRatio(2459.38%)
			  Gray{Float32} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(2018.56%)
				  (1024, 1024) => TrialRatio(2146.11%)
			  RGB{Normed{UInt8,8}} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["RGB"]
				  (128, 128) => TrialRatio(1847.16%)
				  (1024, 1024) => TrialRatio(1293.57%)
			  RGB{Float32} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["RGB"]
				  (128, 128) => TrialRatio(2236.21%)
				  (1024, 1024) => TrialRatio(2137.61%)
			  Gray{Bool} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(153.26%)
				  (1024, 1024) => TrialRatio(155.90%)
		  "index_inbounds_simd" => 5-element BenchmarkTools.BenchmarkGroup:
			  tags: []
			  Gray{Normed{UInt8,8}} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(4971.23%)
				  (1024, 1024) => TrialRatio(4919.05%)
			  Gray{Float32} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(1872.94%)
				  (1024, 1024) => TrialRatio(2073.38%)
			  RGB{Normed{UInt8,8}} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["RGB"]
				  (128, 128) => TrialRatio(4398.30%)
				  (1024, 1024) => TrialRatio(4593.16%)
			  RGB{Float32} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["RGB"]
				  (128, 128) => TrialRatio(2673.97%)
				  (1024, 1024) => TrialRatio(2528.93%)
			  Gray{Bool} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(153.12%)
				  (1024, 1024) => TrialRatio(166.96%)
		  "elt_boundscheck" => 5-element BenchmarkTools.BenchmarkGroup:
			  tags: []
			  Gray{Normed{UInt8,8}} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(1410.12%)
				  (1024, 1024) => TrialRatio(1413.61%)
			  Gray{Float32} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(1906.74%)
				  (1024, 1024) => TrialRatio(1999.44%)
			  RGB{Normed{UInt8,8}} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["RGB"]
				  (128, 128) => TrialRatio(1645.99%)
				  (1024, 1024) => TrialRatio(1184.99%)
			  RGB{Float32} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["RGB"]
				  (128, 128) => TrialRatio(1830.62%)
				  (1024, 1024) => TrialRatio(1605.76%)
			  Gray{Bool} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(128.51%)
				  (1024, 1024) => TrialRatio(130.85%)
		  "index_boundscheck" => 5-element BenchmarkTools.BenchmarkGroup:
			  tags: []
			  Gray{Normed{UInt8,8}} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(4702.50%)
				  (1024, 1024) => TrialRatio(4969.81%)
			  Gray{Float32} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(2351.08%)
				  (1024, 1024) => TrialRatio(2098.17%)
			  RGB{Normed{UInt8,8}} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["RGB"]
				  (128, 128) => TrialRatio(3123.88%)
				  (1024, 1024) => TrialRatio(3590.13%)
			  RGB{Float32} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["RGB"]
				  (128, 128) => TrialRatio(2264.27%)
				  (1024, 1024) => TrialRatio(2178.61%)
			  Gray{Bool} => 2-element BenchmarkTools.BenchmarkGroup:
				  tags: ["Gray"]
				  (128, 128) => TrialRatio(138.64%)
				  (1024, 1024) => TrialRatio(154.85%)

@timholy
Copy link
Member

timholy commented Mar 12, 2020

It will be wonderful to have the benchmarks in a format that can integrate with CI, thanks for tackling this.

Speaking of pausing for infrastructure development, I believe that when I first put together test/benchmarks.jl,

ImageCore.jl/test/benchmarks.jl

Lines 49 to 53 in 73a2f1b

# Rather than using BenchmarkTools (and thus run one test repeatedly,
# accumulating timings), we run the same test interleaving the two
# array types. This is designed to reduce the risk of spurious
# failure, particularly on shared machines like Travis where they may
# get "distracted" by other tasks
turned out to be a significant issue: there was a ton of "noise" in Travis-run benchmark results. While I don't see a good way to address this when comparing a PR vs master, the strategy taken here for comparing op1 vs op2 (two different operations that are, in a sense, "the same" but perhaps running on different types) seems worthwhile. Perhaps the time has come to add a @benchmarkable_cycle op1 op2 ... to BenchmarkTools?

@timholy timholy mentioned this pull request Oct 7, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
no changelog run benchmark
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@johnnychen94 @timholy