criterion performance measurements
+ +overview
+ +want to understand this report?
+ + + +initial position/depth: 2 White
++ | + + |
+ | lower bound | +estimate | +upper bound | + + +
---|---|---|---|
OLS regression | +xxx | +xxx | +xxx | +
R² goodness-of-fit | +xxx | +xxx | +xxx | +
Mean execution time | +7.184156479601918e-4 | +7.248523730957002e-4 | +7.324370498371348e-4 | +
Standard deviation | +1.8639691512974905e-5 | +2.340841664911922e-5 | +3.0546112174050066e-5 | +
Outlying measurements have moderate + (0.23490619993468234%) + effect on estimated standard deviation.
+ +initial position/depth: 5 White
++ | + + |
+ | lower bound | +estimate | +upper bound | + + +
---|---|---|---|
OLS regression | +xxx | +xxx | +xxx | +
R² goodness-of-fit | +xxx | +xxx | +xxx | +
Mean execution time | +0.46041383887512666 | +0.47976388352083177 | +0.4945598152916091 | +
Standard deviation | +1.1613252334666077e-2 | +2.0656366662567663e-2 | +2.8647718665592365e-2 | +
Outlying measurements have moderate + (0.1875%) + effect on estimated standard deviation.
+ +initial position/depth: 6 White
++ | + + |
+ | lower bound | +estimate | +upper bound | + + +
---|---|---|---|
OLS regression | +xxx | +xxx | +xxx | +
R² goodness-of-fit | +xxx | +xxx | +xxx | +
Mean execution time | +4.084563782000032 | +4.153693016624904 | +4.184491083624825 | +
Standard deviation | +1.933709899988889e-2 | +5.144117353665631e-2 | +7.11250890916982e-2 | +
Outlying measurements have moderate + (0.1875%) + effect on estimated standard deviation.
+ + +understanding this report
+ +In this report, each function benchmarked by criterion is assigned + a section of its own. The charts in each section are active; if + you hover your mouse over data points and annotations, you will see + more details.
+ +-
+
- The chart on the left is a + kernel + density estimate (also known as a KDE) of time + measurements. This graphs the probability of any given time + measurement occurring. A spike indicates that a measurement of a + particular time occurred; its height indicates how often that + measurement was repeated. + +
- The chart on the right is the raw data from which the kernel + density estimate is built. The x axis indicates the + number of loop iterations, while the y axis shows measured + execution time for the given number of loop iterations. The + line behind the values is the linear regression prediction of + execution time for a given number of iterations. Ideally, all + measurements will be on (or very near) this line. +
Under the charts is a small table. + The first two rows are the results of a linear regression run + on the measurements displayed in the right-hand chart.
+ +-
+
- OLS regression indicates the + time estimated for a single loop iteration using an ordinary + least-squares regression model. This number is more accurate + than the mean estimate below it, as it more effectively + eliminates measurement overhead and other constant factors. +
- R² goodness-of-fit is a measure of how + accurately the linear regression model fits the observed + measurements. If the measurements are not too noisy, R² + should lie between 0.99 and 1, indicating an excellent fit. If + the number is below 0.99, something is confounding the accuracy + of the linear model. +
- Mean execution time and standard deviation are + statistics calculated from execution time + divided by number of iterations. +
We use a statistical technique called + the bootstrap + to provide confidence intervals on our estimates. The + bootstrap-derived upper and lower bounds on estimates let you see + how accurate we believe those estimates to be. (Hover the mouse + over the table headers to see the confidence levels.)
+ +A noisy benchmarking environment can cause some or many + measurements to fall far from the mean. These outlying + measurements can have a significant inflationary effect on the + estimate of the standard deviation. We calculate and display an + estimate of the extent to which the standard deviation has been + inflated by outliers.
+ + + +