Skip to content

Commit

Permalink
Update benchmark results
Browse files Browse the repository at this point in the history
  • Loading branch information
@ferrolho
ferrolho committed May 7, 2023
1 parent b9ef1d6 commit 4a50b89
Show file tree
Hide file tree
Showing 3 changed files with 261 additions and 123 deletions.
105 changes: 60 additions & 45 deletions docs/src/benchmarks.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
# Benchmarks

To attain maximal performance, it is recommended to pass `-O3`, `--check-bounds=no` as command line flags to `julia`. As of Julia 1.1, maximizing performance for the `dynamics!` algorithm requires either setting the number of BLAS threads to 1 (`using LinearAlgebra; BLAS.set_num_threads(1)`) if using OpenBLAS (the default), or compiling Julia with MKL. See [this issue](https://github.com/JuliaRobotics/RigidBodyDynamics.jl/issues/500) for more information.
To attain maximal performance, it is recommended to pass `-O3` and `--check-bounds=no` as command line flags to `julia`.

> **Warning**
> For Julia versions previous to `v1.8`, maximizing performance for the `dynamics!` algorithm requires either setting the number of BLAS threads to 1 (`using LinearAlgebra; BLAS.set_num_threads(1)`) if using OpenBLAS (the default), or compiling Julia with MKL. See [this issue](https://github.com/JuliaRobotics/RigidBodyDynamics.jl/issues/500) for more information.
Run `perf/runbenchmarks.jl` to see benchmark results for the Atlas robot (v5). Results below are for the following scenarios:

Expand All @@ -9,69 +12,81 @@ Run `perf/runbenchmarks.jl` to see benchmark results for the Atlas robot (v5). R
3. Do inverse dynamics.
4. Do forward dynamics.

Note that results on CI builds are **not at all** representative because of code coverage. Results on a reasonably fast laptop at commit [870bea6](https://github.com/JuliaRobotics/RigidBodyDynamics.jl/commit/870bea668d5b11ce0555fa0552592d2c3cb15c54):
> **Note**
> Results on CI builds are **not at all** representative because of code coverage.
Output of `versioninfo()`:
Here are the results on an Apple MacBook Air (M1, 2020) (8GB RAM, 512GB SSD) at commit [b9ef1d](https://github.com/JuliaRobotics/RigidBodyDynamics.jl/commit/b9ef1d6974beff4d4fbe7dffc6dbfa65f71e0132):

Output of `versioninfo()`:
```
Julia Version 1.5.3
Commit 788b2c77c1 (2020-11-09 13:37 UTC)
Julia Version 1.8.5
Commit 17cfb8e65e* (2023-01-08 06:45 UTC)
Platform Info:
OS: macOS (x86_64-apple-darwin18.7.0)
CPU: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
OS: macOS (arm64-apple-darwin22.1.0)
CPU: 8 × Apple M1
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-9.0.1 (ORCJIT, skylake)
LLVM: libLLVM-13.0.1 (ORCJIT, apple-m1)
Threads: 1 on 4 virtual cores
```

Note that this is a different machine than the one that was used for earlier benchmarks.
> **Note**
> This is a different machine than the one that was used for earlier benchmarks.
Mass matrix:

```
memory estimate: 0 bytes
allocs estimate: 0
--------------
minimum time: 4.415 μs (0.00% GC)
median time: 4.579 μs (0.00% GC)
mean time: 4.916 μs (0.00% GC)
maximum time: 19.794 μs (0.00% GC)
Mass matrix ([`mass_matrix!`](@ref)):
```
BenchmarkTools.Trial: 10000 samples with 10 evaluations.
Range (min … max): 3.796 μs … 6.183 μs ┊ GC (min … max): 0.00% … 0.00%
Time (median): 3.858 μs ┊ GC (median): 0.00%
Time (mean ± σ): 3.879 μs ± 93.310 ns ┊ GC (mean ± σ): 0.00% ± 0.00%
Mass matrix and Jacobian from left hand to right foot:
▁▄▄▆▆██▇█▆▇▄▅▄▂▃▁▃▂▃▃▁▃▁▂ ▁ ▂
▅█████████████████████████████▇█▇█▇▆▇▅▆▅▆▅▅▆▅▆▄▇▆█▇▅▆▆▇▅▄▄ █
3.8 μs Histogram: log(frequency) by time 4.18 μs <
Memory estimate: 0 bytes, allocs estimate: 0.
```
memory estimate: 0 bytes
allocs estimate: 0
--------------
minimum time: 4.860 μs (0.00% GC)
median time: 4.982 μs (0.00% GC)
mean time: 5.399 μs (0.00% GC)
maximum time: 24.712 μs (0.00% GC)

Mass matrix ([`mass_matrix!`](@ref)) and Jacobian ([`geometric_jacobian!`](@ref)) from left hand to right foot:
```julia
BenchmarkTools.Trial: 10000 samples with 10 evaluations.
Range (min … max): 4.042 μs … 6.763 μs ┊ GC (min … max): 0.00% … 0.00%
Time (median): 4.167 μs ┊ GC (median): 0.00%
Time (mean ± σ): 4.187 μs ± 115.303 ns ┊ GC (mean ± σ): 0.00% ± 0.00%

▁▁▂▁▁ ▁▃▄▆▇██▇█▇▆▄▃▃▃▃▁▃▃▃▂▂▁▁ ▃
▃▅███████████████████████████████▇██▇██▆▇▆▄▆▅▆▇▇██▆██▆▆▅▁▅▄ █
4.04 μs Histogram: log(frequency) by time 4.5 μs <

Memory estimate: 0 bytes, allocs estimate: 0.
```

Note the low additional cost of computing a Jacobian when the mass matrix is already computed. This is because RigidBodyDynamics.jl caches intermediate computation results.

Inverse dynamics:
Inverse dynamics ([`inverse_dynamics!`](@ref)):
```julia
BenchmarkTools.Trial: 10000 samples with 10 evaluations.
Range (min … max): 2.700 μs … 5.700 μs ┊ GC (min … max): 0.00% … 0.00%
Time (median): 2.767 μs ┊ GC (median): 0.00%
Time (mean ± σ): 2.773 μs ± 79.491 ns ┊ GC (mean ± σ): 0.00% ± 0.00%

```
memory estimate: 0 bytes
allocs estimate: 0
--------------
minimum time: 4.256 μs (0.00% GC)
median time: 4.541 μs (0.00% GC)
mean time: 4.831 μs (0.00% GC)
maximum time: 21.625 μs (0.00% GC)
▁▁█
▂▂▂▂▂▂▃▅▅▆▇█████▇▆▅▄▃▄▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▁▂▂▁▂▂▂▁▂▂▂▂ ▃
2.7 μs Histogram: frequency by time 2.98 μs <

Memory estimate: 0 bytes, allocs estimate: 0.
```

Forward dynamics:
Forward dynamics ([`dynamics!`](@ref)):
```julia
BenchmarkTools.Trial: 10000 samples with 10 evaluations.
Range (min … max): 10.554 μs … 23.692 μs ┊ GC (min … max): 0.00% … 0.00%
Time (median): 10.667 μs ┊ GC (median): 0.00%
Time (mean ± σ): 10.703 μs ± 212.557 ns ┊ GC (mean ± σ): 0.00% ± 0.00%

```
memory estimate: 0 bytes
allocs estimate: 0
--------------
minimum time: 13.600 μs (0.00% GC)
median time: 14.419 μs (0.00% GC)
mean time: 16.071 μs (0.00% GC)
maximum time: 55.328 μs (0.00% GC)
▃▇█▇▅▂
▁▁▂▂▄███████▇▄▃▂▂▂▁▁▁▁▁▁▁▁▁▁▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ ▂
10.6 μs Histogram: frequency by time 11.3 μs <

Memory estimate: 0 bytes, allocs estimate: 0.
```
Loading

0 comments on commit 4a50b89

Please sign in to comment.