Skip to content

Commit

Permalink
[Doc] Adding benchmark documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
loumalouomega authored Dec 6, 2024
1 parent 6ee2022 commit 6974c9b
Show file tree
Hide file tree
Showing 2 changed files with 187 additions and 0 deletions.
187 changes: 187 additions & 0 deletions docs/pages/Kratos/For_Developers/General/Benchmarking.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,187 @@
---
title: Benchmarking in Kratos Multiphysics
keywords: Benchmarking, Google Benchmark, Performance Measurement
tags: [Benchmarking]
sidebar: kratos_for_developers
summary: Step-by-step guide on adding and executing benchmarks in Kratos Multiphysics using Google Benchmark.
---

## Introduction

Benchmarking is now supported in **Kratos Multiphysics**, enabling developers to measure and analyze the performance of their code effectively. The system integrates seamlessly with [Google Benchmark](https://github.com/google/benchmark), which is included as an external dependency, similar to the **GTest** suite.

### Key Features:
- Automatic generation of a binary per benchmark source file located in the `benchmark` directory.
- Benchmark executables are compiled to the binary `benchmark` directory.
- Activated using the `KRATOS_BUILD_BENCHMARK` CMake option (disabled by default).

### What is Google Benchmark?

[Google Benchmark](https://github.com/google/benchmark) is a library developed by *Google* for microbenchmarking C++ code. It provides:

- Precision and reproducibility in measurements.
- Built-in support for statistical analysis of timing data.
- Customizable inputs to benchmarks for more flexible scenarios.
- Support for benchmarking in multi-threaded and single-threaded environments.

## Writing Benchmarks
### Overview of Google Benchmark API
A *Google Benchmark* consists of a **function** that performs repeated measurements of a specific code snippet. The framework automatically manages:
- Warm-up runs to stabilize performance.
- Statistical aggregation of results (mean, median, standard deviation, etc.).
- Handling of scaling issues with large inputs.
#### Essential Functions:
- `benchmark::State`: Represents the state of the benchmark, used to control iterations.
- `BENCHMARK(...)`: Registers a function as a benchmark.
- `BENCHMARK_MAIN()`: Main entry point for running benchmarks.
#### Benchmark Configuration Options:
1. **Repeats**: Specify the number of iterations with `--benchmark_repetitions=N`.
2. **Time Limit**: Set the time for each benchmark with `--benchmark_min_time=N`.
3. **Custom Arguments**: Pass custom inputs to benchmarks for parameterized testing.

### Example Benchmark (C++ Code)

Below is an example showcasing how to benchmark the nearest point calculation on a triangle in *Kratos*:

~~~cpp
// System includes

// External includes
#include <benchmark/benchmark.h>

// Project includes
#include "geometries/point.h"
#include "geometries/triangle_3d_3.h"
#include "utilities/geometry_utilities/nearest_point_utilities.h"

namespace Kratos
{

// Sample data for benchmarking
Point::Pointer p_point_1(make_shared<Point>( 0.0, 0.0, 0.0));
Point::Pointer p_point_2(make_shared<Point>( 1.0, 0.0, 0.0));
Point::Pointer p_point_3(make_shared<Point>( 0.0, 1.0, 0.0));

Triangle3D3<Point> triangle(p_point_1, p_point_3, p_point_2);
Point nearest_point(0.0, 0.0, 0.0);
Point inside_point(0.2, 0.1, 0.00);

static void BM_TriangleNearestPoint(benchmark::State& state) {
for (auto _ : state) {
NearestPointUtilities::TriangleNearestPoint(inside_point, triangle, nearest_point);
}
}

// Register the function as a benchmark
BENCHMARK(BM_TriangleNearestPoint);

} // namespace Kratos

BENCHMARK_MAIN();
~~~
### Advanced Features of Google Benchmark
#### Custom Arguments
Benchmarks can take custom arguments to test specific cases. Example:
~~~cpp
static void BM_CustomArguments(benchmark::State& state) {
for (auto _ : state) {
int input_size = state.range(0);
// Your code here, e.g., filling arrays of size input_size
}
}
BENCHMARK(BM_CustomArguments)->Arg(100)->Arg(1000)->Arg(10000);
~~~

#### Multithreading Support

Google Benchmark can run benchmarks in multi-threaded mode:

~~~cpp
static void BM_MultiThreaded(benchmark::State& state) {
for (auto _ : state) {
// Thread-safe code
}
}

BENCHMARK(BM_MultiThreaded)->Threads(1)->Threads(2)->Threads(4);
~~~
## Running Benchmarks
### Execution
Once benchmarks are compiled, executables are created in the `benchmark` directory. Run them directly via the command line:
```
./benchmark_name
```
### Configuration Options
- **JSON Output**: Export results in JSON format:
```
./benchmark_name --benchmark_format=json --benchmark_out=output.json
```
- **Statistical Runs**: Run benchmarks multiple times for statistical accuracy:
```
./benchmark_name --benchmark_repetitions=10
```
- **Custom Time Limits**: Specify time per benchmark:
```
./benchmark_name --benchmark_min_time=2.0
```
## Comparing Benchmark Results
The benchmark framework includes a Python utility script to compare results between multiple JSON outputs. This script, located in `kratos/python_scripts/benchmark/generate_plot_google_benchmark.py`, generates visualizations for easier comparison.
### Example Usage:
To compare two JSON results:
```
python generate_plot_google_benchmark.py --filenames result1.json result2.json
```
### Example Output:
A plot similar to the one below will be generated, showing the performance (in terms of CPU time and real time) of different benchmarks across the compared files.
![Benchmark Performance Graph](images/benchmark.png)
## Python Script Explanation
### Purpose:
The provided Python script consolidates and visualizes benchmarking data from multiple JSON files generated by Google Benchmark.
#### Script Functionality:
1. **Load JSON Outputs:** Extract benchmark results from each JSON file.
2. **Merge Data:** Combine results into a single pandas DataFrame for analysis.
3. **Generate Plot:** Create a grouped bar chart showing `cpu_time` and `real_time` for each benchmark.
## About the Google Benchmark Library
### Benefits
- Accurate Timing: Adjusts for background noise and other factors.
- Reproducibility: Consistent and repeatable measurements.
- Customizability: Configure inputs, threads, and scaling scenarios.
### Useful Links
- [Google Benchmark Documentation](https://github.com/google/benchmark)
- [Google Benchmark Usage Guide](https://github.com/google/benchmark/blob/main/docs/user_guide.md)
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 6974c9b

Please sign in to comment.