[Introduce Aggregate Command] Incorporate Automatic Aggregation

[OVI3D0]

Overview

This is an issue based off one of the proposed priorities in this RFC: #627

Background

The proposed aggregate subcommand and enhanced compare subcommand will provide users with powerful tools for aggregating and comparing benchmark test results. However, there is an opportunity to leverage these features in other parts of OpenSearch Benchmark (OSB) to further enhance the user experience and provide more comprehensive performance testing capabilities.

Proposed Design

We propose incorporating automatic aggregation into new and existing features of OSB, such as:

1. Automatically running a test multiple times and aggregating the results: OSB could include an option to automatically run a specified test multiple times and aggregate the results using the proposed aggregate subcommand. This would streamline the process of obtaining reliable and consistent performance measurements by reducing the need for manual intervention.

2. Aggregating results from distributed workload generation (DWG) tests across multiple load generation hosts: In DWG tests, where the workload is generated from multiple hosts, OSB could automatically aggregate the results from each host using the aggregate subcommand, providing a consolidated view of the overall performance under distributed load conditions.

The automatic aggregation feature could include options or configurations to handle failures or errors in individual test runs, such as excluding specific runs, using partial results up to the failure point, or retrying failed runs up to a certain limit.

Proposed Priority

Incorporating automatic aggregation into new and existing features of OSB is a valuable enhancement that will improve the overall user experience and provide more comprehensive performance testing capabilities. It will streamline the process of obtaining reliable and consistent performance measurements, particularly in scenarios involving multiple test runs or distributed workload generation.