[BUG] Date Histogram aggregation and it's variations have increased latency with OS 3.0/Lucene10 upgrade

[expani]

Describe the bug

OS 3.0 * OS 2.19 indicates that the OpenSearch server was running 3.0 whereas the index used was created in OS 2.19. This was done to eliminate any suspicion of the bug arising from an indexing change in Lucene.

Latency and service time has increased irrespective of OS 3.0 or OS 2.19 index used with OS 3.0 server

Metric Name	Operation/Query name	OS 2.19	OS 3.0	OS 3.0 * OS 2.19	Unit
50th percentile latency	date_histogram_minute_agg	40.6969	41.6565	43.0194	ms
90th percentile latency	date_histogram_minute_agg	41.463	43.1025	43.9114	ms
99th percentile latency	date_histogram_minute_agg	43.2176	45.2824	45.1605	ms
100th percentile latency	date_histogram_minute_agg	43.4249	51.6785	51.1855	ms
50th percentile service time	date_histogram_minute_agg	39.3274	40.1942	41.8148	ms
90th percentile service time	date_histogram_minute_agg	39.9173	41.4967	42.4525	ms
99th percentile service time	date_histogram_minute_agg	42.1739	44.3545	44.2888	ms
100th percentile service time	date_histogram_minute_agg	42.3934	49.8299	50.052	ms
50th percentile latency	range-auto-date-histo	2.14E+06	2.19E+06	2.18E+06	ms
90th percentile latency	range-auto-date-histo	2.48E+06	2.54E+06	2.53E+06	ms
99th percentile latency	range-auto-date-histo	2.55E+06	2.61E+06	2.61E+06	ms
100th percentile latency	range-auto-date-histo	2.56E+06	2.62E+06	2.61E+06	ms
50th percentile service time	range-auto-date-histo	9014.39	9218.99	9183.65	ms
90th percentile service time	range-auto-date-histo	9167.03	9306.23	9320.49	ms
99th percentile service time	range-auto-date-histo	9264.83	9410.94	9433.82	ms
100th percentile service time	range-auto-date-histo	9321.41	9472.17	9531.95	ms

Latency and service time ONLY INCREASES WITH OS 2.19 index used with OS 3.0 server

Metric Name	Operation/Query name	OS 2.19	OS 3.0	OS 3.0 * OS 2.19	Unit
50th percentile latency	range-auto-date-histo-with-metrics	5.53E+06	5.53E+06	5.71E+06	ms
90th percentile latency	range-auto-date-histo-with-metrics	6.40E+06	6.40E+06	6.61E+06	ms
99th percentile latency	range-auto-date-histo-with-metrics	6.59E+06	6.60E+06	6.81E+06	ms
100th percentile latency	range-auto-date-histo-with-metrics	6.61E+06	6.62E+06	6.83E+06	ms
50th percentile service time	range-auto-date-histo-with-metrics	22510.1	22540.7	23269.2	ms
90th percentile service time	range-auto-date-histo-with-metrics	22722.3	22712.5	23621.4	ms
99th percentile service time	range-auto-date-histo-with-metrics	22990.4	22860.9	23834.2	ms
100th percentile service time	range-auto-date-histo-with-metrics	23000.1	22918.4	23857.7	ms

Related component

Search:Aggregations

To Reproduce

Run the above mentioned operations with Big5 workload using OS 3.0 server and compare with indices created in OS 2.19 v/s OS 3.0

Expected behavior

All operations should have the same latency or lesser with OS 3.0 irrespective of the underlying index version.

Additional Details

Meta