[BUG] desc_sort_timestamp is slower with OS 3.0 index compared to OS 2.19 index on OS 3.0 server

[expani]

Describe the bug

OS 3.0 * OS 2.19 indicates that the OpenSearch server was running 3.0 whereas the index used was created in OS 2.19. This was done to eliminate any suspicion of the bug arising from an indexing change in Lucene.

Metric Name	Operation/Query name	OS 2.19	OS 3.0	OS 3.0 * OS 2.19	Unit
50th percentile latency	desc_sort_timestamp	7.93071	10.8131	7.7783	ms
90th percentile latency	desc_sort_timestamp	8.45714	11.4532	8.68553	ms
99th percentile latency	desc_sort_timestamp	9.68866	14.6754	10.0167	ms
100th percentile latency	desc_sort_timestamp	11.1424	16.3041	10.6183	ms
50th percentile service time	desc_sort_timestamp	6.51203	9.40036	6.35852	ms
90th percentile service time	desc_sort_timestamp	7.07093	9.89428	7.15328	ms
99th percentile service time	desc_sort_timestamp	8.42815	13.2756	8.50242	ms
100th percentile service time	desc_sort_timestamp	9.56305	14.7515	9.65212	ms

As we can see something in the index layout is causing OS 3.0 server to be slower with OS 3.0 index.

Related component

Search:Performance

To Reproduce

Run desc_sort_timestamp with Big5 workload using OS 3.0 server and compare with indices created in OS 2.19 v/s OS 3.0

Expected behavior

desc_sort_timestamp should be equally faster with OS 3.0 index.

Additional Details

Meta

[expani]

@prudhvigodithi Created an issue for regression with desc_sort_timestamp b/w OS 2.19 and OS 3.0