[BUG] Unable to query across mutliple rollup indexes when rollup has been modified

[sharebear]

What is the bug?
Back in 2023 we started using using rollup jobs to aggregate some data into year based indeces, so the data ends up in the indexes rollup-data-2023, rollup-data-2024 and rollup-data-2025. When visualising data from these indeces we have an index pattern rollup-data-*.

Towards the end of 2024 we have added a field to the rollup job leading to the following scenario:

• rollup-data-2023 : contains only data without the new field
• rollup-data-2024 : contains some data without the new field and some with
• rollup-data-2025 : contains only data with the new field

When performing a query for the new field against the individual indices we get the following (expected) behavour:

• rollup-data-2023 : illegal_argument_exception : Could not find a rollup job that can answer this query because [missing field myNewField.keyword]
• rollup-data-2024 : SUCCESS
• rollup-data-2025 : SUCCESS

This is all understandable so far, the problem comes when trying to execute the same query against the index pattern that matches all three indices.

GET rollup-api-stats-*/_search
{
  "aggs": {
    "2": {
      "terms": {
        "field": "myNewField.keyword",
        "order": {
          "1": "desc"
        },
        "size": 5
      },
      "aggs": {
        "1": {
          "sum": {
            "field": "numberOfCalls"
          }
        }
      }
    }
  },
  "size": 0
}

When doing this all 3 shards fail.

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Could not find a rollup job that can answer this query because [missing field myNewField.keyword]"
      },
      {
        "type": "illegal_argument_exception",
        "reason": "Could not find a rollup job that can answer this query because [missing field myNewField.keyword]"
      },
      {
        "type": "illegal_argument_exception",
        "reason": "Could not find a rollup job that can answer this query because [missing field myNewField.keyword]"
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
...

How can one reproduce the bug?
See description for now, I might try and create a smaller reproduction if more explanation is needed.

What is the expected behavior?
I would expect only one shard to fail, allowing us to create visualisations for the index pattern.

What is your host/environment?

• OS: Linux (Aiven)
• Version: 2.17.1