[Reporting] CSV Export should use ESE search strategy

[tsullivan]

Kibana version: 7.x

See #109167 and #109976 for background context.

Kibana Reporting had an issue that using a SearchSource object serialized on the client, the SearchSource service on the server did not automatically handle the user's preference to include frozen indices in a search. Since a fix was needed urgently, the fix was implemented on the Reporting side. Reporting used its UISettings client to get the value of search:includeFrozen and pass that as an option to data.search.

SearchSource consumers should not have to provide configuration as additional options, since those options can be read internally in SearchSource. Fixing this issue in SearchSource reduces logic needed on the consumer side and promotes consistency from the user's perspective.

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[lukasolson]

SearchSource consumers should not have to provide configuration as additional options, since those options can be read internally in SearchSource.

I think the issue here is that you are consuming the ES_SEARCH_STRATEGY (which used to the be the OSS implementation which didn't include frozen indices because it was not an open-source feature) instead of the enhanced ESE_SEARCH_STRATEGY. Is there any reason you aren't using this strategy (which is the default if you don't pass one in) instead?

[tsullivan]

I think the issue here is that you are consuming the ES_SEARCH_STRATEGY (which used to the be the OSS implementation which didn't include frozen indices because it was not an open-source feature) instead of the enhanced ESE_SEARCH_STRATEGY. Is there any reason you aren't using this strategy (which is the default if you don't pass one in) instead?

CSV export uses the _scroll API to page through all the results of the search. After a little experimentation with the ESE_SEARCH_STRATEGY it looks like scroll queries are not supported:

[2021-10-04T11:14:56.325-07:00][DEBUG][plugins.reporting.runTask.csv_searchsource.csv_searchsource.execute-job.kucz3gq01d7m56b13cai8pcj] executing search request
[2021-10-04T11:14:56.340-07:00][ERROR][plugins.reporting.runTask.csv_searchsource.csv_searchsource.execute-job.kucz3gq01d7m56b13cai8pcj] Error: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: [request_cache] cannot be used in a scroll context;2: [scroll] queries are not supported;
    at getKbnServerError (/home/tsullivan/elastic/kibana/src/plugins/kibana_utils/server/report_server_error.ts:28:10)
    at CatchSubscriber.selector (/home/tsullivan/elastic/kibana/src/plugins/data/server/search/strategies/ese_search/ese_search_strategy.ts:95:15)
    at CatchSubscriber.error (/home/tsullivan/elastic/kibana/node_modules/rxjs/src/internal/operators/catchError.ts:130:23)
    at TapSubscriber._error (/home/tsullivan/elastic/kibana/node_modules/rxjs/src/internal/operators/tap.ts:135:22)
    at TapSubscriber.Subscriber.error (/home/tsullivan/elastic/kibana/node_modules/rxjs/src/internal/Subscriber.ts:113:12)
    at TapSubscriber._error (/home/tsullivan/elastic/kibana/node_modules/rxjs/src/internal/operators/tap.ts:135:22)
    at TapSubscriber.Subscriber.error (/home/tsullivan/elastic/kibana/node_modules/rxjs/src/internal/Subscriber.ts:113:12)
    at TakeUntilSubscriber.Subscriber._error (/home/tsullivan/elastic/kibana/node_modules/rxjs/src/internal/Subscriber.ts:143:22)
    at TakeUntilSubscriber.Subscriber.error (/home/tsullivan/elastic/kibana/node_modules/rxjs/src/internal/Subscriber.ts:113:12)
    at TakeWhileSubscriber.Subscriber._error (/home/tsullivan/elastic/kibana/node_modules/rxjs/src/internal/Subscriber.ts:143:22)
    at TakeWhileSubscriber.Subscriber.error (/home/tsullivan/elastic/kibana/node_modules/rxjs/src/internal/Subscriber.ts:113:12)
    at TapSubscriber._error (/home/tsullivan/elastic/kibana/node_modules/rxjs/src/internal/operators/tap.ts:135:22)
    at TapSubscriber.Subscriber.error (/home/tsullivan/elastic/kibana/node_modules/rxjs/src/internal/Subscriber.ts:113:12)
    at ExpandSubscriber.Subscriber._error (/home/tsullivan/elastic/kibana/node_modules/rxjs/src/internal/Subscriber.ts:143:22)
    at ExpandSubscriber.Subscriber.error (/home/tsullivan/elastic/kibana/node_modules/rxjs/src/internal/Subscriber.ts:113:12)
    at /home/tsullivan/elastic/kibana/node_modules/rxjs/src/internal/util/subscribeToPromise.ts:12:30
[2021-10-04T11:14:56.363-07:00][WARN ][plugins.reporting.runTask.csv_searchsource.csv_searchsource.execute-job.kucz3gq01d7m56b13cai8pcj] No scrollId to clear!
[2021-10-04T11:14:56.391-07:00][ERROR][plugins.reporting.runTask] Error: Max attempts (1) reached for job kucz3gq01d7m56b13cai8pcj. Failed with: {"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: [request_cache] cannot be used in a scroll context;2: [scroll] queries are not supported;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: [request_cache] cannot be used in a scroll context;2: [scroll] queries are not supported;"}
    at Object.run (/home/tsullivan/elastic/kibana/x-pack/plugins/reporting/server/lib/tasks/execute_report.ts:400:58)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at TaskManagerRunner.run (/home/tsullivan/elastic/kibana/x-pack/plugins/task_manager/server/task_running/task_runner.ts:278:22)

[tsullivan]

Closing this issue as CSV export can not use the ESE_SEARCH_STRATEGY