[Feature] Make Service token functionality optional for Output type remote_elasticsearch

[mag-mkorn]

Is your feature request related to a problem? Please describe.
I want to send data to 3rd Party systems that implement the bulk API - in my specific case Cribl. As the output type elasticsearch overrides the api_key field in the Advanced YAML configuration, I use remote_elasticsearch.

In Cribl I have already created static API Keys for authentication that I provide within the Advanced YAML configuration. Therefore, the buillt-in functionality to manage authentication with service tokens is not required for this target system.

I currently set a dummy value as the service tokens and the connection does work. However, the output is tagged Unhealthy in the fleet UI.

Describe the solution you'd like
Make the dynamic authentication functionality using service tokens optional. Add a simple toggle button to enable/disable.

Describe alternatives you've considered
I tried using the elasticsearch output type. This doesn't work as well, as this output type overrides manually configured values for api_key.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[kpollich]

Thanks for raising this, @mag-mkorn. I'm wondering if the official Cribl integration for Elastic Agent might help with this particular use case. I'm not sure we'd want to directly support "non-Elasticsearch outputs that happen to support Elasticsearch's bulk API" officially in Fleet in this way, and I think the integration handles some of the API key juggling for you.

cc @elastic/security-service-integrations as they own the Cribl integration and might be able to weigh in more about this.

[mag-mkorn]

Thanks for raising this, @mag-mkorn. I'm wondering if the official Cribl integration for Elastic Agent might help with this particular use case. I'm not sure we'd want to directly support "non-Elasticsearch outputs that happen to support Elasticsearch's bulk API" officially in Fleet in this way, and I think the integration handles some of the API key juggling for you.

cc @elastic/security-service-integrations as they own the Cribl integration and might be able to weigh in more about this.

Actually, I think we can close this issue. The output type remote_elasticsearch comes with too many restrictions for our scenario, e.g. not supporting synthetics. This makes sense in common deployments, but is not usable for us, as we in fact want to send the data back to the same deployment.

We're not using the cribl integration for multiple reasons. Most importantly, i don't see any additional value provided by the integration. Everything can be done directly in Cribl in a better way. Secondly, the documentation is not really great.