Set basic requirements for inputs under inputrunner

[ruflin]

Inputrunner gives us a fresh starts on inputs. This issue is to discuss the minimal requirements for each input added to the repository. Here an initial list:

• v2: Input must be based on the v2 interface
• No cgo: This removes an cross compile complexity
• Full monitoring support: Input must report metrics
• dataset support: A standard target dataset must be configured. For generic inputs, support must be added to overwrite data_stream.* fields.
• Only local state: If an input adds state, it must be handled fully by the input
• Standard publisher: The standard publisher must be used to allow an easy switch over to shipper eventually
• No config transformations: Configs from Elastic Agent are forwarded to the inputs without transformations

[ruflin]

@andrewkroh @tommyers-elastic Can you chime in here?

[tommyers-elastic]

v2: can we just enforce this in code without input developers having to think about it?

please can you expand on 'full monitoring support' - are we talking about reporting APM data? or telemetry data? or something else?

"Only local state" seems a bit fuzzy to me, why not just specify that inputs here must be stateless.

'No config transformations' - what are the implications of this?

[ruflin]

v2: can we just enforce this in code without input developers having to think about it?

If we can, lets do it.

please can you expand on 'full monitoring support' - are we talking about reporting APM data? or telemetry data? or something else?

I stayed generic as this still needs definition. I expected @cmacknz and team + @andrewkroh to chime on this.

"Only local state" seems a bit fuzzy to me, why not just specify that inputs here must be stateless.

This came out of the conversation with @andrewkroh as they might have a few with state. An I rather have these also in the inputrunner with state then someone else. But the state mangement must not impact any other input

'No config transformations' - what are the implications of this?

This means on the elastic-agent side, the config block can be just forwarded to each input. The same already happens for example for apm-server or endpoint. For metricbeat / filebeat transformation happens which was always to goal to get rid of. The main implication is: The config to start the input look identical to the configs used in Elastic Agent.

[cmacknz]

I stayed generic as this still needs definition. I expected @cmacknz and team + @andrewkroh to chime on this.

In the agent today, monitoring means logs and metrics collection, as controlled by these options in the agent policy.

• Logs is easy, in v2 (8.6+) you write logs to stdout and the agent will collect them automatically.
• Metrics is more complicated. There needs to be a Metricbeat module that knows how to collect metrics from your process. For Beats this is handled by the beat module. For other processes we still need a generalized solution, but we have some ideas. Determine path forward for collection of metrics for components/units elastic-agent#2181

"Only local state" seems a bit fuzzy to me, why not just specify that inputs here must be stateless

Local state doesn't add any complexity for running under the agent, it only adds complexity into inputrunner itself if you want to support it. Each process is started in its own local working directory in the /run sub-directory of the agent installation. The contents of this directory are copied automatically to preserve them between upgrades of the agent. This is where Filebeat keeps its registry for example. You can put whatever files you want in there.

Example of the registry files for the monitoring filestream instance that ships the agent logs to Fleet:

❯ sudo tree /Library/Elastic/Agent/data/elastic-agent-913c02/run/filestream-monitoring/
/Library/Elastic/Agent/data/elastic-agent-913c02/run/filestream-monitoring/
├── meta.json
└── registry
    └── filebeat
        ├── log.json
        └── meta.json

this means on the elastic-agent side, the config block can be just forwarded to each input.

Yes the agent will just forward you the entire agent policy section for your input the configured output and it is your job to parse it.

One thing to keep in mind is that the agent wants to start multiple instances of the same input type in the same process, similar to Filebeat inputs today. This may lead you to using the Beat configuration reloading system which we requires some special handling to work with the way agent sends configurations.

Reading the new agent architecture overview will help here.

I've also included a diagram showing how the configuration would be parsed from a very simple agent policy. From this you would get a single output unit, and a single input unit, and the agent will send you the relevant parts of the agent policy unmodified.

[ruflin]

One thing to keep in mind is that the agent wants to start multiple instances of the same input type in the same process, similar to Filebeat inputs today. This may lead you to using the Beat configuration reloading system which we requires some special handling to work with the way agent sends configurations.

Good point, have not thought of this one. I might have remove some code around this 😇

[andrewkroh]

Full monitoring support: Input must report metrics

I was thinking along the lines of from day 1 we choose to instrument the code using the OTel metrics library. To export the metrics initially we use a promotheus exporter and have Metricbeat pull the data with the prometheus module. In the long run we switch to the OTLP exporter and push the metrics to some collector as configured by the Agent (assuming that's the direction the agent/shipper go in for metrics).

I really like the catalog of metrics documented by Vector. Having a list like this encourages consistent naming for similar measurements between inputs. We should have this.