[Serverless]: No data for running filebeat under Discover tab.

[amolnater-qasource]

Serverless Build details:

VERSION: 8.12.0
BUILD: 68058
COMMIT: 326ef31677a6df2fe3cfaebd4e3fe251d8b26c12

Host OS: All

Preconditions:

1. 8.12.0 Serverless environment should be available.

Steps to reproduce:

1. Setup the filbeat.yml and enable system module.
2. Enable system.yml modules to get the data.
3. Run ./filebeat setup --index-management and observe the template is created under IndexManagement- Data Streams.
4. Run the filebeat using: sudo ./filebeat -e
5. Observe beat runs successfully, however no data under Discover tab.

Screen Recording:

Index.Management.-.Elastic.-.Google.Chrome.2023-10-17.15-39-41.mp4

NOTE:

• We have tested this on Windows 2022 server:
  • Filebeat- threatintel module enabled
  • Metricbeat- system enabled
• SLES 15:
  • Filebeat- system module enabled

However no data is observed under Discover tab of serverless environment.

Expected Result:
Data for running filebeat should be available under Discover tab.

filebeat.yml:
filebeat.zip

sudo filebeat -e output:
filebeat running output.txt

dsl_policy:
dsl_policy.zip

filebeat logs:
logs.zip

[amolnater-qasource]

@manishgupta-qasource Please review.

[manishgupta-qasource]

Secondary review for this ticket is Done

[amolnater-qasource]

FYI @cmacknz @jlind23

[jlind23]

@pierrehilbert @fearful-symmetry would you mind taking a look at this please?

[fearful-symmetry]

Looks like we have some kind of versioning issue:

{"log.level":"error","@timestamp":"2023-10-17T10:05:38.797Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":148},"message":"Failed to connect to backoff(elasticsearch(https://amol-serverless-c4ae3e.es.us-east-1.aws.elastic.cloud:443)): Connection marked as failed because the onConnect callback failed: Elasticsearch is too old. Please upgrade the instance. If you would like to connect to older instances set output.elasticsearch.allow_older_versions to true. ES=8.11.0, Beat=8.12.0","service.name":"filebeat","ecs.version":"1.6.0"}

This can be fixed with the output.elasticsearch.allow_older_versions flag. However, more confused by how this happened, since I thought Serverless wasn't pegged to any stack release and was just locked to 8.0. Going to ask around.

[fearful-symmetry]

Alright, so it turns out that serverless is pinned at 8.11, and we never noticed until after the feature freeze, since, well, it was all 8.11. Serverless will have to bypass version checks.

[cmacknz]

I would be in favor of just removing this version check entirely and just replacing it with a warning log message when ES is older than the Beats. We can default allow_older_versions: true instead and just log the difference.

I'm not sure what problem it is solving, and I know this does cause problems during rolling upgrades of ES nodes.

[fearful-symmetry]

Initial PR here: #36884

[amolnater-qasource]

Hi Team,

Thank you for the fixes.
We have revalidated this issue on latest 8.12.0 Serverless environment and found it fixed now.

Observations:

• Data is now available under Discovering tab on running beat.

Build details:
BUILD: 68058
COMMIT: 326ef31677a6df2fe3cfaebd4e3fe251d8b26c12
Artifact Link: https://snapshots.elastic.co/8.12.0-0f0dbad4/downloads/beats/metricbeat/metricbeat-8.12.0-SNAPSHOT-windows-x86_64.zip

Screen Recording:

Elastic.-.Google.Chrome.2023-10-23.17-05-17.mp4

Hence, we are closing this issue and marking as QA:Validated.

Thanks!

[harshitgupta-qasource]

Bug Conversion

• Test-Case not required as this particular checkpoint is already covered in exploratory testing.

Thanks!