[Stack 9.0.0-SNAPSHOT] failing tests in packages due to file size is too small for ingestion (no documents found) #12499

mrodm · 2025-01-28T11:29:50Z

There are some packages that are failing in the daily CI Jobs for the Elastic stack 9.0.0-SNAPSHOT.

Example of build failing: https://buildkite.com/elastic/integrations/builds/21038

These packages are failing due to errors in the elastic-agent like this one and no documents are ingested into Elasticsearch:

apache_tomcat: @elastic/obs-infraobs-integrations

cannot start ingesting from file "/tmp/service_logs/localhost.log": filesize of "/tmp/service_logs/localhost.log" is 319 bytes, expected at least 1024 bytes for fingerprinting: file size is too small for ingestion

coredns @elastic/obs-infraobs-integrations

cannot start ingesting from file "/tmp/service_logs/coredns.log": filesize of "/tmp/service_logs/coredns.log" is 383 bytes, expected at least 1024 bytes for fingerprinting: file size is too small for ingestion

gitlab @elastic/security-service-integrations

cannot start ingesting from file "/tmp/service_logs/test-gitlab-audit.log": filesize of "/tmp/service_logs/test-gitlab-audit.log" is 781 bytes, expected at least 1024 bytes for fingerprinting: file size is too small for ingestion

microsoft_exchange_server @elastic/sec-windows-platform

cannot start ingesting from file "/tmp/service_logs/imappop_Imap4.log": filesize of "/tmp/service_logs/imappop_Imap4.log" is 246 bytes, expected at least 1024 bytes for fingerprinting: file size is too small for ingestion","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.origin":{"file.line":394,"file.name

ping_federate @elastic/security-service-integrations

cannot start ingesting from file "/tmp/service_logs/test-admin.log": filesize of "/tmp/service_logs/test-admin.log" is 189 bytes, expected at least 1024 bytes for fingerprinting: file size is too small for ingestion

tychon @elastic/security-service-integrations

cannot start ingesting from file "/tmp/service_logs/test-base.ndjson": filesize of "/tmp/service_logs/test-base.ndjson" is 99 bytes, expected at least 1024 bytes for fingerprinting: file size is too small for ingestion

mrodm · 2025-01-28T16:55:31Z

Probably related to these issues (and could be closed if this issue is solved):

[Stack 9.0.0-SNAPSHOT] [ping_federate] Failing test daily: system test: logfile in ping_federate.admin #12321
[Stack 9.0.0-SNAPSHOT] [ping_federate] Failing test daily: system test: logfile in ping_federate.audit #12322
[Stack 9.0.0-SNAPSHOT] [tychon] Failing test daily: system test: default in tychon.cmrs #12327
[Stack 9.0.0-SNAPSHOT] [microsoft_exchange_server] Failing test daily: system test: default in microsoft_exchange_server.smtp #12320
[Stack 9.0.0-SNAPSHOT] [microsoft_exchange_server] Failing test daily: system test: default in microsoft_exchange_server.imap4_pop3 #12318
[Stack 9.0.0-SNAPSHOT] [gitlab] Failing test daily: system test: filestream in gitlab.sidekiq #12308
[Stack 9.0.0-SNAPSHOT] [gitlab] Failing test daily: system test: default in gitlab.pages #12306
[Stack 9.0.0-SNAPSHOT] [coredns] Failing test daily: system test: filestream (variant: v1) in coredns.log #12303

~~Due to some other error there is no issue created (yet) for apache_tomcat.~~

EDIT:

[Stack 9.0.0-SNAPSHOT] [apache_tomcat] Failing test daily: system test: default (variant: v10.1.5) in apache_tomcat.localhost #12522

mrodm · 2025-01-28T17:59:51Z

Looking into the logs, it can be checked how fingerprint mode is enabled:

{"log.level":"debug","@timestamp":"2025-01-28T01:49:24.339Z","message":"fingerprint mode enabled: offset 0, length 1024","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"scanner","log.origin":{"file.line":311,"file.name":"filestream/fswatch.go","function":"github.com/elastic/beats/v7/filebeat/input/filestream.newFileScanner"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-01-28T01:49:24.339Z","message":"recursive glob enabled","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"scanner","log.origin":{"file.line":334,"file.name":"filestream/fswatch.go","function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*fileScanner).resolveRecursiveGlobs"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-01-28T01:49:24.339Z","message":"file identity is set to fingerprint","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"ecs.version":"1.6.0","log.logger":"input.filestream","log.origin":{"file.line":57,"file.name":"filestream/prospector_creator.go","function":"github.com/elastic/beats/v7/filebeat/input/filestream.newProspector"},"service.name":"filebeat","filestream_id":"filestream-coredns.log-48963670-6129-4d62-b9ce-adf88595cf96","ecs.version":"1.6.0"}

In this PR elastic/beats#41762 , the default file_identity is changed from native to fingerprint.

There were some packages that added options to disable this file_identity mode: #12349 (issue #12325)

Another option would be to update the corresponding test docs to have more than 1024 bytes.

efd6 · 2025-01-29T02:33:37Z

Another possible option is to expose the prospector.scanner.fingerprint.length in the integrations so that they can be set lower in tests (or alternatively just set them lower, though this may be unwise given the small size of some of the test cases).

efd6 · 2025-01-29T03:10:32Z

Addressing the SSI integrations:

gitlab gitlab,ping_federate: make system tests pass on 9.0.0 #12510
ping_federate gitlab,ping_federate: make system tests pass on 9.0.0 #12510

mrodm · 2025-01-30T09:39:59Z

Thanks @efd6!

Should the related issues to gitlab and ping_federate mentioned in #12499 (comment) be closed too ?

EDIT:
I see that tychon is also related to SSI integrations, could tychon package be updated similar to gitlab or ping_federate?

chemamartinez · 2025-01-30T18:35:41Z

@mrodm I am already taking a look at the Tychon failure.

We can wait until confirm in the daily build that every issue affecting these packages has been solved. It runs daily at 1:35am UTC: https://buildkite.com/elastic/integrations/builds?query=Check+integrations+local+stacks+-+Stack+Version+v9.0

efd6 · 2025-01-30T19:52:02Z

@mrodm Closed those. Apologies for missing that.

mrodm · 2025-02-04T09:58:26Z

Thanks for solving the issues related to the SSI packages @efd6 @chemamartinez !

@elastic/obs-infraobs-integrations @elastic/sec-windows-platform could you please take a look to the packages related to your teams (apache_tomcat, coredns and microsoft_exchange_server)? Once solved, this would help to run all the tests in those packages successfully for 9.0.0-SNAPSHOT.

Thanks in advance!

ishleenk17 · 2025-02-04T10:08:18Z

Yes @mrodm , we are taking care of these as part of this ticket.

mrodm · 2025-02-12T11:02:41Z

@ishleenk17 could this two issues be closed as part of the work done in #12548.

[Stack 9.0.0-SNAPSHOT] [coredns] Failing test daily: system test: filestream (variant: v1) in coredns.log #12303
[Stack 9.0.0-SNAPSHOT] [apache_tomcat] Failing test daily: system test: default (variant: v10.1.5) in apache_tomcat.localhost #12522

And probably, these two other issues too (created last night), likely that they are the same:

[Stack 9.1.0-SNAPSHOT] [apache_tomcat] Failing test daily: system test: default (variant: v10.1.5) in apache_tomcat.localhost #12740
[Stack 9.1.0-SNAPSHOT] [coredns] Failing test daily: system test: filestream (variant: v1) in coredns.log #12741

mrodm · 2025-02-12T11:07:59Z

In the latest daily builds, microsoft_exchange_server is not failing anymore.

@marc-gr could this two issues be closed thanks to this PR: #12381 ?

[Stack 9.0.0-SNAPSHOT] [microsoft_exchange_server] Failing test daily: system test: default in microsoft_exchange_server.smtp #12320
[Stack 9.0.0-SNAPSHOT] [microsoft_exchange_server] Failing test daily: system test: default in microsoft_exchange_server.imap4_pop3 #12318

ishleenk17 · 2025-02-12T11:08:44Z

@ishleenk17 could this two issues be closed as part of the work done in #12548.

[Stack 9.0.0-SNAPSHOT] [coredns] Failing test daily: system test: filestream (variant: v1) in coredns.log #12303

[Stack 9.0.0-SNAPSHOT] [apache_tomcat] Failing test daily: system test: default (variant: v10.1.5) in apache_tomcat.localhost #12522

And probably, these two other issues too (created last night), likely that they are the same:

[Stack 9.1.0-SNAPSHOT] [apache_tomcat] Failing test daily: system test: default (variant: v10.1.5) in apache_tomcat.localhost #12740

[Stack 9.1.0-SNAPSHOT] [coredns] Failing test daily: system test: filestream (variant: v1) in coredns.log #12741

Yes Mario these would be fixed by #12548.

marc-gr · 2025-02-12T11:12:34Z

In the latest daily builds, microsoft_exchange_server is not failing anymore.

@marc-gr could this two issues be closed thanks to this PR: #12381 ?

[Stack 9.0.0-SNAPSHOT] [microsoft_exchange_server] Failing test daily: system test: default in microsoft_exchange_server.smtp #12320

[Stack 9.0.0-SNAPSHOT] [microsoft_exchange_server] Failing test daily: system test: default in microsoft_exchange_server.imap4_pop3 #12318

yes they should be fixed as part of that one

mrodm · 2025-02-12T11:17:14Z

Closing this issue since all the packages detected that were failing due to the error "file size is too small" have been fixed.

Thanks all for your help!

efd6 self-assigned this Jan 29, 2025

efd6 mentioned this issue Jan 29, 2025

gitlab,ping_federate: make system tests pass on 9.0.0 #12510

Merged

5 tasks

efd6 removed their assignment Jan 29, 2025

kush-elastic mentioned this issue Feb 11, 2025

OBS non-cloud package with 9.0.0 Sanity Issues #12639

Merged

5 tasks

mrodm closed this as completed Feb 12, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Stack 9.0.0-SNAPSHOT] failing tests in packages due to file size is too small for ingestion (no documents found) #12499

[Stack 9.0.0-SNAPSHOT] failing tests in packages due to file size is too small for ingestion (no documents found) #12499

mrodm commented Jan 28, 2025

mrodm commented Jan 28, 2025 •

edited

Loading

mrodm commented Jan 28, 2025 •

edited

Loading

efd6 commented Jan 29, 2025

efd6 commented Jan 29, 2025 •

edited

Loading

mrodm commented Jan 30, 2025 •

edited

Loading

chemamartinez commented Jan 30, 2025

efd6 commented Jan 30, 2025

mrodm commented Feb 4, 2025

ishleenk17 commented Feb 4, 2025

mrodm commented Feb 12, 2025

mrodm commented Feb 12, 2025

ishleenk17 commented Feb 12, 2025 •

edited

Loading

marc-gr commented Feb 12, 2025

mrodm commented Feb 12, 2025

[Stack 9.0.0-SNAPSHOT] failing tests in packages due to file size is too small for ingestion (no documents found) #12499

[Stack 9.0.0-SNAPSHOT] failing tests in packages due to file size is too small for ingestion (no documents found) #12499

Comments

mrodm commented Jan 28, 2025

mrodm commented Jan 28, 2025 • edited Loading

mrodm commented Jan 28, 2025 • edited Loading

efd6 commented Jan 29, 2025

efd6 commented Jan 29, 2025 • edited Loading

mrodm commented Jan 30, 2025 • edited Loading

chemamartinez commented Jan 30, 2025

efd6 commented Jan 30, 2025

mrodm commented Feb 4, 2025

ishleenk17 commented Feb 4, 2025

mrodm commented Feb 12, 2025

mrodm commented Feb 12, 2025

ishleenk17 commented Feb 12, 2025 • edited Loading

marc-gr commented Feb 12, 2025

mrodm commented Feb 12, 2025

mrodm commented Jan 28, 2025 •

edited

Loading

mrodm commented Jan 28, 2025 •

edited

Loading

efd6 commented Jan 29, 2025 •

edited

Loading

mrodm commented Jan 30, 2025 •

edited

Loading

ishleenk17 commented Feb 12, 2025 •

edited

Loading