[Automatic Import] add timestamp to ECS constants

[haetamoudi]

Release Note

Automatic Import now ensures that the field mapping contains the @timestamp field whenever possible.

Summary

Ensure mapping contains @timestamp field whenever possible. #196040

Tested cases:

test case	has @timestamp	is expected result
sample logs with datetime value for time and expires fields	yes (picked time as @timestamp)	✅
sample logs with datetime value for expires field only	no	✅
sample logs with no datetime values	no	✅
sample logs with created_at field that does not contain datetime value	no	✅

Tested values for time field:

value	match correctly @timestamp
2024-02-24T06:56:50.648137154Z	✅
10/01/2023 12:34:56	✅
01-10-2023 12:34:56	✅
Thu, 25 December 2023 10:15:00GMT	✅

Checklist

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
• This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
• Flaky Test Runner was used on any tests changed
• The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

[elasticmachine]

Pinging @elastic/security-scalability (Team:Security-Scalability)

[ebeahan]

Ensure mapping contains @timestamp field whenever possible

For ECS events, @timestamp is a required field. If the provided log samples are missing an obvious field to use for @timestamp or the model doesn't map a field for whatever reason, maybe the produced pipeline should follow the ECS guidance?

If the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline.

[ilyannn]

Ensure mapping contains @timestamp field whenever possible

For ECS events, @timestamp is a required field. If the provided log samples are missing an obvious field to use for @timestamp or the model doesn't map a field for whatever reason, maybe the produced pipeline should follow the ECS guidance?

If the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline.

We could add a processor at the end of all the ingest pipelines that fills out the @timestamp with the current date. That would seem to satisfy the guidance in the fullest possible way.

[ilyannn]

This looks good to me. We can iterate in terms of further testing to see if we need to give more examples or think through the fallback.

[ebeahan]

This looks good to me. We can iterate in terms of further testing to see if we need to give more examples or think through the fallback.

++ I don't want to sidetrack getting the improvements in ASAP, so we can continue the discussion beyond this PR (I'll open an issue 😄 ).

[kibanamachine]

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/12518847786

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 1a50694

Metrics [docs]

✅ unchanged

History

• 💛 Build #262323 was flaky 95fd448
• 💔 Build #262315 failed 434006a
• 💔 Build #262297 failed 1ec4529

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation