Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

196040 automatic import ensure the timestamp field is present #204551

Conversation

haetamoudi
Copy link
Contributor

@haetamoudi haetamoudi commented Dec 17, 2024

Summary

Enforce @timestamp in mapping returned by LLM. #196040

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

  • Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
  • Documentation was added for features that require explanation or tutorials
  • Unit or functional tests were updated or added to match the most common scenarios
  • If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
  • This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
  • Flaky Test Runner was used on any tests changed
  • The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

@elasticmachine
Copy link
Contributor

🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

  • Click to trigger kibana-pull-request for this PR!
  • Click to trigger kibana-deploy-project-from-pr for this PR!

@haetamoudi
Copy link
Contributor Author

If the LLM cannot map any field to @timestamp it will retry multiple times.
I initially wanted to break out of the loop if the logs sample did not have any datetime fields, however datetime can come in multiple formats, and the logic was not very reliable e.g:

2023-10-01T12:34:56Z
2023-10-01T12:34:56.123Z
2023-10-01T12:34:56+00:00
2023-10-01T12:34:56.123+00:00
2023-10-01T12:34:56.123456789Z
2023-10-01 12:34:56
10/01/2023 12:34:56
01-10-2023 12:34:56
2023-10-01T12:34:56.123456789+00:00
2023-10-01T12:34:56.123456789-07:00

Any thoughts?

@@ -37,6 +37,7 @@ Go through each value step by step and modify it with the following process:
9. When you want to use an ECS field as a value for a target, but another field already has the same ECS field as its target, try to find another fitting ECS field. If none is found then the one you are least confident about should have the object replaced with null.
10. If you are not confident for a specific field, you should always set the value to null.
11. These {package_name} log samples are based on source and destination type data, prioritize these compared to other related ECS fields like host.* and observer.*.
12. Ensure it has a @timestamp target field that matches the event creation time.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is that really necessary now that the @timestamp was added to the ECS_FIELDS in constants.ts?

@haetamoudi haetamoudi closed this Dec 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants