[7.17](backport #39392) [Bug] fix high IO after sudden filebeat stop (#35893)

[mergify]

Proposed commit message

In case of corrupted log file (which has good chances to happen in case of sudden unclean system shutdown), we set a flag which causes us to checkpoint immediately, but never do anything else besides that. This causes filebeat to just checkpoint on each log operation (therefore causing a high IO load on the server and also causing filebeat to fall behind).

This change resets the logInvalid flag after a successful checkpointing.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• [ ]

How to test this PR locally

I have not in fact tested the PR. I have only checked that it builds. I'm also not 100% sure of the correctness, but the change is really simple and I hope that a maintainer can quickly confirm whether the fix is correct. In any case the current code clearly will cause the issues I described, since logInvalid is only ever set to true, never to false. Therefore I think that the beat trivially cannot recover.

Related issues

• Closes High io consumption after sudden filebeat stop #35893

Logs

See details in #35893 but the issue we fix otherwise manifests itself with a message of:

WARN memlog/store.go:130 Incomplete or corrupted log file in /var/lib/filebeat/registry/filebeat. Continue with last known complete and consistent state. Reason: unexpected EOF

(or invalid character '\x00' instead of EOF)

---

This is an automatic backport of pull request #39392 done by Mergify.

[mergify]

Cherry-pick of 217f5a6 has failed:

On branch mergify/bp/7.17/pr-39392
Your branch is up to date with 'origin/7.17'.

You are currently cherry-picking commit 217f5a6264.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   CHANGELOG.next.asciidoc
	new file:   libbeat/statestore/backend/memlog/store_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   libbeat/statestore/backend/memlog/diskstore.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[mergify]

This pull request has not been merged yet. Could you please review and merge it @belimawr? 🙏

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[belimawr]

I'm not quite sure why the test failed, I didn't find a clear error message aside from Error: failed modules: kubernetes. I'll try re-running the tests.

[mergify]

This pull request has not been merged yet. Could you please review and merge it @belimawr? 🙏

[mergify]

This pull request has not been merged yet. Could you please review and merge it @belimawr? 🙏