Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable compression by default for Elasticsearch outputs #36681

Merged
merged 6 commits into from
Sep 28, 2023
Merged

Enable compression by default for Elasticsearch outputs #36681

merged 6 commits into from
Sep 28, 2023

Conversation

faec
Copy link
Contributor

@faec faec commented Sep 26, 2023 •
edited
Loading

Change the default compression level for the Elasticsearch output from 0 to 1. Part of https://github.com/elastic/ingest-dev/issues/2458

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

The benchmarks evaluating this change were run by @alexsapran and discussed in https://github.com/elastic/ingest-dev/issues/2399. In summary we expect this to have the following effects on "typical" ingestion:

  • ~70-80% reduction in data volume
  • 20-25% CPU increase
  • ~10% ingestion delay

The exact numbers depend on the specific configuration and the type of data being ingested. The CPU and ingestion changes were fairly stable in tests, and anything noticeably outside that range should be flagged for evaluation if it arises during testing.

@faec faec added enhancement Team:Elastic-Agent Label for the Agent team labels Sep 26, 2023
@faec faec self-assigned this Sep 26, 2023
@faec faec requested a review from a team as a code owner September 26, 2023 19:52
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 26, 2023
@mergify
Copy link
Contributor

mergify bot commented Sep 26, 2023

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @faec? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

@strawgate
Copy link
Contributor

strawgate commented Sep 26, 2023 •
edited
Loading

I'm not too familiar with the monitoring output -- does compression also have to be set for the monitoring output? https://github.com/elastic/beats/blob/main/libbeat/monitoring/report/elasticsearch/elasticsearch.go

@faec
Copy link
Contributor Author

faec commented Sep 26, 2023

I'm not too familiar with the monitoring output -- does compression also have to be set for the monitoring output?

I'm not aware of any discussion or testing of that case, so I wasn't planning to for this first pass. If we want to do that it should probably be included in the benchmarking and followup changes in https://github.com/elastic/ingest-dev/issues/2399.

@cmacknz
Copy link
Member

cmacknz commented Sep 26, 2023

I'm not too familiar with the monitoring output -- does compression also have to be set for the monitoring output?

I believe this is only used for stack monitoring of standalone Beats and is never used when Beats are run under agent.

@faec faec requested review from a team as code owners September 26, 2023 20:45
@elasticmachine
Copy link
Collaborator

elasticmachine commented Sep 26, 2023 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Duration: 100 min 7 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@jlind23 jlind23 requested review from belimawr and rdner September 27, 2023 05:10
@faec faec requested a review from a team as a code owner September 27, 2023 14:00
@strawgate
Copy link
Contributor

The benchmark that shows a 25% CPU doesn't use any processors, right? When processors or more complex inputs are used we'd expect the % CPU impact to be lower (as the output represents a smaller total % of the CPU usage)?

@faec
Copy link
Contributor Author

faec commented Sep 27, 2023

The benchmark that shows a 25% CPU doesn't use any processors, right? When processors or more complex inputs are used we'd expect the % CPU impact to be lower (as the output represents a smaller total % of the CPU usage)?

The quoted figures include some runs with processors (under Agent), but yes, configurations with heavy processor use or similar will see proportionately smaller marginal changes.

@faec faec enabled auto-merge (squash) September 27, 2023 19:49
@jlind23 jlind23 requested a review from cmacknz September 27, 2023 19:52
vigneshshanmugam
vigneshshanmugam approved these changes Sep 27, 2023
View reviewed changes
Copy link
Member

@vigneshshanmugam vigneshshanmugam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@cmacknz cmacknz disabled auto-merge September 28, 2023 13:28
@cmacknz
Copy link
Member

cmacknz commented Sep 28, 2023

Merging, we have enough approvals and this change has been widely communicated.

@cmacknz cmacknz merged commit 231e93a into elastic:main Sep 28, 2023
22 checks passed
@strawgate
Copy link
Contributor

Can we backport the compression changes to latest 7.x?

@cmacknz cmacknz added the backport-7.17 Automated backport to the 7.17 branch with mergify label Sep 29, 2023
mergify bot pushed a commit that referenced this pull request Sep 29, 2023
@faec @mergify
Enable compression by default for Elasticsearch outputs (#36681)
f6f64c8 
* Enable compression by default for Elasticsearch outputs

* update config template

* update changelog

* regenerate config templates

* update docs

* make check

(cherry picked from commit 231e93a)

# Conflicts:
#	libbeat/outputs/elasticsearch/config_test.go
@mergify mergify bot mentioned this pull request Sep 29, 2023
Merged
@cmacknz
Copy link
Member

cmacknz commented Sep 29, 2023

Opened a 7.17 backport. Will wait for the final round of benchmark results before merging.

jlind23 pushed a commit that referenced this pull request Oct 2, 2023
@mergify
[7.17](backport #36681) Enable compression by default for Elasticsear…
eef25a2 
…ch outputs (#36708)
@faec faec mentioned this pull request Oct 3, 2023
Merged
@kilfoyle kilfoyle mentioned this pull request Nov 7, 2023
Merged
@nicpenning
Copy link
Contributor

Any chance this change is causing a memory leak that is being reported across the community?

#37142

@andrewkroh
Copy link
Member

Have you tried setting output.elasticsearch.compression_level: 0 to revert to the old behavior and seeing if the leak persists?

@nicpenning
Copy link
Contributor

I am going to try that when I'm back at the office. I think others have more extreme examples that may be able to test more accurately but I will be happy to test and report back!

@hendry-lim
Copy link
Contributor

hendry-lim commented Nov 19, 2023 •
edited
Loading

Have you tried setting output.elasticsearch.compression_level: 0 to revert to the old behavior and seeing if the leak persists?

image

Yes, it persists. Metricbeat is consuming 4.5 GB of RAM now.

image

@cmacknz
Copy link
Member

cmacknz commented Nov 23, 2023

To close the loop the cause of the memory leak was fixed in elastic/elastic-agent-system-metrics#115

Scholar-Li pushed a commit to Scholar-Li/beats that referenced this pull request Feb 5, 2024
@faec @Scholar-Li
Enable compression by default for Elasticsearch outputs (elastic#36681)
70cdc44 
* Enable compression by default for Elasticsearch outputs

* update config template

* update changelog

* regenerate config templates

* update docs

* make check
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@leehinman leehinman leehinman approved these changes

@andrewkroh andrewkroh andrewkroh approved these changes

@vigneshshanmugam vigneshshanmugam vigneshshanmugam approved these changes

@cmacknz cmacknz cmacknz approved these changes

@belimawr belimawr belimawr approved these changes

@fearful-symmetry fearful-symmetry Awaiting requested review from fearful-symmetry fearful-symmetry is a code owner automatically assigned from elastic/elastic-agent-data-plane

@rdner rdner Awaiting requested review from rdner

Assignees

@faec faec

Labels
backport-7.17 Automated backport to the 7.17 branch with mergify enhancement Team:Elastic-Agent Label for the Agent team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
10 participants
@faec @strawgate @cmacknz @elasticmachine @nicpenning @andrewkroh @hendry-lim @belimawr @vigneshshanmugam @leehinman