Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[APM] Service map can cause OOM in elasticsearch #187707

Open
neptunian opened this issue Jul 5, 2024 · 3 comments
Open

[APM] Service map can cause OOM in elasticsearch #187707

neptunian opened this issue Jul 5, 2024 · 3 comments
Labels
apm:service-maps Service Map feature in APM bug Fixes for quality problems that affect the customer experience stale Used to mark issues that were closed for being stale Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team

Comments

@neptunian
Copy link
Contributor

neptunian commented Jul 5, 2024

Following up from #186417

When testing the service map under the maximum conditions of 1k trace ids with each trace having ~500 spans, the scripted metric aggregation can cause an OOM in elasticsearch depending on the memory available. Looking at the elasticsearch heapdump, I suspect this is due to the # of hash maps and other data structures being created simultaneously, where data can be duplicated and exist at the same time within the reduce phase. This issue did not happen when disabling parallel async requests and having them sync, when calling fetch_service_paths_from_trace_ids. Further investigation needed.

@neptunian
Copy link
Contributor Author

@crespocarlos did some investigation #187445

@neptunian neptunian added the Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team label Jul 5, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

@smith smith added apm:service-maps Service Map feature in APM bug Fixes for quality problems that affect the customer experience labels Jul 5, 2024
@botelastic
Copy link

botelastic bot commented Jan 1, 2025

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@botelastic botelastic bot added the stale Used to mark issues that were closed for being stale label Jan 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
apm:service-maps Service Map feature in APM bug Fixes for quality problems that affect the customer experience stale Used to mark issues that were closed for being stale Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team
Projects
None yet
Development

No branches or pull requests

3 participants