[Synthetics] Improve overview page performance !! #201275
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…w-status-performance
…zad31/kibana into overview-status-performance
…w-status-performance
…w-status-performance
…w-status-performance
…zad31/kibana into overview-status-performance
…w-status-performance
…w-status-performance
shahzad31
added
release_note:skip
There was a problem hiding this comment.
The functionality looks good to me. Have some recommended code changes/improvements. I'll let @dominiqueclarke leave a review as well before approving when she's available.
...bservability_solution/synthetics/server/saved_objects/synthetics_monitor/get_all_monitors.ts
Outdated
Show resolved
Hide resolved
...ck/plugins/observability_solution/synthetics/server/synthetics_service/synthetics_service.ts
Show resolved
Hide resolved
...s/observability_solution/synthetics/server/routes/overview_status/overview_status_service.ts
Outdated
Show resolved
Hide resolved
...s/observability_solution/synthetics/server/routes/overview_status/overview_status_service.ts
Outdated
Show resolved
Hide resolved
...s/observability_solution/synthetics/server/routes/overview_status/overview_status_service.ts
Outdated
Show resolved
Hide resolved
...s/observability_solution/synthetics/server/routes/overview_status/overview_status_service.ts
Show resolved
Hide resolved
...s/observability_solution/synthetics/server/routes/overview_status/overview_status_service.ts
Outdated
Show resolved
Hide resolved
...s/observability_solution/synthetics/server/routes/overview_status/overview_status_service.ts
Outdated
Show resolved
Hide resolved
...s/observability_solution/synthetics/server/routes/overview_status/overview_status_service.ts
Outdated
Show resolved
Hide resolved
botelastic
bot
added
ci:project-deploy-observability
|
Pinging @elastic/obs-ux-management-team (Team:obs-ux-management) |
🤖 GitHub commentsExpand to view the GitHub comments
Just comment with:
|
…w-status-performance
…w-status-performance
|
/oblt-deploy |
💚 Build Succeeded
Metrics [docs]
History
|
|
Starting backport for target branches: 8.x |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Dec 11, 2024
## Summary Improve overview page performance !! Right now UI works for few hundred to 1000 monitors, but it starts degrading after that, this PR makes sure, we refactor queries in such a way that it scale up to 10k-20k monitors easily. ### Queries before Before this PR, we were doing 2 steps queries, first fetch all saved objects and the fetch all summary documents by passings all ids from first phase. This meant that let's say if we have 20k saved objects, first we will need to page through all of them to even start fetching summaries. To fetch summary documents, we were using `top_hits` query which can be memory expensive. ### Queries now In this PR we fetch summaries and saved objects in parallel, since we have space id on documents as well, there was no need to do 2 step queries. Now we fetch both things in parallel and then we hydrate saved object data from summary data. In this PR now we are using top_metrics query to fetch each monitor status instead of `top_hits` I tested on about 20k monitors, app performs reasoably well after the PR <img width="1920" alt="image" src="https://github.com/user-attachments/assets/c143e196-59a4-45b4-86b7-bd22ac4c5d4b"> On a very slow cluster on which kibana is local against a remote cluster ### After <img width="1920" alt="image" src="https://github.com/user-attachments/assets/540d0cdf-2f8c-44d1-af76-81953d9ca0ff"> ### Before <img width="1918" alt="image" src="https://github.com/user-attachments/assets/5fdc314d-bb59-4137-9397-d8aee6bd4806"> --------- Co-authored-by: kibanamachine <[email protected]> (cherry picked from commit b4ccb0c)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this pull request
Dec 12, 2024
## Summary Improve overview page performance !! Right now UI works for few hundred to 1000 monitors, but it starts degrading after that, this PR makes sure, we refactor queries in such a way that it scale up to 10k-20k monitors easily. ### Queries before Before this PR, we were doing 2 steps queries, first fetch all saved objects and the fetch all summary documents by passings all ids from first phase. This meant that let's say if we have 20k saved objects, first we will need to page through all of them to even start fetching summaries. To fetch summary documents, we were using `top_hits` query which can be memory expensive. ### Queries now In this PR we fetch summaries and saved objects in parallel, since we have space id on documents as well, there was no need to do 2 step queries. Now we fetch both things in parallel and then we hydrate saved object data from summary data. In this PR now we are using top_metrics query to fetch each monitor status instead of `top_hits` I tested on about 20k monitors, app performs reasoably well after the PR <img width="1920" alt="image" src="https://github.com/user-attachments/assets/c143e196-59a4-45b4-86b7-bd22ac4c5d4b"> On a very slow cluster on which kibana is local against a remote cluster ### After <img width="1920" alt="image" src="https://github.com/user-attachments/assets/540d0cdf-2f8c-44d1-af76-81953d9ca0ff"> ### Before <img width="1918" alt="image" src="https://github.com/user-attachments/assets/5fdc314d-bb59-4137-9397-d8aee6bd4806"> --------- Co-authored-by: kibanamachine <[email protected]>
kibanamachine
added a commit
that referenced
this pull request
Dec 12, 2024
…03892) # Backport This will backport the following commits from `main` to `8.x`: - [[Synthetics] Improve overview page performance !! (#201275)](#201275) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Shahzad","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-12-11T19:33:33Z","message":"[Synthetics] Improve overview page performance !! (#201275)\n\n## Summary\r\n\r\nImprove overview page performance !!\r\n\r\nRight now UI works for few hundred to 1000 monitors, but it starts\r\ndegrading after that, this PR makes sure, we refactor queries in such a\r\nway that it scale up to 10k-20k monitors easily.\r\n\r\n\r\n### Queries before\r\nBefore this PR, we were doing 2 steps queries, first fetch all saved\r\nobjects and the fetch all summary documents by passings all ids from\r\nfirst phase. This meant that let's say if we have 20k saved objects,\r\nfirst we will need to page through all of them to even start fetching\r\nsummaries. To fetch summary documents, we were using `top_hits` query\r\nwhich can be memory expensive.\r\n\r\n\r\n### Queries now\r\nIn this PR we fetch summaries and saved objects in parallel, since we\r\nhave space id on documents as well, there was no need to do 2 step\r\nqueries. Now we fetch both things in parallel and then we hydrate saved\r\nobject data from summary data. In this PR now we are using top_metrics\r\nquery to fetch each monitor status instead of `top_hits`\r\n\r\n\r\nI tested on about 20k monitors, app performs reasoably well after the PR\r\n<img width=\"1920\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/c143e196-59a4-45b4-86b7-bd22ac4c5d4b\">\r\n\r\n\r\nOn a very slow cluster on which kibana is local against a remote cluster\r\n\r\n### After\r\n<img width=\"1920\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/540d0cdf-2f8c-44d1-af76-81953d9ca0ff\">\r\n\r\n\r\n### Before\r\n<img width=\"1918\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/5fdc314d-bb59-4137-9397-d8aee6bd4806\">\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"b4ccb0c205b2df4312edfe7a087e0bca25242d05","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:prev-minor","ci:project-deploy-observability","Team:obs-ux-management"],"title":"[Synthetics] Improve overview page performance !!","number":201275,"url":"https://github.com/elastic/kibana/pull/201275","mergeCommit":{"message":"[Synthetics] Improve overview page performance !! (#201275)\n\n## Summary\r\n\r\nImprove overview page performance !!\r\n\r\nRight now UI works for few hundred to 1000 monitors, but it starts\r\ndegrading after that, this PR makes sure, we refactor queries in such a\r\nway that it scale up to 10k-20k monitors easily.\r\n\r\n\r\n### Queries before\r\nBefore this PR, we were doing 2 steps queries, first fetch all saved\r\nobjects and the fetch all summary documents by passings all ids from\r\nfirst phase. This meant that let's say if we have 20k saved objects,\r\nfirst we will need to page through all of them to even start fetching\r\nsummaries. To fetch summary documents, we were using `top_hits` query\r\nwhich can be memory expensive.\r\n\r\n\r\n### Queries now\r\nIn this PR we fetch summaries and saved objects in parallel, since we\r\nhave space id on documents as well, there was no need to do 2 step\r\nqueries. Now we fetch both things in parallel and then we hydrate saved\r\nobject data from summary data. In this PR now we are using top_metrics\r\nquery to fetch each monitor status instead of `top_hits`\r\n\r\n\r\nI tested on about 20k monitors, app performs reasoably well after the PR\r\n<img width=\"1920\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/c143e196-59a4-45b4-86b7-bd22ac4c5d4b\">\r\n\r\n\r\nOn a very slow cluster on which kibana is local against a remote cluster\r\n\r\n### After\r\n<img width=\"1920\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/540d0cdf-2f8c-44d1-af76-81953d9ca0ff\">\r\n\r\n\r\n### Before\r\n<img width=\"1918\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/5fdc314d-bb59-4137-9397-d8aee6bd4806\">\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"b4ccb0c205b2df4312edfe7a087e0bca25242d05"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/201275","number":201275,"mergeCommit":{"message":"[Synthetics] Improve overview page performance !! (#201275)\n\n## Summary\r\n\r\nImprove overview page performance !!\r\n\r\nRight now UI works for few hundred to 1000 monitors, but it starts\r\ndegrading after that, this PR makes sure, we refactor queries in such a\r\nway that it scale up to 10k-20k monitors easily.\r\n\r\n\r\n### Queries before\r\nBefore this PR, we were doing 2 steps queries, first fetch all saved\r\nobjects and the fetch all summary documents by passings all ids from\r\nfirst phase. This meant that let's say if we have 20k saved objects,\r\nfirst we will need to page through all of them to even start fetching\r\nsummaries. To fetch summary documents, we were using `top_hits` query\r\nwhich can be memory expensive.\r\n\r\n\r\n### Queries now\r\nIn this PR we fetch summaries and saved objects in parallel, since we\r\nhave space id on documents as well, there was no need to do 2 step\r\nqueries. Now we fetch both things in parallel and then we hydrate saved\r\nobject data from summary data. In this PR now we are using top_metrics\r\nquery to fetch each monitor status instead of `top_hits`\r\n\r\n\r\nI tested on about 20k monitors, app performs reasoably well after the PR\r\n<img width=\"1920\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/c143e196-59a4-45b4-86b7-bd22ac4c5d4b\">\r\n\r\n\r\nOn a very slow cluster on which kibana is local against a remote cluster\r\n\r\n### After\r\n<img width=\"1920\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/540d0cdf-2f8c-44d1-af76-81953d9ca0ff\">\r\n\r\n\r\n### Before\r\n<img width=\"1918\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/5fdc314d-bb59-4137-9397-d8aee6bd4806\">\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"b4ccb0c205b2df4312edfe7a087e0bca25242d05"}}]}] BACKPORT--> Co-authored-by: Shahzad <[email protected]>
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this pull request
Jan 13, 2025
## Summary Improve overview page performance !! Right now UI works for few hundred to 1000 monitors, but it starts degrading after that, this PR makes sure, we refactor queries in such a way that it scale up to 10k-20k monitors easily. ### Queries before Before this PR, we were doing 2 steps queries, first fetch all saved objects and the fetch all summary documents by passings all ids from first phase. This meant that let's say if we have 20k saved objects, first we will need to page through all of them to even start fetching summaries. To fetch summary documents, we were using `top_hits` query which can be memory expensive. ### Queries now In this PR we fetch summaries and saved objects in parallel, since we have space id on documents as well, there was no need to do 2 step queries. Now we fetch both things in parallel and then we hydrate saved object data from summary data. In this PR now we are using top_metrics query to fetch each monitor status instead of `top_hits` I tested on about 20k monitors, app performs reasoably well after the PR <img width="1920" alt="image" src="https://github.com/user-attachments/assets/c143e196-59a4-45b4-86b7-bd22ac4c5d4b"> On a very slow cluster on which kibana is local against a remote cluster ### After <img width="1920" alt="image" src="https://github.com/user-attachments/assets/540d0cdf-2f8c-44d1-af76-81953d9ca0ff"> ### Before <img width="1918" alt="image" src="https://github.com/user-attachments/assets/5fdc314d-bb59-4137-9397-d8aee6bd4806"> --------- Co-authored-by: kibanamachine <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Improve overview page performance !!
Right now UI works for few hundred to 1000 monitors, but it starts degrading after that, this PR makes sure, we refactor queries in such a way that it scale up to 10k-20k monitors easily.
Queries before
Before this PR, we were doing 2 steps queries, first fetch all saved objects and the fetch all summary documents by passings all ids from first phase. This meant that let's say if we have 20k saved objects, first we will need to page through all of them to even start fetching summaries. To fetch summary documents, we were using
top_hitsquery which can be memory expensive.Queries now
In this PR we fetch summaries and saved objects in parallel, since we have space id on documents as well, there was no need to do 2 step queries. Now we fetch both things in parallel and then we hydrate saved object data from summary data. In this PR now we are using top_metrics query to fetch each monitor status instead of
top_hitsI tested on about 20k monitors, app performs reasoably well after the PR
On a very slow cluster on which kibana is local against a remote cluster
After
Before