[Cases] Improve unit test flakiness #212489
Open
+3,935
−4,654
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
/ci |
|
/ci |
…t --include-path /api/status --include-path /api/alerting/rule/ --include-path /api/alerting/rules --include-path /api/actions --include-path /api/security/role --include-path /api/spaces --include-path /api/fleet --include-path /api/dashboards --update'
|
/ci |
|
/ci |
|
/ci |
cnasikas
commented
Mar 4, 2025
| @@ -270,24 +270,6 @@ describe('schema', () => { | |||
| ] | |||
| `); | |||
| }); | |||
|
|
|||
| it.skip('fails when page number is negative', () => { | |||
|
/ci |
cnasikas
added
Team:ResponseOps
cnasikas
changed the title
|
Pinging @elastic/response-ops (Team:ResponseOps) |
|
Pinging @elastic/response-ops-cases (Feature:Cases) |
cnasikas
added
the
release_note:skip
jbudz
approved these changes
Mar 10, 2025
💚 Build Succeeded
Metrics [docs]Async chunks
Unknown metric groupsESLint disabled line counts
Total ESLint disabled count
History
cc @cnasikas |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR attempts (I have lost count of how many there have been so far) to stabilize the flakiness of cases tests.
Approach
Based on our investigations, I believe a common factor that causes all tests to time out is how we wrap the tests with the needed providers. Instead of figuring out why they time out (which is very difficult), I tried a different approach. I rewrote how we initialize the testing dependencies, mocks, and providers on tests. To test my theory, I created a VM instance in GCloud with the same configuration as the one running in the CI, specifically the n2-standard-4 (4 vCPUs, 16 GB Memory) machine type. I ran the tests 100 times, which took almost two days. In 10 of the runs, a random test was timeouted. I noticed that the machine was stressed while running the tests, and the CPU frequently spiked to 100%, especially at the beginning of each test. Then, I increased the timeout for all cases tests to 10 seconds and ran the tests again 100 times. No timeouts occurred. Lastly, I created a VM instance in GCloud with the same configuration as the one running in the CI, specifically the n2-standard-8 (8 vCPUs, 32 GB Memory) machine type. I ran the tests again 100 times. In 1 of the runs, a random test was timeouted. The machine on the CI cannot handle the cases tests. I believe the work in this PR is a step in the right direction either way, and we will benefit from it. I also believe increasing the timeout is a good decision as we need it based on the experiments and the research we have done in the last months.
CPU usage on n2-standard-4
perf.mp4
CPU usage on n2-standard-8
perf-2.mp4
RLT eslint rules
I enabled the RLT eslint rules for Cases and resolved any eslint errors. The process revealed small bugs in the tests, which I fixed them.
Testing utils
I removed the
appMockRenderand any usage in favor of the newrenderWithTestingProvidersutility function and the improvedTestProviderscomponent. TherenderWithTestingProvidersfollows the principals suggested by the RLT team. Specifically:renderWithTestingProvidersis a wrapper of therenderfunction of the RTL library.renderWithTestingProvidersdoes not create the services or any component inside it.renderWithTestingProviderscannot be used inbeforeEachfunctions. It should be called separately on each test.renderWithTestingProvidersaccepts props to override the default mocks.renderWithTestingProviderspasses theTestProvidersin thewrapperargument of the RLTrenderfunction.TestProviderscomponent initializes and memoizes all services and dependencies. It accepts props to override the default mocks.renderWithTestingProviderson each test, even if it means duplication. We favor test isolation.Checklist
Check the PR satisfies the following conditions.
Reviewers should verify this PR satisfies this list as well.
Issues
List
Fixes: #207712
Fixes: #192739
Fixes: #174682
Fixes: #206366
Fixes: #207427
Fixes: #175239
Fixes: #177334
Fixes: #208443
Fixes: #187526
Fixes: #208310
Fixes: #192640
Fixes: #207077
Fixes: #197304
Fixes: #207249
Fixes: #202761
Fixes: #202115
Fixes: #193026
Fixes: #177304
Fixes: #208415
Fixes: #174661
Fixes: #201611
Fixes: #182364
Fixes: #175841
Fixes: #207907
Fixes: #171177
Fixes: #196628
Fixes: #194703
Fixes: #207241
Fixes: #206056
Fixes: #207328
Fixes: #205953
Fixes: #176524
Fixes: #176335
Fixes: #207404
Fixes: #207384
Fixes: #208380
Fixes: #207248
Fixes: #207444
Fixes: #175240
Fixes: #178001