You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I've worked on optimising (unit) test runs in terms of speed of the entire CI workflow, reducing load (more taking into account the load/performance) on firebase emulators, all with the goal of reducing flaky tests.
I've made significant progress and now, our tests are passing, they're quite fast (thanks to Vitest) and are given a lot of time, in terms of timeout, when a large number of tests are making requests against firebase emulators.
However, every now and then, a flaky test comes along. Therefore, I've created a unit test matrix, allowing us to run 36 unit tests as CI workflows and, in effect, helping us pinpoint the flaky tests.
Currently, only one flaky test is found, the frequency of failures is 3/36 (8.3%). I'm pretty sure that the test is not flaky per se, but the load on firebase emulators slows down the response, and, as the test probably runs multiple requests + polls for assertions (using waitFor test helper), the assertions get timed out. I don't think we should extend the default assertion timeout, nor test timeout for all tests (as they're already at 8000ms/15000ms respectively), but rather increase the assertion timeout and test timeout for the particular test. This should be a really quick fix, but I've not yet applied it and have opened this ticked as a TODO, as a lot of work is currently not merged in and I'd like the update to be a simple patch, rather than depending on other branches being mergeds.
The test in question is: Cloud functions -> Data triggers -> createAttendanceForBooking -> should create attendance entry for booking and not overwrite the existing data in slot
The metric (3/36 failure) can be found in the CI workflow: https://github.com/eisbuk/EisBuk/actions/runs/5118822733/jobs/9203298814
The text was updated successfully, but these errors were encountered:
I've worked on optimising (unit) test runs in terms of speed of the entire CI workflow, reducing load (more taking into account the load/performance) on firebase emulators, all with the goal of reducing flaky tests.
I've made significant progress and now, our tests are passing, they're quite fast (thanks to Vitest) and are given a lot of time, in terms of timeout, when a large number of tests are making requests against firebase emulators.
However, every now and then, a flaky test comes along. Therefore, I've created a unit test matrix, allowing us to run 36 unit tests as CI workflows and, in effect, helping us pinpoint the flaky tests.
Currently, only one flaky test is found, the frequency of failures is 3/36 (8.3%). I'm pretty sure that the test is not flaky per se, but the load on firebase emulators slows down the response, and, as the test probably runs multiple requests + polls for assertions (using
waitFor
test helper), the assertions get timed out. I don't think we should extend the default assertion timeout, nor test timeout for all tests (as they're already at 8000ms/15000ms respectively), but rather increase the assertion timeout and test timeout for the particular test. This should be a really quick fix, but I've not yet applied it and have opened this ticked as a TODO, as a lot of work is currently not merged in and I'd like the update to be a simple patch, rather than depending on other branches being mergeds.The test in question is:
Cloud functions
->Data triggers
->createAttendanceForBooking
->should create attendance entry for booking and not overwrite the existing data in slot
The metric (3/36 failure) can be found in the CI workflow: https://github.com/eisbuk/EisBuk/actions/runs/5118822733/jobs/9203298814
The text was updated successfully, but these errors were encountered: