Chore: Fix Flaky E2E Tests

[nickytonline]

I've been noticing more frequent flakiness in the E2E test suite, recently. Here is on from a couple minutes ag0, https://github.com/open-sauced/app/actions/runs/10380912089/job/28741490937?pr=3945, but I've also noticed them on releases to main.

[github-actions]

Thanks for the issue, our team will look into it as soon as possible! If you would like to work on this issue, please wait for us to decide if it's ready. The issue will be ready to work on once we remove the "needs triage" label.

To claim an issue that does not have the "needs triage" label, please leave a comment that says ".take". If you have any questions, please comment on this issue.

For full info on how to contribute, please check out our contributors guide.

[nickytonline]

For some additional context, if I build and run the site locally and run the E2E tests, they all pass. This is pretty much the same setup as what runs in CI, but obviously not the same environment.

[jpmcb]

Do the e2e tests have to rely on github.com being an accessible and available service?

It seems most of the flakey-ness comes from either timeouts reaching github.com or not getting there at all:

    Expected substring: "https://github.com/login"
    Received string:    "http://localhost:3000/s/open-sauced/app"

      58 |
      59 |     await dialog.getByRole("button", { name: "Connect with GitHub", exact: true }).click();
    > 60 |     await expect(page.url()).toContain("https://github.com/login");
         |                              ^
      61 |   });
      62 |
      63 |   test("Adds a repository SBOM to a workspace", async ({ page }) => {

        at /home/runner/work/app/app/e2e/repo-page.spec.ts:60:30

Hypothesis: I wonder if GitHub is rate limiting the workers / runners that are hitting github.com (which probably have a user-agent string that gets more intensive rate limiting or a pool of IP addresses that are broadly identified as bots).

I know the idea behind the e2e tests was to have a re-producible login flow that we could test against. But in my eyes, it'd be better to have super solid, stable tests that don't rely too heavily on the backbone infrastructure of some other service. A good example of this would be the GitHub outage last week: this would entierly cripple our e2e validation and flow. Again, very very tricky because our auth flow does in fact rely on GitHub.

[brandonroberts]

I agree as I came across this also. I don't think we should be testing the full login/redirect flow past the point of whether the button is displayed or not. Testing what we can control from an e2e perspective could be less flakey.

I'll disable the GitHub URL checks and see

[open-sauced]

🎉 This issue has been resolved in version 2.59.0-beta.4 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

[jpmcb]

Re-opening so we can keep tracking flakey e2e tests.

Looks like we still see occasional timeouts: https://github.com/open-sauced/app/actions/runs/10459941789/job/28965031250

I wonder if it'd be worth hosting some of our own powerful runners via self hosted runners . I wonder if the 4 core, 16 gb ram runners GitHub has are underpowered and during peek hours, we get throttled causing the heavy browser based e2e tests to time out.

I could look at bootstrapping some powerful Azure runners to self host our actions. We'd likely see vast performance improvements across runners.

[open-sauced]

🎉 This issue has been resolved in version 2.59.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

[nickytonline]

Do the e2e tests have to rely on github.com being an accessible and available service?

It seems most of the flakey-ness comes from either timeouts reaching github.com or not getting there at all:

    Expected substring: "https://github.com/login"
    Received string:    "http://localhost:3000/s/open-sauced/app"

      58 |
      59 |     await dialog.getByRole("button", { name: "Connect with GitHub", exact: true }).click();
    > 60 |     await expect(page.url()).toContain("https://github.com/login");
         |                              ^
      61 |   });
      62 |
      63 |   test("Adds a repository SBOM to a workspace", async ({ page }) => {

        at /home/runner/work/app/app/e2e/repo-page.spec.ts:60:30

Hypothesis: I wonder if GitHub is rate limiting the workers / runners that are hitting github.com (which probably have a user-agent string that gets more intensive rate limiting or a pool of IP addresses that are broadly identified as bots).

I know the idea behind the e2e tests was to have a re-producible login flow that we could test against. But in my eyes, it'd be better to have super solid, stable tests that don't rely too heavily on the backbone infrastructure of some other service. A good example of this would be the GitHub outage last week: this would entierly cripple our e2e validation and flow. Again, very very tricky because our auth flow does in fact rely on GitHub.

We can intercept the request without actually hitting github.com and just check that it was heading there.

[jpmcb]

We can intercept the request without actually hitting github.com and just check that it was heading there.

That sounds like a great plan.

---

Seeing more frequent timeouts during peak engineering times: https://github.com/open-sauced/app/actions/runs/10579984281/job/29313738273

Error: Timed out 5000ms waiting for expect(locator).toBeHidden()

I'll see if I can carve out some time to drop some powerful self hosted runners

[nickytonline]

Once the self-hosted runners are in place, if the current E2E tests are running well, we should revert the skip tests and test modifications in #3983 to see if they run well again as locally they always pass.