[HOLD for payment 2023-10-12] [HOLD for payment 2023-10-09] E2E tests are broken 💀

[AndrewGable]

Problem

We have E2E tests that are supposed to run on every PR merge, but they are broken and do not run correctly.

Solution

Fix them!

[melvin-bot]

Triggered auto assignment to @mallenexpensify (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.

[melvin-bot]

Bug0 Triage Checklist (Main S/O)

• This "bug" occurs on a supported platform (ensure Platforms in OP are ✅)
• This bug is not a duplicate report (check E/App issues and #expensify-bugs)
  • If it is, comment with a link to the original report, close the issue and add any novel details to the original issue instead
• This bug is reproducible using the reproduction steps in the OP. S/O
  • If the reproduction steps are clear and you're unable to reproduce the bug, check with the reporter and QA first, then close the issue.
  • If the reproduction steps aren't clear and you determine the correct steps, please update the OP.
• This issue is filled out as thoroughly and clearly as possible
  • Pay special attention to the title, results, platforms where the bug occurs, and if the bug happens on staging/production.
• I have reviewed and subscribed to the linked Slack conversation to ensure Slack/Github stay in sync

[hannojg]

👋

[mallenexpensify]

@hannojg do you know what the specific deliverable is here?

[AndrewGable]

The deliverables is the E2E performance tests should run successfully after a PR is merged.

[AndrewGable]

Nice we finally got tests to pass here: https://github.com/Expensify/App/actions/runs/6307355630/job/17124627671

[melvin-bot]

Reviewing label has been removed, please complete the "BugZero Checklist".

[melvin-bot]

The solution for this issue has been 🚀 deployed to production 🚀 in version 1.3.75-12 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

• [No QA]Fix E2E Performance Regression Tests #28109

If no regressions arise, payment will be issued on 2023-10-09. 🎊

After the hold period is over and BZ checklist items are completed, please complete any of the applicable payments for this issue, and check them off once done.

• External issue reporter
• Contributor that fixed the issue
• Contributor+ that helped on the issue and/or PR

For reference, here are some details about the assignees on this issue:

• @hannojg does not require payment (Contractor)

As a reminder, here are the bonuses/penalties that should be applied for any External issue:

• Merged PR within 3 business days of assignment - 50% bonus
• Merged PR more than 9 business days after assignment - 50% penalty

[melvin-bot]

BugZero Checklist: The PR fixing this issue has been merged! The following checklist (instructions) will need to be completed before the issue can be closed:

• [@AndrewGable] The PR that introduced the bug has been identified. Link to the PR:
• [@AndrewGable] The offending PR has been commented on, pointing out the bug it caused and why, so the author and reviewers can learn from the mistake. Link to comment:
• [@AndrewGable] A discussion in #expensify-bugs has been started about whether any other steps should be taken (e.g. updating the PR review checklist) in order to catch this type of bug sooner. Link to discussion:
• [@hannojg] Determine if we should create a regression test for this bug.
• [@hannojg] If we decide to create a regression test for the bug, please propose the regression test steps to ensure the same bug will not reach production again.
• [@mallenexpensify] Link the GH issue for creating/updating the regression test once above steps have been agreed upon:

[AndrewGable]

Good news: Tests are running again!

Bad news: It seems like they are not correctly reporting an e2e that should have failed

[mallenexpensify]

@AndrewGable it looks like @ntdiary reviewed the PR, I'm assuming they're the C+ on this issue, should be assigned to it and compensated for C+, is that right?

[melvin-bot]

Reviewing label has been removed, please complete the "BugZero Checklist".

[melvin-bot]

The solution for this issue has been 🚀 deployed to production 🚀 in version 1.3.77-7 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

• [No QA]Remove check to only look for blockers when run via workflow_call #28659

If no regressions arise, payment will be issued on 2023-10-12. 🎊

After the hold period is over and BZ checklist items are completed, please complete any of the applicable payments for this issue, and check them off once done.

• External issue reporter
• Contributor that fixed the issue
• Contributor+ that helped on the issue and/or PR

For reference, here are some details about the assignees on this issue:

• @hannojg does not require payment (Contractor)

As a reminder, here are the bonuses/penalties that should be applied for any External issue:

• Merged PR within 3 business days of assignment - 50% bonus
• Merged PR more than 9 business days after assignment - 50% penalty

[melvin-bot]

BugZero Checklist: The PR fixing this issue has been merged! The following checklist (instructions) will need to be completed before the issue can be closed:

• [@AndrewGable] The PR that introduced the bug has been identified. Link to the PR:
• [@AndrewGable] The offending PR has been commented on, pointing out the bug it caused and why, so the author and reviewers can learn from the mistake. Link to comment:
• [@AndrewGable] A discussion in #expensify-bugs has been started about whether any other steps should be taken (e.g. updating the PR review checklist) in order to catch this type of bug sooner. Link to discussion:
• [@hannojg] Determine if we should create a regression test for this bug.
• [@hannojg] If we decide to create a regression test for the bug, please propose the regression test steps to ensure the same bug will not reach production again.
• [@mallenexpensify] Link the GH issue for creating/updating the regression test once above steps have been agreed upon:

[melvin-bot]

@AndrewGable, @hannojg, @mallenexpensify Whoops! This issue is 2 days overdue. Let's get this updated quick!

[mallenexpensify]

@ntdiary I believe you need to be paid for the PR review here
If you agree, can you please accept the job and reply here once you have?
https://www.upwork.com/jobs/~016a75689d4031216b

[ntdiary]

@ntdiary I believe you need to be paid for the PR review here If you agree, can you please accept the job and reply here once you have? https://www.upwork.com/jobs/~016a75689d4031216b

@mallenexpensify, wow, thank you! Have accepted. : )

[mallenexpensify]

Contributor+: @ntdiary paid $500 via Upwork

@AndrewGable I'm assuming we don't need a regression test added for this, comment if you disagree!