CRITICAL: Access to a specific report is lost and couldn't see the chats

[m-natarajan]

If you haven’t already, check out our contributing guidelines for onboarding and email [email protected] to request to join our Slack channel!

---

Version Number:
Reproducible in staging?: needs reproduction
Reproducible in production?: needs reproduction
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers): https://staging.new.expensify.com/r/8823458142807525 / https://staging.new.expensify.com/r/71168705
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: @tgolen
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1714659279184109

Action Performed:

1. Open chat report with @Beamanator

Expected Result:

Able to access the chat report and chat loads

Actual Result:

• Don't see the messages sent
• Chat with @Beamanator is not shown in LHN
• When access his chat fro search getting not found page

Workaround:

unknown

Platforms:

Which of our officially supported platforms is this issue occurring on?

• Android: Native
• Android: mWeb Chrome
• iOS: Native
• iOS: mWeb Safari
• MacOS: Chrome / Safari
• MacOS: Desktop

Screenshots/Videos

Add any screenshot/video evidence

View all open jobs on GitHub

[melvin-bot]

Triggered auto assignment to @trjExpensify (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details. Please add this bug to a GH project, as outlined in the SO.

[MelvinBot]

This has been labelled "Needs Reproduction". Follow the steps here: https://stackoverflowteams.com/c/expensify/questions/16989

[melvin-bot]

👋 Friendly reminder that deploy blockers are time-sensitive ⏱ issues! Check out the open Staging deploy checklist to see the list of PRs included in this release, then work quickly on the following:

1. If you find which PR caused the issue/bug, you can reassign it to the person responsible for it.
   • If the author is OOO or won’t get online before the daily deploy is due, you are responsible for finding the best fix/path forward. Don’t hesitate to ask for help!
2. Try to reproduce the issue, if the bug is on production, remove the DeployBlocker label but stay assigned to fix it (or find out which PR broke it to get help from the author).
   • You can adjust the urgency of the issue to better represent the gravity of the bug.
   • If the issue is super low priority, feel free to un-assign yourself.
   • Be careful with PHP warnings, sometimes it is more complex than just adding a null coalescing operator as they might be uncovering some bigger bug.
   • If it was a one-off issue that requires no action (for example, Bedrock was down or it is a duplicated issue), you can close it.

Remember rule #2: Never un-assign yourself from a real DeployBlocker unless you are 100% sure someone else is assigned and will take care of it.

[iwiznia]

Sorry about that noise, was doing a test for https://github.com/Expensify/PHP-Libs/pull/954 and did not realize all that was going to happen 😄

[trjExpensify]

@tgolen the report not found sounds suspiciously like what I experienced here: #41254 (comment)

{"code":666,"jsonCode":403,"type":"Expensify\\Libs\\Error\\ExpError","UUID":"da7984df-628f-41f6-a2b8-ecd4499f7975","message":"Report not found","title":"","data":{"onyxData":[{"onyxMethod":"merge","key":"report_0","value":{"errorFields":{"notFound":{"1714580833933154":"Report not found"}}}}]},"htmlMessage":"","requestID":"87d123434d9c4164-LHR"}

Here are the logs Tim shared in thread as well. @m-natarajan let's make sure you put those in the issues you create, because they're helpful.

@Beamanator's theory is:

"Wild guess because @aldo-expensify has seen us calling OpenReport on report action ids recently I believe"

What are the next steps here? Internal or External? @muttmuure do you want this in NewDot Performance?

[muttmuure]

Does anyone have their onyx state preserved such that they're still experiencing this? (Don't refresh or leave the page where you're seeing the bug.) If so let's start a thread in #newdot-performance and recruit an engineer to debug it live

[trjExpensify]

Not I, I put my logs in the linked issue.

[muttmuure]

OK, I have it:

[melvin-bot]

👋 Friendly reminder that deploy blockers are time-sensitive ⏱ issues! Check out the open Staging deploy checklist to see the list of PRs included in this release, then work quickly on the following:

1. If you find which PR caused the issue/bug, you can reassign it to the person responsible for it.
   • If the author is OOO or won’t get online before the daily deploy is due, you are responsible for finding the best fix/path forward. Don’t hesitate to ask for help!
2. Try to reproduce the issue, if the bug is on production, remove the DeployBlocker label but stay assigned to fix it (or find out which PR broke it to get help from the author).
   • You can adjust the urgency of the issue to better represent the gravity of the bug.
   • If the issue is super low priority, feel free to un-assign yourself.
   • Be careful with PHP warnings, sometimes it is more complex than just adding a null coalescing operator as they might be uncovering some bigger bug.
   • If it was a one-off issue that requires no action (for example, Bedrock was down or it is a duplicated issue), you can close it.

Remember rule #2: Never un-assign yourself from a real DeployBlocker unless you are 100% sure someone else is assigned and will take care of it.

[iwiznia]

Sorry again, used it as a test once more

[trjExpensify]

Unsubscribe

[muttmuure]

Labeling this one UX Reliability

[trjExpensify]

So it sounds like @mallenexpensify could be right and this is an OpenReport error

Ah, which we're working on somewhere else at this point?

[mallenexpensify]

I feel like a lot of these bugs might be related to the write performance issues we're experiencing (or something that's being negatively affected because of the issues with write performance). For some of my bugs, I've punting them to retest next week, after we fix the fire and are on the other side of the merge freeze.

[trjExpensify]

Asking if we're moving this anywhere here, I'm a bit unclear on the next steps: https://expensify.slack.com/archives/C05LX9D6E07/p1715364552147619?thread_ts=1714750131.190839&cid=C05LX9D6E07

[melvin-bot]

@trjExpensify Whoops! This issue is 2 days overdue. Let's get this updated quick!

[muttmuure]

Bumped in channel

[muttmuure]

I think after considering the RCA that Dan provided when I looked into this, I think that the solution here is a mechanism that retrieves the missing parts of the report object. It does seem like what happens is:

• You try to load a report
• Potentially there are site issues
• The report gets stuck in this partially loaded state (with contact details only)
• We don't "retry" getting the rest of the details

[muttmuure]

So I think that this issue will fix this #41112 (comment)

[tgolen]

Just to make sure everyone has the same context... There are lots of suggestions to have some code that monitors for partial report data and then pulls the full report.

We had this exact logic previously, and it was removed in this PR. There was a lot of history behind that decision and it was a real cause of performance issues.

This kind of "magic" code that monitored for partial reports leads to a lot of code that just assumes that a full report object will always be there and code that doesn't know where the data comes from or can't recover when all of a sudden that data isn't there like it expects.

[muttmuure]

thanks for that context, @tgolen!

[melvin-bot]

@muttmuure this issue was created 2 weeks ago. Are we close to a solution? Let's make sure we're treating this as a top priority. Don't hesitate to create a thread in #expensify-open-source to align faster in real time. Thanks!

[melvin-bot]

@muttmuure Whoops! This issue is 2 days overdue. Let's get this updated quick!

[muttmuure]

Updates in channel

[danieldoglas]

@arosiclair I can see you worked on https://github.com/Expensify/Web-Expensify/pull/41296, maybe you can look into this?

[muttmuure]

Since none of the original reporters are experiencing this anymore, I am going to close it