[Performance] General performance investigation for opening app from customer profile trace

[hannojg]

If you haven’t already, check out our contributing guidelines for onboarding and email [email protected] to request to join our Slack channel!

---

What performance issue do we need to solve?

We had a customer meeting and they had really bad performance. We recorded multiple profile traces with them.
This issue is about the profile trace for app open.

The collection of issues:

• [Performance] Opening search for the first time on a big account is causing the app to hang massively #46590
• [Partially backend issue] [Performance] Typing in search is very laggy for big accounts #46591
• [Performance] User unable to navigate to chat in search results (slow API call blocks the whole app?) #46593
• [Performance] After clicking on a search result it took 4x OpenReport requests to open the final report #46594
• https://expensify.slack.com/archives/C049HHMV9SM/p1722434442673559
• https://expensify.slack.com/archives/C049HHMV9SM/p1722438425183629

What is the impact of this on end-users?

Slow or completely hanging/blocked app when starting.

List any benchmarks that show the severity of the issue

https://share.firefox.dev/4fgLVcm

Proposed solution (if any)

None yet, I will go through the profile and see what can be optimised, what exactly caused those lags.

List any benchmarks after implementing the changes to show impacts of the proposed solution (if any)

not available yet

Platforms:

Which of our officially supported platforms is this issue occurring on?

• Android: Native
• Android: mWeb Chrome
• iOS: Native
• iOS: mWeb Safari
• MacOS: Chrome / Safari
• MacOS: Desktop

Version Number: v9.0.11-5
Reproducible in staging?: not tested
Reproducible in production?: yes
Email or phone of affected tester (no customers): customer
Logs: See performance file
Notes/Photos/Videos: See attached video
Expensify/Expensify Issue URL: n/a
Issue reported by: @hannojg
Slack conversation: https://expensify.slack.com/archives/C05LX9D6E07/p1721919928992729

View all open jobs on Upwork

[melvin-bot]

Auto-assigning issues to engineers is no longer supported. If you think this issue should receive engineering attention, please raise it in #whatsnext.

[hannojg]

cc @sakluger (feel free to assign me as I (or someone from my team) will work on this ticket!)

[sakluger]

Hey @hannojg, any updates on this collection of issues?

[hannojg]

Working through the issues one by one, currently working on this one:

• [Performance] Typing in search is very laggy for big accounts #46591

I will also find someone from my team to help with these performance issues!

[sakluger]

Looks like we are adding some timing performance logging: #46807

[sakluger]

I'm going to change this to a Weekly since it's a tracking issue related to four specific performance investigations.

[hannojg]

One important thing we just found out.
So someone from our team created in old dot ~2000 reports. When I opened new dot OpenApp took in total ~3 min to complete (and also returned over 20mb of data).

During that time the whole app seemed block, presumably due to the sequential queue waiting for OpenApp to finish.

I feel like this is something we maybe want to improve on the backend?

During that time I was seeing a constant skeleton loader (I actually thought the app would be broken).

cc @chrispader

[sakluger]

Hanno distributed the issues above to a few other people so we can work on them in parallel. Otherwise, no major updates. I'll be OOO for the next week so I will check back then.

[muttmuure]

How are these coming along @hannojg ?

[sakluger]

Hey @hannojg, do you have any updates on any of the performance issues we're tracking here?

[sakluger]

Looks like steady progress is being made in the PR review.

[sakluger]

Looks like all reviews are complete on #48652, but it's failing one test. Hopefully this will be live very soon!

[sakluger]

The PR was merged but seems to have caused a deploy blocker: #51123

[muttmuure]

This went to prod!

[hannojg]

Hm, did it? I think the PR was reverted

• [CP Staging] Revert "Search suffix tree implementation" #51177

[hannojg]

I am currently working on re-applying + fixing issues

[sakluger]

No updates, we're still working on getting this fixed.

[muttmuure]

How is this going @hannojg?

[hannojg]

New PR will be ready in a moment:

• Search suffix tree implementation #51954

[sakluger]

Hey @hannojg, what's the latest? Looks like there is a bit more cleanup needed before we can merge the PR you linked above?

[hannojg]

Yes exactly, would be awesome if we could work through that list here:

#51954 (comment)

After those are merged I can work on the next set of cleanup PRs. The final goal is to cleanup the existing code so that i can reuse a stable ordering logic for the new performance PR :)

[sakluger]

Latest update here: #51954 (comment)

• There is one cleanup PR left, then a few additional changes needed to fix an ordering bug.

[sakluger]

The final cleanup PR has been created (more info: #51954 (comment))

[sakluger]

As of yesterday, cleanups are complete. Now Hanno is working on fixing the Search suffix tree implementation PR. Latest update here: #51954 (comment)