Investigate workflow job failing: Typecheck/ESLint actions out of memory

[blazejkustra]

Typecheck action that failed

Some context from couple months ago when ESLint was failing: #44425

Issue Owner

Current Issue Owner: @blazejkustra

[melvin-bot]

Triggered auto assignment to @kadiealexander (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details. Please add this bug to a GH project, as outlined in the SO.

[blazejkustra]

I asked @roryabraham for assignment on #typescript-new-expensify channel, waiting for his approval for now.

[melvin-bot]

Triggered auto assignment to @thienlnam, see https://stackoverflow.com/c/expensify/questions/7972 for more details.

[roryabraham]

🟢 from me (as discussed in slack) to try and figure out how to make lint and tsc faster

[blazejkustra]

Coming back with some findings 🔍

TS server is quite fast in general, so any slowness is usually caused by the devs or complex libraries, common culprits are:

• huge union types,
• deep recursive type,
• complex generic types,
• massive files.

Currently, TS check is at around 54 seconds for me:

Baseline ~54s, ~4.5GB of memory

NODE_OPTIONS=--max_old_space_size=8192 tsc --extendedDiagnostics --incremental false

Files:                         5765
Lines of Library:             40640
Lines of Definitions:        510909
Lines of TypeScript:         353141
Lines of JavaScript:           7532
Lines of JSON:                 1267
Lines of Other:                   0
Identifiers:                 976989
Symbols:                    3123303
Types:                      2321176
Instantiations:            17286341
Memory used:               4503960K
Assignability cache size:   2050430
Identity cache size:          40895
Subtype cache size:           41483
Strict subtype cache size:   384382
I/O Read time:                0.83s
Parse time:                   1.15s
ResolveModule time:           0.53s
ResolveTypeReference time:    0.01s
ResolveLibrary time:          0.00s
Program time:                 2.86s
Bind time:                    0.61s
Check time:                  49.83s
I/O Write time:               0.00s
printTime time:               0.08s
Emit time:                    0.08s
Total time:                  53.38s

Investigation

That got me thinking what types are so complicated that E/App takes so long to compile... I looked at the AST tree and saw a lot of onyx symbols used:

  // run a script to count all references
  [ '__type', 213203 ],
  [ 'BaseMappingStringKeyAndSelector', 190291 ], <- onyx
  [ 'BaseMappingFunctionKeyAndSelector', 190291 ], <- onyx
  [ 'Selector', 185896 ], <- onyx
  [ 'Mapping', 178545 ], <- onyx
  [ '__object', 52927 ],
  [ 'RelativeIndexable', 39501 ],
  [ '__jsxAttributes', 39028 ],
  [ 'Array', 35424 ],
  [ 'CollectionMapping', 25118 ], <- onyx
  [ 'G', 20500 ],  <- possibly onyx
  [ 'at', 16904 ],
  [ '__function', 15337 ],
  [ 'String', 10594 ],
  [ 'U', 8215 ],
  [ 'S', 8134 ],

These symbols are used in withOnyx HOC, so I went ahead and prepared a script to automatically migrate most of them to use useOnyx hook instead, and run typecheck with diagnostics again:

~38s, ~2.8GB of memory (40% less memory used 🚀 )

Files:                         5769
Lines of Library:             41147
Lines of Definitions:        510910
Lines of TypeScript:         355488
Lines of JavaScript:           7532
Lines of JSON:                 1269
Lines of Other:                   0
Identifiers:                 979940
Symbols:                    2573629
Types:                      1208298
Instantiations:             5587137
Memory used:               2856605K
Assignability cache size:   1335744
Identity cache size:          22177
Subtype cache size:           41927
Strict subtype cache size:   373653
I/O Read time:                1.20s
Parse time:                   1.74s
ResolveModule time:           0.62s
ResolveTypeReference time:    0.01s
ResolveLibrary time:          0.01s
Program time:                 3.97s
Bind time:                    0.84s
Check time:                  33.85s
printTime time:               0.00s
Emit time:                    0.00s
Total time:                  38.67s

Here is the PR where most of withOnyx are migrated, but there are still around 40 HOCs in use so 38s is not the final time! 🤟

Other things to consider

• Split big files into smaller, especially these three:
  ├─ Check file /users/blazejkustra/documents/swmansion/expensify/swmapp/src/libs/actions/iou.ts (2134ms)
  ├─ Check file /users/blazejkustra/documents/swmansion/expensify/swmapp/src/libs/actions/policy/policy.ts (1161ms)
  ├─ Check file /users/blazejkustra/documents/swmansion/expensify/swmapp/src/libs/actions/report.ts (997ms)

  • intellisense, prettier, lint, ts are super slow in these files, splitting will help to decrease calculations while editing

• Calculating translations is a heavy operation which creates a big union which slows down TS and ESLint
  

  • we should consider rewriting it a bit, maybe instead 'common.address' keys we should use an array key ['common', 'address'] or other approach - needs more investigation

• Caching tsbuildinfo in Github actions (we already do it locally, --incremental flag)

• Split the project into smaller pieces with typescript references

  • this needs more investigations, I don't expect massive gains with this approach as it's hard to split E/App into independent modules

Other general solutions, but not worth the effort for now:

• Use interfaces instead of types (tested it locally, didn't see much difference)
• Split complex types into smaller (this helps typescript to cache them better)

Hot spots, what files takes most time to compile - we should investigate them 👀

Command:

NODE_OPTIONS=--max_old_space_size=8192 tsc --generateTrace trace --incremental false && open trace
npx @typescript/analyze-trace trace

Hot Spots:

├─ Check file /users/blazejkustra/documents/swmansion/expensify/swmapp/src/pages/workspace/accounting/netsuite/import/netsuiteimportmappingpage.tsx (651ms)
│  └─ Check deferred node from (line 96, char 13) to (line 115, char 11) (646ms)
├─ Check file /users/blazejkustra/documents/swmansion/expensify/swmapp/src/libs/errorutils.ts (620ms)
│  └─ Check expression from (line 45, char 12) to (line 45, char 87) (616ms)
│     └─ Check expression from (line 45, char 56) to (line 45, char 86) (615ms)
└─ Check file /users/blazejkustra/documents/swmansion/expensify/swmapp/src/pages/settings/profile/personaldetails/stateselectionpage.tsx (614ms)
   └─ Check deferred node from (line 92, char 36) to (line 101, char 18) (611ms)
      └─ Check expression from (line 100, char 21) to (line 100, char 51) (611ms)
         └─ Check expression from (line 100, char 39) to (line 100, char 50) (611ms)

Materials

For anybody interested, here're some useful links:

• https://github.com/microsoft/TypeScript-wiki/blob/main/Performance-Tracing.md
• https://www.youtube.com/watch?v=IzzvGrZrFSE
• https://www.edgedb.com/blog/an-approach-to-optimizing-typescript-type-checking-performance#how-do-we-quantify-the-type-checker-s-workload
• https://www.youtube.com/watch?v=ZL3z1oBZntk&t=348s

[blazejkustra]

TLDR;

1. To improve the performance of TS and decrease memory usage we should start migrating withOnyx to useOnyx.
2. Rethink our translation system as types are currently too complex
3. Split big files into smaller modules (actions/IOU.ts, actions/Report.ts, actions/Policy.ts)
4. Investigate these three files, for some reason TS is quite slow there:

├─ Check file /src/pages/workspace/accounting/netsuite/import/netsuiteimportmappingpage.tsx (651ms)
├─ Check file /src/libs/errorutils.ts (620ms)
└─ Check file /src/pages/settings/profile/personaldetails/stateselectionpage.tsx (614ms)

[melvin-bot]

This issue has not been updated in over 15 days. @thienlnam, @blazejkustra, @kadiealexander eroding to Monthly issue.

P.S. Is everyone reading this sure this is really a near-term priority? Be brave: if you disagree, go ahead and close it out. If someone disagrees, they'll reopen it, and if they don't: one less thing to do!

[blazejkustra]

I'll be monitoring the performance of TS/ESLint once we got to a point where there are far less withOnyx in the codebase. Let's leave the issue open for now.

[melvin-bot]

This issue has not been updated in over 15 days. @thienlnam, @blazejkustra, @kadiealexander eroding to Monthly issue.

P.S. Is everyone reading this sure this is really a near-term priority? Be brave: if you disagree, go ahead and close it out. If someone disagrees, they'll reopen it, and if they don't: one less thing to do!

[blazejkustra]

Once there are no withOnyx left, I'll post an update to compare TS/ESlint speed