Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: memory usage on giant repos #591

Merged
merged 10 commits into from
May 29, 2024
Merged

Conversation

mshanemc
Copy link
Contributor

What does this PR do?

  • hash files using isogit to avoid walking files in trees outside of packageDirs.
  • moveScale NUT creates a lot of local files
  • reorganize code
  • move some non-variable stuff (os, os/env dependent max file moves) in ShadowRepo to consts for all instances.

What issues does this PR fix or reference?

forcedotcom/cli#2880
@W-15840593@

test perf

on exact code from main
✔ should show 0 files in git status after moving them (10703ms)

on code from main modified with 100_000 local non-project files,
✔ should show 0 files in git status after moving them (17424ms)

this PR with 100_000 local non-project files,
✔ should show 0 files in git status after moving them (12291ms)

@mshanemc mshanemc requested a review from a team as a code owner May 23, 2024 23:03
const getInfo = async (targetTree: Walker, filenameSet: Set<string>): Promise<FileInfo[]> =>
// Unable to properly type the return value of git.walk without using "as", ignoring linter
// eslint-disable-next-line @typescript-eslint/no-unsafe-return
git.walk({
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unlike the StatusMatrix which supports a filter, .walk is going to run the map async callback on every file. That seemed fine in our integration tests, but real-world projects are potentially going to have a LOT of

  1. git history (.git)
  2. tracking (.sf/.sfdx)
  3. potentially other . files

@iowillhoit
Copy link
Contributor

QA NOTES

  • 🟢 Move files are not detected when SF_BETA_TRACK_FILE_MOVES is not set
  • 🟢 Single moved class are detected when SF_BETA_TRACK_FILE_MOVES=true
  • 🟢 Full object dir is detected
  • 🟢 Moving a decomposed md-child to a different parent is not detected as a move
Will Delete [1] files.
 Type        Fullname             Path
 ─────────── ──────────────────── ─────────────────────────────────────────────────────────────────────
 CustomField Property__c.Baths__c force-app/main/foo/objects/Property__c/fields/Baths__c.field-meta.xml

Will Deploy [1] files.
 Type        Fullname           Path
 ─────────── ────────────────── ───────────────────────────────────────────────────────────────────────
 CustomField Broker__c.Baths__c force-app/main/default/objects/Broker__c/fields/Baths__c.field-meta.xml 
  • 🟢 Moved images are detected
  • 🟢 Moving a file out of the project directory shows as a delete
  • 🟢 Moving a full object dir AND editing one internal file
    • 🟢 Commits the files that were just moved
    • 🟢 Will show the edited file as needing deployed
  • 🟢 Nuts for scale moves looks good

@iowillhoit iowillhoit merged commit 4a33284 into main May 29, 2024
32 checks passed
@iowillhoit iowillhoit deleted the sm/file-moves-memory-usage branch May 29, 2024 21:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants