sf project generate manifest returns inconsistent results

[jakubmplatek]

Note
Before you submit your issue, make sure that:

• You're using the latest version of Salesforce CLI.
• You've searched both open and closed issues for related posts.
• You've used the doctor command to diagnose common issues.
• You understand that GitHub Issues don't adhere to any agreement or SLA.
  • If you require immediate assistance, use official channels such as Salesforce Customer Support.

Summary

I'm building automation to run scheduled snapshots of all metadata in my orgs, using sf project generate manifest --from-org command as a base for generating batches of retrieve calls.
I noticed some unexpected changes listed between runs, and found that that initial package manifest generated is inconsistent: it will randomly return a different count of members for a type, or skip a type entirely.

I've tested it by running the generate manifest command in a loop against the same environment and then comparing the generated packages, and found individual runs completely missing some types, including:
ApexClass,MatchingRule,CustomObject,ValidationRule,WebLink,AppMenu,CustomField,CompactLayout,CustomPermission,ListView,CustomMetadata

Tested on different instances, both in prod and sandbox orgs, and in different sf projects, all using the latest cli (2.69.14)

Steps To Reproduce

IMPORTANT
Provide a repository that's configured to reproduce the issue. If you are unable to provide a repo, please explain why not. The more info we have from the start, the faster we can resolve your issue.
We may close your issue if you don't include proper instructions.

• Generate a project with sf project generate or fork dreamhouse-lwc.
• Provide detailed step-by-step instructions on how to reproduce the issue.

authenticate with a salesforce org and run the command a few times to generate multiple manifests (date used to autogenerate a different filename each time)
sf project generate manifest --from-org <alias> --name package_$( date +%s ).xml
compare the contents of the generated package manifests - some of them will have whole metadata types missing, or some members within a metadata type missing

note: the issue is intermittent, so it might take 5-10 runs to see significant differences

Tip

use sf doctor --create-issue to automatically fill the required information

logs generated by doctor:
logs.zip

Expected result

while testing in an org without active users (i.e. no changes to metadata), I expect each package manifest to have strictly equal contents

Actual result

components listed differ between generated manifests

Additional information

System Information

Windows 10, Git for Windows bash

{
    "architecture": "win32-x64",
    "cliVersion": "@salesforce/cli/2.69.14",
    "nodeVersion": "node-v20.11.0",
    "osVersion": "Windows_NT 10.0.19045",
    "rootPath": "C:\\Users\\jplatek\\AppData\\Roaming\\npm\\node_modules\\@salesforce\\cli",
    "shell": "C:\\Program Files\\Git\\usr\\bin\\bash.exe",
    "pluginVersions": [
      "@copado/copado-cli 3.3.0 (user)",
      "@oclif/plugin-autocomplete 3.2.11 (core)",
      "@oclif/plugin-commands 4.1.12 (core)",
      "@oclif/plugin-help 6.2.18 (core)",
      "@oclif/plugin-not-found 3.2.29 (core)",
      "@oclif/plugin-plugins 5.4.19 (core)",
      "@oclif/plugin-search 1.2.16 (core)",
      "@oclif/plugin-update 4.6.14 (core)",
      "@oclif/plugin-version 2.2.16 (core)",
      "@oclif/plugin-warn-if-update-available 3.1.24 (core)",
      "@oclif/plugin-which 3.2.19 (core)",
      "@salesforce/cli 2.69.14 (core)",
      "apex 3.6.6 (core)",
      "api 1.3.2 (core)",
      "auth 3.6.78 (core)",
      "data 3.12.3 (core)",
      "deploy-retrieve 3.15.17 (core)",
      "info 3.4.25 (core)",
      "limits 3.3.42 (core)",
      "marketplace 1.3.7 (core)",
      "org 5.2.7 (core)",
      "packaging 2.9.5 (core)",
      "schema 3.3.43 (core)",
      "settings 2.4.8 (core)",
      "sobject 1.4.46 (core)",
      "telemetry 3.6.26 (core)",
      "templates 56.3.33 (core)",
      "trust 3.7.45 (core)",
      "user 3.6.4 (core)",
      "@salesforce/sfdx-scanner 4.7.0 (user)",
      "sfdmu 4.35.1 (user)",
      "sfdx-git-delta 5.42.1 (user)"
    ]
  }

[github-actions]

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

[iowillhoit]

Hey @jakubmplatek, this is a complicated command that performs a lot of concurrent requests to attempt to gather all of the org's metadata. We are actively working on a few changes that will hopefully improve the reliability of this command. I will link to the existing work item (for internal reference). We will try to post here once this is completed for you to test against. You can also watch the Release Notes for updates.

@W-17431548@

[jakubmplatek]

Thanks for the update @iowillhoit
Is there a rough ETA on this work item? I'm working on automation that would rely on this, so need to know if I need to work on a different approach before this is ready.

[iowillhoit]

It is currently In Progress by one of our engineers but this is not a simple task. It is possible we will have something before the holiday break, but it could be early January.

sf project generate manifest is really a convenience wrapper for making sf org list metadata command calls to an org for every known metadata type and including the results in a manifest file. There are certain types that cannot be collected with a wildcard, so we need to make a lot of calls to collect them. Request retries are made but it's not perfect. If you run this command with the --dev-debug flag and/or setting the JSFORCE_LOG_LEVEL=DEBUG env var you will see what I mean :)

The idea with this work is to throttle requests to hopefully reduce connection resets. It will also provide the ability to exclude certain types of metadata (maybe with the .forceignore, maybe with a command flag).

[jakubmplatek]

I can work with that, thanks.
I personally like that it's not affected by forceignore, because of the way I use it with managed component access stuff (I want them in the manifest, but not necessarily in the repo), so a flag would be preferable. Right now I'm just filtering it while generating my batches, so it would be nice to limit step 1 to improve performance.
Have a good holiday!

[iowillhoit]

Thanks @jakubmplatek you too! I'll pass your feedback along 👍

[iowillhoit]

FYI: Sounds like we are taking the flag approach

[shetzel]

@jakubmplatek - If you'd like to test a draft version of the changes, run sf plugins install [email protected]. Then you can run the project generate manifest command with some additional flags to either include or exclude certain metadata types. Use the --metadata flag to only include specific metadata from the org. Use the --excluded-metadata flag to exclude specific metadata types from the org (such as StandardValueSet). Also, you can limit API calls made by the command to a batch size using the new env var, SF_LIST_METADATA_BATCH_SIZE. Note that this increases the command run time but should help with manifest consistency since it's not making hundreds of concurrent API calls. You'll have to experiment a bit with a batch size that works for your org and environment. Also note that all commands support a --flags-dir flag which tells the command to read flag values from the provided directory (see command help).

Please let me know if this solves your issues.

[jakubmplatek]

Looks like you've nailed it, so far. I ran my test scripts generating 8 manifests each for plugin v 3.15.17 and 3.16.2-dev*, with all other variables unchanged.

With the stable version I got 35 out of 137 types showing differences across the dataset (this is mostly the type missing from the generated package entirely)
With the dev plugin I only saw 3 out of 137 types showing differences (Dashboard, EmailTemplate, Report), which I would assume can be addressed with the batch size. Also worth noting, that none of them have been missing in any of the generated packages (just the count of members differed), although for EmailTemplate the diff is caused by it skipping the actual templates, and only listing folders. Did not delve into analysing the other two, as the member counts are on the order of 10^4-10^5

What is the default batch size when the SF_LIST_METADATA_BATCH_SIZE is not provided, so I can have a starting point for further testing?

[shetzel]

There is no batching of requests by default to match current command behavior. Batching is "opt-in". I tested with batch sizes of 50, 20, and 10 with good results. However, I was using orgs that didn't have a ton of metadata. The manifests were only about 1300 lines total and I never saw a discrepancy.

If you want some visibility of what's going on while the command is running you can set this env var: DEBUG=sf:ConnectionResolver. To see everything, DEBUG=*.

Thanks for the feedback!

[jakubmplatek]

I ran a longer loop and across 78 packages there were 3 faulty ones (not counting the discrepancies with the three big types mentioned earlier), so the issue is still there, but it's definitely more robust now. I can add a fail-check and retry/alert logic to my workflows - with a daily run this failure rate would give me a failure about once a month, so it's not a blocker anymore, plus I didn't see two consecutive runs fail, so a single retry should solve it completely for my org.

Thanks for letting me in on the testing fun!