Intermittent ENOENT Errors During Parallel Metadata Retrieval with sf-cli #2871
Labels
validated
Version information for this issue has been validated
Comments
|
Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support. |
|
Hello @Alfystar 👋 None of the versions of Shared: Update to the latest version of Salesforce CLI (docs) and confirm that you're still seeing your issue. After updating, share the full output of |
github-actions
bot
added
more information required
|
Hey @Alfystar, sorry I am not sure I fully understand the issue. I have some questions for you:
If you were able to create a Github Repo that made it easy to reproduce this issue it would be a huge help. Maybe start from Dreamhouse and add a bunch of fake metadata? |
|
Hey @iowillhoit , I appreciate your questions and the opportunity to clarify. Initially, I attempted to use the Consequently, I decided that the list obtained from Regarding serial retrieval, I observed that running all 99 retrieves in parallel significantly increases the error rate. Reducing this number decreases the errors. I’ve tried with two, but errors still persist. Additionally, I believe I’ve encountered a bug with certain metadata that forces the tool to save results in the code directory instead of the support directory, where I would then have the chance to supervise and possibly perform diffs before aligning the directories. The solution of For my tests, I limited myself to the first N elements of each list to fully test the code. However, the goal is to download everything, and if I were to do it sequentially, it might take days. In my ORG, we have a total of 23K resources (I work for one of your clients, which I’m told has the largest resource footprint. I won’t add more for privacy reasons). As for replicability, I would be happy to run our script on Dreamhouse with maximum logging. However, I would ask you to provide a way to load the org with various types of metadata, as I wouldn’t know how to create everything that exists. |
|
Good morning. I run my script directly inside Dreamhouse scrach org, without any add. Parrallel output:I ran in parallel this bash command, each block in a different bash shell, using python: #Command to retrieve resources 'classes'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_ApexClass.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'apexEmailNotifications'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_ApexEmailNotifications.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'triggers'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_ApexTrigger.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'appMenus'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_AppMenu.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'assignmentRules'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_AssignmentRules.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'aura'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_AuraDefinitionBundle.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'autoResponseRules'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_AutoResponseRules.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'objects'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_CustomObject.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'cleanDataServices'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_CleanDataService.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'communities'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_Community.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'contentassets'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_ContentAsset.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'corsWhitelistOrigins'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_CorsWhitelistOrigin.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'applications'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_CustomApplication.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'labels'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_CustomLabels.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'objectTranslations'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_CustomObjectTranslation.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'tabs'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_CustomTab.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'duplicateRules'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_DuplicateRule.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'entitlementProcesses'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_EntitlementProcess.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'escalationRules'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_EscalationRules.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'flexipages'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_FlexiPage.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'flows'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_Flow.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'flowDefinitions'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_FlowDefinition.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'homePageLayouts'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_HomePageLayout.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'iframeWhiteListUrlSettings'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_IframeWhiteListUrlSettings.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'installedPackages'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_InstalledPackage.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'layouts'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_Layout.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'letterhead'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_Letterhead.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'lwc'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_LightningComponentBundle.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'managedContentTypes'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_ManagedContentType.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'matchingRules'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_MatchingRules.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'milestoneTypes'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_MilestoneType.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'notificationTypeConfig'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_NotificationTypeConfig.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'permissionsets'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_PermissionSet.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'profiles'
mkdir -p force-app/main/default/profiles ; mv 'force-app/main/default/profiles' 'fullRetrieve_Orgs/dreamhouse/tmp_Profile'
sf force source read --target-org dreamhouse -m 'Profile:Admin'
mv 'force-app/main/default/profiles' 'fullRetrieve_Orgs/dreamhouse/retrievePack' && mv 'fullRetrieve_Orgs/dreamhouse/tmp_Profile' 'force-app/main/default/profiles'
#Command to retrieve resources 'profilePasswordPolicies'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_ProfilePasswordPolicy.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'profileSessionSettings'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_ProfileSessionSetting.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'quickActions'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_QuickAction.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'remoteSiteSettings'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_RemoteSiteSetting.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'reportTypes'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_ReportType.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'restrictionRules'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_RestrictionRule.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'roles'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_Role.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'samlssoconfigs'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_SamlSsoConfig.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'settings'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_Settings.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'sharingRules'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_SharingRules.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'staticresources'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_StaticResource.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'topicsForObjects'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_TopicsForObjects.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'translations'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_Translations.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
#Command to retrieve resources 'workflows'
sf project retrieve start --manifest fullRetrieve_Orgs/dreamhouse/package_Workflow.xml --output-dir fullRetrieve_Orgs/dreamhouse/retrievePack
And this is the output with 9 worker: and as far as I know, the 47 packages created, the profiles I make with the CRUD API, are correctly created, and have the response of your tool as the member name. I would add that the metadata that did not go into error was retrieved correctly as far as I can see, as I expected. One worker runI finally tried running the job, limiting the number of workers to 1. Personal conclusionsFrom what I can understand, I believe that on these 3 metadata ( The others, however, are fine in themselves, but the code is not designed to execute read operations in parallel. I am waiting for your response. |
|
Hello @Alfystar, thank you for taking the time to try different approaches on this. Ultimately I agree with the conclusions you shared. The code may not be designed to handle parallel retrieves well. Here is a similar issue related to running parallel deploys (#2786) It is worth pointing out that certain metadata types have restriction on how they can be deployed/retrieved. For example, What happens if you use a different |
|
Hi @iowillhoit Cmd list bash:Output:From what I see I think you got it!
About the restriction for certain metadata types, can you share with me one list or documentation where this constraint are listed? This type of information for my project are gold! |
|
Hi @mdonnalley We are using it for the first time in Prod these days, and we believe that with this new tool, the development and maintenance work of the repository in relation to the org will be greatly facilitated. Thank you for your help, you have been invaluable! |
|
Hey, I think that this week's RC fixed another bug in parallel read/write of config files. If you still have this script around, I'd love to know if it makes this use case work, too. |
|
Hi! Thanks for the update! ❤️ |
|
Hi @mshanemc I'm sorry to tell you that since I presented the problem, I've written too much code to be able to restore the original situation. So far I've solved it by having each retry run in a different folder, and only at the end do I recombine them all together, moreover this approach has allowed me to better manage retrieves that fail and do the automatic retry without interfering with others, I'm sorry, if I manage to recreate a problem I'll write here 😄 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Summary
Dear sf-cli Team,
I am encountering an intermittent issue when attempting to perform parallel metadata retrievals using the sf-cli. My goal is to retrieve different types of resources in parallel to avoid exceeding the 10,000 resource limit per connection.
To obtain the list of resources present in a specific ORG, I use the following command:
sf org list metadata --metadata-type {metaType} --target-org {self.alias} --jsonSubsequently, I create various
package.xmlfiles, one for each resource list. For special metadata such as profiles, I utilize a plugin developed by your team:sfdx-plugin-source-read, which has been quite good, albeit with some limitations.Ultimately, my code generates all the
package.xmlfiles correctly. However, due to the large number of retrieves (99 different non-empty types in my case), I decided to execute them in parallel on the machine, as they are separate connections. This approach led me to discover that, randomly and unpredictably, some resources fail during command execution, claiming that the directory does not exist:…and other similar errors.
However, if I execute the command on the same terminal without any additional code generation, it sometimes works and other times it reproduces the same error. In such cases, creating the directory with mkdir -p allows the command to execute successfully, but the created directory disappears after the process.
I believe there might be an underlying bug of some sort. Could I receive support on this matter?
The thing that strangest me the most is that when I see these errors, from the error it seems that the retrieve is trying to save the output in default/main, but having specified the output directory, this situation shouldn't appear even in the error, I suppose.
Thank you for your assistance.
Best regards, Emanuele Alfano
System Information
{ "architecture": "darwin-arm64", "cliVersion": "@salesforce/cli/2.41.8", "nodeVersion": "node-v20.12.2", "osVersion": "Darwin 23.4.0", "rootPath": "/Users/ea_enel/.local/share/sf/client/2.41.8-dd2c629", "shell": "zsh", "pluginVersions": [ "@oclif/plugin-autocomplete 3.0.17 (core)", "@oclif/plugin-commands 3.3.2 (core)", "@oclif/plugin-help 6.0.21 (core)", "@oclif/plugin-not-found 3.1.8 (core)", "@oclif/plugin-plugins 5.0.19 (core)", "@oclif/plugin-search 1.0.23 (core)", "@oclif/plugin-update 4.2.11 (core)", "@oclif/plugin-version 2.0.18 (core)", "@oclif/plugin-warn-if-update-available 3.0.16 (core)", "@oclif/plugin-which 3.1.9 (core)", "@salesforce/cli 2.41.8 (core)", "apex 3.1.11 (core)", "auth 3.6.9 (core)", "data 3.3.4 (core)", "deploy-retrieve 3.6.13 (core)", "info 3.2.7 (core)", "limits 3.3.5 (core)", "marketplace 1.2.5 (core)", "org 4.1.8 (core)", "packaging 2.4.2 (core)", "schema 3.3.6 (core)", "settings 2.2.6 (core)", "sobject 1.4.1 (core)", "source 3.3.9 (core)", "telemetry 3.3.5 (core)", "templates 56.2.5 (core)", "trust 3.6.8 (core)", "user 3.5.5 (core)", "sfdx-plugin-source-read 1.2.0 (user)" ] }More information
The 2 package.xml i use are:
package_Layout.xml:
package_ExperienceBundle.xml:
in my test scene, i retrive only 1 element, when the problem will be fix, i want retrieve all the element detected with
sf org list metadata --metadata-typeThe text was updated successfully, but these errors were encountered: