force-app fails to deploy to scratch org: background job is being executed

[Charles-Revelle]

Our CI/CD process uses Jenkins builds and scratch orgs to validate commits. The build creates a scratch org and attempts to deploy all folders from the repository. Since Spring '23 (2/15/23) the majority of our builds have been failing on a step to deploy the force-app folder. I have a salesforce support case open that is stalling, but heard that others are seeing this.

The errors received are on files that have not been updated:
"Cannot update or delete a picklist while a picklist background job is being executed."

Steps To Reproduce:

Start Jenkins build with these steps:
CHECKOUT CHANGES IN PULL REQUEST
CHECKING OUT REPOSITORY
CREATE TEMPORARY JWT SERVER KEY
AUTHORIZE DEV HUB TO CREATE SCRATCH-ORG
CREATE SCRATCH-ORG
INSTALLING Marketing MANAGED PACKAGE
DEPLOY MANUAL-METADATA/ONE-TIME-INSTALL FOLDER
DEPLOY MANUAL-METADATA/PRE-INSTALL FOLDER
DEPLOY FORCE-APP FOLDER
Fails at this stage

Expected result

FORCE-APP folder deploys successfully, or with a valid error if something incorrect was committed

Actual result

FORCE-APP deploy fails with this error:

System Information

• Which shell/terminal are you using? cmd.exe

• Using sfdx

  "cliVersion": "sfdx-cli/7.188.1",
  "architecture": "win32-x64",
  "nodeVersion": "node-v18.13.0",
  "pluginVersions": [
  "@oclif/plugin-autocomplete 1.4.5 (core)",
  "@oclif/plugin-commands 2.2.7 (core)",
  "@oclif/plugin-help 5.2.4 (core)",
  "@oclif/plugin-not-found 2.3.18 (core)",
  "@oclif/plugin-plugins 2.3.0 (core)",
  "@oclif/plugin-search 0.0.13 (core)",
  "@oclif/plugin-update 3.1.3 (core)",
  "@oclif/plugin-version 1.2.1 (core)",
  "@oclif/plugin-warn-if-update-available 2.0.26 (core)",
  "@oclif/plugin-which 2.2.14 (core)",
  "alias 2.2.1 (core)",
  "apex 1.5.1 (core)",
  "auth 2.4.2 (core)",
  "community 2.2.1 (core)",
  "config 1.5.1 (core)",
  "custom-metadata 2.1.1 (core)",
  "data 2.2.1 (core)",
  "generator 2.0.17 (core)",
  "info 2.4.2 (core)",
  "limits 2.3.3 (core)",
  "org 2.3.2 (core)",
  "packaging 1.14.3 (core)",
  "schema 2.3.1 (core)",
  "signups 1.4.0 (core)",
  "source 2.5.1 (core)",
  "telemetry 2.1.1 (core)",
  "templates 55.4.1 (core)",
  "trust 2.4.1 (core)",
  "user 2.3.2 (core)",
  "@salesforce/sfdx-plugin-lwc-test 1.0.1 (core)",
  "salesforce-alm 54.8.5 (core)"
  ],
  "osVersion": "Windows_NT 10.0.19045",
  "shell": "cmd.exe",
  "rootPath": "C:\Users\crevelle\AppData\Local\sfdx\client\7.188.1-4a4bb9c

[github-actions]

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

[peternhale]

@Charles-Revelle have reviewed this article?

[Charles-Revelle]

@Charles-Revelle have reviewed this article?

Thank you for sharing @peternhale! I have just reviewed the article, and related Stack Exchange post. I'm not seeing what suggested actions we might take. We are not trying to update the picklists that are presented in the CLI error, and the errors are only being received in CI/CD build deploys to a scratch org. I'm willing to try any fix at this point - just don't see what it is in the articles.

[peternhale]

@Charles-Revelle The error indicates that the org is somehow doing a mass update, so changes cannot be made while the update is running. At this moment, I would recommend speaking with Salesforce Support, given that the issue began at the release of Spring 23.

[github-actions]

We have determined that the issue you reported exists in code owned by another team that uses only the official support channels. To ensure that your issue is addressed, open an official Salesforce customer support ticket with a link to this issue. We encourage anyone experiencing this issue to do the same to increase the priority. We will keep this issue open for the community to collaborate on.

[bdovh]

I have the same error.

Is this an intermittent error?

[tim-c-s]

Hi @Charles-Revelle have you ever managed to somehow fix this issue or find a workaround?
We face a similar issue when trying to deploy data from multiple packages to a scratch-org.

@peternhale any chance you have additional information about this issue? Unfortunately the article you have provided in previous comments in this issue does not seem to exist anymore.
We tried to avoid running into the issue by trying to pause the deployment if there are any running background jobs. Though the according Object (ParallelJobStatus) seems to not be retrievable or queryable, so we can't check if there are any running jobs. Is there another way to query for the running picklist background jobs?
We are also thinking about implementing a fixed timeout between deployment of packages or just retry the package deployment to mitigate the issue.

Any help or additional information regarding this issue would be appreciated.
Best Regards

[bdovh]

in our case sometimes it occurs. Maybe some operation fails to finish in time before deployment? but in the subsequent execution it finishes in time, some random error and random problem looks like

[dinhphieu]

On the same boat, it happens very randomly. We have a retry method to handle this case but sometimes it doesn't even get past 5 retries, which in PR validation time equals roughly 1 hour wasted for a dev

On the other hand, most of the time this doesn't happen, and the deployment gets done in 1 shot within a few minutes. The background job for this specific picklist hasn't been touched in over 2 years either so there's definitely something not right..