Unexpected end of JSON input and Lock file is already being held when deploying in parallel

[MarcTCruz]

Note

• You've searched both open and closed issues for related posts.
  JsonParseError: Parsing --target-org intermittent error on running sf commands #2423

• You understand that GitHub Issues don't adhere to any agreement or SLA.
  Yes.

Summary

I have two different organizations and the mission to deploy most or possibly all from B to A.
There are lots of dependencies and 18k+ xml(totaling 20k+ resouces), so I tried to deploy using concurrency from JS and parallelism since it calls many CLI. Then such script would keep running until nothing has been changed or everything has been deployed.

Steps To Reproduce

Have possible any org(I am using a developer sandbox) with lots(10k+?) of data to be deploying, divide them in small buckets (e.g.: 100) and issue them in parallel.
Command used: sf project deploy start --source-dir "dir0" "dirN" ... --json

Expected result

Deploy gracefully.

Actual result

"stack": "JsonParseError: Unexpected end of JSON input\n at JsonParseError.create (/home/mtpc/.local/bin/sf_files/node_modules/@salesforce/kit/lib/errors.js:51:20)\n at parseJson (/home/mtpc/.local/bin/sf_files/node_modules/@salesforce/kit/lib/json.js:28:39)\n at parseJsonMap (/home/mtpc/.local/bin/sf_files/node_modules/@salesforce/kit/lib/json.js:67:18)\n at DeployCache.read (/home/mtpc/.local/bin/sf_files/node_modules/@salesforce/core/lib/config/configFile.js:174:52)\n at async DeployCache.init (/home/mtpc/.local/bin/sf_files/node_modules/@salesforce/core/lib/config/ttlConfig.js:40:26)\n at async DeployCache.create (/home/mtpc/.local/bin/sf_files/node_modules/@salesforce/kit/lib/creatable.js:57:9)\n at async DeployCache.set (file:///home/mtpc/.local/bin/sf_files/node_modules/@salesforce/plugin-deploy-retrieve/lib/utils/deployCache.js:31:23)\n at async executeDeploy (file:///home/mtpc/.local/bin/sf_files/node_modules/@salesforce/plugin-deploy-retrieve/lib/utils/deploy.js:115:5)\n at async DeployMetadata.run (file:///home/mtpc/.local/bin/sf_files/node_modules/@salesforce/plugin-deploy-retrieve/lib/commands/project/deploy/start.js:192:28)\n at async DeployMetadata._run (/home/mtpc/.local/bin/sf_files/node_modules/@oclif/core/lib/command.js:304:22)",

System Information

bash
xterm-256color

Linux 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64 GNU/Linux

{
  "architecture": "linux-x64",
  "cliVersion": "@salesforce/cli/2.32.8",
  "nodeVersion": "node-v20.11.1",
  "osVersion": "Linux 6.1.0-18-amd64",
  "rootPath": "/home/mtpc/.local/bin/sf_files",
  "shell": "bash",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 3.0.12 (core)",
    "@oclif/plugin-commands 3.2.0 (core)",
    "@oclif/plugin-help 6.0.17 (core)",
    "@oclif/plugin-not-found 3.0.14 (core)",
    "@oclif/plugin-plugins 4.3.2 (core)",
    "@oclif/plugin-search 1.0.20 (core)",
    "@oclif/plugin-update 4.2.0 (core)",
    "@oclif/plugin-version 2.0.14 (core)",
    "@oclif/plugin-warn-if-update-available 3.0.13 (core)",
    "@oclif/plugin-which 3.1.3 (core)",
    "@salesforce/cli 2.32.8 (core)",
    "apex 3.0.26 (core)",
    "auth 3.3.21 (core)",
    "data 3.1.7 (core)",
    "deploy-retrieve 3.2.27 (core)",
    "info 3.0.28 (core)",
    "limits 3.1.13 (core)",
    "marketplace 1.0.26 (core)",
    "org 3.4.1 (core)",
    "packaging 2.1.11 (core)",
    "schema 3.1.7 (core)",
    "settings 2.0.31 (core)",
    "sobject 1.1.17 (core)",
    "source 3.1.18 (core)",
    "telemetry 3.1.15 (core)",
    "templates 56.0.21 (core)",
    "trust 3.3.16 (core)",
    "user 3.3.1 (core)"
  ]
}

Additional information

JsonParseError
output3.txt

Lock file is already being held
output11.txt

[github-actions]

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

[github-actions]

Hello @MarcTCruz 👋 None of the versions of sf you shared match the latest release.

Shared: 2.32.8, 2.32.8
Latest: 2.33.3

Update to the latest version of Salesforce CLI (docs) and confirm that you're still seeing your issue.
You can also try the rc and nightly releases! (docs)

After updating, share the full output of sf version --verbose --json

[github-actions]

Hello @MarcTCruz 👋 None of the versions of sf you shared match the latest release.

Shared: 2.32.8
Latest: 2.33.3

Update to the latest version of Salesforce CLI (docs) and confirm that you're still seeing your issue.
You can also try the rc and nightly releases! (docs)

After updating, share the full output of sf version --verbose --json

[MarcTCruz]

Updated and the problem persists:

{
  "architecture": "linux-x64",
  "cliVersion": "@salesforce/cli/2.33.3",
  "nodeVersion": "node-v20.11.1",
  "osVersion": "Linux 6.1.0-18-amd64",
  "rootPath": "/home/mtpc/.local/bin/sf_files",
  "shell": "bash",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 3.0.13 (core)",
    "@oclif/plugin-commands 3.2.0 (core)",
    "@oclif/plugin-help 6.0.18 (core)",
    "@oclif/plugin-not-found 3.0.14 (core)",
    "@oclif/plugin-plugins 4.3.5 (core)",
    "@oclif/plugin-search 1.0.20 (core)",
    "@oclif/plugin-update 4.2.0 (core)",
    "@oclif/plugin-version 2.0.14 (core)",
    "@oclif/plugin-warn-if-update-available 3.0.14 (core)",
    "@oclif/plugin-which 3.1.4 (core)",
    "@salesforce/cli 2.33.3 (core)",
    "apex 3.0.30 (core)",
    "auth 3.3.23 (core)",
    "data 3.1.9 (core)",
    "deploy-retrieve 3.2.31 (core)",
    "info 3.0.31 (core)",
    "limits 3.1.13 (core)",
    "marketplace 1.0.27 (core)",
    "org 3.4.3 (core)",
    "packaging 2.1.12 (core)",
    "schema 3.1.7 (core)",
    "settings 2.0.31 (core)",
    "sobject 1.1.18 (core)",
    "source 3.1.20 (core)",
    "telemetry 3.1.16 (core)",
    "templates 56.0.21 (core)",
    "trust 3.3.16 (core)",
    "user 3.3.3 (core)"
  ]
}

Edit:
After updating I tried it and error, then I cleaned all Salesforce cache folders, I am till now unable to reproduce, but I will keep it open and try again. In some days I come back with the info.

[mshanemc]

Hey. I agree this might be a bug (kicking off enough parallel deployments to cause our lockfile problems).

But even if we fixed that, a single org is going to queue deployments and deploy them one at a time. So your idea (split deployments into lots of small pieces to run in parallel) wouldn't work.

sf project deploy start --source-dir "dir0" "dirN" ... if they're all in the same command, the CLI will turn them into a single deployment, not 1 deployment per directory.

I guess I'd like more information about what you're trying to do and why. And if you have a reproducer, that'd be great.

[github-actions]

This issue has not received a response in 7 days. It will auto-close in 7 days unless a response is posted.

[MarcTCruz]

Hey. I agree this might be a bug (kicking off enough parallel deployments to cause our lockfile problems).

But even if we fixed that, a single org is going to queue deployments and deploy them one at a time. So your idea (split deployments into lots of small pieces to run in parallel) wouldn't work.

sf project deploy start --source-dir "dir0" "dirN" ... if they're all in the same command, the CLI will turn them into a single deployment, not 1 deployment per directory.

I guess I'd like more information about what you're trying to do and why. And if you have a reproducer, that'd be great.

I understand the deploy process is queued in backend. I am trying with it to make the deploying process a tiny faster since there are lots of dependencies on some different projects to be merged. The uploading part being in parallel instead of starting in sequence may be some improvement in this scenario.

[github-actions]

This issue has not received a response in 7 days. It will auto-close in 7 days unless a response is posted.