High cpu usage

[pavel-lobach]

Salesforce CLI utilizes extremely huge CPU percentage when it executed any command. We execute commands from Node.js using spawn process. The process is built by that way so almost always at least one of the commands is running, either 'sf project retrieve start' or 'sf package installed list'.

As I understand when the project is being retrieved, it just waits for the answer from Salesforce, but in fact it takes almost full CPU for the entire process, which may take about 60 minutes. So this is weird, the cli should be almost idle at the most time.

Here is the screenshot with CPU usage where 3 different processes are retrieving the metadata in separate docker containers. As you see almost 100% of the time the cpu load is about 100% (blue graph).

It is hard to give you the steps to reproduce, do you have any ideas what can we check?

[github-actions]

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

[github-actions]

Hello @pavel-lobach 👋 It looks like you didn't include the full Salesforce CLI version information in your issue.
Please provide the output of version --verbose --json for the CLI you're using (sf or sfdx).

A few more things to check:

• Make sure you've provided detailed steps to reproduce your issue.
  • A repository that clearly demonstrates the bug is ideal.
• Make sure you've installed the latest version of Salesforce CLI. (docs)
  • Better yet, try the rc or nightly versions. (docs)
• Try running the doctor command to diagnose common issues.
• Search GitHub for existing related issues.

Thank you!

[pavel-lobach]

The behavior is the same on the number of recent versions , including the most recent one 2.15.9.

[iowillhoit]

@pavel-lobach, there is other information in sf version --verbose --json that is useful for debugging. Please provide the output.

[pavel-lobach]

@iowillhoit This is the output inside the container with CLI v 2.11.8:

{
"cliVersion": "@salesforce/cli/2.11.8",
"architecture": "linux-x64",
"nodeVersion": "node-v19.9.0",
"osVersion": "Linux 6.1.55-75.123.amzn2023.x86_64",
"shell": "ash",
"rootPath": "/app/node_modules/@salesforce/cli",
"pluginVersions": [
"@oclif/plugin-autocomplete 2.3.8 (core)",
"@oclif/plugin-commands 2.2.26 (core)",
"@oclif/plugin-help 5.2.19 (core)",
"@oclif/plugin-not-found 2.4.1 (core)",
"@oclif/plugin-plugins 3.8.2 (core)",
"@oclif/plugin-search 0.0.22 (core)",
"@oclif/plugin-update 3.2.3 (core)",
"@oclif/plugin-version 1.3.10 (core)",
"@oclif/plugin-warn-if-update-available 2.1.0 (core)",
"@oclif/plugin-which 2.2.32 (core)",
"@salesforce/cli 2.11.8 (core)",
"apex 2.3.15 (core)",
"auth 2.8.17 (core)",
"data 2.5.12 (core)",
"deploy-retrieve 1.18.3 (core)",
"info 2.6.45 (core)",
"limits 2.3.35 (core)",
"login 1.2.31 (core)",
"marketplace 0.2.2 (core)",
"org 2.10.8 (core)",
"packaging 1.27.1 (user)",
"schema 2.3.28 (core)",
"settings 1.4.30 (core)",
"sobject 0.2.7 (core)",
"source 2.10.36 (core)",
"telemetry 2.3.4 (core)",
"templates 55.5.13 (core)",
"trust 2.6.15 (core)",
"user 2.3.32 (core)"
]
}

[github-actions]

Hello @pavel-lobach 👋 None of the versions of sf you shared match the latest release.

Shared: 2.11.8
Latest: 2.15.9

Update to the latest version of Salesforce CLI (docs) and confirm that you're still seeing your issue.
You can also try the rc and nightly releases! (docs)

After updating, share the full output of sf version --verbose --json

[pavel-lobach]

I confirm that the issue is there on the latest version. I had to downgrade to check if it helps or not. But it does not help.

Here is the output with the recent one.

{
"architecture": "linux-x64",
"cliVersion": "@salesforce/cli/2.15.9",
"nodeVersion": "node-v19.9.0",
"osVersion": "Linux 6.1.55-75.123.amzn2023.x86_64",
"rootPath": "/app/node_modules/@salesforce/cli",
"shell": "ash",
"pluginVersions": [
"@oclif/plugin-autocomplete 2.3.10 (core)",
"@oclif/plugin-commands 3.0.4 (core)",
"@oclif/plugin-help 6.0.4 (core)",
"@oclif/plugin-not-found 3.0.2 (core)",
"@oclif/plugin-plugins 3.9.4 (core)",
"@oclif/plugin-search 1.0.4 (core)",
"@oclif/plugin-update 4.1.3 (core)",
"@oclif/plugin-version 2.0.3 (core)",
"@oclif/plugin-warn-if-update-available 3.0.2 (core)",
"@oclif/plugin-which 3.0.6 (core)",
"@salesforce/cli 2.15.9 (core)",
"apex 2.3.20 (core)",
"auth 2.8.25 (core)",
"data 2.6.1 (core)",
"deploy-retrieve 1.19.2 (core)",
"info 2.6.51 (core)",
"limits 2.3.41 (core)",
"login 1.2.39 (core)",
"marketplace 0.3.1 (core)",
"org 2.11.5 (core)",
"packaging 1.27.1 (user)",
"schema 2.3.31 (core)",
"settings 1.4.36 (core)",
"sobject 0.2.14 (core)",
"source 2.10.45 (core)",
"telemetry 2.3.8 (core)",
"templates 55.5.17 (core)",
"trust 2.6.22 (core)",
"user 2.3.39 (core)"
]
}

[shetzel]

One easy thing to change would be the nodejs version. v19 isn't supported anymore so maybe switching to v20.9.0 would have different results.

[pavel-lobach]

Will try to upgrade the node. But not sure that node is the case.

I suppose the CLI always pulls the results from SF. How often? Is it possible to reduce the frequency of pulling of the results? Because to be clear, the command takes about 45-55 minutes. And all the time the process takes so much cpu.

[pavel-lobach]

One more observation. While the container is waiting for the result from Salesforce (about 30-35 minutes) it takes higher CPU % than the process consumes when it starts to extract the received zip file. So the extraction of the zip is easier than the metadata retrieve )

[mshanemc]

I'm able to see that hitting about 300% (that's 3 cores) on a large, slow retrieve, but not the whole time.

It's ok for a while (around 2% CPU, polling freq is 500ms) and responses are small like

[08:48:12.891] DEBUG (sf:MetadataApiRetrieve): 0: "Retrieve status" 1: { "done": false, "id": "09S4p000009D2P2EAK", "status": "InProgress", "success": false, "zipFile": "", "fileProperties": [], "messages": [] }

Then when it gets the real response with the zip file, etc is when the CPU use got crazy for me. And that's kinda reasonable since there's a lot going on (multiple unzipping steps, xml/js conversion/decomposition, a lot of fs operations, and source tracking stuff.

Am going close this, and encourage you to use a supported node version. see https://nodejs.org/en/about/previous-releases

[pavel-lobach]

What we saw is completely opposite things when the CLI commands are run in docker containers. Almost 99% of the time the CPU usage is about 100+%, but when the CLI starts to extract and process, then the load reduces to ~30-50%. This is very strange. Something consumes the CPU when the CLI should be idle.

So the questions that may help are:

1. How can I control the poll frequency?
2. Is there container mode set somehow for CLI? I found SF_CONTAINER_MODE var but not one word anywhere documents it.
3. Please confirm if it is ok or not that 2VCPU node with 4 docker containers running execute concurrent metatada retrieve in parallel, and in this cast the VCPU usage of the node is about 300% on AWS and because of that, the retrieve never finishes. The CPU usage is the whole time, not only when is extracts. The entire retrieve command takes about 30-40 minutes.