GitHub Action SF CLI Running Tests - Fatal Error

[firatesmer]

Hi,
I'm running a regular SF command which starts the deploy command with dry-run. The very same command runs in my local machine (tried in both Mac & Windows) successfully but fails in GitHub CI/CD action, which is a Linux machine.

SF CLI Command:
sf project deploy start --manifest manifest/package.xml --dry-run --wait 60 --test-level RunSpecifiedTests --tests Test1 --tests Test2 --tests Test3 --json --ignore-warnings

Steps To Reproduce

1. Download Salesforce CLI
2. Execute sf project deploy start command

Expected result

Successfully running all the tests and reporting the results.

Actual result

Returns status 1 without any explanation except FATAL ERROR on UI. There are 20 test classes specified in that command but all the test classes have 0 test coverage.

Failed Run (from GitHub action)

Success Run (from local machine - Mac)

System Information

{
  "architecture": "linux-x64",
  "cliVersion": "@salesforce/cli/2.23.20",
  "nodeVersion": "node-v20.10.0",
  "osVersion": "Linux 6.2.0-1018-azure",
  "rootPath": "/home/runner/work/ePaaS-1.0/ePaaS-1.0/sf",
  "shell": "bash",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 3.0.4 (core)",
    "@oclif/plugin-commands 3.1.0 (core)",
    "@oclif/plugin-help 6.0.9 (core)",
    "@oclif/plugin-not-found 3.0.7 (core)",
    "@oclif/plugin-plugins 4.1.14 (core)",
    "@oclif/plugin-search 1.0.10 (core)",
    "@oclif/plugin-update 4.1.7 (core)",
    "@oclif/plugin-version 2.0.11 (core)",
    "@oclif/plugin-warn-if-update-available 3.0.8 (core)",
    "@oclif/plugin-which 3.0.14 (core)",
    "@salesforce/cli 2.23.20 (core)",
    "apex 3.0.14 (core)",
    "auth 3.1.0 (core)",
    "data 3.0.13 (core)",
    "deploy-retrieve 2.2.12 (core)",
    "info 3.0.13 (core)",
    "limits 3.1.0 (core)",
    "marketplace 1.0.15 (core)",
    "org 3.2.0 (core)",
    "packaging 2.0.1 (core)",
    "schema 3.1.0 (core)",
    "settings 2.0.14 (core)",
    "sobject 1.0.7 (core)",
    "source 3.0.10 (core)",
    "telemetry 3.1.8 (core)",
    "templates 56.0.8 (core)",
    "trust 3.2.5 (core)",
    "user 3.2.0 (core)"
  ]
}

Does anyone have any ideas? Thanks in advance.

[github-actions]

Hello @firatesmer 👋 It looks like you didn't include the full Salesforce CLI version information in your issue.
Please provide the output of version --verbose --json for the CLI you're using (sf or sfdx).

A few more things to check:

• Make sure you've provided detailed steps to reproduce your issue.
  • A repository that clearly demonstrates the bug is ideal.
• Make sure you've installed the latest version of Salesforce CLI. (docs)
  • Better yet, try the rc or nightly versions. (docs)
• Try running the doctor command to diagnose common issues.
• Search GitHub for existing related issues.

Thank you!

[github-actions]

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

[shetzel]

You're not providing enough information (see above). Have you checked the deploy status in the org UI? To me, it looks like the CLI is behaving as expected and reporting an error that happened during deployment.

[firatesmer]

Hi @shetzel,
I've updated the ticket. This is the deploy status in the org UI.

Again, when I execute the very same command on my locals (Mac & Win) with the same user, it works as expected. All the tests run and pass and we receive success, 0 as status code.

[shetzel]

You can try getting debug output during the CI run by appending the --dev-debug flag and inspecting the output. Another way to get that output is by running the same command through the doctor command so it generates the files for you. E.g., sf doctor --command "sf project deploy start --manifest manifest/package.xml --dry-run --wait 60 --test-level RunSpecifiedTests --tests Test1 --tests Test2 --tests Test3 --json --ignore-warnings"

Another option is to set the SF_MDAPI_TEMP_DIR env var in your CI script to some known location and check the contents of the directory and manifest to ensure it looks correct.

Is it the same org being deployed to? If everything else is the same as with the successful run then I would debug the CI script.

[firatesmer]

Thanks for the advice. To be honest, it's hard to extract the output of the doctor command since there are files saved into the temp folder of runner. I'll continue later today and share my update but is there an easier way to get the result from the doctor command?
By the way, yes, that is the same org with the same user and same package file.

[shetzel]

The doctor command accepts an --output-dir flag to specify a location. Alternatively, you can set these env vars in the CI script: DEBUG=* and SF_LOG_LEVEL=trace and the debug output will be part of the runner logs.

This doesn't seem like a CLI bug. We have lots of CI tests running via GitHub actions using Ubuntu and Windows that run similar deploy and test commands. It's much more likely that there is something wrong with the CI script.

[mshanemc]

closed because stale/no response

[firatesmer]

Hey,
Sorry for the late response. I wasn't able to find the root cause of the issue but I just found a workaround by changing my CICD script. I was executing the command as command + params (flags) but now I execute the whole thing as command.

I changed this

  let args = [...deploymentAttributes, "--json", "--ignore-warnings"];

  await exec.exec("sf project deploy start", args, {
    listeners: { stderr: errorHandler, stdout: successHandler },
    ignoreReturnCode: true
  });

To this

  let cmd = "sf project deploy start " + deploymentAttributes.join(" ") + " --json --ignore-warnings";

  await exec.exec(cmd, [], {
    listeners: { stderr: errorHandler, stdout: successHandler },
    ignoreReturnCode: true
  });

Thank you both for your time and support.