Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running --tests with multiple --tests parameters does not work as expected #2834

Closed
4 of 9 tasks
csilva-intakedesk opened this issue Apr 19, 2024 · 6 comments
Closed
4 of 9 tasks
Labels
more information required Issue requires more information or a response from the customer validated Version information for this issue has been validated

Comments

@csilva-intakedesk
Copy link

csilva-intakedesk commented Apr 19, 2024

Note
Before you submit your issue, make sure that:

  • > - You're using the latest version of Salesforce CLI.
  • > - You've searched both open and closed issues for related posts.
  • > - You've used the doctor command to diagnose common issues.
  • > - You understand that GitHub Issues don't adhere to any agreement or SLA.
  • > - If you require immediate assistance, use official channels such as Salesforce Customer Support.

Context

Executed within Github Actions

  • Image: ubuntu-22.04
  • Version: 20240414.1.0
  npm install -g @salesforce/cli
  sf version --verbose --json
{
  "architecture": "linux-x64",
  "cliVersion": "@salesforce/cli/2.37.4",
  "nodeVersion": "node-v18.20.2",
  "osVersion": "Linux 6.5.0-1018-azure",
  "rootPath": "/usr/local/lib/node_modules/@salesforce/cli",
  "shell": "bash",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 3.0.13 (core)",
    "@oclif/plugin-commands 3.2.2 (core)",
    "@oclif/plugin-help 6.0.20 (core)",
    "@oclif/plugin-not-found 3.1.2 (core)",
    "@oclif/plugin-plugins 5.0.7 (core)",
    "@oclif/plugin-search 1.0.20 (core)",
    "@oclif/plugin-update 4.2.4 (core)",
    "@oclif/plugin-version 2.0.16 (core)",
    "@oclif/plugin-warn-if-update-available 3.0.15 (core)",
    "@oclif/plugin-which 3.1.7 (core)",
    "@salesforce/cli 2.37.4 (core)",
    "apex 3.1.4 (core)",
    "auth 3.5.5 (core)",
    "data 3.2.5 (core)",
    "deploy-retrieve 3.5.6 (core)",
    "info 3.1.3 (core)",
    "limits 3.2.1 (core)",
    "marketplace 1.1.1 (core)",
    "org 4.0.4 (core)",
    "packaging 2.3.0 (core)",
    "schema 3.2.1 (core)",
    "settings 2.1.2 (core)",
    "sobject 1.2.2 (core)",
    "source 3.2.3 (core)",
    "telemetry 3.1.18 (core)",
    "templates 56.1.1 (core)",
    "trust 3.5.4 (core)",
    "user 3.4.3 (core)"
  ]
}

Summary

The following command works on the development machine

Command:

sf project deploy start --wait 30 --dry-run --verbose --test-level RunSpecifiedTests --tests TriggerHandlerTest --tests ToolkitTest -o [email protected] --manifest package/package.xml --post-destructive-changes destructiveChanges/destructiveChanges.xml

but fails to work in a Github action using Ubuntu. The whole action is a big bash file.

Result

Error (2): Nonexistent flags: --tests TriggerHandlerTest, --tests ToolkitTest
See more help with --help

Steps To Reproduce

sf version --verbose --json
{
  "architecture": "darwin-arm64",
  "cliVersion": "@salesforce/cli/2.15.9",
  "nodeVersion": "node-v20.9.0",
  "osVersion": "Darwin 23.4.0",
  "rootPath": "/usr/local/lib/node_modules/@salesforce/cli",
  "shell": "zsh",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 2.3.10 (core)",
    "@oclif/plugin-commands 3.0.4 (core)",
    "@oclif/plugin-help 6.0.4 (core)",
    "@oclif/plugin-not-found 3.0.2 (core)",
    "@oclif/plugin-plugins 3.9.4 (core)",
    "@oclif/plugin-search 1.0.4 (core)",
    "@oclif/plugin-update 4.1.3 (core)",
    "@oclif/plugin-version 2.0.3 (core)",
    "@oclif/plugin-warn-if-update-available 3.0.2 (core)",
    "@oclif/plugin-which 3.0.6 (core)",
    "@salesforce/cli 2.15.9 (core)",
    "apex 2.3.20 (core)",
    "auth 2.8.25 (core)",
    "custom-metadata 2.2.10 (user)",
    "data 2.6.1 (core)",
    "deploy-retrieve 1.19.2 (core)",
    "dev 1.3.0 (user)",
    "info 2.6.51 (core)",
    "limits 2.3.41 (core)",
    "login 1.2.39 (core)",
    "marketplace 0.3.1 (core)",
    "org 2.11.5 (core)",
    "schema 2.3.31 (core)",
    "settings 1.4.36 (core)",
    "sobject 0.2.14 (core)",
    "source 2.10.45 (core)",
    "telemetry 2.3.8 (core)",
    "templates 55.5.17 (core)",
    "trust 2.6.22 (core)",
    "user 2.3.39 (core)",
    "@salesforce/sfdx-scanner 3.17.0 (user)",
    "sfdmu 4.33.7 (user)",
    "sfdx-git-delta 5.31.1 (user)",
    "skuid-sfdx 0.4.0 (user)"
  ]
}

but it does not work under Github actions that are running under Ubuntu. The result in Ubuntu is:

Result

Error (2): Nonexistent flags: --tests TriggerHandlerTest, --tests ToolkitTest
See more help with --help

Executing the command with one single tag and tests separated by comma, generates a different issue:

Command:

sf project deploy start --wait 30 --dry-run --verbose --test-level RunSpecifiedTests --tests TriggerHandlerTest, ToolkitTest -o [email protected] --manifest package/package.xml --post-destructive-changes destructiveChanges/destructiveChanges.xml

Result

(node:4517) Warning: The previous version of this command used a comma-separated list for tests. We've changed how you specify multiple tests, so if you continue using your current syntax, your tests will probably not run as you expect.

If a test name contains a space, enclose it in double quotes.
For multiple test names, use one of the following formats:

- Repeat the flag for multiple test names: --tests Test1 --tests Test2 --tests "Test With Space"
- Separate the test names with spaces: --tests Test1 Test2 "Test With Space"
(Use `node --trace-warnings ...` to show where the warning was created)
Warning: The previous version of this command used a comma-separated list for tests. We've changed how you specify multiple tests, so if you continue using your current syntax, your tests will probably not run as you expect.

If a test name contains a space, enclose it in double quotes.
For multiple test names, use one of the following formats:

- Repeat the flag for multiple test names: --tests Test1 --tests Test2 --tests "Test With Space"
- Separate the test names with spaces: --tests Test1 Test2 "Test With Space"
Deploying (dry-run) v58.0 metadata to [email protected] using the v60.0 SOAP API.
Deploy ID: 0AfPk000000TXobKAG
Status: {status} | 0/0 Components{errorInfo}{testInfo}{trackingInfo}
Status: In Progress | 0/24 Components
Status: In Progress | 0/24 Components
Status: In Progress | 0/24 Components
Status: In Progress | 15/24 Components
Status: In Progress | 17/24 Components
Status: In Progress | 20/24 Components
Status: In Progress | 20/24 Components

...

Test Results Summary
Passing: 0
Failing: 0
Total: 0
 ›   Warning: Toolkit - Test coverage of selected Apex 
 ›   Class is 0%, at least 75% test coverage is required
Time: 6
 ›   Warning: TriggerHandler - Test coverage of selected 
 ›   Apex Class is 0%, at least 75% test coverage is required
Dry-run complete.

Expected result


Apex Code Coverage
 Name                              % Covered Uncovered Lines                                       
 ───────────────────────────────── ───────── ───────────────────────────────────────────────────── 
 TriggerHandler 88%       81,89,114,115,165,182,183,184,186,187,259,261,278,279 
 Toolkit    100%                                                            

Test Results Summary
Passing: 3
Failing: 0
Total: 3
Time: 16820

Actual result

Result

Error (2): Nonexistent flags: --tests TriggerHandlerTest, --tests ToolkitTest
See more help with --help

System Information

  • zsh on local environtment
  • bash on Github Actions

Local Environment

{
  "architecture": "darwin-arm64",
  "cliVersion": "@salesforce/cli/2.15.9",
  "nodeVersion": "node-v20.9.0",
  "osVersion": "Darwin 23.4.0",
  "rootPath": "/usr/local/lib/node_modules/@salesforce/cli",
  "shell": "zsh",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 2.3.10 (core)",
    "@oclif/plugin-commands 3.0.4 (core)",
    "@oclif/plugin-help 6.0.4 (core)",
    "@oclif/plugin-not-found 3.0.2 (core)",
    "@oclif/plugin-plugins 3.9.4 (core)",
    "@oclif/plugin-search 1.0.4 (core)",
    "@oclif/plugin-update 4.1.3 (core)",
    "@oclif/plugin-version 2.0.3 (core)",
    "@oclif/plugin-warn-if-update-available 3.0.2 (core)",
    "@oclif/plugin-which 3.0.6 (core)",
    "@salesforce/cli 2.15.9 (core)",
    "apex 2.3.20 (core)",
    "auth 2.8.25 (core)",
    "custom-metadata 2.2.10 (user)",
    "data 2.6.1 (core)",
    "deploy-retrieve 1.19.2 (core)",
    "dev 1.3.0 (user)",
    "info 2.6.51 (core)",
    "limits 2.3.41 (core)",
    "login 1.2.39 (core)",
    "marketplace 0.3.1 (core)",
    "org 2.11.5 (core)",
    "schema 2.3.31 (core)",
    "settings 1.4.36 (core)",
    "sobject 0.2.14 (core)",
    "source 2.10.45 (core)",
    "telemetry 2.3.8 (core)",
    "templates 55.5.17 (core)",
    "trust 2.6.22 (core)",
    "user 2.3.39 (core)",
    "@salesforce/sfdx-scanner 3.17.0 (user)",
    "sfdmu 4.33.7 (user)",
    "sfdx-git-delta 5.31.1 (user)",
    "skuid-sfdx 0.4.0 (user)"
  ]
}

Github Action Environment (ERROR environment)

{
  "architecture": "linux-x64",
  "cliVersion": "@salesforce/cli/2.37.4",
  "nodeVersion": "node-v18.20.2",
  "osVersion": "Linux 6.5.0-1018-azure",
  "rootPath": "/usr/local/lib/node_modules/@salesforce/cli",
  "shell": "bash",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 3.0.13 (core)",
    "@oclif/plugin-commands 3.2.2 (core)",
    "@oclif/plugin-help 6.0.20 (core)",
    "@oclif/plugin-not-found 3.1.2 (core)",
    "@oclif/plugin-plugins 5.0.7 (core)",
    "@oclif/plugin-search 1.0.20 (core)",
    "@oclif/plugin-update 4.2.4 (core)",
    "@oclif/plugin-version 2.0.16 (core)",
    "@oclif/plugin-warn-if-update-available 3.0.15 (core)",
    "@oclif/plugin-which 3.1.7 (core)",
    "@salesforce/cli 2.37.4 (core)",
    "apex 3.1.4 (core)",
    "auth 3.5.5 (core)",
    "data 3.2.5 (core)",
    "deploy-retrieve 3.5.6 (core)",
    "info 3.1.3 (core)",
    "limits 3.2.1 (core)",
    "marketplace 1.1.1 (core)",
    "org 4.0.4 (core)",
    "packaging 2.3.0 (core)",
    "schema 3.2.1 (core)",
    "settings 2.1.2 (core)",
    "sobject 1.2.2 (core)",
    "source 3.2.3 (core)",
    "telemetry 3.1.18 (core)",
    "templates 56.1.1 (core)",
    "trust 3.5.4 (core)",
    "user 3.4.3 (core)"
  ]
}

Additional information

Note: Running using a file with a json containing which tests to execute, also does not work.

Tests Done:

  • --tests Test1 --tests Test2 --tests "Test With Space" - FAILS
  • --tests "Test1, Test2, Test With Space" - FAILS
  • --tests Test1, Test2, "Test With Space" - FAILS
  • --tests Test1 Test2 "Test With Space" - FAILS

All of these tests done works with the MBP local environment.

Updates

  • Forcibly installed the same version of the local machine and same error. This doesn't seem to be related to sf cli version but something related to it.
npm install -g @salesforce/[email protected]
 ›   Warning: @salesforce/cli update available from 2.15.9 to 2.37.4.
  • Updating Node and NPM to the latest, also didn't help.
##[debug] Node version after update
v20.12.2
##[debug] NPM version after update
10.5.0
{
  "architecture": "linux-x64",
  "cliVersion": "@salesforce/cli/2.37.4",
  "nodeVersion": "node-v20.12.2",
  "osVersion": "Linux 6.5.0-1018-azure",
  "rootPath": "/usr/local/lib/node_modules/@salesforce/cli",
  "shell": "bash",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 3.0.13 (core)",
    "@oclif/plugin-commands 3.2.2 (core)",
    "@oclif/plugin-help 6.0.20 (core)",
    "@oclif/plugin-not-found 3.1.2 (core)",
    "@oclif/plugin-plugins 5.0.7 (core)",
    "@oclif/plugin-search 1.0.20 (core)",
    "@oclif/plugin-update 4.2.4 (core)",
    "@oclif/plugin-version 2.0.16 (core)",
    "@oclif/plugin-warn-if-update-available 3.0.15 (core)",
    "@oclif/plugin-which 3.1.7 (core)",
    "@salesforce/cli 2.37.4 (core)",
    "apex 3.1.4 (core)",
    "auth 3.5.5 (core)",
    "data 3.2.5 (core)",
    "deploy-retrieve 3.5.6 (core)",
    "info 3.1.3 (core)",
    "limits 3.2.1 (core)",
    "marketplace 1.1.1 (core)",
    "org 4.0.4 (core)",
    "packaging 2.3.0 (core)",
    "schema 3.2.1 (core)",
    "settings 2.1.2 (core)",
    "sobject 1.2.2 (core)",
    "source 3.2.3 (core)",
    "telemetry 3.1.18 (core)",
    "templates 56.1.1 (core)",
    "trust 3.5.4 (core)",
    "user 3.4.3 (core)"
  ]
}
@csilva-intakedesk csilva-intakedesk added the investigating We're actively investigating this issue label Apr 19, 2024
Copy link

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

@github-actions github-actions bot added the validated Version information for this issue has been validated label Apr 19, 2024
csilva-intakedesk added a commit to jvega-intakedesk/salesforce-deployment that referenced this issue Apr 19, 2024
Created: forcedotcom/cli#2834

Adjusted the class to execute with comma separated versions.
@mshanemc
Copy link
Contributor

mshanemc commented Apr 19, 2024

you might want your shell script to echo out the command it's running. I'm sure this is an error in your bash script.

Once you've done that, can you share the actual command that it's running?

I just added a test to do your scenario (two --tests flags salesforcecli/plugin-deploy-retrieve#979) that runs on GHA.
and another to be safe that it's not related to our recent preparse hooks change: salesforcecli/plugin-deploy-retrieve#980

@mshanemc mshanemc added more information required Issue requires more information or a response from the customer and removed investigating We're actively investigating this issue labels Apr 19, 2024
@csilva-intakedesk
Copy link
Author

Hi @mshanemc I have added that before. This is what is passing.

sf project deploy start --wait 30 --dry-run --verbose --test-level RunSpecifiedTests --tests TriggerHandlerTest --tests ToolkitTest -o [email protected] --manifest package/package.xml --post-destructive-changes destructiveChanges/destructiveChanges.xml

I have also tested with other formats such as

--tests TriggerHandlerTest ToolkitTest 

and all are giving errors, different from the initial one that is

Error (2): Nonexistent flags: --tests TriggerHandlerTest, --tests ToolkitTest
See more help with --help

To test locally what I do is to copy, from output the same sequence that the script outputs in the GHA. I will check out the issue you mentioned to see if it is not related to GHA as well.

@mshanemc
Copy link
Contributor

I will check out the issue you mentioned to see if it is not related to GHA as well.

Those aren't issues, they're us running commands like yours (with 2 --tests) using GHA. Hopefully you can see some difference between those and your script

csilva-intakedesk added a commit to jvega-intakedesk/salesforce-deployment that referenced this issue Apr 23, 2024
* IS-788: Adjusting testing parameters.

* IS-788 Adding visibility

* IS-788 : Using the file instead.

* IS-788: Adding debugging information to understand why the tests are failing

* IS-788: Adding detailed coverage information

* IS-788: Adding debug wrapper

* IS-788:? Adding Test word to the component being tested.

* IS-788: Force coverage display

* IS-788: Removing test-coverage formatter

It won't matter because the output fails before it displays the results.

* IS-788: Adding one extra step to check on coverage

* IS-788: Adjusting classes to run

Instead of Json, using the comma separated value

* IS-788: New Script format.

Changing the test action to avoid warnings and only look on apex classes.

* IS-788: Fixing a typo

* IS-788: Adjustments

* IS-788: Adjusting the typo again.

* IS-788: Returning to older strategy which pass tests with comma separated.

* IS-788: Fixing the flag not found error

The issue is that it expects --tests "ClassTest" and not --tests ClassTest

Adjusting

* IS-788: Forgot this

* IS-788: Using an XML Parser

* IS-788: Forgot to remove the json file.

* IS-788: Adding debugging Lines

* IS-788: Troubleshooting differences of CLI

* IS-788: Passing the tests with spaces instead of flags

Documentation seems wrong, adjusting.

* IS-788: Adjusting based on prior functioning script

There is clearly a difference between plugins and sf cli.

* IS-788: Trying to adjust the code to match current cli

* IS-788: This seems to be a bug

Created: forcedotcom/cli#2834

Adjusted the class to execute with comma separated versions.

* IS-788: Separting the tests with spaces

Returning to the very first version of the test.

* IS-788: Returning to run all tests.

Commented sections.

* IS-788: Removing Typo

* IS-788: Checking on node

* IS-788: Forcing version

* IS-788: Adjusting version

* IS-788: Updating node

* IS-788: Adding sudo to update node

* IS-788: More sudo

* IS-788: Last test run with all  updated

* IS-788: Returning to Full test run

* Placing XML content in debug mode.

* Removing debug comments.

* IS-788: Remove apt-get update all

Testing locally with act.

* IS-788: Trying to run command locally

* IS-788: No java

* IS-788: Adding debuging information for local test.

* IS-788: setting java to a pre-fixed version.

* IS-788: Troubleshooting.

* IS-788: Troubleshooting

* IS-788: Troubleshooting

* IS-788: Troubleshooting

* IS-788: Troubleshooting

* IS-788: Troubleshooting

* IS-788: Troubleshooting

* IS-788: Troubleshooting

* IS-788: troubleshooting

* IS-788: Troubleshooting.

* IS-788: Using Snap

* IS-788: Snap

* IS-788: Adding a bash file to test changes.

* IS-788: Some small adjustments to ensure that it won't throw errors.

* IS-788:  Adding validation for if file exists
@mshanemc
Copy link
Contributor

Looks like from that PR you got the bash/quotes figured out. The CLI doesn't need the quotes, but bash var expansion and/or string concat might.

I'll close this issue, but holler if I'm misunderstanding your PR.

@csilva-intakedesk
Copy link
Author

Hi @mshanemc yes, after awhile I managed to get it running. I'm still testing to see if it works with a lot of tests classes. The goal is to only run the test classes for the component being deployed. Thanks for the assistance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
more information required Issue requires more information or a response from the customer validated Version information for this issue has been validated
Projects
None yet
Development

No branches or pull requests

2 participants