-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
fix: quick edit of "agent test" commands
- Loading branch information
1 parent
77cfb0b
commit 9e4daac
Showing
7 changed files
with
62 additions
and
37 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,21 +1,25 @@ | ||
# summary | ||
|
||
Cancel a running test for an Agent. | ||
Cancel an agent test that's currently running in your org. | ||
|
||
# description | ||
|
||
Cancel a running test for an Agent, providing the AiEvaluation ID. | ||
This command requires a job ID, which the original "agent test run" command displays when it completes. You can also use the --use-most-recent flag to see results for the most recently run agent test. | ||
|
||
# flags.job-id.summary | ||
|
||
The AiEvaluation ID. | ||
Job ID of the running agent test that you want to cancel. | ||
|
||
# flags.use-most-recent.summary | ||
|
||
Use the job ID of the most recent test evaluation. | ||
Use the job ID of the most recently-run agent test. | ||
|
||
# examples | ||
|
||
- Cancel a test for an Agent: | ||
- Cancel an agent test currently running in your default org using a job ID: | ||
|
||
<%= config.bin %> <%= command.id %> --job-id AiEvalId | ||
<%= config.bin %> <%= command.id %> --job-id 4KBfake0000003F4AQ | ||
|
||
- Cancel the most recently run agent test in the org with alias "my-org": | ||
|
||
<%= config.bin %> <%= command.id %> --job-id 4KBfake0000003F4AQ --target-org my-org |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,19 +1,31 @@ | ||
# summary | ||
|
||
Get the results of a test evaluation. | ||
Get the results of a completed agent test run. | ||
|
||
# description | ||
|
||
Provide the AiEvaluation ID to get the results of a test evaluation. | ||
This command requires a job ID, which the original "agent test run" command displays when it completes. You can also use the --use-most-recent flag to see results for the most recently run agent test. | ||
|
||
By default, this command outputs test results in human-readable tables for each test case. The tables show whether the test case passed, the expected and actual values, the test score, how long the test took, and more. Use the --result-format to display the test results in JSON or Junit format. Use the --output-dir flag to write the results to a file rather than to the terminal. | ||
|
||
# flags.job-id.summary | ||
|
||
The AiEvaluation ID. | ||
Job ID of the completed agent test run. | ||
|
||
# flags.use-most-recent.summary | ||
|
||
Use the job ID of the most recent test evaluation. | ||
Use the job ID of the most recent agent test run. | ||
|
||
# examples | ||
|
||
- <%= config.bin %> <%= command.id %> --job-id AiEvalId | ||
- Get the results of an agent test run in your default org using its job ID: | ||
|
||
<%= config.bin %> <%= command.id %> --job-id 4KBfake0000003F4AQ | ||
|
||
- Get the results of the most recently run agent test in an org with alias "my-org": | ||
|
||
<%= config.bin %> <%= command.id %> --use-most-recent --target-org my-org | ||
|
||
- Get the results of the most recently run agent test in your default org, and write the JSON-formatted results into a directory called "test-results": | ||
|
||
<%= config.bin %> <%= command.id %> --use-most-recent --output-dir ./test-results --result-format json |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,29 +1,37 @@ | ||
# summary | ||
|
||
Resume a running test for an Agent. | ||
Resume an agent test that you previously started in your org so you can view the test results. | ||
|
||
# description | ||
|
||
Resume a running test for an Agent, providing the AiEvaluation ID. | ||
This command requires a job ID, which the original "agent test run" command displays when it completes. You can also use the --use-most-recent flag to see results for the most recently run agent test. | ||
|
||
Use the --wait flag to specify the number of minutes for this command to wait for the agent test to complete; if the test completes by the end of the wait time, the command displays the test results. If not, the CLI returns control of the terminal to you, and you must run "agent test resume" again. | ||
|
||
By default, this command outputs test results in human-readable tables for each test case. The tables show whether the test case passed, the expected and actual values, the test score, how long the test took, and more. Use the --result-format to display the test results in JSON or Junit format. Use the --output-dir flag to write the results to a file rather than to the terminal. | ||
|
||
# flags.job-id.summary | ||
|
||
The AiEvaluation ID. | ||
Job ID of the original agent test run. | ||
|
||
# flags.use-most-recent.summary | ||
|
||
Use the job ID of the most recent test evaluation. | ||
Use the job ID of the most recent agent test run. | ||
|
||
# flags.wait.summary | ||
|
||
Number of minutes to wait for the command to complete and display results to the terminal window. | ||
|
||
# flags.wait.description | ||
# examples | ||
|
||
- Resume an agent test in your default org using a job ID: | ||
|
||
If the command continues to run after the wait period, the CLI returns control of the terminal window to you. | ||
<%= config.bin %> <%= command.id %> --job-id 4KBfake0000003F4AQ | ||
|
||
# examples | ||
- Resume the most recently-run agent test in an org with alias "my-org" org; wait 10 minutes for the tests to finish: | ||
|
||
<%= config.bin %> <%= command.id %> --use-most-recent --wait 10 --target-org my-org | ||
|
||
- Resume a test for an Agent: | ||
- Resume the most recent agent test in your default org, and write the JSON-formatted results into a directory called "test-results": | ||
|
||
<%= config.bin %> <%= command.id %> --job-id AiEvalId | ||
<%= config.bin %> <%= command.id %> --use-most-recent --output-dir ./test-results --result-format json |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,29 +1,33 @@ | ||
# summary | ||
|
||
Start a test for an Agent. | ||
Start an agent test in your org. | ||
|
||
# description | ||
|
||
Start a test for an Agent, providing the AiEvalDefinitionVersion ID. Returns the job ID. | ||
Use the --name flag to specify the name of the agent test you want to run; find the agent test's name in the Testing Center page in the Setup UI of your org. | ||
|
||
# flags.name.summary | ||
This command starts the agent test in your org, but doesn't by default wait for it to finish. Instead, it displays the "agent test resume" command, with a job ID, that you execute to see the results of the test run, and then returns control of the terminal window to you. Use the --wait flag to specify the number of minutes for the command to wait for the agent test to complete; if the test completes by the end of the wait time, the command displays the test results. If not, run "agent test resume". | ||
|
||
The name of the AiEvaluationDefinition to start. | ||
By default, this command outputs test results in human-readable tables for each test case, if the test completes in time. The tables show whether the test case passed, the expected and actual values, the test score, how long the test took, and more. Use the --result-format to display the test results in JSON or Junit format. Use the --output-dir flag to write the results to a file rather than to the terminal. | ||
|
||
# flags.name.description | ||
# flags.name.summary | ||
|
||
The name of the AiEvaluationDefinition to start. | ||
Name of the agent test to start. | ||
|
||
# flags.wait.summary | ||
|
||
Number of minutes to wait for the command to complete and display results to the terminal window. | ||
|
||
# flags.wait.description | ||
# examples | ||
|
||
- Start a test called MyAgentTest for an agent in your default org, don't wait for the test to finish: | ||
|
||
If the command continues to run after the wait period, the CLI returns control of the terminal window to you. | ||
<%= config.bin %> <%= command.id %> --name MyAgentTest | ||
|
||
# examples | ||
- Start a test for an agent in an org with alias "my-org" and wait for 10 minutes for the test to finish: | ||
|
||
<%= config.bin %> <%= command.id %> --name MyAgentTest --wait 10 --target-org my-org | ||
|
||
- Start a test for an Agent: | ||
- Start a test and write the JSON-formatted results into a directory called "test-results": | ||
|
||
<%= config.bin %> <%= command.id %> --name AiEvalDefVerId | ||
<%= config.bin %> <%= command.id %> --name MyAgentTest --wait 10 --output-dir ./test-results --result-format json |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,11 +1,11 @@ | ||
# flags.result-format.summary | ||
|
||
Format of the test run results. | ||
Format of the agent test run results. | ||
|
||
# flags.output-dir.summary | ||
|
||
Directory to write the test results to. | ||
Directory to write the agent test results into. | ||
|
||
# flags.output-dir.description | ||
|
||
If test run is complete, write the results to the specified directory. If the tests are still running, the test results will not be written. | ||
If the agent test run completes, write the results to the specified directory. If the test is still running, the test results aren't written. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters