Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: quick edit of "agent test" commands #47

Merged
merged 1 commit into from
Dec 17, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 10 additions & 6 deletions messages/agent.test.cancel.md
Original file line number Diff line number Diff line change
@@ -1,21 +1,25 @@
# summary

Cancel a running test for an Agent.
Cancel an agent test that's currently running in your org.

# description

Cancel a running test for an Agent, providing the AiEvaluation ID.
This command requires a job ID, which the original "agent test run" command displays when it completes. You can also use the --use-most-recent flag to see results for the most recently run agent test.

# flags.job-id.summary

The AiEvaluation ID.
Job ID of the running agent test that you want to cancel.

# flags.use-most-recent.summary

Use the job ID of the most recent test evaluation.
Use the job ID of the most recently-run agent test.

# examples

- Cancel a test for an Agent:
- Cancel an agent test currently running in your default org using a job ID:

<%= config.bin %> <%= command.id %> --job-id AiEvalId
<%= config.bin %> <%= command.id %> --job-id 4KBfake0000003F4AQ

- Cancel the most recently run agent test in the org with alias "my-org":

<%= config.bin %> <%= command.id %> --job-id 4KBfake0000003F4AQ --target-org my-org
22 changes: 17 additions & 5 deletions messages/agent.test.results.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,31 @@
# summary

Get the results of a test evaluation.
Get the results of a completed agent test run.

# description

Provide the AiEvaluation ID to get the results of a test evaluation.
This command requires a job ID, which the original "agent test run" command displays when it completes. You can also use the --use-most-recent flag to see results for the most recently run agent test.

By default, this command outputs test results in human-readable tables for each test case. The tables show whether the test case passed, the expected and actual values, the test score, how long the test took, and more. Use the --result-format to display the test results in JSON or Junit format. Use the --output-dir flag to write the results to a file rather than to the terminal.

# flags.job-id.summary

The AiEvaluation ID.
Job ID of the completed agent test run.

# flags.use-most-recent.summary

Use the job ID of the most recent test evaluation.
Use the job ID of the most recent agent test run.

# examples

- <%= config.bin %> <%= command.id %> --job-id AiEvalId
- Get the results of an agent test run in your default org using its job ID:

<%= config.bin %> <%= command.id %> --job-id 4KBfake0000003F4AQ

- Get the results of the most recently run agent test in an org with alias "my-org":

<%= config.bin %> <%= command.id %> --use-most-recent --target-org my-org

- Get the results of the most recently run agent test in your default org, and write the JSON-formatted results into a directory called "test-results":

<%= config.bin %> <%= command.id %> --use-most-recent --output-dir ./test-results --result-format json
26 changes: 17 additions & 9 deletions messages/agent.test.resume.md
Original file line number Diff line number Diff line change
@@ -1,29 +1,37 @@
# summary

Resume a running test for an Agent.
Resume an agent test that you previously started in your org so you can view the test results.

# description

Resume a running test for an Agent, providing the AiEvaluation ID.
This command requires a job ID, which the original "agent test run" command displays when it completes. You can also use the --use-most-recent flag to see results for the most recently run agent test.

Use the --wait flag to specify the number of minutes for this command to wait for the agent test to complete; if the test completes by the end of the wait time, the command displays the test results. If not, the CLI returns control of the terminal to you, and you must run "agent test resume" again.

By default, this command outputs test results in human-readable tables for each test case. The tables show whether the test case passed, the expected and actual values, the test score, how long the test took, and more. Use the --result-format to display the test results in JSON or Junit format. Use the --output-dir flag to write the results to a file rather than to the terminal.

# flags.job-id.summary

The AiEvaluation ID.
Job ID of the original agent test run.

# flags.use-most-recent.summary

Use the job ID of the most recent test evaluation.
Use the job ID of the most recent agent test run.

# flags.wait.summary

Number of minutes to wait for the command to complete and display results to the terminal window.

# flags.wait.description
# examples

- Resume an agent test in your default org using a job ID:

If the command continues to run after the wait period, the CLI returns control of the terminal window to you.
<%= config.bin %> <%= command.id %> --job-id 4KBfake0000003F4AQ

# examples
- Resume the most recently-run agent test in an org with alias "my-org" org; wait 10 minutes for the tests to finish:

<%= config.bin %> <%= command.id %> --use-most-recent --wait 10 --target-org my-org

- Resume a test for an Agent:
- Resume the most recent agent test in your default org, and write the JSON-formatted results into a directory called "test-results":

<%= config.bin %> <%= command.id %> --job-id AiEvalId
<%= config.bin %> <%= command.id %> --use-most-recent --output-dir ./test-results --result-format json
26 changes: 15 additions & 11 deletions messages/agent.test.run.md
Original file line number Diff line number Diff line change
@@ -1,29 +1,33 @@
# summary

Start a test for an Agent.
Start an agent test in your org.

# description

Start a test for an Agent, providing the AiEvalDefinitionVersion ID. Returns the job ID.
Use the --name flag to specify the name of the agent test you want to run; find the agent test's name in the Testing Center page in the Setup UI of your org.

# flags.name.summary
This command starts the agent test in your org, but doesn't by default wait for it to finish. Instead, it displays the "agent test resume" command, with a job ID, that you execute to see the results of the test run, and then returns control of the terminal window to you. Use the --wait flag to specify the number of minutes for the command to wait for the agent test to complete; if the test completes by the end of the wait time, the command displays the test results. If not, run "agent test resume".

The name of the AiEvaluationDefinition to start.
By default, this command outputs test results in human-readable tables for each test case, if the test completes in time. The tables show whether the test case passed, the expected and actual values, the test score, how long the test took, and more. Use the --result-format to display the test results in JSON or Junit format. Use the --output-dir flag to write the results to a file rather than to the terminal.

# flags.name.description
# flags.name.summary

The name of the AiEvaluationDefinition to start.
Name of the agent test to start.

# flags.wait.summary

Number of minutes to wait for the command to complete and display results to the terminal window.

# flags.wait.description
# examples

- Start a test called MyAgentTest for an agent in your default org, don't wait for the test to finish:

If the command continues to run after the wait period, the CLI returns control of the terminal window to you.
<%= config.bin %> <%= command.id %> --name MyAgentTest

# examples
- Start a test for an agent in an org with alias "my-org" and wait for 10 minutes for the test to finish:

<%= config.bin %> <%= command.id %> --name MyAgentTest --wait 10 --target-org my-org

- Start a test for an Agent:
- Start a test and write the JSON-formatted results into a directory called "test-results":

<%= config.bin %> <%= command.id %> --name AiEvalDefVerId
<%= config.bin %> <%= command.id %> --name MyAgentTest --wait 10 --output-dir ./test-results --result-format json
6 changes: 3 additions & 3 deletions messages/shared.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# flags.result-format.summary

Format of the test run results.
Format of the agent test run results.

# flags.output-dir.summary

Directory to write the test results to.
Directory to write the agent test results into.

# flags.output-dir.description

If test run is complete, write the results to the specified directory. If the tests are still running, the test results will not be written.
If the agent test run completes, write the results to the specified directory. If the test is still running, the test results aren't written.
1 change: 0 additions & 1 deletion src/commands/agent/test/resume.ts
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,6 @@ export default class AgentTestResume extends SfCommand<AgentTestResumeResult> {
min: 1,
defaultValue: 5,
summary: messages.getMessage('flags.wait.summary'),
description: messages.getMessage('flags.wait.description'),
}),
'result-format': resultFormatFlag(),
'output-dir': testOutputDirFlag(),
Expand Down
2 changes: 0 additions & 2 deletions src/commands/agent/test/run.ts
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,6 @@ export default class AgentTestRun extends SfCommand<AgentTestRunResult> {
char: 'n',
required: true,
summary: messages.getMessage('flags.name.summary'),
description: messages.getMessage('flags.name.description'),
}),
// we want to pass `undefined` to the API
// eslint-disable-next-line sf-plugin/flag-min-max-default
Expand All @@ -45,7 +44,6 @@ export default class AgentTestRun extends SfCommand<AgentTestRunResult> {
unit: 'minutes',
min: 1,
summary: messages.getMessage('flags.wait.summary'),
description: messages.getMessage('flags.wait.description'),
}),
'result-format': resultFormatFlag(),
'output-dir': testOutputDirFlag(),
Expand Down
Loading