Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Renamed sched.test_cmd to sched.launch and added alias #751

Open
wants to merge 8 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/advanced.rst
Original file line number Diff line number Diff line change
Expand Up @@ -446,7 +446,7 @@ scheduler, but don't configure one. It's time to rectify that.
# srun -N 50 -n 150 ./supermagic -a
# Note that this would run in an sbatch script within an allocation
# that conforms to the rest of the slurm settings.
- {sched.test_cmd} ./supermagic -a
- {sched.launch} ./supermagic -a


- See ``pav show sched --vars <sched_name>`` for a listing of what variables are
Expand Down
2 changes: 1 addition & 1 deletion docs/results/parse.rst
Original file line number Diff line number Diff line change
Expand Up @@ -329,7 +329,7 @@ depends on the **per\_file** attribute for the result parser.
cmds:
# Use the srun --output option to specify that results are
# to be written to separate files.
- {{sched.test_cmd}} --output="%N.out" env
- {{sched.launch}} --output="%N.out" env

result_parse:
regex:
Expand Down
4 changes: 2 additions & 2 deletions docs/test_run_lifecycle.rst
Original file line number Diff line number Diff line change
Expand Up @@ -217,7 +217,7 @@ something like this:
"result": "last_login > {{pav.timestamp - time_limit}}"},
"regex": [{"key": "last_login",
"regex": "Last Login: (\\d+)"}]},
"run": {"cmds": ["{{sched.test_cmd}} ./test1 $ARGS"],
"run": {"cmds": ["{{sched.launch}} ./test1 $ARGS"],
"env": {"ARGS": "-u {{user}}"},
"modules": ["gcc"]},
"variables": {"hours": 5,
Expand Down Expand Up @@ -457,4 +457,4 @@ Mark the Test as Complete

Finally, the test is marked as complete by saving a 'RUN_COMPLETE' file in the
test's run directory. Pavilion uses this to quickly determine which tests
might still be running.
might still be running.
10 changes: 5 additions & 5 deletions docs/tests/scheduling.rst
Original file line number Diff line number Diff line change
Expand Up @@ -111,9 +111,9 @@ command.
| | | 1.
tasks_total | True | 180 | Total tasks to create, based on number of nodes
| | | actually acquired.
test_cmd | True | srun -N 5 -w no | Construct a cmd to run a process under this
launch | True | srun -N 5 -w no | Construct a cmd to run a process under this
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd add a (formerly sched.test_cmd) to the docs here.

| | de[05-10],node2 | scheduler, with the criteria specified by this test.
| | 3 -n 20 |
| | 3 -n 20 | (used to be `sched.test_cmd`)

.. _tests.scheduling.jobs:

Expand Down Expand Up @@ -342,9 +342,9 @@ wrapper command before actually running the intended command.
cmds:
# The run command will be `srun -N1 -p standard valgrind ./supermagic -a`
# It will run `valgrind ./supermagic -a` on the allocation
- '{{sched.test_cmd}} ./supermagic -a'
- '{{sched.launch}} ./supermagic -a'

When using the ``raw`` scheduler, the ``{{sched.test_cmd}}`` normally returns an empty string. You can
When using the ``raw`` scheduler, the ``{{sched.launch}}`` normally returns an empty string. You can
use the wrapper setting to control a different scheduler directly.

.. code-block:: yaml
Expand All @@ -359,5 +359,5 @@ use the wrapper setting to control a different scheduler directly.
run:
cmds:
# With the schedule wrapper, this will be `mpirun -np 2 ./supermagic -a`
- '{{sched.test_cmd}} ./supermagic -a'
- '{{sched.launch}} ./supermagic -a'

2 changes: 1 addition & 1 deletion docs/tests/values.rst
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ not Python3 code.**
cmds:
# In the simple case, expressions are often just a variable
# reference.
- "{{sched.test_cmds}} ./my_cmd"
- "{{sched.launch}} ./my_cmd"

Types of Expressions
^^^^^^^^^^^^^^^^^^^^
Expand Down
4 changes: 2 additions & 2 deletions examples/acceptance/tests/ior.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -273,11 +273,11 @@ scaling_study:
- '[ -d {{scratch.path}}/ior-test-{{dirname}} ] && rm -Rf {{scratch.path}}/ior-test-{{dirname}}'
- 'mkdir {{scratch.path}}/ior-test-{{dirname}} && pushd {{scratch.path}}/ior-test-{{dirname}}'
- '# this writes out the files'
- '{{sched.test_cmd}} $IOR_EXE "${test_variables} -w'
- '{{sched.launch}} $IOR_EXE "${test_variables} -w'
- 'if [[ {{numnodes}} -ne 1 ]]'
- 'then'
- ' # this reads those files'
- ' {{sched.test_cmd}} $IOR_EXE -E -C -Q {{taskspernode}} -r'
- ' {{sched.launch}} $IOR_EXE -E -C -Q {{taskspernode}} -r'
- 'fi'
- 'cd ..'
- 'rm -rf {{scratch.path}}/ior-test-{{dirname}}'
2 changes: 1 addition & 1 deletion examples/acceptance/tests/pynamic.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ base:
# Run commands.
cmds:
- 'cd pynamic-pyMPI-2.6a1'
- '{{sched.test_cmd}} ./pynamic-mpi4py $(date +%s)'
- '{{sched.launch}} ./pynamic-mpi4py $(date +%s)'

result_parse:
regex:
Expand Down
2 changes: 1 addition & 1 deletion examples/acceptance/tests/umt.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,7 @@ _base:
# seconds to update all the information. Sleeping for a few seconds allows for the seff command
# to report the data accurately.
cmds:
- "{{sched.test_cmd}} ./umt2015-crossroads/Teton/SuOlsonTest {{umt_run.gridfilename}}
- "{{sched.launch}} ./umt2015-crossroads/Teton/SuOlsonTest {{umt_run.gridfilename}}
{{umt_run.groups}} {{umt_run.quadtype}} {{umt_run.order}} {{umt_run.polar}}
{{umt_run.azim}} ; job_id=${SLURM_JOB_ID}"
- "sleep 10s"
Expand Down
2 changes: 1 addition & 1 deletion examples/acceptance/tests/vpic.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -265,7 +265,7 @@ _base:
cmds:
- ./build/bin/vpic {{vpic_input_deck}}
- 'top -b -u {{user}} > top_out.txt 2>&1 &'
- '{{vpic_run.time_cmd}} {{sched.test_cmd}} {{vpic_run.numa}} ./lpi-input.Linux --tpp 1'
- '{{vpic_run.time_cmd}} {{sched.launch}} {{vpic_run.numa}} ./lpi-input.Linux --tpp 1'

schedule:
nodes: '{{nnodes}}'
Expand Down
2 changes: 1 addition & 1 deletion examples/pre-packaged/tests/hello_mpi.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -44,4 +44,4 @@ base:
- "{{mpis}}"
cmds:
- "set -x"
- "{{sched.test_cmd}} ./mpi_hello"
- "{{sched.launch}} ./mpi_hello"
2 changes: 1 addition & 1 deletion examples/pre-packaged/tests/hpcg.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ base:

run:
modules: ["{{compilers}}", "{{mpis}}"]
cmds: "{{sched.test_cmd}} ./bin/xhpcg"
cmds: "{{sched.launch}} ./bin/xhpcg"

result_parse:
regex:
Expand Down
2 changes: 1 addition & 1 deletion examples/pre-packaged/tests/imb.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -44,4 +44,4 @@ base:

cmds:
- cd src_c
- "{{sched.test_cmd}} ./IMB-MPI1 -npmin {{sched.alloc_cpu_total}} -mem 50 alltoall"
- "{{sched.launch}} ./IMB-MPI1 -npmin {{sched.alloc_cpu_total}} -mem 50 alltoall"
4 changes: 2 additions & 2 deletions examples/pre-packaged/tests/ior.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ base:
- " rm -rf ior-test-{{block_size}}-{{a_flag}}-{{scratch.name}}"
- " mkdir ior-test-{{block_size}}-{{a_flag}}-{{scratch.name}}"
- " cd ior-test-{{block_size}}-{{a_flag}}-{{scratch.name}}"
- " {{sched.test_cmd}} {{srun_opts}} $bloc/src/ior -a {{a_flag}} -b {{block_size}} -c -e -F -i 2"
- " {{sched.launch}} {{srun_opts}} $bloc/src/ior -a {{a_flag}} -b {{block_size}} -c -e -F -i 2"
- " cd .."
- " rm -rf {{scratch.path}}/ior-test-{{block_size}}-{{a_flag}}-{{scratch.name}}"
- "fi"
Expand Down Expand Up @@ -180,7 +180,7 @@ xroads_load1:
- "rm -rf ior-test-{{outname}}"
- "mkdir ior-test-{{outname}}"
- "cd ior-test-{{outname}}"
- "{{sched.test_cmd}} {{srun_opts}} $bloc/src/ior -f $bloc/inp"
- "{{sched.launch}} {{srun_opts}} $bloc/src/ior -f $bloc/inp"
- "cd .."
- "rm -rf {{scratch.path}}/ior-test-{{outname}}"

Expand Down
2 changes: 1 addition & 1 deletion examples/pre-packaged/tests/stream.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,4 @@ base:
- "{{compilers}}"
- "{{mpis}}"
cmds:
- "{{sched.test_cmd}} ./stream"
- "{{sched.launch}} ./stream"
2 changes: 1 addition & 1 deletion examples/pre-packaged/tests/supermagic.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ base:
- "{{mpis}}"

cmds:
- "{{sched.test_cmd}} ./supermagic"
- "{{sched.launch}} ./supermagic"

result_parse:
regex:
Expand Down
12 changes: 6 additions & 6 deletions lib/pavilion/schedulers/plugins/flux.py
Original file line number Diff line number Diff line change
Expand Up @@ -66,10 +66,10 @@ class FluxVars(SchedulerVariables):

EXAMPLE = SchedulerVariables.EXAMPLE.copy()
EXAMPLE.update({
'test_cmd': 'flux run -x -N 5 -n 20',
'launch': 'flux run -x -N 5 -n 20',
})

def _test_cmd(self):
def _launch(self):
"""Construct a cmd to run a process under this scheduler, with the
criteria specified by this test.
"""
Expand All @@ -91,13 +91,13 @@ def _test_cmd(self):
return ' '.join(cmd)

@dfr_var_method
def test_cmd(self):
def launch(self):
"""Calls the actual test command and then wraps the return with the wrapper
provided in the schedule section of the configuration."""

# Removes all the None values to avoid getting a TypeError while trying to
# join two commands
return ' '.join(filter(lambda item: item is not None, [self._test_cmd(),
return ' '.join(filter(lambda item: item is not None, [self._launch(),
self._sched_config['wrapper']]))


Expand Down Expand Up @@ -210,11 +210,11 @@ def _get_config_elems(self):
yc.ListElem(name='fluxrun_extra',
sub_elem=yc.StrElem(),
help_text="Extra arguments to pass to flux run as part of "
"the 'sched.test_cmd' variable."),
"the 'sched.launch' variable."),
yc.ListElem(name='fluxbatch_extra',
sub_elem=yc.StrElem(),
help_text="Extra arguments to pass to flux batch as part of "
"the 'sched.test_cmd' variable."),
"the 'sched.launch' variable."),
]

defaults = {
Expand Down
10 changes: 5 additions & 5 deletions lib/pavilion/schedulers/plugins/slurm.py
Original file line number Diff line number Diff line change
Expand Up @@ -133,10 +133,10 @@ class SlurmVars(SchedulerVariables):

EXAMPLE = SchedulerVariables.EXAMPLE.copy()
EXAMPLE.update({
'test_cmd': 'srun -N 5 -w node[05-10],node23 -n 20',
'launch': 'srun -N 5 -w node[05-10],node23 -n 20',
})

def _test_cmd(self):
def _launch(self):
"""Construct a cmd to run a process under this scheduler, with the
criteria specified by this test.
"""
Expand Down Expand Up @@ -171,13 +171,13 @@ def _test_cmd(self):
return ' '.join(cmd)

@dfr_var_method
def test_cmd(self):
def launch(self):
"""Calls the actual test command and then wraps the return with the wrapper
provided in the schedule section of the configuration."""

# Removes all the None values to avoid getting a TypeError while trying to
# join two commands
return ' '.join(filter(lambda item: item is not None, [self._test_cmd(),
return ' '.join(filter(lambda item: item is not None, [self._launch(),
self._sched_config['wrapper']]))

def slurm_float(val):
Expand Down Expand Up @@ -276,7 +276,7 @@ def _get_config_elems(self):
yc.ListElem(name='srun_extra',
sub_elem=yc.StrElem(),
help_text="Extra arguments to pass to srun as part of the "
"'sched.test_cmd' variable."),
"'sched.launch' variable."),
yc.ListElem(name='sbatch_extra',
sub_elem=yc.StrElem(),
help_text="Extra arguments to add as sbatch header lines. "
Expand Down
15 changes: 11 additions & 4 deletions lib/pavilion/schedulers/vars.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ class of this that contains all the variable functions it provides.

This class is meant to be inherited from - each scheduler can provide its own set of variables
in addition to these defaults, and may also provide different implementations of
each variable. Most schedulers can get away with overriding one variable - the 'test_cmd'
each variable. Most schedulers can get away with overriding one variable - the 'launch'
method. See the documentation for that method below for more information.

Return values of all variables should be the same format as those allowed by regular test
Expand All @@ -47,6 +47,7 @@ class of this that contains all the variable functions it provides.
'test_min_cpus': '4',
'test_min_mem': '32',
'tasks_total': '180',
'test_cmd': '# This is an alias to "launch"',
}

# Scheduler variable errors are deferred. We'll handle them later we we create
Expand Down Expand Up @@ -155,7 +156,7 @@ def partition(self):

return self._sched_config['partition'] or ''

def _test_cmd(self):
def _launch(self):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're still missing the alias.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrap the alias as a @dfr_var_method - that way it will work regardless of whether or not the function is deferred in any child class.

"""The command to prepend to a line to kick it off under the scheduler.

This should return the command needed to start one or more MPI processes within
Expand All @@ -176,14 +177,20 @@ def _test_cmd(self):

return ''

@var_method
@dfr_var_method
def test_cmd(self):
"""Alias to the launch command"""

return self.launch()

@var_method
def launch(self):
"""Calls the actual test command and then wraps the result with the wrapper
provided in the schedule section of the configuration."""

# Removes all the None values to avoid getting a TypeError while trying to
# join two commands
return ''.join(filter(lambda item: item is not None, [self._test_cmd(),
return ''.join(filter(lambda item: item is not None, [self._launch(),
self._sched_config['wrapper']]))

@dfr_var_method
Expand Down
4 changes: 2 additions & 2 deletions test/data/pav_config_dir/tests/sched_errors.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ a_error:
scheduler: dummy
run:
cmds:
- '{{sched.test_cmd}} echo "foo"'
- '{{sched.launch}} echo "foo"'

b_skipped:
inherits_from: a_error
Expand All @@ -25,4 +25,4 @@ d_no_nodes:

schedule:
nodes: 5
reservation: 'no_exist'
reservation: 'no_exist'
6 changes: 3 additions & 3 deletions test/tests/sched_tests.py
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ def test_sched_var_values(self):
NodeSet(frozenset(['node05', 'node06', 'node07']))]

expected = {
'test_cmd': '',
'launch': '',
'tasks_per_node': '1',
'chunk_ids': ['0', '1'],
'chunk_size': '3',
Expand Down Expand Up @@ -128,7 +128,7 @@ def test_sched_var_values_basic(self):
chunks = None

expected = {
'test_cmd': '',
'launch': '',
'tasks_per_node': '1',
'chunk_ids': [],
'chunk_size': '',
Expand Down Expand Up @@ -598,7 +598,7 @@ def test_wrapper(self):

# The wrapper can by anything a command or even a string
test_cfg['schedule'] = {'wrapper': 'echo'}
test_cfg['run']['cmds'] = ['{{sched.test_cmd}} "this is the wrapper test"']
test_cfg['run']['cmds'] = ['{{sched.launch}} "this is the wrapper test"']

test = self._quick_test(test_cfg, finalize=False)

Expand Down
Loading
Loading