Skip to content

Commit

Permalink
[Envs] Add documentation for environments
Browse files Browse the repository at this point in the history
  • Loading branch information
geoffxy committed Jan 6, 2025
1 parent fe2d3ec commit 5c620c2
Show file tree
Hide file tree
Showing 7 changed files with 152 additions and 3 deletions.
3 changes: 3 additions & 0 deletions website/docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@ each of Conductor's subcommands.
- [`cond restore`](cli/restore.md): Restore results from an archive generated
by `cond archive`
- [`cond clean`](cli/clean.md): Remove Conductor's output files
- [`cond gc`](cli/gc.md): Remove task output files associated with failed tasks
- [`cond explorer`](cli/explorer.md): Launches Conductor's results explorer user
interface

## `cond` Options

Expand Down
17 changes: 17 additions & 0 deletions website/docs/cli/explorer.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,23 @@ This command launches Conductor's results explorer user interface. This tool is
used to interactively examine the experiment result versions managed by
Conductor.

:::note

To use Conductor's results explorer, you need to have installed Conductor with
the `[explorer]` extra.

```bash
pip3 install conductor-cli[explorer]
```

or

```bash
pip3 install conductor-cli[all]
```

:::

## Optional Arguments

### `--port`
Expand Down
104 changes: 104 additions & 0 deletions website/docs/directives/environment.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
---
title: environment()
id: environment
---

```python
environment(name, connect_config, start=None, stop=None)
```

The `environment()` directive lets you define a remote environment where tasks
can run (e.g., a remote server, a VM in the cloud, etc.).

For Conductor, an environment is a machine that it can connect to using SSH. You
use the `environment()` directive to tell Conductor how to connect to the
machine. You can then specify an environment for a `run_command()` or
`run_experiment()` using the `env` argument and Conductor will transparently
execute the task in the environment on your behalf. See
[`run_experiment()`](task-types/run-experiment.md#env) for more details.

Conductor environments work best with Linux-based machines. Your machine must
have `git` and other standard utilities (e.g., `bash`) installed.

:::note

To use environments, you need to have installed Conductor with the `[envs]`
extra.

```bash
pip3 install conductor-cli[envs]
```

or

```bash
pip3 install conductor-cli[all]
```

:::

## Arguments

### `name`

**Type:** String (required)

The name of the environment. This name must be unique within your Conductor
project. An environment name can only contain letters, numbers, hyphens (`-`),
and underscores (`_`). `run_experiment()` and `run_command()` tasks refer to an
environment by this name.

### `connect_config`

**Type:** String (required)

An executable to run, which generates a configuration string (described below)
used to connect to the remote environment using SSH. Conductor runs this
executable after `start` has run, which allows you to dynamically set the SSH
host (e.g., a dynamic IP that is assigned after a cloud VM starts).

This executable must generate a TOML string with the following format

```toml title="Example generated configuration"
host = 192.168.0.1
user = geoffxy
```

Here `host` must refer to a SSH-able host (e.g., IP address, domain name, SSH
hostname) and `user` is the username to log in as. Note that you must have SSH
authentication set up for the remote machine. You can use hosts that are defined
in your local `~/.ssh/config` file.

### `start`

**Type:** String (default: `None`)

An optional executable to run to "start" the remote environment. For example, if
your remote environment is a cloud VM, this script can be used to start the VM.

### `stop`

**Type:** String (default: `None`)

An optional executable to run to "stop" (i.e., shutdown) the remote environment.
For example, if your remote environment is a cloud VM, this script can be used
to shutdown the VM.

## Usage Example

In this example, we define an environment called `my_remote_machine`. We include
a `config.sh` script that generates the config values we want Conductor to use
to SSH into our remote machine. Conductor will use SSH keys to connect (you need
to set up your keys beforehand).

```python
environment(
name="my_remote_machine",
connect_config="./config.sh",
)
```

```bash title="config.sh"
echo "host = 192.168.0.1"
echo "user = geoffxy"
```
10 changes: 9 additions & 1 deletion website/docs/task-types/run-command.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ id: run-command
---

```python
run_command(name, run, parallelizable=False, args=[], options={}, deps=[])
run_command(name, run, parallelizable=False, args=[], options={}, env=None, deps=[])
```

A `run_command()` task runs the command specified in the `run` argument. The
Expand Down Expand Up @@ -104,6 +104,14 @@ run_command(
Conductor will execute the task shown above by running `./run.sh arg1 --foo=3
--bar=true` in `bash`.

### `env`

**Type:** Task identifier (default: `None`)

The identifier of the remote [environment](directives/environment.md) in which
to run this task, if any. This identifier can be fully qualified or relative,
just like identifiers in `deps`.

### `deps`

**Type:** List of task identifiers (default: `[]`)
Expand Down
10 changes: 9 additions & 1 deletion website/docs/task-types/run-experiment-group.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ id: run-experiment-group
---

```python
run_experiment_group(name, run, experiments=[], chain_experiments=False, deps=[])
run_experiment_group(name, run, experiments=[], chain_experiments=False, env=None, deps=[])
```

A `run_experiment_group()` task lets you specify a list of experiments that
Expand Down Expand Up @@ -83,6 +83,14 @@ This argument is useful when you want to run different experiment _groups_
concurrently, but do not want the experiments within one group to run
concurrently.

### `env`

**Type:** Task identifier (default: `None`)

The identifier of the remote [environment](directives/environment.md) in which
to run this experiment group, if any. This identifier can be fully qualified or
relative, just like identifiers in `deps`.

### `deps`

**Type:** List of task identifiers (default: `[]`)
Expand Down
10 changes: 9 additions & 1 deletion website/docs/task-types/run-experiment.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ id: run-experiment
---

```python
run_experiment(name, run, parallelizable=False, args=[], options={}, deps=[])
run_experiment(name, run, parallelizable=False, args=[], options={}, env=None, deps=[])
```

A `run_experiment()` task runs the command specified in the `run` argument. The
Expand Down Expand Up @@ -101,6 +101,14 @@ run_experiment(
Conductor will execute the task shown above by running `./run.sh arg1 --foo=3
--bar=true` in `bash`.

### `env`

**Type:** Task identifier (default: `None`)

The identifier of the remote [environment](directives/environment.md) in which
to run this task, if any. This identifier can be fully qualified or relative,
just like identifiers in `deps`.

### `deps`

**Type:** List of task identifiers (default: `[]`)
Expand Down
1 change: 1 addition & 0 deletions website/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ module.exports = {
collapsed: false,
items: [
'directives/include',
'directives/environment',
],
},
{
Expand Down

0 comments on commit 5c620c2

Please sign in to comment.