Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

319 feature change docker compose service files #323

Merged
merged 4 commits into from
Oct 9, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ jobs:
if: github.event_name == 'pull_request'
env:
AGENT_VERSION: ${{ needs.build-and-push-image.outputs.version }}
COMPOSE_FILE: ./build/docker-compose_cicd.yaml
COMPOSE_FILE: ./build/docker-compose.cicd.yaml
steps:
- name: Checkout repository
uses: actions/checkout@v3
Expand Down
3 changes: 2 additions & 1 deletion .vscode/extensions.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
"ms-python.flake8",
"bierner.markdown-mermaid",
"richardkotze.git-mob",
"ms-vscode-remote.remote-containers"
"ms-vscode-remote.remote-containers",
"valentjn.vscode-ltex"
]
}
9 changes: 8 additions & 1 deletion .vscode/settings.json
Original file line number Diff line number Diff line change
Expand Up @@ -19,5 +19,12 @@
"query": "state:open repo:${owner}/${repository} sort:created-desc"
}
],
"ltex.language": "en-US"
"ltex.language": "en-US",
"docker.commands.composeUp": [
{
"label": "Compose Up",
"template": "xhost +local: && ${composeCommand} ${configurationFile} up"
}
],
"workbench.iconTheme": "vscode-icons"
}
94 changes: 45 additions & 49 deletions build/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,14 @@ facilitating both normal and distributed execution modes.
- [Directory Structure](#directory-structure)
- [Base Service Files](#base-service-files)
- [`agent_service.yaml`](#agent_serviceyaml)
- [`carla-simulator_service.yaml`](#carla-simulator_serviceyaml)
- [`linter_services.yaml`](#linter_servicesyaml)
- [`roscore_service.yaml`](#roscore_serviceyaml)
- [Docker Compose Files](#docker-compose-files)
- [`docker-compose.yaml`](#docker-composeyaml)
- [`docker-compose_dev.yaml`](#docker-compose_devyaml)
- [`docker-compose_cicd.yaml`](#docker-compose_cicdyaml)
- [`docker-compose.carla-simulator.yaml`](#docker-composecarla-simulatoryaml)
- [`docker-compose.linter.yaml`](#docker-composelinteryaml)
- [`docker-compose.leaderboard.yaml`](#docker-composeleaderboardyaml)
- [`docker-compose.devroute.yaml`](#docker-composedevrouteyaml)
- [`docker-compose.dev.yaml`](#docker-composedevyaml)
- [`docker-compose.cicd.yaml`](#docker-composecicdyaml)
- [Execution Modes](#execution-modes)
- [Normal Execution](#normal-execution)
- [Distributed Execution](#distributed-execution)
Expand All @@ -32,13 +33,14 @@ The `build` directory contains the necessary configuration and setup files for b

- **Base Service Files**
- `agent_service.yaml`
- `carla-simulator_service.yaml`
- `linter_services.yaml`
- `roscore_service.yaml`
- **Docker Compose Files**
- `docker-compose.yaml`
- `docker-compose_dev.yaml`
- `docker-compose_cicd.yaml`
- `docker-compose.carla-simulator.yaml`
- `docker-compose.linter.yaml`
- `docker-compose.leaderboard.yaml`
- `docker-compose.devroute.yaml`
- `docker-compose.dev.yaml`
- `docker-compose.cicd.yaml`

## Base Service Files

Expand All @@ -53,7 +55,20 @@ Defines the configuration for the `agent` service, which represents the autonomo
- **Volumes**: Mounts directories like `/workspace` to share code and data between the host and the container.
- **Networks**: Connects the agent to the `carla` and `ros` networks.

### `carla-simulator_service.yaml`
### `roscore_service.yaml`

Defines the `roscore` service for running the ROS master node. Key configurations include:

- **Image**: Uses the official ROS Noetic image.
- **Command**: Starts `roscore`.
- **Environment Variables**: Sets up ROS networking variables.
- **Networks**: Connects to the `ros` network.

## Docker Compose Files

The Docker Compose files allow the execution of different components or whole scenarios that include multiple services.

### `docker-compose.carla-simulator.yaml`

Defines the configuration for the `carla-simulator` service, which runs the CARLA simulator. Key configurations include:

Expand All @@ -63,43 +78,30 @@ Defines the configuration for the `carla-simulator` service, which runs the CARL
- **Volumes**: Shares the X11 UNIX socket and custom CARLA settings.
- **Networks**: Connects to the `carla` network.

### `linter_services.yaml`
### `docker-compose.linter.yaml`

Defines services for code linting and static analysis. Includes:

- **flake8**: For Python linting.
- **mdlint**: For Markdown file linting.
- **Volumes**: Mounts the project directory for linting files within the container.

### `roscore_service.yaml`

Defines the `roscore` service for running the ROS master node. Key configurations include:

- **Image**: Uses the official ROS Noetic image.
- **Command**: Starts `roscore`.
- **Environment Variables**: Sets up ROS networking variables.
- **Networks**: Connects to the `ros` network.

## Docker Compose Files

The Docker Compose files orchestrate multiple services defined in the base service files, allowing for different execution scenarios.

### `docker-compose.yaml`
### `docker-compose.leaderboard.yaml`

- **Includes**:
- `linter_services.yaml`
- `docker-compose.linter.yaml`
- `docker-compose.carla-simulator.yaml`
- `roscore_service.yaml`
- `carla-simulator_service.yaml`
- **Services**:
- Extends the `agent` service from `agent_service.yaml`.
- **Purpose**: Runs the agent with special scenarios included. Solving these scenarios is the primary goal of the project.

### `docker-compose_dev.yaml`
### `docker-compose.devroute.yaml`

- **Includes**:
- `linter_services.yaml`
- `docker-compose.linter.yaml`
- `docker-compose.carla-simulator.yaml`
- `roscore_service.yaml`
- `carla-simulator_service.yaml`
- **Services**:
- Extends the `agent` service from `agent_service.yaml`.
- **Environment Overrides**:
Expand All @@ -108,11 +110,17 @@ The Docker Compose files orchestrate multiple services defined in the base servi
- Runs the agent with simplified settings suitable for development and testing.
- **Purpose**: Provides a minimal setup for development without special scenarios.

### `docker-compose_cicd.yaml`
### `docker-compose.dev.yaml`

- **Services**:
- Defines an `agent-dev` service using the corresponding Dockerfile.
- **Purpose**: Provides a container for attaching a VS Code instance for development.

### `docker-compose.cicd.yaml`

- **Includes**:
- `docker-compose.carla-simulator.yaml`
- `roscore_service.yaml`
- `carla-simulator_service.yaml`
- **Services**:
- Defines an `agent` service using a prebuilt image from the project's container registry.
- **Dependencies**:
Expand All @@ -137,34 +145,22 @@ Distributed execution separates the agent and the CARLA simulator onto different
- Running large vision models that require extensive VRAM.
- The single machine's resources are insufficient to handle both the agent and simulator.

**Note**: In distributed execution, the CARLA simulator must be running on a second desktop PC, and the `CARLA_SIM_HOST` environment variable should be set accordingly.
**Note**: In distributed execution, the CARLA simulator must be running on a second desktop PC, and the `CARLA_SIM_HOST` environment variable should be set accordingly. Further information can be found in [here](../doc/02_development/14_distributed_simulation.md).

## Usage

To run the project using the provided Docker Compose files:

- **Standard Execution with Special Scenarios**:

```bash
docker-compose -f build/docker-compose.yaml up
```

- **Development Execution without Special Scenarios**:

```bash
docker-compose -f build/docker-compose_dev.yaml up
```
To run the project using the provided Docker Compose files simply navigate to the files in the VS Code Explorer and select `Compose Up` after right-clicking the file.

- **CI/CD Execution**:

The `docker-compose_cicd.yaml` is intended to be used within CI/CD pipelines and may be invoked as part of automated scripts.
The `docker-compose.cicd.yaml` is intended to be used within CI/CD pipelines and may be invoked as part of automated scripts.

## Notes

- Ensure that you have NVIDIA GPU support configured if running models that require GPU acceleration.
- The `agent_service.yaml` and other base service files are crucial for defining the common configurations and should not be modified unless necessary.
- When running in distributed mode, update the `CARLA_SIM_HOST` environment variable in the appropriate service configurations to point to the simulator's IP address.
- The linter services defined in `linter_services.yaml` can be used to maintain code quality and should be run regularly during development.
- The linter services defined in `docker-compose.linter.yaml` can be used to maintain code quality and should be run regularly during development.

## Conclusion

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
include:
# linter runs in a seperate workflow
- roscore_service.yaml
- carla-simulator_service.yaml
- docker-compose.carla-simulator.yaml

services:
agent:
Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# compose file for the development environment with distributed mode
include:
- linter_services.yaml
- docker-compose.linter.yaml
- roscore_service.yaml

services:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# compose file for the development environment
# routes_simple.xml
include:
- linter_services.yaml
- docker-compose.linter.yaml
- roscore_service.yaml
- carla-simulator_service.yaml
- docker-compose.carla-simulator.yaml

services:
agent:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
include:
- linter_services.yaml
- docker-compose.linter.yaml
- roscore_service.yaml

services:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
include:
- linter_services.yaml
- docker-compose.linter.yaml
- roscore_service.yaml
- carla-simulator_service.yaml
- docker-compose.carla-simulator.yaml

services:
agent:
Expand Down
File renamed without changes.
13 changes: 0 additions & 13 deletions dc-run-file.sh

This file was deleted.

9 changes: 3 additions & 6 deletions doc/02_development/14_distributed_simulation.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,15 +45,12 @@ Typically, the ip address is the first one in the list.

Replace the ip-address in the following files:

- `docker-compose.distributed.yml`
- `docker-compose.dev.distributed.yml`
- `build/docker-compose.devroute-distributed.yaml`
- `build/docker-compose.leaderboard-distributed.yaml`

### Start the agent on your local machine

```bash
docker compose -f build/docker-compose_distributed.yaml up
docker compose -f build/docker-compose_dev_distributed.yaml up
```
Navigate to the files mentioned above in the VS Code Explorer and select `Compose Up` after right-clicking one of the files.

## How do you know that you do not have enough compute resources?

Expand Down
3 changes: 0 additions & 3 deletions pc_setup_user.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,3 @@ cd
mkdir git
cd git
git clone https://github.com/una-auxme/paf.git

cd paf
./dc-run-file.sh build/docker-compose.yaml
4 changes: 0 additions & 4 deletions xhost_enable.sh

This file was deleted.

Loading