From 86e5fc4a7994f1fe9e4eb82af3918b2c305d9bc6 Mon Sep 17 00:00:00 2001 From: JulianTrommer <46600808+JulianTrommer@users.noreply.github.com> Date: Tue, 12 Nov 2024 09:21:19 +0100 Subject: [PATCH 1/2] Updated docs of GitHub actions --- .github/workflows/build.yml | 2 +- doc/development/build_action.md | 50 +++++++++++++++++++++++++++------ doc/development/drive_action.md | 46 ++++++++++++++++++++++++------ 3 files changed, 81 insertions(+), 17 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index b6682121..04e74f47 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -123,5 +123,5 @@ jobs: rm -rf /tmp/.buildx-cache/cache/latest mv /tmp/.buildx-cache/cache-new/latest /tmp/.buildx-cache/cache/latest - - name: Prune all images older than 1 days from self-hosted runner + - name: Prune all images older than 1 day from self-hosted runner run: docker image prune -a -f --filter "until=24h" \ No newline at end of file diff --git a/doc/development/build_action.md b/doc/development/build_action.md index 191f8ae2..ec534253 100644 --- a/doc/development/build_action.md +++ b/doc/development/build_action.md @@ -8,9 +8,16 @@ - [The `build-and-push-image` job](#the-build-and-push-image-job) - [1. Checkout repository (`actions/checkout@v3`)](#1-checkout-repository-actionscheckoutv3) - [2. Set up Docker Buildx (`docker/setup-buildx-action@v2`)](#2-set-up-docker-buildx-dockersetup-buildx-actionv2) - - [3. Log in to the Container registry (`docker/login-action@v2`)](#3-log-in-to-the-container-registry-dockerlogin-actionv2) - - [4. Test building the image (`docker/build-push-action@v3`)](#4-test-building-the-image-dockerbuild-push-actionv3) - - [5. Build and push the image (`docker/build-push-action@v3`)](#5-build-and-push-the-image-dockerbuild-push-actionv3) + - [3. Cache Docker layers](#3-cache-docker-layers) + - [4. Log in to the Container registry (`docker/login-action@v2`)](#4-log-in-to-the-container-registry-dockerlogin-actionv2) + - [5. Test building the image (`docker/build-push-action@v3`)](#5-test-building-the-image-dockerbuild-push-actionv3) + - [6. Build and push the image (`docker/build-push-action@v3`)](#6-build-and-push-the-image-dockerbuild-push-actionv3) + - [7. Save pull request artifact](#7-save-pull-request-artifact) + - [8. Save merge artifact](#8-save-merge-artifact) + - [9. Upload artifact](#9-upload-artifact) + - [10. Clean up PR Cache](#10-clean-up-pr-cache) + - [11. Clean up merge Cache](#11-clean-up-merge-cache) + - [12. Prune all images older than one day](#12-prune-all-images-older-than-one-day) ## General @@ -21,8 +28,7 @@ to [GitHub Packages](ghcr.io). The image can then be pulled with `docker pull ghcr.io/una-auxme/paf:latest` to get the latest version or `docker pull ghcr.io/una-auxme/paf:` to get a specific version. -If action is triggered by a pull request the created image is then used to execute a test run in the leaderboard, using -the devtest routes. The results of this simulation are then added as a comment to the pull request. +After the action is finished the `drive` action is triggered. ## The `build-and-push-image` job @@ -37,19 +43,47 @@ Set's up Buildx. This is needed to set up the correct driver to allow caching in Detailed description why this is needed can be found [here](https://github.com/docker/build-push-action/issues/163#issuecomment-1053657228). -### 3. Log in to the Container registry ([`docker/login-action@v2`](https://github.com/docker/login-action)) +### 3. Cache Docker layers + +Creates (if not done previously) and sets up the cache for the Docker layers. + +### 4. Log in to the Container registry ([`docker/login-action@v2`](https://github.com/docker/login-action)) Logs in with `GITHUB_TOKEN` into the registry (ghcr.io). Example taken from [here](https://docs.github.com/en/actions/publishing-packages/publishing-docker-images) -### 4. Test building the image ([`docker/build-push-action@v3`](https://github.com/docker/build-push-action/)) +### 5. Test building the image ([`docker/build-push-action@v3`](https://github.com/docker/build-push-action/)) Tries to build the image without pushing it to the repository packages if the workflow was triggered with a pull request. -### 5. Build and push the image ([`docker/build-push-action@v3`](https://github.com/docker/build-push-action/)) +### 6. Build and push the image ([`docker/build-push-action@v3`](https://github.com/docker/build-push-action/)) This action builds the image and pushes it to repository under the `latest` tag if the workflow was triggered with a merge to the `main` branch. To avoid large downloads of the base image the [GitHub Actions cache](https://docs.docker.com/build/building/cache/backends/gha/) is used to cache the image after build. + +### 7. Save pull request artifact + +If the action was triggered by a pull request an artifact is created with the corresponding PR ID (used for commenting on the PR in the `drive` action). + +### 8. Save merge artifact + +If the action was triggered by a merge an artifact is created with an invalid PR ID to signalise the `drive` action that it was not triggered by a PR. + +### 9. Upload artifact + +Uploads the artifact to the given path with a retention of the given number of days. + +### 10. Clean up PR Cache + +Removes the previous obsolete PR cache. + +### 11. Clean up merge Cache + +Removes the previous obsolete test cache. + +### 12. Prune all images older than one day + +Removes all images from the runner that are older than one day to free disk space. diff --git a/doc/development/drive_action.md b/doc/development/drive_action.md index b5570002..eef9ed89 100644 --- a/doc/development/drive_action.md +++ b/doc/development/drive_action.md @@ -4,11 +4,17 @@ - [The drive job](#the-drive-job) - [1. Checkout repository (`actions/checkout@v3`)](#1-checkout-repository-actionscheckoutv3) - - [2. Run agent with docker-compose](#2-run-agent-with-docker-compose) - - [3. Copy simulation results file out of container](#3-copy-simulation-results-file-out-of-container) - - [4. Stop docker-compose stack](#4-stop-docker-compose-stack) - - [5. Comment result in pull request `actions/github-script@v6`](#5-comment-result-in-pull-request-actionsgithub-scriptv6) + - [2. Download artifact](#2-download-artifact) + - [3. Unzip artifact](#3-unzip-artifact) + - [4. Return artifact JSON](#4-return-artifact-json) + - [5. Run agent with docker-compose](#5-run-agent-with-docker-compose) + - [6. Copy simulation results file out of container](#6-copy-simulation-results-file-out-of-container) + - [7. Stop docker-compose stack](#7-stop-docker-compose-stack) + - [8. Create simulation results table](#8-create-simulation-results-table) + - [9. Print simulation results](#9-print-simulation-results) + - [10. Comment result in pull request `actions/github-script@v6`](#10-comment-result-in-pull-request-actionsgithub-scriptv6) - [Simulation results](#simulation-results) + - [11. Prune all images older than one day](#11-prune-all-images-older-than-one-day) ## The drive job @@ -20,7 +26,19 @@ The `drive` job is executed conditionally on `pull_request`, after the build suc Same step as in the [build job](#1-checkout-repository--actionscheckoutv3-) -### 2. Run agent with docker-compose +### 2. Download artifact + +Downloads the artifact that was uploaded during the preceeding `build` action. + +### 3. Unzip artifact + +Extracts the files of the downloaded artifact. + +### 4. Return artifact JSON + +Parses the extracted file in the JSON format to read the information inside the file. + +### 5. Run agent with docker-compose Runs the agent with the [`build/docker-compose.cicd.yaml`](../../build/docker-compose.cicd.yaml) that only contains the bare minimum components for test execution: @@ -30,11 +48,11 @@ bare minimum components for test execution: - Agent container, run through the Carla [`leaderboard_evaluator`](https://github.com/carla-simulator/leaderboard/blob/leaderboard-2.0/leaderboard/leaderboard_evaluator.py). -### 3. Copy simulation results file out of container +### 6. Copy simulation results file out of container Copies the created `simulation_results.json` file out of the agent container into the current container -### 4. Stop docker-compose stack +### 7. Stop docker-compose stack Stops the remaining containers (Carla, roscore) and removes the volumes with: `$ docker-compose down -v`. @@ -42,7 +60,15 @@ Stops the remaining containers (Carla, roscore) and removes the volumes with: This step is important to clean up the remaining containers to have a clean run everytime. This is also the reason for the `if: always()`, that ensures step execution. -### 5. Comment result in pull request [`actions/github-script@v6`](https://github.com/marketplace/actions/github-script) +### 8. Create simulation results table + +Reads the simulation results an creates a table for better readability. + +### 9. Print simulation results + +Prints the simulation results table to the action. + +### 10. Comment result in pull request [`actions/github-script@v6`](https://github.com/marketplace/actions/github-script) This steps uses a JS script to parse the simulation results and add a comment with a results table to the corresponding pull request. @@ -68,3 +94,7 @@ An example comment for this would be: | Yield emergency vehicles infractions | 0.0 | | Scenario timeouts | 62.046 | | Min speed infractions | 0.0 | + +### 11. Prune all images older than one day + +Removes all images from the runner that are older than one day to free disk space. \ No newline at end of file From 3f7378710e8eef39b08dc98e9781b8e0588067d8 Mon Sep 17 00:00:00 2001 From: JulianTrommer <46600808+JulianTrommer@users.noreply.github.com> Date: Tue, 12 Nov 2024 09:30:50 +0100 Subject: [PATCH 2/2] Added newline to drive action --- doc/development/drive_action.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/development/drive_action.md b/doc/development/drive_action.md index eef9ed89..b734105b 100644 --- a/doc/development/drive_action.md +++ b/doc/development/drive_action.md @@ -97,4 +97,4 @@ An example comment for this would be: ### 11. Prune all images older than one day -Removes all images from the runner that are older than one day to free disk space. \ No newline at end of file +Removes all images from the runner that are older than one day to free disk space.