Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat!: rewrite in TypeScript with CacheMap support #25

Merged
merged 11 commits into from
Mar 31, 2024
1 change: 1 addition & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
* text=auto eol=lf
46 changes: 46 additions & 0 deletions .github/workflows/check-dist.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
name: Check Dist

on:
pull_request:
push:

permissions:
contents: read

jobs:
check-dist:
name: Check dist/
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4

- uses: actions/setup-node@v4
with:
node-version: 20

- uses: pnpm/action-setup@v3
with:
version: 8

- name: Install and Build
run: pnpm i

- name: Validate Dist
id: diff
run: |
if [ "$(git diff --ignore-space-at-eol --text dist/ | wc -l)" -gt "0" ]; then
echo "Detected uncommitted changes after build. See status below:"
git diff --ignore-space-at-eol --text dist/
exit 1
fi

# If `dist/` was different than expected, upload it
- if: ${{ failure() && steps.diff.outcome == 'failure' }}
name: Upload Artifact
id: upload
uses: actions/upload-artifact@v4
with:
name: dist
path: dist/
retention-days: 10
37 changes: 17 additions & 20 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,29 +14,26 @@ jobs:
id: meta
with:
images: test
- name: Cache var-cache-apt
id: cache-var-cache-apt
uses: actions/cache@v3

- name: Cache
uses: actions/cache@v4
id: cache
with:
path: var-cache-apt
key: var-cache-apt-${{ hashFiles('.github/workflows/test/Dockerfile') }}
- name: Cache var-lib-apt
id: cache-var-lib-apt
uses: actions/cache@v3
with:
path: var-lib-apt
key: var-lib-apt-${{ hashFiles('.github/workflows/test/Dockerfile') }}
- name: inject var-cache-apt into docker
uses: ./
with:
cache-source: var-cache-apt
cache-target: /var/cache/apt
- name: inject var-lib-apt into docker
path: |
var-cache-apt
var-lib-apt
key: cache-${{ hashFiles('.github/workflows/test/Dockerfile') }}

- name: inject cache into docker
uses: ./
with:
cache-source: var-lib-apt
cache-target: /var/lib/apt
skip-extraction: ${{ steps.cache-var-lib-apt.outputs.cache-hit }}
skip-extraction: false
cache-map: |
{
"var-cache-apt": "/var/cache/apt",
"var-lib-apt": "/var/lib/apt"
}

- name: Build and push
uses: docker/build-push-action@v5
with:
Expand Down
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
node_modules/
.parcel-cache/
96 changes: 68 additions & 28 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
# The BuildKit Cache Dance: save `RUN --mount=type=cache` caches on GitHub Actions
# The BuildKit Cache Dance
Save `RUN --mount=type=cache` caches on GitHub Actions or other CI platforms

The BuildKit Cache Dance allows saving [`RUN --mount=type=cache`](https://docs.docker.com/build/guide/mounts/#add-a-cache-mount)
caches on GitHub Actions.
caches on GitHub Actions or other CI platforms by extracting the cache from the previous build and injecting it into the current build.

Use cases:
- apt-get (`/var/cache/apt`, `/var/lib/apt`)
Expand All @@ -14,7 +15,9 @@ This [`reproducible-containers/buildkit-cache-dance`](https://github.com/reprodu
This action be used for "non-reproducible" containers too.

## Examples
### apt-get

### apt-get GitHub Actions

Dockerfile:
```dockerfile
FROM ubuntu:22.04
Expand All @@ -32,43 +35,39 @@ Action:
```yaml
---
name: Build
on: push
on:
push:

jobs:
build:
name: Build
Build:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: docker/setup-buildx-action@v3
- uses: docker/metadata-action@v5
id: meta
with:
images: YOUR_IMAGE
- name: Cache var-cache-apt
id: cache-var-cache-apt
uses: actions/cache@v3
with:
path: var-cache-apt
key: var-cache-apt-${{ hashFiles('Dockerfile') }}
- name: Cache var-lib-apt
id: cache-var-lib-apt
images: Build

- name: Cache
uses: actions/cache@v3
id: cache
with:
path: var-lib-apt
key: var-lib-apt-${{ hashFiles('Dockerfile') }}
- name: inject var-cache-apt into docker
uses: reproducible-containers/[email protected]
with:
cache-source: var-cache-apt
cache-target: /var/cache/apt
skip-extraction: ${{ steps.cache-var-cache-apt.outputs.cache-hit }}
- name: inject var-lib-apt into docker
uses: reproducible-containers/[email protected]
path: |
var-cache-apt
var-lib-apt
key: cache-${{ hashFiles('.github/workflows/test/Dockerfile') }}

- name: inject cache into docker
uses: reproducible-containers/[email protected]
aminya marked this conversation as resolved.
Show resolved Hide resolved
with:
cache-source: var-lib-apt
cache-target: /var/lib/apt
skip-extraction: ${{ steps.cache-var-lib-apt.outputs.cache-hit }}
cache-map: |
{
"var-cache-apt": "/var/cache/apt",
"var-lib-apt": "/var/lib/apt"
}
skip-extraction: ${{ steps.cache.outputs.cache-hit }}

- name: Build and push
uses: docker/build-push-action@v5
with:
Expand All @@ -79,12 +78,47 @@ jobs:
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}

```

Real-world examples:
- <https://github.com/rootless-containers/slirp4netns/blob/v1.2.2/.github/workflows/release.yaml#L18-L36>
- <https://github.com/containers/fuse-overlayfs/blob/40e0f3c/.github/workflows/release.yaml#L17-L36>

## CLI Usage

In other CI systems, you can run the script directly via `node`:

```shell
curl -LJO https://github.com/reproducible-containers/buildkit-cache-dance/archive/refs/tags/v3.0.0.tar.gz
tar xvf buildkit-cache-dance-3.0.0.tar.gz
```
During injection:

```shell
node ./buildkit-cache-dance-3.0.0/dist/index.js --cache-map '{"var-cache-apt": "/var/cache/apt", "var-lib-apt": "/var/lib/apt"}'
```

After build during extraction:

```shell
node ./buildkit-cache-dance-3.0.0/dist/index.js --extract --cache-map '{"var-cache-apt": "/var/cache/apt", "var-lib-apt": "/var/lib/apt"}'
```

Here are the available options:

```
build-cache-dance [options]
Save 'RUN --mount=type=cache' caches on GitHub Actions or other CI platforms

Options:
--extract Extract the cache from the docker container (extract step). Otherwise, inject the cache (main step)
--cache-map The map of actions source to container destination paths for the cache paths
--scratch-dir Where the action is stores some temporary files for its processing. Default: 'scratch'
--skip-extraction Skip the extraction of the cache from the docker container
--help Show this help
```

## Releases
### v1
v1 follows the original design of [`overmindtech/buildkit-cache-dance`](https://github.com/overmindtech/buildkit-cache-dance/tree/306d31a77191f643c0c4a95083f36c6ddccb4a16).
Expand All @@ -98,6 +132,12 @@ See the [`releases/v1`](https://github.com/reproducible-containers/buildkit-cach
### v2
v2 is composed of the single `reproducible-containers/buildkit-cache-dance` action.

### v3

Rewrote the action in TypeScript and adds support for `cache-map` that gets a string of files that need to be injected as a JSON string. This makes it possible to inject multiple directories in one call and simplifies the usage.

This release also makes it possible to run the script outside GitHub Actions in other CI platforms or locally using command line arguments.

## Acknowledgement
- Thanks to [Alexander Pravdin](https://github.com/speller) for the basic idea in [this comment](https://github.com/moby/buildkit/issues/1512).
- Thanks to the authors of the original [`overmindtech/buildkit-cache-dance`](https://github.com/overmindtech/buildkit-cache-dance).
11 changes: 7 additions & 4 deletions action.yml
Original file line number Diff line number Diff line change
@@ -1,11 +1,14 @@
name: Inject/Extract Cache
description: "Injects the cached data into the docker build(x|kit) process"
inputs:
cache-map:
required: true
description: "The map of actions source to container destination paths for the cache paths"
cache-source:
default: cache
deprecationMessage: "Use `cache-map` instead"
description: "Where the cache is stored in the calling workspace. Default: `cache`"
cache-target:
default: /root/.cache/go-build
deprecationMessage: "Use `cache-map` instead"
description: "Where the cache is stored in the docker container. Default: `/root/.cache/go-build`"
aminya marked this conversation as resolved.
Show resolved Hide resolved
scratch-dir:
default: scratch
Expand All @@ -15,5 +18,5 @@ inputs:
description: "Skip the extraction of the cache from the docker container"
runs:
using: 'node20'
main: 'entrypoint.js'
post: 'entrypoint.js'
main: 'dist/index.js'
post: 'dist/index.js'
Loading