Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: parallelize dockerized builds GitHub actions #28

Open
wants to merge 5 commits into
base: develop
Choose a base branch
from

Commits on Sep 22, 2020

  1. Merge pull request #27 from marcellodesales/develop

    Hotfix: Makefile: fix publish of Github version image
    marcellodesales authored Sep 22, 2020
    Configuration menu
    Copy the full SHA
    982d0f5 View commit details
    Browse the repository at this point in the history
  2. 🔨 Dockerfile: add stages for specifics

    According to the Buildkit library of docker, we can make parallel
    builds easier with multi-stage Dockerfiles.
    
    https://docs.docker.com/develop/develop-images/build_enhancements/#to-enable-buildkit-builds
    
    Here's an example:
    
    $ cat Dockerfile2
    FROM alpine AS dependencies
    
    RUN echo "Downloading dependencies"
    RUN sleep 15 && echo "dependencies done"
    
    FROM dependencies AS windows
    
    RUN echo "Compiling windows"
    RUN sleep 10 && echo "windows done"
    
    FROM dependencies AS linux
    
    RUN echo "Compiling linux"
    RUN sleep 10 && echo "linux done"
    
    $ DOCKER_BUILDKIT=1 docker build -t build-dependencies -f ./Dockerfile2  --target dependencies .
    [+] Building 0.1s (7/7) FINISHED
     => [internal] load build definition from Dockerfile2                                                                                                                                            0.0s
     => => transferring dockerfile: 38B                                                                                                                                                              0.0s
     => [internal] load .dockerignore                                                                                                                                                                0.0s
     => => transferring context: 2B                                                                                                                                                                  0.0s
     => [internal] load metadata for docker.io/library/alpine:latest                                                                                                                                 0.0s
     => [dependencies 1/3] FROM docker.io/library/alpine                                                                                                                                             0.0s
     => CACHED [dependencies 2/3] RUN echo "Downloading dependencies"                                                                                                                                0.0s
     => CACHED [dependencies 3/3] RUN sleep 15 && echo "dependencies done"                                                                                                                           0.0s
     => exporting to image                                                                                                                                                                           0.0s
     => => exporting layers                                                                                                                                                                          0.0s
     => => writing image sha256:b70fc38c238cddf7d88cb495634ce79cb49b0b9416018746e8523433f91a6015                                                                                                     0.0s
     => => naming to docker.io/library/build-dependencies                                                                                                                                            0.0s
    
    * Here, another feature is to use the build cache from the previous image built for dependencies.
    In addition, reusing the cache is shown as CACHED for the dependencies and it avoids building the entire
    docker image stages (when not using BUILDKIT=1)
    
    $ DOCKER_BUILDKIT=1 docker build -t windows-binary -f ./Dockerfile2  --target windows --cache-from=build-dependencies .
    [+] Building 0.0s (10/10) FINISHED
     => [internal] load .dockerignore                                                                                                                                                                0.0s
     => => transferring context: 2B                                                                                                                                                                  0.0s
     => [internal] load build definition from Dockerfile2                                                                                                                                            0.0s
     => => transferring dockerfile: 38B                                                                                                                                                              0.0s
     => [internal] load metadata for docker.io/library/alpine:latest                                                                                                                                 0.0s
     => importing cache manifest from build-dependencies                                                                                                                                             0.0s
     => [dependencies 1/3] FROM docker.io/library/alpine                                                                                                                                             0.0s
     => CACHED [dependencies 2/3] RUN echo "Downloading dependencies"                                                                                                                                0.0s
     => CACHED [dependencies 3/3] RUN sleep 15 && echo "dependencies done"                                                                                                                           0.0s
     => CACHED [windows 1/2] RUN echo "Compiling windows"                                                                                                                                            0.0s
     => CACHED [windows 2/2] RUN sleep 10 && echo "windows done"                                                                                                                                     0.0s
     => exporting to image                                                                                                                                                                           0.0s
     => => exporting layers                                                                                                                                                                          0.0s
     => => writing image sha256:cb2fdc9acfa878f50314d03e644f205729c8314c90d8323fb9203bbc16227275                                                                                                     0.0s
     => => naming to docker.io/library/windows-binary
    
    * Same here for the linux binaries, we can build and only the linux ones will be built with the help
    of the cache built previously.
    
    $ DOCKER_BUILDKIT=1 docker build -t linux-binary -f ./Dockerfile2  --target linux --cache-from=build-dependencies .
    [+] Building 0.1s (10/10) FINISHED
     => [internal] load build definition from Dockerfile2                                                                                                                                            0.0s
     => => transferring dockerfile: 38B                                                                                                                                                              0.0s
     => [internal] load .dockerignore                                                                                                                                                                0.0s
     => => transferring context: 2B                                                                                                                                                                  0.0s
     => [internal] load metadata for docker.io/library/alpine:latest                                                                                                                                 0.0s
     => importing cache manifest from build-dependencies                                                                                                                                             0.0s
     => [dependencies 1/3] FROM docker.io/library/alpine                                                                                                                                             0.0s
     => CACHED [dependencies 2/3] RUN echo "Downloading dependencies"                                                                                                                                0.0s
     => CACHED [dependencies 3/3] RUN sleep 15 && echo "dependencies done"                                                                                                                           0.0s
     => CACHED [linux 1/2] RUN echo "Compiling linux"                                                                                                                                                0.0s
     => CACHED [linux 2/2] RUN sleep 10 && echo "linux done"                                                                                                                                         0.0s
     => exporting to image                                                                                                                                                                           0.0s
     => => exporting layers                                                                                                                                                                          0.0s
     => => writing image sha256:35c85e91b47966432ae3d3c60bc455587bc46016eb528674b51b3f8c86a1e9da                                                                                                     0.0s
     => => naming to docker.io/library/linux-binary                                                                                                                                                  0.0s
    
    Our builds are to run with the same level of separation.
    
    *** For our builds with BUILDKIT, we have the following
    
    => Dockerifle
    
    * dependencies: downloads all the OS and Golang specific dependencies
    
    $ DOCKER_BUILDKIT=1 BIN_VERSION=20.09.10 docker-compose build dependencies
    
    * compiler: compiles the go code with the dependencies layer
    
    $ DOCKER_BUILDKIT=1 BIN_VERSION=20.09.10 PLATFORMS=darwin docker-compose build binaries
    
    * runtime: specific for the linux runtime
    
    $ DOCKER_BUILDKIT=1 PLATFORMS=linux BIN_VERSION=20.09.10 docker-compose build runtime
    marcellodesales committed Sep 22, 2020
    Configuration menu
    Copy the full SHA
    7eec3ad View commit details
    Browse the repository at this point in the history
  3. 🔨 Makefile: add constructs for parallel: docker BUILDKIT

    * build-dependencies: build the docker image with dependencies
    as cache to be reused by others.
    * compile: for specific language
    * runtime: for the linux runtime
    marcellodesales committed Sep 22, 2020
    Configuration menu
    Copy the full SHA
    7ebf42b View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2020

  1. ⚡ develop Workflow: save dependencies docker image

    This is to speed up the compile task so that it makes images faster
    marcellodesales committed Sep 23, 2020
    Configuration menu
    Copy the full SHA
    adf787b View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    6c481b1 View commit details
    Browse the repository at this point in the history