Skip to content

Commit

Permalink
Add ARM image building for regular PRs (apache#24664)
Browse files Browse the repository at this point in the history 
The image building for ARM is currently only done in the main build
only to refresh cache, however there are sometimes cases when
new dependency (for example apache#24635) broke ARM image build and it
was only discovered after merge.

This PR adds extra ARM-based build that should be run after
the AMD64 build. It should not influence the depending steps,
it should just signal failure of the PR if the ARM image cannot
be build.

(cherry picked from commit 2fbd750)
  • Loading branch information
@potiuk
potiuk committed Jun 29, 2022
1 parent 0d659c9 commit 8913c99
Show file tree
Hide file tree
Showing 5 changed files with 507 additions and 317 deletions.
105 changes: 102 additions & 3 deletions .github/workflows/build-images.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -225,21 +225,35 @@ jobs:
- name: "Free space"
run: breeze free-space
- name: >
Build & Push CI images ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
Build & Push AMD64 CI images ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
${{ needs.build-info.outputs.allPythonVersionsListAsString }}
run: breeze build-image --push-image --tag-as-latest --run-in-parallel
if: matrix.platform == 'linux/amd64'
env:
UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgradeToNewerDependencies }}
DOCKER_CACHE: ${{ needs.build-info.outputs.cacheDirective }}
IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
PYTHON_VERSIONS: ${{ needs.build-info.outputs.allPythonVersionsListAsString }}
- name: "Start ARM instance"
run: ./scripts/ci/images/ci_start_arm_instance_and_connect_to_docker.sh
if: matrix.platform == 'linux/arm64'
- name: >
Build ARM CI images ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
${{ needs.build-info.outputs.allPythonVersionsListAsString }}
run: breeze build-image --run-in-parallel
if: matrix.platform == 'linux/arm64'
env:
UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgradeToNewerDependencies }}
DOCKER_CACHE: ${{ needs.build-info.outputs.cacheDirective }}
IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
PYTHON_VERSIONS: ${{ needs.build-info.outputs.allPythonVersionsListAsString }}
- name: Push empty CI image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
if: failure() || cancelled()
if: (failure() || cancelled()) && matrix.platform == 'linux/amd64'
run: breeze build-image --push-image --empty-image --run-in-parallel
env:
IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- name: "Candidates for pip resolver backtrack triggers"
if: failure() || cancelled()
if: (failure() || cancelled()) && matrix.platform == 'linux/amd64'
run: >
breeze find-newer-dependencies --max-age 1
--python "${{ needs.build-info.outputs.defaultPythonVersion }}"
Expand Down Expand Up @@ -360,3 +374,88 @@ jobs:
- name: "Fix ownership"
run: breeze fix-ownership
if: always()


build-ci-images-arm:
timeout-minutes: 80
name: "Build ARM CI images ${{ needs.build-info.outputs.allPythonVersionsListAsString }}"
runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }}
needs: [build-info, build-prod-images]
if: |
needs.build-info.outputs.image-build == 'true' &&
needs.build-info.outputs.upgradeToNewerDependencies != 'false' &&
github.event.pull_request.head.repo.full_name != 'apache/airflow'
env:
RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn)[0] }}
BACKEND: sqlite
outputs: ${{toJSON(needs.build-info.outputs) }}
steps:
- name: Cleanup repo
run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
- uses: actions/checkout@v2
with:
ref: ${{ needs.build-info.outputs.targetCommitSha }}
persist-credentials: false
submodules: recursive
- name: "Retrieve DEFAULTS from the _initialization.sh"
# We cannot "source" the script here because that would be a security problem (we cannot run
# any code that comes from the sources coming from the PR. Therefore, we extract the
# DEFAULT_BRANCH and DEFAULT_CONSTRAINTS_BRANCH and DEBIAN_VERSION via custom grep/awk/sed commands
id: defaults
run: |
DEFAULT_BRANCH=$(grep "export DEFAULT_BRANCH" scripts/ci/libraries/_initialization.sh | \
awk 'BEGIN{FS="="} {print $3}' | sed s'/["}]//g')
echo "DEFAULT_BRANCH=${DEFAULT_BRANCH}" >> $GITHUB_ENV
DEFAULT_CONSTRAINTS_BRANCH=$(grep "export DEFAULT_CONSTRAINTS_BRANCH" \
scripts/ci/libraries/_initialization.sh | \
awk 'BEGIN{FS="="} {print $3}' | sed s'/["}]//g')
echo "DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH}" >> $GITHUB_ENV
DEBIAN_VERSION=$(grep "export DEBIAN_VERSION" scripts/ci/libraries/_initialization.sh | \
awk 'BEGIN{FS="="} {print $3}' | sed s'/["}]//g')
echo "DEBIAN_VERSION=${DEBIAN_VERSION}" >> $GITHUB_ENV
- name: >
Checkout "${{ needs.build-info.outputs.targetBranch }}" branch to 'main-airflow' folder
to use ci/scripts from there.
uses: actions/checkout@v2
with:
path: "main-airflow"
ref: "${{ needs.build-info.outputs.targetBranch }}"
persist-credentials: false
submodules: recursive
- name: >
Override "scripts/ci" with the "${{ needs.build-info.outputs.targetBranch }}" branch
so that the PR does not override it
# We should not override those scripts which become part of the image as they will not be
# changed in the image built - we should only override those that are executed to build
# the image.
run: |
rm -rfv "scripts/ci"
rm -rfv "dev"
mv -v "main-airflow/scripts/ci" "scripts"
mv -v "main-airflow/dev" "."
- name: "Setup python"
uses: actions/setup-python@v2
with:
python-version: ${{ needs.build-info.outputs.defaultPythonVersion }}
- run: ./scripts/ci/install_breeze.sh
- name: "Free space"
run: breeze free-space
- name: "Start ARM instance"
run: ./scripts/ci/images/ci_start_arm_instance_and_connect_to_docker.sh
- name: >
Build ARM CI images ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
${{ needs.build-info.outputs.allPythonVersionsListAsString }}
run: >
breeze build-image --run-in-parallel --parallelism 1
--builder airflow_cache --platform "linux/arm64"
env:
UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgradeToNewerDependencies }}
DOCKER_CACHE: ${{ needs.build-info.outputs.cacheDirective }}
IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
PYTHON_VERSIONS: ${{ needs.build-info.outputs.allPythonVersionsListAsString }}
- name: "Stop ARM instance"
run: ./scripts/ci/images/ci_stop_arm_instance.sh
if: always()
- name: "Fix ownership"
run: breeze fix-ownership
if: always()
77 changes: 77 additions & 0 deletions .github/workflows/ci.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1749,3 +1749,80 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
- name: "Fix ownership"
run: breeze fix-ownership
if: always()

build-ci-arm-images:
timeout-minutes: 80
name: >
${{needs.build-info.outputs.buildJobDescription}} CI ARM images
${{ needs.build-info.outputs.allPythonVersionsListAsString }}
runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }}
needs:
- build-info
- wait-for-ci-images
- wait-for-prod-images
- static-checks
- tests-sqlite
- tests-mysql
- tests-mssql
- tests-postgres
env:
RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn)[0] }}
if: needs.build-info.outputs.upgradeToNewerDependencies != 'false'
steps:
- name: Cleanup repo
run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
if: needs.build-info.outputs.inWorkflowBuild == 'true'
- uses: actions/checkout@v2
with:
ref: ${{ needs.build-info.outputs.targetCommitSha }}
persist-credentials: false
submodules: recursive
if: needs.build-info.outputs.inWorkflowBuild == 'true'
- name: "Setup python"
uses: actions/setup-python@v2
with:
python-version: ${{ needs.build-info.outputs.defaultPythonVersion }}
if: needs.build-info.outputs.inWorkflowBuild == 'true'
- name: "Retrieve DEFAULTS from the _initialization.sh"
# We cannot "source" the script here because that would be a security problem (we cannot run
# any code that comes from the sources coming from the PR. Therefore, we extract the
# DEFAULT_BRANCH and DEFAULT_CONSTRAINTS_BRANCH and DEBIAN_VERSION via custom grep/awk/sed commands
id: defaults
run: |
DEFAULT_BRANCH=$(grep "export DEFAULT_BRANCH" scripts/ci/libraries/_initialization.sh | \
awk 'BEGIN{FS="="} {print $3}' | sed s'/["}]//g')
echo "DEFAULT_BRANCH=${DEFAULT_BRANCH}" >> $GITHUB_ENV
DEFAULT_CONSTRAINTS_BRANCH=$(grep "export DEFAULT_CONSTRAINTS_BRANCH" \
scripts/ci/libraries/_initialization.sh | \
awk 'BEGIN{FS="="} {print $3}' | sed s'/["}]//g')
echo "DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH}" >> $GITHUB_ENV
DEBIAN_VERSION=$(grep "export DEBIAN_VERSION" scripts/ci/libraries/_initialization.sh | \
awk 'BEGIN{FS="="} {print $3}' | sed s'/["}]//g')
echo "DEBIAN_VERSION=${DEBIAN_VERSION}" >> $GITHUB_ENV
if: needs.build-info.outputs.inWorkflowBuild == 'true'
- run: ./scripts/ci/install_breeze.sh
if: needs.build-info.outputs.inWorkflowBuild == 'true'
- name: "Free space"
run: breeze free-space
if: needs.build-info.outputs.inWorkflowBuild == 'true'
- name: "Start ARM instance"
run: ./scripts/ci/images/ci_start_arm_instance_and_connect_to_docker.sh
if: needs.build-info.outputs.inWorkflowBuild == 'true'
- name: >
Build CI ARM images ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
${{ needs.build-info.outputs.allPythonVersionsListAsString }}
run: >
breeze build-image --run-in-parallel --parallelism 1
--builder airflow_cache --platform "linux/arm64"
env:
UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgradeToNewerDependencies }}
DOCKER_CACHE: ${{ needs.build-info.outputs.cacheDirective }}
IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
PYTHON_VERSIONS: ${{ needs.build-info.outputs.allPythonVersionsListAsString }}
if: needs.build-info.outputs.inWorkflowBuild == 'true'
- name: "Stop ARM instance"
run: ./scripts/ci/images/ci_stop_arm_instance.sh
if: always() && needs.build-info.outputs.inWorkflowBuild == 'true'
- name: "Fix ownership"
run: breeze fix-ownership
if: always() && needs.build-info.outputs.inWorkflowBuild == 'true'
Loading

0 comments on commit 8913c99

Please sign in to comment.