Skip to content

Commit

Permalink
Support for JVM package caching (+56 squashed commits)
Browse files Browse the repository at this point in the history 
Squashed commits:
[5cc823b] Remember to checkout windows code
[fd14df3] Windows cache management
[83fbabc] Remove cache dir
[281d41f] Better flow control
[ba7e617] Cleanup
[1089f4e] More path
[b52078c] More path
[08dd96a] Windows
[52a2a81] use script root
[37602fc] Windows
[fde1daa] Setup vcpkg
[2eb5733] More windows
[a9eafc3] More windows
[a4e9d40] Attempt to fix windows cmake
[224b683] Use vcpkg toolchain from Realm Core
[fe89a9b] No need to delete folder as runner is clean
[6728a61] Powershell syntax
[eb953b3] Attempt to fix windows cmake
[a37e15e] jni libs should be built even if jni stub is skipped
[642be64] More cache attempts
[379b512] Fix jni-swig-stub cache
[eb7683b] More cache fixes
[ff58582] Attempt to fix caching
[ef51caa] JNI caching and windows cmdline
[278d06d] Use proper line split
[8d83809] bat fixes
[81c1249] Attempt to build windows jvm
[7bb5f81] Attempt to fix docker build
[2ddf50c] swig also depends on cache
[99cbc9f] Proper stub task
[bd05668] Fix path
[39d73f0] Fix path
[4e5bb0d] Syntax
[e703dca] Syntax
[eff7755] Preliminary support for JVM
[cd6c060] Make sure that simulator builds work for both x64 and arm64 independently.
[51fa175] Fix ios arm64 build
[d81da94] Attempt to test ios
[f43eb95] Disable arm builds completely
[40db240] Add more documentation
[66aa337] Don't publish arm simulator for sync
[675c883] Fix target
[cf4806b] Proper dependency for ios tests
[dad3caf] First attempt at iOS build
[217e592] More fixes
[1cd0dac] Disable macos arm tests because we lack the Android SDK on the runner for now
[e1b4ee4] Matrix macOs builds
[ef78e1f] Fix macos caching
[b62fdfd] Run macos tests on macos, not android tests
[7537d71] Add support for building only macOS arm64
[a0aeb8b] Add macos testing
[87d1513] Re-enable builds
[0b401c9] Working version of delete file
[9be0848] Rework caching logic to work around issues with paths and split builds
[43fdaf3] Play around with paths
[bd55aab] Attempt to download to other cache path
  • Loading branch information
Christian Melchior committed Aug 29, 2022
1 parent 6036315 commit 6feb356
Show file tree
Hide file tree
Showing 4 changed files with 1,179 additions and 490 deletions.
255 changes: 211 additions & 44 deletions .github/workflows/include-check-cache.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
# Some notes on caching and artifacts:
# https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows
# - Caches are restricted to current back and fall back to default branch (master)
# - Caches cannot be downloaded to a new location.
# - Artifacts are restricted to current workflow.
#
name: Check cache
Expand All @@ -20,6 +21,8 @@ on:
outputs:
version-label:
value: ${{ jobs.check-cache.outputs.version-label }}
packages-jvm-cache-hit:
value: ${{ jobs.check-cache.outputs.packages-jvm-cache-hit }}
packages-android-cache-hit:
value: ${{ jobs.check-cache.outputs.packages-android-cache-hit }}
packages-macos-x64-cache-hit:
Expand All @@ -30,6 +33,12 @@ on:
value: ${{ jobs.check-cache.outputs.packages-ios-x64-cache-hit }}
packages-ios-arm64-cache-hit:
value: ${{ jobs.check-cache.outputs.packages-ios-arm64-cache-hit }}
jni-swig-stub-cache-hit:
value: ${{ jobs.check-cache.outputs.jni-swig-stub-cache-hit }}
jni-linux-lib-cache-hit:
value: ${{ jobs.check-cache.outputs.jni-linux-lib-cache-hit }}
jni-windows-lib-cache-hit:
value: ${{ jobs.check-cache-windows.outputs.jni-windows-lib-cache-hit }}
packages-sha:
value: ${{ jobs.check-cache.outputs.packages-sha }}
benchmarks-sha:
Expand All @@ -43,11 +52,14 @@ jobs:
CACHE_SKIP_SAVE: true
outputs:
version-label: ${{ steps.find-library-version.outputs.label }}
packages-jvm-cache-hit: ${{ steps.jvm-cache.outputs.cache-hit }}
packages-android-cache-hit: ${{ steps.android-cache.outputs.cache-hit }}
packages-macos-x64-cache-hit: ${{ steps.macos-x64-cache.outputs.cache-hit }}
packages-macos-arm64-cache-hit: ${{ steps.macos-arm64-cache.outputs.cache-hit }}
packages-ios-x64-cache-hit: ${{ steps.ios-x64-cache.outputs.cache-hit }}
packages-ios-arm64-cache-hit: ${{ steps.ios-arm64-cache.outputs.cache-hit }}
jni-swig-stub-cache-hit: ${{ steps.jni-swig-stub-cache.outputs.cache-hit }}
jni-linux-lib-cache-hit: ${{ steps.jni-linux-lib-cache.outputs.cache-hit }}
packages-sha: ${{ steps.packages-cache-key.outputs.sha }}
benchmarks-sha: ${{ steps.calculate-benchmarks-cache-key.outputs.sha }}

Expand All @@ -72,105 +84,260 @@ jobs:
run: echo "::set-output name=sha::${{ hashFiles('./packages/**') }}"

#
# Check caches for all targets
# For each specific package we need to perform 3 steps:
#
# 1. Check if a cache is available and download it if it is.
# 2. If (1), store this cache as an artifact for jobs downstream to use.
# 3. Cleanup the build folder. This is required so we can download the next
# platform into a fresh cache location. It does not look possible to download
# a cache into a different location.
#
# TODO There doesn't seem to be a good way to check if a cache key exists without download it.
# https://github.com/actions/cache/issues/321
#
# TODO Create a custom action for this until we have a work-around?
# Name of key must match output of `Store build artifacts`.
#

#
# JVM (All platforms)
#
- name: Check JVM cache
id: jvm-cache
uses: nirinchev/cache@d7c96a77c26ab70dd32b202c885cb4b34d95d8a8
with:
path: ./packages/build/m2-buildrepo/jvm
path: ./packages/build/m2-buildrepo
key: packages-m2-jvm-sync-${{ steps.packages-cache-key.outputs.sha }}

- name: Check Android cache
id: android-cache
uses: nirinchev/cache@d7c96a77c26ab70dd32b202c885cb4b34d95d8a8
- name: Save JVM packages
uses: actions/upload-artifact@v3
if: steps.jvm-cache.outputs.cache-hit == 'true'
with:
name: packages-jvm-${{ steps.find-library-version.outputs.label }}
path: ./packages/build/m2-buildrepo/**/*
retention-days: 1

- name: Delete downloaded JVM cache files
id: delete-cache-jvm
uses: JesseTG/[email protected]
if: steps.jvm-cache.outputs.cache-hit == 'true'
with:
path: ./packages/build/m2-buildrepo
key: packages-m2-android-sync-${{ steps.packages-cache-key.outputs.sha }}

- name: Check MacOS X64 cache
id: macos-x64-cache
#
# JNI Stub (JVM)
#

- name: Check JNI Swig stub cache
id: jni-swig-stub-cache
uses: nirinchev/cache@d7c96a77c26ab70dd32b202c885cb4b34d95d8a8
with:
path: ./packages/build/m2-buildrepo/macos-x64
key: packages-m2-macos-x64-sync-${{ steps.packages-cache-key.outputs.sha }}
path: ./packages/jni-swig-stub/build/generated/sources/jni
key: jni-swig-stubs-${{ steps.packages-cache-key.outputs.sha }}

- name: Check MacOS arm64 cache
id: macos-arm64-cache
uses: nirinchev/cache@d7c96a77c26ab70dd32b202c885cb4b34d95d8a8
- name: Save JNI Stub packages
uses: actions/upload-artifact@v3
if: steps.jni-swig-stub-cache.outputs.cache-hit == 'true'
with:
path: ./packages/build/m2-buildrepo/macos-arm64
key: packages-m2-macos-arm64-sync-${{ steps.packages-cache-key.outputs.sha }}
name: jni-stub-${{ steps.find-library-version.outputs.label }}
path: ./packages/jni-swig-stub/build/generated/sources/jni/*
retention-days: 1

- name: Check iOS X64 cache
id: ios-x64-cache
uses: nirinchev/cache@d7c96a77c26ab70dd32b202c885cb4b34d95d8a8
- name: Delete downloaded JVM cache files
id: delete-cache-jni-stub
uses: JesseTG/[email protected]
if: steps.jni-swig-stub-cache.outputs.cache-hit == 'true'
with:
path: ./packages/build/m2-buildrepo/ios-x64
key: packages-m2-ios-x64-sync-${{ steps.packages-cache-key.outputs.sha }}
path: ./packages/jni-swig-stub/build/generated/sources/jni

- name: Check iOS arm64 cache
id: ios-arm64-cache
#
# JNI Linux Lib
#

- name: Check JNI Linux lib cache
id: jni-linux-lib-cache
uses: nirinchev/cache@d7c96a77c26ab70dd32b202c885cb4b34d95d8a8
with:
path: ./packages/build/m2-buildrepo/ios-arm64
key: packages-m2-ios-arm64-sync-${{ steps.packages-cache-key.outputs.sha }}
path: ./packages/cinterop/src/jvmMain/linux-build-dir
key: jni-linux-lib-${{ steps.packages-cache-key.outputs.sha }}

- name: Save JNI Linux lib package
uses: actions/upload-artifact@v3
if: steps.jni-linux-lib-cache.outputs.cache-hit == 'true'
with:
name: jni-linux-lib-${{ steps.find-library-version.outputs.label }}
path: ./packages/cinterop/src/jvmMain/linux-build-dir/librealmc.so
retention-days: 1

- name: Delete downloaded JVM cache files
id: delete-cache-linux-lib
uses: JesseTG/[email protected]
if: steps.jni-linux-lib-cache.outputs.cache-hit == 'true'
with:
path: ./packages/cinterop/src/jvmMain/linux-build-dir

#
# Upload artifacts for all packages found in the cache, so they can be used by
# downstream jobs.
# Android
#
- name: Check Android cache
id: android-cache
uses: nirinchev/cache@d7c96a77c26ab70dd32b202c885cb4b34d95d8a8
with:
path: ./packages/build/m2-buildrepo
key: packages-m2-android-sync-${{ steps.packages-cache-key.outputs.sha }}

- name: Save JVM packages
- name: Save Android packages
uses: actions/upload-artifact@v3
if: steps.jvm-cache.outputs.cache-hit == 'true'
if: steps.android-cache.outputs.cache-hit == 'true'
with:
name: packages-android-${{ steps.find-library-version.outputs.label }}
path: ./packages/build/m2-buildrepo/jvm/**/*
path: ./packages/build/m2-buildrepo/**/*
retention-days: 1

- name: Save Android packages
uses: actions/upload-artifact@v3
- name: Delete downloaded Android cache files
id: delete-cache-android
uses: JesseTG/[email protected]
if: steps.android-cache.outputs.cache-hit == 'true'
with:
name: packages-android-${{ steps.find-library-version.outputs.label }}
path: ./packages/build/m2-buildrepo/android/**/*
path: ./packages/build/m2-buildrepo

#
# MacOS arm64
#
- name: Check MacOS arm64 cache
id: macos-arm64-cache
uses: nirinchev/cache@d7c96a77c26ab70dd32b202c885cb4b34d95d8a8
with:
path: ./packages/build/m2-buildrepo
key: packages-m2-macos-arm64-sync-${{ steps.packages-cache-key.outputs.sha }}

- name: Save MacOS arm64 packages
uses: actions/upload-artifact@v3
if: steps.macos-arm64-cache.outputs.cache-hit == 'true'
with:
name: packages-macos-arm64-${{ steps.find-library-version.outputs.label }}
path: ./packages/build/m2-buildrepo/**/*
retention-days: 1

- name: Delete downloaded MacOS arm64 cache files
id: delete-cache-macos-arm64
uses: JesseTG/[email protected]
if: steps.macos-arm64-cache.outputs.cache-hit == 'true'
with:
path: ./packages/build/m2-buildrepo

#
# MacOS x64
#
- name: Check MacOS X64 cache
id: macos-x64-cache
uses: nirinchev/cache@d7c96a77c26ab70dd32b202c885cb4b34d95d8a8
with:
path: ./packages/build/m2-buildrepo
key: packages-m2-macos-x64-sync-${{ steps.packages-cache-key.outputs.sha }}

- name: Save MacOS x64 packages
uses: actions/upload-artifact@v3
if: steps.macos-x64-cache.outputs.cache-hit == 'true'
with:
name: packages-macos-x64-${{ steps.find-library-version.outputs.label }}
path: ./packages/build/m2-buildrepo/macos-x64/**/*
path: ./packages/build/m2-buildrepo/**/*
retention-days: 1

- name: Save MacOS arm64 packages
- name: Delete downloaded MacOS x64 cache files
id: delete-cache-macos-x64
uses: JesseTG/[email protected]
if: steps.macos-x64-cache.outputs.cache-hit == 'true'
with:
path: ./packages/build/m2-buildrepo

#
# iOS arm64
#
- name: Check iOS arm64 cache
id: ios-arm64-cache
uses: nirinchev/cache@d7c96a77c26ab70dd32b202c885cb4b34d95d8a8
with:
path: ./packages/build/m2-buildrepo
key: packages-m2-ios-arm64-sync-${{ steps.packages-cache-key.outputs.sha }}

- name: Save iOS arm64 packages
uses: actions/upload-artifact@v3
if: steps.macos-arm64-cache.outputs.cache-hit == 'true'
if: steps.ios-arm64-cache.outputs.cache-hit == 'true'
with:
name: packages-macos-arm64-${{ steps.find-library-version.outputs.label }}
path: ./packages/build/m2-buildrepo/macos-arm64/**/*
name: packages-ios-arm64-${{ steps.find-library-version.outputs.label }}
path: ./packages/build/m2-buildrepo/**/*
retention-days: 1

- name: Delete downloaded iOS arm64 cache files
id: delete-cache-ios-arm64
uses: JesseTG/[email protected]
if: steps.ios-arm64-cache.outputs.cache-hit == 'true'
with:
path: ./packages/build/m2-buildrepo

#
# iOS x64
#
- name: Check iOS X64 cache
id: ios-x64-cache
uses: nirinchev/cache@d7c96a77c26ab70dd32b202c885cb4b34d95d8a8
with:
path: ./packages/build/m2-buildrepo
key: packages-m2-ios-x64-sync-${{ steps.packages-cache-key.outputs.sha }}

- name: Save iOS x64 packages
uses: actions/upload-artifact@v3
if: steps.ios-x64-cache.outputs.cache-hit == 'true'
with:
name: packages-ios-x64-${{ steps.find-library-version.outputs.label }}
path: ./packages/build/m2-buildrepo/ios-x64/**/*
path: ./packages/build/m2-buildrepo/**/*
retention-days: 1

- name: Save iOS arm64 packages
- name: Delete downloaded iOS x64 cache files
id: delete-cache-ios-x64
uses: JesseTG/[email protected]
if: steps.ios-x64-cache.outputs.cache-hit == 'true'
with:
path: ./packages/build/m2-buildrepo

# Caches between Linux and Windows cannot be used, so use Windows runners to verify Windows cache state.
# See https://github.com/actions/cache#cache-version
check-cache-windows:
runs-on: windows-latest
name: Check Windows cache
needs: check-cache
env:
CACHE_SKIP_SAVE: true
outputs:
jni-windows-lib-cache-hit: ${{ steps.jni-windows-lib-cache.outputs.cache-hit }}

steps:

#
# JNI Windows Lib
#

- name: Check JNI Windows lib cache
id: jni-windows-lib-cache
uses: nirinchev/cache@d7c96a77c26ab70dd32b202c885cb4b34d95d8a8
with:
path: ./packages/cinterop/src/jvmMain/windows-build-dir
key: jni-windows-lib-${{ needs.check-cache.outputs.packages-sha }}

- name: Save JNI Windows lib package
uses: actions/upload-artifact@v3
if: steps.ios-arm64-cache.outputs.cache-hit == 'true'
if: steps.jni-windows-lib-cache.outputs.cache-hit == 'true'
with:
name: packages-ios-arm64-${{ steps.find-library-version.outputs.label }}
path: ./packages/build/m2-buildrepo/ios-arm64/**/*
name: jni-windows-lib-${{ steps.find-library-version.outputs.label }}
path: ./packages/cinterop/src/jvmMain/windows-build-dir/**/*
retention-days: 1

- name: Delete downloaded JNI Windows lib cache files
id: delete-cache-windows-lib
uses: JesseTG/[email protected]
if: steps.jni-windows-lib-cache.outputs.cache-hit == 'true'
with:
path: ./packages/cinterop/src/jvmMain/windows-build-dir



Loading

0 comments on commit 6feb356

Please sign in to comment.