Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cache action to speed up mvn workflow [skip ci] #9674

Merged
merged 52 commits into from
Dec 1, 2023
Merged

Add cache action to speed up mvn workflow [skip ci] #9674

Changes from 44 commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
e902052
add cache for testing
YanxuanLiu Nov 10, 2023
75229e8
change cache key file
YanxuanLiu Nov 10, 2023
d0f62a8
add cache for all stages
YanxuanLiu Nov 10, 2023
086dbf0
change key path
YanxuanLiu Nov 10, 2023
af5ed08
add mvn help to test
YanxuanLiu Nov 13, 2023
bd2df16
add isolated cache job
YanxuanLiu Nov 21, 2023
dc57563
add checkout
YanxuanLiu Nov 21, 2023
aa887b1
add java setup
YanxuanLiu Nov 22, 2023
3d1c41e
remove distribution
YanxuanLiu Nov 22, 2023
7612e46
add distribution
YanxuanLiu Nov 22, 2023
ce8f3e9
add retry for cache
YanxuanLiu Nov 22, 2023
96353f9
add project param
YanxuanLiu Nov 23, 2023
7e2c75e
make sure cache finished before shim version matrix
YanxuanLiu Nov 23, 2023
0af4d88
add cache
YanxuanLiu Nov 24, 2023
c44ee1b
add cache
YanxuanLiu Nov 24, 2023
b357d6e
add cache
YanxuanLiu Nov 24, 2023
47c6eaa
add check
YanxuanLiu Nov 27, 2023
c4ee2d2
remove test step
YanxuanLiu Nov 27, 2023
7a12ba9
add ls before cache
YanxuanLiu Nov 27, 2023
0a7172a
add check
YanxuanLiu Nov 27, 2023
2ade9fd
removed test steps
YanxuanLiu Nov 27, 2023
65655ae
add new mvn command, and changed key
YanxuanLiu Nov 28, 2023
fdf8bd0
change mvn command
YanxuanLiu Nov 28, 2023
cf4d90d
change mvn command
YanxuanLiu Nov 28, 2023
c97fdec
Update .github/workflows/mvn-verify-check.yml
YanxuanLiu Nov 29, 2023
3c0d047
Update .github/workflows/mvn-verify-check.yml
YanxuanLiu Nov 29, 2023
be5135c
change key format
YanxuanLiu Nov 29, 2023
d6f8659
Merge branch 'branch-24.02' into cache-deps
YanxuanLiu Nov 29, 2023
f643bbc
change time format
YanxuanLiu Nov 29, 2023
845b13d
add date
YanxuanLiu Nov 29, 2023
907efb4
fix bug
YanxuanLiu Nov 29, 2023
f527aa2
fix time
YanxuanLiu Nov 29, 2023
9ee1d3c
chang key name
YanxuanLiu Nov 29, 2023
2b33b20
fix bug
YanxuanLiu Nov 29, 2023
b3075e5
change time def
YanxuanLiu Nov 29, 2023
ae68c17
change time
YanxuanLiu Nov 29, 2023
5090b10
add step to get current date
YanxuanLiu Nov 29, 2023
0602ae9
recover excess deletion.
YanxuanLiu Nov 29, 2023
2dcbc70
change ref to target branch
YanxuanLiu Nov 30, 2023
671fbc0
generate key in cache job
YanxuanLiu Nov 30, 2023
271af46
upadte key
YanxuanLiu Nov 30, 2023
845a40c
echo key for testing
YanxuanLiu Nov 30, 2023
98b0ad5
fix bug of time
YanxuanLiu Nov 30, 2023
67c9236
test date
YanxuanLiu Nov 30, 2023
a1bc646
remove java version matrix
YanxuanLiu Nov 30, 2023
bc16dfc
add runner os
YanxuanLiu Nov 30, 2023
7726775
Update .github/workflows/mvn-verify-check.yml
YanxuanLiu Nov 30, 2023
035ce8b
Update .github/workflows/mvn-verify-check.yml
YanxuanLiu Nov 30, 2023
973e947
add scala 213, and adjusted sequence of steps
YanxuanLiu Nov 30, 2023
e098445
change iteration var name
YanxuanLiu Nov 30, 2023
6fbac2b
Merge branch 'NVIDIA:branch-24.02' into cache-deps
YanxuanLiu Nov 30, 2023
1ac4110
Update .github/workflows/mvn-verify-check.yml
YanxuanLiu Nov 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 71 additions & 4 deletions .github/workflows/mvn-verify-check.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,50 @@ env:
-Dmaven.scaladoc.skip

jobs:
cache-dependencies:
runs-on: ubuntu-latest
strategy:
matrix:
java: ['8']
YanxuanLiu marked this conversation as resolved.
Show resolved Hide resolved
outputs:
dailyCacheKey: ${{ steps.generateCacheKey.outputs.dailyCacheKey }}
steps:
- uses: actions/checkout@v3 # refs/pull/:prNumber/merge
gerashegalov marked this conversation as resolved.
Show resolved Hide resolved
- uses: actions/setup-java@v3
with:
distribution: 'temurin'
java-version: ${{ matrix.java }}
YanxuanLiu marked this conversation as resolved.
Show resolved Hide resolved
- name: populate-daily-cache
run: |
YanxuanLiu marked this conversation as resolved.
Show resolved Hide resolved
set -x
max_retry=3; delay=30; i=1
while true; do
mvn help:evaluate -pl dist -Dexpression=included_buildvers \
-DforceStdout -PnoSnapshots -q | tr -d ',' | \
xargs -n 1 bash -c 'mvn initialize -pl sql-plugin-api -am -Dbuildver=$1' _ && break || {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We still do not populate the cache with Scala 2.13 artifacts: the second line in #9674 (comment)

If one statement works better for your logic we could use a loop

for pom_xml in pom.xml scala2.13/pom.xml; do 
  mvn -f $pom_xml help:evaluate -pl dist \
    -Dexpression=included_buildvers -DforceStdout -PnoSnapshots -q | tr -d ',' | \
       xargs -n 1 bash -c 'mvn -f $1 initialize -pl sql-plugin-api -am -Dbuildver=$2' _  $pom_xml; 
done

if [[ $i -le $max_retry ]]; then
echo "mvn command failed. Retry $i/$max_retry."; ((i++)); sleep $delay; ((delay=delay*2))
else
echo "mvn command failed. Exit 1"; exit 1
fi
}
done
- name: Generate daily cache key
YanxuanLiu marked this conversation as resolved.
Show resolved Hide resolved
id: generateCacheKey
run: |
set -x
echo "DATE=$(date +'%Y-%m-%d')" >> $GITHUB_ENV
echo "${{ env.DATE }}"
cacheKey="${{ hashFiles('**/pom.xml') }}-${{ github.event.pull_request.base.ref }}-${{ env.DATE }}"
echo "dailyCacheKey=$cacheKey" | tee $GITHUB_ENV $GITHUB_OUTPUT
- name: Cache local Maven repository
uses: actions/cache@v3
YanxuanLiu marked this conversation as resolved.
Show resolved Hide resolved
with:
path: ~/.m2
key: ${{ env.dailyCacheKey }}

get-shim-versions-from-dist:
needs: cache-dependencies
runs-on: ubuntu-latest
outputs:
defaultSparkVersion: ${{ steps.allShimVersionsStep.outputs.defaultSparkVersion }}
Expand Down Expand Up @@ -97,7 +140,7 @@ jobs:
echo "scala213Versions=$svJsonStr" >> $GITHUB_OUTPUT

package-tests:
needs: get-shim-versions-from-dist
needs: [get-shim-versions-from-dist, cache-dependencies]
continue-on-error: ${{ matrix.isSnapshot }}
strategy:
matrix: ${{ fromJSON(needs.get-shim-versions-from-dist.outputs.sparkTailVersions) }}
Expand All @@ -112,6 +155,12 @@ jobs:
distribution: adopt
java-version: 8

- name: Cache local Maven repository
uses: actions/cache@v3
with:
path: ~/.m2
key: ${{ needs.cache-dependencies.outputs.dailyCacheKey }}

- name: check runtime before tests
run: |
env | grep JAVA
Expand Down Expand Up @@ -142,7 +191,7 @@ jobs:
done

package-tests-scala213:
needs: get-shim-versions-from-dist
needs: [get-shim-versions-from-dist, cache-dependencies]
continue-on-error: ${{ matrix.isSnapshot }}
strategy:
matrix: ${{ fromJSON(needs.get-shim-versions-from-dist.outputs.scala213Versions) }}
Expand All @@ -157,6 +206,12 @@ jobs:
distribution: adopt
java-version: 8

- name: Cache local Maven repository
uses: actions/cache@v3
with:
path: ~/.m2
key: ${{ needs.cache-dependencies.outputs.dailyCacheKey }}

- name: check runtime before tests
run: |
env | grep JAVA
Expand Down Expand Up @@ -198,7 +253,7 @@ jobs:


verify-all-modules:
needs: get-shim-versions-from-dist
needs: [get-shim-versions-from-dist, cache-dependencies]
runs-on: ubuntu-latest
strategy:
matrix: ${{ fromJSON(needs.get-shim-versions-from-dist.outputs.sparkJDKVersions) }}
Expand All @@ -211,6 +266,12 @@ jobs:
distribution: adopt
java-version: ${{ matrix.java-version }}

- name: Cache local Maven repository
uses: actions/cache@v3
with:
path: ~/.m2
key: ${{ needs.cache-dependencies.outputs.dailyCacheKey }}

- name: check runtime before tests
run: |
env | grep JAVA
Expand Down Expand Up @@ -240,7 +301,7 @@ jobs:
done

install-modules:
needs: get-shim-versions-from-dist
needs: [get-shim-versions-from-dist, cache-dependencies]
runs-on: ubuntu-latest
strategy:
matrix:
Expand All @@ -254,6 +315,12 @@ jobs:
distribution: adopt
java-version: 11

- name: Cache local Maven repository
uses: actions/cache@v3
with:
path: ~/.m2
key: ${{ needs.cache-dependencies.outputs.dailyCacheKey }}

- name: Setup Maven Wrapper
run: mvn wrapper:wrapper -Dmaven=${{ matrix.maven-version }}

Expand Down