Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable JDK binaries caching in CI #159

Merged
merged 6 commits into from
Dec 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
189 changes: 133 additions & 56 deletions .github/workflows/test_workflow.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,11 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 180
steps:
- name: Extract Java Version
id: extract_version
run: |
java_version="${{ matrix.java_version }}"
echo "JAVA_VERSION=${java_version%%+*}" >> $GITHUB_ENV
- uses: actions/checkout@v3
- name: Prepare build JDK
uses: actions/setup-java@v3
Expand All @@ -31,11 +36,20 @@ jobs:
run: |
sudo apt-get update
sudo apt-get install -y curl zip unzip libgtest-dev libgmock-dev
- name: Prepare JDK ${{ matrix.java_version }}
- name: Cache JDK ${{ env.JAVA_VERSION }} [amd64]
uses: actions/cache@v3
with:
path: |
test_${{ env.JAVA_VERSION }}_jdk
key: ${{ env.JAVA_VERSION }}-amd64
- name: Prepare JDK ${{ env.JAVA_VERSION }}
run: |
wget -nv https://download.bell-sw.com/java/${{ matrix.java_version }}/bellsoft-jdk${{ matrix.java_version }}-linux-amd64.tar.gz -O jdk.tar.gz
tar xzf *.tar.gz
find . -type d -name 'jdk*' -maxdepth 1| xargs -I {} mv {} test_jdk
TEST_JDK_DIR="test_${{ env.JAVA_VERSION }}_jdk"
if [ ! -e ${TEST_JDK_DIR}/bin/java ]; then
wget -nv https://download.bell-sw.com/java/${{ matrix.java_version }}/bellsoft-jdk${{ matrix.java_version }}-linux-amd64.tar.gz -O jdk.tar.gz
tar xzf *.tar.gz
find . -type d -name 'jdk*' -maxdepth 1| xargs -I {} mv {} ${TEST_JDK_DIR}
fi
- name: Test
run: |
sudo sysctl vm.mmap_rnd_bits=28
Expand All @@ -44,7 +58,7 @@ jobs:
export TEST_COMMIT=${{ github.sha }}
export TEST_CONFIGURATION=glibc/${{ matrix.java_version }}-amd64
export LIBC=glibc
export JAVA_TEST_HOME=$(pwd)/test_jdk
export JAVA_TEST_HOME=$(pwd)/test_${{ env.JAVA_VERSION }}_jdk
export SANITIZER=${{ matrix.config }}
export JAVA_VERSION=$(${JAVA_TEST_HOME}/bin/java -version 2>&1 | awk -F '"' '/version/ {
split($2, v, "[._]");
Expand Down Expand Up @@ -94,6 +108,11 @@ jobs:
labels: arm-4core-linux
timeout-minutes: 180
steps:
- name: Extract Java Version
id: extract_version
run: |
java_version="${{ matrix.java_version }}"
echo "JAVA_VERSION=${java_version%%+*}" >> $GITHUB_ENV
- uses: actions/checkout@v3
- name: Prepare build JDK
uses: actions/setup-java@v3
Expand All @@ -106,11 +125,20 @@ jobs:
sudo apt remove -y g++
sudo apt autoremove -y
sudo apt install -y curl zip unzip clang make build-essential
- name: Cache JDK ${{ env.JAVA_VERSION }} [aarch64]
uses: actions/cache@v3
with:
path: |
~/test_${{ env.JAVA_VERSION }}_jdk
key: ${{ env.JAVA_VERSION }}-aarch64
- name: Prepare JDK ${{ matrix.java_version }}
run: |
wget -nv https://download.bell-sw.com/java/${{ matrix.java_version }}/bellsoft-jdk${{ matrix.java_version }}-linux-aarch64.tar.gz -O jdk.tar.gz
tar xzf *.tar.gz
find . -type d -name 'jdk*' -maxdepth 1| xargs -I {} mv {} test_jdk
TEST_JDK_DIR="test_${{ env.JAVA_VERSION }}_jdk"
if [ ! -e ${TEST_JDK_DIR}/bin/java ]; then
wget -nv https://download.bell-sw.com/java/${{ matrix.java_version }}/bellsoft-jdk${{ matrix.java_version }}-linux-aarch64.tar.gz -O jdk.tar.gz
tar xzf *.tar.gz
find . -type d -name 'jdk*' -maxdepth 1| xargs -I {} mv {} ${TEST_JDK_DIR}
fi
- name: Test
run: |
sudo sysctl vm.mmap_rnd_bits=28
Expand All @@ -119,7 +147,7 @@ jobs:
export TEST_COMMIT=${{ github.sha }}
export TEST_CONFIGURATION=glibc/${{ matrix.java_version }}-aarch64
export LIBC=glibc
export JAVA_TEST_HOME=$(pwd)/test_jdk
export JAVA_TEST_HOME=$(pwd)/test_${{ env.JAVA_VERSION }}_jdk
export SANITIZER=${{ matrix.config }}
./gradlew :ddprof-test:test${{ matrix.config }}
if [ $? -ne 0 ]; then
Expand Down Expand Up @@ -313,14 +341,22 @@ jobs:
with:
distribution: 'temurin'
java-version: "11"
- name: Cache JDK Oracle JDK 8
uses: actions/cache@v3
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 Code Vulnerability

Workflow depends on a GitHub actions pinned by tag (...read more)

When using a third party action, one needs to provide its GitHub path (owner/project) and can eventually pin it to a Git ref (a branch name, a Git tag, or a commit hash).

No pinned Git ref means the action uses the latest commit of the default branch each time it runs, eventually running newer versions of the code that were not audited by Datadog. Specifying a Git tag is better, but since they are not immutable, using a full length hash is recommended to make sure the action content is actually frozen to some reviewed state.

Be careful however, as even pinning an action by hash can be circumvented by attackers still. For instance, if an action relies on a Docker image which is itself not pinned to a digest, it becomes possible to alter its behaviour through the Docker image without actually changing its hash. You can learn more about this kind of attacks in Unpinnable Actions: How Malicious Code Can Sneak into Your GitHub Actions Workflows. Pinning actions by hash is still a good first line of defense against supply chain attacks.

Additionally, pinning by hash or tag means the action won’t benefit from newer version updates if any, including eventual security patches. Make sure to regularly check if newer versions for an action you use are available. For actions coming from a very trustworthy source, it can make sense to use a laxer pinning policy to benefit from updates as soon as possible.

View in Datadog  Leave us feedback  Documentation

with:
path: |
/usr/lib/jvm/oracle8
key: oracle-jdk8
- name: Prepare JDK ${{ matrix.java_version }}
run: |
sudo sysctl vm.mmap_rnd_bits=28
set -eux;
sudo mkdir -p /usr/lib/jvm/oracle8;
# https://gist.github.com/wavezhang/ba8425f24a968ec9b2a8619d7c2d86a6?permalink_comment_id=4444663#gistcomment-4444663
# jdk1.8.0_361
curl -L --fail "https://javadl.oracle.com/webapps/download/AutoDL?BundleId=247926_0ae14417abb444ebb02b9815e2103550" | sudo tar -xvzf - -C /usr/lib/jvm/oracle8 --strip-components 1
if [ ! -e /usr/lib/jvm/oracle8/bin/java ]; then
sudo mkdir -p /usr/lib/jvm/oracle8;
# https://gist.github.com/wavezhang/ba8425f24a968ec9b2a8619d7c2d86a6?permalink_comment_id=4444663#gistcomment-4444663
# jdk1.8.0_361
curl -L --fail "https://javadl.oracle.com/webapps/download/AutoDL?BundleId=247926_0ae14417abb444ebb02b9815e2103550" | sudo tar -xvzf - -C /usr/lib/jvm/oracle8 --strip-components 1
fi
uname -r
- name: Test
run: |
Expand Down Expand Up @@ -381,28 +417,51 @@ jobs:
image: "alpine:3.14"
options: --cpus 2
steps:
- name: Extract Java Version
id: extract_version
run: |
java_version="${{ matrix.java_version }}"
echo "JAVA_VERSION=${java_version%%+*}" >> $GITHUB_ENV
- uses: actions/checkout@v3
- name: Setup OS
run: apk update && apk add curl moreutils wget hexdump linux-headers bash make g++ clang git cppcheck jq cmake gtest-dev gmock >/dev/null
- name: Prepare build JDK
run: apk update && apk add curl moreutils wget hexdump linux-headers bash make g++ clang git cppcheck jq cmake gtest-dev gmock tar >/dev/null
- name: Cache JDK 11.0.25 [amd64-musl]
uses: actions/cache@v3
with:
path: |
test_11.0.25_jdk
key: 11.0.25-amd64-musl
- name: Cache JDK ${{ env.JAVA_VERSION }} [amd64-musl]
uses: actions/cache@v3
with:
path: |
test_${{ env.JAVA_VERSION }}_jdk
key: ${{ env.JAVA_VERSION }}-amd64-musl
- name: Prepare JDK 11.0.25
run: |
wget -nv https://download.bell-sw.com/java/11.0.25+11/bellsoft-jdk11.0.25+11-linux-x64-musl.tar.gz -O jdk.tar.gz
tar xzf *.tar.gz
find . -type d -name 'jdk*' -maxdepth 1 | xargs -I {} mv {} build_jdk
- name: Prepare JDK ${{ matrix.java_version }}
TEST_JDK_DIR="test_11.0.25_jdk"
if [ ! -e ${TEST_JDK_DIR}/bin/java ]; then
wget -nv https://download.bell-sw.com/java/11.0.25+11/bellsoft-jdk11.0.25+11-linux-x64-musl.tar.gz -O jdk.tar.gz
tar xzf *.tar.gz
find . -type d -name 'jdk*' -maxdepth 1| xargs -I {} mv {} ${TEST_JDK_DIR}
fi
- name: Prepare JDK ${{ env.JAVA_VERSION }}
run: |
wget -nv https://download.bell-sw.com/java/${{ matrix.java_version }}/bellsoft-jdk${{ matrix.java_version }}-linux-x64-musl.tar.gz -O jdk.tar.gz
tar xzf *.tar.gz
find . -type d -name 'jdk*' -maxdepth 1 | xargs -I {} mv {} test_jdk
TEST_JDK_DIR="test_${{ env.JAVA_VERSION }}_jdk"
if [ ! -e ${TEST_JDK_DIR}/bin/java ]; then
wget -nv https://download.bell-sw.com/java/${{ matrix.java_version }}/bellsoft-jdk${{ matrix.java_version }}-linux-x64-musl.tar.gz -O jdk.tar.gz
tar xzf *.tar.gz
find . -type d -name 'jdk*' -maxdepth 1| xargs -I {} mv {} ${TEST_JDK_DIR}
fi
- name: Test
run: |
set +e
export JAVA_HOME=$(pwd)/build_jdk
export JAVA_HOME=$(pwd)/test_11.0.25_jdk
export PATH=$JAVA_HOME/bin:$PATH
export TEST_COMMIT=${{ github.sha }}
export TEST_CONFIGURATION=musl/${{ matrix.java_version }}
export LIBC=musl
export JAVA_TEST_HOME=$(pwd)/test_jdk
export JAVA_TEST_HOME=$(pwd)/test_${{ env.JAVA_VERSION }}_jdk
export SANITIZER=${{ matrix.config }}
export JAVA_VERSION=$(${JAVA_TEST_HOME}/bin/java -version 2>&1 | awk -F '"' '/version/ {
split($2, v, "[._]");
Expand Down Expand Up @@ -466,37 +525,46 @@ jobs:
with:
distribution: 'temurin'
java-version: "11"
- name: Cache JDK ${{ matrix.java_version }} [amd64-zing]
uses: actions/cache@v3
with:
path: |
test_${{ matrix.java_version }}_jdk
key: ${{ matrix.java_version }}-amd64-zing
- name: Prepare JDK ${{ matrix.java_version }}
if: steps.set_config.outputs.config == 'release' || steps.set_config.outputs.config == 'debug'
run: |
sudo apt-get -y update && sudo apt-get -y install curl g++-9 gcc-9
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-9 100 --slave /usr/bin/g++ g++ /usr/bin/g++-9
sudo update-alternatives --set gcc /usr/bin/gcc-9
set -eux
sudo mkdir -p /usr/lib/jvm/zing
if [ "${{ matrix.java_version }}" = "8" ]; then
# jdk1.8.0_372
curl -L --fail "https://cdn.azul.com/zing-zvm/ZVM23.05.0.0/zing23.05.0.0-2-jdk8.0.372-linux_x64.tar.gz" | sudo tar -xvzf - -C /usr/lib/jvm/zing --strip-components 1
elif [ "${{ matrix.java_version }}" = "11" ]; then
# jdk 11.0.19
curl -L --fail "https://cdn.azul.com/zing-zvm/ZVM23.05.0.0/zing23.05.0.0-2-jdk11.0.19-linux_x64.tar.gz" | sudo tar -xvzf - -C /usr/lib/jvm/zing --strip-components 1
elif [ "${{ matrix.java_version }}" = "17" ]; then
# jdk 17.0.7
curl -L --fail "https://cdn.azul.com/zing-zvm/ZVM23.05.0.0/zing23.05.0.0-2-jdk17.0.7-linux_x64.tar.gz" | sudo tar -xvzf - -C /usr/lib/jvm/zing --strip-components 1
elif [ "${{ matrix.java_version }}" = "21" ]; then
# jdk 21.0.2
curl -L --fail "https://cdn.azul.com/zing-zvm/ZVM23.10.0.0/zing23.10.0.0-3-jdk21.0.1-linux_x64.tar.gz" | sudo tar -xvzf - -C /usr/lib/jvm/zing --strip-components 1
TEST_JDK_DIR=test_${{ matrix.java_version }}_jdk
if [ ! -e ${TEST_JDK_DIR}/bin/java ]; then
set -eux
mkdir -p ${TEST_JDK_DIR}
if [ "${{ matrix.java_version }}" = "8" ]; then
# jdk1.8.0_372
curl -L --fail "https://cdn.azul.com/zing-zvm/ZVM23.05.0.0/zing23.05.0.0-2-jdk8.0.372-linux_x64.tar.gz" | sudo tar -xvzf - -C ${TEST_JDK_DIR} --strip-components 1
elif [ "${{ matrix.java_version }}" = "11" ]; then
# jdk 11.0.19
curl -L --fail "https://cdn.azul.com/zing-zvm/ZVM23.05.0.0/zing23.05.0.0-2-jdk11.0.19-linux_x64.tar.gz" | sudo tar -xvzf - -C ${TEST_JDK_DIR} --strip-components 1
elif [ "${{ matrix.java_version }}" = "17" ]; then
# jdk 17.0.7
curl -L --fail "https://cdn.azul.com/zing-zvm/ZVM23.05.0.0/zing23.05.0.0-2-jdk17.0.7-linux_x64.tar.gz" | sudo tar -xvzf - -C ${TEST_JDK_DIR} --strip-components 1
elif [ "${{ matrix.java_version }}" = "21" ]; then
# jdk 21.0.2
curl -L --fail "https://cdn.azul.com/zing-zvm/ZVM23.10.0.0/zing23.10.0.0-3-jdk21.0.1-linux_x64.tar.gz" | sudo tar -xvzf - -C ${TEST_JDK_DIR} --strip-components 1
fi
# rename the bundled libstdc++.so to avoid conflicts with the system one
sudo mv ${TEST_JDK_DIR}/etc/libc++/libstdc++.so.6 ${TEST_JDK_DIR}/etc/libc++/libstdc++.so.6.bak
fi
# rename the bundled libstdc++.so to avoid conflicts with the system one
sudo mv /usr/lib/jvm/zing/etc/libc++/libstdc++.so.6 /usr/lib/jvm/zing/etc/libc++/libstdc++.so.6.bak
- name: Test
if: steps.set_config.outputs.config == 'release' || steps.set_config.outputs.config == 'debug'
run: |
set +e
export TEST_COMMIT=${{ github.sha }}
export TEST_CONFIGURATION=glibc/${{ matrix.java_version }}
export LIBC=glibc
export JAVA_TEST_HOME=/usr/lib/jvm/zing
export JAVA_TEST_HOME=$(pwd)/test_${{ matrix.java_version }}_jdk
export JAVA_HOME=$JAVA_HOME
export PATH=$JAVA_HOME/bin:$PATH
export SANITIZER=${{ matrix.config }}
Expand Down Expand Up @@ -565,26 +633,35 @@ jobs:
with:
distribution: 'temurin'
java-version: "11"
- name: Cache JDK ${{ matrix.java_version }} [aarch64-zing]
uses: actions/cache@v3
with:
path: |
test_${{ matrix.java_version }}_jdk
key: ${{ matrix.java_version }}-aarch64-zing
- name: Prepare JDK ${{ matrix.java_version }}
if: steps.set_config.outputs.config == 'release' || steps.set_config.outputs.config == 'debug'
run: |
sudo apt-get -y update && sudo apt-get -y install curl g++-9 gcc-9
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-9 100 --slave /usr/bin/g++ g++ /usr/bin/g++-9
sudo update-alternatives --set gcc /usr/bin/gcc-9
set -eux
sudo mkdir -p /usr/lib/jvm/zing
if [ "${{ matrix.java_version }}" = "8" ]; then
# jdk1.8.0_432
curl -L --fail "https://cdn.azul.com/zing-zvm/ZVM24.10.0.0/zing24.10.0.0-4-jdk8.0.431-linux_aarch64.tar.gz" | sudo tar -xvzf - -C /usr/lib/jvm/zing --strip-components 1
elif [ "${{ matrix.java_version }}" = "11" ]; then
# jdk 11.0.24
curl -L --fail "https://cdn.azul.com/zing-zvm/ZVM24.10.0.0/zing24.10.0.0-4-jdk11.0.24.0.101-linux_aarch64.tar.gz" | sudo tar -xvzf - -C /usr/lib/jvm/zing --strip-components 1
elif [ "${{ matrix.java_version }}" = "17" ]; then
# jdk 17.0.12
curl -L --fail "https://cdn.azul.com/zing-zvm/ZVM24.10.0.0/zing24.10.0.0-4-jdk17.0.12.0.101-linux_aarch64.tar.gz" | sudo tar -xvzf - -C /usr/lib/jvm/zing --strip-components 1
elif [ "${{ matrix.java_version }}" = "21" ]; then
# jdk 21.0.4
curl -L --fail "https://cdn.azul.com/zing-zvm/ZVM24.10.0.0/zing24.10.0.0-4-jdk21.0.4.0.101-linux_aarch64.tar.gz" | sudo tar -xvzf - -C /usr/lib/jvm/zing --strip-components 1
TEST_JDK_DIR=test_${{ matrix.java_version }}_jdk
if [ ! -e ${TEST_JDK_DIR}/bin/java ]; then
set -eux
mkdir -p ${TEST_JDK_DIR}
if [ "${{ matrix.java_version }}" = "8" ]; then
# jdk1.8.0_431
curl -L --fail "https://cdn.azul.com/zing-zvm/ZVM24.10.0.0/zing24.10.0.0-4-jdk8.0.431-linux_aarch64.tar.gz" | sudo tar -xvzf - -C ${TEST_JDK_DIR} --strip-components 1
elif [ "${{ matrix.java_version }}" = "11" ]; then
# jdk 11.0.24
curl -L --fail "https://cdn.azul.com/zing-zvm/ZVM24.10.0.0/zing24.10.0.0-4-jdk11.0.24.0.101-linux_aarch64.tar.gz" | sudo tar -xvzf - -C ${TEST_JDK_DIR} --strip-components 1
elif [ "${{ matrix.java_version }}" = "17" ]; then
# jdk 17.0.12
curl -L --fail "https://cdn.azul.com/zing-zvm/ZVM24.10.0.0/zing24.10.0.0-4-jdk17.0.12.0.101-linux_aarch64.tar.gz" | sudo tar -xvzf - -C ${TEST_JDK_DIR} --strip-components 1
elif [ "${{ matrix.java_version }}" = "21" ]; then
# jdk 21.0.4
curl -L --fail "https://cdn.azul.com/zing-zvm/ZVM24.10.0.0/zing24.10.0.0-4-jdk21.0.4.0.101-linux_aarch64.tar.gz" | sudo tar -xvzf - -C ${TEST_JDK_DIR} --strip-components 1
fi
fi
- name: Test
if: steps.set_config.outputs.config == 'release' || steps.set_config.outputs.config == 'debug'
Expand All @@ -593,7 +670,7 @@ jobs:
export TEST_COMMIT=${{ github.sha }}
export TEST_CONFIGURATION=glibc/${{ matrix.java_version }}
export LIBC=glibc
export JAVA_TEST_HOME=/usr/lib/jvm/zing
export JAVA_TEST_HOME=$(pwd)/test_${{ matrix.java_version }}_jdk
export JAVA_HOME=$JAVA_HOME
export PATH=$JAVA_HOME/bin:$PATH
export SANITIZER=${{ matrix.config }}
Expand Down
4 changes: 4 additions & 0 deletions ddprof-lib/src/main/java/com/datadoghq/profiler/Platform.java
Original file line number Diff line number Diff line change
Expand Up @@ -323,4 +323,8 @@ public static String getRuntimeVersion() {
public static String getRuntimePatches() {
return RUNTIME.patches;
}

public static boolean isAarch64() {
return System.getProperty("os.arch").toLowerCase().contains("aarch64");
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,10 @@ void jvmVersionTest() throws Exception {

String javaVersion = System.getenv("JAVA_VERSION");
assumeTrue(javaVersion != null);
if (javaVersion.startsWith("8u")) {
// convert 8u432 to nomralized 8.0.432 format which is expected
javaVersion = "8.0." + javaVersion.split("u")[1];
}

String javaHome = System.getenv("JAVA_TEST_HOME");
if (javaHome == null) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.junit.jupiter.api.Assumptions.assumeFalse;

public class BoundMethodHandeMetadataSizeTest extends AbstractProfilerTest {
public class BoundMethodHandleMetadataSizeTest extends AbstractProfilerTest {
@Override
protected String getProfilerCommand() {
return "wall=100us";
Expand All @@ -22,6 +22,7 @@ protected String getProfilerCommand() {
@Test
public void test() throws Throwable {
assumeFalse(Platform.isJ9() && Platform.isJavaVersion(17)); // JVMTI::GetClassSignature() is reliably crashing on a valid 'class' instance ¯\_(ツ)_/¯
assumeFalse(Platform.isAarch64() && Platform.isJavaVersion(8));
registerCurrentThreadForWallClockProfiling();
int numBoundMethodHandles = 10_000;
int x = generateBoundMethodHandles(numBoundMethodHandles);
Expand All @@ -42,7 +43,7 @@ public static String append(String string, int suffix) {
public static int generateBoundMethodHandles(int howMany) throws Throwable {
int total = 0;
MethodHandle append = MethodHandles.lookup()
.findStatic(BoundMethodHandeMetadataSizeTest.class,
.findStatic(BoundMethodHandleMetadataSizeTest.class,
"append",
MethodType.methodType(String.class, String.class, int.class));
for (int i = 0; i < howMany; i++) {
Expand Down
Loading