Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/vcpkg linux #2078

Merged
merged 58 commits into from
May 30, 2024
Merged
Show file tree
Hide file tree
Changes from 50 commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
a5ace26
VCPKG manifest mode with minizip and boost only, ubuntu
sylvlecl May 2, 2024
a532ded
Fix VCPKG install
sylvlecl May 2, 2024
4d88b9f
To be reverted: disable temporarily non-ununtu CIs
sylvlecl May 2, 2024
01c1719
Fix toolchain file path
sylvlecl May 2, 2024
ce6f0ee
Fix wxWidgets search
sylvlecl May 2, 2024
aaf56be
Revert boost from vcpkg for now
sylvlecl May 2, 2024
8215e8c
Save cache even when build fails
sylvlecl May 2, 2024
ff51823
Fix cache key
sylvlecl May 2, 2024
33541eb
Restore windows build
sylvlecl May 2, 2024
54d2717
Fix vcpkg install
sylvlecl May 2, 2024
4fa2567
Try fixing toolchain path
sylvlecl May 2, 2024
ddf60fd
Try fiwing workspace path
sylvlecl May 2, 2024
82f866b
Another try
sylvlecl May 2, 2024
fb49715
Try improving cache logic
sylvlecl May 2, 2024
c64b687
Add boost to windows build
sylvlecl May 2, 2024
a71f0ca
Try with centos
sylvlecl May 2, 2024
1b19dfb
No VCPKG cache for centos, for now
sylvlecl May 2, 2024
eeca546
Try to fix path to vcpkg
sylvlecl May 2, 2024
bcb14cc
Log for debugging
sylvlecl May 2, 2024
64b59bb
Try install cmake with pip instead
sylvlecl May 2, 2024
8d996a1
More logs
sylvlecl May 2, 2024
5e26bbd
Temporarily disable ubuntu & windows
sylvlecl May 2, 2024
c6fa5b9
Try fix path
sylvlecl May 2, 2024
3339c9e
Restore gcc install
sylvlecl May 2, 2024
e675c96
Again another try
sylvlecl May 2, 2024
9e426f8
Fix git version
sylvlecl May 2, 2024
72f62ef
Fix packaging too
sylvlecl May 2, 2024
b3de6f3
Try oracle 8
sylvlecl May 2, 2024
cd23b2c
Fix security error
sylvlecl May 2, 2024
83331e0
Revert "Temporarily disable ubuntu & windows"
sylvlecl May 3, 2024
6b71d90
Hashing after submodule update
sylvlecl May 3, 2024
62a6e8f
Fix restore keys
sylvlecl May 3, 2024
0a52474
Disable ubuntu & windows
sylvlecl May 3, 2024
9f3b2c1
Try cache action downgraded
sylvlecl May 3, 2024
e4953d5
Try cache action v3 on centos7
sylvlecl May 3, 2024
512a091
Add actual caching
sylvlecl May 3, 2024
046436d
Try env vars
sylvlecl May 3, 2024
ff0d261
Skip some tests for testing
sylvlecl May 3, 2024
3e78a3c
fix
sylvlecl May 3, 2024
3e1ea50
fix
sylvlecl May 3, 2024
1671e1e
Revert "Skip some tests for testing"
sylvlecl May 3, 2024
af6210d
Fix vcpkg cache paths on centos
sylvlecl May 3, 2024
66bf8fa
Revert "Disable ubuntu & windows"
sylvlecl May 3, 2024
53dfa9b
Disable centos and oracle on PRs
sylvlecl May 3, 2024
8e49f12
Merge remote-tracking branch 'origin/develop' into feature/vcpkg-linux
sylvlecl May 3, 2024
be5b618
Restore sonarcloud
sylvlecl May 3, 2024
a914c32
Fix windows workflow file
sylvlecl May 3, 2024
40a9f3b
VCPKG for sonar job
sylvlecl May 3, 2024
fd13a66
Merge branch 'feature/vcpkg-dependencies' into feature/vcpkg-linux
sylvlecl May 13, 2024
d4924c9
Support minizip dependency as minizip or minizip-ng
sylvlecl May 11, 2024
1b16bda
Empty commit to test GH caching
flomnes May 14, 2024
e2694b5
Add custom triplets to:
sylvlecl May 16, 2024
e526cba
Fix build type in triplets
sylvlecl May 16, 2024
5bc828e
Fix windows triplets
sylvlecl May 16, 2024
f2546c4
Boost available on linux too
JasonMarechal25 May 29, 2024
46788eb
Merge branch 'develop' into feature/vcpkg-linux
JasonMarechal25 May 29, 2024
38c49f6
Feature/vcpkg linux fixes (#2127)
JasonMarechal25 May 30, 2024
ebae478
Merge branch 'develop' into feature/vcpkg-linux
JasonMarechal25 May 30, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 42 additions & 14 deletions .github/workflows/centos7.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,28 @@ jobs:
yum install -y centos-release-scl
yum install -y devtoolset-10-gcc*

- uses: ./.github/workflows/install-cmake-328
- name: Install cmake 3.28
run: pip3 install cmake==3.28.4

- name: Install VCPKG
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

# Note: we need to use environment variables instead of workflow variables
# because github messes up path variables when running in container,
# see https://github.com/actions/runner/issues/2058
run: |
git submodule update --init vcpkg && ./vcpkg/bootstrap-vcpkg.sh -disableMetrics
echo "VCPKG_ROOT=$GITHUB_WORKSPACE/vcpkg" >> $GITHUB_ENV
echo "VCPKG_CACHE_DIR=$GITHUB_WORKSPACE/vcpkg_cache" >> $GITHUB_ENV
echo "VCPKG_BINARY_SOURCES=clear;files,$GITHUB_WORKSPACE/vcpkg_cache,readwrite" >> $GITHUB_ENV

- name: Restore vcpkg binary dir from cache
id: cache-vcpkg-binary
# Note: we are stuck with v3, because v4 is not compatible with oracle8 image
uses: actions/cache/restore@v3
with:
path: ${{ env.VCPKG_CACHE_DIR }}
key: vcpkg-cache-centos7-${{ hashFiles('src/vcpkg.json', '.git/modules/vcpkg/HEAD') }}
# Allows to restore a cache when deps have only partially changed (like adding a dependency)
restore-keys: vcpkg-cache-centos7-
Copy link
Contributor

@a-zakir a-zakir May 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wow! i would recommend to place the "install vcpkg and the restore cache" jobs in dedicated file

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately it does not seem so easy, because we cannot have "post actions" in composite github actions :(
See actions/runner#1478

I wanted to have the 3 steps "install VCPKG", "restore cache" and "save cache" in the same action, but it's not possible.
If we keep the save step separate, it will be messy because you need to know "internal details" of the other steps (the cahce key basically).

So I'm afraid I will leave it this way for now ...


- name: Init submodule
run: |
Expand Down Expand Up @@ -77,20 +98,21 @@ jobs:
dnf -y config-manager --add-repo https://cli.github.com/packages/rpm/gh-cli.repo
dnf -y install gh


- name: Configure
run: |
source /opt/rh/devtoolset-10/enable
cmake -B _build -S src \
-DCMAKE_C_COMPILER_LAUNCHER=ccache \
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
-DDEPS_INSTALL_DIR=/rte-antares-deps-Release \
-DCMAKE_BUILD_TYPE=Release \
-DBUILD_TESTING=ON \
-DBUILD_not_system=OFF \
-DBUILD_TOOLS=ON \
-DBUILD_UI=OFF \
-DCMAKE_PREFIX_PATH=${{ env.ORTOOLSDIR }}/install \
source /opt/rh/devtoolset-10/enable
source /opt/rh/rh-git227/enable
cmake -B _build -S src \
-DCMAKE_C_COMPILER_LAUNCHER=ccache \
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
-DCMAKE_TOOLCHAIN_FILE=$GITHUB_WORKSPACE/vcpkg/scripts/buildsystems/vcpkg.cmake \
-DDEPS_INSTALL_DIR=/rte-antares-deps-Release \
-DCMAKE_BUILD_TYPE=Release \
-DBUILD_TESTING=ON \
-DBUILD_not_system=OFF \
-DBUILD_TOOLS=ON \
-DBUILD_UI=OFF \
-DCMAKE_PREFIX_PATH=${{ env.ORTOOLSDIR }}/install \

- name: Build
run: |
Expand Down Expand Up @@ -139,4 +161,10 @@ jobs:
run: |
gh release upload "$tag" _build/*.tar.gz _build/*.rpm


- name: Cache vcpkg binary dir
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even in the case of build errors ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question :
I think it makes sense:

  • if the dependencies build works and the project build fails, we need to keep the dependencies to not rebuild them in a later workflow
  • if it's the build of a dependency which fails, I think it is still better to save the cache because other dependencies may have succeeded. Also, because we restore the previous cache, this should keep previously cached items which have not been rebuilt (like the ones that would have been rebuilt after the dependency that failed)

if: always()
id: save-cache-vcpkg-binary
uses: actions/cache/save@v3
with:
path: ${{ env.VCPKG_CACHE_DIR }}
key: vcpkg-cache-centos7-${{ hashFiles('src/vcpkg.json', '.git/modules/vcpkg/HEAD') }}
51 changes: 39 additions & 12 deletions .github/workflows/oracle8.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@ env:
IS_PUSH: ${{ github.event_name == 'push' }}
REF: ${{ inputs.target_branch =='' && github.ref_name || inputs.target_branch}}


jobs:

build:
Expand All @@ -47,7 +46,28 @@ jobs:

- name: Checkout
run: |
git clone $GITHUB_SERVER_URL/$GITHUB_REPOSITORY.git -b ${{ env.REF }} .
git clone $GITHUB_SERVER_URL/$GITHUB_REPOSITORY.git -b ${{ env.REF }} .
git config --global safe.directory '*'

- name: Install VCPKG
# Note: we need to use environment variables instead of workflow variables
# because github messes up path variables when running in container,
# see https://github.com/actions/runner/issues/2058
run: |
git submodule update --init vcpkg && ./vcpkg/bootstrap-vcpkg.sh -disableMetrics
echo "VCPKG_ROOT=$GITHUB_WORKSPACE/vcpkg" >> $GITHUB_ENV
echo "VCPKG_CACHE_DIR=$GITHUB_WORKSPACE/vcpkg_cache" >> $GITHUB_ENV
echo "VCPKG_BINARY_SOURCES=clear;files,$GITHUB_WORKSPACE/vcpkg_cache,readwrite" >> $GITHUB_ENV

- name: Restore vcpkg binary dir from cache
id: cache-vcpkg-binary
# Note: we are stuck with v3, because v4 is not compatible with oracle8 image
uses: actions/cache/restore@v3
with:
path: ${{ env.VCPKG_CACHE_DIR }}
key: vcpkg-cache-oracle8-${{ hashFiles('src/vcpkg.json', '.git/modules/vcpkg/HEAD') }}
# Allows to restore a cache when deps have only partially changed (like adding a dependency)
restore-keys: vcpkg-cache-oracle8-

- name: Config OR-Tools URL
run: |
Expand All @@ -62,9 +82,7 @@ jobs:
rm ortools.zip

- name: Init submodule
run: |
git config --global safe.directory '*'
git submodule update --init --remote src/antares-deps src/tests/resources/Antares_Simulator_Tests
run: git submodule update --init --remote src/antares-deps src/tests/resources/Antares_Simulator_Tests

- name: Install dependencies
run: |
Expand All @@ -79,13 +97,14 @@ jobs:

- name: Configure
run: |
source /opt/rh/gcc-toolset-11/enable
cmake -B _build -S src \
-DCMAKE_BUILD_TYPE=Release \
-DBUILD_TESTING=ON \
-DBUILD_TOOLS=ON \
-DBUILD_UI=OFF \
-DCMAKE_PREFIX_PATH=${{ env.ORTOOLS_DIR }}/install
source /opt/rh/gcc-toolset-11/enable
cmake -B _build -S src \
-DCMAKE_TOOLCHAIN_FILE=$GITHUB_WORKSPACE/vcpkg/scripts/buildsystems/vcpkg.cmake \
-DCMAKE_BUILD_TYPE=Release \
-DBUILD_TESTING=ON \
-DBUILD_TOOLS=ON \
-DBUILD_UI=OFF \
-DCMAKE_PREFIX_PATH=${{ env.ORTOOLS_DIR }}/install

- name: Build
run: |
Expand Down Expand Up @@ -137,3 +156,11 @@ jobs:
tag: ${{ github.event.inputs.release_tag }}
run: |
gh release upload "$tag" _build/*.tar.gz _build/*.rpm

- name: Cache vcpkg binary dir
if: always()
id: save-cache-vcpkg-binary
uses: actions/cache/save@v3
with:
path: ${{ env.VCPKG_CACHE_DIR }}
key: vcpkg-cache-oracle8-${{ hashFiles('src/vcpkg.json', '.git/modules/vcpkg/HEAD') }}
23 changes: 23 additions & 0 deletions .github/workflows/sonarcloud.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,26 @@ jobs:
env:
SONAR_SERVER_URL: "https://sonarcloud.io"
ORTOOLS_DIR: ${{ github.workspace }}/or-tools
# Caching strategy of VCPKG dependencies
VCPKG_BINARY_SOURCES: "clear;files,${{ github.workspace }}/vcpkg_cache,readwrite"

steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # Shallow clones should be disabled for a better relevancy of analysis

- name: Install VCPKG
run: git submodule update --init vcpkg && ./vcpkg/bootstrap-vcpkg.sh -disableMetrics

- name: Restore vcpkg binary dir from cache
id: cache-vcpkg-binary
uses: actions/cache/restore@v4
with:
path: ${{ github.workspace }}/vcpkg_cache
key: vcpkg-cache-ubuntu-${{ hashFiles('src/vcpkg.json', '.git/modules/vcpkg/HEAD') }}
# Allows to restore a cache when deps have only partially changed (like adding a dependency)
restore-keys: vcpkg-cache-ubuntu-

- name: Config OR-Tools URL
run: |
echo "ORTOOLS_URL=https://github.com/rte-france/or-tools/releases/download/$(cat ortools_tag)/ortools_cxx_ubuntu-20.04_static_sirius.zip" >> $GITHUB_ENV
Expand Down Expand Up @@ -81,6 +95,7 @@ jobs:
-DCMAKE_C_COMPILER=/usr/bin/gcc-10 \
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
-DCMAKE_CXX_COMPILER=/usr/bin/g++-10 \
-DCMAKE_TOOLCHAIN_FILE=${{ github.workspace }}/vcpkg/scripts/buildsystems/vcpkg.cmake \
-DDEPS_INSTALL_DIR=./rte-antares-deps-Debug \
-DCODE_COVERAGE=ON \
-DCMAKE_BUILD_TYPE=Debug \
Expand Down Expand Up @@ -109,3 +124,11 @@ jobs:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
run: sonar-scanner --define sonar.host.url="${{ env.SONAR_SERVER_URL }}"

- name: Cache vcpkg binary dir
if: always()
id: save-cache-vcpkg-binary
uses: actions/cache/save@v4
with:
path: ${{ github.workspace }}/vcpkg_cache
key: vcpkg-cache-ubuntu-${{ hashFiles('src/vcpkg.json', '.git/modules/vcpkg/HEAD') }}
56 changes: 39 additions & 17 deletions .github/workflows/ubuntu.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ jobs:
env:
ORTOOLS_DIR: ${{ github.workspace }}/or-tools
os: ubuntu-20.04
# Caching strategy of VCPKG dependencies
VCPKG_BINARY_SOURCES: "clear;files,${{ github.workspace }}/vcpkg_cache,readwrite"

runs-on: ubuntu-20.04
if: "!contains(github.event.head_commit.message, '[skip ci]')"
Expand All @@ -45,17 +47,29 @@ jobs:
with:
ref: ${{ env.REF }}

- name: Install VCPKG
run: git submodule update --init vcpkg && ./vcpkg/bootstrap-vcpkg.sh -disableMetrics

- name: Restore vcpkg binary dir from cache
id: cache-vcpkg-binary
uses: actions/cache/restore@v4
with:
path: ${{ github.workspace }}/vcpkg_cache
key: vcpkg-cache-ubuntu-${{ hashFiles('src/vcpkg.json', '.git/modules/vcpkg/HEAD') }}
# Allows to restore a cache when deps have only partially changed (like adding a dependency)
restore-keys: vcpkg-cache-ubuntu-

- name: ccache
uses: hendrikmuhs/[email protected]
with:
key: ${{ env.os }}

- name: Install libraries
run: |
sudo apt-get update
sudo apt-get install uuid-dev libwxgtk3.0-gtk3-dev
sudo apt-get install libboost-test-dev
sudo apt-get install g++-10 gcc-10
sudo apt-get update
sudo apt-get install uuid-dev libwxgtk3.0-gtk3-dev
sudo apt-get install libboost-test-dev
sudo apt-get install g++-10 gcc-10

- name: Read antares-deps version
id: antares-deps-version
Expand Down Expand Up @@ -95,24 +109,24 @@ jobs:

- name: Configure
run: |
cmake -B _build -S src \
-DCMAKE_C_COMPILER_LAUNCHER=ccache \
-DCMAKE_C_COMPILER=/usr/bin/gcc-10 \
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
-DCMAKE_CXX_COMPILER=/usr/bin/g++-10 \
-DDEPS_INSTALL_DIR=${{github.workspace}}/rte-antares-deps-Release \
-DCMAKE_BUILD_TYPE=Release \
-DBUILD_TESTING=ON \
-DBUILD_not_system=OFF \
-DBUILD_TOOLS=ON \
-DCMAKE_PREFIX_PATH=${{ env.ORTOOLS_DIR }}/install \
-DPython3_EXECUTABLE="${{ env.Python3_ROOT_DIR }}/bin/python"
cmake -B _build -S src \
-DCMAKE_C_COMPILER_LAUNCHER=ccache \
-DCMAKE_C_COMPILER=/usr/bin/gcc-10 \
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
-DCMAKE_CXX_COMPILER=/usr/bin/g++-10 \
-DCMAKE_TOOLCHAIN_FILE=${{ github.workspace }}/vcpkg/scripts/buildsystems/vcpkg.cmake \
-DDEPS_INSTALL_DIR=${{github.workspace}}/rte-antares-deps-Release \
-DCMAKE_BUILD_TYPE=Release \
-DBUILD_TESTING=ON \
-DBUILD_not_system=OFF \
-DBUILD_TOOLS=ON \
-DCMAKE_PREFIX_PATH=${{ env.ORTOOLS_DIR }}/install \
-DPython3_EXECUTABLE="${{ env.Python3_ROOT_DIR }}/bin/python"

- name: Build
run: |
cmake --build _build -j$(nproc)


# simtest
- name: Read simtest version
id: simtest-version
Expand Down Expand Up @@ -296,3 +310,11 @@ jobs:
tag: ${{ github.event.inputs.release_tag }}
run: |
gh release upload "$tag" _build/*.tar.gz _build/*.deb

- name: Cache vcpkg binary dir
if: always()
id: save-cache-vcpkg-binary
uses: actions/cache/save@v4
with:
path: ${{ github.workspace }}/vcpkg_cache
key: vcpkg-cache-ubuntu-${{ hashFiles('src/vcpkg.json', '.git/modules/vcpkg/HEAD') }}
Loading
Loading