Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: fix tests and use SLE containers #94

Merged
merged 5 commits into from
Jul 17, 2023
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .codespell.ignore.words
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@ nd
uptodate
upto
enque
aas
84 changes: 47 additions & 37 deletions .github/workflows/salt-shaptools-ci.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,11 @@
---
name: Package CI
# - this workflow will
# - pytest with python version [2.7, 3.6, 3.7]
# - uploads coverage data to codeClimate and the results are linked as a badges in the README
# - test on
# - SLE 12 SP5
# - python 2.7
# - SLE 15 SP5
# - python 3.6
# - deliver the package content to the configured repository
# - submit the new package content to the upstream repository
on: [push, pull_request] # yamllint disable-line rule:truthy
Expand All @@ -11,7 +14,7 @@ env:
jobs:
tab:
name: 'tabspace checking'
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04
arbulu89 marked this conversation as resolved.
Show resolved Hide resolved

# Use the Bash shell regardless whether the GitHub Actions runner is ubuntu-latest, macos-latest, or windows-latest
defaults:
Expand All @@ -27,7 +30,7 @@ jobs:
run: make test-tab
codespell:
name: 'spell checking'
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04

# Use the Bash shell regardless whether the GitHub Actions runner is ubuntu-latest, macos-latest, or windows-latest
defaults:
Expand All @@ -41,6 +44,7 @@ jobs:

- name: Install linting tools
run: |
sudo apt-get update
sudo apt-get install -y git python3 python3-pip
python3 -m pip install codespell

Expand All @@ -49,7 +53,7 @@ jobs:

shellcheck:
name: 'script syntax check'
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04

# Use the Bash shell regardless whether the GitHub Actions runner is ubuntu-latest, macos-latest, or windows-latest
defaults:
Expand All @@ -63,14 +67,15 @@ jobs:

- name: Install linting tools
run: |
sudo apt-get update
sudo apt-get install -y git python3 python3-pip shellcheck

- name: shellcheck
run: make test-shellcheck

yamllint:
name: 'yaml linting'
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04

# Use the Bash shell regardless whether the GitHub Actions runner is ubuntu-latest, macos-latest, or windows-latest
defaults:
Expand All @@ -84,15 +89,16 @@ jobs:

- name: Install linting tools
run: |
sudo apt-get update
sudo apt-get install -y git python3 python3-pip
python3 -m pip install codespell
python3 -m pip install yamllint

- name: yamllint
run: make test-yamllint

jsonlint:
name: 'json linting'
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04

# Use the Bash shell regardless whether the GitHub Actions runner is ubuntu-latest, macos-latest, or windows-latest
defaults:
Expand All @@ -106,6 +112,7 @@ jobs:

- name: Install linting tools
run: |
sudo apt-get update
sudo apt-get install -y git python3 python3-pip
python3 -m pip install jsonlint

Expand All @@ -114,7 +121,7 @@ jobs:

mlc:
name: 'markup link checker'
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04

# Use the Bash shell regardless whether the GitHub Actions runner is ubuntu-latest, macos-latest, or windows-latest
defaults:
Expand All @@ -137,53 +144,56 @@ jobs:
run: make test-mlc

python:
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04
strategy:
# do not fail if other test fails
fail-fast: false
matrix:
os: [ubuntu-20.04]
python-version: [2.7, 3.6, 3.7]
container:
- registry.suse.com/suse/sles12sp5:latest # python 2.7
yeoldegrove marked this conversation as resolved.
Show resolved Hide resolved
- registry.suse.com/bci/bci-base:15.5 # python 3.6
container:
image: ${{ matrix.container }}
steps:
- name: Auth to SCC and minimal dependencies
run: |
echo "username=${{ secrets.SCC_USERNAME }}" >/etc/zypp/credentials.d/SCCcredentials
echo "password=${{ secrets.SCC_PASSWORD }}" >>/etc/zypp/credentials.d/SCCcredentials
zypper ref -s
zypper -n in -y tar gzip git
- uses: actions/checkout@v2
with:
fetch-depth: 0
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install pytest
pip install pyzmq PyYAML pycrypto msgpack-python jinja2 psutil futures tornado pytest-salt mock pytest-cov
# on old python version, install and clone ...
if test -d /opt/hostedtoolcache/Python/2.7.*; then
pip install enum34
git clone --branch=openSUSE-3000.3 --depth=50 https://github.com/openSUSE/salt ../salt
zypper -n in -y make python
if test -f /usr/bin/python3; then
# minimal python packages and compilers
zypper -n in -y python3-pip python3-devel gcc
# use current salt version shipped with SLE 15
git clone --branch=openSUSE/release/3006.0 --depth=50 https://github.com/openSUSE/salt ../salt
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't remember this hacky thing we had before 😅
If we want to install salt using the same version shipped in SLE, why don't we use zypper?
This way, we would always have the latest rolling version I guess

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@arbulu89 Again, for Leap 42.3 there will be no updated anymore. Also for SLE 12.5 one needs to register an additional Module to install salt.
python-shaptools is also not available for Leap ...

I really would vote to use SLE. One could even think about building images before running the CI steps...

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I forgot is that there also is some magic in tests/run.sh that uses stuff from the salt repo that is not delivered as rpm.

cp salt/modules/*.py ../salt/salt/modules/
cp salt/states/*.py ../salt/salt/states/
cp tests/unit/modules/*.py ../salt/tests/unit/modules/
cp tests/unit/states/*.py ../salt/tests/unit/states/
py.test -vv ../salt/tests/unit/modules/test_hanamod.py ../salt/tests/unit/states/test_hanamod.py  ../salt/tests/unit/modules/test_crmshmod.py  ../salt/tests/unit/modules/test_saptunemod.py ../salt/tests/unit/modules/test_sapcarmod.py ../salt/tests/unit/states/test_crmshmod.py ../salt/tests/unit/modules/test_drbdmod.py ../salt/tests/unit/states/test_drbdmod.py ../salt/tests/unit/states/test_saptunemod.py ../salt/tests/unit/modules/test_netweavermod.py ../salt/tests/unit/states/test_netweavermod.py ../salt/tests/unit/states/test_sapcarmod.py --cov=salt.modules.hanamod --cov=salt.states.hanamod --cov=salt.modules.crmshmod --cov=salt.states.crmshmod --cov=salt.modules.drbdmod --cov=salt.modules.saptunemod --cov=salt.modules.sapcarmod --cov=salt.states.saptunemod --cov=salt.states.drbdmod --cov=salt.modules.netweavermod --cov=salt.states.netweavermod --cov=salt.states.sapcarmod --cov-config .coveragerc --cov-report term --cov-report xml --cov-report html

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed another commit that at least installs the packaged salt version. Had to do a little other change to tets/run.sh.

# python 3.6 - official requirements from salt (works with python >3.6)
pip install -r ../salt/requirements/pytest.txt
pip install -r tests/requirements.3.6.yaml # pinned pytest-cov
else
git clone --branch=openSUSE-3002.2 --depth=50 https://github.com/openSUSE/salt ../salt
# minimal python packages and compilers
zypper -n in -y python-pip python-devel gcc gcc-c++
# python 2.7 - latest available versions for old python release
pip install -r tests/requirements.2.7.yaml
# use current salt version shipped with SLE 12
git clone --branch=openSUSE/release/3000.3 --depth=50 https://github.com/openSUSE/salt ../salt
fi
rm ../salt/tests/conftest.py
git clone --depth=50 https://github.com/SUSE/shaptools.git ../shaptools
pip install -e ../salt
pip install ../shaptools
- name: execute test script
run: make test-python
- name: change absolute path to relative path for successful upload to code climate
run: |
WORKSPACE=`dirname ${{ github.workspace }}`/salt/
sed -i "s#$WORKSPACE##g" coverage.xml
- name: Publish code coverage
uses: paambaati/[email protected]
if: ${{ env.CC_TEST_REPORTER_ID != null && github.event_name != 'pull_request' }}
env:
CC_TEST_REPORTER_ID: ${{ secrets.CC_TEST_REPORTER_ID }}
with:
coverageLocations: coverage.xml:coverage.py


delivery:
needs: [tab, codespell, shellcheck, yamllint, jsonlint, mlc, python]
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04
if: ${{ github.event_name != 'pull_request' }}
container:
image: shap/continuous_deliver
Expand Down Expand Up @@ -211,7 +221,7 @@ jobs:

submit:
needs: [tab, codespell, shellcheck, yamllint, jsonlint, mlc, python, delivery]
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04
if: ${{ github.event_name != 'pull_request' && github.ref == 'refs/heads/master' }}
container:
image: shap/continuous_deliver
Expand Down
11 changes: 8 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
[![Package CI](https://github.com/SUSE/salt-shaptools/actions/workflows/salt-shaptools-ci.yml/badge.svg)](https://github.com/SUSE/salt-shaptools/actions/workflows/salt-shaptools-ci.yml)
[![Test Coverage](https://api.codeclimate.com/v1/badges/770395dbb4bb868502b3/test_coverage)](https://codeclimate.com/github/SUSE/salt-shaptools/test_coverage)
[![Maintainability](https://api.codeclimate.com/v1/badges/770395dbb4bb868502b3/maintainability)](https://codeclimate.com/github/SUSE/salt-shaptools/maintainability)

# SAP Applications and SUSE Linux Enterprise High Availability salt modules and States

Expand Down Expand Up @@ -84,7 +82,14 @@ Your directory layout should looks like ( all the 3 dirs are in same three dir l
```bash
virtualenv saltvirtenv
source saltvirtenv/bin/activate
pip install pyzmq PyYAML pycrypto msgpack-python jinja2 psutil futures tornado pytest-salt-factories mock pytest-cov

# python 3.6 - official requirements from salt (works with python >3.6)
pip install -r ../salt/requirements/pytest.txt
pip install -r tests/requirements.3.6.yaml # pinned pytest-cov
# or
# python 2.7 - latest available versions for old python release
pip install -r tests/requirements.2.7.yaml

pip install -e ../salt
pip install -e ../shaptools
rm ../salt/tests/conftest.py # remove this file from the saltstack repo
Expand Down
2 changes: 1 addition & 1 deletion salt/modules/drbdmod.py
Original file line number Diff line number Diff line change
Expand Up @@ -282,7 +282,7 @@ def _is_no_backing_dev_request(res, output):

def _get_json_output_save(command):
'''
A warpper of get json command to acommandate json output issues
A wrapper of get json command to acommandate json output issues
'''

error_str = '"estimated-seconds-to-finish": nan,'
Expand Down
11 changes: 11 additions & 0 deletions tests/requirements.2.7.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
setuptools==44.1.1
wheel==0.37.1
pytest-cov==2.3.1
jinja2==2.6
pyyaml==3.12
markupsafe==0.23
requests==2.27
certifi==2020.4.5.1
pyzmq==18.0.1
mock==3.0.0
funcsigs==1.0.2
1 change: 1 addition & 0 deletions tests/requirements.3.6.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
pytest-cov==4.0.0