Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enforce 95% unit test coverage for new code #1112

Open
wants to merge 6 commits into
base: mainline
Choose a base branch
from
+123 −9
Open

Enforce 95% unit test coverage for new code #1112

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
e44cdf8
Enforce 95% unit test coverage for new code
farshidz Feb 4, 2025
8ff3ed2
Upload diff coverage report
farshidz Feb 4, 2025
fd8f5a8
Add coverage workflow changes reverted accidentally in #1047
farshidz Feb 4, 2025
64822ff
Update unit tests pipeline to print coverage and report
farshidz Feb 4, 2025
11cdfc6
Fail workflow on step failure
farshidz Feb 4, 2025
b59d44c
Fail when tests fail
farshidz Feb 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 57 additions & 2 deletions .github/workflows/largemodel_unit_test_CI.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,20 +150,75 @@ jobs:
echo "Deleted vespa_tester_app.zip"
- name: Run Large Model Unit Tests
id: run_unit_tests
continue-on-error: true
run: |
# Define these for use by marqo
export VESPA_CONFIG_URL=http://localhost:19071
export VESPA_DOCUMENT_URL=http://localhost:8080
export VESPA_QUERY_URL=http://localhost:8080
export MARQO_MAX_CPU_MODEL_MEMORY=15
export MARQO_MAX_CUDA_MODEL_MEMORY=15
export HF_HUB_ENABLE_HF_TRANSFER=1
export PRIVATE_MODEL_TESTS_AWS_ACCESS_KEY_ID=${{ secrets.PRIVATE_MODEL_TESTS_AWS_ACCESS_KEY_ID }}
export PRIVATE_MODEL_TESTS_AWS_SECRET_ACCESS_KEY=${{ secrets.PRIVATE_MODEL_TESTS_AWS_SECRET_ACCESS_KEY }}
export PRIVATE_MODEL_TESTS_HF_TOKEN=${{ secrets.PRIVATE_MODEL_TESTS_HF_TOKEN }}
export PYTHONPATH="./marqo/tests/integ_tests:./marqo/src"
pytest marqo/tests/integ_tests --largemodel
cd marqo
export PYTHONPATH="./tests:./src:."
set -o pipefail
pytest --largemodel --ignore=tests/test_documentation.py --ignore=tests/compatibility_tests \
--durations=100 --cov=src --cov-branch --cov-context=test \
--cov-report=html:cov_html --cov-report=xml:cov.xml --cov-report term:skip-covered \
--md-report --md-report-flavor gfm --md-report-output pytest_result_summary.md \
tests | tee pytest_output.txt
- name: Check Test Coverage of New Code
id: check_test_coverage
continue-on-error: true
run: |
if [[ "${GITHUB_EVENT_NAME}" == "pull_request" ]]; then
export BASE_BRANCH="${{ github.event.pull_request.base.ref }}"
cd marqo
echo "Running diff-cover against branch $BASE_BRANCH"
git fetch origin $BASE_BRANCH:$BASE_BRANCH
diff-cover cov.xml --html-report diff_cov.html --markdown-report diff_cov.md \
--compare-branch $BASE_BRANCH
else
echo "Skipping diff-cover on Push events"
echo "Skipped diff-cover on Push events" > marqo/diff_cov.md
touch marqo/diff_cov.html
fi
- name: Upload Test Report
uses: actions/upload-artifact@v4
continue-on-error: true
with:
name: marqo-test-report
path: |
marqo/cov_html/
marqo/diff_cov.html
- name: Test and coverage report summary
continue-on-error: true
run: |
echo "# Test Summary" >> $GITHUB_STEP_SUMMARY
cat marqo/pytest_result_summary.md >> $GITHUB_STEP_SUMMARY
echo "# Coverage Summary, Slow Tests and Failed Tests" >> $GITHUB_STEP_SUMMARY
echo '```text' >> $GITHUB_STEP_SUMMARY
awk '/---------- coverage:/ {flag=1} flag' marqo/pytest_output.txt >> $GITHUB_STEP_SUMMARY
echo '```' >> $GITHUB_STEP_SUMMARY
cat marqo/diff_cov.md >> $GITHUB_STEP_SUMMARY
- name: Fail Job If Tests or Coverage Failed
if: ${{ steps.run_unit_tests.outcome == 'failure' || steps.check_test_coverage.outcome == 'failure' }}
run: |
echo "Tests or coverage checks failed. Marking job as failed."
exit 1
shell: bash

Stop-Runner:
name: Stop self-hosted EC2 runner
Expand Down
25 changes: 24 additions & 1 deletion .github/workflows/unit_test_200gb_CI.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -213,9 +213,32 @@ jobs:
- name: Upload Test Report
uses: actions/upload-artifact@v4
continue-on-error: true
with:
name: marqo-test-report
path: marqo/cov_html/
path: |
marqo/cov_html/
marqo/diff_cov.html
- name: Test and coverage report summary
continue-on-error: true
run: |
echo "# Test Summary" >> $GITHUB_STEP_SUMMARY
cat marqo/pytest_result_summary.md >> $GITHUB_STEP_SUMMARY
echo "# Coverage Summary, Slow Tests and Failed Tests" >> $GITHUB_STEP_SUMMARY
echo '```text' >> $GITHUB_STEP_SUMMARY
awk '/---------- coverage:/ {flag=1} flag' marqo/pytest_output.txt >> $GITHUB_STEP_SUMMARY
echo '```' >> $GITHUB_STEP_SUMMARY
cat marqo/diff_cov.md >> $GITHUB_STEP_SUMMARY
- name: Fail Job If Tests or Coverage Failed
if: ${{ steps.run_unit_tests.outcome == 'failure' || steps.check_test_coverage.outcome == 'failure' }}
run: |
echo "Tests or coverage checks failed. Marking job as failed."
exit 1
shell: bash

Stop-Runner:
name: Stop self-hosted EC2 runner
Expand Down
48 changes: 42 additions & 6 deletions .github/workflows/unit_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,14 +22,15 @@ permissions:

jobs:
Test-Marqo:
name: Run Unit Tests
name: Run unit tests
runs-on: ubuntu-latest
environment: marqo-test-suite
steps:
- name: Checkout marqo repo
- name: Check out marqo repo
uses: actions/checkout@v3
with:
path: marqo
fetch-depth: 0 # Needed for diff coverage

- name: Set up Python 3.9
uses: actions/setup-python@v3
Expand All @@ -49,14 +50,49 @@ jobs:
# override base requirements with marqo requirements, if needed:
pip install -r marqo/requirements.dev.txt
- name: Run Unit Tests
- name: Run unit tests
id: run_unit_tests
continue-on-error: true
run: |
cd marqo
export PYTHONPATH="./src"
pytest tests/unit_tests/ --durations=100 --cov=src --cov-branch --cov-context=test --cov-report=html:cov_html --cov-report=lcov:lcov.info
set -o pipefail
pytest --durations=100 --cov=src --cov-branch --cov-context=test \
--cov-report=html:cov_html --cov-report=xml:cov.xml --cov-report term:skip-covered \
--md-report --md-report-flavor gfm --md-report-output pytest_result_summary.md \
tests/unit_tests/ | tee pytest_output.txt
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the plan going forward? Will we set separate coverage threshold for unit tests and integration tests? Or do we combine the coverage report and set one threshold for all the tests.

- name: Upload Test Report
- name: Check new code coverage
id: check_test_coverage
if: github.event_name == 'pull_request'
run: |
cd marqo
diff-cover cov.xml --compare-branch=origin/${{ github.base_ref }} --fail-under=95 \
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If unit test coverage is enforced to be above 95, do we still need to keep the same check in unit_test_200gb_CI workflow?

--html-report diff_cov.html --markdown-report diff_cov.md
- name: Upload test report
uses: actions/upload-artifact@v4
with:
name: marqo-test-report
path: marqo/cov_html/
path: |
marqo/cov_html/
marqo/diff_cov.html
- name: Test and coverage report summary
run: |
echo "# Test Summary" >> $GITHUB_STEP_SUMMARY
cat marqo/pytest_result_summary.md >> $GITHUB_STEP_SUMMARY
echo "# Coverage summary, slow tests and failed tests" >> $GITHUB_STEP_SUMMARY
echo '```text' >> $GITHUB_STEP_SUMMARY
awk '/---------- coverage:/ {flag=1} flag' marqo/pytest_output.txt >> $GITHUB_STEP_SUMMARY
echo '```' >> $GITHUB_STEP_SUMMARY
cat marqo/diff_cov.md >> $GITHUB_STEP_SUMMARY
- name: Fail if tests or coverage failed
if: ${{ steps.run_unit_tests.outcome == 'failure' || steps.check_test_coverage.outcome == 'failure' }}
run: |
echo "Tests or coverage checks failed. Marking job as failed."
exit 1
shell: bash
Loading