Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: fix e2e_benchmark comparison #2850

Merged
merged 1 commit into from
Jan 25, 2024
Merged

ci: fix e2e_benchmark comparison #2850

merged 1 commit into from
Jan 25, 2024

Conversation

m1ghtym0
Copy link
Member

@m1ghtym0 m1ghtym0 commented Jan 24, 2024

Context

The e2e_benchmark test correctly compares the current benchmark with the previous one.
However, there are two problems:

  1. The comparison results do not lead to a failed test due to wrong/missing error handling
  2. The test ALWAYS overwrites the stored results in the S3 bucket with the new results, leading to a degradation of our baseline

Proposed change(s)

  • Update comparison to fail when the new values are outside the threshold (80% of the previous benchmark)
  • Only upload new results if the comparison succeeds -> Baseline can still slowly degrade, but I'd suspect the performance to either be stable or drop significantly in a single step

Additional info

Checklist

  • Add labels (e.g., for changelog category)
  • Is PR title adequate for changelog?

@m1ghtym0 m1ghtym0 added bug fix Fixing a bug no changelog Change won't be listed in release changelog labels Jan 24, 2024
@m1ghtym0 m1ghtym0 requested review from 3u13r and thomasten January 24, 2024 11:25
Copy link

netlify bot commented Jan 24, 2024

Deploy Preview for constellation-docs ready!

Name Link
🔨 Latest commit 078c817
🔍 Latest deploy log https://app.netlify.com/sites/constellation-docs/deploys/65b0f39820232200084ebc6a
😎 Deploy Preview https://deploy-preview-2850--constellation-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@m1ghtym0
Copy link
Member Author

Test with failure case failed correctly: https://github.com/edgelesssys/constellation/actions/runs/7652366550

@m1ghtym0 m1ghtym0 merged commit 26f6fd0 into main Jan 25, 2024
6 of 8 checks passed
@m1ghtym0 m1ghtym0 deleted the fix/e2e_benchmark branch January 25, 2024 10:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug fix Fixing a bug no changelog Change won't be listed in release changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants