Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

K8SPS-331 - chore: migrate from kuttl to chainsaw #560

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

eddycharly
Copy link

@eddycharly eddycharly commented Feb 15, 2024

K8SPS-331 Powered by Pull Request Badge

This PR migrates e2e tests from kuttl to chainsaw.

Chainsaw was created because kuttl stopped to be actively maintained (it's actually a single maintainer now).

We are also solving most of the kuttl limitations that exist today and wrote a command to automatically migrate from kuttl to chainsaw. It supercharges e2e tests with better logs, config maps assertions, assertions trees and many more things 💪

This is a relatively young project but it has already been adopted by a couple of e2e tests heavy projects (kyverno, keptn, fairwinds rbac-manager, grafana-operator, redis-operator and others).

To give more context, you can see similar PRs at:

Sadly, kuttl haven't done a release since jan 2023.

Here you can find a good video showcasing chainsaw: https://www.youtube.com/watch?v=Ejof-wtAdQM&ab_channel=CamilaMacedo

To make things more concrete, there are a couple of issues in this repo, a few files are not valid and kuttl doesn't complain because unmarshalling is not strict so the non-valid fields are simply ignored. This is a problem because the behaviour of the tests is not the expected one.

If you need more infos 👇

@it-percona-cla
Copy link

it-percona-cla commented Feb 15, 2024

CLA assistant check
All committers have signed the CLA.

@eddycharly eddycharly force-pushed the chainsaw branch 2 times, most recently from 3f4db0a to 97304d4 Compare February 15, 2024 21:30
@eddycharly eddycharly marked this pull request as ready for review February 15, 2024 21:30
@eddycharly
Copy link
Author

Ooops, i think i need to do something for jenkins 🤔

@hors
Copy link
Collaborator

hors commented Feb 16, 2024

Ooops, i think i need to do something for jenkins 🤔

please check https://github.com/percona/percona-server-mysql-operator/blob/main/Jenkinsfile

@eddycharly
Copy link
Author

Thanks @hors !

How does chainsaw gets installed on jenkins ?

Signed-off-by: Charles-Edouard Brétéché <[email protected]>
@eddycharly
Copy link
Author

How does chainsaw gets installed on jenkins ?

Found it.

Signed-off-by: Charles-Edouard Brétéché <[email protected]>
@eddycharly
Copy link
Author

@hors @tplavcic why Error: open ./e2e-tests/kuttl.yaml: no such file or directory in the logs ?

@eddycharly
Copy link
Author

There's something i'm not getting here

@eddycharly
Copy link
Author

@hors @tplavcic a bit of help would be greatly appreciated.

Signed-off-by: Charles-Edouard Brétéché <[email protected]>
Signed-off-by: Charles-Edouard Brétéché <[email protected]>
@JNKPercona
Copy link
Collaborator

Test name Status
async-ignore-annotations failure
auto-config failure
config failure
config-router failure
demand-backup failure
gr-demand-backup failure
gr-demand-backup-haproxy failure
gr-haproxy failure
gr-ignore-annotations failure
gr-init-deploy failure
gr-one-pod failure
gr-recreate failure
gr-scaling failure
gr-self-healing failure
gr-tls-cert-manager failure
gr-users failure
haproxy failure
init-deploy failure
limits failure
monitoring failure
one-pod failure
operator-self-healing failure
recreate failure
scaling failure
service-per-pod failure
sidecars failure
smart-update failure
tls-cert-manager failure
users failure
version-service failure
We run 30 out of 30

commit: 8c68083
image: perconalab/percona-server-mysql-operator:PR-560-8c68083

@eddycharly
Copy link
Author

@tplavcic let me know when i should resolve conflicts

@tplavcic
Copy link
Member

@eddycharly We have this problem: https://docs.cloudbees.com/docs/cloudbees-ci-kb/latest/client-and-managed-controllers/why-jenkinsfile-changes-are-not-reflected-in-pr-build
So we need to find some workaround, because I have tried to create a draft PR and it seems it is running chainsaw code: #566

@eddycharly
Copy link
Author

@tplavcic got it !

Maybe we can have both kuttl and chainsaw first, then remove kuttl from jenkins, and finally remove kuttl tests ?

I think in this order it should work 🤷

@tplavcic tplavcic changed the title chore: migrate from kuttl to chainsaw K8SPS-331 - chore: migrate from kuttl to chainsaw Mar 1, 2024
@pull-request-size pull-request-size bot added the size/XXL 1000+ lines label Mar 1, 2024
@tplavcic
Copy link
Member

tplavcic commented Mar 1, 2024

Hi @eddycharly !
We have added chainsaw binary into our pipeline so you can continue and try to run the tests. So please revert your changes in Jenkinsfile (jenkins will not run them anyway) and merge the current main branch to your branch.

FYI: Currently we are working on 0.7.0 release of this operator and we cannot include chainsaw tests into this release, but we will consider it for the next one (we will need to inspect the tool more, have more test runs etc. to be more confident).

Is there a possibility to share a tool for migration? Can this tool be re-run simply/quickly if we have many test changes?

Thank you very much for your work!

@eddycharly
Copy link
Author

@tplavcic i will work on this next week, thanks for the update.

@tplavcic
Copy link
Member

@eddycharly There are some discussions in the team about the direction for our E2E tests so I propose for the moment you don't spend more time on this PR until some decision is made. Thanks!

@eddycharly
Copy link
Author

Hi @tplavcic thanks for letting me know !
Ping me when you make a decision.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants