Skip to content

Make e2e tests idempotent #248

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Conversation

jstudler
Copy link
Contributor

Goal: Make the e2e itempotent so we don't have to manually clean up NetBox to make the tests work.

The idea is to patch .spec.preserveInNetbox of the claim, then assert that .spec.preserveInNetbox of the non-Claim is not true before exiting the test (and cleaning up the k8s resources). This is currently not possible for unknown reasons. Observations/Theories:

  • Observation: The tests fail non-deterministically and in NetBox there are leftovers.
  • Observation: The number of failed tests and the number of leftovers in NetBox does not correlate. e.g. it's possible all tests are successful but there are leftovers.
  • There is something weird with booleans, json serialisation and validation in Golang/CRDs: required boolean field will be invalid if the value was false. go-playground/validator#319
  • Theory: The cached client of chainsaw and NetBox Operator are out of sync, chainsaw observes the .spec.preserveInNetbox=false and cleans up the resources while NetBox Operator still has the old state with .spec.preserveInNetbox=true and thus does not clean up things in NetBox.
  • Theory: There is an issue in the Claim Controller updating the non-Claim spec in time

Another approach to solve this would be to run a script in the cleanup block of the tests that cleans up resources directly in NetBox e.g. everything that contains the "e2e/" prefix in the description. The advantage of this would be that we don't misuse the chainsaw test itself for cleanup.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant