Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Address flakiness in tests #4565

Merged
merged 3 commits into from
Jul 26, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
🔊 Report the random state from factory boy in CI
This may help in reproducing flaky tests.
  • Loading branch information
sergei-maertens committed Jul 26, 2024
commit 09a2b078e65315ac973c1e57eb6364ce1688a8cc
1 change: 1 addition & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ env:
CAMUNDA_API_BASE_URL: http://localhost:8080/engine-rest/
CAMUNDA_USER: demo
CAMUNDA_PASSWORD: demo
TEST_REPORT_RANDOM_STATE: 'true'

jobs:

Expand Down
5 changes: 5 additions & 0 deletions src/openforms/conf/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -535,6 +535,11 @@

X_FRAME_OPTIONS = "DENY"

#
# TESTING
#
TEST_RUNNER = "openforms.tests.runner.RandomStateRunner"

#
# FIXTURES
#
Expand Down
50 changes: 50 additions & 0 deletions src/openforms/tests/runner.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
"""
Test runner with reproducible randomness.

To reproduce tests, set the ``TEST_RANDOM_STATE`` envvar from CI output. To report the
random state locally, set the ``TEST_REPORT_RANDOM_STATE`` envvar to ``true``:

..code-block:: bash

export TEST_REPORT_RANDOM_STATE=true
src/manage.py test src

See https://factoryboy.readthedocs.io/en/stable/recipes.html#using-reproducible-randomness
"""

import base64
import os
import pickle

from django.test.runner import DiscoverRunner

import factory.random


def _setup_random_state():
state = os.environ.get("TEST_RANDOM_STATE")
report_random_state = (
os.environ.get("TEST_REPORT_RANDOM_STATE", "").lower() == "true"
)
decoded_state = None

if state:
try:
decoded_state = pickle.loads(base64.b64decode(state.encode("ascii")))
except ValueError:
pass

if decoded_state:
factory.random.set_random_state(decoded_state)
elif report_random_state:
encoded_state = base64.b64encode(
pickle.dumps(factory.random.get_random_state())
)
print("Current random state: %s" % encoded_state.decode("ascii"))


class RandomStateRunner(DiscoverRunner):

def setup_test_environment(self, **kwargs):
_setup_random_state()
super().setup_test_environment(**kwargs)