Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: Add integration tests for Kubernetes docker-registry deployment #328

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 4 additions & 5 deletions .github/workflows/integration_test.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@

name: Integration tests

on:
Expand Down Expand Up @@ -28,9 +27,9 @@ jobs:
outputs:
channel: ${{ steps.charmcraft.outputs.channel }}
steps:
- uses: actions/checkout@v4
- id: charmcraft
run: echo "channel=$(cat .charmcraft-channel)" >> $GITHUB_OUTPUT
- uses: actions/checkout@v4
- id: charmcraft
run: echo "channel=$(cat .charmcraft-channel)" >> $GITHUB_OUTPUT

integration-tests:
uses: canonical/operator-workflows/.github/workflows/integration_test.yaml@a23cb8af2877316bd87af472b1732318386bd05c
Expand All @@ -44,7 +43,7 @@ jobs:
builder-label: ubuntu-22.04
tester-arch: AMD64
tester-size: xlarge
modules: '["test_k8s", "test_etcd", "test_ceph", "test_upgrade", "test_external_certs"]'
modules: '["test_k8s", "test_etcd", "test_ceph", "test_upgrade", "test_external_certs", "test_registry"]'
# built and test on on self-hosted
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@addyess why do we need to explicitly list each test? That feels like it just waits to be forgotten.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The operator workflows run a unique gh runner for every item in this list. We run all the tests on a unique runner for these 5 test modules b/c they deploy 5 different SKUs. If we pushed the registry tests into another module, they wouldn't have to be added to this list. you COULD push this into the test_k8s module. Or leave it as is and add this modules here.

we could detect this list i suppose. Not good for this PR though

- id: arm64
builder-label: ARM64
Expand Down
7 changes: 5 additions & 2 deletions tests/integration/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
import shlex
import string
from pathlib import Path
from typing import Optional
from typing import AsyncGenerator, Optional

import juju.controller
import juju.utils
Expand Down Expand Up @@ -224,14 +224,17 @@ async def deploy_model(


@pytest_asyncio.fixture(scope="module")
async def kubernetes_cluster(request: pytest.FixtureRequest, ops_test: OpsTest):
async def kubernetes_cluster(
request: pytest.FixtureRequest, ops_test: OpsTest
) -> AsyncGenerator[Model, None]:
"""Deploy kubernetes charms according to the bundle_marker."""
model = "main"
bundle, markings = await Bundle.create(ops_test)

with ops_test.model_context(model) as the_model:
if await is_deployed(the_model, bundle.path):
log.info("Using existing model=%s.", the_model.uuid)
assert ops_test.model, "Model must be present"
yield ops_test.model
return

Expand Down
12 changes: 12 additions & 0 deletions tests/integration/data/test_registries/pod.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# Copyright 2025 Canonical Ltd.
# See LICENSE file for licensing details.

apiVersion: v1
kind: Pod
metadata:
name: test-pod
spec:
containers:
- name: test-container
#image: set by test
command: ["sleep", "100"]
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Copyright 2025 Canonical Ltd.
# See LICENSE file for licensing details.

name: integration-test-docker-registry
description: |-
Used to deploy or refresh within an integration test model
series: jammy
applications:
k8s:
charm: k8s
constraints: cores=2 mem=8G root-disk=16G
num_units: 1
docker-registry:
charm: docker-registry
channel: latest/edge
num_units: 1
81 changes: 81 additions & 0 deletions tests/integration/test_registry.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
#!/usr/bin/env python3

# Copyright 2025 Canonical Ltd.
# See LICENSE file for licensing details.

# pylint: disable=duplicate-code
"""Integration tests."""

import json
import logging
import random
import string
from pathlib import Path
from typing import List

import pytest
import yaml
from juju import model
from kubernetes.utils import create_from_yaml

from . import helpers

# This pytest mark configures the test environment to use the Canonical Kubernetes
# bundle with ceph, for all the test within this module.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment does not fit

pytestmark = [
pytest.mark.bundle(file="test_registries/test-bundle-docker-registry.yaml", apps_local=["k8s"])
]

log = logging.getLogger(__name__)


def _get_data_file_path(name) -> Path:
"""Retrieve the full path of the specified test data file."""
return Path(__file__).parent / "data" / "test_registries" / name
Comment on lines +32 to +34
Copy link
Contributor

@addyess addyess Feb 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, this can just be a CONSTANT

Suggested change
def _get_data_file_path(name) -> Path:
"""Retrieve the full path of the specified test data file."""
return Path(__file__).parent / "data" / "test_registries" / name
TEST_DATA_PATH = Path(__file__).parent / "data/test_registries/pod.yaml"



@pytest.mark.abort_on_fail
async def test_custom_registry(kubernetes_cluster: model.Model, api_client):
"""Test that the charm configures the correct directory and can access a custom registry."""
# List of resources created during the test
created: List = []

docker_registry_unit = kubernetes_cluster.applications["docker-registry"].units[0]
docker_registry_ip = await docker_registry_unit.get_public_address()

config_string = json.dumps(
[{"url": f"http://{docker_registry_ip}:5000", "host": f"{docker_registry_ip}:5000"}]
)

custom_registry_config = {"containerd-custom-registries": config_string}

await kubernetes_cluster.applications["k8s"].set_config(custom_registry_config)
await kubernetes_cluster.wait_for_idle(status="active")

action = await docker_registry_unit.run_action("push", image="busybox:latest", pull=True)
await action.wait()

# Create a pod that uses the busybox image from the custom registry
# Image: {docker_registry_ip}:5000/busybox:latest
test_pod_manifest = list(yaml.safe_load_all(_get_data_file_path("pod.yaml").open()))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's inline this path. It is only used once.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
test_pod_manifest = list(yaml.safe_load_all(_get_data_file_path("pod.yaml").open()))
test_pod_manifest = list(yaml.safe_load_all(TEST_DATA_PATH.open()))


random_pod_name = "test-pod-" + "".join(
random.choices(string.ascii_lowercase + string.digits, k=5)
)
test_pod_manifest[0]["metadata"]["name"] = random_pod_name

test_pod_manifest[0]["spec"]["containers"][0]["image"] = (
f"{docker_registry_ip}:5000/busybox:latest"
)

k8s_unit = kubernetes_cluster.applications["k8s"].units[0]
try:
created.extend(*create_from_yaml(api_client, yaml_objects=test_pod_manifest))
await helpers.wait_pod_phase(k8s_unit, random_pod_name, "Running")
finally:
# Cleanup
for resource in created:
kind = resource.kind
name = resource.metadata.name
event = await k8s_unit.run(f"k8s kubectl delete {kind} {name}")
_ = await event.wait()
Loading