namespace_watcher handler exception

[NerdyShawn]

Our setup is one csec where we track in one namespace a dynamic secret that changes roughly every ~6 hours via conrojb. This secret gets copied to all other namespaces to be available for imagePulls to ecr registry via clusterSecret.

We've recently went to using replace_existing(reference) in our helm chart on clusters but we are seeing issues in the pod logs with the namespace_watcher function. This was to avoid manually having to kill clusterSecret pod to have the upstream secret propagate to the other target namespaces. This doesn't seem to be working currently and we are seeing an exception in the handler.

In this example below here is somecoolnamespace that is one of the ones that is iterated over and throwing the exception on namespace_watcher

[2024-11-05 17:17:46,266] kopf.objects         [WARNING ] [somecoolnamespace] Patching failed with inconsistencies: (('remove', ('status', 'kopf'), {'progress': {'namespace_watcher': {'started': '2024-10-25T13:50:19.639498+00:00', 'stopped': None, 'delayed': '2024-11-05T17:18:46.229134+00:00', 'purpose': 'create', 'retries': 7714, 'success': False, 'failure': False, 'message': "(404)\nReason: Not Found\nHTTP response headers: HTTPHeaderDict({'Audit-Id': '09663ae0-8017-49ba-9a6c-dabb945d7148', 'Cache-Control': 'no-cache, private', 'Content-Type': 'text/plain; charset=utf-8', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '54457f76-eead-463f-875f-38f78cd04aba', 'X-Kubernetes-Pf-Prioritylevel-Uid': '9b512c1a-e179-49c5-88b3-025f4d77b562', 'Date': 'Tue, 05 Nov 2024 17:17:46 GMT', 'Content-Length': '19'})\nHTTP response body: 404 page not found\n\n", 'subrefs': None}}}, None),)





HTTP response body: 404 page not found
HTTP response headers: HTTPHeaderDict({'Audit-Id': '09663ae0-8017-49ba-9a6c-dabb945d7148', 'Cache-Control': 'no-cache, private', 'Content-Type': 'text/plain; charset=utf-8', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '54457f76-eead-463f-875f-38f78cd04aba', 'X-Kubernetes-Pf-Prioritylevel-Uid': '9b512c1a-e179-49c5-88b3-025f4d77b562', 'Date': 'Tue, 05 Nov 2024 17:17:46 GMT', 'Content-Length': '19'})
Reason: Not Found
kubernetes.client.exceptions.ApiException: (404)
    raise ApiException(http_resp=r)
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 234, in request
    return self.request("GET", url,
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 240, in GET
    return self.rest_client.GET(url,
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 373, in request
    response_data = self.request(
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    return self.__call_api(resource_path, method,
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.api_client.call_api(
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api/custom_objects_api.py", line 1159, in get_cluster_custom_object_with_http_info
    return self.get_cluster_custom_object_with_http_info(group, version, plural, name, **kwargs)  # noqa: E501
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api/custom_objects_api.py", line 1060, in get_cluster_custom_object
    clustersecret = custom_objects_api.get_cluster_custom_object(
  File "/src/kubernetes_utils.py", line 27, in patch_clustersecret_status
    patch_clustersecret_status(
  File "/src/handlers.py", line 253, in namespace_watcher
    result = await fn(**kwargs)  # type: ignore
  File "/usr/local/lib/python3.9/site-packages/kopf/_core/actions/invocation.py", line 116, in invoke
    result = await invocation.invoke(
  File "/usr/local/lib/python3.9/site-packages/kopf/_core/actions/execution.py", line 371, in invoke_handler
    result = await invoke_handler(
  File "/usr/local/lib/python3.9/site-packages/kopf/_core/actions/execution.py", line 276, in execute_handler_once
Traceback (most recent call last):
[2024-11-05 17:17:46,228] kopf.objects         [ERROR   ] [somecoolnamespace] Handler 'namespace_watcher' failed with an exception. Will retry.
[2024-11-05 17:17:46,201] kopf.objects         [INFO    ] [somecoolnamespace] Replacing secret ecr-credentials
[2024-11-05 17:17:46,187] kopf.objects         [INFO    ] [somecoolnamespace] cloning secret in namespace somecoolnamespace
[2024-11-05 17:17:46,026] kopf.objects         [WARNING ] [somecoolnamespace] Patching failed with inconsistencies: (('remove', ('status', 'kopf'), {'dummy': '2024-11-05T17:17:45.918857+00:00'}, None),)

This is our applied values to the helm release

helm get values -n clustersecret clustersecret-cluster-secret 
USER-SUPPLIED VALUES:
affinity: {}
image:
  replace_existing: "true"
  repository: quay.io/clustersecret/clustersecret
  tag: 0.0.12
imagePullSecrets: []
kubernetesClusterDomain: cluster.local
nodeSelector: {}
podAnnotations: {}
tolerations: []

an example of the only custom resource of ClusterSecret declared for the operator to use

---
apiVersion: clustersecret.io/v1
kind: ClusterSecret
metadata:
  name: ecr-credentials
  namespace: clustersecret
  labels:
    somelabel: ecr-credentials
  annotations:
    someannotation: ecr-creds-foruse
matchNamespace:
  - '.*'
avoidNamespaces:
  - 'default'
  - 'kube-system'
  - 'flux-system'
type: kubernetes.io/dockerconfigjson
data:
  valueFrom:
    secretKeyRef:
      name: ecr-credentials
      namespace: flux-system