Feature Request: Enable namespaceAnnotations (and namespaceLabels) as targetCustomizations

[EdKingscote]

Is your feature request related to a problem?

#1627 introduced the ability to have namespaceLabels and namespaceAnnotations as part of the bundle deployment options.

This doesn't quite go far enough, as a single set of values can only be applied to all clusters, which means annotations and labels have to be identical across all environments.

I am specifically after namespaceAnnotations in the targetCustomizations, however I think it makes sense to look at both of these elements together to also resolve #2262 in the same feature given the similarities.

I believe this could also provide a quick path for enabling #304 via static configuration as an interim step too (which is my immediate use-case for this right now)

Solution you'd like

Adding the necessary elements to the fleet.yaml doesn't appear to incur complaint, and from deeper analysis in the codebase, I believe the only missing piece of the jigsaw to enable this is in the merge function? as well as the necessary testing and documentation.

This is only the second time ever that I've attempted to write anything in Go, but this feels close to what the missing code should be?

if result.NamespaceAnnotations == nil {
	result.NamespaceAnnotations = custom.NamespaceAnnotations
} else if custom.NamespaceAnnotations != nil {
	result.NamespaceAnnotations.Data = data.MergeMaps(result.NamespaceAnnotations.Data, custom.NamespaceAnnotations.Data)
}

if result.NamespaceLabels == nil {
	result.NamespaceLabels = custom.NamespaceLabels
} else if custom.NamespaceLabels != nil {
	result.NamespaceLabels.Data = data.MergeMaps(result.NamespaceLabels.Data, custom.NamespaceLabels.Data)
}

Alternatives you've considered

I've tried to have my helm chart create the namespace with the appropriate annotations to place in a Rancher project via helm values and takeOwnership: true, however whilst the namespace lands correctly, Rancher doesn't assign appropriate permissions for non-administrative users to see everything else in the namespace, which puts me in a tricky spot!

I also considered using kustomize, but this suggests it won't work when using helm charts for the applications to be deployed.

For now, I think I'm going to end up with the plethora of fleet.yaml files in multiple directories to force split things across my different clusters and work with the top level bundle deployment options.

Anything else?

@raulcabello as you did the original work on namespaceLabels/namespaceAnnotations support - I'd value your thoughts and input!

[manno]

Possible duplicate of #2262

[EdKingscote]

@manno I'm interested in namespaceAnnotations as a targetCustomization, but it is very similar to the namespaceLabels one!

[manno]

Assigning @weyfonk because of ongoing work on #2664

[weyfonk]

Additional QA

Problem

namespaceLabels and namespaceAnnotations, specified as bundle deployment options in fleet.yaml, could only be specified once for all clusters, but could not be overridden on a per-cluster basis.

Solution

Both namespaceLabels and namespaceAnnotations are now supported in fleet.yaml's targetCustomizations.

Testing

Engineering Testing

Manual Testing

N/A

Automated Testing

End-to-end tests have been added to cover merging of namespace labels and annotations specified at the root of fleet.yaml (ie for all clusters) with those set (with different keys) within targetCustomizations. They do not cover:

• setting namespace labels or annotations only in target customizations
• overwriting labels and annotations with the same keys between all-clusters config and target customizations

QA Testing Considerations

N/A

Regressions Considerations

N/A

[sbulage]

System Information

Rancher Version	Fleet Version
v2.10.0-alpha7	fleet:v0.11.0-rc.2

QA Observations:

• Below GitRepo used to reproduce this issue.
• BundleDeployment is not created and throwing error in the fleet-controller pod
• See Error below

GitRepo Used

apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  name: test-ns-annot
  namespace: fleet-default
spec:
  branch: main
  paths:
    - target-customization
  repo: https://github.com/sbulage/test-fleet
  targets:
    - clusterName: imported-0

---

Logs From the fleet-controller pod.

2024-11-05T16:05:11Z	ERROR	Reconciler error	{"controller": "bundle", "controllerGroup": "fleet.cattle.io", "controllerKind": "Bundle", "Bundle": {"name":"test-ns-annot-target-customization","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "test-ns-annot-target-customization", "reconcileID": "32ad86b1-7f44-4d0a-8314-a27c4da2121c", "error": "panic: assignment to entry in nil map [recovered]"}
2024-11-05T16:07:55Z	ERROR	Observed a panic	{"controller": "bundle", "controllerGroup": "fleet.cattle.io", "controllerKind": "Bundle", "Bundle": {"name":"test-ns-annot-target-customization","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "test-ns-annot-target-customization", "reconcileID": "83795076-3f38-4f5c-8f55-239f391d2721", "panic": "assignment to entry in nil map", "panicGoValue": "\"assignment to entry in nil map\"", "stacktrace": "goroutine 351 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x2a7fb88, 0xc000d8d620}, {0x22f0b80, 0x2a549e0})\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:107 +0xbc\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1()\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:105 +0x112\npanic({0x22f0b80?, 0x2a549e0?})\n\t/home/runner/go/pkg/mod/golang.org/[email protected]/src/runtime/panic.go:785 +0x132\ngithub.com/rancher/fleet/internal/cmd/controller/target.(*Target).BundleDeployment(0xc000196b60)\n\t/home/runner/_work/fleet/fleet/internal/cmd/controller/target/target.go:64 +0x33b\ngithub.com/rancher/fleet/internal/cmd/controller/reconciler.(*BundleReconciler).createBundleDeployment(0xc0008160c0, {0x2a7fb88, 0xc000d8d6e0}, {{0x2a86d00?, 0xc000d8d830?}, 0x1?}, 0xc000fee900?, 0x0, {0xc0002f72c0, 0x3f})\n\t/home/runner/_work/fleet/fleet/internal/cmd/controller/reconciler/bundle_controller.go:346 +0x147\ngithub.com/rancher/fleet/internal/cmd/controller/reconciler.(*BundleReconciler).Reconcile(0xc0008160c0, {0x2a7fb88, 0xc000d8d620}, {{{0xc0008cd3f0?, 0x26e2c86?}, {0xc000c70c30?, 0x100?}}})\n\t/home/runner/_work/fleet/fleet/internal/cmd/controller/reconciler/bundle_controller.go:223 +0xad1\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile(0xc000d8d590?, {0x2a7fb88?, 0xc000d8d620?}, {{{0xc0008cd3f0?, 0x0?}, {0xc000c70c30?, 0x0?}}})\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:116 +0xbf\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler(0x2a9d640, {0x2a7fbc0, 0xc000808b90}, {{{0xc0008cd3f0, 0xd}, {0xc000c70c30, 0x22}}})\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:303 +0x3a5\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem(0x2a9d640, {0x2a7fbc0, 0xc000808b90})\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263 +0x20e\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2()\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224 +0x85\ncreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2 in goroutine 201\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:220 +0x490\n"}

[weyfonk]

The issue mentioned in the previous comment should be fixed by #3052.

[sbulage]

System Information

Rancher Version	Fleet Version
v2.10.0-alpha7	fleet:v0.11.0-rc.3

Below Steps Performed

• Below GitRepo used to reproduce this issue.
• Created a GitRepo using fleet.yaml which contains namespaceLabels and namespaceAnnotations present under targetCustomization.
• Waited for application to be created in the specified cluster(targetCustomized).
• Observed that resources are created under the custom namespace mentioned in the targetCustomization.
• Later saw that namespaceLabels and namespaceAnnotations are also added to newly created namespace.

GitRepo Used

apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  name: test-ns-annot
  namespace: fleet-default
spec:
  branch: main
  paths:
    - target-customization-ns-label-annotation
  repo: https://github.com/sbulage/test-fleet
  targets:
    - clusterName: imported-0

Screenshot showing namespaceLables/namespaceAnnotation