Cluser scoped resources and owner references

[nabuskey]

Related to #44

If you create a RG with cluster resources, the owner references cannot point to the correct instance of RG.

RG:

apiVersion: x.symphony.k8s.aws/v1alpha1
kind: ResourceGroup
metadata:
  name: deploymentservice.x.symphony.k8s.aws
  namespace: ns1
spec:
  apiVersion: v1alpha1
  kind: DeploymentServiceTest
  definition:
    spec:
      name: string
      namespace: string | default=default
  resources:
  - name: clusterrole
    definition:
      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRole
      metadata:
        name: pod-reader
      rules:
        - apiGroups: [ "" ]
          resources: [ "pods" ]
          verbs: [ "get", "watch", "list" ]

instance

apiVersion: x.symphony.k8s.aws/v1alpha1
kind: DeploymentServiceTest
metadata:
  name: my-deployment-and-service3
  namespace: ns1
spec:
  name: app3
  namespace: ns1

ClusterRole created:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: pod-reader
  ownerReferences:
  - apiVersion: x.symphony.k8s.aws/v1alpha1
    controller: true
    kind: DeploymentServiceTest
    name: my-deployment-and-service3
    uid: ab1b4ed8-fd0a-4d39-812e-3f9de8c7239f

This is incorrect because the DeploymentServiceTest object is in the ns1 namespace.

The namespace field does not exist on the struct because k8s does not allow namespaced objects owning cluster objects.
https://github.com/kubernetes/apimachinery/blob/c463db196543ee48d2353c6ca3325de6454ad075/pkg/apis/meta/v1/types.go#L291-L320

The ownership relationship is lost and tools like argocd / lineage cannot make sense of these resources.

[a-hilaly]

Not sure how to deal with this one.. My high level thought is that we might want to reject resourcegroups that are namespace scoped and contain cluster-scoped resources. Or maybe warn the user that the tracking via ownerrefrerences might not work as expected...

[nabuskey]

Some thoughts.

1. Not having support for cluster scoped resources. I am fairly confident that we will get a feature request for this. Being able to stitch together cluster scoped resources and namespace scoped resources is quite nice to have. For example, creating ESO ClusterSecretStore along with ConfigMaps / Secrets to offer a cluster wide secret gateway.
2. Broken owner references are not ideal imo. Many tools use the field to figure out what owns what. Technically speaking, what Symphony is doing is not within spec. We could just not add owner references at all for cluster scoped resources, but some people may have issues with that.
3. We could offer cluster version of RG. This does solve this issue, but is it a good user experience? I'm not sure tbh.

[a-hilaly]

We disabled ownerrefrences for now. This probably deserve a proper design doc.