Enable users to customize BTP Operator secret and configMap

[PK85]

Description

We will allow to modify and keep user modifications in the btp-manager-secret.

Reason

Users wants to migrate btp-operator resources and configurations from one k8s cluster into new Kyma cluster.

AC:

• 1) support label on our btp manager secret sap-btp-manager to skip reconciliation. If secret deleted then reconciliation is back.
  • 1.1) In the BTP manager plutono add dashboard about number of skipped, and be able to see a list of ShootName from those skipped ones
• 2) if sap-btp-manager secret is changed, immediately reconcile btp-operator
  • 2.1) CM modification is not visible to the btp-operator-controller pod. Only restart of the pod will load it again
  • 2.2) Secret sap-btp-operator-clusterid is always created and INITIAL_CLUSTER_ID from it is always in use. When pod will restart from 2.1 then btp-operator-controller will fail to start because CLUSTER_ID from CM != INITIAL_CLUSTER_ID from secret. panic: ClusterID changed, which is not supported. Please redeploy with --set cluster.id=test_cluster_id goroutine 1 [running]: main.main() /workspace/main.go:140 +0x1d65
  • 2.3) TODO in the BTP-Manager: if user changes CLUSTER_ID in the btp-manager-secret then BTP-Manager deletes a secret sap-btp-operator-clusterid, checks if CM: sap-btp-operator-config has new CLUSTER_ID set, restart sap-btp-operator-controller-manager-58fbb647ff-72ckx pods, checks if sap-btp-operator-clusterid is back with new INITIAL_CLUSTER_ID.
• 3) E2E integration test
  Issue: End-to-end (e2e) integration test for customizing BTP Manager input secret #921
  PR: E2E BTP Manager secret customization test  #935
• 4) docs
• 5) RNs

Agreed label name:

• kyma-project.io/skip-reconciliation: 'true'

NOTE!:

• I changed manually CLUSTER ID in our secret, then config map was having it but still instances were created with previous previous CLUSTER ID, see code: https://github.com/SAP/sap-btp-service-operator/blob/9cebc45aded4d8b58396f913720168b0d1307f73/main.go#L208

[szwedm]

AC:
...

• 3) E2E integration test, ticket: End-to-end (e2e) integration test for customizing BTP Manager input secret #921

PR with the e2e test: #935

[szwedm]

@ukff see the following code to understand management_namespace config:
https://github.com/SAP/sap-btp-service-operator/blob/cca53330710b7c662001ac10e35f91016c27887d/internal/config/config.go#L19
https://github.com/SAP/sap-btp-service-operator/blob/cca53330710b7c662001ac10e35f91016c27887d/internal/utils/sm_utils.go#L19

[ukff]

User can modify sap-btp-manager Secret by updating following fields:
cluster_id, management_namespace, clientid, clientsecret, sm_url, tokenurl

If User will change cluster_id field in sap-btp-manager Secret, then btp-manager will force re-creation of sap-btp-operator-clusterid by deleting old one and restarting operator deployment sap-btp-operator-controller-manager. To newly created secret will be inserted value from sap-btp-manager Secret. Also value will be added/modified in sap-btp-operator-config ConfigMap.

If User will add/modify field management_namespace in sap-btp-manager Secret, then, sap-btp-service-operator Secret will be moved to corresponding namespace, if namespace is not present, it will be created
Also value will be added/modified in sap-btp-operator-config ConfigMap.

Values from sap-btp-manager secret ("clientid", "clientsecret", "sm_url", "tokenurl", "cluster_id") will be mapped to sap-btp-service-operator

Secret sap-btp-operator-clusterid is always in default namespace.

[szwedm]

If User will change cluster_id field in sap-btp-manager Secret, then btp-manager will force re-creation of sap-btp-operator-clusterid by deleting old one and restarting operator deployment sap-btp-operator-controller-manager. To newly created secret will be inserted value from sap-btp-manager Secret. Also value will be added/modified in sap-btp-operator-config ConfigMap.

Are you sure about the order of resources handling and values insertion?

If User will add/modify field management_namespace in sap-btp-manager Secret, then, sap-btp-service-operator Secret will be moved to corresponding namespace, if namespace is not present, it will be created Also value will be added/modified in sap-btp-operator-config ConfigMap.

When management_namespace environment variable change will be visible in the controller in the pod?

Secret sap-btp-operator-clusterid is always in default namespace.

Which namespace is default namespace?

[jaroslaw-pieszka]

Could you specify all the relevant keys (in secrets and in the config map)?

  MANAGEMENT_NAMESPACE: kyma-system
  RELEASE_NAMESPACE: kyma-system
  management_namespace: kuma-system