Avoid stressing memory by watching arbitrary secrets on the cluster

[kwiatekus]

Description

Exclude secrets and config maps from being auto-loaded to controllers cache

Reasons

Controller's cache can take huge amount of memory (causing exceeded limit and OOM killing by k8s control-plane) in case there is a vast amount of secrets/ config maps in the cluster.

Attachments

    mgr, err := manager.New(ctrl.GetConfigOrDie(), manager.Options{
  	  Scheme: s,
  	  Client: client.Options{
  		  Cache: &client.CacheOptions{
  			  Unstructured: false,
  			   DisableFor: []client.Object{
  			   	&corev1.Secret{},
  			   },
  		  },
  	  },
    })

[kwiatekus]

The fix didnt help.
Warden admission is still OOMKilled on clusters with ~7k secrets

[pPrecel]

The warden admission problem is caused by the Watch operation on secrets in the webhook SetupResourcesController method here. After removing these lines problem disappeared. It needs a more advanced investigation because I guess removing the secret watcher is not a solution.

To test it I've used such script:

#!/bin/bash

kyma provision k3d

kubectl create namespace kyma-system
operatorManifest=$(helm upgrade --install --wait --set global.config.data.logging.level=debug --set admission.enabled=true --dry-run=server -o yaml  warden ./charts/warden/)

# create 2k secrets with huge data
x=1
secretData="${operatorManifest}${operatorManifest}${operatorManifest}${operatorManifest}${operatorManifest}${operatorManifest}${operatorManifest}${operatorManifest}${operatorManifest}${operatorManifest}${operatorManifest}${operatorManifest}"
while [ $x -le 2000 ]; do 
    kubectl create secret generic secret-$x --from-literal data="${secretData}"
    x=$(( $x + 1 ))
done

# install warden
helm upgrade --install --wait --set global.config.data.logging.level=debug --set admission.enabled=true warden ./charts/warden/

To test it without the Watch function I removed it, built and push a new image using the:

docker build -t pprecel/admission:0.1 -f docker/admission/Dockerfile .
docker push pprecel/admission:0.1

and replaced the admissions image in the values.yaml

[halamix2]

Setting predicates to turn off all processing doesn't help

	if err := c.Watch(&source.Kind{
		Type: &corev1.Secret{}},
		&handler.EnqueueRequestForObject{},
		predicate.Funcs{
			CreateFunc:  func(e event.CreateEvent) bool { return false },
			DeleteFunc:  func(e event.DeleteEvent) bool { return false },
			UpdateFunc:  func(e event.UpdateEvent) bool { return false },
			GenericFunc: func(e event.GenericEvent) bool { return false },
		},

I think the sceret watcher is used to await changes just in the webhook secret. if we could watch just one secret it would be great

[halamix2]

Fixed in #188, OOMkilled issue no longer appears