Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CrashLoopBackoff with the cert-manager-plugin #41

Open
JPeer264 opened this issue Dec 22, 2023 · 4 comments
Open

CrashLoopBackoff with the cert-manager-plugin #41

JPeer264 opened this issue Dec 22, 2023 · 4 comments

Comments

@JPeer264
Copy link

It seems that our vcluster cert-manager-plugin sidecar is having a CrashLoopBackoff.

We are running

  • cert-manager: 1.13.3 (helm chart)
  • vcluster: 0.18.1
  • Kubernetes: 1.26.9-do.0 (DigitalOcean)

From the logs it looks like there is some permission error with some existing CRDs. I double checked the service accounts and there are two service accounts within my-namespace: vc-my-namespace and vc-workload-my-namespace (but both without any secrets attached). Maybe it is important to say that recently we upgraded our Kubernetes cluster from 1.25 to 1.26.

Here are the logs from the restarting cert-manager-plugin sidecar:

I1221 16:53:53.918853       1 logr.go:249] plugin: Try creating context...
I1221 16:53:54.185675       1 logr.go:249] plugin: Plugin server listening on localhost:14000
I1221 16:53:54.189780       1 logr.go:249] plugin: Waiting for vcluster to become leader...
I1221 16:53:54.193890       1 logr.go:249] plugin: Starting syncers...
W1221 16:53:54.307447       1 util.go:16] Skip setting owner, because current namespace my-namespace != target namespace 
I1221 16:53:54.307625       1 logr.go:249] plugin: Start syncer certificate
I1221 16:53:54.307879       1 logr.go:249] plugin: Start syncer issuer
I1221 16:53:54.308329       1 logr.go:249] controller issuer controllerGroup cert-manager.io controllerKind Issuer: Starting Controller
I1221 16:53:54.308422       1 logr.go:249] controller issuer controllerGroup cert-manager.io controllerKind Issuer: Starting workers worker count 1
I1221 16:53:54.308357       1 logr.go:249] controller issuer controllerGroup cert-manager.io controllerKind Issuer: Starting EventSource source kind source: *v1.Certificate
I1221 16:53:54.308790       1 logr.go:249] controller issuer controllerGroup cert-manager.io controllerKind Issuer: Starting EventSource source &{{%!s(*v1.Certificate=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] []  []} {<nil>  <nil> <nil> [] [] [] []  <nil> <nil> {  } false [] <nil> <nil> <nil> []} {[] <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}) %!s(*cache.informerCache=&{0xc0002852c0}) %!s(chan error=<nil>) %!s(func()=<nil>)}}
I1221 16:53:54.308846       1 logr.go:249] controller issuer controllerGroup cert-manager.io controllerKind Issuer: Starting EventSource source kind source: *v1.Ingress
I1221 16:53:54.308855       1 logr.go:249] controller issuer controllerGroup cert-manager.io controllerKind Issuer: Starting Controller
I1221 16:53:54.308516       1 logr.go:249] controller issuer controllerGroup cert-manager.io controllerKind Issuer: Starting EventSource source &source.Kind{Type:(*v1.Issuer)(0xc0003ecf20), cache:(*cache.informerCache)(0xc0003ea2b0), started:(chan error)(nil), startCancel:(func())(nil)}
I1221 16:53:54.309124       1 logr.go:249] controller issuer controllerGroup cert-manager.io controllerKind Issuer: Starting EventSource source &source.kindWithCache{kind:source.Kind{Type:(*v1.Issuer)(0xc0003ecdc0), cache:(*cache.informerCache)(0xc0002a0498), started:(chan error)(nil), startCancel:(func())(nil)}}
I1221 16:53:54.309196       1 logr.go:249] plugin: Start syncer secret
I1221 16:53:54.309303       1 logr.go:249] plugin: Successfully started plugin.
I1221 16:53:54.310266       1 logr.go:249] controller secret controllerGroup  controllerKind Secret: Starting EventSource source kind source: *v1.Secret
I1221 16:53:54.310443       1 logr.go:249] controller secret controllerGroup  controllerKind Secret: Starting EventSource source &{{%!s(*v1.Secret=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] []  []} <nil> map[] map[] }) %!s(*cache.informerCache=&{0xc0002852c0}) %!s(chan error=<nil>) %!s(func()=<nil>)}}
I1221 16:53:54.310644       1 logr.go:249] controller secret controllerGroup  controllerKind Secret: Starting EventSource source kind source: *v1.Certificate
I1221 16:53:54.310732       1 logr.go:249] controller secret controllerGroup  controllerKind Secret: Starting EventSource source kind source: *v1.Issuer
I1221 16:53:54.310766       1 logr.go:249] controller secret controllerGroup  controllerKind Secret: Starting Controller
W1221 16:53:54.314512       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Certificate: certificates.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "certificates" in API group "cert-manager.io" at the cluster scope
E1221 16:53:54.314729       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Certificate: failed to list *v1.Certificate: certificates.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "certificates" in API group "cert-manager.io" at the cluster scope
W1221 16:53:54.315073       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
E1221 16:53:54.315222       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
W1221 16:53:54.315358       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
E1221 16:53:54.315506       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Issuer: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
W1221 16:53:55.237477       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
E1221 16:53:55.237513       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
W1221 16:53:55.352838       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
E1221 16:53:55.352886       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Issuer: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
W1221 16:53:55.502413       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Certificate: certificates.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "certificates" in API group "cert-manager.io" at the cluster scope
E1221 16:53:55.502475       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Certificate: failed to list *v1.Certificate: certificates.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "certificates" in API group "cert-manager.io" at the cluster scope
W1221 16:53:56.952989       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
E1221 16:53:56.953276       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
W1221 16:53:57.169682       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
E1221 16:53:57.169723       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Issuer: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
W1221 16:53:57.642061       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Certificate: certificates.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "certificates" in API group "cert-manager.io" at the cluster scope
E1221 16:53:57.642127       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Certificate: failed to list *v1.Certificate: certificates.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "certificates" in API group "cert-manager.io" at the cluster scope
W1221 16:54:01.016241       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
E1221 16:54:01.016575       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Issuer: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
W1221 16:54:01.346556       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
E1221 16:54:01.346599       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
W1221 16:54:03.834723       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Certificate: certificates.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "certificates" in API group "cert-manager.io" at the cluster scope
E1221 16:54:03.835195       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Certificate: failed to list *v1.Certificate: certificates.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "certificates" in API group "cert-manager.io" at the cluster scope
W1221 16:54:10.455922       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
E1221 16:54:10.456534       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Issuer: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
W1221 16:54:11.039885       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
E1221 16:54:11.040501       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
W1221 16:54:12.496913       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Certificate: certificates.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "certificates" in API group "cert-manager.io" at the cluster scope
E1221 16:54:12.497322       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Certificate: failed to list *v1.Certificate: certificates.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "certificates" in API group "cert-manager.io" at the cluster scope
W1221 16:54:26.682542       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
E1221 16:54:26.682691       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
W1221 16:54:27.680297       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
E1221 16:54:27.680653       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Issuer: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
W1221 16:54:31.015092       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Certificate: certificates.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "certificates" in API group "cert-manager.io" at the cluster scope
E1221 16:54:31.015508       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Certificate: failed to list *v1.Certificate: certificates.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "certificates" in API group "cert-manager.io" at the cluster scope
W1221 16:54:53.777878       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
E1221 16:54:53.778049       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Issuer: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
W1221 16:55:06.997712       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
E1221 16:55:06.998159       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
W1221 16:55:20.182484       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Certificate: certificates.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "certificates" in API group "cert-manager.io" at the cluster scope
E1221 16:55:20.182677       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Certificate: failed to list *v1.Certificate: certificates.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "certificates" in API group "cert-manager.io" at the cluster scope
W1221 16:55:45.340782       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
E1221 16:55:45.340864       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
W1221 16:55:49.019376       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
E1221 16:55:49.019421       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Issuer: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope


E1221 16:55:54.309581       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: Could not wait for Cache to sync failed to wait for certificate caches to sync: timed out waiting for cache to be synced
I1221 16:55:54.309653       1 deleg.go:121] Stopping and waiting for non leader election runnables
I1221 16:55:54.309665       1 deleg.go:121] Stopping and waiting for leader election runnables
E1221 16:55:54.309888       1 deleg.go:135] controller-runtime: source: failed to get informer from cache Timeout: failed waiting for *v1.Certificate Informer to sync
E1221 16:55:54.309923       1 deleg.go:135] controller-runtime: source: failed to get informer from cache Timeout: failed waiting for *v1.Issuer Informer to sync
E1221 16:55:54.309940       1 deleg.go:135] controller-runtime: source: failed to get informer from cache Timeout: failed waiting for *v1.Secret Informer to sync
I1221 16:55:54.310017       1 logr.go:249] controller secret controllerGroup  controllerKind Secret: Shutdown signal received, waiting for all workers to finish
I1221 16:55:54.310028       1 logr.go:249] controller secret controllerGroup  controllerKind Secret: All workers finished
I1221 16:55:54.310352       1 logr.go:249] controller secret controllerGroup  controllerKind Secret: Starting workers worker count 1
I1221 16:55:54.310467       1 logr.go:249] controller secret controllerGroup  controllerKind Secret: Shutdown signal received, waiting for all workers to finish
E1221 16:55:54.310888       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"k3s-serving", Namespace:"kube-system"} namespace kube-system name k3s-serving: reconcileID "7579cd32-a8fe-4102-af6e-70f69a8e1728": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.311174       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"sh.helm.release.v1.my-namespace.v67", Namespace:"my-namespace"} namespace my-namespace name sh.helm.release.v1.my-namespace.v67: reconcileID "7bbb5ed4-439e-4adb-a334-53da4629b29c": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.311426       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"sh.helm.release.v1.my-namespace.v68", Namespace:"my-namespace"} namespace my-namespace name sh.helm.release.v1.my-namespace.v68: reconcileID "14f25fc7-a0f2-4a42-876e-5cb718ddebef": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.311591       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"basic-auth", Namespace:"my-namespace"} namespace my-namespace name basic-auth: reconcileID "494fbdea-9885-4dfc-8bbd-1e13b9b16683": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.311779       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"my-namespace-0.node-password.k3s", Namespace:"kube-system"} namespace kube-system name my-namespace-0.node-password.k3s: reconcileID "b349391a-bf97-4f4a-a145-63c7bb64faa2": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.312196       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"quiz-devopscycle-com-tls", Namespace:"my-namespace"} namespace my-namespace name quiz-devopscycle-com-tls: reconcileID "defec9a0-61ae-4d7c-b3e8-0fac2c0a5295": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.312298       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"sh.helm.release.v1.my-namespace.v70", Namespace:"my-namespace"} namespace my-namespace name sh.helm.release.v1.my-namespace.v70: reconcileID "7d1fa0e5-83f5-4fa0-a635-b941e1451696": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.312524       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"sh.helm.release.v1.my-namespace.v71", Namespace:"my-namespace"} namespace my-namespace name sh.helm.release.v1.my-namespace.v71: reconcileID "5af977d6-ad06-431c-ad64-2029e63d0b22": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.312789       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"sh.helm.release.v1.my-namespace.v73", Namespace:"my-namespace"} namespace my-namespace name sh.helm.release.v1.my-namespace.v73: reconcileID "2141b103-7994-4b30-8280-88fdd0cdfc74": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.313070       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"sh.helm.release.v1.my-namespace.v69", Namespace:"my-namespace"} namespace my-namespace name sh.helm.release.v1.my-namespace.v69: reconcileID "b87876d0-9385-4b9b-93e1-3d31efa1ec6c": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.313278       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"api-secrets-my-namespace", Namespace:"my-namespace"} namespace my-namespace name api-secrets-my-namespace: reconcileID "3a783b87-774d-43fd-9f6f-d7e56608ee10": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.313497       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"regcred", Namespace:"my-namespace"} namespace my-namespace name regcred: reconcileID "06afaf48-815f-47f2-870a-0712676c6b4c": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.313680       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"my-namespace-tls", Namespace:"my-namespace"} namespace my-namespace name my-namespace-tls: reconcileID "09fe2b41-0be8-476a-95b2-c49ae470859d": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.313815       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"sh.helm.release.v1.my-namespace.v74", Namespace:"my-namespace"} namespace my-namespace name sh.helm.release.v1.my-namespace.v74: reconcileID "2cbfb54f-ddc3-4434-9a35-af7c6480b671": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.314092       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"sh.helm.release.v1.my-namespace.v66", Namespace:"my-namespace"} namespace my-namespace name sh.helm.release.v1.my-namespace.v66: reconcileID "7d8a7110-bd5f-42c9-90df-4dd2817af1b9": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.314333       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"sh.helm.release.v1.my-namespace.v75", Namespace:"my-namespace"} namespace my-namespace name sh.helm.release.v1.my-namespace.v75: reconcileID "e51e8f75-73cf-4920-afa3-84f33b1f29cf": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.314462       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"sh.helm.release.v1.my-namespace.v72", Namespace:"my-namespace"} namespace my-namespace name sh.helm.release.v1.my-namespace.v72: reconcileID "eabfdc67-c950-4179-a455-4c08b3471bc6": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.314594       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"quiz-redirect-devopsberatung-at-tls", Namespace:"my-namespace"} namespace my-namespace name quiz-redirect-devopsberatung-at-tls: reconcileID "9de7bcad-2fd9-479b-ad28-c6af0d9c3898": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
I1221 16:55:54.314614       1 logr.go:249] controller secret controllerGroup  controllerKind Secret: All workers finished
I1221 16:55:54.314679       1 deleg.go:121] Stopping and waiting for caches
I1221 16:55:54.314992       1 deleg.go:121] Stopping and waiting for webhooks
I1221 16:55:54.315115       1 deleg.go:121] Wait completed, proceeding to shutdown the manager
panic: failed to wait for certificate caches to sync: timed out waiting for cache to be synced

goroutine 286 [running]:
github.com/loft-sh/vcluster-sdk/plugin.(*manager).start.func4()
	/go/vcluster/vendor/github.com/loft-sh/vcluster-sdk/plugin/plugin.go:569 +0x4e
created by github.com/loft-sh/vcluster-sdk/plugin.(*manager).start
	/go/vcluster/vendor/github.com/loft-sh/vcluster-sdk/plugin/plugin.go:566 +0x54f

In the meantime the logs from the syncer sidecar:

2023-12-21 16:53:54	INFO	plugin/plugin.go:225	Registering plugin cert-manager-plugin	{"component": "vcluster"}
2023-12-21 16:53:54	INFO	plugin/plugin.go:304	Register client hook for networking.k8s.io/v1 Ingress in plugin cert-manager-plugin	{"component": "vcluster"}
2023-12-21 16:53:54	INFO	loghelper/klog.go:24	Stopped tunnel to 127.0.0.1:6443	{"component": "vcluster", "component": "k3s", "time": "2023-12-21T16:53:54Z", "level": "info"}
2023-12-21 16:53:54	INFO	loghelper/klog.go:24	Connecting to proxy	{"component": "vcluster", "component": "k3s", "time": "2023-12-21T16:53:54Z", "level": "info", "url": "wss://10.244.0.81:8443/v1-k3s/connect"}
2023-12-21 16:53:54	INFO	loghelper/klog.go:24	Proxy done	{"component": "vcluster", "component": "k3s", "time": "2023-12-21T16:53:54Z", "level": "info", "err": "context canceled", "url": "wss://127.0.0.1:6443/v1-k3s/connect"}
2023-12-21 16:53:54	INFO	loghelper/klog.go:24	error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF	{"component": "vcluster", "component": "k3s", "time": "2023-12-21T16:53:54Z", "level": "info"}
2023-12-21 16:53:54	INFO	loghelper/klog.go:24	Handling backend connection request [my-namespace-0]	{"component": "vcluster", "component": "k3s", "time": "2023-12-21T16:53:54Z", "level": "info"}
2023-12-21 16:54:06	INFO	loghelper/klog.go:24	COMPACT compactRev=110215 targetCompactRev=110246 currentRev=111246	{"component": "vcluster", "component": "k3s", "time": "2023-12-21T16:54:06Z", "level": "info"}
2023-12-21 16:54:06	INFO	loghelper/klog.go:24	COMPACT deleted 31 rows from 31 revisions in 2.931345ms - compacted to 110246/111246	{"component": "vcluster", "component": "k3s", "time": "2023-12-21T16:54:06Z", "level": "info"}
2023-12-21 16:54:37	ERROR	filters/wrap.go:54	timeout or abort while handling: method=GET URI="/api/v1/namespaces?allowWatchBookmarks=true&resourceVersion=111059&timeout=7m46s&timeoutSeconds=466&watch=true" audit-ID="1d12f7af-6add-4be9-82dc-934dc86bf880"	{"component": "vcluster"}
2023-12-21 16:54:54	ERROR	filters/wrap.go:54	timeout or abort while handling: method=GET URI="/v1-k3s/connect" audit-ID="25f381b0-eba5-46ae-9403-31d57bb73929"	{"component": "vcluster"}
2023-12-21 16:55:55	INFO	loghelper/klog.go:24	Stopped tunnel to 10.244.0.81:8443	{"component": "vcluster", "component": "k3s", "time": "2023-12-21T16:55:55Z", "level": "info"}
2023-12-21 16:55:55	INFO	loghelper/klog.go:24	Connecting to proxy	{"component": "vcluster", "component": "k3s", "time": "2023-12-21T16:55:55Z", "level": "info", "url": "wss://127.0.0.1:6443/v1-k3s/connect"}
2023-12-21 16:55:55	INFO	loghelper/klog.go:24	Proxy done	{"component": "vcluster", "component": "k3s", "time": "2023-12-21T16:55:55Z", "level": "info", "err": "context canceled", "url": "wss://10.244.0.81:8443/v1-k3s/connect"}
2023/12/21 16:55:55 websocketproxy: Error when copying from client to backend: websocket: close 1006 (abnormal closure): unexpected EOF
2023-12-21 16:55:55	INFO	loghelper/klog.go:24	Handling backend connection request [my-namespace-0]	{"component": "vcluster", "component": "k3s", "time": "2023-12-21T16:55:55Z", "level": "info"}
2023-12-21 16:55:55	ERROR	filters/timeout.go:142	post-timeout activity - time-elapsed: 1m0.689537949s, GET "/v1-k3s/connect" result: <nil>	{"component": "vcluster"}
2023-12-21 16:55:55	INFO	loghelper/klog.go:24	error in remotedialer server [400]: websocket: bad close code 1006	{"component": "vcluster", "component": "k3s", "time": "2023-12-21T16:55:55Z", "level": "info"}

I would be more than happy to share more information if needed.
@FabianKramm
Copy link
Member

@JPeer264 thanks for creating this issue! We will take a look

@trondvindenes-hvikt
Copy link

I'm experiencing the same issue

1 similar comment
@rarg27
Copy link

rarg27 commented Jan 29, 2024

I'm experiencing the same issue

@Perchework
Copy link

Perchework commented Jan 31, 2024 •
edited
Loading

Same here.
cert-manager 1.11
vcluster 0.18.1
kubernetes 1.26.9

Working with same configuration on vcluster 0.15.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@Perchework @FabianKramm @JPeer264 @rarg27 @trondvindenes-hvikt