Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 Panic in controller during guest customization #177

Open
srm09 opened this issue Jul 12, 2023 · 1 comment
Open

🐛 Panic in controller during guest customization #177

srm09 opened this issue Jul 12, 2023 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@srm09
Copy link
Contributor

srm09 commented Jul 12, 2023

What steps did you take and what happened?
This was introduced after the PR #167 was merged.

What did you expect to happen?
No controller panic when running guest os customization.

Is there anything else you would like to add?
Error stack trace observed as below:

I0712 10:39:08.886378       1 controller.go:118]  "msg"="Observed a panic in reconciler: interface conversion: error is *session.ErrWithReason, not session.ErrWithReason" "VirtualMachine"={"name":"vm-w89h","namespace":"vmsvc-e2e-wkwzz9"} "controller"="virtualmachine" "controllerGroup"="vmoperator.vmware.com" "controllerKind"="VirtualMachine" "name"="vm-w89h" "namespace"="vmsvc-e2e-wkwzz9" "reconcileID"="276a9abc-852f-40a5-86b0-333ee297d49d"
panic: interface conversion: error is *session.ErrWithReason, not session.ErrWithReason [recovered]
panic: interface conversion: error is *session.ErrWithReason, not session.ErrWithReason [recovered]
        panic: interface conversion: error is *session.ErrWithReason, not session.ErrWithReason
panic: interface conversion: error is *session.ErrWithReason, not session.ErrWithReason [recovered]
        panic: interface conversion: error is *session.ErrWithReason, not session.ErrWithReason
        panic: interface conversion: error is *session.ErrWithReason, not session.ErrWithReason

goroutine 2160 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
        /tmp/runner/home/mts/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119 +0x1fa
panic({0x1ea5700, 0xc00085db00})
        /tmp/runner/custom-build/linux64/src/runtime/panic.go:884 +0x213
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/session.customizeCloudInit({{0x25e00f8, 0xc001881d70}, {{0x25e3628, 0xc00151a5f0}, 0x0}, 0xc000ede600}, 0x0?, 0x0?, {0xc0011a4580, 0x0, ...})
        /tmp/runner/cayman_vm-operator/vm-operator/src/pkg/vmprovider/providers/vsphere/session/session_vm_customization.go:304 +0x254

github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/session.(*Session).customize(0x229bc9f?, {{0x25e00f8, 0xc001881d70}, {{0x25e3628, 0xc00151a5f0}, 0x0}, 0xc000ede600}, 0x229bc9f?, 0xc0009a4c00, {0xc0011a4580, ...})
        /tmp/runner/cayman_vm-operator/vm-operator/src/pkg/vmprovider/providers/vsphere/session/session_vm_customization.go:332 +0x44e
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/session.(*Session).prepareVMForPowerOn(0xc0016e9310, {{0x25e00f8, 0xc001881d70}, {{0x25e3628, 0xc00151a5f0}, 0x0}, 0xc000ede600}, 0x40dc2a?, 0xc0006cf100?, 0xc0010f8820)
goroutine 2160 [running]:
        /tmp/runner/cayman_vm-operator/vm-operator/src/pkg/vmprovider/providers/vsphere/session/session_vm_update.go:685 +0x296
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/session.(*Session).UpdateVirtualMachine(0xc0016e9310, {{0x25e00f8, 0xc001881d70}, {{0x25e3628, 0xc00151a5f0}, 0x0}, 0xc000ede600}, 0xc00037ad20, 0xc0016e9348)
        /tmp/runner/cayman_vm-operator/vm-operator/src/pkg/vmprovider/providers/vsphere/session/session_vm_update.go:918 +0xa25
        /tmp/runner/home/mts/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119 +0x1fa
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere.(*vSphereVMProvider).updateVirtualMachine(0xc000a3c640, {{0x25e00f8, 0xc001881d70}, {{0x25e3628, 0xc00151a5f0}, 0x0}, 0xc000ede600}, 0xc00002d3b0?, 0xc00002d3b0)
panic({0x1ea5700, 0xc00085db00})
        /tmp/runner/cayman_vm-operator/vm-operator/src/pkg/vmprovider/providers/vsphere/vmprovider_vm.go:317 +0x1f4
        /tmp/runner/custom-build/linux64/src/runtime/panic.go:884 +0x213
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere.(*vSphereVMProvider).CreateOrUpdateVirtualMachine(0xc000ede7f8?, {0x25e03d0, 0xc001881530}, 0xc000ede600)
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/session.customizeCloudInit({{0x25e00f8, 0xc001881d70}, {{0x25e3628, 0xc00151a5f0}, 0x0}, 0xc000ede600}, 0x0?, 0x0?, {0xc0011a4580, 0x0, ...})
        /tmp/runner/cayman_vm-operator/vm-operator/src/pkg/vmprovider/providers/vsphere/vmprovider_vm.go:86 +0x36a
        /tmp/runner/cayman_vm-operator/vm-operator/src/pkg/vmprovider/providers/vsphere/session/session_vm_customization.go:304 +0x254
github.com/vmware-tanzu/vm-operator/controllers/virtualmachine.(*Reconciler).ReconcileNormal(0xc0001f0d20, 0xc001881530)
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/session.(*Session).customize(0x229bc9f?, {{0x25e00f8, 0xc001881d70}, {{0x25e3628, 0xc00151a5f0}, 0x0}, 0xc000ede600}, 0x229bc9f?, 0xc0009a4c00, {0xc0011a4580, ...})
        /tmp/runner/cayman_vm-operator/vm-operator/src/controllers/virtualmachine/virtualmachine_controller.go:478 +0x232
        /tmp/runner/cayman_vm-operator/vm-operator/src/pkg/vmprovider/providers/vsphere/session/session_vm_customization.go:332 +0x44e
github.com/vmware-tanzu/vm-operator/controllers/virtualmachine.(*Reconciler).Reconcile(0xc0001f0d20, {0x25e00f8, 0xc001881470}, {{{0xc0019ea600?, 0x0?}, {0xc0019ea5f0?, 0x40e007?}}})
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/session.(*Session).prepareVMForPowerOn(0xc0016e9310, {{0x25e00f8, 0xc001881d70}, {{0x25e3628, 0xc00151a5f0}, 0x0}, 0xc000ede600}, 0x40dc2a?, 0xc0006cf100?, 0xc0010f8820)
        /tmp/runner/cayman_vm-operator/vm-operator/src/controllers/virtualmachine/virtualmachine_controller.go:390 +0x566
        /tmp/runner/cayman_vm-operator/vm-operator/src/pkg/vmprovider/providers/vsphere/session/session_vm_update.go:685 +0x296
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x25e00f8?, {0x25e00f8?, 0xc001881470?}, {{{0xc0019ea600?, 0x1d79380?}, {0xc0019ea5f0?, 0x0?}}})
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/session.(*Session).UpdateVirtualMachine(0xc0016e9310, {{0x25e00f8, 0xc001881d70}, {{0x25e3628, 0xc00151a5f0}, 0x0}, 0xc000ede600}, 0xc00037ad20, 0xc0016e9348)
        /tmp/runner/home/mts/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:122 +0xc8
        /tmp/runner/cayman_vm-operator/vm-operator/src/pkg/vmprovider/providers/vsphere/session/session_vm_update.go:918 +0xa25
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0001cf9a0, {0x25e0050, 0xc000a3c4b0}, {0x1f947c0?, 0xc0014a43a0?})
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere.(*vSphereVMProvider).updateVirtualMachine(0xc000a3c640, {{0x25e00f8, 0xc001881d70}, {{0x25e3628, 0xc00151a5f0}, 0x0}, 0xc000ede600}, 0xc00002d3b0?, 0xc00002d3b0)
        /tmp/runner/home/mts/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:323 +0x35f
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0001cf9a0, {0x25e0050, 0xc000a3c4b0})
        /tmp/runner/home/mts/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274 +0x1d9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
        /tmp/runner/home/mts/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235 +0x85
        /tmp/runner/cayman_vm-operator/vm-operator/src/pkg/vmprovider/providers/vsphere/vmprovider_vm.go:317 +0x1f4
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere.(*vSphereVMProvider).CreateOrUpdateVirtualMachine(0xc000ede7f8?, {0x25e03d0, 0xc001881530}, 0xc000ede600)
        /tmp/runner/home/mts/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:231 +0x587
        /tmp/runner/cayman_vm-operator/vm-operator/src/pkg/vmprovider/providers/vsphere/vmprovider_vm.go:86 +0x36a
github.com/vmware-tanzu/vm-operator/controllers/virtualmachine.(*Reconciler).ReconcileNormal(0xc0001f0d20, 0xc001881530)
        /tmp/runner/cayman_vm-operator/vm-operator/src/controllers/virtualmachine/virtualmachine_controller.go:478 +0x232
github.com/vmware-tanzu/vm-operator/controllers/virtualmachine.(*Reconciler).Reconcile(0xc0001f0d20, {0x25e00f8, 0xc001881470}, {{{0xc0019ea600?, 0x0?}, {0xc0019ea5f0?, 0x40e007?}}})
        /tmp/runner/cayman_vm-operator/vm-operator/src/controllers/virtualmachine/virtualmachine_controller.go:390 +0x566
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x25e00f8?, {0x25e00f8?, 0xc001881470?}, {{{0xc0019ea600?, 0x1d79380?}, {0xc0019ea5f0?, 0x0?}}})
        /tmp/runner/home/mts/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:122 +0xc8
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0001cf9a0, {0x25e0050, 0xc000a3c4b0}, {0x1f947c0?, 0xc0014a43a0?})
        /tmp/runner/home/mts/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:323 +0x35f
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0001cf9a0, {0x25e0050, 0xc000a3c4b0})
        /tmp/runner/home/mts/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274 +0x1d9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
        /tmp/runner/home/mts/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
        /tmp/runner/home/mts/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:231 +0x587

Please tell us about your environment.

Value How to Obtain
Supervisor version rpm -qa VMware-wcp on the vCenter appliance
Supervisor node image version rpm -qa VMware-wcpovf on the vCenter appliance
Kubernetes version kubectl version
VM Operator version
kubectl -n vmware-system-vmop get pods \
  -ojsonpath='{range .items[*].spec.containers[*]}{.image}{"\n"}{end}' | \
  sort -u | \
  grep vmop | \
  awk -F'/' '{print $3}' | \
  awk -F: '{print $2}'
@srm09
Copy link
Contributor Author

srm09 commented Jul 12, 2023

Found the offending line, added comment https://github.com/vmware-tanzu/vm-operator/pull/167/files#r1261644348

@srm09 srm09 self-assigned this Jul 12, 2023
@srm09 srm09 added the bug Something isn't working label Jul 12, 2023
@akutz akutz changed the title Panic in controller during guest customization 🐛 Panic in controller during guest customization Jul 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant