Issue with unpublishing volumes

[azkbn]

Bug Report

Description

Logs

Controller:

I0105 17:55:10.242984       1 controller.go:443] "ControllerUnpublishVolume: called" args="{\"node_id\":\"k8s-wrk-01\",\"volume_id\":\"azkbn/pve1/data/9999/vm-9999-pvc-074e9ff4-8054-4ed4-85d9-a8f1fe6871d6.raw\"}"
I0105 17:55:10.246105       1 controller.go:471] "ControllerUnpublishVolume: failed to get proxmox vmrID from ProviderID" cluster="azkbn" nodeID="k8s-wrk-01"

Environment

• Plugin version: v0.9.0
• Kubernetes version:

Client Version: v1.32.0
Kustomize Version: v5.5.0
Server Version: v1.32.0

• CSI capasity:

CLASS               AVAIL         ZONE
proxmox-zfs-ext4    164558972Ki   map[topology.kubernetes.io/region:azkbn topology.kubernetes.io/zone:pve1]
proxmox-data-xfs    223880204Ki   map[topology.kubernetes.io/region:azkbn topology.kubernetes.io/zone:pve1]
proxmox-data-ext4   223880204Ki   map[topology.kubernetes.io/region:azkbn topology.kubernetes.io/zone:pve1]
proxmox-zfs-xfs     164558984Ki   map[topology.kubernetes.io/region:azkbn topology.kubernetes.io/zone:pve1]

• CSI resource on the node:

apiVersion: v1
items:
- apiVersion: storage.k8s.io/v1
  kind: CSINode
  metadata:
    annotations:
      storage.alpha.kubernetes.io/migrated-plugins: kubernetes.io/aws-ebs,kubernetes.io/azure-disk,kubernetes.io/azure-file,kubernetes.io/cinder,kubernetes.io/gce-pd,kubernetes.io/portworx-volume,kubernetes.io/vsphere-volume
    creationTimestamp: "2025-01-04T16:27:14Z"
    name: k8s-ctrl-01
    ownerReferences:
    - apiVersion: v1
      kind: Node
      name: k8s-ctrl-01
      uid: 93d67ccf-5e08-4144-b579-2792869a5711
    resourceVersion: "183799"
    uid: dafc666d-f862-4b91-82d3-dd6fc4a89aea
  spec:
    drivers:
    - allocatable:
        count: 24
      name: csi.proxmox.sinextra.dev
      nodeID: k8s-ctrl-01
      topologyKeys:
      - topology.kubernetes.io/region
      - topology.kubernetes.io/zone
- apiVersion: storage.k8s.io/v1
  kind: CSINode
  metadata:
    annotations:
      storage.alpha.kubernetes.io/migrated-plugins: kubernetes.io/aws-ebs,kubernetes.io/azure-disk,kubernetes.io/azure-file,kubernetes.io/cinder,kubernetes.io/gce-pd,kubernetes.io/portworx-volume,kubernetes.io/vsphere-volume
    creationTimestamp: "2025-01-04T16:27:13Z"
    name: k8s-wrk-01
    ownerReferences:
    - apiVersion: v1
      kind: Node
      name: k8s-wrk-01
      uid: 216d9e4c-8d4b-4d2b-8c0f-d04a51071e58
    resourceVersion: "183825"
    uid: 203ba30a-5d43-48dd-ae71-e48fb67939d0
  spec:
    drivers:
    - allocatable:
        count: 24
      name: csi.proxmox.sinextra.dev
      nodeID: k8s-wrk-01
      topologyKeys:
      - topology.kubernetes.io/region
      - topology.kubernetes.io/zone
kind: List
metadata:
  resourceVersion: ""

• Node describe:

Name:               k8s-ctrl-01
Roles:              control-plane
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    extensions.talos.dev/intel-ucode=20241112
                    extensions.talos.dev/iscsi-tools=v0.1.6
                    extensions.talos.dev/qemu-guest-agent=9.1.2
                    k8.azkbn.home/enable-bgp=
                    k8.azkbn.home/node-type=controlplane
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=k8s-ctrl-01
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=
                    node.cloudprovider.kubernetes.io/platform=proxmox
                    node.kubernetes.io/exclude-from-external-load-balancers=
                    project.io/node-pool=control-plane
                    topology.kubernetes.io/region=azkbn
                    topology.kubernetes.io/zone=pve1

Name:               k8s-wrk-01
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    extensions.talos.dev/intel-ucode=20241112
                    extensions.talos.dev/iscsi-tools=v0.1.6
                    extensions.talos.dev/qemu-guest-agent=9.1.2
                    k8.azkbn.home/enable-bgp=
                    k8.azkbn.home/node-type=worker
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=k8s-wrk-01
                    kubernetes.io/os=linux
                    node.cloudprovider.kubernetes.io/platform=proxmox
                    project.io/node-pool=worker
                    topology.kubernetes.io/region=azkbn
                    topology.kubernetes.io/zone=pve1
Annotations:        csi.volume.kubernetes.io/nodeid: {"csi.proxmox.sinextra.dev":"k8s-wrk-01"}
                    extensions.talos.dev/schematic: 1bd38ebdc556c766ff06e7f6fa2b0e6cd0eb3c2e2db77ed8d3fc0ab2e9c9959c
                    flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"b2:47:f5:9d:37:56"}
                    flannel.alpha.coreos.com/backend-type: vxlan
                    flannel.alpha.coreos.com/kube-subnet-manager: true
                    flannel.alpha.coreos.com/public-ip: 192.168.125.211
                    node.alpha.kubernetes.io/ttl: 0
                    talos.dev/owned-annotations: ["extensions.talos.dev/schematic"]
                    talos.dev/owned-labels:
                      ["extensions.talos.dev/intel-ucode","extensions.talos.dev/iscsi-tools","extensions.talos.dev/qemu-guest-agent","k8.azkbn.home/enable-bgp",...
                    volumes.kubernetes.io/controller-managed-attach-detach: true

• OS version: Talos 1.9.0

Description

I'm getting issue with the following code:

proxmox-csi-plugin/pkg/csi/controller.go

Lines 469 to 482 in 27dd451

	vmrid, zone, err := tools.ProxmoxVMID(ctx, d.Kclient, nodeID)
	if err != nil {
	klog.InfoS("ControllerUnpublishVolume: failed to get proxmox vmrID from ProviderID", "cluster", vol.Cluster(), "nodeID", nodeID)
	vmr, err = cl.GetVmRefByName(nodeID)
	if err != nil {
	klog.ErrorS(err, "ControllerUnpublishVolume: failed to get vm ref by nodeID", "cluster", vol.Cluster(), "nodeID", nodeID)
	return nil, status.Error(codes.Internal, err.Error())
	}
	} else {
	vmr = pxapi.NewVmRef(vmrid)
	vmr.SetNode(zone)
	vmr.SetVmType("qemu")

NodeID in this case is K8S Node name (e.g. k8s-wrk-01). Later it's used to resolve K8S node

proxmox-csi-plugin/pkg/tools/nodes.go

Lines 92 to 101 in 27dd451

	func ProxmoxVMID(ctx context.Context, kclient clientkubernetes.Interface, nodeName string) (int, string, error) {
	node, err := kclient.CoreV1().Nodes().Get(ctx, nodeName, metav1.GetOptions{})
	if err != nil {
	return 0, "", fmt.Errorf("failed to get node: %v", err)
	}
	vmID, err := provider.GetVMID(node.Spec.ProviderID)
	return vmID, node.Labels[corev1.LabelTopologyZone], err
	}

But returned node spec does not have any ProviderID:

  "spec": {
    "podCIDR": "10.0.0.0/24",
    "podCIDRs": [
      "10.0.0.0/24"
    ]
  }

[azkbn]

As far as I understood, ProviderID is assigned by operator from proxmox-cloud-controller-manager. But it would be nice to handle case when there is no ProviderID assigned (for e.g. use volume_id to resolve target proxmox volume, like it's done for delete)

[sergelogvinov]

Hello,

Yep, Proxmox CSI requires the ProviderID https://github.com/sergelogvinov/proxmox-csi-plugin?tab=readme-ov-file#kubernetes-topology-labels, or you can set VM name the same as kubernetes node name (func GetVmRefByName(nodeID)).

You can set it by TalosCCM, ProxmoxCCM or kubelet args https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/docs/install.md#troubleshooting