"The entity was not found in this Azure location" error in cross region snapshot test

[andyzhangx]

Is your feature request related to a problem?/Why is this needed

Describe the solution you'd like in detail

https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_azuredisk-csi-driver/2461/pull-azuredisk-csi-driver-e2e-capz/1823978380622565376/build-log.txt

I0815 08:48:00.102231       1 utils.go:105] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0815 08:48:00.102598       1 utils.go:106] GRPC request: {"node_id":"capz-s0guyf-mp-0000001","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-35901f27-853e-433b-b4bc-165e190d30fb","csi.storage.k8s.io/pvc/name":"pvc-6rg5h","csi.storage.k8s.io/pvc/namespace":"azuredisk-9514","location":"westus2","requestedsizegib":"100","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1723707633890-4625-disk.csi.azure.com"},"volume_id":"/subscriptions/46678f10-4bbb-447e-98e8-d2829589f2d8/resourceGroups/capz-s0guyf/providers/Microsoft.Compute/disks/pvc-35901f27-853e-433b-b4bc-165e190d30fb"}
I0815 08:48:00.697044       1 azure_vmss_cache.go:335] refresh the cache of NonVmssUniformNodesCache in rg &{map[capz-s0guyf:{}]}
I0815 08:48:00.928396       1 controllerserver.go:506] GetDiskLun returned: cannot find Lun for disk pvc-35901f27-853e-433b-b4bc-165e190d30fb. Initiating attaching volume /subscriptions/46678f10-4bbb-447e-98e8-d2829589f2d8/resourceGroups/capz-s0guyf/providers/Microsoft.Compute/disks/pvc-35901f27-853e-433b-b4bc-165e190d30fb to node capz-s0guyf-mp-0000001 (vmState Succeeded).
I0815 08:48:00.937121       1 azuredisk.go:572] volumeAttachments count: 1, nodeName: capz-s0guyf-mp-0000001
I0815 08:48:00.937166       1 controllerserver.go:532] Trying to attach volume /subscriptions/46678f10-4bbb-447e-98e8-d2829589f2d8/resourceGroups/capz-s0guyf/providers/Microsoft.Compute/disks/pvc-35901f27-853e-433b-b4bc-165e190d30fb to node capz-s0guyf-mp-0000001
I0815 08:48:00.937197       1 azure_controller_common.go:201] wait 5000ms for more requests on node capz-s0guyf-mp-0000001, current disk attach: /subscriptions/46678f10-4bbb-447e-98e8-d2829589f2d8/resourceGroups/capz-s0guyf/providers/Microsoft.Compute/disks/pvc-35901f27-853e-433b-b4bc-165e190d30fb
I0815 08:48:05.937846       1 azure_controller_common.go:215] Trying to attach volume /subscriptions/46678f10-4bbb-447e-98e8-d2829589f2d8/resourceGroups/capz-s0guyf/providers/Microsoft.Compute/disks/pvc-35901f27-853e-433b-b4bc-165e190d30fb lun 0 to node capz-s0guyf-mp-0000001, diskMap len:1, map[/subscriptions/46678f10-4bbb-447e-98e8-d2829589f2d8/resourcegroups/capz-s0guyf/providers/microsoft.compute/disks/pvc-35901f27-853e-433b-b4bc-165e190d30fb:0xc000cf4940]
I0815 08:48:05.937925       1 azure_controller_vmss.go:111] azureDisk - update: rg(capz-s0guyf) vm(capz-s0guyf-mp-0000001) - attach disk list(map[/subscriptions/46678f10-4bbb-447e-98e8-d2829589f2d8/resourcegroups/capz-s0guyf/providers/microsoft.compute/disks/pvc-35901f27-853e-433b-b4bc-165e190d30fb:0xc000cf4940])
I0815 08:48:06.072979       1 azure_armclient.go:302] Received error in sendAsync.send: resourceID: http://localhost:7788/subscriptions/46678f10-4bbb-447e-98e8-d2829589f2d8/resourceGroups/capz-s0guyf/providers/Microsoft.Compute/virtualMachineScaleSets/capz-s0guyf-mp-0/virtualMachines/1?api-version=2022-03-01, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {
  "error": {
    "code": "NotFound",
    "message": "The entity was not found in this Azure location.",
    "target": "/subscriptions/46678f10-4bbb-447e-98e8-d2829589f2d8/resourcegroups/capz-s0guyf/providers/microsoft.compute/disks/pvc-35901f27-853e-433b-b4bc-165e190d30fb"
  }
}
I0815 08:48:06.073019       1 azure_armclient.go:549] Received error in put.send: resourceID: /subscriptions/46678f10-4bbb-447e-98e8-d2829589f2d8/resourceGroups/capz-s0guyf/providers/Microsoft.Compute/virtualMachineScaleSets/capz-s0guyf-mp-0/virtualMachines/1, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {
  "error": {
    "code": "NotFound",
    "message": "The entity was not found in this Azure location.",
    "target": "/subscriptions/46678f10-4bbb-447e-98e8-d2829589f2d8/resourcegroups/capz-s0guyf/providers/microsoft.compute/disks/pvc-35901f27-853e-433b-b4bc-165e190d30fb"
  }
}

Describe alternatives you've considered

Additional context

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten