[REGRESSION]: Failed to install VM custom script extension on VMSS. Error: VM has reported a failure when processing extension 'AzureVmssDeploymentTask' (publisher 'Microsoft.Compute' and type 'CustomScriptExtension')

[intikhabalam]

New issue checklist

• I searched for existing GitHub issues
• I read pipeline troubleshooting guide
• I checked how to collect logs

Task name

AzureVmssDeploymentV0

Breaking task version

v230

Last working task version

v229

Regression Description

I am reaching out to address a critical issue we've encountered with the Azure VM Scale Set (VMSS) deployment, specifically using the AzureVmssDeploymentV0 task in Azure DevOps (ADO). This task had been functioning smoothly until recently, with our last successful run recorded on November 1, 2023. However, post-November 3, 2023, we've started facing challenges, which appear to coincide with recent changes in the git repository (https://github.com/microsoft/azure-pipelines-tasks/commits/master/Tasks/AzureVmssDeploymentV0 ).

The primary issue manifests as a duplicated file name segment in URLs during the cs.zip download process by the task. Here's a detailed breakdown of the error we're encountering:

2023-11-09T19:02:34.3068543Z ##[error]Failed to install VM custom script extension on VMSS. Error: VM has reported a failure when processing extension 'AzureVmssDeploymentTask' (publisher 'Microsoft.Compute' and type 'CustomScriptExtension'). Error message: "Failed to download all specified files. Exiting. Error Message: CustomScript failed to download the blob https://stedgesoftware.blob.core.windows.net/vststasks/2717-1699556493949/25119/4//cs.zip/vststasks/2717-1699556493949/25119/4//cs.zip because it does not exist. Please create the blob and try again, the response code and message returned were: "The remote server returned an error: (404) Not Found. The specified blob does not exist." (Service request ID = cdf188e8-b01e-003e-273f-13b646000000)". More information on troubleshooting is available at https://aka.ms/VMExtensionCSEWindowsTroubleshoot. .

The most recent successful deploy was on Wednesday and we didn’t make any configuration changes to the release pipeline since then (according to the history the last change was June).

I’ve tried a few things to get it going:

1. I created a new storage account stedgesoftware2 and configured the release pipeline to use that. This was a workaround for a previous, slightly different error that Mike helped us fix a while ago where there was a file lock. However same result. I’ve changed the pipeline configuration back to use the old storage account.
2. Redeploying the previous working successful release, which now fails. This is to rule out it being something to do with the build artifacts.
3. Turned the stage environment off and back on again.
4. Checked resource health of stedgesoftware resource

I’ve used Azure storage explorer to verify that the files are actually present, so the error message is wrong. However there is something weird further up in the logs:

Previously working release:

Failing release:

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

Azure DevOps Server

Operation system

Windows

Relevant log output

2023-11-09T19:02:34.3068543Z ##[error]Failed to install VM custom script extension on VMSS. Error: VM has reported a failure when processing extension 'AzureVmssDeploymentTask' (publisher 'Microsoft.Compute' and type 'CustomScriptExtension'). Error message: "Failed to download all specified files. Exiting. Error Message: CustomScript failed to download the blob https://stedgesoftware.blob.core.windows.net/vststasks/2717-1699556493949/25119/4//cs.zip/vststasks/2717-1699556493949/25119/4//cs.zip because it does not exist. Please create the blob and try again, the response code and message returned were: "The remote server returned an error: (404) Not Found. The specified blob does not exist." (Service request ID = cdf188e8-b01e-003e-273f-13b646000000)". More information on troubleshooting is available at https://aka.ms/VMExtensionCSEWindowsTroubleshoot. .

Full task logs with system.debug enabled

UNSUCCESSFUL RUN

 [REPLACE THIS WITH YOUR INFORMATION] 

SUCCESSFUL RUN

 [REPLACE THIS WITH YOUR INFORMATION] 

Repro steps

No response

[max-zaytsev]

@intikhabalam thanks for reporting this!
The issue should be mitigated now. Can you please start a new pipeline and check that everything is fine?

[github-actions]

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days