Validate workflow "hangs" on unauthorized template deployment

[matthewponzio]

Summary

I am continuing to PoC the concept of implementing AzOps Repos with various levels of permissions. Our intention would be to recommend something like the following:

Foundation Repo

• Owner: Central Cloud Engineering Team
• Scope: Manages the Azure foundation, inclusive of the entire Management Group hierarchy and the Platform Subscriptions and all constituent Resource Groups / Resources

Landing Zones Repo(s)

• Owner: Central Cloud Engineering Team
• Scope: Manages the Landing Zone (App Workload) Subscriptions - provisioning of the Subscriptions, Policies, RBAC and any centrally managed Resource Groups / Resources, such as networking.

Workload/App Repos

• Owner: App Team
• Scope: Provision and manage Application level Resource Groups and Resources

Based on this structure, the Workload/App Repo would utilize an SPN with limited permissions. For example, it would not have permission to read/write certain network resources.

Issue

I implemented the above setup and have tested it to pretty good success, however when I attempted to test making an unauthorized change from the Workload/App Repo (a change to the virtualNetwork subnet), the Validate pipeline appeared to "hang" during the Validate Step. I let it run for about 20 minutes before I manually cancelled the Workflow. Upon cancellation, you can see in the Validate Step log that it actually worked as intended, returning the following authorization error:

WARNING: [22:43:54][New-AzOpsDeployment] Error returned from WhatIf API: 
InvalidTemplateDeployment - Long running operation failed with status 'Failed'. Additional Info:'The template deployment failed with error: 'Authorization failed for template resource 'corp1-vnet1/corp1-vnet1-subnet1' of type 'Microsoft.Network/virtualNetworks/subnets'. The client '<redacted>' with object id '<redacted' does not have permission to perform action 'Microsoft.Network/virtualNetworks/subnets/write' at scope '/subscriptions/***/resourceGroups/cloud-eng-network/providers/Microsoft.Network/virtualNetworks/corp1-vnet1/subnets/corp1-vnet1-subnet1'.'.

The issue or unexpected behavior is that the pipeline didn't fail "quickly". Admittedly, this is a fairly minor issue, but nevertheless, an authorization error should probably fail the pipeline/workflow immediately.

Expected Behavior

When a Workflow attempts to validate or execute a deployment containing an unauthorized action, the workflow fails almost immediately.

Observed Behavior

The Workflow continued to run, appearing to hang during the Validate Step. Workflow ran for 20 minutes before manual cancellation. Manually cancelling the Workflow worked and revealed the expected authorization error.

Steps to Reproduce

1. Create an AzOps Repo that uses an SPN with some kind of limited permission set
2. Open a PR with a change that would not be authorized by the limited permission set
3. See that the Validate Workflow continues run, appearing to hang
4. Manually cancel the Workflow and verify that the expected authorization was returned

Full Log from Validate Step

Prepare all required actions
Run ./.github/actions/validate-deploy
Run if [ ! -z "$(git diff --name-status HEAD^ HEAD)" ]; then
M root/corporate 1 (***)/cloud-eng-network/microsoft.network_virtualnetworks_subnets-corp1-vnet1_corp1-vnet1-subnet1.json
Run Import-PSFConfig -Path settings.json -Schema MetaJson -EnableException
[[2](https://github.com/redacted/azure-adventureworks-app1/runs/6461309427?check_suite_focus=true#step:4:2)2:43:25][Initialize-AzOpsEnvironment] Processing AzOps environment
[22:43:25][Initialize-AzOpsEnvironment] Starting AzOps environment initialization
[22:43:25][Get-AzOpsSubscription] Excluded subscription states: Disabled,Deleted,Warned,Expired
[22:43:26][Get-AzOpsSubscription] Excluded subscription offers: AzurePass_2014-09-01,FreeTrial_2014-09-01,AAD_20[15](https://github.com/redacted/azure-adventureworks-app1/runs/6461309427?check_suite_focus=true#step:4:15)-09-01
[22:43:27][Get-AzOpsSubscription] Number of subscriptions found: 1
[22:43:27][Get-AzOpsSubscription] Number of subscriptions included: 1
WARNING: [22:43:31][Initialize-AzOpsEnvironment] No management group access, discovery will happen from subscription scope(s)
[22:43:31][Initialize-AzOpsEnvironment] Processing AzOps environment
[22:43:31][Initialize-AzOpsEnvironment] Starting AzOps environment initialization
[22:43:31][Get-AzOpsSubscription] Excluded subscription states: Disabled,Deleted,Warned,Expired
[22:43:31][Get-AzOpsSubscription] Excluded subscription offers: AzurePass_2014-09-01,FreeTrial_2014-09-01,AAD_2015-09-01
[22:43:32][Get-AzOpsSubscription] Number of subscriptions found: 1
[22:43:32][Get-AzOpsSubscription] Number of subscriptions included: 1
WARNING: [22:43:36][Initialize-AzOpsEnvironment] No management group access, discovery will happen from subscription scope(s)
[22:43:36][Invoke-AzOpsPush] Deployment required
[22:43:36][Invoke-AzOpsPush] Adding or modifying:
[22:43:36][Invoke-AzOpsPush]   root/corporate 1 (***)/cloud-eng-network/microsoft.network_virtualnetworks_subnets-corp1-vnet1_corp1-vnet1-subnet1.json
[22:43:36][Invoke-AzOpsPush] Deleting:
[22:43:[37](https://github.com/redacted/azure-adventureworks-app1/runs/6461309427?check_suite_focus=true#step:4:37)][New-AzOpsDeployment] Processing deployment AzOps-microsoft.network_virtualnetworks_subnets-corp1-vnet1-B9EE for template /home/runner/work/azure-adventureworks-app1/azure-adventureworks-app1/root/corporate 1 (***)/cloud-eng-network/microsoft.network_virtualnetworks_subnets-corp1-vnet1_corp1-vnet1-subnet1.json with parameter "" in mode Incremental
Getting the latest status of all resources...
Get-AzResourceGroupDeploymentWhatIfResult:  InvalidTemplateDeployment - Long running operation failed with status
'Failed'. Additional Info:'The template deployment failed with error:
'Authorization failed for template resource 'corp1-vnet1/corp1-vnet1-subnet1'
of type 'Microsoft.Network/virtualNetworks/subnets'. The client
'[72](https://github.com/redacted/azure-adventureworks-app1/runs/6461309427?check_suite_focus=true#step:4:72)0e91c6-f516-44f1-af0f-f21e2[75](https://github.com/recacted/azure-adventureworks-app1/runs/6461309427?check_suite_focus=true#step:4:75)30c88' with object id
'redacted' does not have permission to perform
action 'Microsoft.Network/virtualNetworks/subnets/write' at scope
'/subscriptions/***/resourceGroups/cloud-eng-ne
twork/providers/Microsoft.Network/virtualNetworks/corp1-vnet1/subnets/corp1-vnet
1-subnet1'.'.'

                                             
WARNING: [22:43:54][New-AzOpsDeployment] Error returned from WhatIf API: 
InvalidTemplateDeployment - Long running operation failed with status 'Failed'. Additional Info:'The template deployment failed with error: 'Authorization failed for template resource 'corp1-vnet1/corp1-vnet1-subnet1' of type 'Microsoft.Network/virtualNetworks/subnets'. The client 'redacted' with object id 'redacted' does not have permission to perform action 'Microsoft.Network/virtualNetworks/subnets/write' at scope '/subscriptions/***/resourceGroups/cloud-eng-network/providers/Microsoft.Network/virtualNetworks/corp1-vnet1/subnets/corp1-vnet1-subnet1'.'.'
Exception:  InvalidTemplateDeployment - Long running operation failed with status
'Failed'. Additional Info:'The template deployment failed with error:
'Authorization failed for template resource 'corp1-vnet1/corp1-vnet1-subnet1'
of type 'Microsoft.Network/virtualNetworks/subnets'. The client
'redacted' with object id
'redacted' does not have permission to perform
action 'Microsoft.Network/virtualNetworks/subnets/write' at scope
'/subscriptions/***/resourceGroups/cloud-eng-ne
twork/providers/Microsoft.Network/virtualNetworks/corp1-vnet1/subnets/corp1-vnet
1-subnet1'.'.'

Error: The operation was canceled.

[daltondhcp]

Thank you for reporting. Converted it to an issue #629 so we can further triage and look at potential fix there.

[matthewponzio]

Awesome, thanks @daltondhcp!