Validation failing after enabling "Core.SkipResource": false and "Core.AutoGeneratedTemplateFolderPath": ".az"

[rahul-ve]

Describe the bug

Encountering the below error after enabling the settings "Core.SkipResource": false and "Core.AutoGeneratedTemplateFolderPath": ".az". Tried creating a new resource group but validation fails with:

[06:52:00][Initialize-AzOpsEnvironment] AzOps environment initialization concluded
[06:52:00][Invoke-AzOpsPush] Deployment required
[06:52:00][Invoke-AzOpsPush] Adding or modifying:
[06:52:00][Invoke-AzOpsPush]   root/eslzxx (eslzxx)/eslzxx-sandboxes (eslzxx-sandboxes)/sub-it-sbox-prototype-001 (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)/rg-st-test-sbox-001/microsoft.resources_resourcegroups-rg-st-test-sbox-001.json
[06:52:00][Invoke-AzOpsPush]   settings.json
[06:52:00][Invoke-AzOpsPush] Deleting:

WARNING: [06:52:05][New-AzOpsScope] Failed to: Creating new scope from path /home/runner/work/eslzxx/eslzxx/root/eslzxx (eslzxx)/eslzxx-sandboxes (eslzxx-sandboxes)/sub-it-sbox-prototype-001 (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)/rg-st-test-sbox-001 | Cannot find path '/home/runner/work/eslzxx/eslzxx/root/eslzxx (eslzxx)/eslzxx-sandboxes (eslzxx-sandboxes)/sub-it-sbox-prototype-001 (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)/rg-st-test-sbox-001/.az' because it does not exist.

WARNING: [06:52:05][New-AzOpsScope] Failed to: Creating new scope from path /home/runner/work/eslzxx/eslzxx/root/eslzxx (eslzxx)/eslzxx-sandboxes (eslzxx-sandboxes)/sub-it-sbox-prototype-001 (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)/rg-st-test-sbox-001/microsoft.resources_resourcegroups-rg-st-test-sbox-001.json | Cannot find path '/home/runner/work/eslzxx/eslzxx/root/eslzxx (eslzxx)/eslzxx-sandboxes (eslzxx-sandboxes)/sub-it-sbox-prototype-001 (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)/rg-st-test-sbox-001/.az' because it does not exist.
[06:52:05][Invoke-AzOpsPush] Expecting deploymentList object, it was not found. Exiting runtime with error code 1.
Error: Process completed with exit code 1.

I don't fully understand the error. why is it looking for sub-it-sbox-prototype-001 (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)/rg-st-test-sbox-001/.az folder?

Steps to reproduce

1 . Used the adventureworks template and setup the IaC repo based of the AzOps accelerator template repo.

2 . Then enabled the below two settings to pull in resources that were already present:

"Core.AutoGeneratedTemplateFolderPath": ".az",
"Core.SkipResource": false,

All other settings were left with their defaults.

Pull workflow ran fine.

3 . Tried creating a resource group rg-st-test-sbox-001 under sub sub-it-sbox-prototype-001 but validation step fails with the above error.
The sub sub-it-sbox-prototype-001 was created via the Portal and pulled in, not sure if that matters.

Not sure if this is a bug or incorrect settings.
Would greatly appreciate any pointers in understanding the root cause of this issue.

Thanks
Rahul

[daltondhcp]

When changing the "Core.AutoGeneratedTemplateFolderPath" setting to for example ".az", templates for all autogenerated resources are moved/generated in the specified folder after the next AzOps-Pull run.

So in your scenario, you have two options to create a resource group;
a) Drop a custom bicep/arm template in the "sub-it-sbox-prototype-001" folder. It will then create the RG after validate/push and the rg/.az folder will be pulled back after next AzOps-Pull.
b) Do what you did above, but create the expected path .../rg-st-test-sbox-001/.az/microsoft.resources_resourcegroups-rg-st-test-sbox-001.json and it will work.

Hope this helps!

[rahul-ve]

@daltondhcp - Thanks for the clear explanation!

What tripped me up was the different ways in which we were creating resource groups, going back and checking some old deployments,

1. we sometimes just placed the resource template in a new RG folder and the RG got created without the RG template.

2. Sometimes we used to create the RG by using a template to set tags.. and the structure was;
   with the default "Core.AutoGeneratedTemplateFolderPath": "."

└── sub-it-sbox-prototype-001
    └── rg-st-test-sbox-001
        └── microsoft.resources_resourcegroups-rg-st-test-sbox-001.json

I guess we really dont need the RG folder! Placing the RG template under the sub folder would suffice, wouldn't it?
But we would have to wait until next Pull run for the folder to be populated.

Just checking my understanding, this is similar to option (b) in your comment, except the template goes in the .az folder

└── sub-it-sbox-prototype-001
    └── rg-st-test-sbox-001
        └── .az
             └── microsoft.resources_resourcegroups-rg-st-test-sbox-001.json

Once the default is changed to something like .az do all the subsequent resource templates have to go into that folder? or is it just when creating the RG?

[daltondhcp]

You absolutely got it right and it indeed sounds like the .az folder caused the confusion here. Is there anything you think we could clarify from a documentation perspective to make it more clear?

[rahul-ve]

Not sure if this trips up other users, confused me a fair bit.
May be it would help to expand on the effects of changing the setting "Core.AutoGeneratedTemplateFolderPath": ".az"

Questions like:

• What templates go into ".az" folder? Do all the resources go into that folder?
• If I remember correctly, the templates have to be named exactly, if not, pull workflow creates duplicates, adds a lot of noise.
• What happens when the setting is changed back to default ".", guessing, all new resource templates have to be outside the ".az" folder?

We will not be enabling this option in production, we prefer our dev workflow and legible commit history. large automated commits without proper commit messages makes it hard in the long run.
But for lower environments it is very valuable.

[daltondhcp]

Thank you for the feedback! Would you like to contribute and add a couple of notes about it in the Wiki/FAQ, similar to what we have done with some other settings?
https://github.com/azure/azops/wiki/frequently-asked-questions

[rahul-ve]

Happy to contribute, will submit a pr shortly