Adding Support for Single Template with Multiple Parameter Files

docs/wiki/Frequently-Asked-Questions.md

@@ -15,6 +15,9 @@ This article answers frequently asked questions relating to AzOps.
     - [**I want to discover all resources in specific resource groups in one specific subscription**](#i-want-to-discover-all-resources-in-specific-resource-groups-in-one-specific-subscription)
     - [**I want to discover a specific resource type in specific resource group in one specific subscription**](#i-want-to-discover-a-specific-resource-type-in-specific-resource-group-in-one-specific-subscription)
     - [**I want to discover and manage several Azure Firewall Policy's and rule collections spread out across several resource groups and subscriptions**](#i-want-to-discover-and-manage-several-azure-firewall-policys-and-rule-collections-spread-out-across-several-resource-groups-and-subscriptions)
+  - [Push scenarios and settings](#push-scenarios-and-settings)
+    - [**I want to have multiple different deployments at scope using the same template file but different parameter files**](#i-want-to-have-multiple-different-deployments-at-scope-using-the-same-template-file-but-different-parameter-files)
+    - [**I am getting: Missing defaultValue and no parameter file found, skip deployment**](#i-am-getting-missing-defaultvalue-and-no-parameter-file-found-skip-deployment)
 
 ## Subscriptions or resources not showing up in repository
 
@@ -145,5 +148,56 @@ Yes, ensure the following setting combinations are applied (replace `rgname1`, `
 
 Can AzOps settings be configured to enable this?
 
-Yes, ensure that the variable `AZOPS_CUSTOM_SORT_ORDER` is set to `true` and create a file named `.order` in the same folder as your template files.  
+Yes, ensure that the variable `AZOPS_CUSTOM_SORT_ORDER` is set to `true` and create a file named `.order` in the same folder as your template files.
 Template files listed in the order file will be deployed in the order specified in the file and before any other templates.
+
+## Push scenarios and settings
+
+### **I want to have multiple different deployments at scope using the same template file but different parameter files**
+
+When using custom deployment templates, can I avoid the pattern of duplicating the `.bicep` file for each `parameter` file below?
+```bash
+scope/
+├── template-a.bicep
+├── template-a.bicepparam
+├── template-b.bicep
+├── template-b.bicepparam
+├── template-c.bicep
+└── template-c.parameters.json
+```
+Yes, ensure the following setting combinations are applied (replace `x` with your specific pattern identifier)
+
+```bash
+    "Core.AllowMultipleTemplateParameterFiles": true
+
+    "Core.MultipleTemplateParameterFileSuffix": ".x"
+```
+AzOps module will evaluate each parameter file individually and try to find base template by matching (*regular expression*) according to `MultipleTemplateParameterFileSuffix` pattern identifier.
+```bash
+scope/
+├── template.x1.bicepparam
+├── template.x2.bicepparam
+├── template.x3.parameters.json
+└── template.bicep
+```
+> Note: To avoid having AzOps deploy the base `template.bicep` unintentionally, ensure you have at least one parameter without default value in `template.bicep` and no lingering 1:1 matching parameter file.
+
+### **I am getting: Missing defaultValue and no parameter file found, skip deployment**
+
+To confirm if this applies to you, check the pipeline logs for the following message:
+
+```powershell
+[Resolve-ArmFileAssociation] Template <filepath> with parameter: <missingparam>, missing defaultValue and no parameter file found, skip deployment
+```
+
+What does this mean?
+
+AzOps have detected that parameters used in the template do not have defaultValues, no 1:1 parameter file mapped and that `Core.AllowMultipleTemplateParameterFiles` is set to `true`.
+
+To avoid exiting with error or attempt to deploy the updated base template unintentionally AzOps skips the file and logs it.
+
+The following must be true for this to happen:
+- `Core.AllowMultipleTemplateParameterFiles` is set to `true`
+- A template file is a part of the changeset sent to AzOps
+- Template file contains parameters with no defaultValue
+- Template file does not have 1:1 mapping to parameter file

docs/wiki/Settings.md

@@ -35,8 +35,10 @@ The following configuration values can be modified within the `settings.json` fi
 | 23  | State                                | Folder to store AzOpsState artefact, defaults to `root`                                                                                                                               | `"Core.State: "/root"`                                                                          |
 | 24  | SubscriptionsToIncludeResourceGroups | Filter which Subscription IDs should include Resource Groups in pull [Logic Updated in v2.0.0](https://github.com/Azure/AzOps/releases/tag/2.0.0)                                     | `"Core.SubscriptionsToIncludeResourceGroups": ["*"]`                                            |
 | 25  | TemplateParameterFileSuffix          | Default template file suffix. *Not recommended to change*                                                                                                                             | `"Core.TemplateParameterFileSuffix": ".json"`                                                   |
-| 26  | ThrottleLimit                        | Value declaring number of parallel threads. [Read more](https://github.com/azure/azops/wiki/performance-considerations)                                                               | `"Core.ThrottleLimit": 5`                                                                       |
-| 27  | WhatifExcludedChangeTypes            | Exclude specific change types from WhatIf operations                                                                                                                                  | `"Core.WhatifExcludedChangeTypes": ["NoChange","Ignore"]`                                       |
+| 26  | AllowMultipleTemplateParameterFiles          | Control multiple parameter file behaviour. *Not recommended to change*                                                                                                                             | `"Core.AllowMultipleTemplateParameterFiles": false`                                                   |
+| 27  | MultipleTemplateParameterFileSuffix          | Multiple parameter file suffix identifier. *Example mytemplate.x1.bicepparam*                                                                                                                             | `"Core.MultipleTemplateParameterFileSuffix": ".x"`                                                   |
+| 28  | ThrottleLimit                        | Value declaring number of parallel threads. [Read more](https://github.com/azure/azops/wiki/performance-considerations)                                                               | `"Core.ThrottleLimit": 5`                                                                       |
+| 29  | WhatifExcludedChangeTypes            | Exclude specific change types from WhatIf operations                                                                                                                                  | `"Core.WhatifExcludedChangeTypes": ["NoChange","Ignore"]`                                       |
 
 ## Workflow / Pipeline Settings
 

src/functions/Initialize-AzOpsEnvironment.ps1

@@ -62,6 +62,13 @@
             Stop-PSFFunction -String 'Initialize-AzOpsEnvironment.AzureContext.TooMany' -StringValues $azContextTenants.Count, ($azContextTenants -join ',') -EnableException $true -Cmdlet $PSCmdlet
         }
 
+        # Adjust MultipleTemplateParameterFileSuffix if incorrect MultipleTemplateParameterFileSuffix is set and log warning
+        if (-not $(Get-PSFConfigValue -FullName 'AzOps.Core.MultipleTemplateParameterFileSuffix').StartsWith('.')) {
+            $updateMultipleTemplateParameterFileSuffix = ".$(Get-PSFConfigValue -FullName 'AzOps.Core.MultipleTemplateParameterFileSuffix')"
+            Write-PSFMessage -Level Warning -String 'Initialize-AzOpsEnvironment.MultipleTemplateParameterFileSuffix.Adjustment' -StringValues (Get-PSFConfigValue -FullName 'AzOps.Core.MultipleTemplateParameterFileSuffix'), $updateMultipleTemplateParameterFileSuffix
+            Set-PSFConfig -Module AzOps -Name Core.MultipleTemplateParameterFileSuffix -Value $updateMultipleTemplateParameterFileSuffix
+        }
+
         # Adjust ThrottleLimit from previously default 10 to 5 if system has less than 2 cores
         [int]$cpuCores = if ($IsWindows) { $env:NUMBER_OF_PROCESSORS } else { Invoke-AzOpsNativeCommand -ScriptBlock { nproc --all } -IgnoreExitcode }
         $throttleLimit = (Get-PSFConfig -Module AzOps -Name Core.ThrottleLimit).Value

src/functions/Invoke-AzOpsPush.ps1

@@ -92,25 +92,38 @@
                 #region Directly Associated Template file exists
                 switch ($fileItem.Name) {
                     { $_.EndsWith('.parameters.json') } {
-                        $templatePath = $fileItem.FullName -replace '\.parameters.json', (Get-PSFConfigValue -FullName 'AzOps.Core.TemplateParameterFileSuffix')
-                        $bicepTemplatePath = $fileItem.FullName -replace '.parameters.json', '.bicep'
+                        if ((Get-PSFConfigValue -FullName 'AzOps.Core.AllowMultipleTemplateParameterFiles') -eq $true -and $fileItem.FullName.Split('.')[-3] -match $(Get-PSFConfigValue -FullName 'AzOps.Core.MultipleTemplateParameterFileSuffix').Replace('.','')) {
+                            Write-PSFMessage -Level Verbose @common -String 'Invoke-AzOpsPush.Resolve.MultipleTemplateParameterFile' -StringValues $FilePath
+                            $templatePath = $fileItem.FullName -replace (".$($fileItem.FullName.Split('.')[-3])"), '' -replace '\.parameters.json', '.json'
+                            $bicepTemplatePath = $fileItem.FullName -replace (".$($fileItem.FullName.Split('.')[-3])"), '' -replace '.parameters.json', '.bicep'
+                        }
+                        else {
+                            $templatePath = $fileItem.FullName -replace '\.parameters.json', (Get-PSFConfigValue -FullName 'AzOps.Core.TemplateParameterFileSuffix')
+                            $bicepTemplatePath = $fileItem.FullName -replace '.parameters.json', '.bicep'
+                        }
                         if (Test-Path $templatePath) {
                             Write-PSFMessage -Level Verbose @common -String 'Invoke-AzOpsPush.Resolve.FoundTemplate' -StringValues $FilePath, $templatePath
                             $result.TemplateFilePath = $templatePath
                             return $result
                         }
                         elseif (Test-Path $bicepTemplatePath) {
                             Write-PSFMessage -Level Verbose @common -String 'Invoke-AzOpsPush.Resolve.FoundBicepTemplate' -StringValues $FilePath, $bicepTemplatePath
-                            $transpiledTemplatePaths = ConvertFrom-AzOpsBicepTemplate -BicepTemplatePath $bicepTemplatePath
+                            $transpiledTemplatePaths = ConvertFrom-AzOpsBicepTemplate -BicepTemplatePath $bicepTemplatePath -SkipParam
                             $result.TemplateFilePath = $transpiledTemplatePaths.transpiledTemplatePath
                             return $result
                         }
                     }
                     { $_.EndsWith('.bicepparam') } {
-                        $bicepTemplatePath = $fileItem.FullName -replace '\.bicepparam', '.bicep'
+                        if ((Get-PSFConfigValue -FullName 'AzOps.Core.AllowMultipleTemplateParameterFiles') -eq $true -and $fileItem.FullName.Split('.')[-2] -match $(Get-PSFConfigValue -FullName 'AzOps.Core.MultipleTemplateParameterFileSuffix').Replace('.','')) {
+                            Write-PSFMessage -Level Verbose @common -String 'Invoke-AzOpsPush.Resolve.MultipleTemplateParameterFile' -StringValues $FilePath
+                            $bicepTemplatePath = $fileItem.FullName -replace (".$($fileItem.FullName.Split('.')[-2])"), '' -replace '\.bicepparam', '.bicep'
+                        }
+                        else {
+                            $bicepTemplatePath = $fileItem.FullName -replace '\.bicepparam', '.bicep'
+                        }
                         if (Test-Path $bicepTemplatePath) {
                             Write-PSFMessage -Level Verbose @common -String 'Invoke-AzOpsPush.Resolve.FoundBicepTemplate' -StringValues $FilePath, $bicepTemplatePath
-                            $transpiledTemplatePaths = ConvertFrom-AzOpsBicepTemplate -BicepTemplatePath $bicepTemplatePath
+                            $transpiledTemplatePaths = ConvertFrom-AzOpsBicepTemplate -BicepTemplatePath $bicepTemplatePath -BicepParamTemplatePath $fileItem.FullName
                             $result.TemplateFilePath = $transpiledTemplatePaths.transpiledTemplatePath
                             $result.TemplateParameterFilePath = $transpiledTemplatePaths.transpiledParametersPath
                             return $result
@@ -164,6 +177,17 @@
             }
             else {
                 Write-PSFMessage -Level Verbose @common -String 'Invoke-AzOpsPush.Resolve.ParameterNotFound' -StringValues $FilePath, $parameterPath
+                if ((Get-PSFConfigValue -FullName 'AzOps.Core.AllowMultipleTemplateParameterFiles') -eq $true) {
+                    # Check for template parameters without defaultValue
+                    $defaultValueContent = Get-Content $FilePath
+                    $missingDefaultParam = $defaultValueContent | jq '.parameters | with_entries(select(.value.defaultValue == null))' | ConvertFrom-Json -AsHashtable
+                    if ($missingDefaultParam.Count -ge 1) {
+                        # Skip template deployment when template parameters without defaultValue are found and no parameter file identified
+                        $missingString = foreach ($item in $missingDefaultParam.Keys.GetEnumerator()) {"$item,"}
+                        Write-PSFMessage -Level Verbose -String 'Invoke-AzOpsPush.Resolve.NotFoundParamFileDefaultValue' -StringValues $FilePath, ($missingString | Out-String -NoNewline)
+                        continue
+                    }
+                }
             }
 
             $deploymentName = $fileItem.BaseName -replace '\.json$' -replace ' ', '_'
@@ -350,8 +374,8 @@
         #Sort 'deletionList' based on 'deletionListPriority'
         $deletionList = $deletionList | Sort-Object -Property {$deletionListPriority.IndexOf($_.ScopeObject.Resource)}
 
-        #If addModifySet exists and no deploymentList has been generated at the same time as the StatePath root has additional directories, exit with terminating error
-        if (($addModifySet -and -not $deploymentList) -and (Get-ChildItem -Path $StatePath -Directory)) {
+        #If addModifySet exists and no deploymentList has been generated at the same time as the StatePath root has additional directories and AllowMultipleTemplateParameterFiles is default false, exit with terminating error
+        if (($addModifySet -and -not $deploymentList) -and (Get-ChildItem -Path $StatePath -Directory) -and ((Get-PSFConfigValue -FullName 'AzOps.Core.AllowMultipleTemplateParameterFiles') -eq $false)) {
             Write-PSFMessage -Level Critical @common -String 'Invoke-AzOpsPush.DeploymentList.NotFound'
             throw
         }

src/internal/configurations/Core.ps1

@@ -25,6 +25,8 @@ Set-PSFConfig -Module AzOps -Name Core.SkipResourceType -Value @('Microsoft.VSOn
 Set-PSFConfig -Module AzOps -Name Core.SkipRole -Value $false -Initialize -Validation bool -Description '-'
 Set-PSFConfig -Module AzOps -Name Core.State -Value (Join-Path $pwd -ChildPath "root") -Initialize -Validation string -Description 'Folder to store AzOpsState artefact'
 Set-PSFConfig -Module AzOps -Name Core.SubscriptionsToIncludeResourceGroups -Value @('*') -Initialize -Validation stringarray -Description 'Requires SkipResourceGroup to be false. Subscription ID or Display Name that matches the filter. Powershell filter that matches with like operator is supported.'
-Set-PSFConfig -Module AzOps -Name Core.TemplateParameterFileSuffix -Value '.json' -Initialize -Validation string -Description 'parameter file suffix to look for'
+Set-PSFConfig -Module AzOps -Name Core.TemplateParameterFileSuffix -Value '.json' -Initialize -Validation string -Description 'Parameter file suffix identifier'
+Set-PSFConfig -Module AzOps -Name Core.AllowMultipleTemplateParameterFiles -Value $false -Initialize -Validation string -Description 'Global flag to control multiple parameter file behaviour'
+Set-PSFConfig -Module AzOps -Name Core.MultipleTemplateParameterFileSuffix -Value '.x' -Initialize -Validation string -Description 'Multiple parameter file suffix identifier'
 Set-PSFConfig -Module AzOps -Name Core.ThrottleLimit -Value 5 -Initialize -Validation integer -Description 'Throttle limit used in Foreach-Object -Parallel for resource/subscription discovery'
 Set-PSFConfig -Module AzOps -Name Core.WhatifExcludedChangeTypes -Value @('NoChange', 'Ignore') -Initialize -Validation stringarray -Description 'Exclude specific change types from WhatIf operations.'