Networking addressing documentation #5
-
|
Since you pointed me over here from the Bicep issue, I've really taken a liking to how you have all this set up. I'm building out a POC matching the idea, but I'm at a loss for how you're working out the network addressing. You briefly speak to the fact that it's a /20 address space dividing into a hub and 7 spokes, but how you've actually got it allocated for your subnets is puzzling me. I see that you set up the lower and upper ranges here, using the values from the tenant-specific values defined here and that the third octet is determined by the DeploymentID (142- (2 * DeploymentID)) where the NetworkIDUpper is simply the NetworkID with the third octet value + 1. So then to set up the networks, we iterate through the SubnetInfo values from the tenant-specific parameters and append those Prefixes to the NetworkID (unless it's the snMT02, in which case we use NetworkIDUpper). Some questions then:
snWAF01 snFE01 snAD01 snBE01 AzureBastionSubnet AzureFirewallSubnet GatewaySubnet snMT02 I guess what I'm getting at here is why you didn't just use a 10.0. address space and assign each of these subnets to a 10.0.X/24 so they'd each have 255 addresses and just swap X with the DeploymentID. Yes, you wind up with a lot of room for additional subnets that you're not using, but you get far more IP addresses for each subnet (particularly useful if you're using VMSS instead of individual VMs for the app).
I appreciate the continued help! |
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 4 replies
-
|
Sorry for the delay on responding here. I'd be happy to get you up to speed on how things work. I deploy this solution on my customer projects. Some opt for dividing the /20 into 16 * 256 addresses. However the default here is to divide that into 8 * 512. There are other options, however let's start with those defaults. 8 separate environments per region, works well in most cases. If you clone the AOA tenant then rename it... Any of the parameter files will have ALL subnets using the 512 addresses... MT02 has a dedicated 256 addresses, then the other subnets use the other 256, divided up. The P0 is a hub, so the subnets are different than all other spoke environments. E.g. Gateways etc. Anyway, we can chat some more, to go over the details. This multi tenant model works really well, plus many best practices are built in. |
Beta Was this translation helpful? Give feedback.
-
|
172.18 is just a /20 address space. Each of the tenants uses separate address space, so they don't overlap. A different tenant may use 172.17 or 172.16. Generally speaking 4096 addresses is a reasonable size, which is the /20. AKS is possibly the only resource type that may need more addresses than this. Large organizations may have 50 or 100 orgs or business units that need address space in the Cloud and need to ensure no overlap, since they likely connect via Express Route back to on-prem. /20 is a trade off to allow for this. |
Beta Was this translation helpful? Give feedback.
-
|
I am interested in maybe selecting a better default number of addresses, larger than the /20. Given App gateways have recently started requesting more address space and also for AKS. Although I am starting to use WAF/App gateway less and more FrontDoor. If you have any preferences, I would be interested in feedback. The special network address for the MT02 is because I have to set the bit in the 3rd octed to 1 for that Subnet. |
Beta Was this translation helpful? Give feedback.
-
|
The purpose behind this method for doing IP and network address allocation is to provide 2 simple capabilities. You assign all static IP addresses, simply by providing the last octet IP address
Also the Network addresses are all dynamic from the DeploymentID
So you can have any 8 environments (per region), just don't repeat any numbers. e.g. S1, D2, T3, T4, U5, Q6, P7, P8 |
Beta Was this translation helpful? Give feedback.
-
|
@WhitWaldo Not sure if you ever saw my response here, since I didn't respond straight away.
Since I have a Prefix and Location lookup table {
"AFC1": {
"displayname": "France Central",
"location": "francecentral",
"PREFIX": "AFC1"
},
"ASW1": {
"displayname": "Switzerland West",
"location": "switzerlandwest",
"PREFIX": "ASW1"
},
"AKS1": {
"displayname": "Korea South",
"location": "koreasouth",
"PREFIX": "AKS1"
},https://github.com/brwilkinson/AzureDeploymentFramework/blob/main/ADF/bicep/global/region.json {
"uksouth": {
"displayname": "UK South",
"location": "uksouth",
"PREFIX": "AUS1"
},
"westus3": {
"displayname": "West US 3",
"location": "westus3",
"PREFIX": "AWU3"
},
"germanynorth": {
"displayname": "Germany North",
"location": "germanynorth",
"PREFIX": "AGN1"
},So at the moment I do a Network Range lookup from the Global Region file in each Tenant. e.g.
{
"Global": {
"hubRG": {
"name": "P0"
// "Prefix": "ACU1"
},
"hubKV": {
"name": "VLT01"
// "Prefix": "ACU1"
},
"hubAA": { // Used for DSC State Configuration
"name": "OMSAutomation"
},
"networkId": [
"10.11.",
144
],
"DNSServers": [ // Leave Empty to use AzureDNS
"10.11.144.75",
"10.11.144.76"
],
"RTName": "Hub",
"shutdownSchedulerTimeZone": "Pacific Standard Time",
"patchSchedulerTimeZone": "America/Los_Angeles",
"AppInsightsRegion": "westus2"
}
}So I was wondering how I could remove that from there... and move it to the Region or Prefix File.
The only part of that I haven't decided on, is how to allocate between different Tenants who would need to share that IP space in the same region. |
Beta Was this translation helpful? Give feedback.
-
|
Apologies - definitely missed the earlier responses in my email. Having written out my own global per-tenant ARM template now inspired by your approach, but shifted more towards my own needs, I took a little bit of a different tack. Similar to your approach, I have an While each tenant, like yours, has a /20, each tenant uses a set of flags to identify which features are enabled or not and all Azure services with any sort of networking block then definite their own blocks from that /20. While this is usually a /24 for each, there are some services (virtual network gateway comes to mind) that fail to provision if you assign them sizable blocks, so each resource maps to an "options" array that specifies defaults for each in order to get pre-picked offsets. Of course, if some resources are disabled, there's a whole wasted subnet, but I'd rather have the consistency of knowing, say, that I can always find Application Gateway on 10.X.14.0/24 no matter the tenant I'm looking at, eliminating some cross-tenant complexity. And besides, the approach means I have 255 resources I can handle and so far I haven't used up to but 10.0.21.0/24, so I'm not worried about capping out. All that aside, your framework (and feedback on so many of my Bicep issues) has been wildly influential and helpful in how I pieced together this deployment (between naming, modularizing, file organization, where to keep "constants" and tenant deployments, etc.) and I can't thank you enough for the help. Once I'm sure I've got all the secrets out of here, I'd be more than happy to publish my own for comparison. For the most part, I tried to get away from having separate JSON configuration files and instead build absolutely everything into the ARM template, using deployment scripts where ARM was lacking. As time warrants, I'd be more than happy to circle back and share my own experiences back to your project. The weightiest parts of mine are almost certainly the Managed Service Fabric pieces (and their respective application and service deployments) and Data Explorer (and setup of the tables and service mappings), but I'm sure I can find some common pieces as well. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @WhitWaldo for the update, sounds like you have been making some great progress. I managed to cleanup a lot of items over the past few months in ADF, If you mainly removing any specific items that make it hard to clone a tenant or start a new tenant. E.g. Mainly the global metadata files ... https://github.com/brwilkinson/AzureDeploymentFramework/blob/main/ADF/tenants/AOA/Global-ACU1.json I would be really interested once you have a clean project to see what you have come up with, I know it's hard to remove components to sanitize project after they have been "productionized" in order to share them. |
Beta Was this translation helpful? Give feedback.
-
|
I have pushed a version of the repo now, so above is no longer relevant. Networking is now automated based on:
Based on those you will get an automated Address Space which never overlaps with any other, based on those 4 characteristics. This is calculated inline in the specific templates that require the address e.g. VNET |
Beta Was this translation helpful? Give feedback.
The purpose behind this method for doing IP and network address allocation is to provide 2 simple capabilities.
You assign all static IP addresses, simply by providing the last octet IP address
Also the Network addresses are all dynamic from the DeploymentID
So you can have any 8 environments (per region), just don't repeat any numbers.
e.g. S1, D2, T3, T4, U5, Q6, P7, P8
Sandbox, Dev, Test, UAT, QA, Prod etc. <-- the letters are tags on the type of environment
The number is what determines the Network id 😎✅