K0rdent · a13x5 · Sep 2, 2024 · Aug 26, 2024
@@ -0,0 +1,162 @@
+# Hosted control plane (k0smotron) deployment
+
+## Prerequisites
+
+-   Management Kubernetes cluster (v1.28+) deployed on Azure with HMC installed
+    on it
+-   Default storage class configured on the management cluster
+
+Keep in mind that all control plane components for all managed clusters will
+reside in the management cluster.
+
+## Pre-existing resources
+
+Certain resources will not be created automatically in a hosted control plane
+scenario thus they should be created in advance and provided in the `Deployment`
+object. You can reuse these resources with management cluster as described
+below.
+
+If you deployed your Azure Kubernetes cluster using Cluster API Provider Azure
+(CAPZ) you can obtain all the necessary data with the commands below:
+
+**Location**
+
+```bash
+kubectl get azurecluster <cluster name> -o go-template='{{.spec.location}}'
+```
+
+**Subscription ID**
+
+```bash
+kubectl get azurecluster <cluster name> -o go-template='{{.spec.subscriptionID}}'
+```
+
+**Resource group**
+
+```bash
+kubectl get azurecluster <cluster name> -o go-template='{{.spec.resourceGroup}}'
+```
+
+**vnet name**
+
+```bash
+kubectl get azurecluster <cluster name> -o go-template='{{.spec.networkSpec.vnet.name}}'
+```
+
+**Subnet name**
+
+```bash
+kubectl get azurecluster <cluster name> -o go-template='{{(index .spec.networkSpec.subnets 1).name}}'
+```
+
+**Route table name**
+
+```bash
+kubectl get azurecluster <cluster name> -o go-template='{{(index .spec.networkSpec.subnets 1).routeTable.name}}'
+```
+
+**Security group name**
+
+```bash
+kubectl get azurecluster <cluster name> -o go-template='{{(index .spec.networkSpec.subnets 1).securityGroup.name}}'
+```
+
+
+
+## HMC Deployment manifest
+
+With all the collected data your `Deployment` manifest will look similar to this:
+
+```yaml
+apiVersion: hmc.mirantis.com/v1alpha1
+kind: Deployment
+metadata:
+  name: azure-hosted-cp
+spec:
+  template: azure-hosted-cp
+  config:
+    location: "westus"
+    subscriptionID: ceb131c7-a917-439f-8e19-cd59fe247e03
+    vmSize: Standard_A4_v2
+    clusterIdentity:
+      name: az-cluster-identity
+      namespace: hmc-system
+    resourceGroup: mgmt-cluster
+    network:
+      vnetName: mgmt-cluster-vnet
+      nodeSubnetName: mgmt-cluster-node-subnet
+      routeTableName: mgmt-cluster-node-routetable
+      securityGroupName: mgmt-cluster-node-nsg
+    tenantID: 7db9e0f2-c88a-4116-a373-9c8b6cc9d5eb
+    clientID: 471f65fa-ddee-40b4-90ae-da1a8a114ee1
+    clientSecret: "u_RANDOM"
+```
+
+To simplify creation of the deployment object you can use the template below:
+
+```yaml
+apiVersion: hmc.mirantis.com/v1alpha1
+kind: Deployment
+metadata:
+  name: azure-hosted-cp
+spec:
+  template: azure-hosted-cp
+  config:
+    location: "{{.spec.location}}"
+    subscriptionID: "{{.spec.subscriptionID}}"
+    vmSize: Standard_A4_v2
+    clusterIdentity:
+      name: az-cluster-identity
+      namespace: hmc-system
+    resourceGroup: "{{.spec.resourceGroup}}"
+    network:
+      vnetName: "{{.spec.networkSpec.vnet.name}}"
+      nodeSubnetName: "{{(index .spec.networkSpec.subnets 1).name}}"
+      routeTableName: "{{(index .spec.networkSpec.subnets 1).routeTable.name}}"
+      securityGroupName: "{{(index .spec.networkSpec.subnets 1).securityGroup.name}}"
+    tenantID: 7db9e0f2-c88a-4116-a373-9c8b6cc9d5eb
+    clientID: 471f65fa-ddee-40b4-90ae-da1a8a114ee1
+    clientSecret: "u_RANDOM"
+```
+
+Then you can render it using the command:
+
+```bash
+kubectl get azurecluster <management cluster name> -o go-template="$(cat template.yaml)"
+```
+
+## Cluster creation
+
+After applying `Deployment` object you require to manually set the status of the
+`AzureCluster` object due to current limitations (see k0sproject/k0smotron#668).
+
+To do so you need to execute the following command:
+
+```bash
+kubectl patch azurecluster <cluster name> --type=merge --subresource status --patch 'status: {ready: true}'
+```
+
+## Important notes on the cluster deletion
+
+Because of the aforementioned limitation you also need to make manual steps in
+order to properly delete cluster.
+
+Before removing the cluster make sure to place custom finalizer onto
+`AzureCluster` object. This is needed to prevent it from being deleted instantly
+which will cause cluster deletion to stuck indefinitely.
+
+To place finalizer you can execute the following command:
+
+```bash
+kubectl patch azurecluster <cluster name> --type=merge --patch 'metadata: {finalizers: [manual]}'
+```
+
+When finalizer is placed you can remove the `Deployment` as usual. Check that
+all `AzureMachines` objects are deleted successfully and remove finalizer you've
+placed to finish cluster deletion.
+
+In case if have orphaned `AzureMachines` left you have to delete finalizers on
+them manually after making sure that no VMs are present in Azure.
+
+*Note: since Azure admission prohibits orphaned objects mutation you'll have to
+disable it by deleting it's `mutatingwebhookconfiguration`*
@@ -0,0 +1,23 @@
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*.orig
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj
+.vscode/
@@ -0,0 +1,20 @@
+apiVersion: v2
+name: azure-hosted-cp
+description: |
+  An HMC template to deploy a k8s cluster on Azure with control plane components
+  within the management cluster.
+type: application
+# This is the chart version. This version number should be incremented each time you make changes
+# to the chart and its templates, including the app version.
+# Versions are expected to follow Semantic Versioning (https://semver.org/)
+version: 0.0.1
+# This is the version number of the application being deployed. This version number should be
+# incremented each time you make changes to the application. Versions are not expected to
+# follow Semantic Versioning. They should reflect the version the application is using.
+# It is recommended to use it with quotes.
+appVersion: "1.30.4+k0s.0"
+annotations:
+  hmc.mirantis.com/type: deployment
+  hmc.mirantis.com/infrastructure-providers: azure
+  hmc.mirantis.com/controlplane-providers: k0s
+  hmc.mirantis.com/bootstrap-providers: k0s
@@ -0,0 +1,43 @@
+{{- define "cluster.name" -}}
+    {{- .Release.Name | trunc 63 | trimSuffix "-" }}
+{{- end }}
+
+{{- define "azuremachinetemplate.name" -}}
+    {{- include "cluster.name" . }}-mt
+{{- end }}
+
+{{- define "k0smotroncontrolplane.name" -}}
+    {{- include "cluster.name" . }}-cp
+{{- end }}
+
+{{- define "k0sworkerconfigtemplate.name" -}}
+    {{- include "cluster.name" . }}-machine-config
+{{- end }}
+
+{{- define "machinedeployment.name" -}}
+    {{- include "cluster.name" . }}-md
+{{- end }}
+
+{{- define "azure.json" -}}
+{
+  "cloud": "AzurePublicCloud",
+  "tenantId": "{{ .Values.tenantID }}",
+  "subscriptionId": "{{ .Values.subscriptionID }}",
+  "aadClientId": "{{ .Values.clientID }}",
+  "aadClientSecret": "{{ .Values.clientSecret }}",
+  "resourceGroup": "{{ .Values.resourceGroup }}",
+  "securityGroupName": "{{ .Values.network.securityGroupName }}",
+  "securityGroupResourceGroup": "{{ .Values.resourceGroup }}",
+  "location": "{{ .Values.location }}",
+  "vmType": "vmss",
+  "vnetName": "{{ .Values.network.vnetName }}",
+  "vnetResourceGroup": "{{ .Values.resourceGroup }}",
+  "subnetName": "{{ .Values.network.nodeSubnetName }}",
+  "routeTableName": "{{ .Values.routeTableName }}",
+  "loadBalancerSku": "Standard",
+  "loadBalancerName": "",
+  "maximumLoadBalancerRuleCount": 250,
+  "useManagedIdentityExtension": false,
+  "useInstanceMetadata": true
+}
+{{- end }}
@@ -0,0 +1,31 @@
+apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
+kind: AzureCluster
+metadata:
+  name: {{ include "cluster.name" . }}
+  annotations:
+    cluster.x-k8s.io/managed-by: k0smotron
+spec:
+  identityRef:
+    kind: AzureClusterIdentity
+    name: {{ .Values.clusterIdentity.name }}
+    namespace: {{ .Values.clusterIdentity.namespace }}
+  networkSpec:
+    vnet:
+      resourceGroup: {{ .Values.resourceGroup }}
+      name: {{ .Values.network.vnetName }}
+    subnets:
+      - name: {{ .Values.network.nodeSubnetName }}
+        role: node
+        routeTable:
+          name: {{ .Values.network.routeTableName }}
+        securityGroup:
+          name: {{ .Values.network.securityGroupName }}
+  location: {{ .Values.location }}
+  {{- if .Values.bastion.enabled }}
+  {{- with .Values.bastion.bastionSpec }}
+  bastionSpec:
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
+  {{- end }}
+  subscriptionID: {{ .Values.subscriptionID }}
+  resourceGroup: {{ .Values.resourceGroup }}
@@ -0,0 +1,20 @@
+apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
+kind: AzureMachineTemplate
+metadata:
+  name: {{ include "azuremachinetemplate.name" . }}
+spec:
+  template:
+    spec:
+      osDisk:
+        diskSizeGB: {{ .Values.rootVolumeSize }}
+        osType: Linux
+      {{- if not ( .Values.sshPublicKey | empty) }}
+      sshPublicKey: {{ .Values.sshPublicKey }}
+      {{- end }}
+      vmSize: {{ .Values.vmSize }}
+      {{- if not (quote .Values.image | empty) }}
+      {{- with .Values.image }}
+      image:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- end }}
@@ -0,0 +1,17 @@
+apiVersion: cluster.x-k8s.io/v1beta1
+kind: Cluster
+metadata:
+  name: {{ include "cluster.name" . }}
+spec:
+  {{- with .Values.clusterNetwork }}
+  clusterNetwork:
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
+  controlPlaneRef:
+    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
+    kind: K0smotronControlPlane
+    name: {{ include "k0smotroncontrolplane.name" .  }}
+  infrastructureRef:
+    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
+    kind: AzureCluster
+    name: {{ include "cluster.name" . }}
@@ -0,0 +1,49 @@
+apiVersion: controlplane.cluster.x-k8s.io/v1beta1
+kind: K0smotronControlPlane
+metadata:
+  name: {{ include "k0smotroncontrolplane.name" . }}
+spec:
+  replicas: {{ .Values.controlPlaneNumber }}
+  version: {{ .Values.k0s.version | replace "+" "-" }}
+  {{- with .Values.k0smotron.service }}
+  service:
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
+  controllerPlaneFlags:
+  - "--enable-cloud-provider=true"
+  - "--debug=true"
+  k0sConfig:
+    apiVersion: k0s.k0sproject.io/v1beta1
+    kind: ClusterConfig
+    metadata:
+      name: k0s
+    spec:
+      network:
+        provider: calico
+        calico:
+          mode: vxlan
+      extensions:
+        helm:
+          repositories:
+            - name: cloud-provider-azure
+              url: https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/helm/repo
+            - name: azuredisk-csi-driver
+              url: https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/master/charts
+          charts:
+            - name: cloud-provider-azure
+              namespace: kube-system
+              chartname: cloud-provider-azure/cloud-provider-azure
+              version: 1.30.4
+              order: 1
+              values: |
+                cloudControllerManager:
+                  nodeSelector:
+                    node-role.kubernetes.io/control-plane: null
+            - name: azuredisk-csi-driver
+              namespace: kube-system
+              chartname: azuredisk-csi-driver/azuredisk-csi-driver
+              version: 1.30.3
+              order: 2
+              values: |
+                linux:
+                  kubelet: "/var/lib/k0s/kubelet"
@@ -0,0 +1,15 @@
+apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
+kind: K0sWorkerConfigTemplate
+metadata:
+  name: {{ include "k0sworkerconfigtemplate.name" . }}
+spec:
+  template:
+    spec:
+      version: {{ .Values.k0s.version }}
+      args:
+      - --enable-cloud-provider
+      - --kubelet-extra-args="--cloud-provider=external"
+      files:
+        - path: "/etc/kubernetes/azure.json"
+          permissions: "0644"
+          content: {{ include "azure.json" . | toJson }}