Skip to content

Latest commit

 

History

History
476 lines (436 loc) · 12.9 KB

Install-and-configure-a-multi-master-Kubernetes-cluster-with-kubeadm.md

File metadata and controls

476 lines (436 loc) · 12.9 KB
Raw

Install and configure a multi-master Kubernetes cluster with kubeadm

img

Prerequisites

For this lab, we will use a standard Ubuntu 16.04 or 18.04 installation as a base image for the seven machines needed. The machines will all be configured on the same network, 10.10.10.0/24, and this network needs to have access to the Internet.

The first machine needed is the machine on which the HAProxy load balancer will be installed. We will assign the IP 10.10.10.93 to this machine.

We also need three Kubernetes master nodes. These machines will have the IPs 10.10.10.90, 10.10.10.91, and 10.10.10.92.

Finally, we will also have three Kubernetes worker nodes with the IPs 10.10.10.100, 10.10.10.101, and 10.10.10.102.

We also need an IP range for the pods. This range will be 10.30.0.0/16, but it is only internal to Kubernetes.

I will use my Linux desktop as a client machine to generate all the necessary certificates, but also to manage the Kubernetes cluster. If you don't have a Linux desktop, you can use the HAProxy machine to do the same thing.

img

Installing the client tools

We will need two tools on the client machine: the Cloud Flare SSL tool to generate the different certificates, and the Kubernetes client, kubectl, to manage the Kubernetes cluster.

Installing cfssl

1- Download the binaries.

$ wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
$ wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64

2- Add the execution permission to the binaries.

$ chmod +x cfssl*

3- Move the binaries to /usr/local/bin.

$ sudo mv cfssl_linux-amd64 /usr/local/bin/cfssl
$ sudo mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

4- Verify the installation.

$ cfssl version

Installing kubectl

1- Download the binary.

$wget https://storage.googleapis.com/kubernetes-release/release/v1.15.0/bin/linux/amd64/kubectl

2- Add the execution permission to the binary.

$chmod +x kubectl

3- Move the binary to /usr/local/bin.

$sudo mv kubectl /usr/local/bin

4- Verify the installation.

$ kubectl version

Installing the HAProxy load balancer

As we will deploy three Kubernetes master nodes, we need to deploy an HAPRoxy load balancer in front of them to distribute the traffic.

1- SSH to the 10.10.10.93 Ubuntu machine.

2- Update the machine.

$ sudo apt-get update
$ sudo apt-get upgrade

3- Install HAProxy.

$ sudo apt-get install haproxy

4- Configure HAProxy to load balance the traffic between the three Kubernetes master nodes.

$ sudo vim /etc/haproxy/haproxy.cfg
global
...
default
...
frontend kubernetes
bind 10.10.10.93:6443
option tcplog
mode tcp
default_backend kubernetes-master-nodes


backend kubernetes-master-nodes
mode tcp
balance roundrobin
option tcp-check
server k8s-master-0 10.10.10.90:6443 check fall 3 rise 2
server k8s-master-1 10.10.10.91:6443 check fall 3 rise 2
server k8s-master-2 10.10.10.92:6443 check fall 3 rise 2

5- Restart HAProxy.

$ sudo systemctl restart haproxy

Generating the TLS certificates

These steps can be done on your Linux desktop if you have one or on the HAProxy machine depending on where you installed the cfssl tool.

Creating a certificate authority

1- Create the certificate authority configuration file.

$ vim ca-config.json
{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "kubernetes": {
        "usages": ["signing", "key encipherment", "server auth", "client auth"],
        "expiry": "8760h"
      }
    }
  }
}

2- Create the certificate authority signing request configuration file.

$ vim ca-csr.json
{
  "CN": "Kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
  {
    "C": "IE",
    "L": "Cork",
    "O": "Kubernetes",
    "OU": "CA",
    "ST": "Cork Co."
  }
 ]
}

3- Generate the certificate authority certificate and private key.

$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca

4- Verify that the ca-key.pem and the ca.pem were generated.

$ ls -la

Creating the certificate for the Etcd cluster

1- Create the certificate signing request configuration file.

$ vim kubernetes-csr.json
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
  {
    "C": "IE",
    "L": "Cork",
    "O": "Kubernetes",
    "OU": "Kubernetes",
    "ST": "Cork Co."
  }
 ]
}

2- Generate the certificate and private key.

$ cfssl gencert \
-ca=ca.pem \
-ca-key=ca-key.pem \
-config=ca-config.json \
-hostname=10.10.10.90,10.10.10.91,10.10.10.92,10.10.10.93,127.0.0.1,kubernetes.default \
-profile=kubernetes kubernetes-csr.json | \
cfssljson -bare kubernetes

3- Verify that the kubernetes-key.pem and the kubernetes.pem file were generated.

$ ls -la

4- Copy the certificate to each nodes.

$ scp ca.pem kubernetes.pem kubernetes-key.pem [email protected]:~
$ scp ca.pem kubernetes.pem kubernetes-key.pem [email protected]:~
$ scp ca.pem kubernetes.pem kubernetes-key.pem [email protected]:~
$ scp ca.pem kubernetes.pem kubernetes-key.pem [email protected]:~
$ scp ca.pem kubernetes.pem kubernetes-key.pem [email protected]:~
$ scp ca.pem kubernetes.pem kubernetes-key.pem [email protected]:~

Preparing the nodes for kubeadm

Preparing the 10.10.10.90/91/92/100/101/102 machine

Performing below steps on all systems

Installing Docker latest version
$ sudo -s
# curl -fsSL https://get.docker.com -o get-docker.sh
# sh get-docker.sh
# usermod -aG docker your-user

Installing kubeadm, kublet, and kubectl

1- Add the Google repository key.

# curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -

2- Add the Google repository.

# vim /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io kubernetes-xenial main

3- Update the list of packages and install kubelet, kubeadm and kubectl.

# apt-get update
# apt-get install kubelet kubeadm kubectl

4- Disable the swap.

# swapoff -a
# sed -i '/ swap / s/^/#/' /etc/fstab

Installing and configuring Etcd

Installing and configuring Etcd on the 10.10.10.90/91/92 machine (All 3 master)

1- SSH to the 10.10.10.90 machine.
2- Create a configuration directory for Etcd.

$ sudo mkdir /etc/etcd /var/lib/etcd

3- Move the certificates to the configuration directory.

$ sudo mv ~/ca.pem ~/kubernetes.pem ~/kubernetes-key.pem /etc/etcd

4- Download the etcd binaries.

wget https://github.com/etcd-io/etcd/releases/download/v3.3.13/etcd-v3.3.13-linux-amd64.tar.gz

5- Extract the etcd archive.

$ tar xvzf etcd-v3.3.13-linux-amd64.tar.gz

6- Move the etcd binaries to /usr/local/bin.

$ sudo mv etcd-v3.3.13-linux-amd64/etcd* /usr/local/bin/

7- Create an etcd systemd unit file.

$ sudo vim /etc/systemd/system/etcd.service
[Unit]
Description=etcd
Documentation=https://github.com/coreos


[Service]
ExecStart=/usr/local/bin/etcd \
  --name 10.10.10.90 \
  --cert-file=/etc/etcd/kubernetes.pem \
  --key-file=/etc/etcd/kubernetes-key.pem \
  --peer-cert-file=/etc/etcd/kubernetes.pem \
  --peer-key-file=/etc/etcd/kubernetes-key.pem \
  --trusted-ca-file=/etc/etcd/ca.pem \
  --peer-trusted-ca-file=/etc/etcd/ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth \
  --initial-advertise-peer-urls https://10.10.10.90:2380 \
  --listen-peer-urls https://10.10.40.10:2380 \
  --listen-client-urls https://10.10.10.90:2379,http://127.0.0.1:2379 \
  --advertise-client-urls https://10.10.10.90:2379 \
  --initial-cluster-token etcd-cluster-0 \
  --initial-cluster 10.10.10.90=https://10.10.10.90:2380,10.10.10.91=https://10.10.10.91:2380,10.10.10.92=https://10.10.10.92:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5



[Install]
WantedBy=multi-user.target

8- Reload the daemon configuration.

$ sudo systemctl daemon-reload

9- Enable etcd to start at boot time.

$ sudo systemctl enable etcd

10- Start etcd.

$ sudo systemctl start etcd

11- Verify that the cluster is up and running.

$ ETCDCTL_API=3 etcdctl member list

Perform all the steps on other Master (91 and 92) by replacing IP

Initializing the master nodes

Initializing the Master node 10.10.10.90

1- SSH to the 10.10.10.90 machine.
2- Create the configuration file for kubeadm.

$ vim config.yaml
apiVersion: kubeadm.k8s.io/v1alpha3
kind: ClusterConfiguration
kubernetesVersion: stable
apiServerCertSANs:
- 10.10.10.93
controlPlaneEndpoint: "10.10.10.93:6443"
etcd:
  external:
    endpoints:
    - https://10.10.10.90:2379
    - https://10.10.10.91:2379
    - https://10.10.10.92:2379
    caFile: /etc/etcd/ca.pem
    certFile: /etc/etcd/kubernetes.pem
    keyFile: /etc/etcd/kubernetes-key.pem
networking:
  podSubnet: 10.30.0.0/24
apiServerExtraArgs:
  apiserver-count: "3"

3- Initialize the machine as a master node.

$ sudo kubeadm init --config=config.yaml

4- Copy the certificates to the two other masters.

$ sudo scp -r /etc/kubernetes/pki [email protected]:~
$ sudo scp -r /etc/kubernetes/pki [email protected]:~

Initializing the 2nd master node 10.10.10.91

1- SSH to the 10.10.10.91 machine.
2- Remove the apiserver.crt and apiserver.key.

$ rm ~/pki/apiserver.*

3- Move the certificates to the /etc/kubernetes directory.

$ sudo mv ~/pki /etc/kubernetes/

4 - Create the configuration file for kubeadm.

$ vim config.yaml
apiVersion: kubeadm.k8s.io/v1alpha3
kind: ClusterConfiguration
kubernetesVersion: stable
apiServerCertSANs:
- 10.10.10.93
controlPlaneEndpoint: "10.10.10.93:6443"
etcd:
  external:
    endpoints:
    - https://10.10.10.90:2379
    - https://10.10.10.91:2379
    - https://10.10.10.92:2379
    caFile: /etc/etcd/ca.pem
    certFile: /etc/etcd/kubernetes.pem
    keyFile: /etc/etcd/kubernetes-key.pem
networking:
  podSubnet: 10.30.0.0/24
apiServerExtraArgs:
  apiserver-count: "3"

5- Initialize the machine as a master node.

$ sudo kubeadm init --config=config.yaml

Initializing the 3rd master node 10.10.10.92

1- SSH to the 10.10.10.92 machine.
2- Remove the apiserver.crt and apiserver.key.

$ rm ~/pki/apiserver.*

3- Move the certificates to the /etc/kubernetes directory.

$ sudo mv ~/pki /etc/kubernetes/

4 - Create the configuration file for kubeadm.

$ vim config.yaml
apiVersion: kubeadm.k8s.io/v1alpha3
kind: ClusterConfiguration
kubernetesVersion: stable
apiServerCertSANs:
- 10.10.10.93
controlPlaneEndpoint: "10.10.10.93:6443"
etcd:
external:
  endpoints:
  - https://10.10.10.90:2379
  - https://10.10.10.91:2379
  - https://10.10.10.92:2379
  caFile: /etc/etcd/ca.pem
  certFile: /etc/etcd/kubernetes.pem
  keyFile: /etc/etcd/kubernetes-key.pem
networking:
podSubnet: 10.30.0.0/24
apiServerExtraArgs:
apiserver-count: "3"

5- Initialize the machine as a master node.

$ sudo kubeadm init --config=config.yaml

6- Copy the "kubeadm join" command line printed as the result of the previous command.

Initializing the worker nodes

Initializing the worker node 10.10.10.100/101/102

1- SSH to the 10.10.100.100 machine.
2- Execute the "kubeadm join" command that you copied from the last step of the initialization of the masters.

$ sudo kubeadm join 10.10.40.93:6443 --token [your_token] --discovery-token-ca-cert-hash sha256:[your_token_ca_cert_hash]

Run same command on worker node 101 and 102

Verifying that the workers joined the cluster

1- SSH to one of the master node.
2- Get the list of the nodes.

$ sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes

Configuring kubectl on the client machine

1- SSH to one of the master node. 10.10.10.90
2- Add permissions to the admin.conf file.

$ sudo chmod +r /etc/kubernetes/admin.conf

3- From the client machine, copy the configuration file.

$ scp [email protected]:/etc/kubernetes/admin.conf .

4- Create and configure the kubectl configuration directory.

$ mkdir ~/.kube
$ mv admin.conf ~/.kube/config
$ chmod 600 ~/.kube/config

5- Go back to the SSH session on the master and change back the permissions of the configuration file.

$ sudo chmod 600 /etc/kubernetes/admin.conf

6- check that you can access the Kubernetes API from the client machine.

$ kubectl get nodes

Deploying the overlay network

We are going to use Calico as the overlay network. You can also use static route or another overlay network tool like Weavenet or Flannel.
1- Deploy the overlay network pods from the client machine.

$kubectl apply -f https://docs.projectcalico.org/v3.7/manifests/calico.yaml

2- Check that the pods are deployed properly.

$ kubectl get pods -n kube-system

3- Check that the nodes are in Ready state.

$ kubectl get nodes

Contributor