Raspberry Pi Kubernetes Cluster

Learning how to create a Kubernetes cluster from scratch with some Raspberry Pis.

• Hardware Build
• Setup
  • Master Node
  • Worker Nodes
• Deploy
• Reference

---

Hardware Build

This cluster is 3 nodes:

Component	Quantity
Raspberry Pi 3 Model B+	2
Raspberry Pi 3 Model B	1
Anker PowerPort 6 (60W 6-Port USB Charging Hub)	1
Black Box USB-Powered 10/100 8-Port Switch	1
32GB Sandisk Ultra microSDHC Card	3
Raspberry Pi 3 - 4 Layer Stackable Dob Bone Case	1
Cat6 Ethernet Patch Cable	1
RJ-45 Color Coded Strain Relief Boots	8

Tools

Component	Quantity
UbiGear Ethernet Cable Crimper Kit + 100 RJ45	1

---

Setup

Note: These instructions are for Macs. Much of this setup is following the Kubernetes on Raspbian Lite guide by Alex Ellis.

1. Download Raspbian Stretch Lite.

2. Flash SD Card with the unzipped raspbian image. Etcher is an open-source Electron app for flashing OS images to SD cards and USB drives.

   brew cask install etcher

3. Enable ssh by placing an empty file on the sd card:

   touch /Volumes/boot/ssh

   • Note: may need to mount the sd card after flashing. Example: diskutil mount /dev/disk2s1.
   • Insert flashed sd card in to pi.

4. SSH in to the Pi directly from Ethernet adapter.

   • Enable Internet Sharing
   • Raspbian should have Avahi Daemon running, allowing for connection with raspberrypi.local:

   ping raspberrypi.local
   ssh [email protected]    # default pi password: raspberry

   If the above doesn't work, look for the ethernet adapter (bridge100) inet address:

   ifconfig
   
   # install nmap and discover ethernet devices
   brew install nmap
   sudo nmap -n -sn 192.168.2.1/24
   ssh [email protected]
   

   • Note: ipaddress may differ

5. Set up static IP address:

   # Find local network settings:
   ip -4 addr show | grep global
   
   # Find address of router (or gateway):
   ip route | grep default | aw '{print $3}'
   
   # Find the address of DNS server (likely same as gateway):
   cat /etc/resolv.conf
   
   # List network interface names:
   ls /sys/class/net/

   Edit /etc/dhcpcd.conf:

   # example static IP configuration:
   interface eth0
   static ip_address=192.168.3.2/24
   static routers=192.168.3.1
   

   Reboot:

   sudo reboot
   

6. Setup Locale and modify hostname to (e.g. k8s-master) using raspi-config util and reboot.

   sudo raspi-config

   Note:

   • Select locales with spacebar in raspi-config.
   • SSH in to rasspberry pi with <hostname>.local now (e.g. ssh [email protected]).

7. Setup Docker

   curl -sSL get.docker.com | sh
   sudo usermod pi -aG docker
   newgrp docker

8. Turn off swap space (required for K8s)

   sudo dphys-swapfile swapoff
   sudo dphys-swapfile uninstall
   sudo update-rc.d dphys-swapfile remove

9. Enable cgroups in /boot/cmdline.txt and reboot:

   bflags="$(head -n1 /boot/cmdline.txt) cgroup_enable=cpuset cgroup_enable=memory"
   echo $bflags | sudo tee /boot/cmdline.txt

   Note: cgroup_memory=1 might be needed for some pi3 models.

   sudo reboot

10. Install kubeadm

    curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
    echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
    sudo apt-get update
    sudo apt-get install -y kubeadm

Master Node

Note: Continue on to the Worker Nodes setup below for non-master nodes.

1. Initialize master node

   sudo kubeadm init --token-ttl=0 # takes ~10 mins
   mkdir -p $HOME/.kube
   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
   sudo chown $(id -u):$(id -g) $HOME/.kube/config

2. Save the generated join-token for the other nodes:

   # example
   sudo kubeadm join 192.168.3.2:6443 --token l3m1rn.vyo4bpefx51upqzw --discovery-token-ca-cert-hash sha256:1ba58581a3a95c795fd603894c4ff7f7a205004c20cc17e1cbe62a870019d267

3. Verify everything is running (system pods might show Pending for a while) and install addons:

   kubectl --namespace=kube-system get pods

   • See the installing addons doc
   • Run kubectl apply -f [podnetwork].yaml with one of the addons to deploy it to the cluster.

4. Install a network driver like Weave Net:

   kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

5. Copy cluster config to local machine to connect with cluster without sshing in to the master node:

   # from mac
   scp [email protected]:~/.kube/config ~/.kube/config-pi
   export KUBECONFIG=$KUBECONFIG:$HOME/.kube/config:$HOME/.kube/config-pi
   kubectl config use-context kubernetes-pi # whatever context you've named for your pi cluster config

Worker Nodes

Find the other pi's on the ethernet switch with arp -a, or sudo nmap -n -sn 192.168.3.1/24 (ip may differ).

1. Repeat the general setup from above for each worker node.
2. Change hostnames to k8s-worker-n via sudo raspi-config. After rebooting, it should be possible to ssh in without ips:

   ssh [email protected]
   ssh [email protected]

3. Join the nodes to the cluster:

   sudo kubeadm join 192.168.3.2:6443 --token l3m1rn.vyo4bpefx51upqzw --discovery-token-ca-cert-hash sha256:1ba58581a3a95c795fd603894c4ff7f7a205004c20cc17e1cbe62a870019d267

4. Verify cluster is set up:

   kubectl get nodes

   Note: run this command from master node.. see step 5 from the Master Node setup above.

Deploy

Deploy the K8s Visualizer

1. Clone the visualizer app serve using kubectl proxy:

   git clone https://github.com/raghur/gcp-live-k8s-visualizer.git
   kubectl proxy --www=path/to/gcp-live-k8s-visualizer

2. Navigate to http://localhost:8001/static/

Deploy K8s Dashboard

1. Create the rolebinding listed on the dashboard access control readme

   kubectl apply -f dashboard-admin.yaml

2. Deploy the dashboard:

   kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/alternative/kubernetes-dashboard.yaml
   

3. Allow access to dashboard through master node, by changing the default dashboard service type from ClusterIP to NodePort

   kubectl -n kube-system edit service kubernetes-dashboard

---

Reference

• Kubernetes on Raspbian Lite
• Headless Raspberry Pi Install
• How to SSH into your Raspberry Pi with Mac and Ethernet Cable
• Controlling your cluster from machines other than master
• Organinzing Cluster Access Using Kubeconfig Files
• GCP Live K8s Visualizer