upgrade k8s v1.29 and kubeedge v1.19. (#56)

Signed-off-by: Tomoya Fujita <[email protected]>
fujitatomoya · Feb 4, 2025 · 296212e · 296212e
1 parent 5a54daa
commit 296212e
Show file tree

Hide file tree

Showing 9 changed files with 114 additions and 180 deletions.
diff --git a/README.md b/README.md
@@ -32,13 +32,11 @@ We can use either `Physical Host System` or `Virtualized Instance`.
 
 - Host System
 
-This is practical environment for robot and robotics in general.
-This will set up Kubernetes cluster on host system as followings.
+This is practical environment for robot and robotics application in general.
+This will set up Kubernetes cluster on the following verified host systems.
 
-| Hostname | IP Address | OS | architecture |
-| --- | --- | --- | --- |
-| tomoyafujita | 192.168.1.248 | Ubuntu Focal/Jammy | amd64 |
-| ubuntu | 192.168.1.79 | Ubuntu Focal/Jammy | aarch64 (Raspberry Pi4) |
+  - Ubuntu: Focal/Jammy/Noble
+  - Architecture: amd64/aarch64(Raspberry Pi4)
 
 - Virtualized Instance
 
@@ -52,7 +50,7 @@ This environment is very much useful to try or test your container images or ser
 - [Build ROS / ROS 2 Full Docker Multi-Arch Images](./docs/Build_Docker_Images.md)
   - [ROS Noetic Container Build](./docs//Build_Docker_Images.md#ros-noetic)
   - [ROS Rolling Container Build](./docs//Build_Docker_Images.md#ros-rolling)
-- [Setup Kuberenetes Cluster](./docs/Setup_Kubernetes_Cluster.md)
+- [Setup Kubernetes Cluster](./docs/Setup_Kubernetes_Cluster.md)
   - [Container Network Interface (CNI)](./docs/Setup_Kubernetes_Cluster.md#container-network-interface-cni)
   - [Setup Kubernetes API Server](./docs/Setup_Kubernetes_Cluster.md#setup-kubernetes-api-server)
   - [Join Cluster from Worker Node](./docs/Setup_Kubernetes_Cluster.md#join-the-cluster)

diff --git a/docker/Dockerfile.noetic b/docker/Dockerfile.noetic
@@ -5,7 +5,9 @@
 #  docker pull <user_name>/ros-noetic
 
 FROM ros:noetic
+
 ENV ROS_DISTRO=noetic
+
 LABEL maintainer="Tomoya Fujita <[email protected]>"
 LABEL version="1.0"
 LABEL description="ROS noetic docker full image build"

diff --git a/docker/Dockerfile.rolling b/docker/Dockerfile.rolling
@@ -5,7 +5,9 @@
 #  docker pull <user_name>/ros-rolling
 
 FROM ros:rolling
+
 ENV ROS_DISTRO=rolling
+
 LABEL maintainer="Tomoya Fujita <[email protected]>"
 LABEL version="1.0"
 LABEL description="ROS rolling docker full image build"

diff --git a/docs/Build_Docker_Images.md b/docs/Build_Docker_Images.md
@@ -3,7 +3,7 @@
 Based on `ros:noetic` (ROS 1) and `ros:rolling` (ROS 2), we will build full ROS / ROS 2 docker images and upload those images to [dockerhub](https://hub.docker.com/).
 The reason to build full docker images for ROS and ROS 2 is basic container images do not include useful samples or examples packages, docker images with full packages can be very useful for container application on Kubernetes cluster system.
 
-To build and uploard all ROS container images, you can use [build_docker_images.sh](../scripts/build_docker_images.sh).
+To build and upload all ROS container images, you can use [build_docker_images.sh](../scripts/build_docker_images.sh).
 
 ## Setup Prerequisite
 

diff --git a/docs/Install_Kubernetes_Packages.md b/docs/Install_Kubernetes_Packages.md
@@ -2,20 +2,19 @@
 
 These packages are required to both cluster on host and virtualized environment.
 And all cluster nodes must install the required packages to join the cluster, that is said we need to install these packages on every single host system.
-Even virtualized environment, it will issue Kubernetes API from host system to Kuberentes API server running in the virtualized environment, so that it can provide the consistent user experience.
+Even virtualized environment, it will issue Kubernetes API from host system to Kubernetes API server running in the virtualized environment, so that it can provide the consistent user experience.
 
 To install all related packages to host system, you can use [install_k8s_packages.sh](../scripts/install_k8s_packages.sh).
 
 **The following commands requires `sudo` access permission to install packages.**
 
 ```bash
-> cd ros_k8s/scripts
-> ./install_k8s_packages.sh
+> ./scripts/install_k8s_packages.sh
 ```
 
 The following are manual steps to install packages that you can refer to understand what packages are actually installed.
 
-## Host Setting
+## Raspberry Pi Setting
 
 ### Enable Cgroup Raspberry Pi
 
@@ -43,55 +42,60 @@ net.ifnames=0 dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=LABE
 CRI v1alpha2 removed - kubelet will not register the node if the container runtime doesn't support CRI v1. So to work with Kubernetes 1.26 or later, containerd 1.6.0 is required.
 
 ```bash
-> apt remove -y containerd docker.io
-> apt install -y apt-transport-https ca-certificates curl gnupg-agent software-properties-common gnupg2
-> curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
-> add-apt-repository -y "deb [arch=$(dpkg-architecture -q DEB_BUILD_ARCH)] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
-> apt update -y
-> apt install -y docker-ce docker-ce-cli containerd.io
-> mkdir -p /etc/containerd
-> containerd config default | tee /etc/containerd/config.toml
-> sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml
-> systemctl restart containerd
-> systemctl enable containerd
-> systemctl restart docker
-> systemctl enable docker
-> docker version
+apt remove -y containerd docker.io
+apt install -y apt-transport-https ca-certificates curl gnupg-agent software-properties-common gnupg2
+curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
+add-apt-repository -y "deb [arch=$(dpkg-architecture -q DEB_BUILD_ARCH)] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
+apt update -y
+apt install -y docker-ce docker-ce-cli containerd.io
+mkdir -p /etc/containerd
+containerd config default | tee /etc/containerd/config.toml
+sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml
+systemctl restart containerd
+systemctl enable containerd
+systemctl restart docker
+systemctl enable docker
+docker version
 ```
 
 ## Kubernetes
 
-In this tutorial, we use Kuberentes `v1.26.9-00`.
+In this tutorial, we use Kuberentes `v1.29.x-xx` in default.
 
 ### Kubernetes Install
 
 ```bash
-> apt install -y apt-transport-https curl
-> curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add
-> apt-add-repository -y "deb http://apt.kubernetes.io/ kubernetes-xenial main"
-> apt install -y --allow-downgrades kubeadm=1.26.9-00 kubelet=1.26.9-00 kubectl=1.26.9-00
-> kubeadm version
+### uninstall if installed already
+apt remove -y --allow-change-held-packages kubelet kubectl kubeadm
+
+### install kubernetes
+curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
+echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | tee /etc/apt/sources.list.d/kubernetes.list
+apt-get update
+apt-get install -y kubelet kubeadm kubectl
+apt-mark hold kubelet kubeadm kubectl
 
 ### kubectl version will try to access server that leads to error to exit this script
-> kubectl version --client
+kubeadm version
+kubectl version --client
 ```
 
 ## Golang
 
 ```bash
-> add-apt-repository -y ppa:longsleep/golang-backports
-> apt update
+add-apt-repository -y ppa:longsleep/golang-backports
+apt update
 
 ### golang-1.21 or later should be fine, just installing
-> apt install golang-1.21 golang-go
-> go version
+apt install golang-1.21 golang-go
+go version
 ```
 
 ## Kind
 
 ```bash
-> go install sigs.k8s.io/kind@v0.19.0
+go install sigs.k8s.io/kind@v0.26.0
 
 ## kind needs to be in path
-> $HOME/go/bin/kind version
+$HOME/go/bin/kind version
 ```
diff --git a/docs/Setup_KubeEdge.md b/docs/Setup_KubeEdge.md
@@ -37,8 +37,6 @@ see [Setup Kubernetes API Server](./Setup_Kubernetes_Cluster.md#setup-kubernetes
 In Kubernetes workloads, it requires one of CNI implementation running to set up the cluster.
 see [Deploy CNI Plugin](https://github.com/fujitatomoya/ros_k8s/blob/master/docs/Setup_Kubernetes_Cluster.md#deploy-cni-plugin) to start the CNI for Kubernetes. (this CNI can only be used by Kubernetes worker nodes but KubeEdge.)
 
-**<span style="color: red;">TODO: CNI needs to be uninstalled</span>**
-
 This is only required to bring the Kubenretes API-server running, because we are going to deploy cloudcore to the same physical node with Kubernetes API-server.
 Instead of having CNI deployed to bring the Kubernetes API-server up and running, we are not able to deploy cloudcore to the node since we cannot deploy the containers to any `NotReady` nodes.
 
@@ -51,11 +49,11 @@ see more details for https://kubeedge.io/docs/setup/install-with-keadm/
 `keadm` needs to be installed in both cloud and edge nodes.
 
 ```bash
-> wget https://github.com/kubeedge/kubeedge/releases/download/v1.16.3/keadm-v1.16.3-linux-amd64.tar.gz
-> tar -zxvf keadm-v1.16.3-linux-amd64.tar.gz
-> cp keadm-v1.16.3-linux-amd64/keadm//keadm /usr/local/bin/keadm
+> wget https://github.com/kubeedge/kubeedge/releases/download/v1.19.1/keadm-v1.19.1-linux-amd64.tar.gz
+> tar -zxvf keadm-v1.19.1-linux-amd64.tar.gz
+> cp keadm-v1.19.1-linux-amd64/keadm/keadm /usr/local/bin/keadm
 > keadm version
-version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"4f1da43e6807c127e549e5d4859ed1b66c6f5806", GitTreeState:"clean", BuildDate:"2024-07-12T03:02:51Z", GoVersion:"go1.20.10", Compiler:"gc", Platform:"linux/amd64"}
+version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.1", GitCommit:"e676d31ee10ee8a9a17e74b38134a11bdc1d8350", GitTreeState:"clean", BuildDate:"2024-12-02T12:11:18Z", GoVersion:"go1.21.11", Compiler:"gc", Platform:"linux/amd64"}
 ```
 
 ### Cloud Core
@@ -74,32 +72,42 @@ version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"4f
 > kubectl taint nodes tomoyafujita node-role.kubernetes.io/control-plane:NoSchedule-
 node/tomoyafujita untainted
 
+> kubectl taint nodes --all node-role.kubernetes.io/master-
+> kubectl taint nodes --all node-role.kubernetes.io/control-plane-
+
 > kubectl get nodes -o json | jq '.items[].spec.taints'
 null
 ```
 
 - start KubeEdge `cloudcore`.
 
 ```bash
-> keadm init --advertise-address=192.168.1.248 --kube-config=/root/.kube/config --kubeedge-version=v1.16.3
-Kubernetes version verification passed, KubeEdge installation will start...
+
+> keadm init --advertise-address=<YOUR_IP_ADDRESS> --kube-config=/root/.kube/config --kubeedge-version=v1.19.1
+Kubernetes version verification passed, KubeEdge v1.19.1 installation will start...
 CLOUDCORE started
 =========CHART DETAILS=======
-NAME: cloudcore
-LAST DEPLOYED: Wed Sep 18 14:56:13 2024
+Name: cloudcore
+LAST DEPLOYED: Mon Feb  3 15:36:04 2025
 NAMESPACE: kubeedge
 STATUS: deployed
 REVISION: 1
 
 > kubectl get all -n kubeedge
 NAME                             READY   STATUS    RESTARTS   AGE
-pod/cloudcore-77b5dfdd57-btmlp   1/1     Running   0          24s
+pod/cloudcore-7c54886ddb-kd2vh   1/1     Running   0          42s
+
+NAME                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                             AGE
+service/cloudcore   ClusterIP   10.111.74.141   <none>        10000/TCP,10001/UDP,10002/TCP,10003/TCP,10004/TCP   42s
+
+NAME                                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
+daemonset.apps/edge-eclipse-mosquitto   0         0         0       0            0           <none>          42s
 
 NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
-deployment.apps/cloudcore   1/1     1            1           24s
+deployment.apps/cloudcore   1/1     1            1           42s
 
 NAME                                   DESIRED   CURRENT   READY   AGE
-replicaset.apps/cloudcore-77b5dfdd57   1         1         1       24s
+replicaset.apps/cloudcore-7c54886ddb   1         1         1       42s
 ```
 
 ### Edge Core
@@ -108,50 +116,36 @@ replicaset.apps/cloudcore-77b5dfdd57   1         1         1       24s
 
 ```bash
 > keadm gettoken
-40d9bb8bb2c3818728da3d46f1a78b58f4b9fba8665cc392ded4698d1eb5cab1.eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2OTQ4NDA5ODN9.niGZHHdR7s89K4-919fCNKEVTyudb8DtTmE9p5PFzKg
+30dad7c33966782b1fd0aa21b068d348e2acbf7464339882f10f6ce23d10e0f2.eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE3Mzg3MTIxNzN9.3NuaxcMlxXaCxV4Bebl54oxisL_VUwCtViX6xjAmIr0
 ```
 
 - start KubeEdge `edgecore`
 
 ```bash
-> keadm join --cloudcore-ipport=192.168.1.248:10000 --token=40d9bb8bb2c3818728da3d46f1a78b58f4b9fba8665cc392ded4698d1eb5cab1.eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2OTQ4NDA5ODN9.niGZHHdR7s89K4-919fCNKEVTyudb8DtTmE9p5PFzKg --kubeedge-version=v1.16.3 --remote-runtime-endpoint=unix:///run/containerd/containerd.sock --cgroupdriver systemd
-I0915 06:28:19.544584   10424 command.go:845] 1. Check KubeEdge edgecore process status
-I0915 06:28:19.578838   10424 command.go:845] 2. Check if the management directory is clean
-I0915 06:28:19.579238   10424 join.go:107] 3. Create the necessary directories
-I0915 06:28:19.580874   10424 join.go:184] 4. Pull Images
-Pulling kubeedge/installation-package:v1.12.1 ...
-Pulling kubeedge/pause:3.6 ...
-Pulling eclipse-mosquitto:1.6.15 ...
-I0915 06:28:19.590357   10424 join.go:184] 5. Copy resources from the image to the management directory
-I0915 06:28:27.255374   10424 join.go:184] 6. Start the default mqtt service
-I0915 06:28:27.256005   10424 join.go:107] 7. Generate systemd service file
-I0915 06:28:27.256873   10424 join.go:107] 8. Generate EdgeCore default configuration
-I0915 06:28:27.256988   10424 join.go:270] The configuration does not exist or the parsing fails, and the default configuration is generated
-W0915 06:28:27.263577   10424 validation.go:71] NodeIP is empty , use default ip which can connect to cloud.
-I0915 06:28:27.270836   10424 join.go:107] 9. Run EdgeCore daemon
-I0915 06:28:34.350843   10424 join.go:435]
-I0915 06:28:34.350933   10424 join.go:436] KubeEdge edgecore is running, For logs visit: journalctl -u edgecore.service -xe
+
+> keadm join --cloudcore-ipport=<YOUR_IP_ADDRESS>>:10000 --token=30dad7c33966782b1fd0aa21b068d348e2acbf7464339882f10f6ce23d10e0f2.eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE3Mzg3MTIxNzN9.3NuaxcMlxXaCxV4Bebl54oxisL_VUwCtViX6xjAmIr0 --kubeedge-version=v1.19.1 --remote-runtime-endpoint=unix:///run/containerd/containerd.sock --cgroupdriver systemd
+...<snip>
+I0203 15:39:14.636693  990048 join_others.go:273] KubeEdge edgecore is running, For logs visit: journalctl -u edgecore.service -xe
+I0203 15:39:24.645991  990048 join.go:94] 9. Install Complete!
 
 > systemctl status edgecore
 ● edgecore.service
      Loaded: loaded (/etc/systemd/system/edgecore.service; enabled; vendor preset: enabled)
-     Active: active (running) since Fri 2023-09-15 06:28:34 UTC; 1min 9s ago
-   Main PID: 10603 (edgecore)
-      Tasks: 16 (limit: 9190)
-     Memory: 31.4M
-        CPU: 11.565s
+     Active: active (running) since Mon 2025-02-03 15:39:14 PST; 1min 21s ago
+   Main PID: 990292 (edgecore)
+      Tasks: 24 (limit: 18670)
+     Memory: 33.3M
+        CPU: 6.113s
      CGroup: /system.slice/edgecore.service
-             └─10603 /usr/local/bin/edgecore
+             └─990292 /usr/local/bin/edgecore
 ...<snip>
 ```
 
 - check cluster nodes.
 
 ```bash
-> kubectl get nodes -o wide
-NAME                                    STATUS   ROLES                  AGE    VERSION                    INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
-tomoyafujita-hp-compaq-elite-8300-sff   Ready    control-plane,master   104m   v1.23.17                   192.168.1.248   <none>        Ubuntu 20.04.6 LTS   5.15.0-83-generic   docker://24.0.5
-ubuntu                                  Ready    agent,edge             97s    v1.22.6-kubeedge-v1.12.1   192.168.1.238   <none>        Ubuntu 22.04.3 LTS   5.15.0-1034-raspi   docker://24.0.5
+kubectl get nodes -o wide
+
 ```
 
 ## KubeEdge Test Deployment
@@ -160,11 +154,6 @@ ubuntu                                  Ready    agent,edge             97s    v
 > kubectl apply -f ros2-sample-hostnic.yaml
 deployment.apps/ros2-talker-1 created
 deployment.apps/ros2-listener-1 created
-
-> kubectl get pods -o wide
-NAME                               READY   STATUS    RESTARTS   AGE   IP              NODE                                    NOMINATED NODE   READINESS GATES
-ros2-listener-1-64c4c996b4-scm9k   1/1     Running   0          14m   192.168.1.248   tomoyafujita-hp-compaq-elite-8300-sff   <none>           <none>
-ros2-talker-1-bdd899d8d-drt2f      1/1     Running   0          14m   192.168.1.238   ubuntu                                  <none>           <none>
 ```
 
 ## Break Down KubeEdge