diff --git a/docs/usage/install/underlay/get-started-ovs-zh_CN.md b/docs/usage/install/underlay/get-started-ovs-zh_CN.md index 64f56b4508..c29431cf3f 100644 --- a/docs/usage/install/underlay/get-started-ovs-zh_CN.md +++ b/docs/usage/install/underlay/get-started-ovs-zh_CN.md @@ -27,7 +27,7 @@ Spiderpool 可用作 Underlay 网络场景下提供固定 IP 的一种解决方 * 如果你使用 Underlay 模式，`coordinator` 会在主机上创建 veth 接口，为了防止 NetworkManager 干扰 veth 接口, 导致 Pod 访问异常。我们需要配置 NetworkManager，使其不纳管这些 Veth 接口。 - * 如果你通过 `Iface`r 创建 Vlan 和 Bond 接口，NetworkManager 可能会干扰这些接口，导致 Pod 访问异常。我们需要配置 NetworkManager，使其不纳管这些 Veth 接口。 + * 如果你通过 `Ifacer` 创建 Vlan 和 Bond 接口，NetworkManager 可能会干扰这些接口，导致 Pod 访问异常。我们需要配置 NetworkManager，使其不纳管这些 Veth 接口。 ```shell ~# IFACER_INTERFACE="" @@ -38,6 +38,130 @@ Spiderpool 可用作 Underlay 网络场景下提供固定 IP 的一种解决方 ~# systemctl restart NetworkManager ``` +## 节点上配置 Open vSwitch 网桥 + +如下是创建并配置持久 OVS Bridge 的示例，本文中以 `eth0` 网卡为例，需要在每个节点上执行。 + +### 使用 netplan 持久化 OVS Bridge + +1. 创建 OVS Bridge + + ```bash + ~# ovs-vsctl add-br br1 + ~# ovs-vsctl add-port br1 eth0 + ~# ip link set br1 up + ``` + +2. 在 /etc/netplan 目录下创建 12-br1.yaml 后，通过 `netplan apply` 生效。 + + 为确保在重启主机等场景下 br1 仍然可用，请检查 eth0 网卡是否也被 netplan 纳管。 + + ```yaml: 12-br1.yaml + network: + version: 2 + renderer: networkd + ethernets: + br1: + addresses: + - "/<子网掩码>" # 172.18.10.10/16 + ``` + +3. 创建后，可以在每个节点上查看到如下的网桥信息： + + ```bash + ~# ovs-vsctl show + ec16d9e1-6187-4b21-9c2f-8b6cb75434b9 + Bridge br1 + Port eth0 + Interface eth0 + Port br1 + Interface br1 + type: internal + Port veth97fb4795 + Interface veth97fb4795 + ovs_version: "2.17.3" + + ~# ip a show br1 + 208: br1: mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 + link/ether 00:50:56:b4:5f:fd brd ff:ff:ff:ff:ff:ff + inet 172.18.10.10/16 brd 172.18.255.255 scope global noprefixroute br1 + valid_lft forever preferred_lft forever + inet6 fe80::4f28:8ef1:6b82:a9e4/64 scope link noprefixroute + valid_lft forever preferred_lft forever + ``` + +### 使用 NetworkManager 持久化 OVS Bridge + +通过 NetworkManager 持久化 OVS Bridge 是一种不局限操作系统，更通用的一种方式。 + +1. 使用 NetworkManager 持久化 OVS Bridge，你需要安装 OVS NetworkManager 插件，示例如下： + + ```bash + ~# sudo dnf install -y NetworkManager-ovs + ~# sudo systemctl restart NetworkManager + ``` + +2. 创建 ovs 网桥、端口和接口。 + + ```bash + ~# sudo nmcli con add type ovs-bridge conn.interface br1 con-name br1 + ~# sudo nmcli con add type ovs-port conn.interface br1-port master br1 con-name br1-port + ~# sudo nmcli con add type ovs-interface slave-type ovs-port conn.interface br1 master br1-port con-name br1-int + ``` + +3. 在网桥上创建另一个端口，并选择我们的物理设备中的 eth0 网卡作为其以太网接口，以便真正的流量可以在网络上流转。 + + ```bash + ~# sudo nmcli con add type ovs-port conn.interface ovs-port-eth0 master br1 con-name ovs-port-eth0 + ~# sudo nmcli con add type ethernet conn.interface eth0 master ovs-port-eth0 con-name ovs-port-eth0-int + ``` + +4. 配置与激活 ovs 网桥。 + + 通过设置静态 IP 的方式配置网桥 + + ```bash + ~# sudo nmcli con modify br1-int ipv4.method static ipv4.address "/<子网掩码>" # 172.18.10.10/16 + ``` + + 激活网桥。 + + ```bash + ~# sudo nmcli con down "eth0" + ~# sudo nmcli con up ovs-port-eth0-int + ~# sudo nmcli con up br1-int + ``` + +5. 创建后，可以在每个节点上查看到类似如下的信息。 + + ```bash + ~# nmcli c + br1-int dbb1c9be-e1ab-4659-8d4b-564e3f8858fa ovs-interface br1 + br1 a85626c1-2392-443b-a767-f86a57a1cff5 ovs-bridge br1 + br1-port fe30170f-32d2-489e-9ca3-62c1f5371c6c ovs-port br1-port + ovs-port-eth0 a43771a9-d840-4d2d-b1c3-c501a6da80ed ovs-port ovs-port-eth0 + ovs-port-eth0-int 1334f49b-dae4-4225-830b-4d101ab6fad6 ethernet eth0 + + ~# ovs-vsctl show + 203dd6d0-45f4-4137-955e-c4c36b9709e6 + Bridge br1 + Port ovs-port-eth0 + Interface eth0 + type: system + Port br1-port + Interface br1 + type: internal + ovs_version: "3.2.1" + + ~# ip a show br1 + 208: br1: mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 + link/ether 00:50:56:b4:5f:fd brd ff:ff:ff:ff:ff:ff + inet 172.18.10.10/16 brd 172.18.255.255 scope global noprefixroute br1 + valid_lft forever preferred_lft forever + inet6 fe80::4f28:8ef1:6b82:a9e4/64 scope link noprefixroute + valid_lft forever preferred_lft forever + ``` + ## 安装 Spiderpool 1. 安装 Spiderpool。 @@ -86,7 +210,7 @@ Spiderpool 可用作 Underlay 网络场景下提供固定 IP 的一种解决方 serviceCIDR: - 10.233.0.0/18 ``` - + 如果状态为 `NotReady`,这将会阻止 Pod 被创建。目前 Spiderpool: * 优先通过查询 `kube-system/kubeadm-config` ConfigMap 获取集群的 Pod 和 Service 子网。 * 如果 `kubeadm-config` 不存在导致无法获取集群子网，那么 Spiderpool 会从 `Kube-controller-manager Pod` 中获取集群 Pod 和 Service 的子网。 如果您集群的 Kube-controller-manager 组件以 `systemd` 等方式而不是以静态 Pod 运行。那么 Spiderpool 仍然无法获取集群的子网信息。 @@ -112,37 +236,7 @@ Spiderpool 可用作 Underlay 网络场景下提供固定 IP 的一种解决方 一旦创建完成，Spiderpool 将会自动同步其状态。 -3. 在每个节点上配置 Open vSwitch 网桥。 - - 创建网桥并配置网桥，以 `eth0` 为例。 - - ```bash - ~# ovs-vsctl add-br br1 - ~# ovs-vsctl add-port br1 eth0 - ~# ip addr add /<子网掩码> dev br1 - ~# ip link set br1 up - ~# ip route add default via <默认网关IP> dev br1 - ``` - - 请把以上命令配置在系统行动脚本中，以在主机重启时能够生效 - - 创建后，可以在每个节点上查看到如下的网桥信息： - - ```bash - ~# ovs-vsctl show - ec16d9e1-6187-4b21-9c2f-8b6cb75434b9 - Bridge br1 - Port eth0 - Interface eth0 - Port br1 - Interface br1 - type: internal - Port veth97fb4795 - Interface veth97fb4795 - ovs_version: "2.17.3" - ``` - -4. 创建 SpiderIPPool 实例。 +3. 创建 SpiderIPPool 实例。 Pod 会从该 IP 池中获取 IP，进行 Underlay 的网络通讯，所以该 IP 池的子网需要与接入的 Underlay 子网对应。以下是创建相关的 SpiderIPPool 示例： @@ -162,7 +256,7 @@ Spiderpool 可用作 Underlay 网络场景下提供固定 IP 的一种解决方 EOF ``` -5. 验证安装： +4. 验证安装： ```bash ~# kubectl get po -n kube-system |grep spiderpool @@ -177,7 +271,7 @@ Spiderpool 可用作 Underlay 网络场景下提供固定 IP 的一种解决方 ~# ``` -6. Spiderpool 为简化书写 JSON 格式的 Multus CNI 配置，它提供了 SpiderMultusConfig CR 来自动管理 Multus NetworkAttachmentDefinition CR。如下是创建 Ovs SpiderMultusConfig 配置的示例： +5. Spiderpool 为简化书写 JSON 格式的 Multus CNI 配置，它提供了 SpiderMultusConfig CR 来自动管理 Multus NetworkAttachmentDefinition CR。如下是创建 Ovs SpiderMultusConfig 配置的示例： * 确认 ovs-cni 所需的网桥名称，本例子以 br1 为例: diff --git a/docs/usage/install/underlay/get-started-ovs.md b/docs/usage/install/underlay/get-started-ovs.md index d41ff2d8dc..41f3535eeb 100644 --- a/docs/usage/install/underlay/get-started-ovs.md +++ b/docs/usage/install/underlay/get-started-ovs.md @@ -38,6 +38,130 @@ Spiderpool can be used as a solution to provide fixed IPs in an Underlay network ~# systemctl restart NetworkManager ``` +## Configure Open vSwitch bridge on the node + +The following is an example of creating and configuring a persistent OVS Bridge. This article takes the `eth0` network card as an example and needs to be executed on each node. + +### Use netplan to persist OVS Bridge + +1. Create OVS Bridge + + ```bash + ~# ovs-vsctl add-br br1 + ~# ovs-vsctl add-port br1 eth0 + ~# ip link set br1 up + ``` + +2. After creating 12-br1.yaml in the /etc/netplan directory, run `netplan apply` to take effect. + + To ensure that br1 is still available in scenarios such as restarting the host, please check whether the eth0 network card is also managed by netplan. + + ```yaml: 12-br1.yaml + network: + version: 2 + renderer: networkd + ethernets: + br1: + addresses: + - "/" # 172.18.10.10/16 + ``` + +3. After creation, you can view the following bridge information on each node: + + ```bash + ~# ovs-vsctl show + ec16d9e1-6187-4b21-9c2f-8b6cb75434b9 + Bridge br1 + Port eth0 + Interface eth0 + Port br1 + Interface br1 + type: internal + Port veth97fb4795 + Interface veth97fb4795 + ovs_version: "2.17.3" + + ~# ip a show br1 + 208: br1: mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 + link/ether 00:50:56:b4:5f:fd brd ff:ff:ff:ff:ff:ff + inet 172.18.10.10/16 brd 172.18.255.255 scope global noprefixroute br1 + valid_lft forever preferred_lft forever + inet6 fe80::4f28:8ef1:6b82:a9e4/64 scope link noprefixroute + valid_lft forever preferred_lft forever + ``` + +### Use NetworkManager to persist OVS Bridge + +Persisting OVS Bridge through NetworkManager is a more general method that is not limited to operating systems. + +1. To use NetworkManager to persist OVS Bridge, you need to install the OVS NetworkManager plug-in. The example is as follows: + + ```bash + ~# sudo dnf install -y NetworkManager-ovs + ~# sudo systemctl restart NetworkManager + ``` + +2. Create ovs bridges, ports and interfaces. + + ```bash + ~# sudo nmcli con add type ovs-bridge conn.interface br1 con-name br1 + ~# sudo nmcli con add type ovs-port conn.interface br1-port master br1 con-name br1-port + ~# sudo nmcli con add type ovs-interface slave-type ovs-port conn.interface br1 master br1-port con-name br1-int + ``` + +3. Create another port on the bridge and select the eth0 NIC in the physical device as its Ethernet interface so that real traffic can flow on the network. + + ```bash + ~# sudo nmcli con add type ovs-port conn.interface ovs-port-eth0 master br1 con-name ovs-port-eth0 + ~# sudo nmcli con add type ethernet conn.interface eth0 master ovs-port-eth0 con-name ovs-port-eth0-int + ``` + +4. Configure and activate the ovs bridge. + + Configure the bridge by setting a static IP + + ```bash + ~# sudo nmcli con modify br1-int ipv4.method static ipv4.address "/<子网掩码>" # 172.18.10.10/16 + ``` + + Activate bridge + + ```bash + ~# sudo nmcli con down "eth0" + ~# sudo nmcli con up ovs-port-eth0-int + ~# sudo nmcli con up br1-int + ``` + +5. After creation, you can view information similar to the following on each node. + + ```bash + ~# nmcli c + br1-int dbb1c9be-e1ab-4659-8d4b-564e3f8858fa ovs-interface br1 + br1 a85626c1-2392-443b-a767-f86a57a1cff5 ovs-bridge br1 + br1-port fe30170f-32d2-489e-9ca3-62c1f5371c6c ovs-port br1-port + ovs-port-eth0 a43771a9-d840-4d2d-b1c3-c501a6da80ed ovs-port ovs-port-eth0 + ovs-port-eth0-int 1334f49b-dae4-4225-830b-4d101ab6fad6 ethernet eth0 + + ~# ovs-vsctl show + 203dd6d0-45f4-4137-955e-c4c36b9709e6 + Bridge br1 + Port ovs-port-eth0 + Interface eth0 + type: system + Port br1-port + Interface br1 + type: internal + ovs_version: "3.2.1" + + ~# ip a show br1 + 208: br1: mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 + link/ether 00:50:56:b4:5f:fd brd ff:ff:ff:ff:ff:ff + inet 172.18.10.10/16 brd 172.18.255.255 scope global noprefixroute br1 + valid_lft forever preferred_lft forever + inet6 fe80::4f28:8ef1:6b82:a9e4/64 scope link noprefixroute + valid_lft forever preferred_lft forever + ``` + ## Install Spiderpool 1. Install Spiderpool. @@ -88,7 +212,7 @@ Spiderpool can be used as a solution to provide fixed IPs in an Underlay network * Spiderpool prioritizes obtaining the cluster's Pod and Service subnets by querying the kube-system/kubeadm-config ConfigMap. * If the kubeadm-config does not exist, causing the failure to obtain the cluster subnet, Spiderpool will attempt to retrieve the cluster Pod and Service subnets from the kube-controller-manager Pod. - + If the kube-controller-manager component in your cluster runs in systemd mode instead of as a static Pod, Spiderpool still cannot retrieve the cluster's subnet information. If both of the above methods fail, Spiderpool will synchronize the status.phase as NotReady, preventing Pod creation. To address such abnormal situations, we can manually create the kubeadm-config ConfigMap and correctly configure the cluster's subnet information: @@ -110,37 +234,7 @@ Spiderpool can be used as a solution to provide fixed IPs in an Underlay network EOF ``` -3. To configure Open vSwitch bridges on each node: - - Create a bridge and configure it using `eth0`` as an example. - - ```bash - ~# ovs-vsctl add-br br1 - ~# ovs-vsctl add-port br1 eth0 - ~# ip addr add / dev br1 - ~# ip link set br1 up - ~# ip route add default via dev br1 - ``` - - Pleade include these commands in your system startup script to ensure they take effect after host restarts. - - After creating the bridge, you will be able to view its information on each node: - - ```bash - ~# ovs-vsctl show - ec16d9e1-6187-4b21-9c2f-8b6cb75434b9 - Bridge br1 - Port eth0 - Interface eth0 - Port br1 - Interface br1 - type: internal - Port veth97fb4795 - Interface veth97fb4795 - ovs_version: "2.17.3" - ``` - -4. Create a SpiderIPPool instance. +3. Create a SpiderIPPool instance. The Pod will obtain an IP address from the IP pool for underlying network communication, so the subnet of the IP pool needs to correspond to the underlying subnet being accessed. @@ -162,7 +256,7 @@ Spiderpool can be used as a solution to provide fixed IPs in an Underlay network EOF ``` -5. Verify the installation： +4. Verify the installation： ```bash ~# kubectl get po -n kube-system |grep spiderpool @@ -177,7 +271,7 @@ Spiderpool can be used as a solution to provide fixed IPs in an Underlay network ~# ``` -6. To simplify writing Multus CNI configuration in JSON format, Spiderpool provides SpiderMultusConfig CR to automatically manage Multus NetworkAttachmentDefinition CR. Here is an example of creating an ovs-cni SpiderMultusConfig configuration: +5. To simplify writing Multus CNI configuration in JSON format, Spiderpool provides SpiderMultusConfig CR to automatically manage Multus NetworkAttachmentDefinition CR. Here is an example of creating an ovs-cni SpiderMultusConfig configuration: * Confirm the bridge name for ovs-cni. Take the host bridge: `br1` as an example: