Experiment pods cannot use Kubernetes "ClusterIP" Services (like kube-dns)

[stephen-soltesz]

Since very early, M-Lab experiment pod configurations required explicit reset of the content of /etc/resolv.conf to use nameserver 8.8.8.8 instead of the default kube-dns service IP address.

Recently we discovered that there may be a fundamental limitation with our pod network configuration that combines multus with ipvlan to provide "passthrough" access to the public network. Because pod traffic has direct access to the public network, the iptables rules managing k8s Services are bypassed, this includes access to the kube-dns service.

[nkinkade]

M-Lab's experiment pods (ndt, wehe, etc.) use the ipvlan CNI plugin to assign a public IP to a virtual interface inside the pod. The ipvlan plugin configures a "passthrough" connection between the virtual interface and an interface on the host in the default network namespace (eth0 in our case). The default route in experiment pods is the ipvlan interface. Traffic across the ipvlan interface does not traverse iptables, totally bypassing anything magic kube-proxy sets up to handle ClusterIP Services.

Currently, our experiment pods don't have a need for cluster Services, other than DNS. We have historically got around the inability to use cluster DNS (provided by a CoreDNS deployment) by explicitly setting the nameserver in /etc/resolv.conf to 8.8.8.8.

Flannel traffic can still work inside of experiment pods, as there are routes for 192.168.0.0/16 which route out the cni0 bridge interface to the host.