Xcluster - Experimental Cluster Environment

A very lightweight and configurable cluster environment primarily intended for development and test of network functions.

To see how xcluster can be used with Kubernetes please see the Quick Start section. See also the Kubernetes overlay.

An xcluster consists of a number of identical (kvm) VMs. The disk image is shared among the VMs and the qemu-img "backing_file" function is used to allow individual writes (much like a layered file-system). On start specified packages (overlays) are installed.

The VMs are given "roles" depending on their hostname;

vm-001 - vm-200   Cluster nodes
vm-201 - vm-220   Router VMs
vm-221 - vm-240   Tester VMs
vm-250 -          Reserved

More info;

• Quick Start
• 100 VMs
• Troubleshooting
• Misc info. Prettify xterms, use master-branch, and more...
• Networking. Default network and DNS setup.
• Network name space. Setup a netns for running xcluster.
• Overlays. How they work and how they are created
• Overlay index
• Disk-image and kernel. How they are created and extended.
• Build from scratch. If the binary release can't be used.
• Xcluster for CI. Headless operation.
• Local docker registry.
• Test
• Fedora. Not maintained.
• Alpine image in xcluster

Overlays

In the earliest stage of start all VMs are mounting an iso image (cdrom) that may contain any number of tar-files called "overlays". The files are unpacked on root "/" in sort order and provides a primitive packaging system. The iso image is created from ovl directories. Example;

# Start an xcluster with overlays (xc is an alias for the xcluster.sh script)
xc mkcdrom iptools; xc start

NOTE; You do not normally install SW on a running xcluster, instead you create an overlay, include it in xc mkcdrom and re-start. Since re-start is so fast it is slower and clumsier to copy sw to running VMs, e.g. with ssh. read more.

To create a new ovl from a template:

xcadmin mkovl --template=template-k8s my-ovl

The ovls are created in $HOME/tmp/ovl by default.

Execution environment and dependencies

Xcluster is developed on Ubuntu 22.04 LTS (>=7.0.0). It should also work on Ubuntu 20.04 LTS but this is not tested. It will probably not work on other distributions (mainly due to variations in libs). Xcluster can be started on Fedora but this environment is not maintained. If you run on another distribution than Ubuntu 22.04 you may run into problems with pre-built images and cached overlays from the binary release when you add own programs (because of library version probems). In that case there may be no other option than to rebuild all images and overlays locally from scratch.

First you must be able to run a kvm;

> kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
# If it looks as above; fine! go on...
# If "kvm-ok" does not exist, install as below;
sudo apt install -y qemu-kvm
# To start kvm you must be in the "kvm" group;
sudo usermod -aG kvm $USER
id
# (you may have to logout/in to enable the new group)

Some additional packets may have to be installed. Below is the bare minimum for development you will need more;

sudo apt install -y xterm genisoimage jq screen

For image handling you will also need docker.

Environment and default options

Options to the xcluster functions can be specified on the command line as "long options" for instance --nrouters=2 or as an environment variable;

xc start --nrouter=2
# same as;
__nrouters=2 xc start
# same as;
export __nrouters=2
xc start

This makes it very easy to set default options. The current environment settings and default options can be printed with;

xc env
# If you need to use the settings in a script do;
eval $($XCLUSTER env)

Timezone

The timezone in the VMs is set (by you) using the timezone overlay.

The $ARCHIVE variable

Xcluster uses the $ARCHIVE directory to store for instance downloaded archives. It defaults to $HOME/Downloads but you might want something better.

Quick start

Verify that kvm is installed and can be used and install dependencies if necessary (see above);

kvm-ok
id     # (you must be member of the "kvm" group and preferably "docker")
sudo apt install -y xterm pxz genisoimage jq   # (if necessary)

To get a k8s cluster with dual-stack running do;

XCDIR=$HOME/tmp   # Change to your preference
mkdir -p $XCDIR
cd $XCDIR
ver=<latest-xcluster-release>
curl -L https://github.com/Nordix/xcluster/releases/download/$ver/xcluster-$ver.tar.xz | tar xJ
cd $XCDIR/xcluster
. ./Envsettings.k8s
curl -L http://artifactory.nordix.org/artifactory/cloud-native/xcluster/images/hd-k8s.img.xz | xz -d > $__image
xc mkcdrom; xc start   # (no xterms? See below) (use "xc starts" to start without xterms)
vm 2     # Opens a terminal on vm-002

# In the terminal (on cluster) test things, for example;
kubectl get nodes  # (may take ~10 sec to appear)
kubectl version --short
kubectl get node vm-002 -o json | jq .spec   # "podCIDRs" is dual-stack
nslookup www.google.se   # (doesn't work? See below)
wget -O /dev/null http://www.google.se # (doesn't work? See below)
# Traffic test with mconnect
kubectl apply -f /etc/kubernetes/mconnect/mconnect.yaml # (image is pre-pulled)
kubectl get svc
assign-lb-ip -svc mconnect-lb -ip 10.0.0.0
kubectl get svc
mconnect -address mconnect.default.svc.xcluster:5001 -nconn 100

No xterms? Start with "xtermopt=-hold xc start" to keep the window. See also the troubleshooting doc

Nslookup doesn't work? See DNS-troubleshooting

External access does not work (wget http://www.google.se)? Check your host firewall settings. If this does not work then images can not be loaded from external sites (docker.io).

Run a test-suite;

log=/tmp/$USER-xcluster.log
xcadmin k8s_test test-template basic > $log

Suggested reading;

• ovl/private-reg setup a local private registry
• ovl/k8s-xcluster to use different CNI-plugins

K8s test and development with xcluster

NOTE: xcluster is intended for test and development of network functions. It is not intended for general k8s application development.

If you want to use xcluster for some more serious test/development you must run xcluster inside an own netns as described here. In netns "bridged" networking is used, see the networking topology.

While the user-space networking works (sort of) the performance is horrible, slow and lossy. This will cause all sorts of weird problems that you may not relate to poor network performance at first.

While not a requirement it is strongly recommended that you use a private docker registry. Set the XOVLS to automatically include the private-reg ovl;

export XOVLS="private-reg"

To keep up with xcluster updates use a clone rather than a release as described here.