Multi Region Deployment

[euh2]

I have a K8s cluster spanning two regions and quiet a few nodes in each region. The RTT between the two regions is around 7ms. Decent given the geographical distance. I first tried setting up a Ceph Cluster with Rook. But the Ping time is too large for Ceph to work properly. Ceph is not the right choice for my setup and requirements.
Actually I only need the option to failover from one primary region to a secondary (standby) region. Storage only needs a replica of 2. One in the primary region and the second in the standby region.

So before I set up DRBD, is this actually doable with this operator?

I know I can set affinity and such on pods, but is it possible to instruct this controller to replicate my volumes to the second region? I don't need in-region replication. All satellites within one region can have Diskless DRBDs.

I've worked with DRBD on non-k8s before, but I don't see any obvious answer to this scenario.

[WanzenBug]

I believe this is possible. See the examples here: https://linbit.com/drbd-user-guide/linstor-guide-1_0-en/#s-kubernetes-volume-accessibility-and-locality

Specifically, I believe you are looking for

replicasOnDifferent: topology.kubernetes.io/region

You probably are also looking for some DRBD tuning (using protocol C is probably also too slow).

[euh2]

Thanks! This is exactly what I need!

Now I'm having trouble with the InitContainer drbd-module-loader.

Need a git checkout to regenerate drbd/.drbd_git_revision
make[1]: Entering directory '/tmp/pkg/drbd-9.2.6/drbd'

    Calling toplevel makefile of kernel source tree, which I believe is in
    KDIR=/lib/modules/6.1.0-18-cloud-amd64/build

cat /dev/null   > Module.symvers
make -C /lib/modules/6.1.0-18-cloud-amd64/build   M=/tmp/pkg/drbd-9.2.6/drbd  modules "PRE_CFLAGS="
/usr/src/linux-headers-6.1.0-18-common/Makefile:1108: /usr/src/linux-headers-6.1.0-18-common/scripts/Makefile.debug: No such file or directory
make[2]: *** No rule to make target '/usr/src/linux-headers-6.1.0-18-common/scripts/Makefile.debug'.  Stop.
make[1]: *** [Makefile:184: kbuild] Error 2
make[1]: Leaving directory '/tmp/pkg/drbd-9.2.6/drbd'
make: *** [Makefile:126: module] Error 2

Could not find the expexted *.ko, see stderr for more details

The VMs are based on Debian 12 Cloud Image. I assumed that the drbd-module-loader Container based on Debian 11 would still work, but maybe I'm wrong about that?

apiVersion: piraeus.io/v1
kind: LinstorSatelliteConfiguration
metadata:
  name: overrides
spec:
  podTemplate:
    spec:
      initContainers:
        - name: drbd-module-loader
          image: quay.io/piraeusdatastore/drbd9-bullseye:v9.2.6

# uname -r
6.1.0-18-cloud-amd64

Maybe the issue is the cloud image itself:

# cat /usr/src/linux-headers-6.1.0-18-cloud-amd64/Makefile
include /usr/src/linux-headers-6.1.0-18-common/Makefile

The error above suggests: /usr/src/linux-headers-6.1.0-18-common/scripts/Makefile.debug: No such file or directory. But this file exists:

# ls /usr/src/linux-headers-6.1.0-18-common/scripts/Makefile.debug
/usr/src/linux-headers-6.1.0-18-common/scripts/Makefile.debug

Thankful for any help here!

[WanzenBug]

I guess we just have to build a new version of the image for debian 12:
https://github.com/piraeusdatastore/piraeus/tree/master/dockerfiles/drbd-driver-loader
Shouldn't be any different than the already existing versions. Would you be able to build your own images to test?

[euh2]

It seems to work now!

I followed the Docs to build a Container based on Debian 12:

FROM debian:bookworm

ARG DRBD_VERSION=9.2.6

RUN apt-get update && apt-get upgrade -y && apt-get install -y kmod gnupg wget make gcc patch elfutils curl 'linux-kbuild-*' && apt-get clean

RUN wget https://pkg.linbit.com/downloads/drbd/9/drbd-${DRBD_VERSION}.tar.gz -O /drbd.tar.gz && \
    wget https://raw.githubusercontent.com/LINBIT/drbd/master/docker/entry.sh -O /entry.sh && chmod +x /entry.sh

ENV LB_HOW compile
ENTRYPOINT /entry.sh

[WanzenBug]

Should be upstream soon: piraeusdatastore/piraeus#174