README.md

HomeOps

Features

• Talos bare-metal K8s OS
• Lots of self-hosted services
• Flux GitOps with this repository (kubernetes directory)
• Cilium container networking interface (CNI) and layer 4 loadbalancing
• go-task shorthand for useful commands (Taskfile and taskfiles) for multiple clusters
• SOPS secrets stored in Git
• Renovate bot dependency updates
• Cloudflared HTTP tunnel
• K8s gateway for local DNS resolution to the cluster and NGINX ingress controller
• Both internal & external services with a service gateway
• OIDC authentication with LDAP
• Automatic Cloudflare DNS updates with external-dns
• CloudNative-PG with automatic failover
• kube-prometheus-stack with various Grafana dashboards
• Rook Ceph cluster storage

This setup is inspired by this homelab gitops template. You can find similar setups with the k8s at home search.

See also my homelab repo for how I provision machines in my home.

Usage

Setup

Install go-task

task init

Provision the Talos nodes.

task talos:bootstrap

Install flux.

task flux:{verify,bootstrap}

Verify the installation.

kubectl -n flux-system get pods -o wide

task kubernetes:resources

DNS and Tunnel

Setup a Cloudflare Tunnel.

cloudflared tunnel login
cloudflared tunnel create cluster

Add the tunnel's credentials.json to the value in cloudflared-secret and tunnel ID to cluster-secrets.sops.yaml.

Add a Cloudflare API token with these permissions to the value in external-dns-secret.

• Zone - DNS - Edit
• Account - Cloudflare Tunnel - Read

Github Webhook

Setup a webook to reconcile flux when changes are pushed to Github. Note: this only works with Let's Encrypt Production certificates.

Get webook path:

kubectl -n flux-system get receiver github-receiver -o jsonpath='{.status.webhookPath}'

Append to self-hosted domain:

https://flux-webhook.${DOMAIN}/hook/12ebd1e363c641dc3c2e430ecf3cee2b3c7a5ac9e1234506f6f5f3ce1230e123

Generate a webook token openssl rand -hex 16 and add to secret: kubernetes/<cluster>/apps/flux-system/webhooks/app/github/secret.sops.yaml.

Add the webook to the repository's "Settings/Webhooks" > "Add webhook" button. Add the URL and token.

Directories

This Git repository contains the following directories under Kubernetes. Check out cluster-template for more details on how this FluxCD setup works.

📁 kubernetes
├── 📁 main # main cluster
│   ├── 📁 apps # applications
│   ├── 📁 bootstrap # bootstrap procedures
│   ├── 📁 flux # core flux configuration
│   └── 📁 templates # re-useable components
└── 📁 ...

Deployments

Most helm deployments in this repo utilize this useful app-template chart.

Hardware

Resources

• DataHoarder Wiki: Hardware
• 2020 DIY NAS
• 2020 Economy DIY NAS
• Home server case recommendations

Memory

• Why use ECC (discussion)
• If you love your data, use ECC RAM.
• Error rates increase rapidly with rising altitude.

Storage

Controller

I used a widely-known and inexpensive method to add additional SATA storage via a Host Bus Adapter (HBA). I purchased a Dell Perc H310 a long while back. Mine did come from overseas, but it turned out to be legit. This video shows how it can be flashed to an LSI 9211-8i IT (see also 1, 2).

Here are other recommended controllers.

2.5" drive stackers

These printable stackers are great for stacking SSDs in a homelab.

Raspberry Pi cluster

One cluster uses Raspberry Pi 4B (x 5) but the 4 GB RAM models are hungry for more memory. Micro SD cards are insufficient for etcd's demanding read/writes, so I recommend SATA over USB 3.0. Check out this guide for compatible SSD interfaces. I use a PicoCluster case.

Home automation

IoT

• USB Zigbee/Z-Wave receiver and upgrade Zigbee firmware for compatibility with Home Assistant (notice this issue)
• Zigbee/Matter receiver

Software

See my homelab repo for how I provision proxmox, SSH keys, dotfiles and other tasks in my home.

Check disks

Here's a handy script to automatically test disks with badblocks and SMART: Spearfoot/disk-burnin-and-testing.

Testing disks takes a long time for larger drives, but it's worth it to be thorough before determining whether to make a return. This is a destructive test, so it's probably best to use /dev/disk/by-id to be certain you're targeting the correct drive.

Use tune2fs -l <partition> to identify the block size.

sudo badblocks -wsv -b 4096 /dev/sda > sda_badblocks.txt

Here's some additional advice from /r/DataHoarders.

JBOD

MergerFS is a union filesystem for pooling drives together. It's a great pair with SnapRAID. An alternative is SnapRAID-BTRFS.

mkdir /mnt/disk{1,2,3,4}
mkdir /mnt/parity1 # adjust this command based on your parity setup
mkdir /mnt/storage # this will be the main mergerfs mount point (a collection of your drives)

Mount drives to these folders, then add /etc/fstab entries by ID.

ls /dev/disk/by-id

You must also include an entry for the MergerFS union, such as:

/mnt/disk* /mnt/storage fuse.mergerfs allow_other,use_ino,cache.files=partial,dropcacheonclose=true,category.create=mfs,fsname=mergerfs,minfreespace=10G 0 0

See also perfectmediaserver: MergerFS

For data that's irreplaceable RAID is not a backup.

ZFS

Install zfs-dkms and zfs-utils, and be sure to have linux-headers installed for dkms to work. Update the ZFS libraries together using a AUR helper.

OS Installation

Use Ventoy to bundle bootable ISO and IMG images on a single USB.

Setup Proxmox on the hosts with Arch Linux guests. Post setup for Proxmox.

Media

For a media server, it's a good idea to understand digital video.