Since starting my PhD at CMU, I've set up a compute cluster on my lab desk for both research and personal use. It's pretty cool, so I thought I'd talk some about it.

Features

• I'm using Kubernetes on k3s, with headscale and k3s secrets bootstrapped with agenix. Meaning, I can plug a computer into the internet, one-click install my nix config, and the new computer automatically connects up with my cluster. My SSH public key is automatically distributed as well.
• Using a simple ansible script, combined with NixOS's atomic configurations, I can modify, update, and manage all my machines at once.
• The NixOS setup my servers are using are actually a part of a larger mono-config, so features like my very decked out shell and editor configurations propagate
• I pay two bucks a month for an OVH VPS with unlimited transfer and an IPv4, and route all my traffic, including website serving, over headscale.
• I'm not picking up power nor networking costs, since my servers are on my desk at CMU
• I'm running several great pieces of software, including my websites through nginx, rook-ceph (which holds my brainrot, paper backups, and other things), gitea, Matrix, Iodine, a private docker registry, Syncthing, Hydra (for home-grown CI/CD), and some other experimental stuff.
• Because virtually everything is declarative, if my setup exploded tomorrow I could get everything back up and running in <1h (modulo ~20gb of packages). I'm confident everything could survive a move, or two, or ten.

Trials and Tribulations

It took me maybe a month of intermittent effort to get everything set up, and I've had to overcome several hurdles:

• Kubernetes initialization. I replicated some of the bare-linux settings required for kubernetes, described in this blog. This included disabling swap, adding some kernel modules, and changing some kernel-level network configurations.
• k8s vs k3s. I spent a couple weeks trying to get kubernetes working, but for whatever reason the networking and automatic registration just did not want to work consistently. I had much better luck with k3s^^
• Docker Registry. Kubernetes uses a local docker registry to pull images for my websites. Getting this set up was an incredible hassle. I ultimately needed to disable TLS (a fine compromise; my websites have no interesting data and are open source anyways) and whitelist the docker registry IP for k3s. Requests, however, are still ultimately authenticated.
• rook-ceph. Rook-ceph persistent volume claim management is a huge pain to set up, as there is no clear documentation for NixOS unfortunately. What I personally found most helpful here was the NixOS wiki surprisingly: nixos.wiki/wiki/Kubernetes. I had to go ahead and make the respective nixos-specific changes to operator.yaml, but after that everything went well.

Todos (Oct 2025)

• Set up backup management
• Set up git-crypt