k3s-nixos
Since starting my PhD at CMU, I've set up a compute cluster on my lab desk for both research and personal use. It's pretty cool, so I thought I'd talk some about it.
• I'm using Kubernetes on k3s, with headscale and k3s secrets bootstrapped with agenix. Meaning, I can plug a computer into the internet, one-click install my nix config, and the new computer automatically connects up with my cluster. My SSH public key is automatically distributed as well.
• Using a simple ansible script, combined with NixOS's atomic configurations, I can modify, update, and manage all my machines at once.
• The NixOS setup my servers are using are actually a part of a larger mono-config, so features like my very decked out shell and editor configurations propagate
• I pay two bucks a month for an OVH VPS with unlimited transfer and an IPv4, and route all my traffic, including website serving, over headscale.
• I'm not picking up power nor networking costs, since my servers are on my desk at CMU
• I'm running several great pieces of software, including my websites through nginx, rook-ceph (which holds my brainrot, paper backups, and other things), gitea, Matrix, Iodine, a private docker registry, Syncthing, Hydra (for home-grown CI/CD), and some other experimental stuff.
• Because virtually everything is declarative, if my setup exploded tomorrow I could get everything back up and running in <1h (modulo ~20gb of packages). I'm confident everything could survive a move, or two, or ten.
It took me maybe a month of intermittent effort to get everything set up, and I've had to overcome several hurdles:
• Kubernetes initialization. I replicated some of the bare-linux settings required for kubernetes, described in this blog. This included disabling swap, adding some kernel modules, and changing some kernel-level network configurations.
• k8s vs k3s. I spent a couple weeks trying to get kubernetes working, but for whatever reason the networking and automatic registration just did not want to work consistently. I had much better luck with k3s^^
• Docker Registry. Kubernetes uses a local docker registry to pull images for my websites. Getting this set up was an incredible hassle. I ultimately needed to disable TLS (a fine compromise; my websites have no interesting data and are open source anyways) and whitelist the docker registry IP for k3s. Requests, however, are still ultimately authenticated.
• rook-ceph. Rook-ceph persistent volume claim management is a huge pain to set up, as there is no clear documentation for NixOS unfortunately. What I personally found most helpful here was the NixOS wiki surprisingly: nixos.wiki/wiki/Kubernetes. I had to go ahead and make the respective nixos-specific changes to
• Set up git-crypt
Features
• I'm using Kubernetes on k3s, with headscale and k3s secrets bootstrapped with agenix. Meaning, I can plug a computer into the internet, one-click install my nix config, and the new computer automatically connects up with my cluster. My SSH public key is automatically distributed as well.
• Using a simple ansible script, combined with NixOS's atomic configurations, I can modify, update, and manage all my machines at once.
• The NixOS setup my servers are using are actually a part of a larger mono-config, so features like my very decked out shell and editor configurations propagate
• I pay two bucks a month for an OVH VPS with unlimited transfer and an IPv4, and route all my traffic, including website serving, over headscale.
• I'm not picking up power nor networking costs, since my servers are on my desk at CMU
• I'm running several great pieces of software, including my websites through nginx, rook-ceph (which holds my brainrot, paper backups, and other things), gitea, Matrix, Iodine, a private docker registry, Syncthing, Hydra (for home-grown CI/CD), and some other experimental stuff.
• Because virtually everything is declarative, if my setup exploded tomorrow I could get everything back up and running in <1h (modulo ~20gb of packages). I'm confident everything could survive a move, or two, or ten.
Trials and Tribulations
It took me maybe a month of intermittent effort to get everything set up, and I've had to overcome several hurdles:
• Kubernetes initialization. I replicated some of the bare-linux settings required for kubernetes, described in this blog. This included disabling swap, adding some kernel modules, and changing some kernel-level network configurations.
• k8s vs k3s. I spent a couple weeks trying to get kubernetes working, but for whatever reason the networking and automatic registration just did not want to work consistently. I had much better luck with k3s^^
• Docker Registry. Kubernetes uses a local docker registry to pull images for my websites. Getting this set up was an incredible hassle. I ultimately needed to disable TLS (a fine compromise; my websites have no interesting data and are open source anyways) and whitelist the docker registry IP for k3s. Requests, however, are still ultimately authenticated.
• rook-ceph. Rook-ceph persistent volume claim management is a huge pain to set up, as there is no clear documentation for NixOS unfortunately. What I personally found most helpful here was the NixOS wiki surprisingly: nixos.wiki/wiki/Kubernetes. I had to go ahead and make the respective nixos-specific changes to
operator.yaml, but after that everything went well.
Todos (Oct 2025)
• Set up backup management• Set up git-crypt
← 2025