I currently run all my self-hosted apps either on Podman in a VM or in LXCs on Proxmox. For hardware, I’m using a Chinese-made mini computer with an Intel N150 and 16GB of DDR5 RAM that I bought before the whole AI hype started. I also have a Synology NAS that I use mainly for media and photo storage.
I’ve been thinking about tinkering with Kubernetes in my homelab for a while now (I already use it extensively at work, so I’m quite familiar with it), and I started looking around for used hardware to use as bare metal nodes. Nothing fancy—I’m looking for 1 or 2 mini servers or SFF with at least 16GB of memory and a decent CPU (4–6 cores). But with current prices, even decently priced used hardware (~200–250€) is quite difficult to find in Europe, and most of it is HP stuff with Lenovo being a rare breed around here. I won’t even get started on newly bought hardware…
If you’ve bought hardware in this market recently, how did it go for you? Or are most of you holding out for now, waiting for better times?
I run a single node cluster.
My single node has 256 gb of ram and 24 cores. I do this because, if you want a lot of ram/cores/storage, it is cheaper to get a used “tower server” type device and then upgrade it as you go over time, than it is to buy entirely new devices for every bit of ram you want to add to the cluster.
I like kubernetes because I like configuration as code, gitops, the way it abstracts over components so I can swap components out easily, the way that helm charts are an easier way of orchestrating containers, and a bunch of other things.
Clustering is merely one of many benefits of kubernetes, one that isn’t particularly important to me. Although, my opinion on that has changed somewhat recently. Waiting for a reboot is annoying, since I am rebooting the whole thing and I have to wait for each service to go down or come up before the machine reboots properly. But if I was running kubernetes as a virtual machines inside incus with multiple nodes, I could update each node one by one without the whole thing going down. Or, I could snapshot them, allowing me to reboot the host without waiting for kubernetes. But these things are mostly just somewhat nice to have, rather than a core feature I really require.
have you found many self-hosted services that suppprt that kind of HA? I can’t imagine services like torrent clients allowing you to stream writes to one node while replicating to the other, though maybe I’m misunderstanding the setup
They don’t have to support it. It’s more that the database they are connecting to supports HA, and/or both are using shared storage. So when one container dies, kubernetes restarts another container on another node with the same shared storage attached.
Docker is often configured to automatically restart containers when a container dies, just in case it’s a one off bug or something like that, and kubernetes is like a more resilient version of that.
I’m aware of databases that support HA, but the vast majority of self-hosted apps I’ve encountered use file storage, even if they have a database as well. It sounds like you’re proposing shared storage like an NFS share. But if you’re upgrading nodes, at some point you have to upgrade the node hosting the shared storage right? Wouldn’t that take down all services? Unless you use a distributed storage system, but I’ve heard those can get very complicated…
Kubernetes makes distributed storage easy.
Basically, all the components get deployed for you, since that’s part of what kubernetes is good at.
And then, services/containers can provision storage by requesting storage via making a “claim” and whatever distributed storage providee you have gives it to you.
Have you tried using Ceph or other distributed storage systems in your kubernetes cluster?
Oh wow okay, if I’d go down that route I would definitely do multiple VMs on that host. In my opinion, the whole clustering and self healing / HA aspect of Kubernetes is why I want to switch to it. I can do gitops with Podman/Docker as well, in fact I already do that including a renovate pipeline on my sel-hosted Forgejo instance. But having the redundancy of several nodes, if one goes down the service will still work or at least will be re-deployed within a couple of seconds, provided there is distributed storage (longhorn, ceph or even nfs cis) if its a stateful app.