Rethinking k8s
Currently my Kubernetes cluster is based on Kubespray, with a single node on the control plane, dedicated to that task. The control node has experienced some hard drive errors which has caused it to go read-only a number of times, failing it out. Thinking about what to do has led me to thinking about a broader reconfiguration of my home data centre, with two major approaches.
Conservative option: Use LXC on the desktop
The easiest thing to do at this point would be to add an LXC container on the desktop system, install MicroK8s on it, and reconfigure the k8s gateway container already in that LXD cluster to point at it instead of the laptops. The desktop has two network cards and one is configured for the k8s network. I’ll keep this network but it’ll be virtual and I can even stop using the ancient 100Mbit “switch”.
This option is the least work, the least research, and the least risk. If I’d gone this way this article wouldn’t have a reason to exist. I’d just post “Retired my broken laptop k8s cluster today”, or something.
The primary reason I started thinking whether this was the right way to go is that this system only has 6GB of RAM, which is not that much when it’s split for several different virtual systems. There are five or six of them, depending.
Advantages:
- Easy, already know how, low risk
Disadvantages:
- Boring
- Doesn’t scale (hahahaha)
- Memory could well be an issue.
Bolder option: Move everything to the laptops
I’ve used the laptops as Kubernetes cluster, and that worked out okay, but this was a sort of traditional setup with a control node and several (three) worker nodes. Control nodes can also be worker nodes, and if they are, then each k8s node is no more or less important than any other, and with the right software, you could add or remove any node to grow or shrink the cluster as necessary. Recent-ish updates to MicroK8s seem to be moving this project in that direction, though while I’ve deployed it this way using pretty reliable virtual machines for a few months, the real proof will be whether this works as old, should-be-retired laptops shudder through their various aches and pains.
Storage is an issue because I will no longer have a central server that can provide storage. I didn’t do that before, but had services requiring local storage (like the database service) running on the desktop, which has mirrored hard drives. I can’t rely on the individual hard drives: one from the six original broken laptops has already died, and another is giving errors. So, I need persistent storage that is always available regardless of on what node a particular pod winds up.
I believe Gluster will do the trick. I’ve never
used it, but I’ve had to look at it from time to time and I have a vague
high-level understanding of its architecture. From my reading it can support
M mirrors for any non-zero value of M, so I should be able to expand my
cluster or shrink it to one node and always have the same data available on
the node.
Advantanges:
- Every node has access to the same storage
- More redundancy
- Nodes can withstand some power outage (though their switch won’t)
- It’s more interesting
- More storage, since each laptop has a larger hard drive
Disadvantages:
- Have to move Postgres to Kubernetes–maybe using CockroachDB
- Also have to move GitLab to Kubernetes–maybe also using CockroachDB?
So then what happened
It is somewhat inconvenient to work on the laptops in terms of both their physical frailty (all have fairly bad hinges, one has a dotty keyboard, one needs an external monitor) and physical access (they are in a basement storage room) and it would be a right pain to go to a bunch of effort and then find that the basic ideas don’t work at all.
So my plan is to build this out virtually, develop a procedure for deployment, and then shift that to the laptops. The environments won’t be quite the same, but it’ll eliminate a lot of the uncertainty and surface potential issues, even with the laptops' particular configurations, before I actually start pulling them out and working on them.