Using the first brick
Before I add bricks and expand the cluster, I want to see if I can use what I’ve got so far. I’m not concerned about the Kubernetes side, but as mentioned I have never used Gluster before, and I’m sure there are some kinks to work out.
My initial idea for Gluster was that, with all nodes transparently mirroring the same data, I could just deploy jobs to whatever nodes and the storage would be the same and it would just work. I might come back to this idea later, but it’s probably unworkable. Kubernetes probably guards against dispatching pods to nodes that have only a one-in-X nodes chance of having the local storage expected. The hostPath-type volume looks like it might be dumb enough to work–in that it may leave this potential issue up to the user to worry about. The local volume type demands setting up node affinity to ensure the expected storage is available.
One or both of these may well work, with the advantage of the pod being forced to write its data locally, minimizing network traffic. Whether they will work, and at what cost, is unknown. (As well, use may be quite non-intuitive, so that’s a drawback as well.) The lower-hanging fruit it just to use the GlusterFS volume type right off the bat.
Following the example, I create endpoints for the Gluster service, and then a basic pod that mounts a volume, and that’s basically it. Along the way I also reconfigure a reverse proxy already on this network to serve as go-between for my precious little cluster and the unwashed masses.
I haven’t used endpoints in
before. I’ve implemented pods and services on top of those: a pod puts a
container on a node, and a service creates a proxy on every node for that pod.
Endpoints (the k8s objects) are defined as collections of endpoints that
implement the “actual service”, so I think in my case they’ve been created for
me to help the service (object) reach the pod, and if I run
kubectl get endpoints that seems to be confirmed: there’s one for every pod I’ve created.
In this case I don’t create the service, because it already exists, but Kubernetes can’t just figure it out itself, so I create it. Following the example, I wind up with the following:
apiVersion: v1 kind: Endpoints metadata: name: glusterfs-cluster subsets: - addresses: - ip: 10.0.0.16 nodeName: brick0 ports: - port: 1
The example doesn’t specify the
nodeName, but according to
kubectl explain endpoints.subsets.addresses.nodeName, specifying nodeName “can be used to
determine endpoints local to a node.” This sounds to me like the pod will
prioritize the local Gluster mount, but who knows. I’ll leave it in there for
I don’t need to create the Gluster pod, but I do need to create the Gluster service which proxies the endpoints on every node:
apiVersion: v1 kind: Service metadata: name: glusterfs-cluster spec: ports: - port: 1
I haven’t designated namespaces for anything yet so for now everything’s going in the default namespace. I am not sure at this point how I will want to manage that. In my development k8s clusters I use namespaces keep things tidy, not for keeping one user from clobbering another user’s resources, but there’s still the question of how I’d make this available to all namespaces and if there’s a way to carve it up so I couldn’t accidentally clobber myself. I’ll want to come back to this.
I want to create something whizbang simple here and I have just the use case: webserver. So I create an Nginx deployment and a service for it. It looks like this:
kind: Deployment metadata: labels: app: nginx name: nginx spec: replicas: 1 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:latest volumeMounts: - mountPath: "/usr/share/nginx/html" name: glustervol ports: - containerPort: 80 protocol: TCP volumes: - name: glustervol glusterfs: endpoints: glusterfs-cluster path: gv0 readOnly: true restartPolicy: Always --- apiVersion: v1 kind: Service metadata: name: nginx spec: ports: - nodePort: 32000 port: 32000 protocol: TCP targetPort: 80 selector: app: nginx type: NodePort
I save this to a file and
kubectl apply -f <file>. And this works pretty
much right away. I put an index file on the Gluster volume and it gets served
when the service is
path specification refers to the name of the Gluster volume, but I
didn’t create this volume just for serving web pages, so I create a directory
in the volume,
/website, and add that to the path specification. The
container is reinitialized and once the index file is moved there, it’s
served up as expected.
So this is looking pretty good so far.
The reverse proxy
I have a virtual machine which shields the k8s cluster from the mean old world, taking requests and routing them appropriately to the underlying service. It’s a simple matter to map incoming requests to the k8s API server using HAProxy.
I’ve added the following snippets to the default config that came with the distribution’s package (in this case, CentOS but who cares):
frontend k8s-ctrl bind *:6443 mode tcp option tcplog default_backend k8s-ctrl backend k8s-ctrl balance roundrobin mode tcp server brick0 10.0.0.16:16443 check
Obviously port 6443 must have very limited exposure. Also, MicroK8s uses port 16443 for its API server instead of the more common 6443, and I still with 6443 on the frontend, just for consistency with other k8s clusters.
HAProxy works well for this sort of thing; for a single node I could also
simply forward the port using
iptables. But soon I’ll be adding more nodes
to the backend.
For mapping virtual hosts to the various servers, I’ve in the past had to use either Nginx or Apache for extra functionality provided by plugins available only for those platforms. For this cluster, none such are needed and HAProxy should be able to handle basic virtual hosts.
With this cluster I’d also like to explore the use of Ingress resources. With that I believe I’d be able to set up HAProxy to route X.k8s.example.org to backendservers/X for any service route “X” and then not have to add a new stanza to HAProxy every time I deploy a new service.
At this point I’ve proved to myself everything I want for the one-node cluster. Now let’s see how expanding it goes.