Using the first brick

Before I add bricks and expand the cluster, I want to see if I can use what I’ve got so far. I’m not concerned about the Kubernetes side, but as mentioned I have never used Gluster before, and I’m sure there are some kinks to work out.

My initial idea for Gluster was that, with all nodes transparently mirroring the same data, I could just deploy jobs to whatever nodes and the storage would be the same and it would just work. I might come back to this idea later, but it’s probably unworkable. Kubernetes probably guards against dispatching pods to nodes that have only a one-in-X nodes chance of having the local storage expected. The hostPath-type volume looks like it might be dumb enough to work–in that it may leave this potential issue up to the user to worry about. The local volume type demands setting up node affinity to ensure the expected storage is available.

One or both of these may well work, with the advantage of the pod being forced to write its data locally, minimizing network traffic. Whether they will work, and at what cost, is unknown. (As well, use may be quite non-intuitive, so that’s a drawback as well.) The lower-hanging fruit it just to use the GlusterFS volume type right off the bat.

Following the example, I create endpoints for the Gluster service, and then a basic pod that mounts a volume, and that’s basically it. Along the way I also reconfigure a reverse proxy already on this network to serve as go-between for my precious little cluster and the unwashed masses.

Endpoints

I haven’t used endpoints in Kubernetes before. I’ve implemented pods and services on top of those: a pod puts a container on a node, and a service creates a proxy on every node for that pod. Endpoints (the k8s objects) are defined as collections of endpoints that implement the “actual service”, so I think in my case they’ve been created for me to help the service (object) reach the pod, and if I run kubectl get endpoints that seems to be confirmed: there’s one for every pod I’ve created.

In this case I don’t create the service, because it already exists, but Kubernetes can’t just figure it out itself, so I create it. Following the example, I wind up with the following:

apiVersion: v1
kind: Endpoints
metadata:
  name: glusterfs-cluster
subsets:
- addresses:
  - ip: 10.0.0.16
    nodeName: brick0
  ports:
  - port: 1

The example doesn’t specify the nodeName, but according to kubectl explain endpoints.subsets.addresses.nodeName, specifying nodeName “can be used to determine endpoints local to a node.” This sounds to me like the pod will prioritize the local Gluster mount, but who knows. I’ll leave it in there for now.

I don’t need to create the Gluster pod, but I do need to create the Gluster service which proxies the endpoints on every node:

apiVersion: v1
kind: Service
metadata:
  name: glusterfs-cluster
spec:
  ports:
  - port: 1

I haven’t designated namespaces for anything yet so for now everything’s going in the default namespace. I am not sure at this point how I will want to manage that. In my development k8s clusters I use namespaces keep things tidy, not for keeping one user from clobbering another user’s resources, but there’s still the question of how I’d make this available to all namespaces and if there’s a way to carve it up so I couldn’t accidentally clobber myself. I’ll want to come back to this.

Pod

I want to create something whizbang simple here and I have just the use case: webserver. So I create an Nginx deployment and a service for it. It looks like this:

kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        volumeMounts:
        - mountPath: "/usr/share/nginx/html"
          name: glustervol
        ports:
        - containerPort: 80
          protocol: TCP
      volumes:
      - name: glustervol
        glusterfs:
          endpoints: glusterfs-cluster
          path: gv0
          readOnly: true
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  ports:
  - nodePort: 32000
    port: 32000
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: NodePort

I save this to a file and kubectl apply -f <file>. And this works pretty much right away. I put an index file on the Gluster volume and it gets served when the service is curled.

The path specification refers to the name of the Gluster volume, but I didn’t create this volume just for serving web pages, so I create a directory in the volume, /website, and add that to the path specification. The container is reinitialized and once the index file is moved there, it’s served up as expected.

So this is looking pretty good so far.

The reverse proxy

I have a virtual machine which shields the k8s cluster from the mean old world, taking requests and routing them appropriately to the underlying service. It’s a simple matter to map incoming requests to the k8s API server using HAProxy.

I’ve added the following snippets to the default config that came with the distribution’s package (in this case, CentOS but who cares):

frontend k8s-ctrl
  bind    *:6443
  mode    tcp
  option  tcplog
  default_backend k8s-ctrl

backend k8s-ctrl
  balance  roundrobin
  mode     tcp
  server   brick0 10.0.0.16:16443 check

Obviously port 6443 must have very limited exposure. Also, MicroK8s uses port 16443 for its API server instead of the more common 6443, and I still with 6443 on the frontend, just for consistency with other k8s clusters.

HAProxy works well for this sort of thing; for a single node I could also simply forward the port using iptables. But soon I’ll be adding more nodes to the backend.

For mapping virtual hosts to the various servers, I’ve in the past had to use either Nginx or Apache for extra functionality provided by plugins available only for those platforms. For this cluster, none such are needed and HAProxy should be able to handle basic virtual hosts.

With this cluster I’d also like to explore the use of Ingress resources. With that I believe I’d be able to set up HAProxy to route X.k8s.example.org to backendservers/X for any service route “X” and then not have to add a new stanza to HAProxy every time I deploy a new service.

At this point I’ve proved to myself everything I want for the one-node cluster. Now let’s see how expanding it goes.

Comments