Consuming File System artifacts from Kubernetes Pods

When you are deploying an application which contains artifacts written on file system dynamically withing kubernetes (k8s), for example a tomcat server exposed to outside to deploy war files, you need to make sure the file system state is preserved always. Otherwise if the pod goes down, you might loose data.

So one solution is to mount an external disk. Yes indeed you can do that. But how robust is that solution. Say something happened to the external disk. How can you recover the data? Use several disks and rsync to sync the data. Sounds a robust solution. Say you want to increase the reliability. And what happens if rsync process get killed. How much will it cost to make it's reliability closer to 100%?

We have a robust, simple solution. It's using gluster to save data. [1] [2]

We install a pod named gluster for each node. There is an additional disk attached to each node which will be used as the data storage for gluster. This disk is formatted in a special format and attached to the gluster.

This disk can be attached to a specific path and we can mention it as below (eg :- /dev/sdb) in the topology.json. Following is a sample. When your run gkdeploy script to install gluster, you have to place topology.json in the same location.

{  
   "clusters":[  
      {  
         "nodes":[  
            {  
               "node":{  
                  "hostnames":{  
                     "manage":[  
                        "node0"
                     ],
                     "storage":[  
                        "192.168.10.100"
                     ]
                  },
                  "zone":1
               },
               "devices":[  
                  "/dev/sdb"
               ]
            },
            {  
               "node":{  
                  "hostnames":{  
                     "manage":[  
                        "node1"
                     ],
                     "storage":[  
                        "192.168.10.101"
                     ]
                  },
                  "zone":1
               },
               "devices":[  
                  "/dev/sdb"
               ]
            },
            {  
               "node":{  
                  "hostnames":{  
                     "manage":[  
                        "node2"
                     ],
                     "storage":[  
                        "192.168.10.102"
                     ]
                  },
                  "zone":1
               },
               "devices":[  
                  "/dev/sdb"
               ]
            }
         ]
      }
   ]
}

Once you install gluster in k8s now comes the question on how they interconnect? There is an another component called heketi [3] which is generally used to manage volumes. Here heketi is used to manage gluster cluster. You can use heketi cli [4] to do advanced operations. Heketi is included in the gk-deploy script from glusterfs.

Note that this blog is written to explain a problem and a solution. Here I haven't included any low level operational details or installation details.

Once gluster is installed in your k8s setup, next question is to how to use it? First you need to create a storage class in k8s with the heketi cli url. Here this url should be accessible to master node of k8s deployment. It looks like below.

	kind: StorageClass
	apiVersion: storage.k8s.io/v1beta1
	metadata:
	name: glusterfs-storage
	provisioner: kubernetes.io/glusterfs
	parameters:
	resturl: "http://${HEKETI_URL}"

After that there should be a pvc (Persisted Volume Claim) to use this storage class.

	apiVersion: v1
	kind: PersistentVolumeClaim
	metadata:
	name: gluster-pvc
	annotations:
	volume.beta.kubernetes.io/storage-class: glusterfs-storage
	spec:
	accessModes:
	- ReadWriteOnce
	resources:
	requests:
	storage: 2Gi

Then you can use pvc in your pod conflagration as a volume mount.

	volumeMounts:
	-
	mountPath: /mnt/files/diff
	name: sample-files

	volumes:
	-
	name: synapse-files
	persistentVolumeClaim:
	claimName: gluster-pvc

Hope this helped someone to create a solution.
Good luck on deployment!!!

[1] https://github.com/gluster/glusterfs
[2] https://github.com/gluster/gluster-kubernetes
[3] https://github.com/heketi/heketi
[4] https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/container-native_storage_for_openshift_container_platform/chap-documentation-red_hat_gluster_storage_container_native_with_openshift_platform-heketi_cli