Kubernetes 101 : Storage

Jigar R

5 min read

·

1 day ago

--

Listen

Share

Press enter or click to view image in full size

This post is about Storage options in Kubernetes.

In past, we have discussed

• part 1 — API Server, scheduler, etcd database and kubelet at high level
• part 2 — resource limits on pod, replicaset and deployment

In part-2, we looked into how to rollout updates. But what happens to the data? In this post, we will focus on storage.

In previous posts, I mentioned that pods get scheduled on one of the node. So far we haven’t specified any storage information; but our pods would still read/write data. Kubelet and container-runtime allocates an isolated area on the node for a pod. Any data written to it is deleted whenever pod terminates or moves to another node. This is why it’s also called ephemeral storage. We can also utilize node’s memory using following YAML file:

apiVersion: v1
kind: Pod
metadata:
  name: test-pd
spec:
  containers:
  - image: registry.k8s.io/test-webserver
    name: test-container
    volumeMounts:
    - mountPath: /cache
      name: cache-volume
  volumes:
  - name: cache-volume
    emptyDir:                     <-- emptyDir is the type of the volume
      sizeLimit: 500Mi
      medium: Memory              <-- without this we will write on disk 

If we need to keep the data, then we have several options. In the following section, I will present some solutions along with their limitations to set the stage for better solution.

We can also provide a dedicated disk drive to the pod. For example, we can specify that /data/foo should be used for pod-a. We can do so using hostPath volume. Let’s say pod-a got scheduled on node-B; kubelet and runtime will try to bind /data/foo for pod-a but if it does not find it then the container would not start. We might be tempted to use nodeSelector and always schedule pod-a on node-a. But this is not an ideal solution.

Instead, Imagine storing data outside of the node — on an external drive. Whenever a new pod gets created, you assign a dedicated disk to the pod. And this disk can be attached to whatever node the pod is scheduled. Now, we don’t depend on a specific host. This is exactly what we can get using a Persistent Volume.

Claiming storage for a pod

Different applications need different types of storage options. For example, a write-heavy database application needs a completely different type of storage than a web application. Therefore, depending on the use case, pod may require different types of persistent storage. These storages are either static or dynamic (created-on-demand).

Kubernetes administrators create

• Storage classes — to enable creation of Persistent Volumes dynamically
• Static Persistent Volume — for applications that require specific configuration (for example, NFS server, EBS volume that was created prior to adopting Kubernetes)

On the other hand, typical Kubernetes users don’t need to worry about the background details. They would specify

• create claim persistent volume by creating PVC (Persistent Volume Claim)
• linking claim to the Pod

In the following case, we have a storage class ebs-gp3. A PVC app-data-pvc is created and later linked to demo-pod.

# see storage classes 
$ kubectl get storageclass

NAME       PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
ebs-gp3    ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   5m

# create pvc specific to a storage class 
$ cat demo-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-data-pvc
spec:
  # storageClassName: ebs-gp3 <-- one can specify specific storage class
  #                              when unspecified default storage class is used
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
      

# linking claim to the pod
# cat demo-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: demo-pod
spec:
  containers:
  - name: app
    image: busybox:1.36
    command: ["sh", "-c", "echo 'hello from persistent storage' > /data/hello.txt; sleep 3600"]
    volumeMounts:
    - name: app-data           # <-- container needs app-data volume
      mountPath: /data
  volumes:
  - name: app-data             # <-- app-data volume is linked to 
    persistentVolumeClaim:
      claimName: app-data-pvc  #     the pvc that we created earlier

# a persistent volume would be created
$ kubectl get pv
NAME         CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                  STORAGECLASS   AGE
pvc-1axxxx   5Gi        RWO            Delete           Bound    default/app-data-pvc   ebs-gp3        35s

## To view storage class related information use 
$ kubectl describe storageclass ebs-gp3
$ kubectl get storageclass ebs-gp3 -o yaml

Scenario 1 : The pod ran into error and got scheduled on a new pod, will my data survive?

Yes, the data will survive. Even if the user deletes the pod! But if someone deletes the pvc then we will be in trouble. This is because we have “Delete” reclaim policy on the Persistent-Volume (PV).

Scenario 2: The pod filled up the disk and it can not write any more data

In this case, if our storage is expandable then we just need to update related pvc.

# see storage classs
$ kubectl get storageclass

NAME       PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
ebs-gp3    ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   5m

# in our case ALLOWVOLUMEEXPANSION is true

Scenario 3: The pod runs two containers; can they share the volume?

Yes they can; see following sample YAML file

apiVersion: v1
kind: Pod
metadata:
  name: two-container-app
spec:
  containers:
  - name: app
    image: my-app:1.0
    volumeMounts:
    - name: app-data
      mountPath: /var/lib/app

  - name: log-shipper
    image: busybox:1.36
    command: ["sh", "-c", "tail -F /logs/app.log"]
    volumeMounts:
    - name: app-data
      mountPath: /logs
      readOnly: true

  volumes:
  - name: app-data
    persistentVolumeClaim:
      claimName: app-data-pvc

Scenario 4: Instead of creating pods, I am using deployment with 3 replicas. What would happen?

Our PVC specifies “ReadWriteOnce” meaning only one node can mount the volume. So if nodeA mounted volume first then replicas scheduled on nodeA would be able to use the volume. Other replicas would stay in waiting state.

# create pvc specific to a storage class 
$ cat demo-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-data-pvc
spec:
  accessModes:
    - ReadWriteOnce             <-- see access mode
  resources:
    requests:
      storage: 5Gi

There are 4 access modes

• ReadWriteOnce
• ReadOnlyMany — many nodes can mount volume as read-only
• ReadWriteMany
• ReadWriteOncePod

We might be tempted to switch to “ReadWriteMany” mode, but it can create conflicts if both replicas open and write the same file. Most apps can work without having a persistent-volume. In case if we need one then we would have to make sure that each pod write in a different file/directory.

The problem would be even worse for critical applications like databases. This is where “Statefulsets” are useful.

======

In this post, we covered following key concepts

• ephemeral storage
• hostpath volume type
• Persistent Volume Claim, Persistent Volume and Storage Class

References:

• https://kubernetes.io/docs/concepts/storage/volumes/
• https://kubernetes.io/docs/concepts/storage/persistent-volumes/
• https://kubernetes.io/docs/tasks/configure-pod-container/configure-volume-storage/
• https://kubernetes.io/docs/tutorials/configuration/configure-persistent-volume-storage/
• j_cadmus, CC BY 2.0, via Wikimedia Commons

Other related posts

• https://jigarr.medium.com/list/kubernetes-f5d0178b859a