Storage in Kubernetes

By default, containers running in pods on Kubernetes have access to the filesystem, but as we're about to find out, there are some big limitations to this.

Assignment

The api application in SynergyChat can be configured to save its data (the messages) to a file on the filesystem. That way, even if the program is restarted, the messages will still be there.

Update the api service's ConfigMap to include a new environment variable:

API_DB_FILEPATH: /var/lib/synergychat/api/db.json

Next, update the api service's deployment to use the new value.

Take a look at the logs of the new pod (kubectl logs <podname>). You should see a message saying that the filesystem will be used for the database. If you do, everything is working as intended!

Now, let's test the persistent storage:

Open a webpage to http://synchat.internal.
Create a few messages as different users.
Refresh the page
The messages should still be there because they're saved in the server's memory.
Now, delete the api pod
Once k8s replaces the deleted pod with a new one, refresh the page again.

Perhaps to your surprise, the messages are gone! What??? Why??? We saved them to the filesystem, didn't we? Well, we did, but in Kubernetes the filesystem is ephemeral. That means that when a pod is deleted, the filesystem is deleted with it.

This has to do with the philosophy behind Kubernetes and even containers in general: when we spin up a new one, it should always be a blank slate, which makes reproducing and debugging issues much easier. No messy state to worry about.

So how do we get persistent storage??? We'll cover that next.

Saving data to a containers's filesystem will persist the data between pod recreations.