In early November we introduced a feature enabling all of our Nine managed GKE customers to restore their backed up Kubernetes resources themselves.
In this blog, we will examine how Nine backups work, look at how you can create your own restores and take a look into the future regarding self-service and products at Nine .
How does Nine back up my data
When we talk about data, it is important to note that we are talking about Kubernetes cluster data, e.g. configuration and persistent block storage.
Non-GKE services like CloudSQL have their own backup plans and cannot be restored via this method.
Nine uses Velero which is an open-source tool, that allows the back up, restore and migration of k8s resources.
Velero consists of two parts:
Firstly, the Velero server that runs in each Nine managed GKE cluster. Secondly,a command-line client that runs locally on your machine.
By default, Nine backs up the whole cluster daily at 1 AM UTC and the backups are stored for 30 days.
If you wish to have more frequent backup of certain namespaces or the entire cluster, we can additionally configure that for you.
This is configured through the Velero “Schedule” custom resource in the k8s cluster:
1apiVersion: velero.io/v1 2kind: Schedule 3metadata: 4 name: velero-dailybackup 5 namespace: nine-velero 6spec: 7 schedule: 0 1 * * * 8 template: 9 includedNamespaces: 10 - '*' 11 ttl: 720h0m0s
As usual, the interesting part is in the spec, so let’s take a look how the schedule can be configured:
spec.schedule
Here, we are using the cron notation:
1 # everyday at 1 AM 2 schedule: 0 1 * * *
Each of the positions have a different meaning:
Position | Period | Values |
---|---|---|
1 | Minute | 0-59,* |
2 | Hour | 0-23,* |
3 | Day of Month | 1-31,* |
4 | Month | 1-12,* |
5 | Day of Week | 0-7,* |
This allows us to configure the schedules as we want. Here are some examples:
1 # all two hours 2 schedule: "0 */2 * * *" 3 4 # daily at 11 AM 5 schedule: "0 11 * * *" 6 7 # each sunday at 4 AM 8 schedule: "0 4 * * 0" 9 10 # Monthly each 1st of the month at 8 AM 11 schedule: "0 8 1 * *"
spec.template.includedNamespaces
This field controls which namespaces should be backed up by the schedule.
1 template: 2 # all namespaces 3 includedNamespaces: 4 - '*' 5 6 # test and stage namespaces 7 includedNamespaces: 8 - test-ns 9 - stage-ns
spec.template.ttl
The TTL field allows us to specify how long the backup should be stored. Velero automatically deletes all backups older than the TTL value.
1 # 24h * 30 2 ttl: 720h0m0s
Summing this up, if you want a custom backup schedule e.g. monthly backup and storing that backup for a year, you can let us know and we will configure this for you.
Restorative Justice
Backups have existed since day one for Nine managed GKE, however, the restores were out of the customer’s reach.
If you wanted to restore your data, you had to create either a ticket at our service desk or even worse, if you wanted a restore outside our office hours, you had to call us incurring an on-call fee.
While we are always happy to help you out, your inconvenience of always having to contact us was a thorn in our side. We had to find a solution to give you more freedom while maintaining the role of the trustful contact.
Solution
From now on, our customers can manually create their own restores from backups.
To create your own restore, you have to install the Velero client locally. The installation guide can be found here.
Once installed, the following command can be executed on the cluster to list the backups:
1$ velero backup get -n nine-velero 2------------------------------- 3 4NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR 5velero-dailybackup-20211122010018 Completed 0 0 2021-11-22 02:00:18 +0100 CET 29d default <none>
We found the backup velero-dailybackup-20211122010018
and now we can create a
restore out of it.
There are generally two approaches to create restores:
- restoring into a newly created namespace
- restoring into an existing namespace
For this example, let’s say we have the following namespaces in our cluster:
1$ kubectl get namespace 2------------------------------- 3 4nine-velero Active 2y50d 5production Active 2y50d 6stage Active 2y50d
Restoring into newly created namespace
Let’s create a namespace:
1$ kubectl create namespace production-restore 2$ kubectl get namespace 3------------------------------- 4 5nine-velero Active 2y50d 6production Active 2y50d 7stage Active 2y50d 8production-restore Active 1s
We want to now restore our production namespace into the newly created
production-restore
namespace.
1$ velero restore create -n nine-velero --from-backup velero-dailybackup-20211122010018 --include-namespaces production --namespace-mappings production:production-restore
Let’s take a look at the important parameters:
–include-namespaces
Here, Velero is given the information about which of the backed up namespaces should be in the restore.
You can add multiple namespaces, by using comma-separated values:
1$ velero --include-namespaces production,stage ...
–namespace-mappings
The namespace mappings parameter controls how the namespace in the backup will be mapped to the desired restore namespace.
The form we have to use is:
1src1:dst1,src2:dst2,...
Restoring into an existing namespace
The problem with restoring resources in an already existing namespace is that Velero will not overwrite existing resources in it.
We have to either delete the complete namespace to make sure all the resources will be restored, or we have to individually delete each resource that we want to restore.
For example:
1# delete an entire namespace 2$ kubectl delete namespace production 3 4# delete a pod and a deployment in the production namespace 5$ kubectl delete pod mypod-123 6$ kubectl delete deployment mydep
To create a restore now into the production
namespace, we can run the
following command:
1$ velero restore create -n nine-velero --from-backup velero-dailybackup-20211122010018 --include-namespaces production
If we delete the namespace, all the resources will now be restored. If we only delete the pod and the deployment, only those two resources will be restored.
Open questions
If you have any open questions, you can either take a look at our support pages or ask us via a service desk ticket. We are always glad to help!
Beyond the horizon
While self-servicing restores might not be the biggest or the most special feature of all times, it does show what the future for Nine managed products holds.
The mission is clear: Give our customers the ability to be more agile and not be dependent on our reaction time.
If you want a restore, you can get it right now! There is no need to create a ticket or even talk to our customer service desk when you need a restore in either a rollback or a critical emergency situation, the power to make these changes is now firmly in your hands.
This flexibility and independence, while really attractive to some, might not be everyone’s cup of tea. Some customers will not want to create their own restores. And this is okay. While we focus on providing our customers with more options to take action themselves to automate the use of our products as well as to enable DevOps oriented and automated workflows, our close contact to the customer is and will stay of the utmost importance.