Announcing self-service for restores: A small step for UX, giant implications for the future

Davor Gajic Dez 13, 2021

In early November we introduced a feature enabling all of our Nine managed GKE customers to restore their backed up Kubernetes resources themselves.

In this blog, we will examine how Nine backups work, look at how you can create your own restores and take a look into the future regarding self-service and products at Nine .

How does Nine back up my data

When we talk about data, it is important to note that we are talking about Kubernetes cluster data, e.g. configuration and persistent block storage.

Non-GKE services like CloudSQL have their own backup plans and cannot be restored via this method.

Nine uses Velero which is an open-source tool, that allows the back up, restore and migration of k8s resources.

Velero consists of two parts:

Firstly, the Velero server that runs in each Nine managed GKE cluster. Secondly,a command-line client that runs locally on your machine.

By default, Nine backs up the whole cluster daily at 1 AM UTC and the backups are stored for 30 days.

If you wish to have more frequent backup of certain namespaces or the entire cluster, we can additionally configure that for you.

This is configured through the Velero “Schedule” custom resource in the k8s cluster:

 1apiVersion: velero.io/v1
 2kind: Schedule
 3metadata:
 4  name: velero-dailybackup
 5  namespace: nine-velero
 6spec:
 7  schedule: 0 1 * * *
 8  template:
 9    includedNamespaces:
10    - '*'
11    ttl: 720h0m0s

As usual, the interesting part is in the spec, so let’s take a look how the schedule can be configured:

spec.schedule

Here, we are using the cron notation:

1  # everyday at 1 AM
2  schedule: 0 1 * * *

Each of the positions have a different meaning:

Position Period Values
1 Minute 0-59,*
2 Hour 0-23,*
3 Day of Month 1-31,*
4 Month 1-12,*
5 Day of Week 0-7,*

This allows us to configure the schedules as we want. Here are some examples:

 1  # all two hours
 2  schedule: "0 */2 * * *"
 3
 4  # daily at 11 AM
 5  schedule: "0 11 * * *"
 6
 7  # each sunday at 4 AM
 8  schedule: "0 4 * * 0"
 9
10  # Monthly each 1st of the month at 8 AM
11  schedule: "0 8 1 * *"

spec.template.includedNamespaces

This field controls which namespaces should be backed up by the schedule.

1  template:
2    # all namespaces
3    includedNamespaces:
4    - '*'
5
6    # test and stage namespaces
7    includedNamespaces:
8    - test-ns
9    - stage-ns

spec.template.ttl

The TTL field allows us to specify how long the backup should be stored. Velero automatically deletes all backups older than the TTL value.

1    # 24h * 30
2    ttl: 720h0m0s

Summing this up, if you want a custom backup schedule e.g. monthly backup and storing that backup for a year, you can let us know and we will configure this for you.

Restorative Justice

Backups have existed since day one for Nine managed GKE, however, the restores were out of the customer’s reach.

If you wanted to restore your data, you had to create either a ticket at our service desk or even worse, if you wanted a restore outside our office hours, you had to call us incurring an on-call fee.

While we are always happy to help you out, your inconvenience of always having to contact us was a thorn in our side. We had to find a solution to give you more freedom while maintaining the role of the trustful contact.

Solution

From now on, our customers can manually create their own restores from backups.

To create your own restore, you have to install the Velero client locally. The installation guide can be found here.

Once installed, the following command can be executed on the cluster to list the backups:

1$ velero backup get -n nine-velero
2-------------------------------
3
4NAME                                STATUS      ERRORS   WARNINGS   CREATED                          EXPIRES   STORAGE LOCATION   SELECTOR
5velero-dailybackup-20211122010018   Completed   0        0          2021-11-22 02:00:18 +0100 CET    29d       default            <none>

We found the backup velero-dailybackup-20211122010018 and now we can create a restore out of it.

There are generally two approaches to create restores:

  • restoring into a newly created namespace
  • restoring into an existing namespace

For this example, let’s say we have the following namespaces in our cluster:

1$ kubectl get namespace
2-------------------------------
3
4nine-velero   Active   2y50d
5production    Active   2y50d
6stage         Active   2y50d

Restoring into newly created namespace

Let’s create a namespace:

1$ kubectl create namespace production-restore
2$ kubectl get namespace
3-------------------------------
4
5nine-velero           Active   2y50d
6production            Active   2y50d
7stage                 Active   2y50d
8production-restore    Active   1s

We want to now restore our production namespace into the newly created production-restore namespace.

1$ velero restore create -n nine-velero --from-backup velero-dailybackup-20211122010018 --include-namespaces production --namespace-mappings production:production-restore

Let’s take a look at the important parameters:

–include-namespaces

Here, Velero is given the information about which of the backed up namespaces should be in the restore.

You can add multiple namespaces, by using comma-separated values:

1$ velero --include-namespaces production,stage ...

–namespace-mappings

The namespace mappings parameter controls how the namespace in the backup will be mapped to the desired restore namespace.

The form we have to use is:

1src1:dst1,src2:dst2,...

Restoring into an existing namespace

The problem with restoring resources in an already existing namespace is that Velero will not overwrite existing resources in it.

We have to either delete the complete namespace to make sure all the resources will be restored, or we have to individually delete each resource that we want to restore.

For example:

1# delete an entire namespace
2$ kubectl delete namespace production
3
4# delete a pod and a deployment in the production namespace
5$ kubectl delete pod mypod-123
6$ kubectl delete deployment mydep

To create a restore now into the production namespace, we can run the following command:

1$ velero restore create -n nine-velero --from-backup velero-dailybackup-20211122010018 --include-namespaces production

If we delete the namespace, all the resources will now be restored. If we only delete the pod and the deployment, only those two resources will be restored.

Open questions

If you have any open questions, you can either take a look at our support pages or ask us via a service desk ticket. We are always glad to help!

Beyond the horizon

While self-servicing restores might not be the biggest or the most special feature of all times, it does show what the future for Nine managed products holds.

The mission is clear: Give our customers the ability to be more agile and not be dependent on our reaction time.

If you want a restore, you can get it right now! There is no need to create a ticket or even talk to our customer service desk when you need a restore in either a rollback or a critical emergency situation, the power to make these changes is now firmly in your hands.

This flexibility and independence, while really attractive to some, might not be everyone’s cup of tea. Some customers will not want to create their own restores. And this is okay. While we focus on providing our customers with more options to take action themselves to automate the use of our products as well as to enable DevOps oriented and automated workflows, our close contact to the customer is and will stay of the utmost importance.