Skip to content

Commit

Permalink
chore(README): add some adjustments
Browse files Browse the repository at this point in the history
Signed-off-by: Valentin Maillot <[email protected]>
  • Loading branch information
vmaillot committed Nov 24, 2023
1 parent 9588a37 commit a29243b
Showing 1 changed file with 25 additions and 30 deletions.
55 changes: 25 additions & 30 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,8 @@
# Kubernetes etcd backup CronJob

This CronJob creates an POD which runs `/backup.sh` on a kubernetes cluster to create the described backup. After finishing, it copies the files to an configured PV and expires old backups according to its configuration.

The backup script generates a snapshot.db files with the date when it is performed:

This CronJob creates a Pod which runs `/backup.sh` on a Kubernetes cluster to create the described backup. After finishing, it copies the files to a configured PV and expires old backups according to its configuration.

The backup script generates a `snapshot.db` file with the date when it is performed.

## Installation

Expand All @@ -17,36 +15,33 @@ kubectl create namespace etcd-backup
If you run etcd in your cluster you can read the etcd configuration and the location of the required certificates from your clusters etcd pod. The following commands will give you the necessary information:

```
kubectl describe pod -n kube-system etcd-<name of your etcd pod> |less
kubectl describe pod -n kube-system etcd-<name of your etcd pod> | less
```

Get the IP address of the etcd endpoint and put it in the config map.
Then get the location of the following certificates:
Get the IP address of the etcd endpoint and put it in the config map. Then get the location of the following certificates:
- peer-cert-file etcd-peer.crt
- peer-key-file etcd-peer.key
- trusted-ca-file etcd-ca.crt

If you run etcd outside of your cluster, you can get the information from the etcd configuration file. The default location is `etc/etcd.env`. The certificate information is in the TLS section. You need the `ETCD_ADVERTISE_CLIENT_URLS`, `ETCD_PEER_TRUSTED_CA_FILE`, `ETCD_PEER_CERT_FILE` and `ETCD_PEER_KEY_FILE` variables. The following example shows the default values:
If you run etcd outside of your cluster, you can get the information from the etcd configuration file. The default location is `/etc/etcd.env`. The certificate information is in the TLS section. You need the `ETCD_ADVERTISE_CLIENT_URLS`, `ETCD_PEER_TRUSTED_CA_FILE`, `ETCD_PEER_CERT_FILE` and `ETCD_PEER_KEY_FILE` variables. The following example shows the default values:

- ETCD_ADVERTISE_CLIENT_URLS=https://192.168.122.151:2379
- ...
- ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
- ETCD_PEER_CERT_FILE=/etc/ssl/etcd/ssl/member-node1.pem
- ETCD_PEER_KEY_FILE=/etc/ssl/etcd/ssl/member-node1-key.pem



Get the certificates from the kubernetes host and put them into a secret:
Get the certificates from the Kubernetes host and put them into a secret:
```
kubectl create secret generic etcd-peer-tls --from-file=tls.crt --from-file=tls.key -n etcd-backup
kubectl create secret generic etcd-peer-tls --from-file=tls.crt --from-file=tls.key -n etcd-backup
kubectl create secret generic etcd-server-ca --from-file=service-ca.crt -n etcd-backup
```
Add the endpoint IP address to the config map, without `https://` or port:

Add the endpoint IP address to the [ConfigMap](./backup-config.yaml), without scheme or port:
```
ENDPOINT: "192.168.122.151"
```


### Create the backup configuration

Then adjust the storage configuration to your needs in `backup-storage.yaml` and deploy it. The example uses NFS but you can use any storage class you want:
Expand All @@ -59,29 +54,28 @@ Configure the backup-script:
kubectl create -f backup-config.yaml
```


Then deploy, and configure the cronjob:
Then deploy the CronJob:
```
kubectl create -f backup-cronjob.yaml
```

## Creating manual backup / testing
## Creating manual backup for testing purpose

To test the backup, or create an manual backup, you can run a job:
To test the backup or create an manual backup you can run a job:
```
backupName=$(date "+etcd-backup-manual-%F-%H-%M-%S")
kubectl create job --from=cronjob/etcd-backup ${backupName}
```

To see if everything works as it should, you can check the logs:
To see if everything works as it should you can check the logs:
```
kubectl logs -l job-name=${backupName}
```
Then check on your Storage, if the files are there as excepted.

## Configuration

Configuration can be changed in configmap `backup-config`:
Configuration can be changed in the ConfigMap `backup-config`:

```
kubectl edit -n etcd-backup cm/backup-config
Expand All @@ -94,42 +88,43 @@ The following options are used:
- `ETCD_BACKUP_EXPIRE_TYPE`:
- `days`: Keep backups newer than `backup.keepdays`.
- `count`: Keep a number of backups. `backup.keepcount` is used to determine how much.
- `never`: Dont expire backups, keep all of them.
- `ETCD_BACKUP_KEEP_DAYS`: Days to keep the backup. Only used if `backup.expiretype` is set to `days`
- `ETCD_BACKUP_KEEP_COUNT`: Number of backups to keep. Only used if `backup.expiretype` is set to `count`
- `never`: Do not expire backups, keep all of them.
- `ETCD_BACKUP_KEEP_DAYS`: Days to keep the backup. Only used if `backup.expiretype` is set to `days`.
- `ETCD_BACKUP_KEEP_COUNT`: Number of backups to keep. Only used if `backup.expiretype` is set to `count`.
- `ETCD_BACKUP_UMASK`: Umask used inside the script to set restrictive permission on written files, as they contain sensitive information.
- `ENDPOINT`: The IP address of the etcd endpoint ,without port, e.g. `"192.168.39.86"`.
- `ENDPOINT`: The IP address of the etcd endpoint, without scheme or port, e.g. `"192.168.39.86"`.

Changing the schedule be done in the CronJob directly, with `spec.schedule`:
```
kubectl edit -n etcd-backup cronjob/etcd-backup
```
Default is `0 0 * * *` which means the cronjob runs one time a day at midnight.
Default is `0 0 * * *` which means the CronJob runs one time a day at midnight.

## Monitoring

To be able to get alerts when backups are failing or not being scheduled you can deploy this [PrometheusRule](https://github.com/adfinis-sygroup/openshift-etcd-backup/etcd-backup-cronjob-monitor.PrometheusRule.yaml).
To be able to get alerts when backups are failing or not being scheduled you can deploy this [PrometheusRule](https://github.com/adfinis/kubernetes-etcd-backup/blob/main/etcd-backup-cronjob-monitor.PrometheusRule.yaml).

```
kubectl create -n etcd-backup -f etcd-backup-cronjob-monitor.PrometheusRule.yaml
```

# Helm chart

To easily deploy the solution a helm chart is available on upstream Adfinis charts [repository](https://github.com/adfinis-sygroup/helm-charts/tree/master/charts/kubectl-etcd-backup).
To easily deploy the solution a Helm chart is available on upstream Adfinis charts [repository](https://github.com/adfinis-sygroup/helm-charts/tree/master/charts/kubernetes-etcd-backup).

## Installation

Fist create the namespace:
```
kubectl create namespace etcd-backup
```
Then create the secretes as described above.

Then create the secrets as described above.
Finally update the `values.yaml` file according to your needs.

```
helm repo add adfinis https://charts.adfinis.com
helm install etcd-backup adfinis/kubectl-etcd-backup
helm install etcd-backup adfinis/kubernetes-etcd-backup
```

## Development
Expand Down Expand Up @@ -162,4 +157,4 @@ If a commit does not contain a conventional commit style message you can fix
it during the squash and merge operation on the PR.

## References
* https://docs.openshift.com/container-platform/4.7/backup_and_restore/backing-up-etcd.html
* https://docs.openshift.com/container-platform/4.14/backup_and_restore/control_plane_backup_and_restore/backing-up-etcd.html

0 comments on commit a29243b

Please sign in to comment.