rebuild-from-scratch

Rebuilding from etcd backup

When would I need to rebuild from an etcd backup?

Kuberentes uses etcd for storing the all of cluster data including deployments, secrets, configmap, etc. RKE allows us to take an etcd backup and rebuild a cluster using the backup to seed the data inside the cluster. Then is very helpful for disaster situation like a large scale storage outage or accidental deletion for cluster.

Reproducing in a lab

• Prerequisites
  • Latest RKE
  • Latest kubectl
  • 6 VMs (2 core, 4GB of RAM, 20GB root)
  • SSH access to root on all nodes
  • Internet access to github and docker hub.
  • S3 bucket.
  • Running Docker Install Script
• Edit the cluster.yml to include your node IPs and S3 settings

  vi ./cluster.yml
  

• Stand up the cluster

  bash ./build.sh
  

• Verify the cluster is up and healthy

  bash ./verify.sh
  

• Break the cluster

  bash ./break.sh
  

Restoring/Recovering

• Edit the cluster.yml to comment out the 3 orginal nodes and uncomment the 3 new nodes.

  vi ./cluster.yml
  

• NOTE: If S3 is not an option, you can use local etcd backups but you'll need to rsync /opt/rke/etcd-snapshots from the old nodes to the new nodes.
• Using the following script to restore the etcd data from S3 into 3 new nodes.

  bash ./restore.sh SnapshotNameHere
  

Documention

• Restoring from Backup
• Recurring Snapshots