Rancher Restore failed

SUSE Rancher Prime
1

I’m testing Rancher for the last couple of weeks, and it looks really promising. However, I hit a snag when trying to restore Rancher. Here’s my setup.

Rancher-1 (single node cluster, running k3s) - version 2.10.1
kubectl version
Client Version: v1.31.4+k3s1
Kustomize Version: v5.4.2
Server Version: v1.31.4+k3s1

Apps (charts) installed

  • Longhorn
  • Cert-Manager
  • Rancher Backup

Use it to create a new “RKE2” cluster (on another VM). Copied the command provided and ran it on another VM.

Registered an existing (vanilla K8s) cluster with Rancher.
So, this Rancher has 3 clusters:

  • Local (Rancher)
  • RKE2
  • Vanilla K8s

Then, I created a Rancher backup. Downloaded the tarball from the PVC (manually searched for that file) to my laptop.

Rancher-2 (single node cluster, running k3s) - version 2.10.2
kubectl version:
Client Version: v1.31.5+k3s1
Kustomize Version: v5.4.2
Server Version: v1.31.5+k3s1

Deployed this second rancher on another VM.
Installed the Cert-Manager and Rancher-Backup app.
Checked the ‘rancher-backup’ deployment in the ‘cattle-resources-system’ namespace. It is using the local-path storageClass.

I exec into this rancher-backup pod and cd /var/lib/backups and created a dummy file.
Then, I searched for that filename on the host to locate the actual location on the filesystem.
I then uploaded the rancher backup (from Rancher-1) to this location, and create a ‘Restore’ from the Rancher UI.

It failed with the following errors:

ERRO[2025/02/10 03:07:53] Error restoring namespaced resources [error restoring rke2 of type provisioning.cattle.io/v1, Resource=clusters: restoreResource: err updating resource admission webhook “rancher.cattle.io.clusters.provisioning.cattle. io” denied the request: creatorID annotation cannot be changed error restoring k8s-v130 of type provisioning.cattle. io/v1, Resource=clusters: restoreResource: err updating resource admission webhook “rancher.cattle.io.clusters.provisioning.cattle. io” denied the request: creatorID annotation cannot be changed error restoring local of type provisioning.cattle.io/v1, Resource=clusters: restoreResource: err updating resource admission webhook “rancher.cattle.io.clusters.provisioning.cattle. io” denied the request: creatorID annotation cannot be changed]
ERRO[2025/02/10 03:07:53] error syncing ‘restore-x62vn’: handler restore: error restoring namespaced resources, check logs for exact error, requeuing

Same error above repeats again and again.
What logs is it asking me to check? The above error came from the ‘rancher-backup’ pod.

Any help/tips is much appreciated.

Thanks!

2

Check the restore docs :

==> restore into a new cluster. Note that the order ofr restoration and rancher install is important.

3

Thanks @bpedersen2 ! I didn’t realize one should run the restore first before installing Rancher. But hit another snag … on this new host, I’ve installed k3s and also the rancher-backup via helm

There is no volumeMounts to specify a location for the pod/container to access the backup tarball.
— snippet of the deployment ‘rancher-backup’ in the cattle-resources-system ns -----

spec:                                                                             
  containers:                                                                     
  - env:                                                                          
    - name: CHART_NAMESPACE                                                       
      value: cattle-resources-system                                              
    image: rancher/backup-restore-operator:v6.0.0                                 
    imagePullPolicy: Always                                                       
    name: rancher-backup                                                          
    resources: {}                                                                 
    terminationMessagePath: /dev/termination-log                                  
    terminationMessagePolicy: File                                                
  dnsPolicy: ClusterFirst

The deployment manifest on the original Rancher has a volumeMount section:

spec:
  containers:
  - env:
    - name: CHART_NAMESPACE
      value: cattle-resources-system
    image: rancher/backup-restore-operator:v6.0.0
    imagePullPolicy: Always
    name: rancher-backup
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/lib/backups
      name: local-storage
      :
  volumes:
  - name: local-storage
    persistentVolumeClaim: 
      claimName: pvc-data

So, I modified the rancher-backup deployment on the new host and copied the backup tarball to the PVC/PV location (using the k3s local-path storage-class).

Created the restore.yaml

apiVersion: resources.cattle. io/v1
kind: Restore
metadata:
  name: restore-migration
  annotations:
spec:
  backupFilename: rancher-backup-s0.tar.gz
  deleteTimeoutSeconds: 10
  prune: false

Get this error mesg:
kubectl logs -n cattle-resources-system --tail 100 -f -l app.kubernetes. io/instance=rancher-backup
INFO[2025/02/11 02:08:58] Starting backup-restore controller version v6.0.0 (afac4f6)
INFO[2025/02/11 02:08:58] No PVC or S3 details provided for storing backups by default. User must specify storageLocation on each Backup CR
:
INFO[2025/02/11 02:13:04] Restoring from backup rancher-backup-s0.tar.gz
ERRO[2025/02/11 02:13:04] error syncing ‘restore-migration’: handler restore: Backup location not specified on the restore CR, and not configured at the operator level, requeuing

So, it is saying “No PVC or S3 details provided for storing backups by default. User must specify storageLocation on each Backup CR”

kubectl explain restore.spec.storageLocation showed it accepts only ‘s3’ - nothing about PVC or directory paths.

Can one do a restore without a S3 bucket?

4

Ok, I give up - no more hairs left to pull :confounded:

I setup a MinIO server and then store the rancher-backup file at the s3 bucket.
Created the restore resource and applied it.

It works as described at https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster

Thanks again @bpedersen2 for that link.

If anyone knows whether a restore can be done without needing a s3 bucket, please let me know.