# Error after migration

**URL:** <https://forums.suse.com/t/error-after-migration/21426>\
**Category:** SUSE Storage\
**Created:** [October 14, 2021, 4:33pm UTC](https://forums.suse.com/t/error-after-migration/21426 "2021-10-14T16:33:52Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Marx\_N](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/marx_n/32/8124_2.png) [@Marx\_N](https://forums.suse.com/u/Marx_N)\
**Post date:** [October 14, 2021, 4:33pm UTC](https://forums.suse.com/t/error-after-migration/21426/1 "2021-10-14T16:33:53Z")

</div>

Hello, some time in the past after one of upgrade migration (I’m using currently 1.2.2) backup tab stopped to work, showing an error:

> error listing backup volume names: Failed to execute: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.2.2/longhorn [backup ls --volume-only nfs://omv:/backup/longhorn], output Failed to execute: ls [-1 /var/lib/longhorn-backupstore-mounts/omv/backup/longhorn/backupstore/volumes], output ls: cannot access ‘/var/lib/longhorn-backupstore-mounts/omv/backup/longhorn/backupstore/volumes’: Permission denied , error exit status 2 , stderr, time=“2021-10-14T16:29:34Z” level=warning msg=“failed to list first level dirs for path: backupstore/volumes reason: Failed to execute: ls [-1 /var/lib/longhorn-backupstore-mounts/omv/backup/longhorn/backupstore/volumes], output ls: cannot access ‘/var/lib/longhorn-backupstore-mounts/omv/backup/longhorn/backupstore/volumes’: Permission denied\n, error exit status 2” pkg=backupstore time=“2021-10-14T16:29:34Z” level=error msg=“Failed to execute: ls [-1 /var/lib/longhorn-backupstore-mounts/omv/backup/longhorn/backupstore/volumes], output ls: cannot access ‘/var/lib/longhorn-backupstore-mounts/omv/backup/longhorn/backupstore/volumes’: Permission denied\n, error exit status 2” , error exit status 1

Besides that, everything works as expected.

Here’s my deployment: [https://github.com/Marx2/homelab/blob/main/cluster/core/longhorn/helm-release.yaml](https://github.com/Marx2/homelab/blob/main/cluster/core/longhorn/helm-release.yaml)

Can you help me fix this problem?

Edit: I’m using 3 nodes  
Follder /var/lib/longhorn-backupstore-mounts/omv/backup/longhorn/backupstore/volumes exists only on one of them, and as a root I can’t access it:

> root@longhorn-manager-chksw:/# ls -al /var/lib/longhorn-backupstore-mounts/omv/backup/longhorn/backupstore/  
> ls: cannot open directory ‘/var/lib/longhorn-backupstore-mounts/omv/backup/longhorn/backupstore/’: Permission denied

---

<div class="post-metadata">

**Author:** ![JenTing](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/jenting/32/7290_2.png) [@JenTing](https://forums.suse.com/u/JenTing)\
**Post date:** [October 15, 2021, 1:01am UTC](https://forums.suse.com/t/error-after-migration/21426/2 "2021-10-15T01:01:19Z")

</div>

Could you please go to the node and try to umount the NFS mount point manually, Then, longhorn will remount it when it needs to access the remote backup target.

---

<div class="post-metadata">

**Author:** ![Marx\_N](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/marx_n/32/8124_2.png) [@Marx\_N](https://forums.suse.com/u/Marx_N)\
**Post date:** [October 15, 2021, 6:04am UTC](https://forums.suse.com/t/error-after-migration/21426/3 "2021-10-15T06:04:53Z")

</div>

Hi, I did it  
Mount is:

```auto
/dev/mapper/pve-root on /var/lib/longhorn-setting type ext4 (ro,relatime,errors=remount-ro)
omv:/backup/longhorn on /var/lib/longhorn-backupstore-mounts/omv/backup/longhorn type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.42.1.61,local_lock=none,addr=192.168.1.230)

```

After unmounting I went to Dashboard and it got remounted automatically like this:

```auto
/dev/mapper/pve-root on /var/lib/longhorn-setting type ext4 (ro,relatime,errors=remount-ro)
omv:/backup/longhorn on /var/lib/longhorn-backupstore-mounts/omv/backup/longhorn type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,acregmin=1,acregmax=1,acdirmin=1,acdirmax=1,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.42.1.61,local_lock=none,addr=192.168.1.230)

```

Unfortunatelly gooing int backup tab still shows the same error. It’s also visible in manager’s logs:

`time="2021-10-15T05:59:34Z" level=error msg="Error listing backup volumes from backup target" controller=longhorn-backup-target cred= error="error listing backup volume names: Failed to execute: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.2.2/longhorn [backup ls --volume-only nfs://omv:/backup/longhorn], output Failed to execute: ls [-1 /var/lib/longhorn-backupstore-mounts/omv/backup/longhorn/backupstore/volumes], output ls: cannot access '/var/lib/longhorn-backupstore-mounts/omv/backup/longhorn/backupstore/volumes': Permission denied\n, error exit status 2\n, stderr, time=\"2021-10-15T05:59:34Z\" level=warning msg=\"failed to list first level dirs for path: backupstore/volumes reason: Failed to execute: ls [-1 /var/lib/longhorn-backupstore-mounts/omv/backup/longhorn/backupstore/volumes], output ls: cannot access '/var/lib/longhorn-backupstore-mounts/omv/backup/longhorn/backupstore/volumes': Permission denied\\n, error exit status 2\" pkg=backupstore\ntime=\"2021-10-15T05:59:34Z\" level=error msg=\"Failed to execute: ls [-1 /var/lib/longhorn-backupstore-mounts/omv/backup/longhorn/backupstore/volumes], output ls: cannot access '/var/lib/longhorn-backupstore-mounts/omv/backup/longhorn/backupstore/volumes': Permission denied\\n, error exit status 2\"\n, error exit status 1" interval=5m0s node=wezyr url="nfs://omv:/backup/longhorn"`

Can it be somehow connected with this bug?

> <https://github.com/longhorn/longhorn/issues/2964>
>
> \*\*Describe the bug\*\*
> After upgrade longhorn to 1.20, some container are unable …to start corectly (e.g prometheus).
> 
> Looks like root cause is wrong Longhorn volume permisions inside container when container is not running as root.
> 
> Even with \`fsGroup\` specified, permissions are not set for volume.
> 
> \*\*To Reproduce\*\*
> \`\`\`
> \---
> apiVersion: v1
> kind: PersistentVolumeClaim
> metadata:
> name: broken-longhorn
> namespace: default
> spec:
> accessModes:
> - ReadWriteOnce
> resources:
> requests:
> storage: 1Gi
> storageClassName: longhorn
> 
> \---
> apiVersion: apps/v1
> kind: Deployment
> metadata:
> name: broken-longhorn
> namespace: default
> labels:
> app.kubernetes.io/name: broken-longhorn
> spec:
> replicas: 1
> strategy:
> type: Recreate
> selector:
> matchLabels:
> app.kubernetes.io/name: broken-longhorn
> template:
> metadata:
> labels:
> app.kubernetes.io/name: broken-longhorn
> spec:
> containers:
> - name: broken-longhorn
> image: ubuntu:focal-20210723
> command:
> - "/bin/sh"
> - "-ec"
> - |
> tail -f /dev/null
> imagePullPolicy: IfNotPresent
> volumeMounts:
> - mountPath: /data
> name: data
> securityContext:
> runAsUser: 65534
> runAsNonRoot: true
> runAsGroup: 65534
> fsGroup: 65534
> fsGroupChangePolicy: Always
> volumes:
> - name: data
> persistentVolumeClaim:
> claimName: broken-longhorn
> \`\`\`
> 
> \`\`\`
> % kubectl get pvc broken-longhorn -o yaml
> apiVersion: v1
> kind: PersistentVolumeClaim
> metadata:
> annotations:
> kubectl.kubernetes.io/last-applied-configuration: |
> {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"broken-longhorn","namespace":"default"},"spec":{"accessModes":\["ReadWriteOnce"\],"resources":{"requests":{"storage":"1Gi"}},"storageClassName":"longhorn"}}
> pv.kubernetes.io/bind-completed: "yes"
> pv.kubernetes.io/bound-by-controller: "yes"
> volume.beta.kubernetes.io/storage-provisioner: driver.longhorn.io
> creationTimestamp: "2021-09-01T07:48:46Z"
> finalizers:
> - kubernetes.io/pvc-protection
> name: broken-longhorn
> namespace: default
> resourceVersion: "25249959"
> uid: 9ab5ca66-0794-4ad2-8aa5-73e96fa603fc
> spec:
> accessModes:
> - ReadWriteOnce
> resources:
> requests:
> storage: 1Gi
> storageClassName: longhorn
> volumeMode: Filesystem
> volumeName: pvc-9ab5ca66-0794-4ad2-8aa5-73e96fa603fc
> status:
> accessModes:
> - ReadWriteOnce
> capacity:
> storage: 1Gi
> phase: Bound
> 
> % kubectl get pv pvc-9ab5ca66-0794-4ad2-8aa5-73e96fa603fc -o yaml
> apiVersion: v1
> kind: PersistentVolume
> metadata:
> annotations:
> pv.kubernetes.io/provisioned-by: driver.longhorn.io
> creationTimestamp: "2021-09-01T07:48:48Z"
> finalizers:
> - kubernetes.io/pv-protection
> - external-attacher/driver-longhorn-io
> name: pvc-9ab5ca66-0794-4ad2-8aa5-73e96fa603fc
> resourceVersion: "25250154"
> uid: d4db7690-4763-45bf-a5c7-9b7ae0b5d584
> spec:
> accessModes:
> - ReadWriteOnce
> capacity:
> storage: 1Gi
> claimRef:
> apiVersion: v1
> kind: PersistentVolumeClaim
> name: broken-longhorn
> namespace: default
> resourceVersion: "25249908"
> uid: 9ab5ca66-0794-4ad2-8aa5-73e96fa603fc
> csi:
> driver: driver.longhorn.io
> volumeAttributes:
> fromBackup: ""
> numberOfReplicas: "3"
> staleReplicaTimeout: "30"
> storage.kubernetes.io/csiProvisionerIdentity: 1630473362980-8081-driver.longhorn.io
> volumeHandle: pvc-9ab5ca66-0794-4ad2-8aa5-73e96fa603fc
> persistentVolumeReclaimPolicy: Delete
> storageClassName: longhorn
> volumeMode: Filesystem
> status:
> phase: Bound
> 
> % kubectl get csidriver driver.longhorn.io -o yaml
> apiVersion: storage.k8s.io/v1
> kind: CSIDriver
> metadata:
> annotations:
> driver.longhorn.io/kubernetes-version: v1.20.7+k3s1
> driver.longhorn.io/version: v1.2.0
> creationTimestamp: "2021-08-31T17:47:21Z"
> name: driver.longhorn.io
> resourceVersion: "24953648"
> uid: 274cd12a-6aca-47a9-bfd8-32261eb5033a
> spec:
> attachRequired: true
> fsGroupPolicy: ReadWriteOnceWithFSType
> podInfoOnMount: true
> volumeLifecycleModes:
> - Persistent
> \`\`\`
> 
> \`\`\`
> $ kubectl exec -t -i broken-longhorn-c4ccbbb6f-79djg -- bash
> nobody@broken-longhorn-c4ccbbb6f-79djg:/$ ls -la /data/
> total 24
> drwxr-xr-x 3 root root 4096 Sep 1 07:49 .
> drwxr-xr-x 1 root root 4096 Sep 1 07:49 ..
> drwx------ 2 root root 16384 Sep 1 07:49 lost+found
> nobody@broken-longhorn-c4ccbbb6f-79djg:/$ touch /data/test
> touch: cannot touch '/data/test': Permission denied
> \`\`\`
> 
> \*\*Expected behavior\*\*
> When \`fsGroup\` is provided, it should be used to chown destination mount.
> 
> \*\*Environment:\*\*
> - Longhorn version: 1.20
> - Installation method (e.g. Rancher Catalog App/Helm/Kubectl): helm
> - Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: k3os/k3s
> - Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): baremetal
> 
> \*\*Additional context\*\*
> \`\`\`
> % kubectl version
> Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:52:14Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"darwin/amd64"}
> Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.7+k3s1", GitCommit:"aa768cbdabdb44c95c5c1d9562ea7f5ded073bc0", GitTreeState:"clean", BuildDate:"2021-05-20T01:07:13Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"linux/amd64"}
> \`\`\`

I can’t test it, because I use Longhorn from Helm chart, and I don’t know, how to pass flag

> ```
> - --default-fstype=ext4
> 
> ```

---

<div class="post-metadata">

**Author:** ![Marx\_N](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/marx_n/32/8124_2.png) [@Marx\_N](https://forums.suse.com/u/Marx_N)\
**Post date:** [October 17, 2021, 11:48am UTC](https://forums.suse.com/t/error-after-migration/21426/4 "2021-10-17T11:48:48Z")

</div>

I’ve made this changen in Helm chart also, but doesn’t seem to work:

```auto
 values:
    persistence:
      defaultFsType: ext4

```

---

<div class="post-metadata">

**Author:** ![JenTing](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/jenting/32/7290_2.png) [@JenTing](https://forums.suse.com/u/JenTing)\
**Post date:** [October 20, 2021, 2:33am UTC](https://forums.suse.com/t/error-after-migration/21426/5 "2021-10-20T02:33:21Z")

</div>

I guess the NFS server is inside the Kubernetes cluster.

- Can you check how your NFS was deployed?
- Was the NFS server accessible by other in-cluster Pods after the upgrade?

---

<div class="post-metadata">

**Author:** ![PhanLe0110](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/phanle0110/32/7022_2.png) [@PhanLe0110](https://forums.suse.com/u/PhanLe0110)\
**Post date:** [October 20, 2021, 9:55pm UTC](https://forums.suse.com/t/error-after-migration/21426/6 "2021-10-20T21:55:28Z")

</div>

The bug [[BUG] Longhorn 1.2.0 - wrong volume permissions inside container / broken fsGroup · Issue #2964 · longhorn/longhorn · GitHub](https://github.com/longhorn/longhorn/issues/2964) is not related to this issue.  
That bug is fixed in Longhorn v1.2.2 that is the version you are using.

---

<div class="post-metadata">

**Author:** ![Marx\_N](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/marx_n/32/8124_2.png) [@Marx\_N](https://forums.suse.com/u/Marx_N)\
**Post date:** [October 25, 2021, 7:19am UTC](https://forums.suse.com/t/error-after-migration/21426/7 "2021-10-25T07:19:03Z")

</div>

maybe it’s not fixed? Or maybe it is, but I need to do something on my own to make it working (like reset, restart, reconfigure sth.)? How can I check it?  
I suppose reinstalling would fix it, but as I have no backup working, I can’t do that

---

<div class="post-metadata">

**Author:** ![PhanLe0110](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/phanle0110/32/7022_2.png) [@PhanLe0110](https://forums.suse.com/u/PhanLe0110)\
**Post date:** [December 14, 2021, 12:23am UTC](https://forums.suse.com/t/error-after-migration/21426/8 "2021-12-14T00:23:28Z")

</div>

Sorry for the late response. The fix only works for the newly provisioned PV.

---

<div class="post-metadata">

**Author:** ![andreasnielsen87](https://avatars.discourse-cdn.com/v4/letter/a/aca169/32.png) [@andreasnielsen87](https://forums.suse.com/u/andreasnielsen87)\
**Post date:** [December 20, 2021, 10:22am UTC](https://forums.suse.com/t/error-after-migration/21426/9 "2021-12-20T10:22:20Z")

</div>

You will lose any ability to add or remove nodes from the cluster, perform etcd backups or disaster recovery, or edit any of the cluster configuration if you launch a Kubernetes cluster in one Rancher instance and then try to import it into another Rancher instance using the imported cluster feature.

---

<div class="post-metadata">

**Author:** ![Lorenzo\_Gaviglio](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/lorenzo_gaviglio/32/11464_2.png) [@Lorenzo\_Gaviglio](https://forums.suse.com/u/Lorenzo_Gaviglio)\
**Post date:** [August 15, 2024, 1:27pm UTC](https://forums.suse.com/t/error-after-migration/21426/10 "2024-08-15T13:27:15Z")

</div>

Hey Marx, did you get any solution for this problem? I’m fasing a similar situation and I don’t know how to solve it