# Cannot delete Namespaces from Rancher cluster

**URL:** <https://forums.suse.com/t/cannot-delete-namespaces-from-rancher-cluster/37276>\
**Category:** SUSE Rancher Prime\
**Created:** [March 15, 2022, 5:45pm UTC](https://forums.suse.com/t/cannot-delete-namespaces-from-rancher-cluster/37276 "2022-03-15T17:45:40Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![cjohn001](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/cjohn001/32/5702_2.png) [@cjohn001](https://forums.suse.com/u/cjohn001)\
**Post date:** [March 15, 2022, 5:45pm UTC](https://forums.suse.com/t/cannot-delete-namespaces-from-rancher-cluster/37276/1 "2022-03-15T17:45:40Z")

</div>

Hello together,  
I have a rke provisioned rancher cluster (local) running and a rancher provisioned cluster with workloads.  
I am now trying to delete a namespace in the cluster and get the following error

I have openebs installed for pvc provisioning. However, in the namespace there is no pvc included. I also checked the certificates in both clusters, those are all valid till 2030. I can also create new namespaces create deployments and delete resources except a namespace. Can someone tell me how I can fix this? Thanks for your help

Error deleting  
Internal error occurred: failed calling webhook “[admission-webhook.openebs.io](http://admission-webhook.openebs.io)”: Post “[https://admission-server-svc.openebs.svc:443/validate?timeout=5s](https://admission-server-svc.openebs.svc:443/validate?timeout=5s)”: x509: certificate has expired or is not yet valid: current time 2022-03-15T17:37:47Z is after 2021-12-26T18:24:28Z

---

<div class="post-metadata">

**Author:** ![cjohn001](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/cjohn001/32/5702_2.png) [@cjohn001](https://forums.suse.com/u/cjohn001)\
**Post date:** [March 15, 2022, 7:13pm UTC](https://forums.suse.com/t/cannot-delete-namespaces-from-rancher-cluster/37276/2 "2022-03-15T19:13:11Z")

</div>

> [@cjohn001](#):
>
> [admission-webhook.openebs.io](http://admission-webhook.openebs.io)

Solved

> <https://github.com/openebs/openebs/issues/2951>
>
> \## Description
> Trying to create the following PVC:
> \`\`\`
> \---
> kind: PersistentV…olumeClaim
> apiVersion: v1
> metadata:
> namespace: default
> name: pvtest
> spec:
> accessModes:
> - ReadWriteOnce
> storageClassName: openebs-hostpath
> resources:
> requests:
> storage: 1Gi
> \`\`\`
> 
> One cluster is OK:
> \- \`k apply -f tpv.yaml\` --\> \`persistentvolumeclaim/pvtest created\`
> \- \`k delete -f tpv.yaml\` --\> \`persistentvolumeclaim "pvtest" deleted\`
> 
> Another cluster:
> \- \`k apply -f tpv.yaml\` --\>
> \`\`\`
> Error from server (InternalError): error when creating "tpv.yaml": Internal error occurred: failed calling webhook "admission-webhook.openebs.io": Post https://admission-server-svc.openebs.svc:443/validate?timeout=30s: x509: certificate has expired or is not yet valid
> \`\`\`
> 
> Both clusters were deployed in the same way, have the same configuration, and both worked just fine for about 6 months until today (I didn't change anything on their config)
> 
> \## Expected Behavior
> \- The second cluster should just create a PVC, then delete.
> \- Actually, I've noticed this while trying to drop and delete again one namespace. 1st cluster was OK, 2nd is frozen on \`terminating\` that namespace forever, after some time I've found that the root cause for this is the inability to delete that namespace's PVCs. They're now reported as "lost".
> 
> Get namespaces:
> \`\`\`
> sds Terminating 36d
> \`\`\`
> PVCs in that namespace:
> \`\`\`
> sds-0 Lost pvc-07c870dd-41c9-11ea-a753-022eab783cb2 0 openebs-hostpath 36d
> sds-1 Lost pvc-07c6a170-41c9-11ea-a753-022eab783cb2 0 openebs-hostpath 36d
> sds-2 Lost pvc-07c949ec-41c9-11ea-a753-022eab783cb2 0 openebs-hostpath 36d
> sds-3 Lost pvc-07c77cf6-41c9-11ea-a753-022eab783cb2 0 openebs-hostpath 36d
> \`\`\`
> 
> \* \`kubectl get nodes\`:
> \`\`\`
> ip-192-168-25-150.us-west-2.compute.internal Ready \<none\> 180d v1.13.8-eks-cd3eb0
> ip-192-168-49-247.us-west-2.compute.internal Ready \<none\> 180d v1.13.8-eks-cd3eb0
> ip-192-168-64-125.us-west-2.compute.internal Ready \<none\> 180d v1.13.8-eks-cd3eb0
> ip-192-168-83-36.us-west-2.compute.internal Ready \<none\> 180d v1.13.8-eks-cd3eb0
> \`\`
> \* \`kubectl get pods --all-namespaces\`:
> \`\`\`
> NAMESPACE NAME READY STATUS RESTARTS AGE
> default local-storage-nfs-nfs-server-provisioner-0 1/1 Running 1 36d
> dev-analytics-api-prod dev-analytics-api-7c7f56b8fc-448mh 1/1 Running 0 7h11m
> dev-analytics-api-prod dev-analytics-api-7c7f56b8fc-jsksj 1/1 Running 0 7h11m
> dev-analytics-api-prod dev-analytics-api-migration-1567762361901059583 0/1 Completed 0 180d
> dev-analytics-ui dev-analytics-ui-6c984d9958-8z26v 1/1 Running 0 11d
> json2hat json2hat-1581814860-dckpn 0/1 Completed 0 17d
> json2hat json2hat-1582419660-jj47j 0/1 Completed 0 10d
> json2hat json2hat-1583024460-rb7t8 0/1 Completed 0 3d15h
> kibana kibana-7567969554-mvvp6 0/1 Running 0 31m
> kibana kibana-7567969554-vnfcq 0/1 Running 0 31m
> kube-system aws-node-lrh7d 1/1 Running 0 180d
> kube-system aws-node-qnnjk 1/1 Running 0 180d
> kube-system aws-node-rkn9z 1/1 Running 0 180d
> kube-system aws-node-xp6mf 1/1 Running 1 180d
> kube-system coredns-79d667b89f-22p4h 1/1 Running 0 42d
> kube-system coredns-79d667b89f-fb8x5 1/1 Running 0 42d
> kube-system kube-proxy-dv7bn 1/1 Running 0 180d
> kube-system kube-proxy-mfpwx 1/1 Running 1 180d
> kube-system kube-proxy-rs7d7 1/1 Running 0 180d
> kube-system kube-proxy-sp9jm 1/1 Running 0 180d
> mariadb backups-page-dcc76bd4d-kbzh9 1/1 Running 0 36d
> mariadb mariadb-backups-1583203500-p4zmb 0/1 Completed 0 37h
> mariadb mariadb-backups-1583289900-cvpqr 0/1 Completed 0 13h
> mariadb mariadb-master-0 1/1 Running 3 153d
> mariadb mariadb-slave-0 1/1 Running 0 153d
> openebs maya-apiserver-c54455947-44k2r 1/1 Running 0 42d
> openebs openebs-admission-server-b945f5f94-bwlgk 1/1 Running 0 30m
> openebs openebs-localpv-provisioner-6c778bbc9c-7ckjp 1/1 Running 0 27m
> openebs openebs-ndm-259t6 1/1 Running 1 180d
> openebs openebs-ndm-dxhml 1/1 Running 0 180d
> openebs openebs-ndm-ht9dw 1/1 Running 0 180d
> openebs openebs-ndm-m725n 1/1 Running 0 180d
> openebs openebs-ndm-operator-796785b89f-dfgq7 1/1 Running 0 36d
> openebs openebs-provisioner-7db48b57f4-wq5nd 1/1 Running 1 42d
> openebs openebs-snapshot-operator-5664bf5777-cssqm 2/2 Running 7 180d
> sortinghat-api sortinghat-api-5f9d67cd8d-hx5kq 1/1 Running 0 180d
> \`\`\`
> \* \`kubectl get services\`:
> \`\`\`
> NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
> kubernetes ClusterIP 10.100.0.1 \<none\> 443/TCP 182d
> local-storage-nfs-nfs-server-provisioner ClusterIP 10.100.255.228 \<none\> 2049/TCP,20048/TCP,51413/TCP,51413/UDP 36d
> \`\`\`
> \* \`kubectl get sc\`:
> \`\`\`
> NAME PROVISIONER AGE
> gp2 (default) kubernetes.io/aws-ebs 182d
> local-storage kubernetes.io/no-provisioner 182d
> nfs-openebs-localstorage cluster.local/local-storage-nfs-nfs-server-provisioner 36d
> openebs-device openebs.io/local 182d
> openebs-hostpath openebs.io/local 182d
> openebs-jiva-default openebs.io/provisioner-iscsi 182d
> openebs-snapshot-promoter volumesnapshot.external-storage.k8s.io/snapshot-promoter 182d
> \`\`\`
> \* \`kubectl get pv\`:
> \`\`\`
> NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
> pvc-050a7761-d075-11e9-95db-024cac4c3a40 4Gi RWO Delete Bound mariadb/data-mariadb-master-0 openebs-hostpath 180d
> pvc-050e5e66-d075-11e9-95db-024cac4c3a40 4Gi RWO Delete Bound mariadb/data-mariadb-slave-0 openebs-hostpath 180d
> pvc-57f3604e-41c8-11ea-a753-022eab783cb2 4Gi RWO Delete Bound default/data-local-storage-nfs-nfs-server-provisioner-0 openebs-hostpath 36d
> pvc-63fe9708-41c8-11ea-a753-022eab783cb2 4Gi RWX Delete Bound mariadb/mariadb-backups nfs-openebs-localstorage 36d
> \`\`\`
> \* \`kubectl get pvc\`:
> \`\`\`
> NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
> data-local-storage-nfs-nfs-server-provisioner-0 Bound pvc-57f3604e-41c8-11ea-a753-022eab783cb2 4Gi RWO openebs-hostpath 36d
> \`\`\`
> \* OS (from \`/etc/os-release\`):
> \`\`\`
> NAME="Ubuntu"
> VERSION="18.04.2 LTS (Bionic Beaver)"
> ID=ubuntu
> ID\_LIKE=debian
> PRETTY\_NAME="Ubuntu 18.04.2 LTS"
> VERSION\_ID="18.04"
> HOME\_URL="https://www.ubuntu.com/"
> SUPPORT\_URL="https://help.ubuntu.com/"
> BUG\_REPORT\_URL="https://bugs.launchpad.net/ubuntu/"
> PRIVACY\_POLICY\_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
> VERSION\_CODENAME=bionic
> UBUNTU\_CODENAME=bionic
> \`\`\`
> \* Kernel (from \`uname -a\`):
> \`\`\`
> Linux ip-172-31-34-53 4.15.0-1043-aws #45-Ubuntu SMP Mon Jun 24 14:07:03 UTC 2019 x86\_64 x86\_64 x86\_64 GNU/Linux
> \`\`\`
> 
> OpenEBS version: \`1.1.0\`
