# Problem adding AKS cluster to Rancher

**URL:** <https://forums.suse.com/t/problem-adding-aks-cluster-to-rancher/38391>\
**Category:** SUSE Rancher Prime\
**Created:** [July 6, 2022, 11:30pm UTC](https://forums.suse.com/t/problem-adding-aks-cluster-to-rancher/38391 "2022-07-06T23:30:04Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dmitry\_Shultz](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/dmitry_shultz/32/5232_2.png) [@Dmitry\_Shultz](https://forums.suse.com/u/Dmitry_Shultz)\
**Post date:** [July 6, 2022, 11:30pm UTC](https://forums.suse.com/t/problem-adding-aks-cluster-to-rancher/38391/1 "2022-07-06T23:30:04Z")

</div>

We are trying to add AKS cluster to Rancher and having some problem with it. Here is the Slack post with the full video recording that reproduces it in real time: [Slack](https://rancher-users.slack.com/archives/C3ASABBD1/p1657063107070329)

As you can see Rancher pod crashes in several minutes after we try to add AKS cluster using Rancher UI. If Rancher is deployed in HA mode - all pods eventually die and are getting into the crash loop.

We can provide any info on demand to help to resolve this. Any pointer for where to look/zoom in would be greatly appreciated.

---

<div class="post-metadata">

**Author:** ![Dmitry\_Shultz](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/dmitry_shultz/32/5232_2.png) [@Dmitry\_Shultz](https://forums.suse.com/u/Dmitry_Shultz)\
**Post date:** [July 11, 2022, 5:22pm UTC](https://forums.suse.com/t/problem-adding-aks-cluster-to-rancher/38391/2 "2022-07-11T17:22:36Z")

</div>

If we are doing anything wrong/stupid here (we never used AKS cluster with Rancher running in AKS before) - please let us know.

---

<div class="post-metadata">

**Author:** ![Dmitry\_Shultz](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/dmitry_shultz/32/5232_2.png) [@Dmitry\_Shultz](https://forums.suse.com/u/Dmitry_Shultz)\
**Post date:** [July 13, 2022, 11:23pm UTC](https://forums.suse.com/t/problem-adding-aks-cluster-to-rancher/38391/3 "2022-07-13T23:23:03Z")

</div>

We missed the error log in the aks-config-operator logs, looks like this:

```auto
Doing /etc/rancher/ssl
W0712 18:30:13.902837 10 loader.go:221] Config not found: /home/aks-operator/.kube/config
time="2022-07-12T18:30:14Z" level=info msg="Starting aks.cattle.io/v1, Kind=AKSClusterConfig controller"
time="2022-07-12T18:30:14Z" level=info msg="Starting /v1, Kind=Secret controller"
time="2022-07-12T18:30:14Z" level=info msg="Checking configuration for cluster [aksnonprod]"
E0712 18:30:15.575463 10 runtime.go:78] Observed a panic: runtime.boundsError{x:1, y:0, signed:true, code:0x0} (runtime error: index out of range [1] with length 0)
goroutine 252 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x14c27a0, 0xc0008a6090)
	/go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/runtime/runtime.go:74 +0x95
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/runtime/runtime.go:48 +0x86
panic(0x14c27a0, 0xc0008a6090)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/rancher/aks-operator/controller.BuildUpstreamClusterState(0x1760658, 0xc000875e40, 0x17609d8, 0xc0003ce0c0, 0x7f25ec64eff0, 0xc000345f80, 0xc000396118, 0xd18c2e2800, 0x3, 0x6fc23ac00)
	/go/src/github.com/rancher/aks-operator/controller/aks-cluster-config-handler.go:674 +0x131f
github.com/rancher/aks-operator/controller.(*Handler).checkAndUpdate(0xc0003271c0, 0xc000396000, 0x0, 0x0, 0x0)
	/go/src/github.com/rancher/aks-operator/controller/aks-cluster-config-handler.go:339 +0x55c
github.com/rancher/aks-operator/controller.(*Handler).OnAksConfigChanged(0xc0003271c0, 0xc000040200, 0x1a, 0xc000396000, 0xc0002b08a0, 0xc0ab8d4d8b16b31f, 0x11b0ace1)
	/go/src/github.com/rancher/aks-operator/controller/aks-cluster-config-handler.go:121 +0x9d
github.com/rancher/aks-operator/controller.(*Handler).recordError.func1(0xc000040200, 0x1a, 0xc000396000, 0x14eb180, 0xeca0c0, 0x12cb0c0)
	/go/src/github.com/rancher/aks-operator/controller/aks-cluster-config-handler.go:171 +0x67
github.com/rancher/aks-operator/pkg/generated/controllers/aks.cattle.io/v1.FromAKSClusterConfigHandlerToHandler.func1(0xc000040200, 0x1a, 0x1739d68, 0xc000396000, 0xe0, 0xf, 0x14eb180, 0x40a84c)
	/go/src/github.com/rancher/aks-operator/pkg/generated/controllers/aks.cattle.io/v1/aksclusterconfig.go:105 +0x6b
github.com/rancher/lasso/pkg/controller.SharedControllerHandlerFunc.OnChange(0xc000329820, 0xc000040200, 0x1a, 0x1739d68, 0xc000396000, 0x1a, 0xc000026c01, 0x40a56c, 0x2030288)
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210616224652-fc3ebd901c08/pkg/controller/sharedcontroller.go:29 +0x4e
github.com/rancher/lasso/pkg/controller.(*SharedHandler).OnChange(0xc000327140, 0xc000040200, 0x1a, 0x1739d68, 0xc000396000, 0xc000915d01, 0x0)
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210616224652-fc3ebd901c08/pkg/controller/sharedhandler.go:69 +0x14c
github.com/rancher/lasso/pkg/controller.(*controller).syncHandler(0xc0000e09a0, 0xc000040200, 0x1a, 0xc000915e58, 0x4)
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210616224652-fc3ebd901c08/pkg/controller/controller.go:215 +0xd1
github.com/rancher/lasso/pkg/controller.(*controller).processSingleItem(0xc0000e09a0, 0x13ce1e0, 0xc00085c200, 0x0, 0x0)
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210616224652-fc3ebd901c08/pkg/controller/controller.go:197 +0xe7
github.com/rancher/lasso/pkg/controller.(*controller).processNextWorkItem(0xc0000e09a0, 0x203000)
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210616224652-fc3ebd901c08/pkg/controller/controller.go:174 +0x54
github.com/rancher/lasso/pkg/controller.(*controller).runWorker(...)
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210616224652-fc3ebd901c08/pkg/controller/controller.go:163
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00085c2c0)
	/go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00085c2c0, 0x1734000, 0xc0002b0930, 0x1, 0xc00008b080)
	/go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/wait/wait.go:156 +0x9b
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00085c2c0, 0x3b9aca00, 0x0, 0xc00051fc01, 0xc00008b080)
	/go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc00085c2c0, 0x3b9aca00, 0xc00008b080)
	/go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/wait/wait.go:90 +0x4d
created by github.com/rancher/lasso/pkg/controller.(*controller).run
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210616224652-fc3ebd901c08/pkg/controller/controller.go:134 +0x33b
panic: runtime error: index out of range [1] with length 0 [recovered]
	panic: runtime error: index out of range [1] with length 0

```

It looks like /home/aks-operator/.kube/config is not in place. Is this a bug i.e. config should be created automatically or we are missing something?

---

<div class="post-metadata">

**Author:** ![Dmitry\_Shultz](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/dmitry_shultz/32/5232_2.png) [@Dmitry\_Shultz](https://forums.suse.com/u/Dmitry_Shultz)\
**Post date:** [July 27, 2022, 9:26pm UTC](https://forums.suse.com/t/problem-adding-aks-cluster-to-rancher/38391/4 "2022-07-27T21:26:39Z")

</div>

Issue created [Rancher 2.2.6 aks-config-operator /home/aks-operator/.kube/config not found · Issue #38285 · rancher/rancher · GitHub](https://github.com/rancher/rancher/issues/38285) - no response yet ☹

At this point I’m wondering if anybody successfully added any Azure AKS cluster to Rancher recently. What versions are working for you?

---

<div class="post-metadata">

**Author:** ![Dmitry\_Shultz](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/dmitry_shultz/32/5232_2.png) [@Dmitry\_Shultz](https://forums.suse.com/u/Dmitry_Shultz)\
**Post date:** [July 29, 2022, 8:00pm UTC](https://forums.suse.com/t/problem-adding-aks-cluster-to-rancher/38391/5 "2022-07-29T20:00:00Z")

</div>

Other people are seeing the same crash stack trace:

> <https://github.com/rancher/rancher/issues/37863>
>
> \*\*Rancher Server Setup\*\*
> \- Rancher version: 2.6.5
> \- Installation option (Docke…r install/Helm Chart): Helm Chart
> - If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc): Brand new RKE2 cluster (v1.22.9+rke2r2)
> \- Proxy/Cert Details: Rancher sitting behind NGINX+ reverse proxy, used for inbound connections only, no outbound proxy.
> 
> \*\*Information about the Cluster\*\*
> \- Kubernetes version: v1.22.9+rke2r2
> \- Cluster Type (Local/Downstream): Downstream
> - If downstream, what type of cluster? (Custom/Imported or specify provider for Hosted/Infrastructure Provider): AKS (v1.21.9).
> \<!--
> \* Custom = Running a docker command on a node
> \* Imported = Running kubectl apply onto an existing k8s cluster 
> \* Hosted = EKS, GKE, AKS, etc
> \* Infrastructure Provider = Rancher provisioning the nodes using different node drivers (e.g. AWS, Digital Ocean, etc)
> \--\>
> 
> \*\*User Information\*\*
> \- What is the role of the user logged in? (Admin/Cluster Owner/Cluster Member/Project Owner/Project Member/Custom): Owner
> - If custom, define the set of permissions:
> 
> 
>    
> \*\*Describe the bug\*\*
> 
> When importing an AKS (v1.21.9) cluster, the aks-config-operator pod goes into CrashLoopBackoff state with the following error:
> \`\`\`
> PS C:\\Users\\egk01\> kubectl logs -p -n cattle-system aks-config-operator-67b5b98448-98bsf
> Doing /etc/rancher/ssl
> W0530 16:00:54.422468 9 loader.go:221\] Config not found: /home/aks-operator/.kube/config
> time="2022-05-30T16:00:54Z" level=info msg="Starting aks.cattle.io/v1, Kind=AKSClusterConfig controller"
> time="2022-05-30T16:00:54Z" level=info msg="Starting /v1, Kind=Secret controller"
> time="2022-05-30T16:00:55Z" level=info msg="Checking configuration for cluster \[uk1ds-kc-001\]"
> time="2022-05-30T16:00:55Z" level=info msg="Checking configuration for cluster \[uk1d-kc-001\]"
> E0530 16:00:55.553476 9 runtime.go:78\] Observed a panic: runtime.boundsError{x:1, y:0, signed:true, code:0x0} (runtime error: index out of range \[1\] with length 0)
> goroutine 216 \[running\]:
> k8s.io/apimachinery/pkg/util/runtime.logPanic(0x14c27a0, 0xc00022a168)
> /go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/runtime/runtime.go:74 +0x95
> k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
> /go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/runtime/runtime.go:48 +0x86
> panic(0x14c27a0, 0xc00022a168)
> /usr/local/go/src/runtime/panic.go:965 +0x1b9
> github.com/rancher/aks-operator/controller.BuildUpstreamClusterState(0x1760658, 0xc0008a4d40, 0x17609d8, 0xc000217bc0, 0x7faa033d41a0, 0xc0002261c0, 0xc00048ad70, 0xd18c2e2800, 0x3, 0x6fc23ac00)
> /go/src/github.com/rancher/aks-operator/controller/aks-cluster-config-handler.go:674 +0x131f
> github.com/rancher/aks-operator/controller.(\*Handler).checkAndUpdate(0xc000240400, 0xc00048ac58, 0x0, 0x0, 0x0)
> /go/src/github.com/rancher/aks-operator/controller/aks-cluster-config-handler.go:339 +0x55c
> github.com/rancher/aks-operator/controller.(\*Handler).OnAksConfigChanged(0xc000240400, 0xc00058c0a0, 0x1a, 0xc00048ac58, 0x7faa03445258, 0x1, 0x100000000000028)
> /go/src/github.com/rancher/aks-operator/controller/aks-cluster-config-handler.go:121 +0x9d
> github.com/rancher/aks-operator/controller.(\*Handler).recordError.func1(0xc00058c0a0, 0x1a, 0xc00048ac58, 0xc000294300, 0x21, 0xc000294330)
> /go/src/github.com/rancher/aks-operator/controller/aks-cluster-config-handler.go:171 +0x67
> github.com/rancher/aks-operator/pkg/generated/controllers/aks.cattle.io/v1.FromAKSClusterConfigHandlerToHandler.func1(0xc00058c0a0, 0x1a, 0x1739d68, 0xc00048ac58, 0xc0006b9c38, 0x41659b, 0xc0006b9c18, 0x4eaeba)
> /go/src/github.com/rancher/aks-operator/pkg/generated/controllers/aks.cattle.io/v1/aksclusterconfig.go:105 +0x6b
> github.com/rancher/lasso/pkg/controller.SharedControllerHandlerFunc.OnChange(0xc000214920, 0xc00058c0a0, 0x1a, 0x1739d68, 0xc00048ac58, 0x1a, 0x756ea14a212d01, 0x40a43f, 0xc000030000)
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210616224652-fc3ebd901c08/pkg/controller/sharedcontroller.go:29 +0x4e
> github.com/rancher/lasso/pkg/controller.(\*SharedHandler).OnChange(0xc000240380, 0xc00058c0a0, 0x1a, 0x1739d68, 0xc00048ac58, 0xc0004b5c01, 0x0)
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210616224652-fc3ebd901c08/pkg/controller/sharedhandler.go:69 +0x14c
> github.com/rancher/lasso/pkg/controller.(\*controller).syncHandler(0xc000242210, 0xc00058c0a0, 0x1a, 0xc0004b5d38, 0xc0006b9da0)
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210616224652-fc3ebd901c08/pkg/controller/controller.go:215 +0xd1
> github.com/rancher/lasso/pkg/controller.(\*controller).processSingleItem(0xc000242210, 0x13ce1e0, 0xc0004d0e10, 0x0, 0x0)
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210616224652-fc3ebd901c08/pkg/controller/controller.go:197 +0xe7
> github.com/rancher/lasso/pkg/controller.(\*controller).processNextWorkItem(0xc000242210, 0x203000)
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210616224652-fc3ebd901c08/pkg/controller/controller.go:174 +0x54
> github.com/rancher/lasso/pkg/controller.(\*controller).runWorker(...)
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210616224652-fc3ebd901c08/pkg/controller/controller.go:163
> k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0004d0ea0)
> /go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/wait/wait.go:155 +0x5f
> k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0004d0ea0, 0x1734000, 0xc000561bf0, 0xc0004d0e01, 0xc0002486c0)
> /go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/wait/wait.go:156 +0x9b
> k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0004d0ea0, 0x3b9aca00, 0x0, 0x1, 0xc0002486c0)
> /go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/wait/wait.go:133 +0x98
> k8s.io/apimachinery/pkg/util/wait.Until(0xc0004d0ea0, 0x3b9aca00, 0xc0002486c0)
> /go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/wait/wait.go:90 +0x4d
> created by github.com/rancher/lasso/pkg/controller.(\*controller).run
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210616224652-fc3ebd901c08/pkg/controller/controller.go:134 +0x33b
> panic: runtime error: index out of range \[1\] with length 0 \[recovered\]
> panic: runtime error: index out of range \[1\] with length 0
> 
> goroutine 216 \[running\]:
> k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
> /go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/runtime/runtime.go:55 +0x109
> panic(0x14c27a0, 0xc00022a168)
> /usr/local/go/src/runtime/panic.go:965 +0x1b9
> github.com/rancher/aks-operator/controller.BuildUpstreamClusterState(0x1760658, 0xc0008a4d40, 0x17609d8, 0xc000217bc0, 0x7faa033d41a0, 0xc0002261c0, 0xc00048ad70, 0xd18c2e2800, 0x3, 0x6fc23ac00)
> /go/src/github.com/rancher/aks-operator/controller/aks-cluster-config-handler.go:674 +0x131f
> github.com/rancher/aks-operator/controller.(\*Handler).checkAndUpdate(0xc000240400, 0xc00048ac58, 0x0, 0x0, 0x0)
> /go/src/github.com/rancher/aks-operator/controller/aks-cluster-config-handler.go:339 +0x55c
> github.com/rancher/aks-operator/controller.(\*Handler).OnAksConfigChanged(0xc000240400, 0xc00058c0a0, 0x1a, 0xc00048ac58, 0x7faa03445258, 0x1, 0x100000000000028)
> /go/src/github.com/rancher/aks-operator/controller/aks-cluster-config-handler.go:121 +0x9d
> github.com/rancher/aks-operator/controller.(\*Handler).recordError.func1(0xc00058c0a0, 0x1a, 0xc00048ac58, 0xc000294300, 0x21, 0xc000294330)
> /go/src/github.com/rancher/aks-operator/controller/aks-cluster-config-handler.go:171 +0x67
> github.com/rancher/aks-operator/pkg/generated/controllers/aks.cattle.io/v1.FromAKSClusterConfigHandlerToHandler.func1(0xc00058c0a0, 0x1a, 0x1739d68, 0xc00048ac58, 0xc0006b9c38, 0x41659b, 0xc0006b9c18, 0x4eaeba)
> /go/src/github.com/rancher/aks-operator/pkg/generated/controllers/aks.cattle.io/v1/aksclusterconfig.go:105 +0x6b
> github.com/rancher/lasso/pkg/controller.SharedControllerHandlerFunc.OnChange(0xc000214920, 0xc00058c0a0, 0x1a, 0x1739d68, 0xc00048ac58, 0x1a, 0x756ea14a212d01, 0x40a43f, 0xc000030000)
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210616224652-fc3ebd901c08/pkg/controller/sharedcontroller.go:29 +0x4e
> github.com/rancher/lasso/pkg/controller.(\*SharedHandler).OnChange(0xc000240380, 0xc00058c0a0, 0x1a, 0x1739d68, 0xc00048ac58, 0xc0004b5c01, 0x0)
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210616224652-fc3ebd901c08/pkg/controller/sharedhandler.go:69 +0x14c
> github.com/rancher/lasso/pkg/controller.(\*controller).syncHandler(0xc000242210, 0xc00058c0a0, 0x1a, 0xc0004b5d38, 0xc0006b9da0)
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210616224652-fc3ebd901c08/pkg/controller/controller.go:215 +0xd1
> github.com/rancher/lasso/pkg/controller.(\*controller).processSingleItem(0xc000242210, 0x13ce1e0, 0xc0004d0e10, 0x0, 0x0)
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210616224652-fc3ebd901c08/pkg/controller/controller.go:197 +0xe7
> github.com/rancher/lasso/pkg/controller.(\*controller).processNextWorkItem(0xc000242210, 0x203000)
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210616224652-fc3ebd901c08/pkg/controller/controller.go:174 +0x54
> github.com/rancher/lasso/pkg/controller.(\*controller).runWorker(...)
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210616224652-fc3ebd901c08/pkg/controller/controller.go:163
> k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0004d0ea0)
> /go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/wait/wait.go:155 +0x5f
> k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0004d0ea0, 0x1734000, 0xc000561bf0, 0xc0004d0e01, 0xc0002486c0)
> /go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/wait/wait.go:156 +0x9b
> k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0004d0ea0, 0x3b9aca00, 0x0, 0x1, 0xc0002486c0)
> /go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/wait/wait.go:133 +0x98
> k8s.io/apimachinery/pkg/util/wait.Until(0xc0004d0ea0, 0x3b9aca00, 0xc0002486c0)
> /go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/wait/wait.go:90 +0x4d
> created by github.com/rancher/lasso/pkg/controller.(\*controller).run
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210616224652-fc3ebd901c08/pkg/controller/controller.go:134 +0x33b
> \`\`\`
> 
> In this error, "uk1ds-kc-001" is a cluster which has already been successfully imported, and "uk1d-kc-001" is a the cluster which I am attempting to import.
> 
> At the same time, the rancher pods continually crash with the same error (with a bit more preceding detail):
> \`\`\`
> 2022/05/30 16:00:10 \[INFO\] \[mgmt-cluster-rbac-delete\] Creating namespace c-t89dk
> 2022/05/30 16:00:10 \[INFO\] \[mgmt-cluster-rbac-delete\] Creating Default project for cluster c-t89dk
> 2022/05/30 16:00:10 \[INFO\] \[mgmt-project-rbac-create\] Creating namespace p-z8w8t
> 2022/05/30 16:00:10 \[ERROR\] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=c-t89dk, err=Operation cannot be fulfilled on namespaces "c-t89dk": the object has been modified; please apply your changes to the latest version and try again
> 2022/05/30 16:00:10 \[INFO\] \[mgmt-cluster-rbac-delete\] Creating System project for cluster c-t89dk
> 2022/05/30 16:00:10 \[INFO\] \[mgmt-project-rbac-create\] Creating namespace p-gccb2
> 2022/05/30 16:00:10 \[INFO\] \[mgmt-project-rbac-create\] Creating creator projectRoleTemplateBinding for user user-t6wbb for project p-z8w8t
> 2022/05/30 16:00:10 \[INFO\] \[mgmt-cluster-rbac-delete\] Updating cluster c-t89dk
> 2022/05/30 16:00:10 \[ERROR\] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=c-t89dk, err=Operation cannot be fulfilled on namespaces "c-t89dk": the object has been modified; please apply your changes to the latest version and try again
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-project-rbac-create\] Creating creator projectRoleTemplateBinding for user user-t6wbb for project p-gccb2
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-project-rbac-create\] Creating creator clusterRoleTemplateBinding for user user-t6wbb for cluster c-t89dk
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-crtb-controller\] Setting InitialRolesPopulated condition on project p-z8w8t
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-prtb-controller\] Creating role/clusterRole p-z8w8t-projectowner
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-crtb-controller\] Setting InitialRolesPopulated condition on project p-gccb2
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-project-rbac-create\] Updating project p-z8w8t
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-crtb-controller\] Creating role/clusterRole c-t89dk-clusterowner
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-prtb-controller\] Creating roleBinding for membership in project p-z8w8t for subject user-t6wbb
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-project-rbac-create\] Updating project p-gccb2
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-prtb-controller\] Creating role/clusterRole p-gccb2-projectowner
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-crtb-controller\] Creating clusterRoleBinding for membership in cluster c-t89dk for subject user-t6wbb
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-prtb-controller\] Creating role/clusterRole c-t89dk-clustermember
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-prtb-controller\] Creating roleBinding for membership in project p-gccb2 for subject user-t6wbb
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-prtb-controller\] Creating clusterRoleBinding for membership in cluster c-t89dk for subject user-t6wbb
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-project-rbac-create\] Updating project p-z8w8t
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-crtb-controller\] Creating role cluster-owner in namespace c-t89dk
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-crtb-controller\] Setting InitialRolesPopulated condition on cluster
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-cluster-rbac-delete\] Updating cluster c-t89dk
> 2022/05/30 16:00:11 \[ERROR\] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=p-z8w8t, err=Operation cannot be fulfilled on namespaces "p-z8w8t": the object has been modified; please apply your changes to the latest version and try again
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-prtb-controller\] Creating role project-owner in namespace c-t89dk
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-crtb-controller\] Creating roleBinding for subject user-t6wbb with role cluster-owner in namespace c-t89dk
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-prtb-controller\] Creating roleBinding for subject user-t6wbb with role project-owner in namespace c-t89dk
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-project-rbac-create\] Updating project p-gccb2
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-crtb-controller\] Creating role cluster-owner in namespace p-z8w8t
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-prtb-controller\] Updating clusterRoleBinding crb-7yl3oxhpng for cluster membership in cluster c-t89dk for subject user-t6wbb
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-prtb-controller\] Creating role project-owner in namespace p-z8w8t
> 2022/05/30 16:00:11 \[ERROR\] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=p-gccb2, err=Operation cannot be fulfilled on namespaces "p-gccb2": the object has been modified; please apply your changes to the latest version and try again
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-crtb-controller\] Creating roleBinding for subject user-t6wbb with role cluster-owner in namespace p-z8w8t
> 2022/05/30 16:00:11 \[ERROR\] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=p-z8w8t, err=Operation cannot be fulfilled on namespaces "p-z8w8t": the object has been modified; please apply your changes to the latest version and try again
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-prtb-controller\] Creating roleBinding for subject user-t6wbb with role project-owner in namespace c-t89dk
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-crtb-controller\] Creating role cluster-owner in namespace p-gccb2
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-prtb-controller\] Creating role project-owner in namespace p-gccb2
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-prtb-controller\] Creating role admin in namespace p-z8w8t
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-crtb-controller\] Creating roleBinding for subject user-t6wbb with role cluster-owner in namespace p-gccb2
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-prtb-controller\] Creating roleBinding for subject user-t6wbb with role project-owner in namespace p-z8w8t
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-prtb-controller\] Creating role admin in namespace p-gccb2
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-prtb-controller\] Creating roleBinding for subject user-t6wbb with role admin in namespace p-z8w8t
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-prtb-controller\] Creating roleBinding for subject user-t6wbb with role project-owner in namespace p-gccb2
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-auth-prtb-controller\] Creating roleBinding for subject user-t6wbb with role admin in namespace p-gccb2
> 2022/05/30 16:00:11 \[INFO\] \[mgmt-cluster-rbac-delete\] Updating cluster c-t89dk
> 2022/05/30 16:00:11 \[INFO\] waiting for cluster import \[c-t89dk\] to start
> 2022/05/30 16:00:11 \[INFO\] waiting for cluster import \[c-t89dk\] to start
> 2022/05/30 16:00:11 \[INFO\] waiting for cluster import \[c-t89dk\] to start
> 2022/05/30 16:00:11 \[INFO\] waiting for cluster import \[c-t89dk\] to start
> 2022/05/30 16:00:11 \[INFO\] waiting for cluster import \[c-t89dk\] to start
> 2022/05/30 16:00:11 \[INFO\] waiting for cluster import \[c-t89dk\] to start
> 2022/05/30 16:00:13 \[INFO\] setting initial upstreamSpec on cluster \[c-t89dk\]
> E0530 16:00:14.107171 33 runtime.go:78\] Observed a panic: runtime.boundsError{x:1, y:0, signed:true, code:0x0} (runtime error: index out of range \[1\] with length 0)
> goroutine 2809 \[running\]:
> k8s.io/apimachinery/pkg/util/runtime.logPanic({0x3cf6500, 0xc00c10af78})
> /go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/runtime/runtime.go:74 +0x7d
> k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc001f1a1d0})
> /go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/runtime/runtime.go:48 +0x75
> panic({0x3cf6500, 0xc00c10af78})
> /usr/lib64/go/1.17/src/runtime/panic.go:1038 +0x215
> github.com/rancher/aks-operator/controller.BuildUpstreamClusterState({0x47f9dd0, 0xc000084048}, {0x47fc220, 0xc00747db00}, {0x7f031d6bafc0, 0xc0011de930}, 0xc00e452b40)
> /go/pkg/mod/github.com/rancher/aks-operator@v1.0.5/controller/aks-cluster-config-handler.go:674 +0xa97
> github.com/rancher/rancher/pkg/controllers/management/clusterupstreamrefresher.BuildAKSUpstreamSpec({0x47fc220, 0xc00747db00}, {0x7f031d6bafc0, 0xc0011de930}, 0xc00e455000)
> /go/src/github.com/rancher/rancher/pkg/controllers/management/clusterupstreamrefresher/aks\_upstream\_spec.go:14 +0x65
> github.com/rancher/rancher/pkg/controllers/management/aks.(\*aksOperatorController).setInitialUpstreamSpec(0xc004874dd0, 0xc00e455000)
> /go/src/github.com/rancher/rancher/pkg/controllers/management/aks/aks\_cluster\_handler.go:269 +0x9f
> github.com/rancher/rancher/pkg/controllers/management/aks.(\*aksOperatorController).onClusterChange(0xc004874dd0, {0xc00e4afb68, 0x40b2d4}, 0x3e41980)
> /go/src/github.com/rancher/rancher/pkg/controllers/management/aks/aks\_cluster\_handler.go:154 +0x738
> github.com/rancher/rancher/pkg/generated/controllers/management.cattle.io/v3.FromClusterHandlerToHandler.func1({0xc00bcfc600, 0x18e93bc5b778}, {0x47b41b0, 0xc00e455000})
> /go/src/github.com/rancher/rancher/pkg/generated/controllers/management.cattle.io/v3/cluster.go:105 +0x44
> github.com/rancher/lasso/pkg/controller.SharedControllerHandlerFunc.OnChange(0xc00e455000, {0xc00bcfc600, 0x4015ceb}, {0x47b41b0, 0xc00e455000})
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220412224715-5f3517291ad4/pkg/controller/sharedcontroller.go:29 +0x38
> github.com/rancher/lasso/pkg/controller.(\*SharedHandler).OnChange(0xc0008ba780, {0xc00bcfc600, 0x7}, {0x47b41b0, 0xc00af29000})
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220412224715-5f3517291ad4/pkg/controller/sharedhandler.go:75 +0x23f
> github.com/rancher/lasso/pkg/controller.(\*controller).syncHandler(0xc000bfd3f0, {0xc00bcfc600, 0x7})
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220412224715-5f3517291ad4/pkg/controller/controller.go:220 +0x93
> github.com/rancher/lasso/pkg/controller.(\*controller).processSingleItem(0xc000bfd3f0, {0x35bd920, 0xc001f1a1d0})
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220412224715-5f3517291ad4/pkg/controller/controller.go:201 +0x10e
> github.com/rancher/lasso/pkg/controller.(\*controller).processNextWorkItem(0xc000bfd3f0)
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220412224715-5f3517291ad4/pkg/controller/controller.go:178 +0x46
> github.com/rancher/lasso/pkg/controller.(\*controller).runWorker(...)
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220412224715-5f3517291ad4/pkg/controller/controller.go:167
> k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f031d6d4050)
> /go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/wait/wait.go:155 +0x67
> k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0, {0x47886e0, 0xc003bb30b0}, 0x1, 0xc0018b6fc0)
> /go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/wait/wait.go:156 +0xb6
> k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x8, 0x3b9aca00, 0x0, 0x0, 0x0)
> /go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/wait/wait.go:133 +0x89
> k8s.io/apimachinery/pkg/util/wait.Until(0x7, 0x0, 0x0)
> /go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/wait/wait.go:90 +0x25
> created by github.com/rancher/lasso/pkg/controller.(\*controller).run
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220412224715-5f3517291ad4/pkg/controller/controller.go:135 +0x2c6
> panic: runtime error: index out of range \[1\] with length 0 \[recovered\]
> panic: runtime error: index out of range \[1\] with length 0
> 
> goroutine 2809 \[running\]:
> k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc001f1a1d0})
> /go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/runtime/runtime.go:55 +0xd8
> panic({0x3cf6500, 0xc00c10af78})
> /usr/lib64/go/1.17/src/runtime/panic.go:1038 +0x215
> github.com/rancher/aks-operator/controller.BuildUpstreamClusterState({0x47f9dd0, 0xc000084048}, {0x47fc220, 0xc00747db00}, {0x7f031d6bafc0, 0xc0011de930}, 0xc00e452b40)
> /go/pkg/mod/github.com/rancher/aks-operator@v1.0.5/controller/aks-cluster-config-handler.go:674 +0xa97
> github.com/rancher/rancher/pkg/controllers/management/clusterupstreamrefresher.BuildAKSUpstreamSpec({0x47fc220, 0xc00747db00}, {0x7f031d6bafc0, 0xc0011de930}, 0xc00e455000)
> /go/src/github.com/rancher/rancher/pkg/controllers/management/clusterupstreamrefresher/aks\_upstream\_spec.go:14 +0x65
> github.com/rancher/rancher/pkg/controllers/management/aks.(\*aksOperatorController).setInitialUpstreamSpec(0xc004874dd0, 0xc00e455000)
> /go/src/github.com/rancher/rancher/pkg/controllers/management/aks/aks\_cluster\_handler.go:269 +0x9f
> github.com/rancher/rancher/pkg/controllers/management/aks.(\*aksOperatorController).onClusterChange(0xc004874dd0, {0xc00e4afb68, 0x40b2d4}, 0x3e41980)
> /go/src/github.com/rancher/rancher/pkg/controllers/management/aks/aks\_cluster\_handler.go:154 +0x738
> github.com/rancher/rancher/pkg/generated/controllers/management.cattle.io/v3.FromClusterHandlerToHandler.func1({0xc00bcfc600, 0x18e93bc5b778}, {0x47b41b0, 0xc00e455000})
> /go/src/github.com/rancher/rancher/pkg/generated/controllers/management.cattle.io/v3/cluster.go:105 +0x44
> github.com/rancher/lasso/pkg/controller.SharedControllerHandlerFunc.OnChange(0xc00e455000, {0xc00bcfc600, 0x4015ceb}, {0x47b41b0, 0xc00e455000})
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220412224715-5f3517291ad4/pkg/controller/sharedcontroller.go:29 +0x38
> github.com/rancher/lasso/pkg/controller.(\*SharedHandler).OnChange(0xc0008ba780, {0xc00bcfc600, 0x7}, {0x47b41b0, 0xc00af29000})
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220412224715-5f3517291ad4/pkg/controller/sharedhandler.go:75 +0x23f
> github.com/rancher/lasso/pkg/controller.(\*controller).syncHandler(0xc000bfd3f0, {0xc00bcfc600, 0x7})
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220412224715-5f3517291ad4/pkg/controller/controller.go:220 +0x93
> github.com/rancher/lasso/pkg/controller.(\*controller).processSingleItem(0xc000bfd3f0, {0x35bd920, 0xc001f1a1d0})
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220412224715-5f3517291ad4/pkg/controller/controller.go:201 +0x10e
> github.com/rancher/lasso/pkg/controller.(\*controller).processNextWorkItem(0xc000bfd3f0)
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220412224715-5f3517291ad4/pkg/controller/controller.go:178 +0x46
> github.com/rancher/lasso/pkg/controller.(\*controller).runWorker(...)
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220412224715-5f3517291ad4/pkg/controller/controller.go:167
> k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f031d6d4050)
> /go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/wait/wait.go:155 +0x67
> k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0, {0x47886e0, 0xc003bb30b0}, 0x1, 0xc0018b6fc0)
> /go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/wait/wait.go:156 +0xb6
> k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x8, 0x3b9aca00, 0x0, 0x0, 0x0)
> /go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/wait/wait.go:133 +0x89
> k8s.io/apimachinery/pkg/util/wait.Until(0x7, 0x0, 0x0)
> /go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/wait/wait.go:90 +0x25
> created by github.com/rancher/lasso/pkg/controller.(\*controller).run
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220412224715-5f3517291ad4/pkg/controller/controller.go:135 +0x2c6
> \`\`\`
> 
> Before first discovered, I left this import running overnight and when I came back in the morning Rancher was completely unavailable as all pods were unavailable.
> 
> I have completely rebuilt Rancher on a new RKE2 cluster (previously discovered on a RKE v1 cluster) but this still remains. As far as I am aware, the AKS clusters were built in the exact same way.
> 
> \*\*To Reproduce\*\*
> 
> \*\*This is not easily reproducible, the resource group our service principal has permissions on contains 2 AKS clusters, one imports successfully and the other does not\*\*.
> 
> \* In Azure, create a service principal and give "Contributor" role to resource group containing relevant AKS cluster.
> \* Navigate to Cluster Management -\> Import Existing.
> \* Select "Azure AKS" and give the cluster a name.
> \* Select the credentials in the "Cloud Credentials" drop-down then click Next.
> \* Select the cluster from the "Cluster to register" drop-down, then click "Register Cluster".
> 
> \*\*Result\*\*
> 
> aks-config-operator and rancher pods continually crash and the cluster fails to import.
> 
> \*\*Expected Result\*\*
> 
> The cluster is imported and becomes visible in the "Cluster Management" section of Rancher, with all Rancher features available.
> 
> \*\*Screenshots\*\*
> N/A, logs should be sufficient.
> 
> \*\*Additional context\*\*
> None that I can think of which would be relevant.

> <https://github.com/rancher/rancher/issues/38377>
>
> \*\*Rancher Server Setup\*\*
> \- Rancher version: 2.5.10
> \- Installation option (Dock…er install/Helm Chart): Helm Chart
> - If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc):
> \`\`\`
> NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
> rancher cattle-system 5 2021-10-31 17:49:07.723621129 +0000 UTC deployed rancher-2.5.10 v2.5.10
> \`\`\`
> \- Proxy/Cert Details:
> 
> \*\*Information about the Cluster\*\*
> \- Kubernetes version: 1.21.10
> \- Cluster Type (Local/Downstream): Local
> - If downstream, what type of cluster? (Custom/Imported or specify provider for Hosted/Infrastructure Provider): AWS
> \<!--
> \* Custom = Running a docker command on a node
> \* Imported = Running kubectl apply onto an existing k8s cluster
> \* Hosted = EKS, GKE, AKS, etc
> \* Infrastructure Provider = Rancher provisioning the nodes using different node drivers (e.g. AWS, Digital Ocean, etc)
> \--\>
> 
> \*\*User Information\*\*
> \- What is the role of the user logged in? (Admin/Cluster Owner/Cluster Member/Project Owner/Project Member/Custom): Admin
> - If custom, define the set of permissions:
> 
> \*\*Describe the bug\*\*
> 
> Rancher crashes with a boundsError -- I see \`eks-operator\` in there, which I've set to 0 replicas but the issue persists, so I'm asking here
> deployments
> !\[image\](https://user-images.githubusercontent.com/109804792/180446070-a89735b1-005c-45b7-9945-dfe5880051e5.png)
> pods
> !\[image\](https://user-images.githubusercontent.com/109804792/180448213-60a2c1a4-3ef7-4bdd-9752-84cf4ada6d19.png)
> events for the rancher pod
> !\[image\](https://user-images.githubusercontent.com/109804792/180448297-64887d63-1fc9-44e4-8b53-3b28b06ceae0.png)
> 
> \*\*To Reproduce\*\*
> 
> I'm not sure, sorry. My best guess is an invalid EKS cluster definition somehow got past a schema check?
> 
> \*\*Result\*\*
> Rancher server breaks, returning 503 or 502
> !\[image\](https://user-images.githubusercontent.com/109804792/180450565-19f7463c-5d4f-40f7-b791-eb8a5eef2b30.png)
> 
> \*\*Expected Result\*\*
> 
> Rancher functions normally
> 
> \*\*Screenshots\*\*
> 
> 
> \*\*Additional context\*\*
> 
> Here's the full error
> \`\`\`
> E0722 12:59:58.631866 7 runtime.go:78\] Observed a panic: runtime.boundsError{x:0, y:0, signed:true, code:0x0} (runtime error: index out of range \[0\] with length 0)
> goroutine 1523 \[running\]:
> k8s.io/apimachinery/pkg/util/runtime.logPanic(0x3658a80, 0xc09820cba0)
> /go/pkg/mod/k8s.io/apimachinery@v0.20.6/pkg/util/runtime/runtime.go:74 +0x95
> k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
> /go/pkg/mod/k8s.io/apimachinery@v0.20.6/pkg/util/runtime/runtime.go:48 +0x86
> panic(0x3658a80, 0xc09820cba0)
> /usr/local/go/src/runtime/panic.go:965 +0x1b9
> github.com/rancher/eks-operator/controller.BuildUpstreamClusterState(0xc037b89ae0, 0x20, 0xc0631001c8, 0x14, 0xc0453237d0, 0xc0818104b8, 0x1, 0x1, 0xc0818104c8, 0xc062c43f00, ...)
> /go/pkg/mod/github.com/rancher/eks-operator@v1.0.9/controller/eks-cluster-config-handler.go:865 +0x1931
> github.com/rancher/rancher/pkg/controllers/management/clusterupstreamrefresher.BuildEKSUpstreamSpec(0x3f3d538, 0xc04cdf3050, 0xc04a2ad100, 0x30, 0xc0d89d79a0, 0x1)
> /go/src/github.com/rancher/rancher/pkg/controllers/management/clusterupstreamrefresher/eks\_upstream\_spec.go:80 +0xc92
> github.com/rancher/rancher/pkg/controllers/management/clusterupstreamrefresher.getComparableUpstreamSpec(0x3f3d538, 0xc04cdf3050, 0xc04a2ad100, 0x30, 0xc0d89d79a0, 0x1)
> /go/src/github.com/rancher/rancher/pkg/controllers/management/clusterupstreamrefresher/cluster\_upstream\_refresher.go:263 +0x14c
> github.com/rancher/rancher/pkg/controllers/management/clusterupstreamrefresher.(\*clusterRefreshController).refreshClusterUpstreamSpec(0xc04cdb7740, 0xc04a2ad100, 0x38cc6bf, 0x3, 0xa303efde8, 0x59d1700, 0x1)
> /go/src/github.com/rancher/rancher/pkg/controllers/management/clusterupstreamrefresher/cluster\_upstream\_refresher.go:147 +0xef
> github.com/rancher/rancher/pkg/controllers/management/clusterupstreamrefresher.(\*clusterRefreshController).onClusterChange(0xc04cdb7740, 0xc0065a5200, 0x7, 0xc04a2ad100, 0x37761a0, 0x383d380, 0x7ff83d307a98)
> /go/src/github.com/rancher/rancher/pkg/controllers/management/clusterupstreamrefresher/cluster\_upstream\_refresher.go:75 +0x225
> github.com/rancher/rancher/pkg/generated/controllers/management.cattle.io/v3.FromClusterHandlerToHandler.func1(0xc0065a5200, 0x7, 0x3ef5738, 0xc04a2ad100, 0xc04a2ad100, 0x7ff83d307a98, 0xc04a2ad100, 0x1)
> /go/src/github.com/rancher/rancher/pkg/generated/controllers/management.cattle.io/v3/cluster.go:105 +0x6b
> github.com/rancher/lasso/pkg/controller.SharedControllerHandlerFunc.OnChange(0xc04d858010, 0xc0065a5200, 0x7, 0x3ef5738, 0xc04a2ad100, 0x0, 0xc04a2ad100, 0x0, 0x0)
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210408231703-9ddd9378d08d/pkg/controller/sharedcontroller.go:29 +0x4e
> github.com/rancher/lasso/pkg/controller.(\*SharedHandler).OnChange(0xc001b351a0, 0xc0065a5200, 0x7, 0x3ef5738, 0xc04a2ad100, 0xc022e42c01, 0x0)
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210408231703-9ddd9378d08d/pkg/controller/sharedhandler.go:66 +0x123
> github.com/rancher/lasso/pkg/controller.(\*controller).syncHandler(0xc000dbd340, 0xc0065a5200, 0x7, 0xc032d3be58, 0x0)
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210408231703-9ddd9378d08d/pkg/controller/controller.go:210 +0xd1
> github.com/rancher/lasso/pkg/controller.(\*controller).processSingleItem(0xc000dbd340, 0x3089ba0, 0xc04da70400, 0x0, 0x0)
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210408231703-9ddd9378d08d/pkg/controller/controller.go:192 +0xe7
> github.com/rancher/lasso/pkg/controller.(\*controller).processNextWorkItem(0xc000dbd340, 0x203001)
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210408231703-9ddd9378d08d/pkg/controller/controller.go:169 +0x54
> github.com/rancher/lasso/pkg/controller.(\*controller).runWorker(...)
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210408231703-9ddd9378d08d/pkg/controller/controller.go:158
> k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc010473450)
> /go/pkg/mod/k8s.io/apimachinery@v0.20.6/pkg/util/wait/wait.go:155 +0x5f
> k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc010473450, 0x3ed54c0, 0xc022aa9050, 0x1, 0xc0019563c0)
> /go/pkg/mod/k8s.io/apimachinery@v0.20.6/pkg/util/wait/wait.go:156 +0x9b
> k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc010473450, 0x3b9aca00, 0x0, 0x1, 0xc0019563c0)
> /go/pkg/mod/k8s.io/apimachinery@v0.20.6/pkg/util/wait/wait.go:133 +0x98
> k8s.io/apimachinery/pkg/util/wait.Until(0xc010473450, 0x3b9aca00, 0xc0019563c0)
> /go/pkg/mod/k8s.io/apimachinery@v0.20.6/pkg/util/wait/wait.go:90 +0x4d
> created by github.com/rancher/lasso/pkg/controller.(\*controller).run
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210408231703-9ddd9378d08d/pkg/controller/controller.go:129 +0x33b
> panic: runtime error: index out of range \[0\] with length 0 \[recovered\]
> panic: runtime error: index out of range \[0\] with length 0
> 
> goroutine 1523 \[running\]:
> k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
> /go/pkg/mod/k8s.io/apimachinery@v0.20.6/pkg/util/runtime/runtime.go:55 +0x109
> panic(0x3658a80, 0xc09820cba0)
> /usr/local/go/src/runtime/panic.go:965 +0x1b9
> github.com/rancher/eks-operator/controller.BuildUpstreamClusterState(0xc037b89ae0, 0x20, 0xc0631001c8, 0x14, 0xc0453237d0, 0xc0818104b8, 0x1, 0x1, 0xc0818104c8, 0xc062c43f00, ...)
> /go/pkg/mod/github.com/rancher/eks-operator@v1.0.9/controller/eks-cluster-config-handler.go:865 +0x1931
> github.com/rancher/rancher/pkg/controllers/management/clusterupstreamrefresher.BuildEKSUpstreamSpec(0x3f3d538, 0xc04cdf3050, 0xc04a2ad100, 0x30, 0xc0d89d79a0, 0x1)
> /go/src/github.com/rancher/rancher/pkg/controllers/management/clusterupstreamrefresher/eks\_upstream\_spec.go:80 +0xc92
> github.com/rancher/rancher/pkg/controllers/management/clusterupstreamrefresher.getComparableUpstreamSpec(0x3f3d538, 0xc04cdf3050, 0xc04a2ad100, 0x30, 0xc0d89d79a0, 0x1)
> /go/src/github.com/rancher/rancher/pkg/controllers/management/clusterupstreamrefresher/cluster\_upstream\_refresher.go:263 +0x14c
> github.com/rancher/rancher/pkg/controllers/management/clusterupstreamrefresher.(\*clusterRefreshController).refreshClusterUpstreamSpec(0xc04cdb7740, 0xc04a2ad100, 0x38cc6bf, 0x3, 0xa303efde8, 0x59d1700, 0x1)
> /go/src/github.com/rancher/rancher/pkg/controllers/management/clusterupstreamrefresher/cluster\_upstream\_refresher.go:147 +0xef
> github.com/rancher/rancher/pkg/controllers/management/clusterupstreamrefresher.(\*clusterRefreshController).onClusterChange(0xc04cdb7740, 0xc0065a5200, 0x7, 0xc04a2ad100, 0x37761a0, 0x383d380, 0x7ff83d307a98)
> /go/src/github.com/rancher/rancher/pkg/controllers/management/clusterupstreamrefresher/cluster\_upstream\_refresher.go:75 +0x225
> github.com/rancher/rancher/pkg/generated/controllers/management.cattle.io/v3.FromClusterHandlerToHandler.func1(0xc0065a5200, 0x7, 0x3ef5738, 0xc04a2ad100, 0xc04a2ad100, 0x7ff83d307a98, 0xc04a2ad100, 0x1)
> /go/src/github.com/rancher/rancher/pkg/generated/controllers/management.cattle.io/v3/cluster.go:105 +0x6b
> github.com/rancher/lasso/pkg/controller.SharedControllerHandlerFunc.OnChange(0xc04d858010, 0xc0065a5200, 0x7, 0x3ef5738, 0xc04a2ad100, 0x0, 0xc04a2ad100, 0x0, 0x0)
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210408231703-9ddd9378d08d/pkg/controller/sharedcontroller.go:29 +0x4e
> github.com/rancher/lasso/pkg/controller.(\*SharedHandler).OnChange(0xc001b351a0, 0xc0065a5200, 0x7, 0x3ef5738, 0xc04a2ad100, 0xc022e42c01, 0x0)
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210408231703-9ddd9378d08d/pkg/controller/sharedhandler.go:66 +0x123
> github.com/rancher/lasso/pkg/controller.(\*controller).syncHandler(0xc000dbd340, 0xc0065a5200, 0x7, 0xc032d3be58, 0x0)
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210408231703-9ddd9378d08d/pkg/controller/controller.go:210 +0xd1
> github.com/rancher/lasso/pkg/controller.(\*controller).processSingleItem(0xc000dbd340, 0x3089ba0, 0xc04da70400, 0x0, 0x0)
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210408231703-9ddd9378d08d/pkg/controller/controller.go:192 +0xe7
> github.com/rancher/lasso/pkg/controller.(\*controller).processNextWorkItem(0xc000dbd340, 0x203001)
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210408231703-9ddd9378d08d/pkg/controller/controller.go:169 +0x54
> github.com/rancher/lasso/pkg/controller.(\*controller).runWorker(...)
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210408231703-9ddd9378d08d/pkg/controller/controller.go:158
> k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc010473450)
> /go/pkg/mod/k8s.io/apimachinery@v0.20.6/pkg/util/wait/wait.go:155 +0x5f
> k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc010473450, 0x3ed54c0, 0xc022aa9050, 0x1, 0xc0019563c0)
> /go/pkg/mod/k8s.io/apimachinery@v0.20.6/pkg/util/wait/wait.go:156 +0x9b
> k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc010473450, 0x3b9aca00, 0x0, 0x1, 0xc0019563c0)
> /go/pkg/mod/k8s.io/apimachinery@v0.20.6/pkg/util/wait/wait.go:133 +0x98
> k8s.io/apimachinery/pkg/util/wait.Until(0xc010473450, 0x3b9aca00, 0xc0019563c0)
> /go/pkg/mod/k8s.io/apimachinery@v0.20.6/pkg/util/wait/wait.go:90 +0x4d
> created by github.com/rancher/lasso/pkg/controller.(\*controller).run
> /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20210408231703-9ddd9378d08d/pkg/controller/controller.go:129 +0x33b
> \`\`\`
> 
> I found similar issues \[here\](https://github.com/rancher/rancher/issues/28533) and \[here\](https://github.com/rancher/rancher/issues/37863), but I didn't really get any info out of them
> I know k8s 1.21 is not listed on the support matrix for 2.5.10, but we've used it for a couple of months without issue
> I was thinking about trying an upgrade to 2.5.14, but I'm not sure it's a good idea (don't want to break the server for good)
> 
> Please let me know if more information is needed

Something seems to be broken, any help/workaround from the Rancher team will be appreciated.