# Kubectl exec fails with 403 forbidden

**URL:** <https://forums.suse.com/t/kubectl-exec-fails-with-403-forbidden/42920>\
**Category:** SUSE Rancher Prime\
**Created:** [April 10, 2024, 4:08pm UTC](https://forums.suse.com/t/kubectl-exec-fails-with-403-forbidden/42920 "2024-04-10T16:08:40Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![jDmacD](https://avatars.discourse-cdn.com/v4/letter/j/58956e/32.png) [@jDmacD](https://forums.suse.com/u/jDmacD)\
**Post date:** [April 10, 2024, 4:08pm UTC](https://forums.suse.com/t/kubectl-exec-fails-with-403-forbidden/42920/1 "2024-04-10T16:08:40Z")

</div>

I am having a persistent issue where I cannot `kubectl exec` into a pod using

- The UI based terminal in rancher
- The rancher cli from a local machine
- kubectl using a Rancher genenerated kubeconfig from a local machine

I have a single node k3s cluster with Rancher deployed. From this I am creating a 3 node rke2 cluster hosted on vsphere.

```auto
Client Version: v1.29.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.12+rke2r1

```

I am using the Default Admin (admin) account. To the rke2 cluster I have deployed a single pod with one container (wbitt/multitool) to the `default` namespace which is part of the `Default` project.

I can exec straight to the running container via the `Execute Shell` option on the pod itself without issue. This leads me to believe that this is not permissions problem, ignoring the fact that I am Default Admin.

When I try to exec with any other method it fails, for example:

```sh
kubectl -n default exec -it multitool -- sh
Error from server:

```

Cranking up the verbosity reveals a more detailed error

```sh
[nix-shell:~/code/kubernetes/clusters/tools]$ rancher kubectl -v=7 -n default exec -it multitool -- sh
I0410 15:08:03.235139 25175 loader.go:395] Config loaded from file: /run/user/1000/rancher-3936582872
I0410 15:08:03.238706 25175 round_trippers.go:463] GET https://company.ie/k8s/clusters/c-m-5wrsnz9q/api/v1/namespaces/default/pods/multitool
I0410 15:08:03.238733 25175 round_trippers.go:469] Request Headers:
I0410 15:08:03.238740 25175 round_trippers.go:473] Accept: application/json, */*
I0410 15:08:03.238759 25175 round_trippers.go:473] User-Agent: kubectl/v1.29.3 (linux/amd64) kubernetes/6813625
I0410 15:08:03.238782 25175 round_trippers.go:473] Authorization: Bearer <masked>
I0410 15:08:03.349554 25175 round_trippers.go:574] Response Status: 200 OK in 110 milliseconds
I0410 15:08:03.351282 25175 podcmd.go:88] Defaulting container name to container-0
I0410 15:08:03.351850 25175 round_trippers.go:463] POST https://company.ie/k8s/clusters/c-m-5wrsnz9q/api/v1/namespaces/default/pods/multitool/exec?command=sh&container=container-0&stdin=true&stdout=true&tty=true
I0410 15:08:03.351874 25175 round_trippers.go:469] Request Headers:
I0410 15:08:03.351883 25175 round_trippers.go:473] X-Stream-Protocol-Version: v5.channel.k8s.io
I0410 15:08:03.351888 25175 round_trippers.go:473] X-Stream-Protocol-Version: v4.channel.k8s.io
I0410 15:08:03.351891 25175 round_trippers.go:473] X-Stream-Protocol-Version: v3.channel.k8s.io
I0410 15:08:03.351894 25175 round_trippers.go:473] X-Stream-Protocol-Version: v2.channel.k8s.io
I0410 15:08:03.351897 25175 round_trippers.go:473] X-Stream-Protocol-Version: channel.k8s.io
I0410 15:08:03.351899 25175 round_trippers.go:473] User-Agent: kubectl/v1.29.3 (linux/amd64) kubernetes/6813625
I0410 15:08:03.351902 25175 round_trippers.go:473] Authorization: Bearer <masked>
I0410 15:08:03.435878 25175 round_trippers.go:574] Response Status: 403 Forbidden in 83 milliseconds
I0410 15:08:03.436434 25175 helpers.go:246] server response object: [{
  "metadata": {}
}]
Error from server:
exit status 1

```

`Response Status: 403 Forbidden` is obviously the issue here, but it is not clear exactly what is forbidden.

Similar issues are present on the forum, [this post](https://forums.suse.com/t/kubectl-exec-log-get-error-from-server/36852/6) for example, but the answer make no sense.

Most of the posts point to a websockets issue. I have followed the steps specified in [this kb article](https://www.suse.com/support/kb/doc/?id=000020189) and received a stream of json. To me that rules out a websocket issue.

---

<div class="post-metadata">

**Author:** ![jDmacD](https://avatars.discourse-cdn.com/v4/letter/j/58956e/32.png) [@jDmacD](https://forums.suse.com/u/jDmacD)\
**Post date:** [April 11, 2024, 10:39am UTC](https://forums.suse.com/t/kubectl-exec-fails-with-403-forbidden/42920/2 "2024-04-11T10:39:53Z")

</div>

The cluster hosting Rancher has the Cilium CNI installed and is using the Gateway API. To be 100% sure this is not a websocket issue I deployed a [websocat](https://github.com/vi/websocat) pod and tested it form my local command line which works perfectly.

---

<div class="post-metadata">

**Author:** ![jDmacD](https://avatars.discourse-cdn.com/v4/letter/j/58956e/32.png) [@jDmacD](https://forums.suse.com/u/jDmacD)\
**Post date:** [April 15, 2024, 2:01pm UTC](https://forums.suse.com/t/kubectl-exec-fails-with-403-forbidden/42920/3 "2024-04-15T14:01:18Z")

</div>

Clarifications and updates

> [@jDmacD](#):
>
> The UI based terminal in rancher

This actually _does_ work. There is just a horrible delay on the command

When I said the following;

> [@jDmacD](#):
>
> I can exec straight to the running container via the `Execute Shell` option on the pod itself without issue. This leads me to believe that this is not permissions problem, ignoring the fact that I am Default Admin.

I meant this UI functionality

 ![ui_execute](https://us1.discourse-cdn.com/flex022/uploads/suse/original/2X/3/3805fdd1f5235f88f9cc2e7126d148d15994c1b5.png)

I have two clusters.

- A manually created k3s cluster that Rancher is deployed to
- An rke 2 cluster on vSphere created by Rancher

Dowloading a kubeconfig for either cluster and attempting `kubectl exec` fails. In both cases sshing onto the nodes and using the kubconfigs `/etc/rancher/k3s/k3s.yaml` and `/etc/rancher/rke2/rke2.yaml` work as expected.

I have done further testing with websocat. This command

```auto
websocat \
  "wss://company.ie/k8s/clusters/c-m5wrsnz9q/api/v1/namespaces/default/pods/multitool/exec?command=hostname&container=container-0&stdin=true&stdout=true&tty=true" \
--header "Authorization: Bearer token-m4v5s:xxxxxxxxx"

```

Will perform an exec and return the hostname. This is more confusing as it is further proof that this is not a websocket issue.

---

<div class="post-metadata">

**Author:** ![bpedersen2](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/bpedersen2/32/9899_2.png) [@bpedersen2](https://forums.suse.com/u/bpedersen2)\
**Post date:** [April 24, 2024, 2:21pm UTC](https://forums.suse.com/t/kubectl-exec-fails-with-403-forbidden/42920/4 "2024-04-24T14:21:32Z")

</div>

Sounds a bit related to [[BUG] Rancher can no longer provision harvester machines after restart · Issue #44912 · rancher/rancher · GitHub](https://github.com/rancher/rancher/issues/44912) and [[BUG] The Kubeconfig from `Support -\> Download Kubeconfig`, for use with Terraform eventually goes bad / stops working, spitting back `unauthorized` · Issue #5669 · harvester/harvester · GitHub](https://github.com/harvester/harvester/issues/5669)
