# Timeout on kubectl exec on with nginx ssl termination

**URL:** <https://forums.suse.com/t/timeout-on-kubectl-exec-on-with-nginx-ssl-termination/3916>\
**Category:** Rancher 1.x\
**Created:** [September 7, 2016, 9:12am UTC](https://forums.suse.com/t/timeout-on-kubectl-exec-on-with-nginx-ssl-termination/3916 "2016-09-07T09:12:46Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![markcu](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/markcu/32/1476_2.png) [@markcu](https://forums.suse.com/u/markcu)\
**Post date:** [September 7, 2016, 9:12am UTC](https://forums.suse.com/t/timeout-on-kubectl-exec-on-with-nginx-ssl-termination/3916/1 "2016-09-07T09:12:46Z")

</div>

I’m trying to debug an issue using `kubectl` and Rancher. I’ve upgraded our Rancher setup to 1.2.0-pre2 because I’d like to use `kubectl exec`. But when I run it, it’ll get a timeout error. I’ve tried running on the same machine as I have Rancher running and I get the same error. I can certainly do `kubectl exec` from within the UI kubectl shell and other kubectl commands work fine.

The only thing I can think of is that we have an nginx ssl termination in front of rancher, as per the [official instructions](http://docs.rancher.com/rancher/v1.2/en/installing-rancher/installing-server/basic-ssl-config/). I’ve tried playing around with the nginx config, adding support for http2 and so on but no luck.

Any suggestions welcome?

Snippet from verbose output from kubectl (–v=9):

`I0907 09:09:52.464221 1 exec.go:179] defaulting container name to mongo I0907 09:09:52.464789 1 round_trippers.go:299] curl -k -v -XPOST -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" https://rancher.dev.stpl.stormtest.com/r/projects/1a8/kubernetes/api/v1/namespaces/default/pods/mongo-kwf8k/exec?command=ls&command=C%3A%2FProgram&command=Files%2FGit&container=mongo&container=mongo&stderr=true&stdout=true&tty=true I0907 09:09:52.465308 1 round_trippers.go:299] curl -k -v -XPOST -H "User-Agent: kubectl/v1.3.0 (linux/amd64) kubernetes/2831379" -H "Authorization: Basic OTU1OUVDRTE5QjUwQUIxQkU3NzQ6VnRHTUdUQ3lZU0d6Tm85OXB3Z0FaNXB5c2RkeWgxckoyNnlBNlFTZQ==" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" https://rancher.dev.stpl.stormtest.com/r/projects/1a8/kubernetes/api/v1/namespaces/default/pods/mongo-kwf8k/exec?command=ls&command=C%3A%2FProgram&command=Files%2FGit&container=mongo&container=mongo&stderr=true&stdout=true&tty=true I0907 09:09:52.639324 1 round_trippers.go:318] POST https://rancher.dev.stpl.stormtest.com/r/projects/1a8/kubernetes/api/v1/namespaces/default/pods/mongo-kwf8k/exec?command=ls&command=C%3A%2FProgram&command=Files%2FGit&container=mongo&container=mongo&stderr=true&stdout=true&tty=true 101 Switching Protocols in 173 milliseconds I0907 09:09:52.640316 1 round_trippers.go:324] Response Headers: I0907 09:09:52.640783 1 round_trippers.go:327] Server: nginx/1.9.15 I0907 09:09:52.641125 1 round_trippers.go:327] Date: Wed, 07 Sep 2016 09:09:47 GMT I0907 09:09:52.641484 1 round_trippers.go:327] Connection: upgrade I0907 09:09:52.641725 1 round_trippers.go:327] Upgrade: SPDY/3.1 I0907 09:09:52.642070 1 round_trippers.go:327] X-Stream-Protocol-Version: v2.channel.k8s.io I0907 09:09:52.642303 1 round_trippers.go:318] POST https://rancher.dev.stpl.stormtest.com/r/projects/1a8/kubernetes/api/v1/namespaces/default/pods/mongo-kwf8k/exec?command=ls&command=C%3A%2FProgram&command=Files%2FGit&container=mongo&container=mongo&stderr=true&stdout=true&tty=true 101 Switching Protocols in 177 milliseconds I0907 09:09:52.642738 1 round_trippers.go:324] Response Headers: I0907 09:09:52.643155 1 round_trippers.go:327] X-Stream-Protocol-Version: v2.channel.k8s.io I0907 09:09:52.643580 1 round_trippers.go:327] Server: nginx/1.9.15 I0907 09:09:52.643958 1 round_trippers.go:327] Date: Wed, 07 Sep 2016 09:09:47 GMT I0907 09:09:52.644205 1 round_trippers.go:327] Connection: upgrade I0907 09:09:52.644501 1 round_trippers.go:327] Upgrade: SPDY/3.1 F0907 09:10:22.657095 1 helpers.go:108] error: Timeout occured`

---

<div class="post-metadata">

**Author:** ![markcu](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/markcu/32/1476_2.png) [@markcu](https://forums.suse.com/u/markcu)\
**Post date:** [September 13, 2016, 9:45am UTC](https://forums.suse.com/t/timeout-on-kubectl-exec-on-with-nginx-ssl-termination/3916/2 "2016-09-13T09:45:24Z")

</div>

Small update to this. I spun up a second rancher instance without nginx ssl termination and I was able to run remove commands. It’s definitely the nginx configuration.

---

<div class="post-metadata">

**Author:** ![markcu](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/markcu/32/1476_2.png) [@markcu](https://forums.suse.com/u/markcu)\
**Post date:** [September 13, 2016, 11:55am UTC](https://forums.suse.com/t/timeout-on-kubectl-exec-on-with-nginx-ssl-termination/3916/3 "2016-09-13T11:55:54Z")

</div>

Found the problem and it’s frigg’in dumb. For kubectl to work “upgrade” needs to be capitalized:

> proxy\_set\_header Connection “Upgrade”;

So your nginx configuration should be:

> upstream rancher {  
> server rancher-server:8080;  
> }
> 
> server {  
> listen 443 ssl;  
> server\_name ;  
> ssl\_certificate \<cert\_file\>;  
> ssl\_certificate\_key \<key\_file\>;
> 
> ```
> location / {
> proxy_set_header Host $host;
> proxy_set_header X-Forwarded-Proto $scheme;
> proxy_set_header X-Forwarded-Port $server_port;
> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
> proxy_pass http://rancher;
> proxy_http_version 1.1;
> proxy_set_header Upgrade $http_upgrade;
> proxy_set_header Connection "Upgrade";
> # This allows the ability for the execute shell window to remain open for up to 15 minutes. Without this parameter, the default is 1 minute and will automatically close.
> proxy_read_timeout 900s;
> }
> 
> ```
> 
> }
> 
> server {  
> listen 80;  
> server\_name ;  
> return 301 https://$server\_name$request\_uri;  
> }

---

<div class="post-metadata">

**Author:** ![thatcher](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/thatcher/32/4690_2.png) [@thatcher](https://forums.suse.com/u/thatcher)\
**Post date:** [July 12, 2017, 8:24pm UTC](https://forums.suse.com/t/timeout-on-kubectl-exec-on-with-nginx-ssl-termination/3916/4 "2017-07-12T20:24:03Z")

</div>

OMG. You’re my life saver. Such a small and annoying little thing! I was using the jwilder\_nginx proxy, and it has upgrade configured by default; but not Upgrade.

---

<div class="post-metadata">

**Author:** ![vincent](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/vincent/32/7156_2.png) [@vincent](https://forums.suse.com/u/vincent)\
**Post date:** [July 12, 2017, 8:44pm UTC](https://forums.suse.com/t/timeout-on-kubectl-exec-on-with-nginx-ssl-termination/3916/5 "2017-07-12T20:44:22Z")

</div>

Related: [https://github.com/rancher/rancher/issues/9155](https://github.com/rancher/rancher/issues/9155) . The header is supposed to be treated as case-insensitive, so this is a bug in k8s. I don’t know that that person ever put a bug in for k8s if you’d like to do so yourself. You could also probably put in a 1 byte PR for jwilder/nginx-proxy to make it capital by default, since that is much more common than lowercase.
