# Vsphere Provisioning - "Stuck at SSH Step" **URL:** \ **Category:** SUSE Rancher Prime\ **Created:** [May 17, 2018, 8:49pm UTC](https://forums.suse.com/t/vsphere-provisioning-stuck-at-ssh-step/10465 "2018-05-17T20:49:47Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Gneisler](https://avatars.discourse-cdn.com/v4/letter/g/59ef9b/32.png) [@Gneisler](https://forums.suse.com/u/Gneisler)\ **Post date:** [May 17, 2018, 8:49pm UTC](https://forums.suse.com/t/vsphere-provisioning-stuck-at-ssh-step/10465/1 "2018-05-17T20:49:47Z")
Using Rancher 2.0.0 I am trying to build nodes through vSphere node driver. I am getting the VM created with Rancher OS latest, however I get stuck on rancher “Waiting for SSH to be Available” It then after several attempts deletes the VM and recreates it. If rancher just created the VM why would it not have access to the VM to SSH? Is there a step I am missing? I’d really love to be able to provision nodes through Rancher UI, however I am stuck at this point. Thanks in advance. Garrett ---
**Author:** ![JockDaRock](https://avatars.discourse-cdn.com/v4/letter/j/58f4c7/32.png) [@JockDaRock](https://forums.suse.com/u/JockDaRock)\ **Post date:** [May 21, 2018, 7:19pm UTC](https://forums.suse.com/t/vsphere-provisioning-stuck-at-ssh-step/10465/2 "2018-05-21T19:19:14Z")
Any help from the rancher team on this so we can properly kick the tires would be appreciated. Kind Regards ---
**Author:** ![superseb](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/superseb/32/2424_2.png) [@superseb](https://forums.suse.com/u/superseb)\ **Post date:** [May 21, 2018, 8:37pm UTC](https://forums.suse.com/t/vsphere-provisioning-stuck-at-ssh-step/10465/3 "2018-05-21T20:37:33Z")
What ISO are you booting? Is the network it is attached to based on DHCP? Does it get an IP address? Is the host running the `rancher/rancher` container able to SSH to that IP (you can manually verify while the host is still up). Does the ISO fully boot (check VMware console)? ---
**Author:** ![JockDaRock](https://avatars.discourse-cdn.com/v4/letter/j/58f4c7/32.png) [@JockDaRock](https://forums.suse.com/u/JockDaRock)\ **Post date:** [May 21, 2018, 8:48pm UTC](https://forums.suse.com/t/vsphere-provisioning-stuck-at-ssh-step/10465/4 "2018-05-21T20:48:20Z")
Hey @superseb , thank you for the reply. I can’t speak for @Gneisler, but I am booting from the default ISO. Which from the settings looks like a boot2docker iso. The network is statically defined for me. So NO DHCP. If you could tell me how to pass a statically defined IP address to the node through the use of Rancher, i would greatly appreciate that. The ISO does fully boot, but the eth0 address does not have an IP and therefore I can’t reach it from the rancher/rancher container via ssh. Do you have any other questions or thoughts on your end? Kind Regards, Jock ---
**Author:** ![Gneisler](https://avatars.discourse-cdn.com/v4/letter/g/59ef9b/32.png) [@Gneisler](https://forums.suse.com/u/Gneisler)\ **Post date:** [May 21, 2018, 9:03pm UTC](https://forums.suse.com/t/vsphere-provisioning-stuck-at-ssh-step/10465/5 "2018-05-21T21:03:54Z")
@superseb I am running Rancher 1.3.0, DHCP setup and my VM does come up and obtains an IP. 10.10.10.170 in this case and it fully boots. I am able to get to login on SSH (Putty), however I dont know the password and I think that is part of the problem that Rancher can’t SSH into the newly created box either. Yes, my Rancher Host (10.10.10.100) Can SSH into 10.10.10.170, however password is still unknown. ---
**Author:** ![Gneisler](https://avatars.discourse-cdn.com/v4/letter/g/59ef9b/32.png) [@Gneisler](https://forums.suse.com/u/Gneisler)\ **Post date:** [May 21, 2018, 9:15pm UTC](https://forums.suse.com/t/vsphere-provisioning-stuck-at-ssh-step/10465/6 "2018-05-21T21:15:17Z")
Here is my logs from the node creation (Note, I do have another Custom Cluster running) : 2018-05-21 19:51:24.323176 I | mvcc: store.index: compact 2808835 2018-05-21 19:51:24.452858 I | mvcc: finished scheduled compaction at 2808835 (took 114.099566ms) 2018-05-21 19:56:24.344569 I | mvcc: store.index: compact 2809313 2018-05-21 19:56:24.476282 I | mvcc: finished scheduled compaction at 2809313 (took 115.453652ms) 2018-05-21 20:01:24.371517 I | mvcc: store.index: compact 2809793 2018-05-21 20:01:24.501974 I | mvcc: finished scheduled compaction at 2809793 (took 114.967135ms) 2018-05-21 20:06:24.393680 I | mvcc: store.index: compact 2810272 2018-05-21 20:06:24.526386 I | mvcc: finished scheduled compaction at 2810272 (took 115.516733ms) E0521 20:08:39.056161 1 streamwatcher.go:109] Unable to decode an event from the watch stream: json: cannot unmarshal string into Go struct field dynamicEvent.Object of type v3.NodeStatus 2018-05-21 20:11:24.423893 I | mvcc: store.index: compact 2810749 2018-05-21 20:11:24.554361 I | mvcc: finished scheduled compaction at 2810749 (took 113.94832ms) 2018-05-21 20:16:24.446828 I | mvcc: store.index: compact 2811227 2018-05-21 20:16:24.580335 I | mvcc: finished scheduled compaction at 2811227 (took 117.357646ms) 2018-05-21 20:21:24.470289 I | mvcc: store.index: compact 2811708 2018-05-21 20:21:24.603625 I | mvcc: finished scheduled compaction at 2811708 (took 116.863508ms) 2018-05-21 20:26:24.493156 I | mvcc: store.index: compact 2812186 2018-05-21 20:26:24.627190 I | mvcc: finished scheduled compaction at 2812186 (took 116.671218ms) E0521 20:29:21.065702 1 streamwatcher.go:109] Unable to decode an event from the watch stream: json: cannot unmarshal string into Go struct field dynamicEvent.Object of type v3.NodeStatus 2018-05-21 20:31:24.517165 I | mvcc: store.index: compact 2812664 2018-05-21 20:31:24.649320 I | mvcc: finished scheduled compaction at 2812664 (took 116.825798ms) 2018-05-21 20:33:21.362663 W | etcdserver: apply entries took too long [117.338042ms for 1 entries] 2018-05-21 20:33:21.362710 W | etcdserver: avoid queries with large range/delete range! 2018-05-21 20:36:24.538559 I | mvcc: store.index: compact 2813143 2018-05-21 20:36:24.673483 I | mvcc: finished scheduled compaction at 2813143 (took 118.002293ms) 2018-05-21 20:41:24.561623 I | mvcc: store.index: compact 2813623 2018-05-21 20:41:24.691912 I | mvcc: finished scheduled compaction at 2813623 (took 114.756605ms) 2018/05/21 20:46:07 [INFO] Running cluster events cleanup 2018/05/21 20:46:07 [INFO] Done running cluster events cleanup 2018-05-21 20:46:24.590199 I | mvcc: store.index: compact 2814101 2018-05-21 20:46:24.724396 I | mvcc: finished scheduled compaction at 2814101 (took 117.250385ms) 2018-05-21 20:51:24.613519 I | mvcc: store.index: compact 2814583 2018-05-21 20:51:24.745679 I | mvcc: finished scheduled compaction at 2814583 (took 116.752901ms) E0521 20:53:51.075275 1 streamwatcher.go:109] Unable to decode an event from the watch stream: json: cannot unmarshal string into Go struct field dynamicEvent.Object of type v3.NodeStatus 2018-05-21 20:56:24.635070 I | mvcc: store.index: compact 2815062 2018-05-21 20:56:24.769395 I | mvcc: finished scheduled compaction at 2815062 (took 119.229249ms) 2018/05/21 20:57:37 [INFO] stdout: Creating CA: management-state/node/nodes/test-master1/certs/ca.pem 2018/05/21 20:57:38 [INFO] stdout: Creating client certificate: management-state/node/nodes/test-master1/certs/cert.pem 2018/05/21 20:57:38 [INFO] stdout: Running pre-create checks… 2018/05/21 20:57:43 [INFO] stdout: Creating machine… 2018/05/21 20:57:43 [INFO] stdout: (test-master1) Image cache directory does not exist, creating it at management-state/node/nodes/test-master1/cache… 2018/05/21 20:57:43 [INFO] stdout: (test-master1) Downloading management-state/node/nodes/test-master1/cache/boot2docker.iso from [https://releases.rancher.com/os/latest/rancheros.iso](https://releases.rancher.com/os/latest/rancheros.iso)… 2018/05/21 20:57:49 [INFO] stdout: (test-master1) 0%!.(MISSING)…10%!.(MISSING)…20%!.(MISSING)…30%!.(MISSING)…40%!.(MISSING)…50%!.(MISSING)…60%!.(MISSING)…70%!.(MISSING)…80%!.(MISSING)…90%!.(MISSING)…100%!(NOVERB) 2018/05/21 20:57:49 [INFO] stdout: (test-master1) Generating SSH Keypair… 2018/05/21 20:57:55 [INFO] stdout: (test-master1) Creating VM… 2018/05/21 20:58:04 [INFO] stdout: (test-master1) Uploading Boot2docker ISO … 2018/05/21 20:58:11 [INFO] stdout: (test-master1) adding network: Data Network V101 2018/05/21 20:58:11 [INFO] stdout: (test-master1) Reconfiguring VM 2018/05/21 20:58:37 [INFO] stdout: (test-master1) Waiting for VMware Tools to come online… 2018/05/21 21:00:08 [INFO] stdout: (test-master1) Provisioning certs and ssh keys… 2018/05/21 21:00:09 [INFO] stdout: Waiting for machine to be running, this may take a few minutes… 2018/05/21 21:00:11 [INFO] stdout: Detecting operating system of created instance… 2018/05/21 21:00:11 [INFO] stdout: Waiting for SSH to be available… 2018-05-21 21:01:24.664589 I | mvcc: store.index: compact 2815543 2018-05-21 21:01:24.804049 I | mvcc: finished scheduled compaction at 2815543 (took 117.123078ms) 2018-05-21 21:06:24.695000 I | mvcc: store.index: compact 2816122 2018-05-21 21:06:24.845629 I | mvcc: finished scheduled compaction at 2816122 (took 134.439012ms) 2018/05/21 21:08:04 [INFO] stdout: The default lines below are for a sh/bash shell, you can specify the shell you’re using, with the --shell flag. 2018/05/21 21:08:04 [INFO] stdout: 2018/05/21 21:08:04 [INFO] Generating and uploading node config test-master1 E0521 21:08:04.877775 1 generic\_controller.go:204] NodeController c-td7ld/m-bz9qd [node-controller] failed with : Error creating machine: Error detecting OS: Too many retries waiting for SSH to be available. Last error: Maximum number of retries (60) exceeded 2018/05/21 21:08:04 [INFO] NodeController c-td7ld/m-bz9qd completed with dropped err: [node-controller] failed with : Error 2018/05/21 21:08:05 [INFO] NodeController c-td7ld/m-bz9qd completed with dropped err: [node-controller] failed with : Error E0521 21:09:39.082617 1 streamwatcher.go:109] Unable to decode an event from the watch stream: json: cannot unmarshal string into Go struct field dynamicEvent.Object of type v3.NodeStatus 2018/05/21 21:10:24 [INFO] stdout: Creating CA: management-state/node/nodes/test-master1/certs/ca.pem 2018/05/21 21:10:25 [INFO] stdout: Creating client certificate: management-state/node/nodes/test-master1/certs/cert.pem 2018/05/21 21:10:26 [INFO] stdout: Running pre-create checks… 2018/05/21 21:10:30 [INFO] stdout: Creating machine… 2018/05/21 21:10:30 [INFO] stdout: (test-master1) Image cache directory does not exist, creating it at management-state/node/nodes/test-master1/cache… 2018/05/21 21:10:30 [INFO] stdout: (test-master1) Downloading management-state/node/nodes/test-master1/cache/boot2docker.iso from [https://releases.rancher.com/os/latest/rancheros.iso](https://releases.rancher.com/os/latest/rancheros.iso)… 2018/05/21 21:10:36 [INFO] stdout: (test-master1) 0%!.(MISSING)…10%!.(MISSING)…20%!.(MISSING)…30%!.(MISSING)…40%!.(MISSING)…50%!.(MISSING)…60%!.(MISSING)…70%!.(MISSING)…80%!.(MISSING)…90%!.(MISSING)…100%!(NOVERB) 2018/05/21 21:10:36 [INFO] stdout: (test-master1) Generating SSH Keypair… 2018/05/21 21:10:42 [INFO] stdout: (test-master1) Creating VM… 2018/05/21 21:10:55 [INFO] stdout: (test-master1) Uploading Boot2docker ISO … 2018/05/21 21:11:02 [INFO] stdout: (test-master1) adding network: Data Network V101 2018/05/21 21:11:02 [INFO] stdout: (test-master1) Reconfiguring VM 2018-05-21 21:11:24.722537 I | mvcc: store.index: compact 2816600 2018-05-21 21:11:24.852673 I | mvcc: finished scheduled compaction at 2816600 (took 114.539928ms) 2018/05/21 21:11:39 [INFO] stdout: (test-master1) Waiting for VMware Tools to come online… ---
**Author:** ![superseb](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/superseb/32/2424_2.png) [@superseb](https://forums.suse.com/u/superseb)\ **Post date:** [May 22, 2018, 6:59am UTC](https://forums.suse.com/t/vsphere-provisioning-stuck-at-ssh-step/10465/7 "2018-05-22T06:59:23Z")
Did you specify `rancher` as SSH User? ---
**Author:** ![Gneisler](https://avatars.discourse-cdn.com/v4/letter/g/59ef9b/32.png) [@Gneisler](https://forums.suse.com/u/Gneisler)\ **Post date:** [May 22, 2018, 3:38pm UTC](https://forums.suse.com/t/vsphere-provisioning-stuck-at-ssh-step/10465/8 "2018-05-22T15:38:52Z")
I have downloaded the rsa key from the created node. Converted this to a ppk file and attempted to ssh using putty using the newly created ppk file, logging in as rancher. Key is rejected. ![image](https://us1.discourse-cdn.com/flex022/uploads/suse/original/2X/c/caa6437b25565ec94e8dcd7f9c24508835344b38.png) My problem with this is Rancher itself can’t SSH to a node it creates…which means I am missing some configuration to specify the SSH password?! What is the default password? What am I missing? ---
**Author:** ![Gneisler](https://avatars.discourse-cdn.com/v4/letter/g/59ef9b/32.png) [@Gneisler](https://forums.suse.com/u/Gneisler)\ **Post date:** [May 22, 2018, 6:12pm UTC](https://forums.suse.com/t/vsphere-provisioning-stuck-at-ssh-step/10465/9 "2018-05-22T18:12:55Z")
I think this is relevant > > > \*\*RancherOS Version: (ros os version)\*\* > v1.2.0 > > \*\*Where are you running Ranch…erOS? (docker-machine, AWS, GCE, baremetal, etc.)\*\* > VMWare ESXi 6.5 > > I'm trying to launch RancherOS on ESXi 6.5 via the Rancher Web UI but I'm getting stuck on "Waiting for SSH". I've tried to launch an instance via docker-machine manually and get the same problem. > > I believe the issue is that docker-machine is unable to inject its SSH keys into RancherOS, but I'm not sure where to go from there. > > Does anyone have any ideas? > > docker-machine command: > \`\`\` > docker-machine -D create \\ > --engine-registry-mirror "http://???" \\ > --driver vmwarevsphere \\ > --vmwarevsphere-username ???\\ > --vmwarevsphere-password ???\\ > --vmwarevsphere-vcenter ??? \\ > --vmwarevsphere-network ??? \\ > --vmwarevsphere-datastore ??? \\ > --vmwarevsphere-disk-size 10000 \\ > --vmwarevsphere-cpu-count 2 \\ > --vmwarevsphere-memory-size 2048 \\ > --vmwarevsphere-boot2docker-url "http://releases.rancher.com/os/latest/rancheros.iso" \\ > --vmwarevsphere-cloudinit "https://???" \\ > rancheros-test > \`\`\`\` > > docker-machine debug log: > \`\`\` > Docker Machine Version: 0.14.0, build 89b8332 > Found binary path at /usr/local/bin/docker-machine > Launching plugin server for driver vmwarevsphere > Plugin server listening at address 127.0.0.1:37875 > () Calling .GetVersion > Using API Version 1 > () Calling .SetConfigRaw > () Calling .GetMachineName > (flag-lookup) Calling .GetMachineName > (flag-lookup) Calling .DriverName > (flag-lookup) Calling .GetCreateFlags > Found binary path at /usr/local/bin/docker-machine > Launching plugin server for driver vmwarevsphere > Plugin server listening at address 127.0.0.1:39809 > () Calling .GetVersion > Using API Version 1 > () Calling .SetConfigRaw > () Calling .GetMachineName > (rancheros-test) Calling .GetMachineName > (rancheros-test) Calling .DriverName > (rancheros-test) Calling .GetCreateFlags > (rancheros-test) Calling .SetConfigFromFlags > Reading certificate data from /root/.docker/machine/certs/ca.pem > Decoding PEM data... > Parsing certificate... > Reading certificate data from /root/.docker/machine/certs/cert.pem > Decoding PEM data... > Parsing certificate... > Running pre-create checks... > (rancheros-test) Calling .PreCreateCheck > (rancheros-test) DBG | Connecting to vSphere for pre-create checks... > (rancheros-test) Calling .GetConfigRaw > Creating machine... > (rancheros-test) Calling .Create > (rancheros-test) Downloading /root/.docker/machine/cache/boot2docker.iso from http://releases.rancher.com/os/latest/rancheros.iso... > (rancheros-test) 0%....10%....20%....30%....40%....50%....60%....70%....80%....90%....100% > (rancheros-test) Generating SSH Keypair... > (rancheros-test) Creating VM... > (rancheros-test) Uploading Boot2docker ISO ... > (rancheros-test) adding network: vlan45-docker > (rancheros-test) Reconfiguring VM > (rancheros-test) setting guestinfo.cloud-init.data.url to https://??? > (rancheros-test) > (rancheros-test) Waiting for VMware Tools to come online... > (rancheros-test) Provisioning certs and ssh keys... > (rancheros-test) DBG | Creating Tar key bundle... > (rancheros-test) Calling .GetConfigRaw > (rancheros-test) Calling .DriverName > (rancheros-test) Calling .DriverName > Waiting for machine to be running, this may take a few minutes... > (rancheros-test) Calling .GetState > Detecting operating system of created instance... > Waiting for SSH to be available... > Getting to WaitForSSH function... > (rancheros-test) Calling .GetSSHHostname > (rancheros-test) Calling .GetSSHPort > (rancheros-test) Calling .GetSSHKeyPath > (rancheros-test) Calling .GetSSHKeyPath > (rancheros-test) Calling .GetSSHUsername > Using SSH client type: external > Using SSH private key: /root/.docker/machine/machines/rancheros-test/id\_rsa (-rw-------) > &{\[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@172.16.???.??? -o IdentitiesOnly=yes -i /root/.docker/machine/machines/rancheros-test/id\_rsa -p 22\] /usr/bin/ssh \} > About to run SSH command: > exit 0 > SSH cmd err, output: exit status 255: > Error getting ssh command 'exit 0' : ssh command error: > command : exit 0 > err : exit status 255 > output : > ... > this repeats for a few minutes > ... > Error creating machine: Error detecting OS: Too many retries waiting for SSH to be available. Last error: Maximum number of retries (60) exceeded > notifying bugsnag: \[Error creating machine: Error detecting OS: Too many retries waiting for SSH to be available. Last error: Maximum number of retries (60) exceeded\] > \`\`\` ---
**Author:** ![Gneisler](https://avatars.discourse-cdn.com/v4/letter/g/59ef9b/32.png) [@Gneisler](https://forums.suse.com/u/Gneisler)\ **Post date:** [May 23, 2018, 4:17pm UTC](https://forums.suse.com/t/vsphere-provisioning-stuck-at-ssh-step/10465/10 "2018-05-23T16:17:57Z")
I attempted Rancher OS v1.4.0\_rc2 and got past the SSH step. I did hit another error, however I will wait until 1.4.0 is out of RC to attempt again. Consider this closed, thank you for everyones help. ---
**Author:** ![Dongfang\_Liao](https://avatars.discourse-cdn.com/v4/letter/d/e9c0ed/32.png) [@Dongfang\_Liao](https://forums.suse.com/u/Dongfang_Liao)\ **Post date:** [May 23, 2018, 4:51pm UTC](https://forums.suse.com/t/vsphere-provisioning-stuck-at-ssh-step/10465/11 "2018-05-23T16:51:16Z")
> [@Gneisler](#): > > er OS v1.4.0\_rc2 and got past the SSH step. I did hit another error, however I will wait until 1.4.0 is out of RC to attempt again. i have same problem too, it is vsphere 6.5 with rancher 2.0. it shows in the rancher2 console that cannot bring up etcd cluster, waiting for ssh. i can login to the node OS from console, it is boot2docker os, but etcd container has no ‘4001’ listening, if i ran etcdctl member list, it shows me not able to connect to localhost:4001. any help will be very appreciated! ---
**Author:** ![Gneisler](https://avatars.discourse-cdn.com/v4/letter/g/59ef9b/32.png) [@Gneisler](https://forums.suse.com/u/Gneisler)\ **Post date:** [August 16, 2018, 9:11pm UTC](https://forums.suse.com/t/vsphere-provisioning-stuck-at-ssh-step/10465/12 "2018-08-16T21:11:34Z")
> [@Gneisler](#): > > I attempted Rancher OS v1.4.0\_rc2 and got past the SSH step. I did hit another error, however I will wait until 1.4.0 is out of RC to attempt again. > > Consider this closed, thank you for everyones help. Just would like to say thank you to the rancher team for releasing 1.4.0 Rancher OS. I am now able to spin up nodes through vmware just with a click of the button. Thanks you! ---
**Author:** ![TimothyMeade](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/timothymeade/32/4742_2.png) [@TimothyMeade](https://forums.suse.com/u/TimothyMeade)\ **Post date:** [May 12, 2019, 7:07pm UTC](https://forums.suse.com/t/vsphere-provisioning-stuck-at-ssh-step/10465/13 "2019-05-12T19:07:45Z")
I have hit this exact same issue. Is there a work around? rancher v2.2.2 . Should we revert? What does work? We need to spin up 2 clusters soonest. Thanks