# Creating new clusters is very flaky - resolved

**URL:** <https://forums.suse.com/t/creating-new-clusters-is-very-flaky-resolved/13134>\
**Category:** SUSE Rancher Prime\
**Created:** [January 28, 2019, 6:24pm UTC](https://forums.suse.com/t/creating-new-clusters-is-very-flaky-resolved/13134 "2019-01-28T18:24:33Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ryan](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/ryan/32/2768_2.png) [@ryan](https://forums.suse.com/u/ryan)\
**Post date:** [January 28, 2019, 6:24pm UTC](https://forums.suse.com/t/creating-new-clusters-is-very-flaky-resolved/13134/1 "2019-01-28T18:24:33Z")

</div>

Greetings,

I am unable to create a new cluster. Every time I try, I get different error messages.

Rancher version: 2.1.4  
All three nodes are ubuntu 16.10 VMs, up to date with all latest packages.

I decided to “clean slate” everything:

- ‘docker stop’ everything on each node.
- ‘docker system prune -a’ on each node.
- On rancher, I deleted each node and the cluster.

I rebuilt the cluster:

- On rancher, created new cluster
- “custom” node type
- “Canal” network provider
- “custom” cloud provider
- I chose each node to serve all three roles: etcd, control plane, and worker
- I cut and paste the docker run command into each VM
- Rancher reports three nodes, but I am getting this error:

```auto
This cluster is currently Provisioning; areas that interact directly with it will not be available until the API is ready.

[network] Host [10.10.55.223] is not able to connect to the following ports: [10.10.55.224:2379, 10.10.55.224:2380]. Please check network policies and firewall rules

```

I have deleted everything and tried this a few times, all end with error. What am I doing wrong?

---

<div class="post-metadata">

**Author:** ![ryan](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/ryan/32/2768_2.png) [@ryan](https://forums.suse.com/u/ryan)\
**Post date:** [January 29, 2019, 6:59pm UTC](https://forums.suse.com/t/creating-new-clusters-is-very-flaky-resolved/13134/2 "2019-01-29T18:59:06Z")

</div>

Frustrated, I decided to completely start over with all new installs:

- I built a new rancher server (2.1.15 - saw some cluster bugs were fixed)
- I built 3 new Ubuntu 16.04 server VMs. Installed docker (v 17.03.2)
- I built a new cluster in Rancher using “Custom” nodes. Each node serves all three roles.
- I cut and paste the rancher provisioning script into each terminal and left.

6 hours later, the new cluster was in error state:  
`[controlPlane] Failed to bring up Control Plane: Failed to verify healthcheck: Failed to check https://localhost:6443/healthz for service [kube-apiserver] on host [10.10.55.222]: Get https://localhost:6443/healthz: read tcp [::1]:43414-&gt;[::1]:6443: read: connection reset by peer, log: I0129 15:07:03.469022 1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.`

I rebooted each node and waited. I saw the messages in rancher cycle through the standard cluster provisioning messages.

I came back to the exact same error.

---

<div class="post-metadata">

**Author:** ![ryan](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/ryan/32/2768_2.png) [@ryan](https://forums.suse.com/u/ryan)\
**Post date:** [January 30, 2019, 6:43pm UTC](https://forums.suse.com/t/creating-new-clusters-is-very-flaky-resolved/13134/3 "2019-01-30T18:43:56Z")

</div>

I resolved this issue.

Re-using RKE provisioned K8S nodes is not as simple as just stopping all the containers. I found this script from superseb that cleaned up the nodes and made them work again:

[https://gist.githubusercontent.com/superseb/2cf186726807a012af59a027cb41270d/raw/7cfbce916809e7b2474a73a3da367b1a7f4ac9cf/cleanup.sh](https://gist.githubusercontent.com/superseb/2cf186726807a012af59a027cb41270d/raw/7cfbce916809e7b2474a73a3da367b1a7f4ac9cf/cleanup.sh)

This seems like something that needs to go into the docs!

---

<div class="post-metadata">

**Author:** ![DineshC001](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/dineshc001/32/4629_2.png) [@DineshC001](https://forums.suse.com/u/DineshC001)\
**Post date:** [February 3, 2019, 6:02pm UTC](https://forums.suse.com/t/creating-new-clusters-is-very-flaky-resolved/13134/4 "2019-02-03T18:02:56Z")

</div>

You should follow these steps to remove rancher cleanly - [https://rancher.com/docs/rancher/v2.x/en/admin-settings/removing-rancher/user-cluster-nodes/](https://rancher.com/docs/rancher/v2.x/en/admin-settings/removing-rancher/user-cluster-nodes/)
