Three "All" Node in HA?

etlweather · June 25, 2018, 5:18pm

Short answer - the cluster shouldn’t go down…

Same or similar issues:

github.com/rancher/rancher

"Cluster not available" shown in UI if worker node reboots

opened 07:23AM - 03 Jun 18 UTC

closed 10:06PM - 06 Sep 18 UTC

jomeier

kind/bug priority/1 status/more-info area/rke

Hi, what is the reason for rancher 2 UI to decide, when a cluster gets unavai…lable? I created a test cluster with rke consisting of three nodes: one with etcd and control, two nodes which are workers. Rancher 2 latest runs on a different server (not on this cluster). Than I tried to simulate destruction of nodes by rebooting them. If I reboot the 1st worker, rancher UI is fine and tells me after a few seconds, that this node is not available. If I reboot the 2nd worker, rancher UI tells me - in a big wide square - that my cluster is not available. I not even can "launch a kubectl" console in the UI because it's blocked. I assumed, that rebooting workers shouldn't make the whole cluster unavailable. It's intersting, that during both reboots I can access the cluster's API on a linux shell with kubectl. Only rancher UI acts strange. Greetz, Josef

github.com/rancher/rancher

When control plane becomes unavailable (host powered down) when there is another control plane in the cluster , not able to deploy nodes in the worker plane and other control plane.

opened 09:56PM - 24 May 18 UTC

closed 09:15PM - 06 Aug 18 UTC

sangeethah

kind/bug

**Rancher versions:v2.0.2 **Steps to Reproduce:** Create a cluster with foll…owing node configurations: 1 control (n1) 1 etcd (n2) 1 worker (n3) Add 1 more control node (n4) Power down control node - n1. Wait for the node to be marked "unavailable". Try to create a daemon set. 3 pods get created out of which only 1 pod was able to start sucessfully which is on the new control node. There is an attempt made to start a pod in the worker node that fails with following error: ``` Normal SuccessfulMountVolume 5m kubelet, ip-172-31-3-155 MountVolume.SetUp succeeded for volume "default-token-mqlnn" Normal SandboxChanged 4m (x12 over 5m) kubelet, ip-172-31-3-155 Pod sandbox changed, it will be killed and re-created. Warning FailedCreatePodSandBox 14s (x100 over 5m) kubelet, ip-172-31-3-155 Failed create pod sandbox: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "hellotest-qjr5t_default" network: error getting ClusterInformation: Get https://10.43.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default: dial tcp 10.43.0.1:443: getsockopt: no route to host ``` Other pod is attempted to start in the control node that is unavailable. Deploying a pod with some scale , results in all the pods getting deployed on the worker node and it get stuck in "ContainerCreating" state. ``` NAME READY STATUS RESTARTS AGE hello1-74f74757b9-5swtv 0/1 ContainerCreating 0 39m hello1-74f74757b9-8rdp2 0/1 ContainerCreating 0 39m hello1-74f74757b9-gz72c 0/1 ContainerCreating 0 39m hello1-74f74757b9-xql8l 0/1 ContainerCreating 0 39m ``` **Results:**

Topic		Replies	Views
HA - UI failure with single etcd node reboot SUSE Rancher Prime	7	1023	November 19, 2019
Need for Rancher HA / Impact if down? SUSE Rancher Prime	13	5335	June 14, 2018
HA-Cluster not reliable [solved] SUSE Rancher Prime	1	1910	August 17, 2018
API-Server unreachable with 1/3 node down, HA? SUSE Rancher Prime	0	842	January 29, 2019
Rancher API Server SUSE Rancher Prime	0	1152	July 9, 2020

Three "All" Node in HA?

Related topics