# Liveness probe failed connection refused

**URL:** <https://forums.suse.com/t/liveness-probe-failed-connection-refused/20837>\
**Category:** SUSE Rancher Prime\
**Created:** [August 1, 2021, 4:33pm UTC](https://forums.suse.com/t/liveness-probe-failed-connection-refused/20837 "2021-08-01T16:33:01Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![kamal\_boumahdi](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/kamal_boumahdi/32/7837_2.png) [@kamal\_boumahdi](https://forums.suse.com/u/kamal_boumahdi)\
**Post date:** [August 1, 2021, 4:33pm UTC](https://forums.suse.com/t/liveness-probe-failed-connection-refused/20837/1 "2021-08-01T16:33:02Z")

</div>

Hello everyone, I hope you’re having a great time.  
here’s the issue I’m having :

```auto
~$ kubectl -n cattle-system get po
NAME READY STATUS RESTARTS AGE
helm-operation-8plqf 0/2 Completed 0 46m
helm-operation-95r5c 0/2 Completed 0 44m
helm-operation-bjkrm 0/2 Completed 0 43m
helm-operation-cszqz 0/2 Completed 0 45m
helm-operation-mvh84 0/2 Completed 0 44m
rancher-df6c8dcd6-2fhmf 0/1 Evicted 0 33m
rancher-df6c8dcd6-2hdsn 0/1 Evicted 0 3m8s
rancher-df6c8dcd6-2p8gt 0/1 Evicted 0 27m
...
rancher-df6c8dcd6-xnljn 1/1 Running 0 16m
rancher-df6c8dcd6-z8s5j 0/1 Evicted 0 16m
rancher-df6c8dcd6-zf82l 0/1 Evicted 0 3m10s
rancher-df6c8dcd6-zqg4k 0/1 Evicted 0 53m
rancher-df6c8dcd6-zxc26 0/1 Evicted 0 27m
rancher-df6c8dcd6-zxwl5 0/1 Evicted 0 3m2s
rancher-webhook-69b7997768-5r59d 1/1 Running 0 45m

```

so I it seems that these pods keep getting created then evicted, I try to describe one of them and here is what I get:

```auto
~$ kubectl describe po rancher-df6c8dcd6-2fhmf -n cattle-system
Name: rancher-df6c8dcd6-2fhmf
Namespace: cattle-system
Priority: 0
Node: joxstar/
Start Time: Sun, 01 Aug 2021 15:36:43 +0000
Labels: app=rancher
                pod-template-hash=df6c8dcd6
                release=rancher
Annotations: cni.projectcalico.org/podIP: 10.42.0.148/32
                cni.projectcalico.org/podIPs: 10.42.0.148/32
Status: Failed
Reason: Evicted
Message: The node was low on resource: ephemeral-storage. Container rancher was using 416152Ki, which exceeds its request of 0.
IP:
IPs: <none>
Controlled By: ReplicaSet/rancher-df6c8dcd6
Containers:
  rancher:
    Image: rancher/rancher:v2.5.9
    Port: 80/TCP
    Host Port: 0/TCP
    Args:
      --http-listen-port=80
      --https-listen-port=443
      --add-local=true
    Liveness: http-get http://:80/healthz delay=60s timeout=1s period=30s #success=1 #failure=3
    Readiness: http-get http://:80/healthz delay=5s timeout=1s period=30s #success=1 #failure=3
    Environment:
      CATTLE_NAMESPACE: cattle-system
      CATTLE_PEER_SERVICE: rancher
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from rancher-token-8dc7l (ro)
Volumes:
  rancher-token-8dc7l:
    Type: Secret (a volume populated by a Secret)
    SecretName: rancher-token-8dc7l
    Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Warning FailedScheduling 42m (x6 over 47m) default-scheduler 0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
  Normal Scheduled 41m default-scheduler Successfully assigned cattle-system/rancher-df6c8dcd6-2fhmf to joxstar
  Normal Pulled 41m kubelet Container image "rancher/rancher:v2.5.9" already present on machine
  Normal Created 41m kubelet Created container rancher
  Normal Started 41m kubelet Started container rancher
  Warning Unhealthy 41m kubelet Readiness probe failed: Get http://10.42.0.148:80/healthz: dial tcp 10.42.0.148:80: connect: connection refused
  Warning Evicted 29m kubelet The node was low on resource: ephemeral-storage. Container rancher was using 416152Ki, which exceeds its request of 0.
  Normal Killing 29m kubelet Stopping container rancher

```

Now I understand that these pods are consuming resources, **but why are they not deleted?**  
also what’s causing these failures?

```auto
 Liveness: http-get http://:80/healthz delay=60s timeout=1s period=30s #success=1 #failure=3
 Readiness: http-get http://:80/healthz delay=5s timeout=1s period=30s #success=1 #failure=3

```

I know I must have messed up somewhere (being such a noob), but how do I go about fixing it?  
any help is very much appreciated.  
possibly helpful information:

- it’s an RKE sigle node cluster.
- using ubuntu 20.04 with VMware workstation.
- I installed nginx as a layer 4 LB.

---

<div class="post-metadata">

**Author:** ![superseb](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/superseb/32/2424_2.png) [@superseb](https://forums.suse.com/u/superseb)\
**Post date:** [August 2, 2021, 11:51am UTC](https://forums.suse.com/t/liveness-probe-failed-connection-refused/20837/2 "2021-08-02T11:51:11Z")

</div>

The default threshold for Kubernetes to clean up terminated pods is 12500, we changed it to 1000 so unless you have a 1000 it won’t automatically cleanup. You can do it manually though.

In general, you are running out of resources, did you check the storage on the machines and the logs of the kubelet? The failures probably come from the resource problem as well, but you can check the logs of the Rancher pods (it is described on [Rancher Docs: Rancher HA](https://rancher.com/docs/rancher/v2.5/en/troubleshooting/rancherha/))

---

<div class="post-metadata">

**Author:** ![kamal\_boumahdi](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/kamal_boumahdi/32/7837_2.png) [@kamal\_boumahdi](https://forums.suse.com/u/kamal_boumahdi)\
**Post date:** [August 3, 2021, 4:58pm UTC](https://forums.suse.com/t/liveness-probe-failed-connection-refused/20837/3 "2021-08-03T16:58:14Z")

</div>

Thanks a lot, I really appreciate your input.  
So if I understand you correctly, **these failures might be related to storage**.

```auto
 Liveness: http-get http://:80/healthz delay=60s timeout=1s period=30s #success=1 #failure=3
 Readiness: http-get http://:80/healthz delay=5s timeout=1s period=30s #success=1 #failure=3

```

Storage on the machines:  
this is from the node,

```auto
~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 2.8G 0 2.8G 0% /dev
tmpfs 570M 3.2M 567M 1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv 19G 15G 3.0G 84% /
tmpfs 2.8G 0 2.8G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 2.8G 0 2.8G 0% /sys/fs/cgroup
/dev/loop1 56M 56M 0 100% /snap/core18/1944
/dev/loop2 68M 68M 0 100% /snap/lxd/20326
/dev/loop0 56M 56M 0 100% /snap/core18/2074
/dev/loop5 33M 33M 0 100% /snap/snapd/12398
/dev/loop3 71M 71M 0 100% /snap/lxd/21029
/dev/sda2 976M 200M 710M 22% /boot
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/d392c473ddafd39f835ef5af5dac0ee49c3f8f264fcf55035f5cf4215487bd9c/merged
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/861d417b513fd84f101af1b0471dd7c0d3960739438e3203b3324c2c7d98ea78/merged
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/2f857e84ddde915a39d9097c514957d02a5f9e46ac37cf45a5cf6db7eb979501/merged
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/0bf82cfd0cce5200da8a180620e52043ebbe335c766a676b6a60e40511368d64/merged
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/3cb14e053278c7f6b716f45ab2a77a0262ad458bfaac076339ebfe765a66d075/merged
shm 64M 0 64M 0% /var/lib/docker/containers/b7345745d75470f836823eba82d1f7928584adda15ee8c2967696364b802efc2/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/3c0ea6d5bcddb59168195cd411a3bb21e399d5a2eb9332f4918908bd44fd54c8/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/01b97faf4b9296a108ce3b59eefba859d5d7f6e094d5b8dbf719ab5864bff8ad/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/a0656042661e238f94bde8a5c34fc33fb01f0b76fb2672e6a9f5ea2f703c5a41/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/ae22fabae178bcd1bccac89f5fc2659b17a2ec16549d6f37da2913bdb1a6ffdf/mounts/shm
/dev/loop6 33M 33M 0 100% /snap/snapd/12704
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/bec4ed7ecc3edd7de4e1c48c6ea563cad598003825eebc0136051194295a3e04/merged
shm 64M 0 64M 0% /var/lib/docker/containers/3cd89bb3a7866f9dad3bfd55128d0a486a6ca122c8d121165fdbb9e7f809ca05/mounts/shm
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/07dbbb8d8d6805e0a131536ce725ea1dfa8f2191814b314f81b7e6f125ec4191/merged
shm 64M 0 64M 0% /var/lib/docker/containers/462461b3357720d2e78ce39f4ad920fe2ee590e7bbb4f86469e002ed776a06c0/mounts/shm
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/a6eb2912e31ffdf3d9fa7dbd1a5da1ca89b532aa179e64a39f10c4a3ad00f3e9/merged
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/47c91123ff2d97608d2203847cd492c8e96902f7065caadbab786d412aeb70b7/merged
shm 64M 0 64M 0% /var/lib/docker/containers/dc9dbbe39c796c14bb383a1a07ae881a2a0af0221a4d60e2e8c51a7c1795706b/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/f58bd8ef88866c56f2e9577e05e06f3a265076ec3bdbf79c38086f6c13a371ca/mounts/shm
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/5f282847f0321d51132c1c24ddb6618d372cec4b062998896e3863e3b912da4e/merged
shm 64M 0 64M 0% /var/lib/docker/containers/38020b77c7da34ef82455b0f2f434985e5dd6b3fa0852501a7b3b8f7db815a6e/mounts/shm
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/799e81a76e4e70cba1415f5810fea86cd12cb40017691124981a4c9b61acb84d/merged
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/4748a0449675b165e9e1d56cccb80598bdc759e9a348fb4f5048886bfe3f9d85/merged
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/5e2e128834df14909a5503cb3e08842448fde687e1b339926f5c6ca186d15921/merged
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/9ebc701293ef0121f46b092936341cbe02b0a73f30ec70d7852212dff025fffd/merged
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/9a36ff4639bf20f48d9c8d432c7487332206d7e2c5e1ad12e2b9c1e3c640dc60/merged
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/ce91f0972465fdb63c226d1d886daa5b31492ed9e3e50636144943a5b8d7ef7b/merged
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/9d4752e9e5d123ad8fd257dccd167a91867b36ec009a214dd490fc843dc54bc0/merged
shm 64M 0 64M 0% /var/lib/docker/containers/76a3489d96bf610c02965402d4833377432111d4f616715ad820b6a29c136585/mounts/shm
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/7e6f4f1aa0241d5a464eeb5b0fb286f3a29ff05135e9c4d794a233dd7fb55c4e/merged
shm 64M 0 64M 0% /var/lib/docker/containers/0472c91a8ce4dc05b079070b2be730cee3ee7d4969871eb8f1e98a859ca21fdd/mounts/shm
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/435c66b1affa2b2eeb8af1c0f9d3f5006d1081831946f965e39245e28a9f9d26/merged
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/3e6666543a7c0e8d178154f653e29a23accf02b3326c35b2cad5e2b515ebd660/merged
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/87e44c2973d0d32c5a941c0fda2754e23b9550ea0b202071f1ec62d51993a88e/merged
shm 64M 0 64M 0% /var/lib/docker/containers/51033b72d0230e98b624865d8ce2fd81c6b48b288de99aa2c76f5d66ae489491/mounts/shm
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/000d7f046735d3c061ca73472044590c692542f0a2eb16035493306d86e2fe59/merged
shm 64M 0 64M 0% /var/lib/docker/containers/38fcaac2c17558abce2c6ae9b6c2c73d4025aed01fda89f1d345950254e3afed/mounts/shm
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/14822a707b31d2b85390838c89ef32cd3b1850be6337a04319726c518bb89786/merged
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/028581af7ee7a14f5dec181aeaaf374eafd295701f32bc733d23a8ace7f23961/merged
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/6d9522db823484a9819ddd2dd60a6c23b78e43d24c8ec6226becb8fac52ff088/merged
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/d795b1fe639fb44f78f9cdf50f1122f256a4a9679fa07d5f0a5284ef0c937c32/merged
shm 64M 0 64M 0% /var/lib/docker/containers/a87741123c91134bb9a36eb795df4c24c53b575f9e27b536bc75c279d73c6e3e/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/5f533ee3b152670f220fde48a647e56d71ac8fffad1ebc84ec099bae6d1c04f3/mounts/shm
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/2b113c3ca500ce89d29527d1ed704403af204c49ce0d58d686987aed728a9cb9/merged
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/a3150886ca02e1462fee695d971373606cf0acb8b3bed38e3c499a8572264c06/merged
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/c5d2caf6bb454b32710bae0d4f2937149a99927583f32d20420ec7be69b0b3e4/merged
shm 64M 0 64M 0% /var/lib/docker/containers/0f6efd41812028fbfa413ad21a994f526f414de4fb414aa2524af41353c42d86/mounts/shm
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/76ac325e2b7b8d845b3fd194163e8a6c498520963a6758a49514cd629b6928f0/merged
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/b1fb21283c8ca6268666f5b9e0fbf47a53bebe08db90a639910f7069d06f3408/merged
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/53ee260a8864066fe4e887a6a83a4b501308c5ff57c216895cfc7a60287a1789/merged
shm 64M 0 64M 0% /var/lib/docker/containers/95ede95613e9da50f134a094c740e78dca4958a720c88d797ee59f8004d589c2/mounts/shm
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/60259319bb18d9db884217194d4c52d20ee38a988a777b061cb553037befc862/merged
shm 64M 0 64M 0% /var/lib/docker/containers/208c748fdf715b726b9d55383e3bf53be4b6c042c0a534cfec31f46ac9df6934/mounts/shm
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/ce59b119fc5bb3f5907bb900d368fa8b868adb70b901ac26aaad62b35d9f184c/merged
shm 64M 0 64M 0% /var/lib/docker/containers/03d8f07c57c95449a13145c90798c78da56df28b0cf6cdadce5ad2a26338a0b3/mounts/shm
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/847ffa689ca3f860d61c774988631458fd003668f701098726377895d8ee154e/merged
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/8072e39ef7fe6981ad7f1e57a092fe7e75a08c0af73d255f328c1b676e171efd/merged
shm 64M 0 64M 0% /var/lib/docker/containers/efa5c71cafddf20d7a16c66f5adab45765398b3040bccf8c9e9aba5343d3f1b5/mounts/shm
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/16e1e2dd57c9a2fc629f9e82589f016304f37e5bc5b5f0fe780ae556a36c3f40/merged
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/d2ebe14847c2a6ab6cced64fe885e164252a9ba61bfd7e1fc854b56fc3956848/merged
shm 64M 0 64M 0% /var/lib/docker/containers/9321c1d5fc825f882bbace743b09c000d2021ea599d4e95b225e0e74ac21bf74/mounts/shm
overlay 19G 15G 3.0G 84% /var/lib/docker/overlay2/128d261f32f41934ceb0689ea854b20aa9f6ee9cbb0b54dd22c665e958b87697/merged
tmpfs 570M 0 570M 0% /run/user/1000

```

and if I’m reading this correctly, at 84% it does seem almost saturated.

But according to the Rancher logs, this **Readiness probe** failure took place well **before the storage issue** ( a little over 10 mins before).

Which makes me think that **something else is causing these pods to fail** (or be considered unhealthy), and since for some reason (which I don’t know) **they’re just hanging there instead of being completely irradiated.**

While **more pods are created** to satisfy the desired state (to ultimately meet the same fate). We eventually **end up with the storage problem**.

Not sure though, but if that’s right then I need to find out what’s causing this **Readiness probe** failure in the first place.

what’s your take on that? thanks again for your help, in the mean time I’ll spin up another VM with exaggerated storage capacity and see if I get the same issue just to be sure.

---

<div class="post-metadata">

**Author:** ![superseb](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/superseb/32/2424_2.png) [@superseb](https://forums.suse.com/u/superseb)\
**Post date:** [August 3, 2021, 8:58pm UTC](https://forums.suse.com/t/liveness-probe-failed-connection-refused/20837/4 "2021-08-03T20:58:30Z")

</div>

The logs of the Rancher pods/containers will most likely give the best indication why the checks are failing, see previous reply.

---

<div class="post-metadata">

**Author:** ![kamal\_boumahdi](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/kamal_boumahdi/32/7837_2.png) [@kamal\_boumahdi](https://forums.suse.com/u/kamal_boumahdi)\
**Post date:** [August 6, 2021, 6:16pm UTC](https://forums.suse.com/t/liveness-probe-failed-connection-refused/20837/5 "2021-08-06T18:16:49Z")

</div>

I got rid of the storage problem by switching to other VMs, yet it seems like that wasn’t the problem.  
kubectl -n cattle-system get pods returns that everything is okay.

```auto
~$ kubectl -n cattle-system get pods -l app=rancher -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
rancher-cbbdf9fd5-n4cgr 1/1 Running 0 23m 10.42.0.10 fuji1 <none> <none>
rancher-cbbdf9fd5-nj99v 1/1 Running 0 23m 10.42.0.11 fuji1 <none> <none>
rancher-cbbdf9fd5-pmz5b 1/1 Running 0 23m 10.42.0.9 fuji1 <none> <none>

```

but describe shows the same issue as before, for all three pods

```auto
~$ kubectl -n cattle-system describe pods -l app=rancher
Name: rancher-cbbdf9fd5-n4cgr
Namespace: cattle-system
Priority: 0
Node: fuji1/192.168.157.141
Start Time: Fri, 06 Aug 2021 17:32:54 +0000
Labels: app=rancher
              pod-template-hash=cbbdf9fd5
              release=rancher
Annotations: cni.projectcalico.org/podIP: 10.42.0.10/32
              cni.projectcalico.org/podIPs: 10.42.0.10/32
Status: Running
IP: 10.42.0.10
IPs:
  IP: 10.42.0.10
Controlled By: ReplicaSet/rancher-cbbdf9fd5
Containers:
  rancher:
    Container ID: docker://318db6fdff3225e4066ffc56999370103e776a9a175c9358f637334d46b89b0a
    Image: rancher/rancher:v2.5.9
    Image ID: docker-pullable://rancher/rancher@sha256:4cfd6f4ecf64a8bb117738cc30c57c63617eaf4b3bdbb49dff024f36e3e589bb
    Port: 80/TCP
    Host Port: 0/TCP
    Args:
      --no-cacerts
      --http-listen-port=80
      --https-listen-port=443
      --add-local=true
    State: Running
      Started: Fri, 06 Aug 2021 17:38:47 +0000
    Ready: True
    Restart Count: 0
    Liveness: http-get http://:80/healthz delay=60s timeout=1s period=30s #success=1 #failure=3
    Readiness: http-get http://:80/healthz delay=5s timeout=1s period=30s #success=1 #failure=3
    Environment:
      CATTLE_NAMESPACE: cattle-system
      CATTLE_PEER_SERVICE: rancher
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from rancher-token-bqsls (ro)
Conditions:
  Type Status
  Initialized True
  Ready True
  ContainersReady True
  PodScheduled True
Volumes:
  rancher-token-bqsls:
    Type: Secret (a volume populated by a Secret)
    SecretName: rancher-token-bqsls
    Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal Scheduled 23m default-scheduler Successfully assigned cattle-system/rancher-cbbdf9fd5-n4cgr to fuji1
  Normal Pulling 23m kubelet Pulling image "rancher/rancher:v2.5.9"
  Normal Pulled 18m kubelet Successfully pulled image "rancher/rancher:v2.5.9"
  Normal Created 18m kubelet Created container rancher
  Normal Started 17m kubelet Started container rancher
  Warning Unhealthy 16m (x3 over 17m) kubelet Readiness probe failed: Get http://10.42.0.10:80/healthz: dial tcp 10.42.0.10:80: connect: connection refused
  Warning Unhealthy 16m kubelet Liveness probe failed: Get http://10.42.0.10:80/healthz: dial tcp 10.42.0.10:80: connect: connection refused

Name: rancher-cbbdf9fd5-nj99v
Namespace: cattle-system
Priority: 0
Node: fuji1/192.168.157.141
Start Time: Fri, 06 Aug 2021 17:32:54 +0000
Labels: app=rancher
              pod-template-hash=cbbdf9fd5
              release=rancher
Annotations: cni.projectcalico.org/podIP: 10.42.0.11/32
              cni.projectcalico.org/podIPs: 10.42.0.11/32
Status: Running
IP: 10.42.0.11
IPs:
  IP: 10.42.0.11
Controlled By: ReplicaSet/rancher-cbbdf9fd5
Containers:
  rancher:
    Container ID: docker://2ba8702efdaa5503d27804bc1a751ea661618049081dbfef530d04d0d6651546
    Image: rancher/rancher:v2.5.9
    Image ID: docker-pullable://rancher/rancher@sha256:4cfd6f4ecf64a8bb117738cc30c57c63617eaf4b3bdbb49dff024f36e3e589bb
    Port: 80/TCP
    Host Port: 0/TCP
    Args:
      --no-cacerts
      --http-listen-port=80
      --https-listen-port=443
      --add-local=true
    State: Running
      Started: Fri, 06 Aug 2021 17:38:47 +0000
    Ready: True
    Restart Count: 0
    Liveness: http-get http://:80/healthz delay=60s timeout=1s period=30s #success=1 #failure=3
    Readiness: http-get http://:80/healthz delay=5s timeout=1s period=30s #success=1 #failure=3
    Environment:
      CATTLE_NAMESPACE: cattle-system
      CATTLE_PEER_SERVICE: rancher
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from rancher-token-bqsls (ro)
Conditions:
  Type Status
  Initialized True
  Ready True
  ContainersReady True
  PodScheduled True
Volumes:
  rancher-token-bqsls:
    Type: Secret (a volume populated by a Secret)
    SecretName: rancher-token-bqsls
    Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal Scheduled 23m default-scheduler Successfully assigned cattle-system/rancher-cbbdf9fd5-nj99v to fuji1
  Normal Pulling 23m kubelet Pulling image "rancher/rancher:v2.5.9"
  Normal Pulled 18m kubelet Successfully pulled image "rancher/rancher:v2.5.9"
  Normal Created 18m kubelet Created container rancher
  Normal Started 17m kubelet Started container rancher
  Warning Unhealthy 16m (x3 over 17m) kubelet Readiness probe failed: Get http://10.42.0.11:80/healthz: dial tcp 10.42.0.11:80: connect: connection refused
  Warning Unhealthy 16m kubelet Liveness probe failed: Get http://10.42.0.11:80/healthz: dial tcp 10.42.0.11:80: connect: connection refused

Name: rancher-cbbdf9fd5-pmz5b
Namespace: cattle-system
Priority: 0
Node: fuji1/192.168.157.141
Start Time: Fri, 06 Aug 2021 17:32:54 +0000
Labels: app=rancher
              pod-template-hash=cbbdf9fd5
              release=rancher
Annotations: cni.projectcalico.org/podIP: 10.42.0.9/32
              cni.projectcalico.org/podIPs: 10.42.0.9/32
Status: Running
IP: 10.42.0.9
IPs:
  IP: 10.42.0.9
Controlled By: ReplicaSet/rancher-cbbdf9fd5
Containers:
  rancher:
    Container ID: docker://93cd1749b5cc11a64bb9c6d88fb4e00f0e8c965c29ff8e7cf0ff61db70b12497
    Image: rancher/rancher:v2.5.9
    Image ID: docker-pullable://rancher/rancher@sha256:4cfd6f4ecf64a8bb117738cc30c57c63617eaf4b3bdbb49dff024f36e3e589bb
    Port: 80/TCP
    Host Port: 0/TCP
    Args:
      --no-cacerts
      --http-listen-port=80
      --https-listen-port=443
      --add-local=true
    State: Running
      Started: Fri, 06 Aug 2021 17:38:47 +0000
    Ready: True
    Restart Count: 0
    Liveness: http-get http://:80/healthz delay=60s timeout=1s period=30s #success=1 #failure=3
    Readiness: http-get http://:80/healthz delay=5s timeout=1s period=30s #success=1 #failure=3
    Environment:
      CATTLE_NAMESPACE: cattle-system
      CATTLE_PEER_SERVICE: rancher
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from rancher-token-bqsls (ro)
Conditions:
  Type Status
  Initialized True
  Ready True
  ContainersReady True
  PodScheduled True
Volumes:
  rancher-token-bqsls:
    Type: Secret (a volume populated by a Secret)
    SecretName: rancher-token-bqsls
    Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal Scheduled 23m default-scheduler Successfully assigned cattle-system/rancher-cbbdf9fd5-pmz5b to fuji1
  Normal Pulling 23m kubelet Pulling image "rancher/rancher:v2.5.9"
  Normal Pulled 18m kubelet Successfully pulled image "rancher/rancher:v2.5.9"
  Normal Created 18m kubelet Created container rancher
  Normal Started 17m kubelet Started container rancher
  Warning Unhealthy 16m kubelet Liveness probe failed: Get http://10.42.0.9:80/healthz: dial tcp 10.42.0.9:80: connect: connection refused
  Warning Unhealthy 16m (x4 over 17m) kubelet Readiness probe failed: Get http://10.42.0.9:80/healthz: dial tcp 10.42.0.9:80: connect: connection refused

```

and here are the logs. “fleet-local” has been modified ? I didn’t do that, does it have to do with RKEs version and Ranchers perhaps?

```auto
~$ kubectl -n cattle-system logs -l app=rancher
2021/08/06 17:44:42 [INFO] APIVersion project.cattle.io/v3 Kind PipelineExecution
2021/08/06 17:44:42 [INFO] APIVersion project.cattle.io/v3 Kind App
2021/08/06 17:44:42 [INFO] APIVersion project.cattle.io/v3 Kind AppRevision
2021/08/06 17:44:42 [INFO] APIVersion project.cattle.io/v3 Kind Pipeline
2021/08/06 17:44:42 [INFO] APIVersion project.cattle.io/v3 Kind PipelineSetting
2021/08/06 17:44:42 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeProviderConfig
2021/08/06 17:44:42 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeRepository
2021/08/06 17:44:42 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeCredential
2021/08/06 17:44:42 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind NodeMetrics
2021/08/06 17:44:42 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind PodMetrics
2021/08/06 17:44:42 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeProviderConfig
2021/08/06 17:44:42 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeRepository
2021/08/06 17:44:42 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeCredential
2021/08/06 17:44:42 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind NodeMetrics
2021/08/06 17:44:42 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind PodMetrics
2021/08/06 17:44:48 [INFO] Watching metadata for gitjob.cattle.io/v1, Kind=GitJob
2021/08/06 17:44:48 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterGroup
2021/08/06 17:44:48 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterRegistration
2021/08/06 17:45:17 [WARNING] error updating ns fleet-local status: Operation cannot be fulfilled on namespaces "fleet-local": the object has been modified; please apply your changes to the latest version and try again
2021/08/06 17:45:17 [WARNING] error updating ns cluster-fleet-local-local-1a3d67d0a899 status: Operation cannot be fulfilled on namespaces "cluster-fleet-local-local-1a3d67d0a899": the object has been modified; please apply your changes to the latest version and try again
2021/08/06 17:44:41 [INFO] APIVersion project.cattle.io/v3 Kind PipelineExecution
2021/08/06 17:44:41 [INFO] APIVersion project.cattle.io/v3 Kind App
2021/08/06 17:44:41 [INFO] APIVersion project.cattle.io/v3 Kind AppRevision
2021/08/06 17:44:41 [INFO] APIVersion project.cattle.io/v3 Kind Pipeline
2021/08/06 17:44:41 [INFO] APIVersion project.cattle.io/v3 Kind PipelineSetting
2021/08/06 17:44:41 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeProviderConfig
2021/08/06 17:44:41 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeRepository
2021/08/06 17:44:41 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeCredential
2021/08/06 17:44:41 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind NodeMetrics
2021/08/06 17:44:41 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind PodMetrics

```

the warning about the unhealthiness of the pods described earlier

```auto
~$ kubectl -n cattle-system get events
LAST SEEN TYPE REASON OBJECT MESSAGE
21m Normal CREATE ingress/cm-acme-http-solver-fllqn Ingress cattle-system/cm-acme-http-solver-fllqn
13m Normal UPDATE ingress/cm-acme-http-solver-fllqn Ingress cattle-system/cm-acme-http-solver-fllqn
<unknown> Normal Scheduled pod/cm-acme-http-solver-nnffq Successfully assigned cattle-system/cm-acme-http-solver-nnffq to fuji1
21m Normal Pulling pod/cm-acme-http-solver-nnffq Pulling image "quay.io/jetstack/cert-manager-acmesolver:v1.0.4"
15m Normal Pulled pod/cm-acme-http-solver-nnffq Successfully pulled image "quay.io/jetstack/cert-manager-acmesolver:v1.0.4"
15m Normal Created pod/cm-acme-http-solver-nnffq Created container acmesolver
15m Normal Started pod/cm-acme-http-solver-nnffq Started container acmesolver
<unknown> Normal Scheduled pod/helm-operation-9hhdw Successfully assigned cattle-system/helm-operation-9hhdw to fuji1
13m Normal Pulling pod/helm-operation-9hhdw Pulling image "rancher/shell:v0.1.6"
12m Normal Pulled pod/helm-operation-9hhdw Successfully pulled image "rancher/shell:v0.1.6"
12m Normal Created pod/helm-operation-9hhdw Created container helm
12m Normal Started pod/helm-operation-9hhdw Started container helm
12m Normal Pulled pod/helm-operation-9hhdw Container image "rancher/shell:v0.1.6" already present on machine
12m Normal Created pod/helm-operation-9hhdw Created container proxy
12m Normal Started pod/helm-operation-9hhdw Started container proxy
<unknown> Normal Scheduled pod/helm-operation-glggv Successfully assigned cattle-system/helm-operation-glggv to fuji1
11m Normal Pulled pod/helm-operation-glggv Container image "rancher/shell:v0.1.6" already present on machine
11m Normal Created pod/helm-operation-glggv Created container helm
11m Normal Started pod/helm-operation-glggv Started container helm
11m Normal Pulled pod/helm-operation-glggv Container image "rancher/shell:v0.1.6" already present on machine
11m Normal Created pod/helm-operation-glggv Created container proxy
11m Normal Started pod/helm-operation-glggv Started container proxy
<unknown> Normal Scheduled pod/helm-operation-j45bz Successfully assigned cattle-system/helm-operation-j45bz to fuji1
12m Normal Pulling pod/helm-operation-j45bz Pulling image "rancher/shell:v0.1.6"
12m Normal Pulled pod/helm-operation-j45bz Successfully pulled image "rancher/shell:v0.1.6"
12m Normal Created pod/helm-operation-j45bz Created container helm
12m Normal Started pod/helm-operation-j45bz Started container helm
12m Normal Pulled pod/helm-operation-j45bz Container image "rancher/shell:v0.1.6" already present on machine
12m Normal Created pod/helm-operation-j45bz Created container proxy
12m Normal Started pod/helm-operation-j45bz Started container proxy
<unknown> Normal Scheduled pod/helm-operation-n8w9w Successfully assigned cattle-system/helm-operation-n8w9w to fuji1
10m Normal Pulled pod/helm-operation-n8w9w Container image "rancher/shell:v0.1.6" already present on machine
10m Normal Created pod/helm-operation-n8w9w Created container helm
10m Normal Started pod/helm-operation-n8w9w Started container helm
10m Normal Pulled pod/helm-operation-n8w9w Container image "rancher/shell:v0.1.6" already present on machine
10m Normal Created pod/helm-operation-n8w9w Created container proxy
10m Normal Started pod/helm-operation-n8w9w Started container proxy
<unknown> Normal Scheduled pod/helm-operation-wsljx Successfully assigned cattle-system/helm-operation-wsljx to fuji1
11m Normal Pulled pod/helm-operation-wsljx Container image "rancher/shell:v0.1.6" already present on machine
11m Normal Created pod/helm-operation-wsljx Created container helm
11m Normal Started pod/helm-operation-wsljx Started container helm
11m Normal Pulled pod/helm-operation-wsljx Container image "rancher/shell:v0.1.6" already present on machine
11m Normal Created pod/helm-operation-wsljx Created container proxy
11m Normal Started pod/helm-operation-wsljx Started container proxy
<unknown> Normal Scheduled pod/rancher-cbbdf9fd5-n4cgr Successfully assigned cattle-system/rancher-cbbdf9fd5-n4cgr to fuji1
21m Normal Pulling pod/rancher-cbbdf9fd5-n4cgr Pulling image "rancher/rancher:v2.5.9"
15m Normal Pulled pod/rancher-cbbdf9fd5-n4cgr Successfully pulled image "rancher/rancher:v2.5.9"
15m Normal Created pod/rancher-cbbdf9fd5-n4cgr Created container rancher
15m Normal Started pod/rancher-cbbdf9fd5-n4cgr Started container rancher
14m Warning Unhealthy pod/rancher-cbbdf9fd5-n4cgr Readiness probe failed: Get http://10.42.0.10:80/healthz: dial tcp 10.42.0.10:80: connect: connection refused
14m Warning Unhealthy pod/rancher-cbbdf9fd5-n4cgr Liveness probe failed: Get http://10.42.0.10:80/healthz: dial tcp 10.42.0.10:80: connect: connection refused
<unknown> Normal Scheduled pod/rancher-cbbdf9fd5-nj99v Successfully assigned cattle-system/rancher-cbbdf9fd5-nj99v to fuji1
21m Normal Pulling pod/rancher-cbbdf9fd5-nj99v Pulling image "rancher/rancher:v2.5.9"
15m Normal Pulled pod/rancher-cbbdf9fd5-nj99v Successfully pulled image "rancher/rancher:v2.5.9"
15m Normal Created pod/rancher-cbbdf9fd5-nj99v Created container rancher
15m Normal Started pod/rancher-cbbdf9fd5-nj99v Started container rancher
14m Warning Unhealthy pod/rancher-cbbdf9fd5-nj99v Readiness probe failed: Get http://10.42.0.11:80/healthz: dial tcp 10.42.0.11:80: connect: connection refused
14m Warning Unhealthy pod/rancher-cbbdf9fd5-nj99v Liveness probe failed: Get http://10.42.0.11:80/healthz: dial tcp 10.42.0.11:80: connect: connection refused
<unknown> Normal Scheduled pod/rancher-cbbdf9fd5-pmz5b Successfully assigned cattle-system/rancher-cbbdf9fd5-pmz5b to fuji1
21m Normal Pulling pod/rancher-cbbdf9fd5-pmz5b Pulling image "rancher/rancher:v2.5.9"
16m Normal Pulled pod/rancher-cbbdf9fd5-pmz5b Successfully pulled image "rancher/rancher:v2.5.9"
15m Normal Created pod/rancher-cbbdf9fd5-pmz5b Created container rancher
15m Normal Started pod/rancher-cbbdf9fd5-pmz5b Started container rancher
14m Warning Unhealthy pod/rancher-cbbdf9fd5-pmz5b Readiness probe failed: Get http://10.42.0.9:80/healthz: dial tcp 10.42.0.9:80: connect: connection refused
14m Warning Unhealthy pod/rancher-cbbdf9fd5-pmz5b Liveness probe failed: Get http://10.42.0.9:80/healthz: dial tcp 10.42.0.9:80: connect: connection refused
21m Normal SuccessfulCreate replicaset/rancher-cbbdf9fd5 Created pod: rancher-cbbdf9fd5-n4cgr
21m Normal SuccessfulCreate replicaset/rancher-cbbdf9fd5 Created pod: rancher-cbbdf9fd5-nj99v
21m Normal SuccessfulCreate replicaset/rancher-cbbdf9fd5 Created pod: rancher-cbbdf9fd5-pmz5b
<unknown> Normal Scheduled pod/rancher-webhook-69b7997768-ptdph Successfully assigned cattle-system/rancher-webhook-69b7997768-ptdph to fuji1
12m Normal Pulling pod/rancher-webhook-69b7997768-ptdph Pulling image "rancher/rancher-webhook:v0.1.1"
11m Normal Pulled pod/rancher-webhook-69b7997768-ptdph Successfully pulled image "rancher/rancher-webhook:v0.1.1"
11m Normal Created pod/rancher-webhook-69b7997768-ptdph Created container rancher-webhook
11m Normal Started pod/rancher-webhook-69b7997768-ptdph Started container rancher-webhook
12m Normal SuccessfulCreate replicaset/rancher-webhook-69b7997768 Created pod: rancher-webhook-69b7997768-ptdph
12m Normal ScalingReplicaSet deployment/rancher-webhook Scaled up replica set rancher-webhook-69b7997768 to 1
21m Normal ScalingReplicaSet deployment/rancher Scaled up replica set rancher-cbbdf9fd5 to 3
21m Normal CREATE ingress/rancher Ingress cattle-system/rancher
13m Normal UPDATE ingress/rancher Ingress cattle-system/rancher
21m Normal CreateCertificate ingress/rancher Successfully created Certificate "tls-rancher-ingress"
21m Normal Started challenge/tls-rancher-ingress-c55rq-193660333-2832573432 Challenge scheduled for processing
21m Normal Presented challenge/tls-rancher-ingress-c55rq-193660333-2832573432 Presented challenge using HTTP-01 challenge mechanism
21m Normal Created order/tls-rancher-ingress-c55rq-193660333 Created Challenge resource "tls-rancher-ingress-c55rq-193660333-2832573432" for domain "rancher.my.org"
21m Normal IssuerNotReady certificaterequest/tls-rancher-ingress-c55rq Referenced issuer does not have a Ready status condition
21m Normal Issuing certificate/tls-rancher-ingress Issuing certificate as Secret does not exist
21m Normal Generated certificate/tls-rancher-ingress Stored new private key in temporary Secret resource "tls-rancher-ingress-n5f5l"
21m Normal Requested certificate/tls-rancher-ingress Created new CertificateRequest resource "tls-rancher-ingress-c55rq"

```

so what do you think? can you point me in the right direction ?

---

<div class="post-metadata">

**Author:** ![superseb](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/superseb/32/2424_2.png) [@superseb](https://forums.suse.com/u/superseb)\
**Post date:** [August 6, 2021, 9:43pm UTC](https://forums.suse.com/t/liveness-probe-failed-connection-refused/20837/6 "2021-08-06T21:43:53Z")

</div>

Not sure if I am following, the output shows Running and 0 restarts. Are you saying that changes and they restart randomly? Can you get logs for each pod (or for the pod that is restarting) and post them in full?

---

<div class="post-metadata">

**Author:** ![kamal\_boumahdi](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/kamal_boumahdi/32/7837_2.png) [@kamal\_boumahdi](https://forums.suse.com/u/kamal_boumahdi)\
**Post date:** [August 8, 2021, 1:14pm UTC](https://forums.suse.com/t/liveness-probe-failed-connection-refused/20837/7 "2021-08-08T13:14:40Z")

</div>

earlier this morning I checked, and this what I found:

```auto
$ kubectl -n cattle-system get pods -l app=rancher -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
rancher-cbbdf9fd5-n4cgr 1/1 Running 2 43h 10.42.0.34 fuji1 <none> <none>
rancher-cbbdf9fd5-nj99v 1/1 Running 2 43h 10.42.0.39 fuji1 <none> <none>
rancher-cbbdf9fd5-pmz5b 1/1 Running 2 43h 10.42.0.33 fuji1 <none> <none>

```

and the logs go like this:  
for **rancher-cbbdf9fd5-n4cgr**

```auto
$ kubectl -n cattle-system logs rancher-cbbdf9fd5-n4cgr
2021/08/08 12:48:47 [INFO] Rancher version v2.5.9 (3c5418944) is starting
2021/08/08 12:48:47 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:true AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Agent:false Features: ClusterRegistry:}
2021/08/08 12:48:47 [INFO] Listening on /tmp/log.sock
2021/08/08 12:48:54 [INFO] Starting API controllers
2021/08/08 12:48:55 [INFO] Starting management.cattle.io/v3, Kind=User controller
2021/08/08 12:48:55 [INFO] Starting management.cattle.io/v3, Kind=Token controller
2021/08/08 12:48:55 [INFO] Starting management.cattle.io/v3, Kind=GroupMember controller
2021/08/08 12:48:55 [INFO] Starting management.cattle.io/v3, Kind=UserAttribute controller
2021/08/08 12:48:55 [INFO] Starting /v1, Kind=Secret controller
2021/08/08 12:48:55 [INFO] Starting management.cattle.io/v3, Kind=Group controller
2021/08/08 12:48:58 [INFO] Starting apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition controller
2021/08/08 12:48:58 [INFO] Starting /v1, Kind=Secret controller
2021/08/08 12:48:58 [INFO] Starting management.cattle.io/v3, Kind=Feature controller
2021/08/08 12:48:58 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2021/08/08 12:48:58 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2021/08/08 12:48:58 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2021/08/08 12:48:58 [INFO] Starting management.cattle.io/v3, Kind=Preference controller
2021/08/08 12:48:58 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2021/08/08 12:48:58 [INFO] Starting management.cattle.io/v3, Kind=Cluster controller
2021/08/08 12:48:58 [INFO] Starting catalog.cattle.io/v1, Kind=ClusterRepo controller
2021/08/08 12:48:58 [INFO] Starting management.cattle.io/v3, Kind=Setting controller
2021/08/08 12:48:58 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2021/08/08 12:48:58 [INFO] Starting /v1, Kind=ConfigMap controller
I0808 12:48:58.578644 7 leaderelection.go:243] attempting to acquire leader lease kube-system/cattle-controllers...
2021/08/08 12:48:59 [INFO] Refreshing all schemas
2021/08/08 12:49:01 [INFO] Refreshing all schemas
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind Binding
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind ComponentStatus
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind ConfigMap
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind Endpoints
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind Event
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind LimitRange
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind Namespace
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind Node
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind PersistentVolumeClaim
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind PersistentVolume
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind Pod
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind PodTemplate
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind ReplicationController
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind ResourceQuota
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind Secret
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind ServiceAccount
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind Service
2021/08/08 12:49:01 [INFO] APIVersion apiregistration.k8s.io/v1 Kind APIService
2021/08/08 12:49:01 [INFO] APIVersion apiregistration.k8s.io/v1beta1 Kind APIService
2021/08/08 12:49:01 [INFO] APIVersion extensions/v1beta1 Kind Ingress
2021/08/08 12:49:01 [INFO] APIVersion apps/v1 Kind ControllerRevision
2021/08/08 12:49:01 [INFO] APIVersion apps/v1 Kind DaemonSet
2021/08/08 12:49:01 [INFO] APIVersion apps/v1 Kind Deployment
2021/08/08 12:49:01 [INFO] APIVersion apps/v1 Kind ReplicaSet
2021/08/08 12:49:01 [INFO] APIVersion apps/v1 Kind StatefulSet
2021/08/08 12:49:01 [INFO] APIVersion events.k8s.io/v1beta1 Kind Event
2021/08/08 12:49:01 [INFO] APIVersion authentication.k8s.io/v1 Kind TokenReview
2021/08/08 12:49:01 [INFO] APIVersion authentication.k8s.io/v1beta1 Kind TokenReview
2021/08/08 12:49:01 [INFO] APIVersion authorization.k8s.io/v1 Kind LocalSubjectAccessReview
2021/08/08 12:49:01 [INFO] APIVersion authorization.k8s.io/v1 Kind SelfSubjectAccessReview
2021/08/08 12:49:01 [INFO] APIVersion authorization.k8s.io/v1 Kind SelfSubjectRulesReview
2021/08/08 12:49:01 [INFO] APIVersion authorization.k8s.io/v1 Kind SubjectAccessReview
2021/08/08 12:49:01 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind LocalSubjectAccessReview
2021/08/08 12:49:01 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SelfSubjectAccessReview
2021/08/08 12:49:01 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SelfSubjectRulesReview
2021/08/08 12:49:01 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SubjectAccessReview
2021/08/08 12:49:01 [INFO] APIVersion autoscaling/v1 Kind HorizontalPodAutoscaler
2021/08/08 12:49:01 [INFO] APIVersion autoscaling/v2beta1 Kind HorizontalPodAutoscaler
2021/08/08 12:49:01 [INFO] APIVersion autoscaling/v2beta2 Kind HorizontalPodAutoscaler
2021/08/08 12:49:01 [INFO] APIVersion batch/v1 Kind Job
2021/08/08 12:49:01 [INFO] APIVersion batch/v1beta1 Kind CronJob
2021/08/08 12:49:01 [INFO] APIVersion certificates.k8s.io/v1beta1 Kind CertificateSigningRequest
2021/08/08 12:49:01 [INFO] APIVersion networking.k8s.io/v1 Kind NetworkPolicy
2021/08/08 12:49:01 [INFO] APIVersion networking.k8s.io/v1beta1 Kind Ingress
2021/08/08 12:49:01 [INFO] APIVersion policy/v1beta1 Kind PodDisruptionBudget
2021/08/08 12:49:01 [INFO] APIVersion policy/v1beta1 Kind PodSecurityPolicy
2021/08/08 12:49:01 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind ClusterRoleBinding
2021/08/08 12:49:01 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind ClusterRole
2021/08/08 12:49:01 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind RoleBinding
2021/08/08 12:49:01 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind Role
2021/08/08 12:49:01 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind ClusterRoleBinding
2021/08/08 12:49:01 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind ClusterRole
2021/08/08 12:49:01 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind RoleBinding
2021/08/08 12:49:01 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind Role
2021/08/08 12:49:01 [INFO] APIVersion storage.k8s.io/v1 Kind CSINode
2021/08/08 12:49:01 [INFO] APIVersion storage.k8s.io/v1 Kind StorageClass
2021/08/08 12:49:01 [INFO] APIVersion storage.k8s.io/v1 Kind VolumeAttachment
2021/08/08 12:49:01 [INFO] APIVersion storage.k8s.io/v1beta1 Kind CSIDriver
2021/08/08 12:49:01 [INFO] APIVersion storage.k8s.io/v1beta1 Kind CSINode
2021/08/08 12:49:01 [INFO] APIVersion storage.k8s.io/v1beta1 Kind StorageClass
2021/08/08 12:49:01 [INFO] APIVersion storage.k8s.io/v1beta1 Kind VolumeAttachment
2021/08/08 12:49:01 [INFO] APIVersion admissionregistration.k8s.io/v1 Kind MutatingWebhookConfiguration
2021/08/08 12:49:01 [INFO] APIVersion admissionregistration.k8s.io/v1 Kind ValidatingWebhookConfiguration
2021/08/08 12:49:01 [INFO] APIVersion admissionregistration.k8s.io/v1beta1 Kind MutatingWebhookConfiguration
2021/08/08 12:49:01 [INFO] APIVersion admissionregistration.k8s.io/v1beta1 Kind ValidatingWebhookConfiguration
2021/08/08 12:49:01 [INFO] APIVersion apiextensions.k8s.io/v1 Kind CustomResourceDefinition
2021/08/08 12:49:01 [INFO] APIVersion apiextensions.k8s.io/v1beta1 Kind CustomResourceDefinition
2021/08/08 12:49:01 [INFO] APIVersion scheduling.k8s.io/v1 Kind PriorityClass
2021/08/08 12:49:01 [INFO] APIVersion scheduling.k8s.io/v1beta1 Kind PriorityClass
2021/08/08 12:49:01 [INFO] APIVersion coordination.k8s.io/v1 Kind Lease
2021/08/08 12:49:01 [INFO] APIVersion coordination.k8s.io/v1beta1 Kind Lease
2021/08/08 12:49:01 [INFO] APIVersion node.k8s.io/v1beta1 Kind RuntimeClass
2021/08/08 12:49:01 [INFO] APIVersion discovery.k8s.io/v1beta1 Kind EndpointSlice
2021/08/08 12:49:01 [INFO] APIVersion acme.cert-manager.io/v1 Kind Challenge
2021/08/08 12:49:01 [INFO] APIVersion acme.cert-manager.io/v1 Kind Order
2021/08/08 12:49:01 [INFO] APIVersion acme.cert-manager.io/v1beta1 Kind Order
2021/08/08 12:49:01 [INFO] APIVersion acme.cert-manager.io/v1beta1 Kind Challenge
2021/08/08 12:49:01 [INFO] APIVersion acme.cert-manager.io/v1alpha3 Kind Order
2021/08/08 12:49:01 [INFO] APIVersion acme.cert-manager.io/v1alpha3 Kind Challenge
2021/08/08 12:49:01 [INFO] APIVersion acme.cert-manager.io/v1alpha2 Kind Challenge
2021/08/08 12:49:01 [INFO] APIVersion acme.cert-manager.io/v1alpha2 Kind Order
2021/08/08 12:49:01 [INFO] APIVersion catalog.cattle.io/v1 Kind ClusterRepo
2021/08/08 12:49:01 [INFO] APIVersion catalog.cattle.io/v1 Kind App
2021/08/08 12:49:01 [INFO] APIVersion catalog.cattle.io/v1 Kind Operation
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1 Kind ClusterIssuer
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1 Kind CertificateRequest
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1 Kind Issuer
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1 Kind Certificate
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1beta1 Kind CertificateRequest
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1beta1 Kind Issuer
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1beta1 Kind Certificate
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1beta1 Kind ClusterIssuer
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1alpha3 Kind Issuer
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1alpha3 Kind Certificate
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1alpha3 Kind ClusterIssuer
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1alpha3 Kind CertificateRequest
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1alpha2 Kind Certificate
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1alpha2 Kind ClusterIssuer
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1alpha2 Kind CertificateRequest
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1alpha2 Kind Issuer
2021/08/08 12:49:01 [INFO] APIVersion crd.projectcalico.org/v1 Kind NetworkSet
2021/08/08 12:49:01 [INFO] APIVersion crd.projectcalico.org/v1 Kind IPAMHandle
2021/08/08 12:49:01 [INFO] APIVersion crd.projectcalico.org/v1 Kind BGPPeer
2021/08/08 12:49:01 [INFO] APIVersion crd.projectcalico.org/v1 Kind GlobalNetworkSet
2021/08/08 12:49:01 [INFO] APIVersion crd.projectcalico.org/v1 Kind NetworkPolicy
2021/08/08 12:49:01 [INFO] APIVersion crd.projectcalico.org/v1 Kind IPAMBlock
2021/08/08 12:49:01 [INFO] APIVersion crd.projectcalico.org/v1 Kind GlobalNetworkPolicy
2021/08/08 12:49:01 [INFO] APIVersion crd.projectcalico.org/v1 Kind FelixConfiguration
2021/08/08 12:49:01 [INFO] APIVersion crd.projectcalico.org/v1 Kind HostEndpoint
2021/08/08 12:49:01 [INFO] APIVersion crd.projectcalico.org/v1 Kind IPPool
2021/08/08 12:49:01 [INFO] APIVersion crd.projectcalico.org/v1 Kind IPAMConfig
2021/08/08 12:49:01 [INFO] APIVersion crd.projectcalico.org/v1 Kind ClusterInformation
2021/08/08 12:49:01 [INFO] APIVersion crd.projectcalico.org/v1 Kind BlockAffinity
2021/08/08 12:49:01 [INFO] APIVersion crd.projectcalico.org/v1 Kind BGPConfiguration
2021/08/08 12:49:01 [INFO] APIVersion gitjob.cattle.io/v1 Kind GitJob
2021/08/08 12:49:01 [INFO] APIVersion monitoring.coreos.com/v1 Kind PrometheusRule
2021/08/08 12:49:01 [INFO] APIVersion monitoring.coreos.com/v1 Kind Prometheus
2021/08/08 12:49:01 [INFO] APIVersion monitoring.coreos.com/v1 Kind Alertmanager
2021/08/08 12:49:01 [INFO] APIVersion monitoring.coreos.com/v1 Kind ServiceMonitor
2021/08/08 12:49:01 [INFO] APIVersion rancher.cattle.io/v1 Kind Project
2021/08/08 12:49:01 [INFO] APIVersion rancher.cattle.io/v1 Kind RoleTemplate
2021/08/08 12:49:01 [INFO] APIVersion rancher.cattle.io/v1 Kind RoleTemplateBinding
2021/08/08 12:49:01 [INFO] APIVersion rancher.cattle.io/v1 Kind Cluster
2021/08/08 12:49:01 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind BundleDeployment
2021/08/08 12:49:01 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind BundleNamespaceMapping
2021/08/08 12:49:01 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind Bundle
2021/08/08 12:49:01 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind Cluster
2021/08/08 12:49:01 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind Content
2021/08/08 12:49:01 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind ClusterRegistration
2021/08/08 12:49:01 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind GitRepoRestriction
2021/08/08 12:49:01 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind ClusterRegistrationToken
2021/08/08 12:49:01 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind GitRepo
2021/08/08 12:49:01 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind ClusterGroup
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind MultiClusterAppRevision
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind RkeK8sSystemImage
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind EtcdBackup
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind GlobalRoleBinding
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind RoleTemplate
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind Cluster
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind Catalog
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind MultiClusterApp
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind DynamicSchema
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind MonitorMetric
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ClusterRoleTemplateBinding
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlertRule
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind Feature
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind Token
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlert
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind NodePool
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind Project
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind TemplateVersion
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind Preference
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ClusterTemplateRevision
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlert
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind RkeAddon
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ClusterMonitorGraph
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind Group
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind CatalogTemplate
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind PodSecurityPolicyTemplate
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ClusterScan
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind RkeK8sServiceOption
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind Setting
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ProjectNetworkPolicy
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind SamlToken
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ProjectLogging
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ClusterTemplate
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind KontainerDriver
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind GlobalRole
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind CisBenchmarkVersion
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlertGroup
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ClusterCatalog
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ProjectCatalog
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind TemplateContent
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind GroupMember
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind User
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind UserAttribute
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind FleetWorkspace
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind PodSecurityPolicyTemplateProjectBinding
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ClusterLogging
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlertGroup
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind Template
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ComposeConfig
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind CisConfig
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind NodeTemplate
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ClusterRegistrationToken
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind Notifier
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind AuthConfig
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind GlobalDnsProvider
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ProjectMonitorGraph
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlertRule
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ProjectRoleTemplateBinding
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind NodeDriver
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind Node
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind GlobalDns
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind CatalogTemplateVersion
2021/08/08 12:49:01 [INFO] APIVersion project.cattle.io/v3 Kind App
2021/08/08 12:49:01 [INFO] APIVersion project.cattle.io/v3 Kind PipelineSetting
2021/08/08 12:49:01 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeProviderConfig
2021/08/08 12:49:01 [INFO] APIVersion project.cattle.io/v3 Kind AppRevision
2021/08/08 12:49:01 [INFO] APIVersion project.cattle.io/v3 Kind Pipeline
2021/08/08 12:49:01 [INFO] APIVersion project.cattle.io/v3 Kind PipelineExecution
2021/08/08 12:49:01 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeRepository
2021/08/08 12:49:01 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeCredential
2021/08/08 12:49:01 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind NodeMetrics
2021/08/08 12:49:01 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind PodMetrics
2021/08/08 12:49:03 [INFO] Running in clustered mode with ID 10.42.0.34, monitoring endpoint cattle-system/rancher
2021/08/08 12:49:17 [INFO] Starting cluster controllers for local
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=CatalogTemplateVersion controller
2021/08/08 12:49:21 [INFO] Starting project.cattle.io/v3, Kind=SourceCodeRepository controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=GlobalDns controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=GlobalRole controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=ProjectRoleTemplateBinding controller
2021/08/08 12:49:21 [INFO] Starting project.cattle.io/v3, Kind=Pipeline controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=MultiClusterApp controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=CisConfig controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=PodSecurityPolicyTemplate controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=RkeK8sSystemImage controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=Catalog controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=CisBenchmarkVersion controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=GlobalRoleBinding controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=RoleTemplate controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=NodePool controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=Project controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=ClusterRoleTemplateBinding controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=ClusterTemplateRevision controller
2021/08/08 12:49:21 [INFO] Starting project.cattle.io/v3, Kind=SourceCodeCredential controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=MultiClusterAppRevision controller
2021/08/08 12:49:21 [INFO] Starting project.cattle.io/v3, Kind=App controller
2021/08/08 12:49:21 [INFO] Starting project.cattle.io/v3, Kind=PipelineExecution controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=GroupMember controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=Token controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=DynamicSchema controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=NodeTemplate controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=Group controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=NodeDriver controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=AuthConfig controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=ClusterCatalog controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=User controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=RkeAddon controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=UserAttribute controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=CatalogTemplate controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=ProjectCatalog controller
2021/08/08 12:49:21 [INFO] Starting /v1, Kind=Endpoints controller
2021/08/08 12:49:21 [INFO] Starting /v1, Kind=Namespace controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=KontainerDriver controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=ProjectLogging controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=PodSecurityPolicyTemplateProjectBinding controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=ClusterRegistrationToken controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=Node controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=RkeK8sServiceOption controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=ClusterTemplate controller
2021/08/08 12:49:23 [INFO] Starting cluster agent for local [owner=false]
2021/08/08 12:49:23 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2021/08/08 12:49:23 [INFO] Starting /v1, Kind=Namespace controller
2021/08/08 12:49:23 [INFO] Starting /v1, Kind=Service controller
2021/08/08 12:49:23 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2021/08/08 12:49:23 [INFO] Starting /v1, Kind=Endpoints controller
2021/08/08 12:49:23 [INFO] Starting /v1, Kind=Pod controller
2021/08/08 12:49:23 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2021/08/08 12:49:23 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2021/08/08 12:49:23 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRegistrationToken
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=CisBenchmarkVersion
2021/08/08 12:49:28 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=RoleBinding
2021/08/08 12:49:28 [INFO] Watching metadata for storage.k8s.io/v1, Kind=VolumeAttachment
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=ResourceQuota
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=PodTemplate
2021/08/08 12:49:28 [INFO] Watching metadata for coordination.k8s.io/v1, Kind=Lease
2021/08/08 12:49:28 [INFO] Watching metadata for storage.k8s.io/v1beta1, Kind=CSIDriver
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=PersistentVolume
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=MultiClusterAppRevision
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ComposeConfig
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=RoleTemplate
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=UserAttribute
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodeDriver
2021/08/08 12:49:28 [INFO] Watching metadata for batch/v1, Kind=Job
2021/08/08 12:49:28 [INFO] Watching metadata for project.cattle.io/v3, Kind=PipelineSetting
2021/08/08 12:49:28 [INFO] Watching metadata for autoscaling/v2beta2, Kind=HorizontalPodAutoscaler
2021/08/08 12:49:28 [INFO] Watching metadata for storage.k8s.io/v1, Kind=StorageClass
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=MultiClusterApp
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterTemplate
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeAddon
2021/08/08 12:49:28 [INFO] Watching metadata for project.cattle.io/v3, Kind=PipelineExecution
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlertRule
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=Template
2021/08/08 12:49:28 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=ClusterRepo
2021/08/08 12:49:28 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Cluster
2021/08/08 12:49:28 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=NetworkSet
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=Cluster
2021/08/08 12:49:28 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=FelixConfiguration
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=Preference
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlert
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=Namespace
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=Node
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlertRule
2021/08/08 12:49:28 [INFO] Watching metadata for rancher.cattle.io/v1, Kind=RoleTemplate
2021/08/08 12:49:28 [INFO] Watching metadata for cert-manager.io/v1, Kind=Issuer
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterScan
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=Feature
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=Pod
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=GroupMember
2021/08/08 12:49:28 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=GlobalNetworkPolicy
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=AuthConfig
2021/08/08 12:49:28 [INFO] Watching metadata for project.cattle.io/v3, Kind=AppRevision
2021/08/08 12:49:28 [INFO] Watching metadata for events.k8s.io/v1beta1, Kind=Event
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=User
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=Notifier
2021/08/08 12:49:28 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterRegistrationToken
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=Group
2021/08/08 12:49:28 [INFO] Watching metadata for discovery.k8s.io/v1beta1, Kind=EndpointSlice
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=Service
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectCatalog
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=ReplicationController
2021/08/08 12:49:28 [INFO] Watching metadata for apps/v1, Kind=ReplicaSet
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalRoleBinding
2021/08/08 12:49:28 [INFO] Watching metadata for acme.cert-manager.io/v1, Kind=Challenge
2021/08/08 12:49:28 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=Operation
2021/08/08 12:49:28 [INFO] Watching metadata for apiregistration.k8s.io/v1, Kind=APIService
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectNetworkPolicy
2021/08/08 12:49:28 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=ClusterInformation
2021/08/08 12:49:28 [INFO] Watching metadata for storage.k8s.io/v1, Kind=CSINode
2021/08/08 12:49:28 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterRegistration
2021/08/08 12:49:28 [INFO] Watching metadata for apps/v1, Kind=Deployment
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=SamlToken
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=KontainerDriver
2021/08/08 12:49:28 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=NetworkPolicy
2021/08/08 12:49:28 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=BundleDeployment
2021/08/08 12:49:28 [INFO] Watching metadata for networking.k8s.io/v1beta1, Kind=Ingress
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=Endpoints
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=LimitRange
2021/08/08 12:49:28 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeRepository
2021/08/08 12:49:28 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=IPPool
2021/08/08 12:49:28 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Content
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRoleTemplateBinding
2021/08/08 12:49:28 [INFO] Watching metadata for rancher.cattle.io/v1, Kind=Cluster
2021/08/08 12:49:28 [INFO] Watching metadata for acme.cert-manager.io/v1, Kind=Order
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlert
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=Event
2021/08/08 12:49:28 [INFO] Watching metadata for apps/v1, Kind=StatefulSet
2021/08/08 12:49:28 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=ServiceMonitor
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=Node
2021/08/08 12:49:28 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeProviderConfig
2021/08/08 12:49:28 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=BundleNamespaceMapping
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterMonitorGraph
2021/08/08 12:49:28 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=Prometheus
2021/08/08 12:49:28 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=IPAMBlock
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplate
2021/08/08 12:49:28 [INFO] Watching metadata for node.k8s.io/v1beta1, Kind=RuntimeClass
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplateVersion
2021/08/08 12:49:28 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=PrometheusRule
2021/08/08 12:49:28 [INFO] Watching metadata for networking.k8s.io/v1, Kind=NetworkPolicy
2021/08/08 12:49:28 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Bundle
2021/08/08 12:49:28 [INFO] Watching metadata for rancher.cattle.io/v1, Kind=RoleTemplateBinding
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalDnsProvider
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=CisConfig
2021/08/08 12:49:28 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRole
2021/08/08 12:49:28 [INFO] Watching metadata for certificates.k8s.io/v1beta1, Kind=CertificateSigningRequest
2021/08/08 12:49:28 [INFO] Watching metadata for scheduling.k8s.io/v1, Kind=PriorityClass
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=Token
2021/08/08 12:49:28 [INFO] Watching metadata for cert-manager.io/v1, Kind=CertificateRequest
2021/08/08 12:49:28 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=Alertmanager
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterTemplateRevision
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectRoleTemplateBinding
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=Secret
2021/08/08 12:49:28 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=BlockAffinity
2021/08/08 12:49:28 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=HostEndpoint
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlertGroup
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodeTemplate
2021/08/08 12:49:28 [INFO] Watching metadata for batch/v1beta1, Kind=CronJob
2021/08/08 12:49:28 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=App
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=Project
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=FleetWorkspace
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlertGroup
2021/08/08 12:49:28 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterGroup
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=TemplateContent
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectMonitorGraph
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=ConfigMap
2021/08/08 12:49:28 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding
2021/08/08 12:49:28 [INFO] Watching metadata for policy/v1beta1, Kind=PodDisruptionBudget
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=Setting
2021/08/08 12:49:28 [INFO] Watching metadata for cert-manager.io/v1, Kind=ClusterIssuer
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=PodSecurityPolicyTemplateProjectBinding
2021/08/08 12:49:28 [INFO] Watching metadata for project.cattle.io/v3, Kind=Pipeline
2021/08/08 12:49:28 [INFO] Watching metadata for apiextensions.k8s.io/v1, Kind=CustomResourceDefinition
2021/08/08 12:49:28 [INFO] Watching metadata for admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalDns
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterLogging
2021/08/08 12:49:28 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=GlobalNetworkSet
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodePool
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterCatalog
2021/08/08 12:49:28 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=IPAMConfig
2021/08/08 12:49:28 [INFO] Watching metadata for apps/v1, Kind=DaemonSet
2021/08/08 12:49:28 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=GitRepo
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=DynamicSchema
2021/08/08 12:49:28 [INFO] Watching metadata for policy/v1beta1, Kind=PodSecurityPolicy
2021/08/08 12:49:28 [INFO] Watching metadata for project.cattle.io/v3, Kind=App
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=MonitorMetric
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=TemplateVersion
2021/08/08 12:49:28 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=BGPConfiguration
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=Catalog
2021/08/08 12:49:28 [INFO] Watching metadata for cert-manager.io/v1, Kind=Certificate
2021/08/08 12:49:28 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=Role
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=ServiceAccount
2021/08/08 12:49:28 [INFO] Watching metadata for rancher.cattle.io/v1, Kind=Project
2021/08/08 12:49:28 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=GitRepoRestriction
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sSystemImage
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sServiceOption
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=PersistentVolumeClaim
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=PodSecurityPolicyTemplate
2021/08/08 12:49:28 [INFO] Watching metadata for admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration
2021/08/08 12:49:28 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=BGPPeer
2021/08/08 12:49:28 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=IPAMHandle
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalRole
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectLogging
2021/08/08 12:49:28 [INFO] Watching metadata for gitjob.cattle.io/v1, Kind=GitJob
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=EtcdBackup
2021/08/08 12:49:28 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeCredential
2021/08/08 12:49:28 [INFO] Watching metadata for apps/v1, Kind=ControllerRevision
I0808 12:49:38.802997 7 trace.go:205] Trace[1819992954]: "Reflector ListAndWatch" name:pkg/mod/github.com/rancher/client-go@v1.20.0-rancher.1/tools/cache/reflector.go:168 (08-Aug-2021 12:49:28.725) (total time: 10077ms):
Trace[1819992954]: ---"Objects listed" 10077ms (12:49:00.802)
Trace[1819992954]: [10.077395935s] [10.077395935s] END
2021/08/08 12:50:04 [INFO] Listening on :443
2021/08/08 12:50:04 [INFO] Listening on :80
2021/08/08 12:50:04 [INFO] Listening on :444
2021/08/08 12:50:04 [INFO] Updating TLS secret for serving-cert (count: 8): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.34:10.42.0.34 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:f3a4b6f81a3e624a595617f5e18f34a1a0186c063d72b1262fd09343af54a325]
2021/08/08 12:50:05 [INFO] Active TLS secret serving-cert (ver=36375) (count 8): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:429e226403a5b18ac214ac57165b349fcd12a069caef133d72f269769e175315]
2021/08/08 12:50:05 [INFO] Starting /v1, Kind=Secret controller
2021/08/08 12:50:05 [INFO] Active TLS secret serving-cert (ver=36375) (count 8): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:429e226403a5b18ac214ac57165b349fcd12a069caef133d72f269769e175315]
2021/08/08 12:50:07 [INFO] Updating TLS secret for serving-cert (count: 9): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.34:10.42.0.34 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:cbb8784b086c09c2d80b5a2c202cd7e584e513ad03f6a216463a80a2e76d0558]
2021/08/08 12:50:07 [INFO] Active TLS secret serving-cert (ver=36388) (count 9): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.34:10.42.0.34 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:cbb8784b086c09c2d80b5a2c202cd7e584e513ad03f6a216463a80a2e76d0558]
2021/08/08 12:50:07 [INFO] Active TLS secret serving-cert (ver=36388) (count 9): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.34:10.42.0.34 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:cbb8784b086c09c2d80b5a2c202cd7e584e513ad03f6a216463a80a2e76d0558]
2021/08/08 12:50:18 [INFO] Adding peer wss://10.42.0.33/v3/connect, 10.42.0.33
2021/08/08 12:50:19 [ERROR] Failed to connect to peer wss://10.42.0.33/v3/connect [local ID=10.42.0.34]: websocket: bad handshake
2021/08/08 12:50:24 [ERROR] Failed to connect to peer wss://10.42.0.33/v3/connect [local ID=10.42.0.34]: websocket: bad handshake
2021/08/08 12:50:24 [INFO] Handling backend connection request [10.42.0.33]
2021/08/08 12:50:30 [INFO] Active TLS secret serving-cert (ver=36529) (count 10): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.34:10.42.0.34 listener.cattle.io/cn-10.42.0.39:10.42.0.39 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:886488085d31e3ebc49f83d909f43c692a45719f4293c566b41e72dd78dcf92c]
2021/08/08 12:50:31 [INFO] Adding peer wss://10.42.0.39/v3/connect, 10.42.0.39
2021/08/08 12:50:31 [INFO] Stopping cluster agent for local
2021/08/08 12:50:31 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRole workers
W0808 12:50:31.013667 7 reflector.go:437] pkg/mod/github.com/rancher/client-go@v1.20.0-rancher.1/tools/cache/reflector.go:168: watch of *v1.ClusterRole ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W0808 12:50:31.013760 7 reflector.go:437] pkg/mod/github.com/rancher/client-go@v1.20.0-rancher.1/tools/cache/reflector.go:168: watch of *v1.APIService ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
2021/08/08 12:50:31 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=Role workers
2021/08/08 12:50:31 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=RoleBinding workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=Pod workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=Endpoints workers
2021/08/08 12:50:31 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding workers
2021/08/08 12:50:31 [INFO] Shutting down apiregistration.k8s.io/v1, Kind=APIService workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=Service workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=Namespace workers
2021/08/08 12:50:31 [INFO] Registering project network policy
2021/08/08 12:50:31 [INFO] Not registering CIS controller for non RKE cluster: local
2021/08/08 12:50:31 [INFO] registering podsecuritypolicy cluster handler for cluster local
2021/08/08 12:50:31 [INFO] registering podsecuritypolicy project handler for cluster local
2021/08/08 12:50:31 [INFO] registering podsecuritypolicy namespace handler for cluster local
2021/08/08 12:50:31 [INFO] registering podsecuritypolicy serviceaccount handler for cluster local
2021/08/08 12:50:31 [INFO] registering podsecuritypolicy template handler for cluster local
2021/08/08 12:50:31 [INFO] Registering monitoring for cluster "local"
2021/08/08 12:50:31 [INFO] Registering istio for cluster "local"
2021/08/08 12:50:32 [INFO] Registering namespaceHandler for adding labels
2021/08/08 12:50:32 [INFO] Starting cluster controllers for local
2021/08/08 12:50:32 [INFO] Starting project.cattle.io/v3, Kind=SourceCodeProviderConfig controller
2021/08/08 12:50:32 [INFO] Starting management.cattle.io/v3, Kind=ProjectAlertGroup controller
2021/08/08 12:50:32 [INFO] Starting management.cattle.io/v3, Kind=ClusterScan controller
2021/08/08 12:50:32 [INFO] Starting management.cattle.io/v3, Kind=ClusterLogging controller
2021/08/08 12:50:32 [INFO] Starting management.cattle.io/v3, Kind=Notifier controller
2021/08/08 12:50:32 [INFO] Starting project.cattle.io/v3, Kind=PipelineSetting controller
2021/08/08 12:50:32 [INFO] Starting management.cattle.io/v3, Kind=ClusterAlertGroup controller
2021/08/08 12:50:32 [INFO] Starting management.cattle.io/v3, Kind=ProjectAlert controller
2021/08/08 12:50:32 [INFO] Starting management.cattle.io/v3, Kind=ProjectAlertRule controller
2021/08/08 12:50:32 [INFO] Starting management.cattle.io/v3, Kind=ProjectNetworkPolicy controller
2021/08/08 12:50:32 [INFO] Starting management.cattle.io/v3, Kind=ClusterAlertRule controller
2021/08/08 12:50:33 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2021/08/08 12:50:33 [INFO] Starting batch/v1, Kind=Job controller
2021/08/08 12:50:33 [INFO] Starting /v1, Kind=LimitRange controller
2021/08/08 12:50:33 [INFO] Starting /v1, Kind=Service controller
2021/08/08 12:50:33 [INFO] Starting /v1, Kind=Secret controller
2021/08/08 12:50:33 [INFO] Starting monitoring.coreos.com/v1, Kind=Prometheus controller
2021/08/08 12:50:33 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2021/08/08 12:50:33 [INFO] Starting cluster agent for local [owner=true]
2021/08/08 12:50:33 [INFO] Starting /v1, Kind=ReplicationController controller
2021/08/08 12:50:33 [INFO] Starting apps/v1, Kind=DaemonSet controller
2021/08/08 12:50:33 [INFO] Starting extensions/v1beta1, Kind=Ingress controller
2021/08/08 12:50:33 [INFO] Starting /v1, Kind=ConfigMap controller
2021/08/08 12:50:33 [INFO] Starting /v1, Kind=ResourceQuota controller
2021/08/08 12:50:33 [INFO] Starting apps/v1, Kind=Deployment controller
2021/08/08 12:50:33 [INFO] Starting apps/v1, Kind=ReplicaSet controller
2021/08/08 12:50:33 [INFO] Starting /v1, Kind=Node controller
2021/08/08 12:50:33 [INFO] Starting policy/v1beta1, Kind=PodSecurityPolicy controller
2021/08/08 12:50:33 [INFO] Starting /v1, Kind=Event controller
2021/08/08 12:50:33 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2021/08/08 12:50:33 [INFO] Starting /v1, Kind=Namespace controller
2021/08/08 12:50:33 [INFO] Starting apps/v1, Kind=StatefulSet controller
2021/08/08 12:50:33 [INFO] Starting batch/v1beta1, Kind=CronJob controller
2021/08/08 12:50:33 [INFO] Starting /v1, Kind=Pod controller
2021/08/08 12:50:33 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2021/08/08 12:50:33 [INFO] Starting /v1, Kind=ServiceAccount controller
2021/08/08 12:50:33 [INFO] Starting /v1, Kind=Endpoints controller
2021/08/08 12:50:33 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2021/08/08 12:50:33 [INFO] Starting networking.k8s.io/v1, Kind=NetworkPolicy controller
2021/08/08 12:50:33 [INFO] Starting monitoring.coreos.com/v1, Kind=ServiceMonitor controller
2021/08/08 12:50:34 [INFO] Handling backend connection request [10.42.0.39]

```

for **rancher-cbbdf9fd5-nj99v**

```auto
$ kubectl -n cattle-system logs rancher-cbbdf9fd5-nj99v
2021/08/08 12:48:45 [INFO] Rancher version v2.5.9 (3c5418944) is starting
2021/08/08 12:48:45 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:true AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Agent:false Features: ClusterRegistry:}
2021/08/08 12:48:45 [INFO] Listening on /tmp/log.sock
2021/08/08 12:48:52 [INFO] Starting API controllers
2021/08/08 12:48:52 [INFO] Starting management.cattle.io/v3, Kind=Group controller
2021/08/08 12:48:52 [INFO] Starting management.cattle.io/v3, Kind=User controller
2021/08/08 12:48:52 [INFO] Starting management.cattle.io/v3, Kind=Token controller
2021/08/08 12:48:52 [INFO] Starting management.cattle.io/v3, Kind=UserAttribute controller
2021/08/08 12:48:52 [INFO] Starting /v1, Kind=Secret controller
2021/08/08 12:48:52 [INFO] Starting management.cattle.io/v3, Kind=GroupMember controller
I0808 12:48:54.061007 7 leaderelection.go:243] attempting to acquire leader lease kube-system/cattle-controllers...
2021/08/08 12:48:54 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2021/08/08 12:48:54 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2021/08/08 12:48:54 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2021/08/08 12:48:54 [INFO] Starting management.cattle.io/v3, Kind=Cluster controller
2021/08/08 12:48:54 [INFO] Starting apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition controller
2021/08/08 12:48:54 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2021/08/08 12:48:54 [INFO] Starting catalog.cattle.io/v1, Kind=ClusterRepo controller
2021/08/08 12:48:54 [INFO] Starting management.cattle.io/v3, Kind=Setting controller
2021/08/08 12:48:54 [INFO] Starting /v1, Kind=ConfigMap controller
2021/08/08 12:48:54 [INFO] Starting /v1, Kind=Secret controller
2021/08/08 12:48:54 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2021/08/08 12:48:54 [INFO] Starting management.cattle.io/v3, Kind=Feature controller
2021/08/08 12:48:54 [INFO] Starting management.cattle.io/v3, Kind=Preference controller
2021/08/08 12:48:54 [INFO] Refreshing all schemas
2021/08/08 12:48:58 [INFO] Refreshing all schemas
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind Binding
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind ComponentStatus
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind ConfigMap
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind Endpoints
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind Event
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind LimitRange
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind Namespace
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind Node
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind PersistentVolumeClaim
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind PersistentVolume
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind Pod
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind PodTemplate
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind ReplicationController
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind ResourceQuota
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind Secret
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind ServiceAccount
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind Service
2021/08/08 12:48:59 [INFO] APIVersion apiregistration.k8s.io/v1 Kind APIService
2021/08/08 12:48:59 [INFO] APIVersion apiregistration.k8s.io/v1beta1 Kind APIService
2021/08/08 12:48:59 [INFO] APIVersion extensions/v1beta1 Kind Ingress
2021/08/08 12:48:59 [INFO] APIVersion apps/v1 Kind ControllerRevision
2021/08/08 12:48:59 [INFO] APIVersion apps/v1 Kind DaemonSet
2021/08/08 12:48:59 [INFO] APIVersion apps/v1 Kind Deployment
2021/08/08 12:48:59 [INFO] APIVersion apps/v1 Kind ReplicaSet
2021/08/08 12:48:59 [INFO] APIVersion apps/v1 Kind StatefulSet
2021/08/08 12:48:59 [INFO] APIVersion events.k8s.io/v1beta1 Kind Event
2021/08/08 12:48:59 [INFO] APIVersion authentication.k8s.io/v1 Kind TokenReview
2021/08/08 12:48:59 [INFO] APIVersion authentication.k8s.io/v1beta1 Kind TokenReview
2021/08/08 12:48:59 [INFO] APIVersion authorization.k8s.io/v1 Kind LocalSubjectAccessReview
2021/08/08 12:48:59 [INFO] APIVersion authorization.k8s.io/v1 Kind SelfSubjectAccessReview
2021/08/08 12:48:59 [INFO] APIVersion authorization.k8s.io/v1 Kind SelfSubjectRulesReview
2021/08/08 12:48:59 [INFO] APIVersion authorization.k8s.io/v1 Kind SubjectAccessReview
2021/08/08 12:48:59 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind LocalSubjectAccessReview
2021/08/08 12:48:59 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SelfSubjectAccessReview
2021/08/08 12:48:59 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SelfSubjectRulesReview
2021/08/08 12:48:59 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SubjectAccessReview
2021/08/08 12:48:59 [INFO] APIVersion autoscaling/v1 Kind HorizontalPodAutoscaler
2021/08/08 12:48:59 [INFO] APIVersion autoscaling/v2beta1 Kind HorizontalPodAutoscaler
2021/08/08 12:48:59 [INFO] APIVersion autoscaling/v2beta2 Kind HorizontalPodAutoscaler
2021/08/08 12:48:59 [INFO] APIVersion batch/v1 Kind Job
2021/08/08 12:48:59 [INFO] APIVersion batch/v1beta1 Kind CronJob
2021/08/08 12:48:59 [INFO] APIVersion certificates.k8s.io/v1beta1 Kind CertificateSigningRequest
2021/08/08 12:48:59 [INFO] APIVersion networking.k8s.io/v1 Kind NetworkPolicy
2021/08/08 12:48:59 [INFO] APIVersion networking.k8s.io/v1beta1 Kind Ingress
2021/08/08 12:48:59 [INFO] APIVersion policy/v1beta1 Kind PodDisruptionBudget
2021/08/08 12:48:59 [INFO] APIVersion policy/v1beta1 Kind PodSecurityPolicy
2021/08/08 12:48:59 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind ClusterRoleBinding
2021/08/08 12:48:59 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind ClusterRole
2021/08/08 12:48:59 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind RoleBinding
2021/08/08 12:48:59 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind Role
2021/08/08 12:48:59 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind ClusterRoleBinding
2021/08/08 12:48:59 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind ClusterRole
2021/08/08 12:48:59 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind RoleBinding
2021/08/08 12:48:59 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind Role
2021/08/08 12:48:59 [INFO] APIVersion storage.k8s.io/v1 Kind CSINode
2021/08/08 12:48:59 [INFO] APIVersion storage.k8s.io/v1 Kind StorageClass
2021/08/08 12:48:59 [INFO] APIVersion storage.k8s.io/v1 Kind VolumeAttachment
2021/08/08 12:48:59 [INFO] APIVersion storage.k8s.io/v1beta1 Kind CSIDriver
2021/08/08 12:48:59 [INFO] APIVersion storage.k8s.io/v1beta1 Kind CSINode
2021/08/08 12:48:59 [INFO] APIVersion storage.k8s.io/v1beta1 Kind StorageClass
2021/08/08 12:48:59 [INFO] APIVersion storage.k8s.io/v1beta1 Kind VolumeAttachment
2021/08/08 12:48:59 [INFO] APIVersion admissionregistration.k8s.io/v1 Kind MutatingWebhookConfiguration
2021/08/08 12:48:59 [INFO] APIVersion admissionregistration.k8s.io/v1 Kind ValidatingWebhookConfiguration
2021/08/08 12:48:59 [INFO] APIVersion admissionregistration.k8s.io/v1beta1 Kind MutatingWebhookConfiguration
2021/08/08 12:48:59 [INFO] APIVersion admissionregistration.k8s.io/v1beta1 Kind ValidatingWebhookConfiguration
2021/08/08 12:48:59 [INFO] APIVersion apiextensions.k8s.io/v1 Kind CustomResourceDefinition
2021/08/08 12:48:59 [INFO] APIVersion apiextensions.k8s.io/v1beta1 Kind CustomResourceDefinition
2021/08/08 12:48:59 [INFO] APIVersion scheduling.k8s.io/v1 Kind PriorityClass
2021/08/08 12:48:59 [INFO] APIVersion scheduling.k8s.io/v1beta1 Kind PriorityClass
2021/08/08 12:48:59 [INFO] APIVersion coordination.k8s.io/v1 Kind Lease
2021/08/08 12:48:59 [INFO] APIVersion coordination.k8s.io/v1beta1 Kind Lease
2021/08/08 12:48:59 [INFO] APIVersion node.k8s.io/v1beta1 Kind RuntimeClass
2021/08/08 12:48:59 [INFO] APIVersion discovery.k8s.io/v1beta1 Kind EndpointSlice
2021/08/08 12:48:59 [INFO] APIVersion acme.cert-manager.io/v1 Kind Challenge
2021/08/08 12:48:59 [INFO] APIVersion acme.cert-manager.io/v1 Kind Order
2021/08/08 12:48:59 [INFO] APIVersion acme.cert-manager.io/v1beta1 Kind Order
2021/08/08 12:48:59 [INFO] APIVersion acme.cert-manager.io/v1beta1 Kind Challenge
2021/08/08 12:48:59 [INFO] APIVersion acme.cert-manager.io/v1alpha3 Kind Order
2021/08/08 12:48:59 [INFO] APIVersion acme.cert-manager.io/v1alpha3 Kind Challenge
2021/08/08 12:48:59 [INFO] APIVersion acme.cert-manager.io/v1alpha2 Kind Challenge
2021/08/08 12:48:59 [INFO] APIVersion acme.cert-manager.io/v1alpha2 Kind Order
2021/08/08 12:48:59 [INFO] APIVersion catalog.cattle.io/v1 Kind ClusterRepo
2021/08/08 12:48:59 [INFO] APIVersion catalog.cattle.io/v1 Kind App
2021/08/08 12:48:59 [INFO] APIVersion catalog.cattle.io/v1 Kind Operation
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1 Kind ClusterIssuer
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1 Kind CertificateRequest
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1 Kind Issuer
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1 Kind Certificate
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1beta1 Kind CertificateRequest
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1beta1 Kind Issuer
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1beta1 Kind Certificate
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1beta1 Kind ClusterIssuer
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1alpha3 Kind Issuer
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1alpha3 Kind Certificate
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1alpha3 Kind ClusterIssuer
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1alpha3 Kind CertificateRequest
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1alpha2 Kind Certificate
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1alpha2 Kind ClusterIssuer
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1alpha2 Kind CertificateRequest
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1alpha2 Kind Issuer
2021/08/08 12:48:59 [INFO] APIVersion crd.projectcalico.org/v1 Kind NetworkSet
2021/08/08 12:48:59 [INFO] APIVersion crd.projectcalico.org/v1 Kind IPAMHandle
2021/08/08 12:48:59 [INFO] APIVersion crd.projectcalico.org/v1 Kind BGPPeer
2021/08/08 12:48:59 [INFO] APIVersion crd.projectcalico.org/v1 Kind GlobalNetworkSet
2021/08/08 12:48:59 [INFO] APIVersion crd.projectcalico.org/v1 Kind NetworkPolicy
2021/08/08 12:48:59 [INFO] APIVersion crd.projectcalico.org/v1 Kind IPAMBlock
2021/08/08 12:48:59 [INFO] APIVersion crd.projectcalico.org/v1 Kind GlobalNetworkPolicy
2021/08/08 12:48:59 [INFO] APIVersion crd.projectcalico.org/v1 Kind FelixConfiguration
2021/08/08 12:48:59 [INFO] APIVersion crd.projectcalico.org/v1 Kind HostEndpoint
2021/08/08 12:48:59 [INFO] APIVersion crd.projectcalico.org/v1 Kind IPPool
2021/08/08 12:48:59 [INFO] APIVersion crd.projectcalico.org/v1 Kind IPAMConfig
2021/08/08 12:48:59 [INFO] APIVersion crd.projectcalico.org/v1 Kind ClusterInformation
2021/08/08 12:48:59 [INFO] APIVersion crd.projectcalico.org/v1 Kind BlockAffinity
2021/08/08 12:48:59 [INFO] APIVersion crd.projectcalico.org/v1 Kind BGPConfiguration
2021/08/08 12:48:59 [INFO] APIVersion gitjob.cattle.io/v1 Kind GitJob
2021/08/08 12:48:59 [INFO] APIVersion monitoring.coreos.com/v1 Kind PrometheusRule
2021/08/08 12:48:59 [INFO] APIVersion monitoring.coreos.com/v1 Kind Prometheus
2021/08/08 12:48:59 [INFO] APIVersion monitoring.coreos.com/v1 Kind Alertmanager
2021/08/08 12:48:59 [INFO] APIVersion monitoring.coreos.com/v1 Kind ServiceMonitor
2021/08/08 12:48:59 [INFO] APIVersion rancher.cattle.io/v1 Kind Project
2021/08/08 12:48:59 [INFO] APIVersion rancher.cattle.io/v1 Kind RoleTemplate
2021/08/08 12:48:59 [INFO] APIVersion rancher.cattle.io/v1 Kind RoleTemplateBinding
2021/08/08 12:48:59 [INFO] APIVersion rancher.cattle.io/v1 Kind Cluster
2021/08/08 12:48:59 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind BundleDeployment
2021/08/08 12:48:59 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind BundleNamespaceMapping
2021/08/08 12:48:59 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind Bundle
2021/08/08 12:48:59 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind Cluster
2021/08/08 12:48:59 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind Content
2021/08/08 12:48:59 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind ClusterRegistration
2021/08/08 12:48:59 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind GitRepoRestriction
2021/08/08 12:48:59 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind ClusterRegistrationToken
2021/08/08 12:48:59 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind GitRepo
2021/08/08 12:48:59 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind ClusterGroup
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind MultiClusterAppRevision
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind RkeK8sSystemImage
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind EtcdBackup
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind GlobalRoleBinding
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind RoleTemplate
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind Cluster
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind Catalog
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind MultiClusterApp
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind DynamicSchema
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind MonitorMetric
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ClusterRoleTemplateBinding
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlertRule
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind Feature
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind Token
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlert
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind NodePool
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind Project
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind TemplateVersion
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind Preference
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ClusterTemplateRevision
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlert
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind RkeAddon
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ClusterMonitorGraph
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind Group
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind CatalogTemplate
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind PodSecurityPolicyTemplate
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ClusterScan
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind RkeK8sServiceOption
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind Setting
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ProjectNetworkPolicy
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind SamlToken
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ProjectLogging
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ClusterTemplate
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind KontainerDriver
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind GlobalRole
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind CisBenchmarkVersion
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlertGroup
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ClusterCatalog
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ProjectCatalog
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind TemplateContent
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind GroupMember
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind User
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind UserAttribute
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind FleetWorkspace
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind PodSecurityPolicyTemplateProjectBinding
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ClusterLogging
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlertGroup
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind Template
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ComposeConfig
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind CisConfig
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind NodeTemplate
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ClusterRegistrationToken
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind Notifier
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind AuthConfig
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind GlobalDnsProvider
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ProjectMonitorGraph
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlertRule
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ProjectRoleTemplateBinding
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind NodeDriver
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind Node
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind GlobalDns
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind CatalogTemplateVersion
2021/08/08 12:48:59 [INFO] APIVersion project.cattle.io/v3 Kind App
2021/08/08 12:48:59 [INFO] APIVersion project.cattle.io/v3 Kind PipelineSetting
2021/08/08 12:48:59 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeProviderConfig
2021/08/08 12:48:59 [INFO] APIVersion project.cattle.io/v3 Kind AppRevision
2021/08/08 12:48:59 [INFO] APIVersion project.cattle.io/v3 Kind Pipeline
2021/08/08 12:48:59 [INFO] APIVersion project.cattle.io/v3 Kind PipelineExecution
2021/08/08 12:48:59 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeRepository
2021/08/08 12:48:59 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeCredential
2021/08/08 12:48:59 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind NodeMetrics
2021/08/08 12:48:59 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind PodMetrics
2021/08/08 12:49:01 [INFO] Running in clustered mode with ID 10.42.0.39, monitoring endpoint cattle-system/rancher
2021/08/08 12:49:11 [INFO] Starting cluster controllers for local
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=ClusterRoleTemplateBinding controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=CatalogTemplate controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=ClusterTemplateRevision controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=Group controller
2021/08/08 12:49:15 [INFO] Starting project.cattle.io/v3, Kind=SourceCodeRepository controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=GlobalDns controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=ClusterRegistrationToken controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=GlobalRoleBinding controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=RkeK8sServiceOption controller
2021/08/08 12:49:15 [INFO] Starting project.cattle.io/v3, Kind=App controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=GroupMember controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=Node controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=PodSecurityPolicyTemplate controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=AuthConfig controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=KontainerDriver controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=Token controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=RoleTemplate controller
2021/08/08 12:49:15 [INFO] Starting project.cattle.io/v3, Kind=PipelineExecution controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=CatalogTemplateVersion controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=MultiClusterAppRevision controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=RkeAddon controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=ProjectCatalog controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=GlobalRole controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=NodePool controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=CisBenchmarkVersion controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=User controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=RkeK8sSystemImage controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=CisConfig controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=ClusterCatalog controller
2021/08/08 12:49:15 [INFO] Starting /v1, Kind=Endpoints controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=NodeTemplate controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=Project controller
2021/08/08 12:49:15 [INFO] Starting /v1, Kind=Namespace controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=DynamicSchema controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=ProjectLogging controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=ProjectRoleTemplateBinding controller
2021/08/08 12:49:16 [INFO] Starting project.cattle.io/v3, Kind=SourceCodeCredential controller
2021/08/08 12:49:16 [INFO] Starting management.cattle.io/v3, Kind=MultiClusterApp controller
2021/08/08 12:49:16 [INFO] Starting management.cattle.io/v3, Kind=UserAttribute controller
2021/08/08 12:49:16 [INFO] Starting management.cattle.io/v3, Kind=PodSecurityPolicyTemplateProjectBinding controller
2021/08/08 12:49:16 [INFO] Starting management.cattle.io/v3, Kind=NodeDriver controller
2021/08/08 12:49:16 [INFO] Starting management.cattle.io/v3, Kind=Catalog controller
2021/08/08 12:49:16 [INFO] Starting management.cattle.io/v3, Kind=ClusterTemplate controller
2021/08/08 12:49:16 [INFO] Starting project.cattle.io/v3, Kind=Pipeline controller
2021/08/08 12:49:20 [INFO] Starting cluster agent for local [owner=false]
2021/08/08 12:49:20 [INFO] Starting /v1, Kind=Pod controller
2021/08/08 12:49:20 [INFO] Starting /v1, Kind=Service controller
2021/08/08 12:49:20 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2021/08/08 12:49:20 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2021/08/08 12:49:20 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2021/08/08 12:49:20 [INFO] Starting /v1, Kind=Namespace controller
2021/08/08 12:49:20 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2021/08/08 12:49:20 [INFO] Starting /v1, Kind=Endpoints controller
2021/08/08 12:49:20 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplate
2021/08/08 12:49:22 [INFO] Watching metadata for rancher.cattle.io/v1, Kind=RoleTemplate
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=RoleTemplate
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=GroupMember
2021/08/08 12:49:22 [INFO] Watching metadata for apps/v1, Kind=Deployment
2021/08/08 12:49:22 [INFO] Watching metadata for acme.cert-manager.io/v1, Kind=Order
2021/08/08 12:49:22 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=BlockAffinity
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=DynamicSchema
2021/08/08 12:49:22 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterGroup
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlert
2021/08/08 12:49:22 [INFO] Watching metadata for /v1, Kind=ResourceQuota
2021/08/08 12:49:22 [INFO] Watching metadata for project.cattle.io/v3, Kind=PipelineSetting
2021/08/08 12:49:22 [INFO] Watching metadata for storage.k8s.io/v1, Kind=VolumeAttachment
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodeDriver
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=Cluster
2021/08/08 12:49:22 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=GitRepo
2021/08/08 12:49:22 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=Prometheus
2021/08/08 12:49:22 [INFO] Watching metadata for cert-manager.io/v1, Kind=CertificateRequest
2021/08/08 12:49:22 [INFO] Watching metadata for apps/v1, Kind=ReplicaSet
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectRoleTemplateBinding
2021/08/08 12:49:22 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=Alertmanager
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectMonitorGraph
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=Preference
2021/08/08 12:49:22 [INFO] Watching metadata for admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration
2021/08/08 12:49:22 [INFO] Watching metadata for /v1, Kind=LimitRange
2021/08/08 12:49:22 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=GitRepoRestriction
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=Setting
2021/08/08 12:49:22 [INFO] Watching metadata for networking.k8s.io/v1beta1, Kind=Ingress
2021/08/08 12:49:22 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=ClusterRepo
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sServiceOption
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=Feature
2021/08/08 12:49:22 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=FelixConfiguration
2021/08/08 12:49:22 [INFO] Watching metadata for batch/v1, Kind=Job
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=TemplateContent
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=FleetWorkspace
2021/08/08 12:49:22 [INFO] Watching metadata for /v1, Kind=ConfigMap
2021/08/08 12:49:22 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=BGPConfiguration
2021/08/08 12:49:22 [INFO] Watching metadata for /v1, Kind=Pod
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=Project
2021/08/08 12:49:22 [INFO] Watching metadata for storage.k8s.io/v1beta1, Kind=CSIDriver
2021/08/08 12:49:22 [INFO] Watching metadata for storage.k8s.io/v1, Kind=StorageClass
2021/08/08 12:49:22 [INFO] Watching metadata for apiregistration.k8s.io/v1, Kind=APIService
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=TemplateVersion
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplateVersion
2021/08/08 12:49:22 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=IPPool
2021/08/08 12:49:22 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=NetworkSet
2021/08/08 12:49:22 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Content
2021/08/08 12:49:22 [INFO] Watching metadata for /v1, Kind=Secret
2021/08/08 12:49:22 [INFO] Watching metadata for apiextensions.k8s.io/v1, Kind=CustomResourceDefinition
2021/08/08 12:49:22 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=ServiceMonitor
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlertRule
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterLogging
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRoleTemplateBinding
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=KontainerDriver
2021/08/08 12:49:23 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeCredential
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=MonitorMetric
2021/08/08 12:49:23 [INFO] Watching metadata for scheduling.k8s.io/v1, Kind=PriorityClass
2021/08/08 12:49:23 [INFO] Watching metadata for apps/v1, Kind=StatefulSet
2021/08/08 12:49:23 [INFO] Watching metadata for rancher.cattle.io/v1, Kind=Project
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalDns
2021/08/08 12:49:23 [INFO] Watching metadata for /v1, Kind=Node
2021/08/08 12:49:23 [INFO] Watching metadata for rancher.cattle.io/v1, Kind=RoleTemplateBinding
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalRole
2021/08/08 12:49:23 [INFO] Watching metadata for cert-manager.io/v1, Kind=ClusterIssuer
2021/08/08 12:49:23 [INFO] Watching metadata for batch/v1beta1, Kind=CronJob
2021/08/08 12:49:23 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=RoleBinding
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sSystemImage
2021/08/08 12:49:23 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=Role
2021/08/08 12:49:23 [INFO] Watching metadata for policy/v1beta1, Kind=PodSecurityPolicy
2021/08/08 12:49:23 [INFO] Watching metadata for coordination.k8s.io/v1, Kind=Lease
2021/08/08 12:49:23 [INFO] Watching metadata for /v1, Kind=PersistentVolume
2021/08/08 12:49:23 [INFO] Watching metadata for project.cattle.io/v3, Kind=PipelineExecution
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterTemplateRevision
2021/08/08 12:49:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=HostEndpoint
2021/08/08 12:49:23 [INFO] Watching metadata for acme.cert-manager.io/v1, Kind=Challenge
2021/08/08 12:49:23 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterRegistrationToken
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=Node
2021/08/08 12:49:23 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Cluster
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodeTemplate
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=Notifier
2021/08/08 12:49:23 [INFO] Watching metadata for apps/v1, Kind=ControllerRevision
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRegistrationToken
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=Template
2021/08/08 12:49:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=GlobalNetworkPolicy
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlertGroup
2021/08/08 12:49:23 [INFO] Watching metadata for rancher.cattle.io/v1, Kind=Cluster
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=PodSecurityPolicyTemplate
2021/08/08 12:49:23 [INFO] Watching metadata for storage.k8s.io/v1, Kind=CSINode
2021/08/08 12:49:23 [INFO] Watching metadata for cert-manager.io/v1, Kind=Issuer
2021/08/08 12:49:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=NetworkPolicy
2021/08/08 12:49:23 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterRegistration
2021/08/08 12:49:23 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=BundleDeployment
2021/08/08 12:49:23 [INFO] Watching metadata for /v1, Kind=Service
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeAddon
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterTemplate
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlertGroup
2021/08/08 12:49:23 [INFO] Watching metadata for networking.k8s.io/v1, Kind=NetworkPolicy
2021/08/08 12:49:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=GlobalNetworkSet
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=CisConfig
2021/08/08 12:49:23 [INFO] Watching metadata for node.k8s.io/v1beta1, Kind=RuntimeClass
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalRoleBinding
2021/08/08 12:49:23 [INFO] Watching metadata for /v1, Kind=PodTemplate
2021/08/08 12:49:23 [INFO] Watching metadata for admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration
2021/08/08 12:49:23 [INFO] Watching metadata for /v1, Kind=ReplicationController
2021/08/08 12:49:23 [INFO] Watching metadata for discovery.k8s.io/v1beta1, Kind=EndpointSlice
2021/08/08 12:49:23 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=BundleNamespaceMapping
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=MultiClusterAppRevision
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=MultiClusterApp
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=Group
2021/08/08 12:49:23 [INFO] Watching metadata for certificates.k8s.io/v1beta1, Kind=CertificateSigningRequest
2021/08/08 12:49:23 [INFO] Watching metadata for /v1, Kind=Namespace
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=CisBenchmarkVersion
2021/08/08 12:49:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=ClusterInformation
2021/08/08 12:49:23 [INFO] Watching metadata for /v1, Kind=Event
2021/08/08 12:49:23 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=App
2021/08/08 12:49:23 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=PrometheusRule
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalDnsProvider
2021/08/08 12:49:23 [INFO] Watching metadata for policy/v1beta1, Kind=PodDisruptionBudget
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=UserAttribute
2021/08/08 12:49:23 [INFO] Watching metadata for project.cattle.io/v3, Kind=AppRevision
2021/08/08 12:49:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=IPAMConfig
2021/08/08 12:49:23 [INFO] Watching metadata for /v1, Kind=Endpoints
2021/08/08 12:49:23 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeProviderConfig
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=Catalog
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ComposeConfig
2021/08/08 12:49:23 [INFO] Watching metadata for /v1, Kind=PersistentVolumeClaim
2021/08/08 12:49:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=IPAMHandle
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodePool
2021/08/08 12:49:23 [INFO] Watching metadata for apps/v1, Kind=DaemonSet
2021/08/08 12:49:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=IPAMBlock
2021/08/08 12:49:23 [INFO] Watching metadata for events.k8s.io/v1beta1, Kind=Event
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=User
2021/08/08 12:49:23 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRole
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=SamlToken
2021/08/08 12:49:23 [INFO] Watching metadata for project.cattle.io/v3, Kind=App
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectCatalog
2021/08/08 12:49:23 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding
2021/08/08 12:49:23 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeRepository
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=PodSecurityPolicyTemplateProjectBinding
2021/08/08 12:49:23 [INFO] Watching metadata for /v1, Kind=ServiceAccount
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterCatalog
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectLogging
2021/08/08 12:49:23 [INFO] Watching metadata for autoscaling/v2beta2, Kind=HorizontalPodAutoscaler
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterMonitorGraph
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=Token
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=AuthConfig
2021/08/08 12:49:23 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Bundle
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlertRule
2021/08/08 12:49:23 [INFO] Watching metadata for gitjob.cattle.io/v1, Kind=GitJob
2021/08/08 12:49:23 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=Operation
2021/08/08 12:49:23 [INFO] Watching metadata for project.cattle.io/v3, Kind=Pipeline
2021/08/08 12:49:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=BGPPeer
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=EtcdBackup
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterScan
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlert
2021/08/08 12:49:23 [INFO] Watching metadata for cert-manager.io/v1, Kind=Certificate
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectNetworkPolicy
2021/08/08 12:50:04 [INFO] Listening on :443
2021/08/08 12:50:04 [INFO] Listening on :80
2021/08/08 12:50:04 [INFO] Listening on :444
2021/08/08 12:50:04 [INFO] Updating TLS secret for serving-cert (count: 8): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.39:10.42.0.39 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:977d70d9405a56ac2287891f4372a5623e689d54cc67c12a4413af9182c58173]
2021/08/08 12:50:05 [INFO] Active TLS secret serving-cert (ver=36375) (count 8): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:429e226403a5b18ac214ac57165b349fcd12a069caef133d72f269769e175315]
2021/08/08 12:50:05 [INFO] Starting /v1, Kind=Secret controller
2021/08/08 12:50:05 [INFO] Active TLS secret serving-cert (ver=36375) (count 8): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:429e226403a5b18ac214ac57165b349fcd12a069caef133d72f269769e175315]
2021/08/08 12:50:07 [INFO] Active TLS secret serving-cert (ver=36388) (count 9): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.34:10.42.0.34 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:cbb8784b086c09c2d80b5a2c202cd7e584e513ad03f6a216463a80a2e76d0558]
2021/08/08 12:50:18 [INFO] Adding peer wss://10.42.0.33/v3/connect, 10.42.0.33
2021/08/08 12:50:19 [ERROR] Failed to connect to peer wss://10.42.0.33/v3/connect [local ID=10.42.0.39]: websocket: bad handshake
2021/08/08 12:50:24 [ERROR] Failed to connect to peer wss://10.42.0.33/v3/connect [local ID=10.42.0.39]: websocket: bad handshake
2021/08/08 12:50:24 [INFO] Adding peer wss://10.42.0.34/v3/connect, 10.42.0.34
2021/08/08 12:50:24 [ERROR] Failed to connect to peer wss://10.42.0.34/v3/connect [local ID=10.42.0.39]: websocket: bad handshake
2021/08/08 12:50:29 [ERROR] Failed to connect to peer wss://10.42.0.33/v3/connect [local ID=10.42.0.39]: websocket: bad handshake
2021/08/08 12:50:29 [ERROR] Failed to connect to peer wss://10.42.0.34/v3/connect [local ID=10.42.0.39]: websocket: bad handshake
2021/08/08 12:50:30 [INFO] Updating TLS secret for serving-cert (count: 10): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.34:10.42.0.34 listener.cattle.io/cn-10.42.0.39:10.42.0.39 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:886488085d31e3ebc49f83d909f43c692a45719f4293c566b41e72dd78dcf92c]
2021/08/08 12:50:30 [INFO] Active TLS secret serving-cert (ver=36529) (count 10): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.34:10.42.0.34 listener.cattle.io/cn-10.42.0.39:10.42.0.39 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:886488085d31e3ebc49f83d909f43c692a45719f4293c566b41e72dd78dcf92c]
2021/08/08 12:50:30 [INFO] Active TLS secret serving-cert (ver=36529) (count 10): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.34:10.42.0.34 listener.cattle.io/cn-10.42.0.39:10.42.0.39 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:886488085d31e3ebc49f83d909f43c692a45719f4293c566b41e72dd78dcf92c]
2021/08/08 12:50:31 [INFO] Handling backend connection request [10.42.0.34]
2021/08/08 12:50:31 [INFO] Handling backend connection request [10.42.0.33]

```

and for **rancher-cbbdf9fd5-pmz5b**

```auto
$ kubectl -n cattle-system logs rancher-cbbdf9fd5-pmz5b
2021/08/08 12:48:48 [INFO] Rancher version v2.5.9 (3c5418944) is starting
2021/08/08 12:48:48 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:true AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Agent:false Features: ClusterRegistry:}
2021/08/08 12:48:48 [INFO] Listening on /tmp/log.sock
2021/08/08 12:49:00 [INFO] Starting API controllers
2021/08/08 12:49:01 [INFO] Starting management.cattle.io/v3, Kind=UserAttribute controller
2021/08/08 12:49:01 [INFO] Starting management.cattle.io/v3, Kind=Group controller
2021/08/08 12:49:01 [INFO] Starting management.cattle.io/v3, Kind=User controller
2021/08/08 12:49:01 [INFO] Starting management.cattle.io/v3, Kind=GroupMember controller
2021/08/08 12:49:01 [INFO] Starting management.cattle.io/v3, Kind=Token controller
2021/08/08 12:49:01 [INFO] Starting /v1, Kind=Secret controller
2021/08/08 12:49:03 [INFO] Starting management.cattle.io/v3, Kind=Preference controller
2021/08/08 12:49:03 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2021/08/08 12:49:03 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2021/08/08 12:49:03 [INFO] Starting management.cattle.io/v3, Kind=Cluster controller
I0808 12:49:03.891804 7 leaderelection.go:243] attempting to acquire leader lease kube-system/cattle-controllers...
2021/08/08 12:49:03 [INFO] Starting apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition controller
2021/08/08 12:49:03 [INFO] Starting /v1, Kind=ConfigMap controller
2021/08/08 12:49:03 [INFO] Starting catalog.cattle.io/v1, Kind=ClusterRepo controller
2021/08/08 12:49:03 [INFO] Starting management.cattle.io/v3, Kind=Feature controller
2021/08/08 12:49:03 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2021/08/08 12:49:03 [INFO] Starting management.cattle.io/v3, Kind=Setting controller
2021/08/08 12:49:03 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2021/08/08 12:49:03 [INFO] Starting /v1, Kind=Secret controller
2021/08/08 12:49:03 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
I0808 12:49:04.048828 7 leaderelection.go:253] successfully acquired lease kube-system/cattle-controllers
2021/08/08 12:49:04 [INFO] Refreshing all schemas
2021/08/08 12:49:06 [INFO] Steve auth startup complete
2021/08/08 12:49:09 [INFO] Refreshing all schemas
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind Binding
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind ComponentStatus
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind ConfigMap
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind Endpoints
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind Event
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind LimitRange
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind Namespace
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind Node
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind PersistentVolumeClaim
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind PersistentVolume
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind Pod
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind PodTemplate
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind ReplicationController
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind ResourceQuota
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind Secret
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind ServiceAccount
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind Service
2021/08/08 12:49:09 [INFO] APIVersion apiregistration.k8s.io/v1 Kind APIService
2021/08/08 12:49:09 [INFO] APIVersion apiregistration.k8s.io/v1beta1 Kind APIService
2021/08/08 12:49:09 [INFO] APIVersion extensions/v1beta1 Kind Ingress
2021/08/08 12:49:09 [INFO] APIVersion apps/v1 Kind ControllerRevision
2021/08/08 12:49:09 [INFO] APIVersion apps/v1 Kind DaemonSet
2021/08/08 12:49:09 [INFO] APIVersion apps/v1 Kind Deployment
2021/08/08 12:49:09 [INFO] APIVersion apps/v1 Kind ReplicaSet
2021/08/08 12:49:09 [INFO] APIVersion apps/v1 Kind StatefulSet
2021/08/08 12:49:09 [INFO] APIVersion events.k8s.io/v1beta1 Kind Event
2021/08/08 12:49:09 [INFO] APIVersion authentication.k8s.io/v1 Kind TokenReview
2021/08/08 12:49:09 [INFO] APIVersion authentication.k8s.io/v1beta1 Kind TokenReview
2021/08/08 12:49:09 [INFO] APIVersion authorization.k8s.io/v1 Kind LocalSubjectAccessReview
2021/08/08 12:49:09 [INFO] APIVersion authorization.k8s.io/v1 Kind SelfSubjectAccessReview
2021/08/08 12:49:09 [INFO] APIVersion authorization.k8s.io/v1 Kind SelfSubjectRulesReview
2021/08/08 12:49:09 [INFO] APIVersion authorization.k8s.io/v1 Kind SubjectAccessReview
2021/08/08 12:49:09 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind LocalSubjectAccessReview
2021/08/08 12:49:09 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SelfSubjectAccessReview
2021/08/08 12:49:09 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SelfSubjectRulesReview
2021/08/08 12:49:09 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SubjectAccessReview
2021/08/08 12:49:09 [INFO] APIVersion autoscaling/v1 Kind HorizontalPodAutoscaler
2021/08/08 12:49:09 [INFO] APIVersion autoscaling/v2beta1 Kind HorizontalPodAutoscaler
2021/08/08 12:49:09 [INFO] APIVersion autoscaling/v2beta2 Kind HorizontalPodAutoscaler
2021/08/08 12:49:09 [INFO] APIVersion batch/v1 Kind Job
2021/08/08 12:49:09 [INFO] APIVersion batch/v1beta1 Kind CronJob
2021/08/08 12:49:09 [INFO] APIVersion certificates.k8s.io/v1beta1 Kind CertificateSigningRequest
2021/08/08 12:49:09 [INFO] APIVersion networking.k8s.io/v1 Kind NetworkPolicy
2021/08/08 12:49:09 [INFO] APIVersion networking.k8s.io/v1beta1 Kind Ingress
2021/08/08 12:49:09 [INFO] APIVersion policy/v1beta1 Kind PodDisruptionBudget
2021/08/08 12:49:09 [INFO] APIVersion policy/v1beta1 Kind PodSecurityPolicy
2021/08/08 12:49:09 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind ClusterRoleBinding
2021/08/08 12:49:09 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind ClusterRole
2021/08/08 12:49:09 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind RoleBinding
2021/08/08 12:49:09 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind Role
2021/08/08 12:49:09 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind ClusterRoleBinding
2021/08/08 12:49:09 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind ClusterRole
2021/08/08 12:49:09 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind RoleBinding
2021/08/08 12:49:09 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind Role
2021/08/08 12:49:09 [INFO] APIVersion storage.k8s.io/v1 Kind CSINode
2021/08/08 12:49:09 [INFO] APIVersion storage.k8s.io/v1 Kind StorageClass
2021/08/08 12:49:09 [INFO] APIVersion storage.k8s.io/v1 Kind VolumeAttachment
2021/08/08 12:49:09 [INFO] APIVersion storage.k8s.io/v1beta1 Kind CSIDriver
2021/08/08 12:49:09 [INFO] APIVersion storage.k8s.io/v1beta1 Kind CSINode
2021/08/08 12:49:09 [INFO] APIVersion storage.k8s.io/v1beta1 Kind StorageClass
2021/08/08 12:49:09 [INFO] APIVersion storage.k8s.io/v1beta1 Kind VolumeAttachment
2021/08/08 12:49:09 [INFO] APIVersion admissionregistration.k8s.io/v1 Kind MutatingWebhookConfiguration
2021/08/08 12:49:09 [INFO] APIVersion admissionregistration.k8s.io/v1 Kind ValidatingWebhookConfiguration
2021/08/08 12:49:09 [INFO] APIVersion admissionregistration.k8s.io/v1beta1 Kind MutatingWebhookConfiguration
2021/08/08 12:49:09 [INFO] APIVersion admissionregistration.k8s.io/v1beta1 Kind ValidatingWebhookConfiguration
2021/08/08 12:49:09 [INFO] APIVersion apiextensions.k8s.io/v1 Kind CustomResourceDefinition
2021/08/08 12:49:09 [INFO] APIVersion apiextensions.k8s.io/v1beta1 Kind CustomResourceDefinition
2021/08/08 12:49:09 [INFO] APIVersion scheduling.k8s.io/v1 Kind PriorityClass
2021/08/08 12:49:09 [INFO] APIVersion scheduling.k8s.io/v1beta1 Kind PriorityClass
2021/08/08 12:49:09 [INFO] APIVersion coordination.k8s.io/v1 Kind Lease
2021/08/08 12:49:09 [INFO] APIVersion coordination.k8s.io/v1beta1 Kind Lease
2021/08/08 12:49:09 [INFO] APIVersion node.k8s.io/v1beta1 Kind RuntimeClass
2021/08/08 12:49:09 [INFO] APIVersion discovery.k8s.io/v1beta1 Kind EndpointSlice
2021/08/08 12:49:09 [INFO] APIVersion acme.cert-manager.io/v1 Kind Challenge
2021/08/08 12:49:09 [INFO] APIVersion acme.cert-manager.io/v1 Kind Order
2021/08/08 12:49:09 [INFO] APIVersion acme.cert-manager.io/v1beta1 Kind Order
2021/08/08 12:49:09 [INFO] APIVersion acme.cert-manager.io/v1beta1 Kind Challenge
2021/08/08 12:49:09 [INFO] APIVersion acme.cert-manager.io/v1alpha3 Kind Order
2021/08/08 12:49:09 [INFO] APIVersion acme.cert-manager.io/v1alpha3 Kind Challenge
2021/08/08 12:49:09 [INFO] APIVersion acme.cert-manager.io/v1alpha2 Kind Challenge
2021/08/08 12:49:09 [INFO] APIVersion acme.cert-manager.io/v1alpha2 Kind Order
2021/08/08 12:49:09 [INFO] APIVersion catalog.cattle.io/v1 Kind ClusterRepo
2021/08/08 12:49:09 [INFO] APIVersion catalog.cattle.io/v1 Kind App
2021/08/08 12:49:09 [INFO] APIVersion catalog.cattle.io/v1 Kind Operation
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1 Kind ClusterIssuer
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1 Kind CertificateRequest
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1 Kind Issuer
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1 Kind Certificate
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1beta1 Kind CertificateRequest
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1beta1 Kind Issuer
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1beta1 Kind Certificate
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1beta1 Kind ClusterIssuer
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1alpha3 Kind Issuer
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1alpha3 Kind Certificate
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1alpha3 Kind ClusterIssuer
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1alpha3 Kind CertificateRequest
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1alpha2 Kind Certificate
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1alpha2 Kind ClusterIssuer
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1alpha2 Kind CertificateRequest
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1alpha2 Kind Issuer
2021/08/08 12:49:09 [INFO] APIVersion crd.projectcalico.org/v1 Kind NetworkSet
2021/08/08 12:49:09 [INFO] APIVersion crd.projectcalico.org/v1 Kind IPAMHandle
2021/08/08 12:49:09 [INFO] APIVersion crd.projectcalico.org/v1 Kind BGPPeer
2021/08/08 12:49:09 [INFO] APIVersion crd.projectcalico.org/v1 Kind GlobalNetworkSet
2021/08/08 12:49:09 [INFO] APIVersion crd.projectcalico.org/v1 Kind NetworkPolicy
2021/08/08 12:49:09 [INFO] APIVersion crd.projectcalico.org/v1 Kind IPAMBlock
2021/08/08 12:49:09 [INFO] APIVersion crd.projectcalico.org/v1 Kind GlobalNetworkPolicy
2021/08/08 12:49:09 [INFO] APIVersion crd.projectcalico.org/v1 Kind FelixConfiguration
2021/08/08 12:49:09 [INFO] APIVersion crd.projectcalico.org/v1 Kind HostEndpoint
2021/08/08 12:49:09 [INFO] APIVersion crd.projectcalico.org/v1 Kind IPPool
2021/08/08 12:49:09 [INFO] APIVersion crd.projectcalico.org/v1 Kind IPAMConfig
2021/08/08 12:49:09 [INFO] APIVersion crd.projectcalico.org/v1 Kind ClusterInformation
2021/08/08 12:49:09 [INFO] APIVersion crd.projectcalico.org/v1 Kind BlockAffinity
2021/08/08 12:49:09 [INFO] APIVersion crd.projectcalico.org/v1 Kind BGPConfiguration
2021/08/08 12:49:09 [INFO] APIVersion gitjob.cattle.io/v1 Kind GitJob
2021/08/08 12:49:09 [INFO] APIVersion monitoring.coreos.com/v1 Kind PrometheusRule
2021/08/08 12:49:09 [INFO] APIVersion monitoring.coreos.com/v1 Kind Prometheus
2021/08/08 12:49:09 [INFO] APIVersion monitoring.coreos.com/v1 Kind Alertmanager
2021/08/08 12:49:09 [INFO] APIVersion monitoring.coreos.com/v1 Kind ServiceMonitor
2021/08/08 12:49:09 [INFO] APIVersion rancher.cattle.io/v1 Kind Project
2021/08/08 12:49:09 [INFO] APIVersion rancher.cattle.io/v1 Kind RoleTemplate
2021/08/08 12:49:09 [INFO] APIVersion rancher.cattle.io/v1 Kind RoleTemplateBinding
2021/08/08 12:49:09 [INFO] APIVersion rancher.cattle.io/v1 Kind Cluster
2021/08/08 12:49:09 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind BundleDeployment
2021/08/08 12:49:09 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind BundleNamespaceMapping
2021/08/08 12:49:09 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind Bundle
2021/08/08 12:49:09 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind Cluster
2021/08/08 12:49:09 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind Content
2021/08/08 12:49:09 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind ClusterRegistration
2021/08/08 12:49:09 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind GitRepoRestriction
2021/08/08 12:49:09 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind ClusterRegistrationToken
2021/08/08 12:49:09 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind GitRepo
2021/08/08 12:49:09 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind ClusterGroup
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind MultiClusterAppRevision
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind RkeK8sSystemImage
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind EtcdBackup
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind GlobalRoleBinding
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind RoleTemplate
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind Cluster
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind Catalog
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind MultiClusterApp
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind DynamicSchema
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind MonitorMetric
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ClusterRoleTemplateBinding
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlertRule
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind Feature
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind Token
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlert
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind NodePool
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind Project
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind TemplateVersion
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind Preference
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ClusterTemplateRevision
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlert
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind RkeAddon
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ClusterMonitorGraph
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind Group
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind CatalogTemplate
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind PodSecurityPolicyTemplate
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ClusterScan
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind RkeK8sServiceOption
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind Setting
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ProjectNetworkPolicy
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind SamlToken
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ProjectLogging
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ClusterTemplate
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind KontainerDriver
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind GlobalRole
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind CisBenchmarkVersion
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlertGroup
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ClusterCatalog
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ProjectCatalog
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind TemplateContent
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind GroupMember
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind User
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind UserAttribute
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind FleetWorkspace
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind PodSecurityPolicyTemplateProjectBinding
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ClusterLogging
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlertGroup
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind Template
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ComposeConfig
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind CisConfig
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind NodeTemplate
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ClusterRegistrationToken
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind Notifier
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind AuthConfig
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind GlobalDnsProvider
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ProjectMonitorGraph
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlertRule
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ProjectRoleTemplateBinding
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind NodeDriver
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind Node
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind GlobalDns
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind CatalogTemplateVersion
2021/08/08 12:49:09 [INFO] APIVersion project.cattle.io/v3 Kind App
2021/08/08 12:49:09 [INFO] APIVersion project.cattle.io/v3 Kind PipelineSetting
2021/08/08 12:49:09 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeProviderConfig
2021/08/08 12:49:09 [INFO] APIVersion project.cattle.io/v3 Kind AppRevision
2021/08/08 12:49:09 [INFO] APIVersion project.cattle.io/v3 Kind Pipeline
2021/08/08 12:49:09 [INFO] APIVersion project.cattle.io/v3 Kind PipelineExecution
2021/08/08 12:49:09 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeRepository
2021/08/08 12:49:09 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeCredential
2021/08/08 12:49:09 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind NodeMetrics
2021/08/08 12:49:09 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind PodMetrics
2021/08/08 12:49:11 [INFO] Running in clustered mode with ID 10.42.0.33, monitoring endpoint cattle-system/rancher
2021/08/08 12:49:35 [INFO] Starting cluster controllers for local
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=NodeTemplate controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=NodePool controller
2021/08/08 12:49:39 [INFO] Starting project.cattle.io/v3, Kind=PipelineExecution controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=ClusterRegistrationToken controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=PodSecurityPolicyTemplateProjectBinding controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=DynamicSchema controller
2021/08/08 12:49:39 [INFO] Starting project.cattle.io/v3, Kind=Pipeline controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=CisConfig controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=MultiClusterApp controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=Token controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=ClusterTemplateRevision controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=ProjectCatalog controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=CatalogTemplate controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=ProjectRoleTemplateBinding controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=NodeDriver controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=AuthConfig controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=Node controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=KontainerDriver controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=ClusterRoleTemplateBinding controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=GlobalRoleBinding controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=GroupMember controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=UserAttribute controller
2021/08/08 12:49:39 [INFO] Starting project.cattle.io/v3, Kind=SourceCodeRepository controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=ClusterCatalog controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=GlobalRole controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=Project controller
2021/08/08 12:49:39 [INFO] Starting project.cattle.io/v3, Kind=SourceCodeCredential controller
2021/08/08 12:49:39 [INFO] Starting /v1, Kind=Endpoints controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=User controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=Catalog controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=RoleTemplate controller
2021/08/08 12:49:39 [INFO] Starting project.cattle.io/v3, Kind=App controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=RkeK8sServiceOption controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=RkeK8sSystemImage controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=ClusterTemplate controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=ProjectLogging controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=CisBenchmarkVersion controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=PodSecurityPolicyTemplate controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=MultiClusterAppRevision controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=RkeAddon controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=CatalogTemplateVersion controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=GlobalDns controller
2021/08/08 12:49:39 [INFO] Starting /v1, Kind=Namespace controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=Group controller
2021/08/08 12:49:41 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=FelixConfiguration
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=EtcdBackup
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterLogging
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sServiceOption
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectNetworkPolicy
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=Template
2021/08/08 12:49:41 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeCredential
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplate
2021/08/08 12:49:41 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Bundle
2021/08/08 12:49:41 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=NetworkSet
2021/08/08 12:49:41 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=ClusterInformation
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=Cluster
2021/08/08 12:49:41 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeRepository
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=Node
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=Catalog
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=ConfigMap
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=DynamicSchema
2021/08/08 12:49:41 [INFO] Watching metadata for apps/v1, Kind=ControllerRevision
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=ServiceAccount
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=MonitorMetric
2021/08/08 12:49:41 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=Operation
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=PodTemplate
2021/08/08 12:49:41 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=IPAMBlock
2021/08/08 12:49:41 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=Prometheus
2021/08/08 12:49:41 [INFO] Watching metadata for admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration
2021/08/08 12:49:41 [INFO] Watching metadata for policy/v1beta1, Kind=PodSecurityPolicy
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=SamlToken
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=Preference
2021/08/08 12:49:41 [INFO] Watching metadata for storage.k8s.io/v1beta1, Kind=CSIDriver
2021/08/08 12:49:41 [INFO] Watching metadata for cert-manager.io/v1, Kind=Issuer
2021/08/08 12:49:41 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=PrometheusRule
2021/08/08 12:49:41 [INFO] Watching metadata for rancher.cattle.io/v1, Kind=Cluster
2021/08/08 12:49:41 [INFO] Watching metadata for certificates.k8s.io/v1beta1, Kind=CertificateSigningRequest
2021/08/08 12:49:41 [INFO] Watching metadata for rancher.cattle.io/v1, Kind=Project
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=MultiClusterApp
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=PersistentVolume
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectLogging
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=RoleTemplate
2021/08/08 12:49:41 [INFO] Watching metadata for networking.k8s.io/v1beta1, Kind=Ingress
2021/08/08 12:49:41 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=IPPool
2021/08/08 12:49:41 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=ClusterRepo
2021/08/08 12:49:41 [INFO] Watching metadata for acme.cert-manager.io/v1, Kind=Order
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectCatalog
2021/08/08 12:49:41 [INFO] Watching metadata for cert-manager.io/v1, Kind=CertificateRequest
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectMonitorGraph
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=FleetWorkspace
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlert
2021/08/08 12:49:41 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=BundleNamespaceMapping
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalDnsProvider
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=LimitRange
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=AuthConfig
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodePool
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalRole
2021/08/08 12:49:41 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=BGPPeer
2021/08/08 12:49:41 [INFO] Watching metadata for apps/v1, Kind=DaemonSet
2021/08/08 12:49:41 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=IPAMHandle
2021/08/08 12:49:41 [INFO] Watching metadata for batch/v1beta1, Kind=CronJob
2021/08/08 12:49:41 [INFO] Watching metadata for batch/v1, Kind=Job
2021/08/08 12:49:41 [INFO] Watching metadata for admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration
2021/08/08 12:49:41 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=GlobalNetworkSet
2021/08/08 12:49:41 [INFO] Watching metadata for storage.k8s.io/v1, Kind=CSINode
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=Service
2021/08/08 12:49:41 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Cluster
2021/08/08 12:49:41 [INFO] Watching metadata for rancher.cattle.io/v1, Kind=RoleTemplateBinding
2021/08/08 12:49:41 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=BlockAffinity
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=MultiClusterAppRevision
2021/08/08 12:49:41 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRole
2021/08/08 12:49:41 [INFO] Watching metadata for cert-manager.io/v1, Kind=ClusterIssuer
2021/08/08 12:49:41 [INFO] Watching metadata for networking.k8s.io/v1, Kind=NetworkPolicy
2021/08/08 12:49:41 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterGroup
2021/08/08 12:49:41 [INFO] Watching metadata for apps/v1, Kind=Deployment
2021/08/08 12:49:41 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=IPAMConfig
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterCatalog
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=KontainerDriver
2021/08/08 12:49:41 [INFO] Watching metadata for coordination.k8s.io/v1, Kind=Lease
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterTemplate
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlertGroup
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=User
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalDns
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlert
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodeTemplate
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodeDriver
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterScan
2021/08/08 12:49:41 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=NetworkPolicy
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplateVersion
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=Node
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlertGroup
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=PersistentVolumeClaim
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=UserAttribute
2021/08/08 12:49:41 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=Role
2021/08/08 12:49:41 [INFO] Watching metadata for project.cattle.io/v3, Kind=PipelineSetting
2021/08/08 12:49:41 [INFO] Watching metadata for apps/v1, Kind=ReplicaSet
2021/08/08 12:49:41 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=BundleDeployment
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=Endpoints
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=Feature
2021/08/08 12:49:41 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterRegistrationToken
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ComposeConfig
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=PodSecurityPolicyTemplate
2021/08/08 12:49:41 [INFO] Watching metadata for gitjob.cattle.io/v1, Kind=GitJob
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=ResourceQuota
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=TemplateContent
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=CisConfig
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=TemplateVersion
2021/08/08 12:49:41 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=BGPConfiguration
2021/08/08 12:49:41 [INFO] Watching metadata for policy/v1beta1, Kind=PodDisruptionBudget
2021/08/08 12:49:41 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=ServiceMonitor
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sSystemImage
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeAddon
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlertRule
2021/08/08 12:49:41 [INFO] Watching metadata for scheduling.k8s.io/v1, Kind=PriorityClass
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=Secret
2021/08/08 12:49:41 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=GitRepoRestriction
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=Event
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=GroupMember
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalRoleBinding
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=CisBenchmarkVersion
2021/08/08 12:49:41 [INFO] Watching metadata for autoscaling/v2beta2, Kind=HorizontalPodAutoscaler
2021/08/08 12:49:41 [INFO] Watching metadata for project.cattle.io/v3, Kind=Pipeline
2021/08/08 12:49:41 [INFO] Watching metadata for project.cattle.io/v3, Kind=PipelineExecution
2021/08/08 12:49:41 [INFO] Watching metadata for acme.cert-manager.io/v1, Kind=Challenge
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterMonitorGraph
2021/08/08 12:49:41 [INFO] Watching metadata for storage.k8s.io/v1, Kind=VolumeAttachment
2021/08/08 12:49:41 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Content
2021/08/08 12:49:41 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterRegistration
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=Notifier
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=Namespace
2021/08/08 12:49:41 [INFO] Watching metadata for events.k8s.io/v1beta1, Kind=Event
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=Token
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterTemplateRevision
2021/08/08 12:49:41 [INFO] Watching metadata for rancher.cattle.io/v1, Kind=RoleTemplate
2021/08/08 12:49:41 [INFO] Watching metadata for storage.k8s.io/v1, Kind=StorageClass
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=Pod
2021/08/08 12:49:41 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=GitRepo
2021/08/08 12:49:41 [INFO] Watching metadata for project.cattle.io/v3, Kind=AppRevision
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=Setting
2021/08/08 12:49:41 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=Alertmanager
2021/08/08 12:49:41 [INFO] Watching metadata for discovery.k8s.io/v1beta1, Kind=EndpointSlice
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=ReplicationController
2021/08/08 12:49:41 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding
2021/08/08 12:49:41 [INFO] Watching metadata for node.k8s.io/v1beta1, Kind=RuntimeClass
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRoleTemplateBinding
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRegistrationToken
2021/08/08 12:49:41 [INFO] Watching metadata for apiregistration.k8s.io/v1, Kind=APIService
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=Project
2021/08/08 12:49:41 [INFO] Watching metadata for project.cattle.io/v3, Kind=App
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectRoleTemplateBinding
2021/08/08 12:49:41 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=App
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=PodSecurityPolicyTemplateProjectBinding
2021/08/08 12:49:41 [INFO] Watching metadata for apiextensions.k8s.io/v1, Kind=CustomResourceDefinition
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=Group
2021/08/08 12:49:41 [INFO] Watching metadata for cert-manager.io/v1, Kind=Certificate
2021/08/08 12:49:41 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeProviderConfig
2021/08/08 12:49:41 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=RoleBinding
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlertRule
2021/08/08 12:49:41 [INFO] Watching metadata for apps/v1, Kind=StatefulSet
2021/08/08 12:49:41 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=GlobalNetworkPolicy
2021/08/08 12:49:41 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=HostEndpoint
2021/08/08 12:49:50 [INFO] Starting cluster agent for local [owner=false]
2021/08/08 12:49:50 [INFO] Starting /v1, Kind=Namespace controller
2021/08/08 12:49:50 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2021/08/08 12:49:50 [INFO] Starting /v1, Kind=Endpoints controller
2021/08/08 12:49:50 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2021/08/08 12:49:50 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2021/08/08 12:49:50 [INFO] Starting /v1, Kind=Pod controller
2021/08/08 12:49:50 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2021/08/08 12:49:50 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2021/08/08 12:49:50 [INFO] Starting /v1, Kind=Service controller
2021/08/08 12:50:04 [INFO] Reconciling GlobalRoles
2021/08/08 12:50:04 [INFO] Reconciling RoleTemplates
2021/08/08 12:50:04 [INFO] Listening on :443
2021/08/08 12:50:04 [INFO] Listening on :80
2021/08/08 12:50:04 [INFO] Listening on :444
2021/08/08 12:50:04 [INFO] Updating TLS secret for serving-cert (count: 8): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:429e226403a5b18ac214ac57165b349fcd12a069caef133d72f269769e175315]
2021/08/08 12:50:04 [INFO] Active TLS secret serving-cert (ver=36375) (count 8): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:429e226403a5b18ac214ac57165b349fcd12a069caef133d72f269769e175315]
2021/08/08 12:50:05 [INFO] Starting /v1, Kind=Secret controller
2021/08/08 12:50:05 [INFO] Active TLS secret serving-cert (ver=36375) (count 8): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:429e226403a5b18ac214ac57165b349fcd12a069caef133d72f269769e175315]
2021/08/08 12:50:05 [INFO] adding kontainer driver rancherKubernetesEngine
2021/08/08 12:50:05 [INFO] adding kontainer driver googleKubernetesEngine
2021/08/08 12:50:05 [INFO] adding kontainer driver azureKubernetesService
2021/08/08 12:50:05 [INFO] adding kontainer driver amazonElasticContainerService
2021/08/08 12:50:05 [INFO] adding kontainer driver baiducloudcontainerengine
2021/08/08 12:50:05 [INFO] adding kontainer driver aliyunkubernetescontainerservice
2021/08/08 12:50:05 [INFO] adding kontainer driver tencentkubernetesengine
2021/08/08 12:50:05 [INFO] adding kontainer driver huaweicontainercloudengine
2021/08/08 12:50:05 [INFO] adding kontainer driver oraclecontainerengine
2021/08/08 12:50:05 [INFO] adding kontainer driver linodekubernetesengine
2021/08/08 12:50:05 [INFO] adding kontainer driver opentelekomcloudcontainerengine
2021/08/08 12:50:05 [INFO] Starting catalog controller
2021/08/08 12:50:05 [INFO] Starting project-level catalog controller
2021/08/08 12:50:05 [INFO] Starting cluster-level catalog controller
2021/08/08 12:50:06 [INFO] Refreshing driverMetadata in 1440 minutes
2021/08/08 12:50:06 [INFO] update kontainerdriver opentelekomcloudcontainerengine
2021/08/08 12:50:06 [INFO] update kontainerdriver oraclecontainerengine
2021/08/08 12:50:06 [INFO] update kontainerdriver rancherkubernetesengine
2021/08/08 12:50:06 [INFO] update kontainerdriver amazonelasticcontainerservice
2021/08/08 12:50:06 [INFO] update kontainerdriver baiducloudcontainerengine
2021/08/08 12:50:06 [INFO] update kontainerdriver googlekubernetesengine
2021/08/08 12:50:06 [INFO] update kontainerdriver linodekubernetesengine
2021/08/08 12:50:06 [INFO] update kontainerdriver tencentkubernetesengine
2021/08/08 12:50:06 [INFO] update kontainerdriver aliyunkubernetescontainerservice
2021/08/08 12:50:06 [INFO] update kontainerdriver azurekubernetesservice
2021/08/08 12:50:06 [INFO] update kontainerdriver huaweicontainercloudengine
2021/08/08 12:50:06 [ERROR] error parsing azure-group-cache-size, skipping update strconv.Atoi: parsing "": invalid syntax
2021/08/08 12:50:06 [ERROR] error parsing azure-group-cache-size, skipping update strconv.Atoi: parsing "": invalid syntax
2021/08/08 12:50:07 [INFO] Active TLS secret serving-cert (ver=36388) (count 9): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.34:10.42.0.34 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:cbb8784b086c09c2d80b5a2c202cd7e584e513ad03f6a216463a80a2e76d0558]
2021/08/08 12:50:07 [INFO] Starting catalog.cattle.io/v1, Kind=App controller
2021/08/08 12:50:07 [INFO] Starting management.cattle.io/v3, Kind=ComposeConfig controller
2021/08/08 12:50:07 [INFO] Starting management.cattle.io/v3, Kind=ClusterAlertRule controller
2021/08/08 12:50:07 [INFO] Starting /v1, Kind=Pod controller
2021/08/08 12:50:07 [INFO] Starting management.cattle.io/v3, Kind=EtcdBackup controller
2021/08/08 12:50:07 [INFO] Starting management.cattle.io/v3, Kind=GlobalDnsProvider controller
2021/08/08 12:50:07 [INFO] Starting management.cattle.io/v3, Kind=ClusterAlertGroup controller
2021/08/08 12:50:07 [INFO] Starting apps/v1, Kind=DaemonSet controller
2021/08/08 12:50:07 [INFO] Starting catalog.cattle.io/v1, Kind=Operation controller
2021/08/08 12:50:07 [INFO] Starting management.cattle.io/v3, Kind=ClusterScan controller
2021/08/08 12:50:07 [INFO] Rancher startup complete
2021/08/08 12:50:07 [INFO] CleanupDuplicateBindings, checking configmap
2021/08/08 12:50:07 [INFO] bindings cleanup already ran before, not calling again
2021/08/08 12:50:09 [INFO] driverMetadata: refreshing data from upstream https://releases.rancher.com/kontainer-driver-metadata/release-v2.5/data.json
2021/08/08 12:50:09 [INFO] Retrieve data.json from local path /var/lib/rancher-data/driver-metadata/data.json
2021/08/08 12:50:10 [INFO] Refreshing driverMetadata in 1440 minutes
2021/08/08 12:50:11 [INFO] kontainerdriver azurekubernetesservice listening on address 127.0.0.1:34217
2021/08/08 12:50:11 [INFO] kontainerdriver amazonelasticcontainerservice listening on address 127.0.0.1:32817
2021/08/08 12:50:11 [INFO] kontainerdriver googlekubernetesengine listening on address 127.0.0.1:37729
2021/08/08 12:50:11 [INFO] kontainerdriver amazonelasticcontainerservice stopped
2021/08/08 12:50:11 [INFO] dynamic schema for kontainerdriver amazonelasticcontainerservice updating
2021/08/08 12:50:11 [INFO] kontainerdriver azurekubernetesservice stopped
2021/08/08 12:50:11 [INFO] dynamic schema for kontainerdriver azurekubernetesservice updating
2021/08/08 12:50:11 [INFO] kontainerdriver googlekubernetesengine stopped
2021/08/08 12:50:11 [INFO] dynamic schema for kontainerdriver googlekubernetesengine updating
2021/08/08 12:50:13 [INFO] driverMetadata: refreshing data from upstream https://releases.rancher.com/kontainer-driver-metadata/release-v2.5/data.json
2021/08/08 12:50:13 [INFO] Retrieve data.json from local path /var/lib/rancher-data/driver-metadata/data.json
2021/08/08 12:50:18 [INFO] Stopping cluster agent for local
2021/08/08 12:50:19 [INFO] Shutting down /v1, Kind=Service workers
2021/08/08 12:50:19 [INFO] Shutting down apiregistration.k8s.io/v1, Kind=APIService workers
2021/08/08 12:50:19 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding workers
2021/08/08 12:50:19 [INFO] Shutting down /v1, Kind=Pod workers
2021/08/08 12:50:19 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRole workers
2021/08/08 12:50:19 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=Role workers
2021/08/08 12:50:19 [INFO] Shutting down /v1, Kind=Endpoints workers
2021/08/08 12:50:19 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=RoleBinding workers
2021/08/08 12:50:19 [INFO] Shutting down /v1, Kind=Namespace workers
W0808 12:50:19.018088 7 reflector.go:437] pkg/mod/github.com/rancher/client-go@v1.20.0-rancher.1/tools/cache/reflector.go:168: watch of *v1.Role ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
2021/08/08 12:50:19 [INFO] Registering project network policy
2021/08/08 12:50:19 [INFO] Not registering CIS controller for non RKE cluster: local
2021/08/08 12:50:19 [INFO] registering podsecuritypolicy cluster handler for cluster local
2021/08/08 12:50:19 [INFO] registering podsecuritypolicy project handler for cluster local
2021/08/08 12:50:19 [INFO] registering podsecuritypolicy namespace handler for cluster local
2021/08/08 12:50:19 [INFO] registering podsecuritypolicy serviceaccount handler for cluster local
2021/08/08 12:50:19 [INFO] registering podsecuritypolicy template handler for cluster local
2021/08/08 12:50:19 [INFO] Registering monitoring for cluster "local"
2021/08/08 12:50:19 [INFO] Registering istio for cluster "local"
2021/08/08 12:50:20 [INFO] Registering namespaceHandler for adding labels
2021/08/08 12:50:20 [INFO] Starting cluster controllers for local
2021/08/08 12:50:20 [INFO] Starting management.cattle.io/v3, Kind=SamlToken controller
2021/08/08 12:50:20 [INFO] Starting project.cattle.io/v3, Kind=SourceCodeProviderConfig controller
2021/08/08 12:50:20 [INFO] Starting management.cattle.io/v3, Kind=ClusterLogging controller
2021/08/08 12:50:20 [INFO] Starting management.cattle.io/v3, Kind=ProjectAlert controller
2021/08/08 12:50:20 [INFO] Starting management.cattle.io/v3, Kind=Notifier controller
2021/08/08 12:50:20 [INFO] Starting management.cattle.io/v3, Kind=ProjectAlertGroup controller
2021/08/08 12:50:20 [INFO] Starting project.cattle.io/v3, Kind=PipelineSetting controller
2021/08/08 12:50:20 [INFO] Starting management.cattle.io/v3, Kind=ProjectNetworkPolicy controller
2021/08/08 12:50:20 [INFO] Starting management.cattle.io/v3, Kind=ProjectAlertRule controller
2021/08/08 12:50:21 [INFO] Starting cluster agent for local [owner=true]
2021/08/08 12:50:21 [INFO] Starting batch/v1beta1, Kind=CronJob controller
2021/08/08 12:50:21 [INFO] Starting extensions/v1beta1, Kind=Ingress controller
2021/08/08 12:50:21 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2021/08/08 12:50:21 [INFO] Starting /v1, Kind=LimitRange controller
2021/08/08 12:50:21 [INFO] Starting /v1, Kind=ConfigMap controller
2021/08/08 12:50:21 [INFO] Starting monitoring.coreos.com/v1, Kind=Prometheus controller
2021/08/08 12:50:21 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2021/08/08 12:50:21 [INFO] Starting batch/v1, Kind=Job controller
2021/08/08 12:50:21 [INFO] Starting /v1, Kind=ServiceAccount controller
2021/08/08 12:50:21 [INFO] Starting /v1, Kind=Endpoints controller
2021/08/08 12:50:21 [INFO] Starting /v1, Kind=Service controller
2021/08/08 12:50:21 [INFO] Starting networking.k8s.io/v1, Kind=NetworkPolicy controller
2021/08/08 12:50:21 [INFO] Starting /v1, Kind=Secret controller
2021/08/08 12:50:21 [INFO] Starting policy/v1beta1, Kind=PodSecurityPolicy controller
2021/08/08 12:50:21 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2021/08/08 12:50:21 [INFO] Starting apps/v1, Kind=Deployment controller
2021/08/08 12:50:21 [INFO] Starting apps/v1, Kind=ReplicaSet controller
2021/08/08 12:50:21 [INFO] Starting apps/v1, Kind=StatefulSet controller
2021/08/08 12:50:21 [INFO] Starting /v1, Kind=ResourceQuota controller
2021/08/08 12:50:21 [INFO] Starting apps/v1, Kind=DaemonSet controller
2021/08/08 12:50:21 [INFO] Updating workload [ingress-nginx/nginx-ingress-controller] with public endpoints [[{"nodeName":"local:machine-fhh7b","addresses":["192.168.157.141"],"port":80,"protocol":"TCP","podName":"ingress-nginx:nginx-ingress-controller-2sd2j","allNodes":false},{"nodeName":"local:machine-fhh7b","addresses":["192.168.157.141"],"port":443,"protocol":"TCP","podName":"ingress-nginx:nginx-ingress-controller-2sd2j","allNodes":false}]]
2021/08/08 12:50:21 [INFO] Starting /v1, Kind=Pod controller
2021/08/08 12:50:21 [INFO] Starting /v1, Kind=Node controller
2021/08/08 12:50:21 [INFO] Starting /v1, Kind=Event controller
2021/08/08 12:50:21 [INFO] Starting monitoring.coreos.com/v1, Kind=ServiceMonitor controller
2021/08/08 12:50:21 [INFO] Starting /v1, Kind=ReplicationController controller
2021/08/08 12:50:21 [INFO] Starting /v1, Kind=Namespace controller
2021/08/08 12:50:21 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2021/08/08 12:50:21 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2021/08/08 12:50:21 [INFO] Updating service [cm-acme-http-solver-gps6s] with public endpoints [[{"addresses":["192.168.157.141"],"port":32446,"protocol":"TCP","serviceName":"cattle-system:cm-acme-http-solver-gps6s","allNodes":true}]]
2021/08/08 12:50:24 [INFO] Adding peer wss://10.42.0.34/v3/connect, 10.42.0.34
2021/08/08 12:50:29 [INFO] Handling backend connection request [10.42.0.34]
2021/08/08 12:50:30 [INFO] Active TLS secret serving-cert (ver=36529) (count 10): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.34:10.42.0.34 listener.cattle.io/cn-10.42.0.39:10.42.0.39 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:886488085d31e3ebc49f83d909f43c692a45719f4293c566b41e72dd78dcf92c]
2021/08/08 12:50:31 [INFO] Adding peer wss://10.42.0.39/v3/connect, 10.42.0.39
2021/08/08 12:50:31 [INFO] Stopping cluster agent for local
2021/08/08 12:50:31 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=RoleBinding workers
W0808 12:50:31.032683 7 reflector.go:437] pkg/mod/github.com/rancher/client-go@v1.20.0-rancher.1/tools/cache/reflector.go:168: watch of *v1.StatefulSet ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
2021/08/08 12:50:31 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRole workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=Namespace workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=ReplicationController workers
2021/08/08 12:50:31 [INFO] Shutting down monitoring.coreos.com/v1, Kind=ServiceMonitor workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=Event workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=Node workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=Pod workers
2021/08/08 12:50:31 [INFO] Shutting down apps/v1, Kind=DaemonSet workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=ResourceQuota workers
2021/08/08 12:50:31 [INFO] Shutting down apps/v1, Kind=StatefulSet workers
2021/08/08 12:50:31 [INFO] Shutting down apps/v1, Kind=ReplicaSet workers
2021/08/08 12:50:31 [INFO] Shutting down apps/v1, Kind=Deployment workers
2021/08/08 12:50:31 [INFO] Shutting down apiregistration.k8s.io/v1, Kind=APIService workers
2021/08/08 12:50:31 [INFO] Shutting down policy/v1beta1, Kind=PodSecurityPolicy workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=Secret workers
2021/08/08 12:50:31 [INFO] Shutting down batch/v1beta1, Kind=CronJob workers
2021/08/08 12:50:31 [INFO] Shutting down networking.k8s.io/v1, Kind=NetworkPolicy workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=Service workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=Endpoints workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=ServiceAccount workers
2021/08/08 12:50:31 [INFO] Shutting down batch/v1, Kind=Job workers
2021/08/08 12:50:31 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=Role workers
2021/08/08 12:50:31 [INFO] Shutting down monitoring.coreos.com/v1, Kind=Prometheus workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=ConfigMap workers
2021/08/08 12:50:31 [INFO] Shutting down extensions/v1beta1, Kind=Ingress workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=LimitRange workers
2021/08/08 12:50:31 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding workers
2021/08/08 12:50:31 [INFO] Starting cluster controllers for local
2021/08/08 12:50:31 [INFO] Starting /v1, Kind=Service controller
2021/08/08 12:50:31 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2021/08/08 12:50:31 [INFO] Starting cluster agent for local [owner=false]
2021/08/08 12:50:31 [INFO] Starting /v1, Kind=Endpoints controller
2021/08/08 12:50:31 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2021/08/08 12:50:31 [INFO] Starting /v1, Kind=Namespace controller
2021/08/08 12:50:31 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2021/08/08 12:50:31 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2021/08/08 12:50:31 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2021/08/08 12:50:31 [INFO] Starting /v1, Kind=Pod controller
2021/08/08 12:50:34 [INFO] Handling backend connection request [10.42.0.39]

```

I’m sensing that the samething is happening with all 3 pods yet I’m clueless to what is really going on.  
I see a handshake failure there, but what could possibly be causing it?

---

<div class="post-metadata">

**Author:** ![kamal\_boumahdi](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/kamal_boumahdi/32/7837_2.png) [@kamal\_boumahdi](https://forums.suse.com/u/kamal_boumahdi)\
**Post date:** [August 10, 2021, 12:08pm UTC](https://forums.suse.com/t/liveness-probe-failed-connection-refused/20837/8 "2021-08-10T12:08:34Z")

</div>

could it be that I miss configured the load balancer?  
I’m using nginx version: nginx/1.20.1  
this is the IP of the machine where I ran **rke up : 192.168.157.140**  
and the **cluster.yml** shows :

```auto
$ cat cluster.yml
# If you intened to deploy Kubernetes in an air-gapped environment,
# please consult the documentation on how to configure custom RKE images.
nodes:
- address: 192.168.157.141
  port: "22"
  internal_address: ""
  role:
  - controlplane
  - worker
  - etcd
  hostname_override: fuji1
  user: fuji
  docker_socket: /var/run/docker.sock
  ssh_key: ""
  ssh_key_path: ~/.ssh/id_rsa
  ssh_cert: ""
  ssh_cert_path: ""
  labels: {}
  taints: []
services:
  etcd:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    external_urls: []
    ca_cert: ""
    cert: ""
    key: ""
    path: ""
    uid: 0
    gid: 0
    snapshot: null
    retention: ""
    creation: ""
    backup_config: null
  kube-api:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    service_cluster_ip_range: 10.43.0.0/16
    service_node_port_range: ""
    pod_security_policy: false
    always_pull_images: false
    secrets_encryption_config: null
    audit_log: null
    admission_configuration: null
    event_rate_limit: null
  kube-controller:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    cluster_cidr: 10.42.0.0/16
    service_cluster_ip_range: 10.43.0.0/16
  scheduler:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
  kubelet:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    cluster_domain: cluster.local
    infra_container_image: ""
    cluster_dns_server: 10.43.0.10
    fail_swap_on: false
    generate_serving_certificate: false
  kubeproxy:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
network:
  plugin: canal
  options: {}
  mtu: 0
  node_selector: {}
  update_strategy: null
authentication:
  strategy: x509
  sans: []
  webhook: null
addons: ""
addons_include: []
system_images:
  etcd: rancher/coreos-etcd:v3.4.3-rancher1
  alpine: rancher/rke-tools:v0.1.56
  nginx_proxy: rancher/rke-tools:v0.1.56
  cert_downloader: rancher/rke-tools:v0.1.56
  kubernetes_services_sidecar: rancher/rke-tools:v0.1.56
  kubedns: rancher/k8s-dns-kube-dns:1.15.0
  dnsmasq: rancher/k8s-dns-dnsmasq-nanny:1.15.0
  kubedns_sidecar: rancher/k8s-dns-sidecar:1.15.0
  kubedns_autoscaler: rancher/cluster-proportional-autoscaler:1.7.1
  coredns: rancher/coredns-coredns:1.6.5
  coredns_autoscaler: rancher/cluster-proportional-autoscaler:1.7.1
  nodelocal: rancher/k8s-dns-node-cache:1.15.7
  kubernetes: rancher/hyperkube:v1.17.4-rancher1
  flannel: rancher/coreos-flannel:v0.11.0-rancher1
  flannel_cni: rancher/flannel-cni:v0.3.0-rancher5
  calico_node: rancher/calico-node:v3.13.0
  calico_cni: rancher/calico-cni:v3.13.0
  calico_controllers: rancher/calico-kube-controllers:v3.13.0
  calico_ctl: rancher/calico-ctl:v2.0.0
  calico_flexvol: rancher/calico-pod2daemon-flexvol:v3.13.0
  canal_node: rancher/calico-node:v3.13.0
  canal_cni: rancher/calico-cni:v3.13.0
  canal_flannel: rancher/coreos-flannel:v0.11.0
  canal_flexvol: rancher/calico-pod2daemon-flexvol:v3.13.0
  weave_node: weaveworks/weave-kube:2.5.2
  weave_cni: weaveworks/weave-npc:2.5.2
  pod_infra_container: rancher/pause:3.1
  ingress: rancher/nginx-ingress-controller:nginx-0.25.1-rancher1
  ingress_backend: rancher/nginx-ingress-controller-defaultbackend:1.5-rancher1
  metrics_server: rancher/metrics-server:v0.3.6
  windows_pod_infra_container: rancher/kubelet-pause:v0.1.3
ssh_key_path: ~/.ssh/id_rsa
ssh_cert_path: ""
ssh_agent_auth: false
authorization:
  mode: rbac
  options: {}
ignore_docker_version: false
kubernetes_version: ""
private_registries: []
ingress:
  provider: ""
  options: {}
  node_selector: {}
  extra_args: {}
  dns_policy: ""
  extra_envs: []
  extra_volumes: []
  extra_volume_mounts: []
  update_strategy: null
cluster_name: ""
cloud_provider:
  name: ""
prefix_path: ""
addon_job_timeout: 0
bastion_host:
  address: ""
  port: ""
  user: ""
  ssh_key: ""
  ssh_key_path: ""
  ssh_cert: ""
  ssh_cert_path: ""
monitoring:
  provider: ""
  options: {}
  node_selector: {}
  update_strategy: null
  replicas: null
restore:
  restore: false
  snapshot_name: ""
dns: null

```

and this is what I did in nginx :

```auto
$ cat /etc/nginx/nginx.conf
stream {

        upstream TCP80{
                server 192.168.157.141:80;
        }
        upstream TCP443{
                server 192.168.157.141:443;
        }

        server {
                listen 80;
                proxy_pass TCP80;
        }
        server{
                listen 443;
                proxy_pass TCP443;
        }
}
events { }

```

---

<div class="post-metadata">

**Author:** ![superseb](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/superseb/32/2424_2.png) [@superseb](https://forums.suse.com/u/superseb)\
**Post date:** [August 11, 2021, 10:41am UTC](https://forums.suse.com/t/liveness-probe-failed-connection-refused/20837/9 "2021-08-11T10:41:33Z")

</div>

None of the logs show a restart, to know why its restarting, we need the logs from the pod that exited (this might need `kubectl logs -p` for previous pod logs. When you are using Rancher, what problems do you encounter?

---

<div class="post-metadata">

**Author:** ![kamal\_boumahdi](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/kamal_boumahdi/32/7837_2.png) [@kamal\_boumahdi](https://forums.suse.com/u/kamal_boumahdi)\
**Post date:** [August 11, 2021, 10:23pm UTC](https://forums.suse.com/t/liveness-probe-failed-connection-refused/20837/10 "2021-08-11T22:23:28Z")

</div>

I get a blank page when I browse to the host name [https://rancher.my.org](https://rancher.my.org)

---

<div class="post-metadata">

**Author:** ![superseb](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/superseb/32/2424_2.png) [@superseb](https://forums.suse.com/u/superseb)\
**Post date:** [August 12, 2021, 8:28am UTC](https://forums.suse.com/t/liveness-probe-failed-connection-refused/20837/11 "2021-08-12T08:28:40Z")

</div>

All the time or randomly? Simplest way to start debugging is to see what the response is on the command line, for example, `curl -vLk https://rancher.my.org` and start from there. Also, if you think the load balancer is the issue, exclude it by pointing the DNS to a node directly and see if that works.

The example NGINX configuration for layer 4 can be found here: [Rancher Docs: Setting up an NGINX Load Balancer](https://rancher.com/docs/rancher/v2.5/en/installation/resources/k8s-tutorials/infrastructure-tutorials/nginx/)

---

<div class="post-metadata">

**Author:** ![kamal\_boumahdi](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/kamal_boumahdi/32/7837_2.png) [@kamal\_boumahdi](https://forums.suse.com/u/kamal_boumahdi)\
**Post date:** [August 16, 2021, 1:46am UTC](https://forums.suse.com/t/liveness-probe-failed-connection-refused/20837/12 "2021-08-16T01:46:29Z")

</div>

well the blank page is all the time, the restarts are random.  
as for debugging from within, here’s what I get:

```auto
~$ curl -vLk https://rancher.my.org
* Trying 64.190.62.111:443...
* TCP_NODELAY set
* Connected to rancher.my.org (64.190.62.111) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=rancher.my.org
* start date: Jul 26 00:00:00 2021 GMT
* expire date: Jul 26 23:59:59 2022 GMT
* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=Encryption Everywhere DV TLS CA - G1
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x558754f60e10)
> GET / HTTP/2
> Host: rancher.my.org
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200
< cache-control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< content-type: text/html; charset=UTF-8
< date: Mon, 16 Aug 2021 01:10:27 GMT
< expires: Mon, 26 Jul 1997 05:00:00 GMT
< last-modified: Mon, 16 Aug 2021 01:10:27 GMT
< pragma: no-cache
< server: NginX
< vary: Accept-Encoding
< x-adblock-key: MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBANnylWw2vLY4hUn9w06zQKbhKBfvjFUCsdFlb6TdQhxb9RXWXuI4t31c+o8fYOv/s8q1LGPga3DE1L/tHU4LENMCAwEAAQ==_xd/MGsOJ8/p4oGkEaB6Vmp67y3786AgZqepPA/7wf5voD5M/bARtM6XqJg1ah8THtifPlehNIVapKbYwIsJ52Q==
< x-cache-miss-from: parking-7d65c95449-4j5qd
<
<!DOCTYPE html><html lang="en" data-adblockkey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBANnylWw2vLY4hUn9w06zQKbhKBfvjFUCsdFlb6TdQhxb9RXWXuI4t31c+o8fYOv/s8q1LGPga3DE1L/tHU4LENMCAwEAAQ==_xd/MGsOJ8/p4oGkEaB6Vmp67y3786AgZqepPA/7wf5voD5M/bARtM6XqJg1ah8THtifPlehNIVapKbYwIsJ52Q==><head><meta charset="utf-8"><title>my.org&nbsp;-&nbspSocial networking المصادر والمعلومات.</title><meta name="viewport" content="width=device-width,initial-scale=1.0,maximum-scale=1.0,user-scalable=0"><meta name="description" content="my.org هل هو أول وأفضل مصادر جميع المعلومات التي تبحث عنها. من ضوء الموضوعات العامة إلى مزيد من الموضوعات التي كنت لتتوقعها هنا، my.org تمتلكها جميعًا. نأمل أن تجد ما تبحث عنه!"><link
        rel="icon"
        type="image/png"
        href="//img.sedoparking.com/templates/logos/sedo_logo.png"
/><style>
        /*! normalize.css v7.0.0 | MIT License | github.com/necolas/normalize.css */html{line-height:1.15;-ms-text-size-adjust:100%;-webkit-text-size-adjust:100%}body{margin:0}article,aside,footer,header,nav,section{display:block}h1{font-size:2em;margin:.67em 0}figcaption,figure,main{display:block}figure{margin:1em 40px}hr{box-sizing:content-box;height:0;overflow:visible}pre{font-family:monospace,monospace;font-size:1em}a{background-color:transparent;-webkit-text-decoration-skip:objects}abbr[title]{border-bottom:none;text-decoration:underline;text-decoration:underline dotted}b,strong{font-weight:inherit}b,strong{font-weight:bolder}code,kbd,samp{font-family:monospace,monospace;font-size:1em}dfn{font-style:italic}mark{background-color:#ff0;color:#000}small{font-size:80%}sub,sup{font-size:75%;line-height:0;position:relative;vertical-align:baseline}sub{bottom:-0.25em}sup{top:-0.5em}audio,video{display:inline-block}audio:not([controls]){display:none;height:0}img{border-style:none}svg:not(:root){overflow:hidden}button,input,optgroup,select,textarea{font-family:sans-serif;font-size:100%;line-height:1.15;margin:0}button,input{overflow:visible}button,select{text-transform:none}button,html [type=button],[type=reset],[type=submit]{-webkit-appearance:button}button::-moz-focus-inner,[type=button]::-moz-focus-inner,[type=reset]::-moz-focus-inner,[type=submit]::-moz-focus-inner{border-style:none;padding:0}button:-moz-focusring,[type=button]:-moz-focusring,[type=reset]:-moz-focusring,[type=submit]:-moz-focusring{outline:1px dotted ButtonText}fieldset{padding:.35em .75em .625em}legend{box-sizing:border-box;color:inherit;display:table;max-width:100%;padding:0;white-space:normal}progress{display:inline-block;vertical-align:baseline}textarea{overflow:auto}[type=checkbox],[type=radio]{box-sizing:border-box;padding:0}[type=number]::-webkit-inner-spin-button,[type=number]::-webkit-outer-spin-button{height:auto}[type=search]{-webkit-appearance:textfield;outline-offset:-2px}[type=search]::-webkit-search-cancel-button,[type=search]::-webkit-search-decoration{-webkit-appearance:none}::-webkit-file-upload-button{-webkit-appearance:button;font:inherit}details,menu{display:block}summary{display:list-item}canvas{display:inline-block}template{display:none}[hidden]{display:none}.announcement{background:#262626;text-align:center;padding:0 5px}.announcement p{color:#717171}.announcement a{color:#717171}.container-header{margin:0 auto 0 auto;text-align:center}.container-header__content{color:#717171}.container-content{margin:25px auto 20px auto;text-align:center;background:url("//img.sedoparking.com/templates/bg/arrows-1-colors-3.png") #fbfbfb no-repeat center top;background-size:100%}.container-content__container-relatedlinks,.container-content __container-ads,.container-content__ webarchive{width:30%;display:inline-block}.container-content __container-relatedlinks{margin-top:9%}.container-content__ container-ads{margin-top:8%}.container-content __container-ads--twot{margin-top:7%}.container-content__ webarchive{margin-top:8%}.container-content __header{color:#717171;font-size:15px;margin:0}.container-content--lp{width:90%;min-height:820px}.container-content--rp{width:90%;min-height:820px}.container-content--wa{width:90%}.container-content--twot{width:90%;min-height:820px}.two-tier-ads-list{padding:0 0 1.6em 0}.two-tier-ads-list__ list-element{list-style:none;padding:10px 0 5px 0;display:inline-block}.two-tier-ads-list__list-element-image{content:url("//img.sedoparking.com/templates/images/bullet_justads.gif");float:left;padding-top:32px}.two-tier-ads-list__list-element-content{display:inline-block}.two-tier-ads-list __list-element-header-link{font-size:37px;font-weight:bold;text-decoration:underline;color:#0a48ff}.two-tier-ads-list__ list-element-text{padding:3px 0 6px 0;margin:.11em 0;line-height:18px;color:#000}.two-tier-ads-list __list-element-link{font-size:1em;text-decoration:underline;color:#0a48ff}.two-tier-ads-list__ list-element-link:link,.two-tier-ads-list __list-element-link:visited{text-decoration:underline}.two-tier-ads-list__ list-element-link:hover,.two-tier-ads-list __list-element-link:active,.two-tier-ads-list__ list-element-link:focus{text-decoration:none}.webarchive-block{text-align:center}.webarchive-block __header-link{color:#0a48ff;font-size:20px}.webarchive-block__ list{padding:0}.webarchive-block __list-element{word-wrap:break-word;list-style:none}.webarchive-block__ list-element-link{line-height:30px;font-size:20px;color:rgba(10,72,255,.7)}.webarchive-block __list-element-link:link,.webarchive-block__ list-element-link:visited{text-decoration:none}.webarchive-block __list-element-link:hover,.webarchive-block__ list-element-link:active,.webarchive-block __list-element-link:focus{text-decoration:underline}.container-buybox{text-align:center}.container-buybox__ content-buybox{display:inline-block;text-align:left}.container-buybox __content-heading{font-size:15px}.container-buybox__ content-text{font-size:12px}.container-buybox __content-link{color:#919da6}.container-buybox__ content-link--no-decoration{text-decoration:none}.container-searchbox{margin-bottom:50px;text-align:center}.container-searchbox __content{display:inline-block;font-family:arial,sans-serif;font-size:12px}.container-searchbox__ searchtext-label{display:none}.container-searchbox __input,.container-searchbox__ button{border:0 none}.container-searchbox __button{cursor:pointer;font-size:12px;margin-left:15px;border:0 none;padding:2px 8px;color:#638296}.container-disclaimer{text-align:center}.container-disclaimer__ content{display:inline-block}.container-disclaimer __content-text,.container-disclaimer a{font-size:10px}.container-disclaimer__ content-text{color:#555}.container-disclaimer a{color:#555}.container-imprint{text-align:center}.container-imprint __content{display:inline-block}.container-imprint__ content-text,.container-imprint __content-link{font-size:10px;color:#555}.container-privacyPolicy{text-align:center}.container-privacyPolicy__ content{display:inline-block}.container-privacyPolicy __content-link{font-size:10px;color:#555}.container-cookie-message{position:fixed;bottom:0;width:100%;background:#5f5f5f;font-size:12px;padding-top:15px;padding-bottom:15px}.container-cookie-message__ content-text{color:#fff}.container-cookie-message __content-text{margin-left:15%;margin-right:15%}.container-cookie-message__ content-interactive{text-align:left;margin:0 15px;font-size:10px}.container-cookie-message __content-interactive-header,.container-cookie-message__ content-interactive-text{color:#fff}.container-cookie-message __content-interactive-header{font-size:small}.container-cookie-message__ content-interactive-text{margin-top:10px;margin-right:0px;margin-bottom:5px;margin-left:0px;font-size:larger}.container-cookie-message a{color:#fff}.cookie-modal-window{position:fixed;background-color:rgba(200,200,200,.75);top:0;right:0;bottom:0;left:0;-webkit-transition:all .3s;-moz-transition:all .3s;transition:all .3s;text-align:center}.cookie-modal-window __content-header{font-size:150%;margin:0 0 15px}.cookie-modal-window__ content{text-align:initial;margin:10% auto;padding:40px;background:#fff;display:inline-block;max-width:550px}.cookie-modal-window __content-text{line-height:1.5em}.cookie-modal-window__ close{width:100%;margin:0}.cookie-modal-window __content-body table{width:100%;border-collapse:collapse}.cookie-modal-window__ content-body table td{padding-left:15px}.cookie-modal-window __content-necessary-cookies-row{background-color:#dee1e3}.disabled{display:none;z-index:-999}.btn{display:inline-block;border-style:solid;border-radius:5px;padding:15px 25px;text-align:center;text-decoration:none;cursor:pointer;margin:5px;transition:.3s}.btn--success{background-color:#218838;border-color:#218838;color:#fff;font-size:x-large}.btn--success:hover{background-color:#1a6b2c;border-color:#1a6b2c;color:#fff;font-size:x-large}.btn--success-sm{background-color:#218838;border-color:#218838;color:#fff;font-size:initial}.btn--success-sm:hover{background-color:#1a6b2c;border-color:#1a6b2c;color:#fff;font-size:initial}.btn--secondary{background-color:#8c959c;border-color:#8c959c;color:#fff;font-size:medium}.btn--secondary:hover{background-color:#727c83;border-color:#727c83;color:#fff;font-size:medium}.btn--secondary-sm{background-color:#8c959c;border-color:#8c959c;color:#fff;font-size:initial}.btn--secondary-sm:hover{background-color:#727c83;border-color:#727c83;color:#fff;font-size:initial}.switch input{opacity:0;width:0;height:0}.switch{position:relative;display:inline-block;width:60px;height:34px}.switch__ slider{position:absolute;cursor:pointer;top:0;left:0;right:0;bottom:0;background-color:#5a6268;-webkit-transition:.4s;transition:.4s}.switch __slider:before{position:absolute;content:"";height:26px;width:26px;left:4px;bottom:4px;background-color:#fff;-webkit-transition:.4s;transition:.4s}.switch__ slider--round{border-radius:34px}.switch __slider--round:before{border-radius:50%}input:checked+.switch__ slider{background-color:#007bff}input:focus+.switch __slider{box-shadow:0 0 1px #007bff}input:checked+.switch__ slider:before{-webkit-transform:translateX(26px);-ms-transform:translateX(26px);transform:translateX(26px)}body{background-color:#262626;font-family:Arial,Helvetica,Verdana,"Lucida Grande",sans-serif}body.cookie-message-enabled{padding-bottom:300px}.container-footer{padding-top:0;padding-left:5%;padding-right:5%;padding-bottom:10px}

    </style><script type="text/javascript">
        var dto = {"uiOptimize":false,"singleDomainName":"my.org","domainName":"my.org","domainPrice":0,"domainCurrency":"","adultFlag":false,"pu":"//rancher.my.org","dnsh":true,"dpsh":false,"toSell":false,"cdnHost":"http://img.sedoparking.com","adblockkey":" data-adblockkey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBANnylWw2vLY4hUn9w06zQKbhKBfvjFUCsdFlb6TdQhxb9RXWXuI4t31c+o8fYOv/s8q1LGPga3DE1L/tHU4LENMCAwEAAQ==_xd/MGsOJ8/p4oGkEaB6Vmp67y3786AgZqepPA/7wf5voD5M/bARtM6XqJg1ah8THtifPlehNIVapKbYwIsJ52Q==","tid":3032,"buybox":false,"buyboxTopic":true,"disclaimer":true,"imprint":false,"searchbox":true,"noFollow":false,"slsh":false,"ppsh":true,"dnhlsh":true,"toSellUrl":"","toSellText":"","searchboxPath":"//rancher.my.org/parking.php","searchParams":{"ses":"Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0xJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejYmdHJhY2txdWVyeT0x"},"gts":false,"imprintUrl":false,"contentType":2,"t":"content","pus":"ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=","postActionParameter":false,"gFeedSES":{"default":"OAlmYThmNWY3MzlkM2YxNDQ2ZWM3ZjQzMTZiMjRkYWI1ZAkJMwkwCQkxMDcxMjE3NzgJU29jaWFsIG5ldHdvcmtpbmcJMzAzMgkJNDcJOTIJMTYyOTA3NjIyNwkwCU4JMAkxCTAJMTI5OAk4MTU4OTE4NgkxOTYuNjUuMjI4LjE0Mwkx","alternate":"OAk0OTFlZjFkN2FjMjlhZGM2OWM5MDNkMDdkY2E5ZmZmOQkwCTMJMAkJMTA3MTIxNzc4CVNvY2lhbCBuZXR3b3JraW5nCTMwMzIJCTQ3CTkyCTE2MjkwNzYyMjcJMAlOCTAJMQkwCTEyOTgJODE1ODkxODYJMTk2LjY1LjIyOC4xNDMJMQ%3D%3D"},"visitorViewIdJsAds":false,"jsParameter":false,"ads":[],"adv":0,"advt":2,"rls":[{"term":"Dating and Relationships"}],"numberRelatedLinks":6,"waUrl":"/search/portal.php?l=OAlmYThmNWY3MzlkM2YxNDQ2ZWM3ZjQzMTZiMjRkYWI1ZAkJMwkwCQkxMDcxMjE3NzgJU29jaWFsIG5ldHdvcmtpbmcJMzAzMgkJNDcJOTIJMTYyOTA3NjIyNwkwCU4JMAkxCTAJMTI5OAk4MTU4OTE4NgkxOTYuNjUuMjI4LjE0Mwkx","tsc":true,"tscQs":"200=MTA3MTIxNzc4&21=MTk2LjY1LjIyOC4xNDM=&681=MTYyOTA3NjIyNzczNjgzYTU2NDI3Yzk0OWEwZjc3MGU0MGQzYTNjM2Zm&crc=2408e98de656f67a89fd5c1a4a4c2d5d7a72a87d&cv=1","lang":"ar","maid":92,"sedoParkingUrl":"https://www.sedo.com/services/parking.php3","dbg":false,"signedLink":"l","visitorViewId":"v","registrar_params":[],"clickTrack":[],"cookieMessage":false,"cookieMessageInteractive":false,"executeTrackingPixels":false,"bannerTypes":[]};
    </script></head><body ><div class="container-header" id="container-header"><h1 class="container-header __content">my.org</h1></div><div id="container-content" class="container-content container-content--wa"><div class="container-content__ webarchive" id="container-webarchive"><div class="container-content __content-webarchive" id="wa-default"><h2 class="container-content__ header">
                الفئات الأكثر شعبية
            </h2><div class="webarchive-block"><h3 class="webarchive-block__header"><a
                                class="webarchive-block__header-link"
                                href="//rancher.my.org/info/%D8%AA%D9%85%D9%88%D9%8A%D9%84.html>?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=تمويل&amp;keyword=تمويل">
                            تمويل
                        </a></h3><ul class="webarchive-block __list"><li class="webarchive-block__ list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%AA%D8%A3%D9%85%D9%8A%D9%86-%D8%A7%D9%84%D8%A3%D9%85%D9%84%D8%A7%D9%83.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=تأمين الأملاك&amp;keyword=تأمين الأملاك">
                                    تأمين الأملاك
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A3%D8%B3%D9%87%D9%85.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=أسهم&amp;keyword=أسهم">
                                    أسهم
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A7%D8%B3%D8%AA%D8%AB%D9%85%D8%A7%D8%B1%D8%A7%D8%AA.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=استثمارات&amp;keyword=استثمارات">
                                    استثمارات
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%AA%D8%A3%D9%85%D9%8A%D9%86.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=تأمين&amp;keyword=تأمين">
                                    تأمين
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A8%D8%B7%D8%A7%D9%82%D8%A7%D8%AA-%D8%A5%D8%A6%D8%AA%D9%85.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=بطاقات إئتم&amp;keyword=بطاقات إئتم">
                                    بطاقات إئتم
                                </a></li></ul></div><div class="webarchive-block"><h3 class="webarchive-block__header"><a
                                class="webarchive-block__header-link"
                                href="//rancher.my.org/info/%D8%A7%D9%84%D9%83%D8%AA%D8%B1%D9%88%D9%86%D9%8A%D8%A7%D8%AA.html>?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=الكترونيات&amp;keyword=الكترونيات">
                            الكترونيات
                        </a></h3><ul class="webarchive-block __list"><li class="webarchive-block__ list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%AF%D9%8A-%D9%81%D9%8A-%D8%AF%D9%8A.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=دي في دي&amp;keyword=دي في دي">
                                    دي في دي
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D9%87%D9%88%D8%A7%D8%AA%D9%81-%D8%A7%D9%84%D9%85%D8%AD%D9%85%D9%88%D9%84.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=هواتف المحمول&amp;keyword=هواتف المحمول">
                                    هواتف المحمول
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A7%D9%84%D9%83%D8%A7%D9%85%D9%8A%D8%B1%D8%A7%D8%AA-%D8%A7%D9%84%D8%B1%D9%82%D9%85%D9%8A%D8%A9.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=الكاميرات الرقمية&amp;keyword=الكاميرات الرقمية">
                                    الكاميرات الرقمية
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A7%D9%84%D9%87%D9%88%D8%A7%D8%AA%D9%81.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=الهواتف&amp;keyword=الهواتف">
                                    الهواتف
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A7%D9%84%D8%AD%D9%88%D8%A7%D8%B3%D8%A8-%D8%A7%D9%84%D9%83%D9%81%D9%8A%D8%A9.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=الحواسب الكفية&amp;keyword=الحواسب الكفية">
                                    الحواسب الكفية
                                </a></li></ul></div><div class="webarchive-block"><h3 class="webarchive-block__header"><a
                                class="webarchive-block__header-link"
                                href="//rancher.my.org/info/%D8%B3%D9%8A%D8%A7%D8%AD%D8%A9.html>?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=سياحة&amp;keyword=سياحة">
                            سياحة
                        </a></h3><ul class="webarchive-block __list"><li class="webarchive-block__ list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D9%81%D9%86%D8%A7%D8%AF%D9%82.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=فنادق&amp;keyword=فنادق">
                                    فنادق
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A2%D8%AE%D8%B1-%D9%84%D8%AD%D8%B8%D8%A9.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=آخر لحظة&amp;keyword=آخر لحظة">
                                    آخر لحظة
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%B9%D8%B7%D9%84%D8%A7%D8%AA.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=عطلات&amp;keyword=عطلات">
                                    عطلات
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%AA%D8%A3%D8%AC%D9%8A%D8%B1-%D8%B3%D9%8A%D8%A7%D8%B1%D8%A7%D8%AA.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=تأجير سيارات&amp;keyword=تأجير سيارات">
                                    تأجير سيارات
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%B1%D8%AD%D9%84%D8%A7%D8%AA-%D8%A8%D8%AD%D8%B1%D9%8A%D8%A9.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=رحلات بحرية&amp;keyword=رحلات بحرية">
                                    رحلات بحرية
                                </a></li></ul></div><div class="webarchive-block"><h3 class="webarchive-block__header"><a
                                class="webarchive-block__header-link"
                                href="//rancher.my.org/info/%D8%A3%D9%84%D8%B9%D8%A7%D8%A8.html>?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=ألعاب&amp;keyword=ألعاب">
                            ألعاب
                        </a></h3><ul class="webarchive-block __list"><li class="webarchive-block__ list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/Casino.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=Casino&amp;keyword=Casino">
                                    Casino
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/Poker.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=Poker&amp;keyword=Poker">
                                    Poker
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/Sports-Betting.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=Sports Betting&amp;keyword=Sports Betting">
                                    Sports Betting
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A7%D9%83%D8%B3-%D8%A8%D9%88%D9%83%D8%B3.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=اكس بوكس&amp;keyword=اكس بوكس">
                                    اكس بوكس
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A8%D9%84%D8%A7%D9%8A-%D8%B3%D8%AA%D9%8A%D8%B4%D9%86.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=بلاي ستيشن&amp;keyword=بلاي ستيشن">
                                    بلاي ستيشن
                                </a></li></ul></div><div class="webarchive-block"><h3 class="webarchive-block__header"><a
                                class="webarchive-block__header-link"
                                href="//rancher.my.org/info/%D8%A5%D9%86%D8%AA%D8%B1%D9%86%D8%AA%21.html>?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=إنترنت!&amp;keyword=إنترنت!">
                            إنترنت!
                        </a></h3><ul class="webarchive-block __list"><li class="webarchive-block__ list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A3%D9%85%D8%A7%D9%86-%D8%A7%D9%84%D8%B4%D8%A8%D9%83%D8%A9.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=أمان الشبكة&amp;keyword=أمان الشبكة">
                                    أمان الشبكة
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D9%85%D8%B2%D9%88%D8%AF%D9%8A-%D8%AE%D8%AF%D9%85%D8%A9-%D8%A7%D9%84%D8%A5%D9%86%D8%AA%D8%B1%D9%86%D8.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=مزودي خدمة الإنترنت&amp;keyword=مزودي خدمة الإنترنت">
                                    مزودي خدمة الإنترنت
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%AF%D9%8A-%D8%A7%D8%B3-%D8%A7%D9%84.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=دي اس ال&amp;keyword=دي اس ال">
                                    دي اس ال
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A5%D8%B3%D8%AA%D8%B6%D8%A7%D9%81%D8%A9-%D9%85%D9%88%D8%A7%D9%82%D8%B9.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=إستضافة مواقع&amp;keyword=إستضافة مواقع">
                                    إستضافة مواقع
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%AA%D8%B5%D9%85%D9%8A%D9%85-%D9%85%D9%88%D8%A7%D9%82%D8%B9.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=تصميم مواقع&amp;keyword=تصميم مواقع">
                                    تصميم مواقع
                                </a></li></ul></div><div class="webarchive-block"><h3 class="webarchive-block__header"><a
                                class="webarchive-block__header-link"
                                href="//rancher.my.org/info/%D8%A3%D8%B3%D9%84%D9%88%D8%A8-%D8%A7%D9%84%D8%AD%D9%8A%D8%A7%D8%A9.html>?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=أسلوب الحياة&amp;keyword=أسلوب الحياة">
                            أسلوب الحياة
                        </a></h3><ul class="webarchive-block __list"><li class="webarchive-block__ list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A3%D8%AF%D9%88%D9%8A%D8%A9.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=أدوية&amp;keyword=أدوية">
                                    أدوية
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D9%81%D9%8A%D8%AA%D8%A7%D9%85%D9%8A%D9%86%D8%A7%D8%AA.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=فيتامينات&amp;keyword=فيتامينات">
                                    فيتامينات
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A7%D9%84%D9%86%D9%88%D9%85.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=النوم&amp;keyword=النوم">
                                    النوم
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%B1%D8%AC%D9%8A%D9%85.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=رجيم&amp;keyword=رجيم">
                                    رجيم
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D9%81%D9%82%D8%AF%D8%A7%D9%86-%D8%A7%D9%84%D9%88%D8%B2%D9%86.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=فقدان الوزن&amp;keyword=فقدان الوزن">
                                    فقدان الوزن
                                </a></li></ul></div><div class="webarchive-block"><h3 class="webarchive-block__header"><a
                                class="webarchive-block__header-link"
                                href="//rancher.my.org/info/%D9%87%D8%AF%D8%A7%D9%8A%D8%A7.html>?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=هدايا&amp;keyword=هدايا">
                            هدايا
                        </a></h3><ul class="webarchive-block __list"><li class="webarchive-block__ list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%AC%D9%88%D9%84%D9%81.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=جولف&amp;keyword=جولف">
                                    جولف
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%B3%D9%84%D8%A7%D8%AA-%D9%87%D8%AF%D8%A7%D9%8A%D8%A7.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=سلات هدايا&amp;keyword=سلات هدايا">
                                    سلات هدايا
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D9%85%D8%AC%D9%88%D9%87%D8%B1%D8%A7%D8%AA.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=مجوهرات&amp;keyword=مجوهرات">
                                    مجوهرات
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%B2%D9%87%D9%88%D8%B1.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=زهور&amp;keyword=زهور">
                                    زهور
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A7%D9%84%D8%B2%D9%88%D8%A7%D8%AC.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=الزواج&amp;keyword=الزواج">
                                    الزواج
                                </a></li></ul></div><div class="webarchive-block"><h3 class="webarchive-block__header"><a
                                class="webarchive-block__header-link"
                                href="//rancher.my.org/info/%D8%AD%D9%88%D8%A7%D8%B3%D8%A8-%D8%A2%D9%84%D9%8A%D8%A9.html>?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=حواسب آلية&amp;keyword=حواسب آلية">
                            حواسب آلية
                        </a></h3><ul class="webarchive-block __list"><li class="webarchive-block__ list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A3%D9%84%D8%B9%D8%A7%D8%A8-%D8%A7%D9%84%D8%AD%D8%A7%D8%B3%D8%A8-%D8%A7%D9%84%D8%A2%D9%84%D9%8A.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=ألعاب الحاسب الآلي&amp;keyword=ألعاب الحاسب الآلي">
                                    ألعاب الحاسب الآلي
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%AE%D9%88%D8%A7%D8%AF%D9%85.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=خوادم&amp;keyword=خوادم">
                                    خوادم
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A7%D9%84%D8%AD%D8%A7%D8%B3%D8%A8-%D8%A7%D9%84%D9%86%D9%82%D8%A7%D9%84.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=الحاسب النقال&amp;keyword=الحاسب النقال">
                                    الحاسب النقال
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D9%82%D8%B7%D8%B9-%D8%A3%D8%AC%D9%87%D8%B2%D8%A9-%D8%A7%D9%84%D8%AD%D8%A7%D8%B3%D8%A8-%D8%A7%D9%84.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=قطع أجهزة الحاسب الآلي&amp;keyword=قطع أجهزة الحاسب الآلي">
                                    قطع أجهزة الحاسب الآلي
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A8%D8%B1%D8%A7%D9%85%D8%AC.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=برامج&amp;keyword=برامج">
                                    برامج
                                </a></li></ul></div><div class="webarchive-block"><h3 class="webarchive-block__header"><a
                                class="webarchive-block__header-link"
                                href="//rancher.my.org/info/%D8%A3%D8%B5%D8%AF%D9%82%D8%A7%D8%A1.html>?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=أصدقاء&amp;keyword=أصدقاء">
                            أصدقاء
                        </a></h3><ul class="webarchive-block __list"><li class="webarchive-block__ list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A7%D9%84%D8%B2%D9%88%D8%A7%D8%AC.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=الزواج&amp;keyword=الزواج">
                                    الزواج
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/Eastern.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=Eastern&amp;keyword=Eastern">
                                    Eastern
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/Christmas.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=Christmas&amp;keyword=Christmas">
                                    Christmas
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%B9%D9%8A%D8%AF-%D9%85%D9%8A%D9%84%D8%A7%D8%AF.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=عيد ميلاد&amp;keyword=عيد ميلاد">
                                    عيد ميلاد
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A8%D8%B7%D8%A7%D9%82%D8%A7%D8%AA-%D9%85%D8%B9%D8%A7%D9%8A%D8%AF%D8%A9.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=بطاقات معايدة&amp;keyword=بطاقات معايدة">
                                    بطاقات معايدة
                                </a></li></ul></div><div class="webarchive-block"><h3 class="webarchive-block__header"><a
                                class="webarchive-block__header-link"
                                href="//rancher.my.org/info/%D8%A7%D9%84%D8%AA%D8%B1%D9%81%D9%8A%D9%87.html>?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=الترفيه&amp;keyword=الترفيه">
                            الترفيه
                        </a></h3><ul class="webarchive-block __list"><li class="webarchive-block__ list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D9%85%D9%88%D8%B3%D9%8A%D9%82%D9%89.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=موسيقى&amp;keyword=موسيقى">
                                    موسيقى
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%AA%D8%B0%D8%A7%D9%83%D8%B1.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=تذاكر&amp;keyword=تذاكر">
                                    تذاكر
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/MP3s.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=MP3s&amp;keyword=MP3s">
                                    MP3s
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A3%D9%81%D9%84%D8%A7%D9%85.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=أفلام&amp;keyword=أفلام">
                                    أفلام
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%B1%D9%8A%D8%A7%D8%B6%D8%A9.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=رياضة&amp;keyword=رياضة">
                                    رياضة
                                </a></li></ul></div><div class="webarchive-block"><h3 class="webarchive-block__header"><a
                                class="webarchive-block__header-link"
                                href="//rancher.my.org/info/%D8%A7%D9%84%D8%AA%D8%B3%D9%88%D9%82.html>?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=التسوق&amp;keyword=التسوق">
                            التسوق
                        </a></h3><ul class="webarchive-block __list"><li class="webarchive-block__ list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A7%D9%84%D9%85%D9%88%D8%B6%D9%87.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=الموضه&amp;keyword=الموضه">
                                    الموضه
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%B2%D9%87%D9%88%D8%B1.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=زهور&amp;keyword=زهور">
                                    زهور
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D9%85%D8%A8%D9%8A%D8%B9%D8%A7%D8%AA.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=مبيعات&amp;keyword=مبيعات">
                                    مبيعات
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D9%85%D9%84%D8%A7%D8%A8%D8%B3-%D9%86%D8%B3%D8%A7%D8%A6%D9%8A%D8%A9.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=ملابس نسائية&amp;keyword=ملابس نسائية">
                                    ملابس نسائية
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D9%85%D8%B2%D8%A7%D8%AF%D8%A7%D8%AA.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=مزادات&amp;keyword=مزادات">
                                    مزادات
                                </a></li></ul></div><div class="webarchive-block"><h3 class="webarchive-block__header"><a
                                class="webarchive-block__header-link"
                                href="//rancher.my.org/info/%D8%AA%D8%AC%D8%A7%D8%B1%D8%A9.html>?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=تجارة&amp;keyword=تجارة">
                            تجارة
                        </a></h3><ul class="webarchive-block __list"><li class="webarchive-block__ list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%AE%D8%B7%D8%A9-%D8%AA%D8%AC%D8%A7%D8%B1%D9%8A%D8%A9.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=خطة تجارية&amp;keyword=خطة تجارية">
                                    خطة تجارية
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D9%88%D8%B8%D8%A7%D8%A6%D9%81.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=وظائف&amp;keyword=وظائف">
                                    وظائف
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%AA%D8%AF%D8%B1%D9%8A%D8%A8-%D8%AA%D9%82%D9%86%D9%8A.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=تدريب تقني&amp;keyword=تدريب تقني">
                                    تدريب تقني
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A5%D8%AF%D8%A7%D8%B1%D8%A9.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=إدارة&amp;keyword=إدارة">
                                    إدارة
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A7%D9%84%D8%B9%D9%85%D9%84-%D9%85%D9%86-%D8%A7%D9%84%D9%85%D9%86%D8%B2%D9%84.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=العمل من المنزل&amp;keyword=العمل من المنزل">
                                    العمل من المنزل
                                </a></li></ul></div></div></div></div><div class="container-searchbox" id="container-searchbox"><div class="container-searchbox__content" id="sb-default"><form class="searchform" id="searchform" action="//rancher.my.org/parking.php" method="get" name="searchform"><input
                        class="container-searchbox__input"
                        type="hidden"
                        name="ses"
                        value="Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0xJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejYmdHJhY2txdWVyeT0x"><input
                        class="container-searchbox__input"
                        type="hidden"
                        name="token"
                        value=""><label
                        class="container-searchbox__searchtext-label"
                        for="searchtext">
                    Search
                </label><input
                        class="container-searchbox__input"
                        type="text"
                        name="keyword"
                        value=""
                        id="searchtext"
                        maxlength="70"><button
                        class="container-searchbox__button"
                        type="submit">
                    بحث
                </button></form></div></div><div class="container-buybox" id="container-buybox"></div><div class="container-footer" id="container-footer"><div class="container-disclaimer" id="container-disclaimer"><div class="container-disclaimer __content"><p class="container-disclaimer__ content-text">
        This webpage was generated by the domain owner using <a href="https://www.sedo.com/services/parking.php3">Sedo Domain Parking</a>. Disclaimer: Sedo maintains no relationship with third party advertisers. Reference to any specific service or trade mark is not controlled by Sedo nor does it constitute or imply its association, endorsement or recommendation.
    </p></div></div><div class="container-imprint" id="container-imprint"></div><div class="container-privacyPolicy" id="container-privacyPolicy"><div class="container-privacyPolicy__content"><div id="privacy-policy-link"><a
                class="container-privacyPolicy__content-link"
                href="#"
                onclick="window.open(
                        '//sedoparking.com/privacy-policy/us/index.html',
                        'privacy-policy','width=600,height=400');
                        return false;">
            نهج الخصوصية
        </a></div></div></div></div><!-- CAF --><script type="text/javascript">
                        if (dto.tsc) {
    var request = new XMLHttpRequest();
    request.open('GET', dto.pu + '/search/tsc.php?' + dto.tscQs);
    request.send();
}
* Connection #0 to host rancher.my.org left intact
            </script></body></html>

```

which I don’t know what to make of other than what appears to be **some sort of redirecting** towards an html page.  
I believe that **the Load balancer is set up correctly** , because it goes along with the example you shared. even more, when I browse [https://192.168.157.140](https://192.168.157.140), I get the **warning about the Rancher generated cert**. which implies that the load balancer is working properly.  
I also find it strange that after acknowledging the warning, I get :

> default backend - 404

and also :

```auto
~$ curl -vLk https://192.168.157.140
* Trying 192.168.157.140:443...
* TCP_NODELAY set
* Connected to 192.168.157.140 (192.168.157.140) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
* start date: Aug 6 15:16:44 2021 GMT
* expire date: Aug 6 15:16:44 2022 GMT
* issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x557b50b72e10)
> GET / HTTP/2
> Host: 192.168.157.140
> user-agent: curl/7.68.0
> accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 404
< server: openresty/1.15.8.1
< date: Mon, 16 Aug 2021 01:42:41 GMT
< content-type: text/plain; charset=utf-8
< content-length: 21
< strict-transport-security: max-age=15724800; includeSubDomains
<
* Connection #0 to host 192.168.157.140 left intact
default backend - 404

```

which I **don’t get when browsing** [http://rancher.my.org](http://rancher.my.org)  
I don’t know how to diagnose the situation, so I’d like to know what you think.  
I want to try excluding the load balancer by pointing the DNS to a node directly, I don’t know how but I’ll look it up.  
thanks for the assistance you have provided already, I’ve been learning through you.

---

<div class="post-metadata">

**Author:** ![kamal\_boumahdi](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/kamal_boumahdi/32/7837_2.png) [@kamal\_boumahdi](https://forums.suse.com/u/kamal_boumahdi)\
**Post date:** [August 31, 2021, 12:00pm UTC](https://forums.suse.com/t/liveness-probe-failed-connection-refused/20837/13 "2021-08-31T12:00:03Z")

</div>

Solved!  
To everyone who might run into this problem in the future.  
These **Liveliness probe** and **Readiness probe** kept failing due to connection being refused, which is due to a certain port being closed.  
You have to actually open them in order for connection to be accepted.  
You’ll find the list of all the ports that need to be opened as well as how to open them in this documentation page: [Rancher Docs: Requirements](https://rancher.com/docs/rke/latest/en/os/)  
PLEASE don’t repeat my mistake 🙂

---

<div class="post-metadata">

**Author:** ![Tulio\_Manrique](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/tulio_manrique/32/6821_2.png) [@Tulio\_Manrique](https://forums.suse.com/u/Tulio_Manrique)\
**Post date:** [October 12, 2021, 5:43pm UTC](https://forums.suse.com/t/liveness-probe-failed-connection-refused/20837/14 "2021-10-12T17:43:44Z")

</div>

could you tell me which ports was the problem,right now i have a cluster and the prometheus restart becasuse dont pass the health check but its fine when you go on the sites and saw the metrics

---

<div class="post-metadata">

**Author:** ![kamal\_boumahdi](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/kamal_boumahdi/32/7837_2.png) [@kamal\_boumahdi](https://forums.suse.com/u/kamal_boumahdi)\
**Post date:** [October 18, 2021, 12:09am UTC](https://forums.suse.com/t/liveness-probe-failed-connection-refused/20837/15 "2021-10-18T00:09:05Z")

</div>

if I remember correctly, it was 6443, 80 or 443

---

<div class="post-metadata">

**Author:** ![Ravi\_Sagar](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/ravi_sagar/32/9606_2.png) [@Ravi\_Sagar](https://forums.suse.com/u/Ravi_Sagar)\
**Post date:** [November 10, 2022, 1:11pm UTC](https://forums.suse.com/t/liveness-probe-failed-connection-refused/20837/16 "2022-11-10T13:11:14Z")

</div>

open the ports where?

---

<div class="post-metadata">

**Author:** ![user449993](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/user449993/32/9651_2.png) [@user449993](https://forums.suse.com/u/user449993)\
**Post date:** [November 23, 2022, 8:33am UTC](https://forums.suse.com/t/liveness-probe-failed-connection-refused/20837/17 "2022-11-23T08:33:37Z")

</div>

I have just encountered similar problems. in my case it was not the ports that I had to open but to add the configuration that is mentioned here for NetworkManager

[https://docs.rke2.io/known\_issues/#networkmanager](https://docs.rke2.io/known_issues/#networkmanager)

I also disabled selinux and rebooted the machine. after doing this, the errors with the liveness probes and other connectivity issues I’ve had disappeared.