# Rancher in CrashLoopBackOff cycle

**URL:** <https://forums.suse.com/t/rancher-in-crashloopbackoff-cycle/18921>\
**Category:** SUSE Rancher Prime\
**Created:** [November 30, 2020, 8:39pm UTC](https://forums.suse.com/t/rancher-in-crashloopbackoff-cycle/18921 "2020-11-30T20:39:36Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![funkymalc](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/funkymalc/32/6910_2.png) [@funkymalc](https://forums.suse.com/u/funkymalc)\
**Post date:** [November 30, 2020, 8:39pm UTC](https://forums.suse.com/t/rancher-in-crashloopbackoff-cycle/18921/1 "2020-11-30T20:39:36Z")

</div>

This morning I woke to discover Rancher 2.4.10 on K3 is not available. The K3 cluster seems to be ok, however, all pods in the cattle-system namespace are in a perpetual CrashLoopBackOff cycle:

NAME READY STATUS RESTARTS AGE  
cattle-node-agent-crq87 0/1 CrashLoopBackOff 61 4h51m  
cattle-node-agent-2mnbp 0/1 CrashLoopBackOff 50 3h52m  
rancher-56b887d7d7-zsqc8 0/1 Running 18 75m  
rancher-56b887d7d7-dlp6c 0/1 CrashLoopBackOff 18 75m  
rancher-56b887d7d7-9rn47 0/1 CrashLoopBackOff 18 75m  
cattle-cluster-agent-79467cd4b4-pr69p 0/1 CrashLoopBackOff 11 31m

The cattle node agents and cattle-cluster-agent logs show they are not able to connect to CATTLE\_SERVER. This would seem to make sense since none of the rancher pods have a ready status.

The rancher pods themselves have a log that looks like this (and elected leaders continually shifting):

> ```
> 2020/11/30 20:23:03 [INFO] Rancher version v2.4.10 (677ed79a3) is starting
> 
> 2020/11/30 20:23:03 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:auto Embedded:false HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:true AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features:}
> 
> 2020/11/30 20:23:03 [INFO] Listening on /tmp/log.sock
> 
> I1130 20:23:03.887028 7 http.go:122] HTTP2 has been explicitly disabled
> 
> 2020/11/30 20:23:04 [INFO] Starting API controllers
> 
> 2020/11/30 20:23:04 [INFO] Running in clustered mode with ID 10.42.0.123, monitoring endpoint cattle-system/rancher
> 
> 2020/11/30 20:23:05 [INFO] Starting API controllers
> 
> I1130 20:23:09.951631 7 http.go:122] HTTP2 has been explicitly disabled
> 
> I1130 20:23:09.952407 7 http.go:122] HTTP2 has been explicitly disabled
> 
> 2020/11/30 20:23:09 [INFO] Starting cluster controllers for local
> 
> 2020/11/30 20:23:10 [INFO] Starting cluster agent for local [owner=false]
> 
> 2020/11/30 20:23:10 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
> 
> 2020/11/30 20:23:10 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
> 
> I1130 20:23:19.533368 7 trace.go:116] Trace[1059014376]: "Reflector ListAndWatch" name:github.com/rancher/norman/controller/generic_controller.go:237 (started: 2020-11-30 20:23:05.07245314 +0000 UTC m=+1.476564592) (total time: 14.460864782s):
> 
> Trace[1059014376]: [14.39649303s] [14.39649303s] Objects listed
> 
> 2020/11/30 20:23:20 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
> 
> 2020/11/30 20:23:20 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
> 
> I1130 20:23:21.253718 7 leaderelection.go:242] attempting to acquire leader lease kube-system/cattle-controllers...
> 
> I1130 20:23:21.265434 7 leaderelection.go:252] successfully acquired lease kube-system/cattle-controllers
> 
> 2020/11/30 20:23:21 [INFO] Starting catalog controller
> 
> 2020/11/30 20:23:21 [INFO] Starting project-level catalog controller
> 
> 2020/11/30 20:23:21 [INFO] Starting cluster-level catalog controller
> 
> 2020/11/30 20:23:21 [INFO] Starting management controllers
> 
> 2020/11/30 20:23:21 [INFO] Refreshing all schemas
> 
> 2020/11/30 20:23:21 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
> 
> 2020/11/30 20:23:21 [INFO] Starting apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition controller
> 
> 2020/11/30 20:23:21 [INFO] Refreshing all schemas
> 
> 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind Binding
> 
> 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind ComponentStatus
> 
> 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind ConfigMap
> 
> 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind Endpoints
> 
> 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind Event
> 
> 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind LimitRange
> 
> 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind Namespace
> 
> 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind Node
> 
> 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind PersistentVolumeClaim
> 
> 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind PersistentVolume
> 
> 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind Pod
> 
> 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind PodTemplate
> 
> 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind ReplicationController
> 
> 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind ResourceQuota
> 
> 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind Secret
> 
> 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind ServiceAccount
> 
> 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind Service
> 
> 2020/11/30 20:23:21 [INFO] APIVersion apiregistration.k8s.io/v1 Kind APIService
> 
> 2020/11/30 20:23:21 [INFO] APIVersion apiregistration.k8s.io/v1beta1 Kind APIService
> 
> 2020/11/30 20:23:21 [INFO] APIVersion extensions/v1beta1 Kind Ingress
> 
> 2020/11/30 20:23:21 [INFO] APIVersion apps/v1 Kind ControllerRevision
> 
> 2020/11/30 20:23:21 [INFO] APIVersion apps/v1 Kind DaemonSet
> 
> 2020/11/30 20:23:21 [INFO] APIVersion apps/v1 Kind Deployment
> 
> 2020/11/30 20:23:21 [INFO] APIVersion apps/v1 Kind ReplicaSet
> 
> 2020/11/30 20:23:21 [INFO] APIVersion apps/v1 Kind StatefulSet
> 
> 2020/11/30 20:23:21 [INFO] APIVersion events.k8s.io/v1beta1 Kind Event
> 
> 2020/11/30 20:23:21 [INFO] APIVersion authentication.k8s.io/v1 Kind TokenReview
> 
> 2020/11/30 20:23:21 [INFO] APIVersion authentication.k8s.io/v1beta1 Kind TokenReview
> 
> 2020/11/30 20:23:21 [INFO] APIVersion authorization.k8s.io/v1 Kind LocalSubjectAccessReview
> 
> 2020/11/30 20:23:21 [INFO] APIVersion authorization.k8s.io/v1 Kind SelfSubjectAccessReview
> 
> 2020/11/30 20:23:21 [INFO] APIVersion authorization.k8s.io/v1 Kind SelfSubjectRulesReview
> 
> 2020/11/30 20:23:21 [INFO] APIVersion authorization.k8s.io/v1 Kind SubjectAccessReview
> 
> 2020/11/30 20:23:21 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind LocalSubjectAccessReview
> 
> 2020/11/30 20:23:21 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SelfSubjectAccessReview
> 
> 2020/11/30 20:23:21 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SelfSubjectRulesReview
> 
> 2020/11/30 20:23:21 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SubjectAccessReview
> 
> 2020/11/30 20:23:21 [INFO] APIVersion autoscaling/v1 Kind HorizontalPodAutoscaler
> 
> 2020/11/30 20:23:21 [INFO] APIVersion autoscaling/v2beta1 Kind HorizontalPodAutoscaler
> 
> 2020/11/30 20:23:21 [INFO] APIVersion autoscaling/v2beta2 Kind HorizontalPodAutoscaler
> 
> 2020/11/30 20:23:21 [INFO] APIVersion batch/v1 Kind Job
> 
> 2020/11/30 20:23:21 [INFO] APIVersion batch/v1beta1 Kind CronJob
> 
> 2020/11/30 20:23:21 [INFO] APIVersion certificates.k8s.io/v1beta1 Kind CertificateSigningRequest
> 
> 2020/11/30 20:23:21 [INFO] APIVersion networking.k8s.io/v1 Kind NetworkPolicy
> 
> 2020/11/30 20:23:21 [INFO] APIVersion networking.k8s.io/v1beta1 Kind Ingress
> 
> 2020/11/30 20:23:21 [INFO] APIVersion policy/v1beta1 Kind PodDisruptionBudget
> 
> 2020/11/30 20:23:21 [INFO] APIVersion policy/v1beta1 Kind PodSecurityPolicy
> 
> 2020/11/30 20:23:21 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind ClusterRoleBinding
> 
> 2020/11/30 20:23:21 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind ClusterRole
> 
> 2020/11/30 20:23:21 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind RoleBinding
> 
> 2020/11/30 20:23:21 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind Role
> 
> 2020/11/30 20:23:21 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind ClusterRoleBinding
> 
> 2020/11/30 20:23:21 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind ClusterRole
> 
> 2020/11/30 20:23:21 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind RoleBinding
> 
> 2020/11/30 20:23:21 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind Role
> 
> 2020/11/30 20:23:21 [INFO] APIVersion storage.k8s.io/v1 Kind CSINode
> 
> 2020/11/30 20:23:21 [INFO] APIVersion storage.k8s.io/v1 Kind StorageClass
> 
> 2020/11/30 20:23:21 [INFO] APIVersion storage.k8s.io/v1 Kind VolumeAttachment
> 
> 2020/11/30 20:23:21 [INFO] APIVersion storage.k8s.io/v1beta1 Kind CSIDriver
> 
> 2020/11/30 20:23:21 [INFO] APIVersion storage.k8s.io/v1beta1 Kind CSINode
> 
> 2020/11/30 20:23:21 [INFO] APIVersion storage.k8s.io/v1beta1 Kind StorageClass
> 
> 2020/11/30 20:23:21 [INFO] APIVersion storage.k8s.io/v1beta1 Kind VolumeAttachment
> 
> 2020/11/30 20:23:21 [INFO] APIVersion admissionregistration.k8s.io/v1 Kind MutatingWebhookConfiguration
> 
> 2020/11/30 20:23:21 [INFO] APIVersion admissionregistration.k8s.io/v1 Kind ValidatingWebhookConfiguration
> 
> 2020/11/30 20:23:21 [INFO] APIVersion admissionregistration.k8s.io/v1beta1 Kind MutatingWebhookConfiguration
> 
> 2020/11/30 20:23:21 [INFO] APIVersion admissionregistration.k8s.io/v1beta1 Kind ValidatingWebhookConfiguration
> 
> 2020/11/30 20:23:21 [INFO] APIVersion apiextensions.k8s.io/v1 Kind CustomResourceDefinition
> 
> 2020/11/30 20:23:21 [INFO] APIVersion apiextensions.k8s.io/v1beta1 Kind CustomResourceDefinition
> 
> 2020/11/30 20:23:21 [INFO] APIVersion scheduling.k8s.io/v1 Kind PriorityClass
> 
> 2020/11/30 20:23:21 [INFO] APIVersion scheduling.k8s.io/v1beta1 Kind PriorityClass
> 
> 2020/11/30 20:23:21 [INFO] APIVersion coordination.k8s.io/v1 Kind Lease
> 
> 2020/11/30 20:23:21 [INFO] APIVersion coordination.k8s.io/v1beta1 Kind Lease
> 
> 2020/11/30 20:23:21 [INFO] APIVersion node.k8s.io/v1beta1 Kind RuntimeClass
> 
> 2020/11/30 20:23:21 [INFO] APIVersion discovery.k8s.io/v1beta1 Kind EndpointSlice
> 
> 2020/11/30 20:23:21 [INFO] APIVersion helm.cattle.io/v1 Kind HelmChart
> 
> 2020/11/30 20:23:21 [INFO] APIVersion k3s.cattle.io/v1 Kind Addon
> 
> 2020/11/30 20:23:21 [INFO] APIVersion monitoring.coreos.com/v1 Kind PodMonitor
> 
> 2020/11/30 20:23:21 [INFO] APIVersion monitoring.coreos.com/v1 Kind PrometheusRule
> 
> 2020/11/30 20:23:21 [INFO] APIVersion monitoring.coreos.com/v1 Kind Alertmanager
> 
> 2020/11/30 20:23:21 [INFO] APIVersion monitoring.coreos.com/v1 Kind ServiceMonitor
> 
> 2020/11/30 20:23:21 [INFO] APIVersion monitoring.coreos.com/v1 Kind Prometheus
> 
> 2020/11/30 20:23:21 [INFO] APIVersion acme.cert-manager.io/v1alpha2 Kind Challenge
> 
> 2020/11/30 20:23:21 [INFO] APIVersion acme.cert-manager.io/v1alpha2 Kind Order
> 
> 2020/11/30 20:23:21 [INFO] APIVersion cert-manager.io/v1alpha2 Kind ClusterIssuer
> 
> 2020/11/30 20:23:21 [INFO] APIVersion cert-manager.io/v1alpha2 Kind CertificateRequest
> 
> 2020/11/30 20:23:21 [INFO] APIVersion cert-manager.io/v1alpha2 Kind Certificate
> 
> 2020/11/30 20:23:21 [INFO] APIVersion cert-manager.io/v1alpha2 Kind Issuer
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind GlobalRole
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterLogging
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind AuthConfig
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind CatalogTemplate
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ProjectLogging
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind TemplateContent
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ProjectCatalog
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind NodePool
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind CisConfig
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind GroupMember
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind MultiClusterAppRevision
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind GlobalDnsProvider
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind GlobalDns
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind RkeK8sServiceOption
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind CisBenchmarkVersion
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterTemplateRevision
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlertRule
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlert
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind RoleTemplate
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Token
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind MonitorMetric
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Preference
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Catalog
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlertGroup
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind DynamicSchema
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind RkeK8sSystemImage
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind User
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ProjectRoleTemplateBinding
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind RkeAddon
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind TemplateVersion
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind KontainerDriver
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind GlobalRoleBinding
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind CatalogTemplateVersion
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlertRule
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ProjectMonitorGraph
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterMonitorGraph
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Feature
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Group
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Template
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Project
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Cluster
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlertGroup
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind MultiClusterApp
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Notifier
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterRoleTemplateBinding
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind EtcdBackup
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ProjectNetworkPolicy
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind SamlToken
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind NodeTemplate
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind PodSecurityPolicyTemplateProjectBinding
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Setting
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterTemplate
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind UserAttribute
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterRegistrationToken
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind PodSecurityPolicyTemplate
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterScan
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterCatalog
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind NodeDriver
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlert
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Node
> 
> 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ComposeConfig
> 
> 2020/11/30 20:23:21 [INFO] APIVersion project.cattle.io/v3 Kind PipelineExecution
> 
> 2020/11/30 20:23:21 [INFO] APIVersion project.cattle.io/v3 Kind PipelineSetting
> 
> 2020/11/30 20:23:21 [INFO] APIVersion project.cattle.io/v3 Kind Pipeline
> 
> 2020/11/30 20:23:21 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeRepository
> 
> 2020/11/30 20:23:21 [INFO] APIVersion project.cattle.io/v3 Kind AppRevision
> 
> 2020/11/30 20:23:21 [INFO] APIVersion project.cattle.io/v3 Kind App
> 
> 2020/11/30 20:23:21 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeCredential
> 
> 2020/11/30 20:23:21 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeProviderConfig
> 
> 2020/11/30 20:23:21 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind NodeMetrics
> 
> 2020/11/30 20:23:21 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind PodMetrics
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=PersistentVolume
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for cert-manager.io/v1alpha2, Kind=ClusterIssuer
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectMonitorGraph
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlertRule
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=ServiceAccount
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodeTemplate
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=PersistentVolumeClaim
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for batch/v1, Kind=Job
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sServiceOption
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodeDriver
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for autoscaling/v1, Kind=HorizontalPodAutoscaler
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for networking.k8s.io/v1beta1, Kind=Ingress
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Template
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=CisConfig
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=PodSecurityPolicyTemplateProjectBinding
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for project.cattle.io/v3, Kind=PipelineSetting
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for project.cattle.io/v3, Kind=App
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=Secret
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=Event
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for networking.k8s.io/v1, Kind=NetworkPolicy
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlert
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sSystemImage
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRegistrationToken
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ComposeConfig
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for cert-manager.io/v1alpha2, Kind=Certificate
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for cert-manager.io/v1alpha2, Kind=Issuer
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=Pod
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for coordination.k8s.io/v1, Kind=Lease
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectLogging
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for certificates.k8s.io/v1beta1, Kind=CertificateSigningRequest
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for project.cattle.io/v3, Kind=Pipeline
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=LimitRange
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=Service
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalRoleBinding
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=ResourceQuota
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=PodMonitor
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplateVersion
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterScan
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectRoleTemplateBinding
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for policy/v1beta1, Kind=PodSecurityPolicy
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for storage.k8s.io/v1, Kind=CSINode
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for apps/v1, Kind=DaemonSet
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectCatalog
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for helm.cattle.io/v1, Kind=HelmChart
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for scheduling.k8s.io/v1, Kind=PriorityClass
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=PodTemplate
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=ServiceMonitor
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for project.cattle.io/v3, Kind=PipelineExecution
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=TemplateVersion
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for policy/v1beta1, Kind=PodDisruptionBudget
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for apps/v1, Kind=ControllerRevision
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlertRule
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=Namespace
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for cert-manager.io/v1alpha2, Kind=CertificateRequest
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalRole
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for apps/v1, Kind=ReplicaSet
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=DynamicSchema
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=RoleBinding
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalDns
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for events.k8s.io/v1beta1, Kind=Event
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=UserAttribute
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for storage.k8s.io/v1, Kind=StorageClass
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=ReplicationController
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=AuthConfig
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for batch/v1beta1, Kind=CronJob
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=TemplateContent
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Preference
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterCatalog
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Cluster
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=GroupMember
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Notifier
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeProviderConfig
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=Endpoints
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRoleTemplateBinding
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for apps/v1, Kind=StatefulSet
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=SamlToken
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterTemplateRevision
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterMonitorGraph
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=KontainerDriver
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Node
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=MonitorMetric
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplate
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Token
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlertGroup
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=User
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for discovery.k8s.io/v1beta1, Kind=EndpointSlice
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for storage.k8s.io/v1, Kind=VolumeAttachment
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for project.cattle.io/v3, Kind=AppRevision
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=ConfigMap
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlertGroup
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=PrometheusRule
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectNetworkPolicy
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for apps/v1, Kind=Deployment
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterTemplate
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=MultiClusterApp
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for apiextensions.k8s.io/v1, Kind=CustomResourceDefinition
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=MultiClusterAppRevision
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Setting
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for acme.cert-manager.io/v1alpha2, Kind=Order
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlert
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Feature
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for acme.cert-manager.io/v1alpha2, Kind=Challenge
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=Role
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for k3s.cattle.io/v1, Kind=Addon
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRole
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=RoleTemplate
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodePool
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=PodSecurityPolicyTemplate
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for node.k8s.io/v1beta1, Kind=RuntimeClass
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=CisBenchmarkVersion
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeCredential
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Catalog
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Group
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterLogging
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=EtcdBackup
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalDnsProvider
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeRepository
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeAddon
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=Alertmanager
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=Prometheus
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Project
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=Node
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for storage.k8s.io/v1beta1, Kind=CSIDriver
> 
> 2020/11/30 20:23:24 [INFO] Watching metadata for apiregistration.k8s.io/v1, Kind=APIService
> 
> I1130 20:23:51.218155 7 trace.go:116] Trace[664750235]: "Reflector ListAndWatch" name:github.com/rancher/norman/controller/generic_controller.go:237 (started: 2020-11-30 20:23:21.341298323 +0000 UTC m=+17.745409776) (total time: 29.876820003s):
> 
> Trace[664750235]: [29.82704505s] [29.82704505s] Objects listed
> 
> 2020/11/30 20:23:51 [INFO] update kontainerdriver rancherkubernetesengine
> 
> 2020/11/30 20:23:51 [INFO] update kontainerdriver baiducloudcontainerengine
> 
> 2020/11/30 20:23:51 [INFO] update kontainerdriver aliyunkubernetescontainerservice
> 
> 2020/11/30 20:23:51 [INFO] update kontainerdriver tencentkubernetesengine
> 
> 2020/11/30 20:23:51 [INFO] update kontainerdriver huaweicontainercloudengine
> 
> 2020/11/30 20:23:51 [INFO] update kontainerdriver oraclecontainerengine
> 
> 2020/11/30 20:23:51 [INFO] update kontainerdriver googlekubernetesengine
> 
> 2020/11/30 20:23:51 [INFO] update kontainerdriver amazonelasticcontainerservice
> 
> 2020/11/30 20:23:51 [INFO] update kontainerdriver azurekubernetesservice
> 
> 2020/11/30 20:23:51 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-nz6xz [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/kafka-test" not found
> 
> 2020/11/30 20:23:51 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-xrbjr [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:23:51 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-r9nwj [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:23:51 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-t86rm [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:23:51 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-f98ql [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:23:51 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-glm86 [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:23:51 [ERROR] UserAttributeController u-ixoevjmsq5 [mgmt-auth-userattributes-controller] failed with : refresh daemon not yet initialized
> 
> 2020/11/30 20:23:51 [INFO] Refreshing driverMetadata in 1440 minutes
> 
> 2020/11/30 20:23:51 [ERROR] UserAttributeController u-ygauthgjnd [mgmt-auth-userattributes-controller] failed with : refresh daemon not yet initialized
> 
> 2020/11/30 20:23:51 [ERROR] error parsing azure-group-cache-size, skipping update strconv.Atoi: parsing "": invalid syntax
> 
> 2020/11/30 20:23:51 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject github_team://2956598 with role project-member in namespace
> 
> 2020/11/30 20:23:51 [INFO] Reconciling GlobalRoles
> 
> 2020/11/30 20:23:51 [INFO] Starting management.cattle.io/v3, Kind=Cluster controller
> 
> 2020/11/30 20:23:51 [INFO] Reconciling RoleTemplates
> 
> 2020/11/30 20:23:51 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject github_team://2956598 with role project-member in namespace
> 
> 2020/11/30 20:23:52 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-nz6xz [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/kafka-test" not found
> 
> 2020/11/30 20:23:52 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-xrbjr [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:23:52 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-r9nwj [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:23:52 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-t86rm [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:23:52 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-f98ql [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:23:52 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-glm86 [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:23:52 [ERROR] UserAttributeController u-ixoevjmsq5 [mgmt-auth-userattributes-controller] failed with : refresh daemon not yet initialized
> 
> 2020/11/30 20:23:52 [ERROR] UserAttributeController u-ygauthgjnd [mgmt-auth-userattributes-controller] failed with : refresh daemon not yet initialized
> 
> 2020/11/30 20:23:52 [INFO] driverMetadata: refreshing data from upstream https://releases.rancher.com/kontainer-driver-metadata/release-v2.4/data.json
> 
> 2020/11/30 20:23:52 [INFO] Retrieve data.json from local path /var/lib/rancher-data/driver-metadata/data.json
> 
> 2020/11/30 20:23:52 [INFO] adding kontainer driver rancherKubernetesEngine
> 
> 2020/11/30 20:23:52 [INFO] adding kontainer driver googleKubernetesEngine
> 
> 2020/11/30 20:23:52 [INFO] adding kontainer driver azureKubernetesService
> 
> 2020/11/30 20:23:52 [INFO] adding kontainer driver amazonElasticContainerService
> 
> 2020/11/30 20:23:52 [INFO] adding kontainer driver baiducloudcontainerengine
> 
> 2020/11/30 20:23:52 [INFO] adding kontainer driver aliyunkubernetescontainerservice
> 
> 2020/11/30 20:23:52 [INFO] adding kontainer driver tencentkubernetesengine
> 
> 2020/11/30 20:23:52 [INFO] adding kontainer driver huaweicontainercloudengine
> 
> 2020/11/30 20:23:52 [INFO] adding kontainer driver oraclecontainerengine
> 
> 2020/11/30 20:23:52 [INFO] Rancher startup complete
> 
> 2020/11/30 20:23:53 [ERROR] UserAttributeController u-ixoevjmsq5 [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ixoevjmsq5-secret" not found
> 
> 2020/11/30 20:23:53 [ERROR] UserAttributeController u-ygauthgjnd [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ygauthgjnd-secret" not found
> 
> 2020/11/30 20:23:53 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-nz6xz [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/kafka-test" not found
> 
> 2020/11/30 20:23:53 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-xrbjr [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:23:53 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-r9nwj [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:23:53 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-t86rm [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:23:53 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-f98ql [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:23:53 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-glm86 [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:23:53 [INFO] Refreshing driverMetadata in 1440 minutes
> 
> 2020/11/30 20:23:53 [INFO] driverMetadata: refreshing data from upstream https://releases.rancher.com/kontainer-driver-metadata/release-v2.4/data.json
> 
> 2020/11/30 20:23:53 [INFO] Retrieve data.json from local path /var/lib/rancher-data/driver-metadata/data.json
> 
> 2020/11/30 20:23:55 [ERROR] UserAttributeController u-ixoevjmsq5 [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ixoevjmsq5-secret" not found
> 
> 2020/11/30 20:23:55 [ERROR] UserAttributeController u-ygauthgjnd [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ygauthgjnd-secret" not found
> 
> 2020/11/30 20:23:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-nz6xz [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/kafka-test" not found
> 
> 2020/11/30 20:23:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-xrbjr [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:23:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-r9nwj [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:23:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-t86rm [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:23:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-f98ql [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:23:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-glm86 [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:23:56 [INFO] kontainerdriver googlekubernetesengine listening on address 127.0.0.1:44949
> 
> 2020/11/30 20:23:56 [INFO] kontainerdriver googlekubernetesengine stopped
> 
> 2020/11/30 20:23:56 [INFO] dynamic schema for kontainerdriver googlekubernetesengine updating
> 
> 2020/11/30 20:23:56 [INFO] kontainerdriver azurekubernetesservice listening on address 127.0.0.1:44187
> 
> 2020/11/30 20:23:56 [INFO] kontainerdriver amazonelasticcontainerservice listening on address 127.0.0.1:44079
> 
> 2020/11/30 20:23:56 [INFO] kontainerdriver azurekubernetesservice stopped
> 
> 2020/11/30 20:23:56 [INFO] kontainerdriver amazonelasticcontainerservice stopped
> 
> 2020/11/30 20:23:56 [INFO] dynamic schema for kontainerdriver amazonelasticcontainerservice updating
> 
> 2020/11/30 20:23:56 [INFO] dynamic schema for kontainerdriver azurekubernetesservice updating
> 
> time="2020-11-30 20:23:57" level=info msg="Telemetry Client v0.5.12"
> 
> time="2020-11-30 20:23:57" level=info msg="Listening on 0.0.0.0:8114"
> 
> time="2020-11-30 20:23:57" level=info msg="Collecting anonymous data from https://localhost:443/v3"
> 
> time="2020-11-30 20:23:57" level=error msg="Error collecting data: Get https://localhost:443/v3: dial tcp [::1]:443: connect: connection refused"
> 
> 2020/11/30 20:23:59 [ERROR] UserAttributeController u-ixoevjmsq5 [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ixoevjmsq5-secret" not found
> 
> 2020/11/30 20:23:59 [ERROR] UserAttributeController u-ygauthgjnd [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ygauthgjnd-secret" not found
> 
> 2020/11/30 20:23:59 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-nz6xz [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/kafka-test" not found
> 
> 2020/11/30 20:23:59 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-xrbjr [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:23:59 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-r9nwj [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:23:59 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-t86rm [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:23:59 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-f98ql [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:23:59 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-glm86 [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> I1130 20:23:59.619824 7 trace.go:116] Trace[879221466]: "Reflector ListAndWatch" name:github.com/rancher/steve/pkg/clustercache/controller.go:164 (started: 2020-11-30 20:23:24.341763453 +0000 UTC m=+20.745874908) (total time: 35.278020551s):
> 
> Trace[879221466]: [35.232208593s] [35.232208593s] Objects listed
> 
> 2020/11/30 20:24:07 [ERROR] UserAttributeController u-ixoevjmsq5 [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ixoevjmsq5-secret" not found
> 
> 2020/11/30 20:24:07 [ERROR] UserAttributeController u-ygauthgjnd [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ygauthgjnd-secret" not found
> 
> 2020/11/30 20:24:07 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-nz6xz [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/kafka-test" not found
> 
> 2020/11/30 20:24:07 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-xrbjr [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:24:07 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-r9nwj [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:24:07 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-t86rm [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:24:07 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-f98ql [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:24:07 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-glm86 [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:24:17 [ERROR] ClusterController local exit status 1, The connection to the server localhost:443 was refused - did you specify the right host or port?
> 
> 2020/11/30 20:24:23 [ERROR] UserAttributeController u-ixoevjmsq5 [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ixoevjmsq5-secret" not found
> 
> 2020/11/30 20:24:23 [ERROR] UserAttributeController u-ygauthgjnd [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ygauthgjnd-secret" not found
> 
> 2020/11/30 20:24:23 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-nz6xz [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/kafka-test" not found
> 
> 2020/11/30 20:24:23 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-xrbjr [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:24:23 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-r9nwj [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:24:23 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-t86rm [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:24:23 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-f98ql [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:24:23 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-glm86 [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:24:34 [ERROR] ClusterController c-wh54c [cluster-deploy] failed with : Get https://172.20.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent?timeout=45s: waiting for cluster [c-wh54c] agent to connect
> 
> 2020/11/30 20:24:35 [ERROR] ClusterController c-cv652 [cluster-deploy] failed with : Get https://10.100.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent?timeout=45s: waiting for cluster [c-cv652] agent to connect
> 
> 2020/11/30 20:24:42 [ERROR] ClusterController local exit status 1, The connection to the server localhost:443 was refused - did you specify the right host or port?
> 
> 2020/11/30 20:24:51 [INFO] Stopping cluster agent for c-cv652
> 
> 2020/11/30 20:24:51 [INFO] Stopping cluster agent for c-wh54c
> 
> 2020/11/30 20:24:55 [ERROR] UserAttributeController u-ixoevjmsq5 [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ixoevjmsq5-secret" not found
> 
> 2020/11/30 20:24:55 [ERROR] UserAttributeController u-ygauthgjnd [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ygauthgjnd-secret" not found
> 
> 2020/11/30 20:24:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-nz6xz [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/kafka-test" not found
> 
> 2020/11/30 20:24:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-r9nwj [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:24:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-glm86 [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:24:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-xrbjr [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:24:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-t86rm [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:24:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-f98ql [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found
> 
> 2020/11/30 20:25:08 [ERROR] ClusterController local exit status 1, The connection to the server localhost:443 was refused - did you specify the right host or port?
> 
> 2020/11/30 20:25:21 [ERROR] ClusterController c-cv652 [cluster-deploy] failed with : Get https://10.100.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent?timeout=45s: waiting for cluster [c-cv652] agent to connect
> 
> 2020/11/30 20:25:21 [ERROR] ClusterController c-wh54c [cluster-deploy] failed with : Get https://172.20.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent?timeout=45s: waiting for cluster [c-wh54c] agent to connect
> 
> 2020/11/30 20:25:28 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=Role workers
> 
> 2020/11/30 20:25:28 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRole workers
> 
> 2020/11/30 20:25:28 [INFO] Shutting down RoleBindingController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down ClusterRoleBindingController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down EndpointsController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down ServiceController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down NamespaceController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down PodController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down APIServiceController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition workers
> 
> 2020/11/30 20:25:28 [INFO] Shutting down apiregistration.k8s.io/v1, Kind=APIService workers
> 
> 2020/11/30 20:25:28 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRole workers
> 
> 2020/11/30 20:25:28 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=Role workers
> 
> 2020/11/30 20:25:28 [INFO] Shutting down ClusterRoleController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down SourceCodeRepositoryController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down PipelineExecutionController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down SourceCodeCredentialController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down PipelineController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down AppController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down AuthConfigController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down DynamicSchemaController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down GroupController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down GroupMemberController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down RKEAddonController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down RKEK8sServiceOptionController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down RKEK8sSystemImageController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down GlobalDNSController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down MultiClusterAppRevisionController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down MultiClusterAppController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down PodSecurityPolicyTemplateController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down ProjectLoggingController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down NodeDriverController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down SecretController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down NamespaceController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down SecretController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down SecretController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down EndpointsController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down ClusterRoleBindingController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down CatalogTemplateController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down ProjectController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down NodePoolController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down NodeTemplateController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down SettingController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down ProjectCatalogController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down ClusterCatalogController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down CatalogController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down CatalogTemplateVersionController controller
> 
> 2020/11/30 20:25:28 [INFO] Shutting down FeatureController controller
> 
> 2020/11/30 20:25:28 [FATAL] leaderelection lost for cattle-controllers
> 
> ```

Any idea what might be causing this?