# Rancher flooding logs with errors

**URL:** <https://forums.suse.com/t/rancher-flooding-logs-with-errors/39653>\
**Category:** SUSE Rancher Prime\
**Created:** [December 13, 2022, 3:09pm UTC](https://forums.suse.com/t/rancher-flooding-logs-with-errors/39653 "2022-12-13T15:09:32Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![mloibl](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/mloibl/32/9686_2.png) [@mloibl](https://forums.suse.com/u/mloibl)\
**Post date:** [December 13, 2022, 3:09pm UTC](https://forums.suse.com/t/rancher-flooding-logs-with-errors/39653/1 "2022-12-13T15:09:32Z")

</div>

We are currently using Rancher version 2.6.9. While debugging some oidc errors we are experiencing trouble with our error logs: They are being flooded by error messages from which we are unable to identify the source of.

You may find the logs of the last 5 minutes in the [gist](https://gist.github.com/mloibl/6bc609c932088a3581e8c971cd00f203).

We could identify several types of errors:

`error syncing 'p-n7k7r/creator-project-owner': handler mgmt-auth-prtb-controller: clusters.management.cattle.io "c-fvg4w" not found, requeuing`

`error syncing 'p-pqzmm/creator-project-owner': handler auth-prov-v2-prtb: failed to update fleet-local/r-cluster-local-view-p-pqzmm-creator-project-owner-nk3rmcfzaj rbac.authorization.k8s.io/v1, Kind=RoleBinding for auth-prov-v2-prtb-rolebinding p-pqzmm/creator-project-owner: RoleBinding.rbac.authorization.k8s.io "r-cluster-local-view-p-pqzmm-creator-project-owner-nk3rmcfzaj" is invalid: [metadata.ownerReferences.apiVersion: Invalid value: "": version must not be empty, metadata.ownerReferences.kind: Invalid value: "": kind must not be empty, metadata.ownerReferences.name: Invalid value: "": name must not be empty], requeuing`

`error syncing 'grb-ftw5p': handler grb-cluster-sync: Index with name by-cluster does not exist, requeuing`

`error syncing 'c-p6msc/p-jq749': handler system-image-upgrade-controller: upgrade cluster c-p6msc system service alerting failed: template system-library-rancher-monitoring incompatible with rancher version or cluster's [c-p6msc] kubernetes version, requeuing`

From our understanding there could be some jobs running in the background referencing already deleted objects. Any suggestions how we could clean this up?

Kind regards

Moritz

---

<div class="post-metadata">

**Author:** ![daemonadmin](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/daemonadmin/32/9861_2.png) [@daemonadmin](https://forums.suse.com/u/daemonadmin)\
**Post date:** [March 28, 2023, 1:51pm UTC](https://forums.suse.com/t/rancher-flooding-logs-with-errors/39653/2 "2023-03-28T13:51:06Z")

</div>

We have the same issue. Any suggestions will be appreciated.

---

<div class="post-metadata">

**Author:** ![spirilis](https://avatars.discourse-cdn.com/v4/letter/s/8e8cbc/32.png) [@spirilis](https://forums.suse.com/u/spirilis)\
**Post date:** [March 28, 2023, 6:36pm UTC](https://forums.suse.com/t/rancher-flooding-logs-with-errors/39653/3 "2023-03-28T18:36:19Z")

</div>

I have something vaguely similar spamming my logs with 2.6.11, prtb-related:

2023/03/28 18:35:21 [ERROR] error syncing ‘p-89qlv/prtb-z88r4’: handler mgmt-auth-prtb-controller: cannot determine project and cluster from p-89qlv, requeuing  
2023/03/28 18:35:21 [ERROR] error syncing ‘p-89qlv/prtb-l4lfp’: handler mgmt-auth-prtb-controller: cannot determine project and cluster from p-89qlv, requeuing  
2023/03/28 18:35:21 [ERROR] error syncing ‘p-89qlv/prtb-2rqq7’: handler mgmt-auth-prtb-controller: cannot determine project and cluster from p-89qlv, requeuing  
2023/03/28 18:35:21 [ERROR] error syncing ‘p-89qlv/prtb-plm5c’: handler mgmt-auth-prtb-controller: cannot determine project and cluster from p-89qlv, requeuing

---

<div class="post-metadata">

**Author:** ![spirilis](https://avatars.discourse-cdn.com/v4/letter/s/8e8cbc/32.png) [@spirilis](https://forums.suse.com/u/spirilis)\
**Post date:** [March 28, 2023, 6:47pm UTC](https://forums.suse.com/t/rancher-flooding-logs-with-errors/39653/4 "2023-03-28T18:47:37Z")

</div>

Fwiw- it looks like these prtb’s are from a project that was deleted, but the project namespace (in the rancher-hosting RKE cluster) is still in a “Terminating” state. Guess this is remnants of an old bug. I am going to try deleting the remaining resources in that namespace.

---

<div class="post-metadata">

**Author:** ![spirilis](https://avatars.discourse-cdn.com/v4/letter/s/8e8cbc/32.png) [@spirilis](https://forums.suse.com/u/spirilis)\
**Post date:** [March 28, 2023, 7:12pm UTC](https://forums.suse.com/t/rancher-flooding-logs-with-errors/39653/5 "2023-03-28T19:12:03Z")

</div>

Well if anyone else encounters the specific errors I found, it turns out someone deleted some projects in the cluster and the project-namespaces (IN THE CLUSTER HOSTING RANCHER) were hanging around because the mgmt-auth-prtb-controller finalizer couldn’t complete because the “projectName:” field in the prtb object had the project ID, but not the cluster.

Editing the YAML for the prtb and prepending c-(clusterID): to the project ID in the projectName: field cleared the backlog and now all those project-namespaces successfully terminated, and the logs quit spamming Rancher’s pods.

E.g.:

```auto
apiVersion: management.cattle.io/v3
kind: ProjectRoleTemplateBinding
metadata:
  annotations:
    field.cattle.io/creatorId: user-hkvlr
  creationTimestamp: "2020-01-30T15:58:43Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2020-03-16T13:51:30Z"
  finalizers:
  - controller.cattle.io/mgmt-auth-prtb-controller
  generation: 3
  labels:
    cattle.io/creator: norman
  name: prtb-z8vjh
  namespace: p-r7hr6
  resourceVersion: "50750630"
  uid: a18bd8e2-f936-468a-ab63-e4d04e757472
projectName: p-r7hr6
roleTemplateName: project-owner
userName: u-x9pbj
userPrincipalName: local://u-x9pbj

```

Changing that “projectName” field to:

```auto
projectName: c-c4hlm:p-r7hr6

```

did the trick.

Quick for/awk/sed script to rip through a terminating namespace’s PRTBs to fix:

```auto
K8SNS="p-r7hr6"
for prtb in `kubectl -n $K8SNS get projectroletemplatebindings.v3.management.cattle.io | grep ^prtb- | awk '{print $1}'`; do
    kubectl -n $K8SNS get projectroletemplatebindings.v3.management.cattle.io $prtb -o yaml > tmp.yml
    sed -i 's/^projectName: \(.*\)$/projectName: c-c4hlm:\1/' tmp.yml
    kubectl -n $K8SNS apply -f tmp.yml
done

```

Replace the “c-c4hlm” with the correct cluster ID as needed.