Cattle Cluster Agent failing to get cacert

SUSE Rancher Prime
1

Cattle Cluster Agent keeps failing with
ERROR: Failed to pull the cacert from the rancher server settings at https://myserver/v3/settings/cacerts. I’m using self signed certs. I can get to the URL just fine in a browser. Any ideas why this might happen?

2

I am also getting this error on a new cluster, initialized from RKE.

After the cluster is initialized, I set the rancher-server admin password, navigate to the local cluster, navigate to Projects/Namespaces create a new Rancher-Server project, add the cattle-system namespace to it

And then I see this:

3

Similar to: https://github.com/rancher/rancher/issues/13615

4

It’s not similar, failed to pull or having an incorrect checksum are different things.

This is the code that’s being run (https://github.com/rancher/rancher/blob/master/package/run.sh#L142), what is the response when you run curl --insecure -s -fL $CATTLE_SERVER/v3/settings/cacerts inside the container?

5

The root cause in my case was that the DNS could not resolve any domain within the cattle-cluster-agent workload. I had been trying to install rancher on a baremetal Ubuntu Bionic Beaver (18.04) host, which ships with it’s own DNS servers (listening on 127.0.0.1:53) as part of the systemd-resolved service. This had resulted in a port-conflict with kube-dns since they were both trying to listen on port 53.

The problems with dns here manifested themselves as the Failed to pull the cacert from the rancher server settings at https://myserver/v3/settings/cacerts error since myserver could not be resolved.

–

To disable ubuntu’s default dns server to free up port 53, I ended up doing the following

  1. Add ‘127.0.0.1 localhost’ to /etc/hosts file
  2. update /etc/resolv.conf to only include:
    nameserver 8.8.8.8
    nameserver 8.8.4.4
  3. Run the following:
    sudo systemctl disable systemd-resolved.service
    sudo service systemd-resolved stop
6

Supported OS’ include:

Operating System
Ubuntu 16.04 (64-bit)
Red Hat Enterprise Linux 7.5 (64-bit)
RancherOS 1.3.0 (64-bit)

For tracking Ubuntu 18.04 support: https://github.com/rancher/rancher/issues/13888