Issues creating k3s cluster

catsup2548 · September 3, 2025, 1:22am

I am working standing up a k3s cluster in my org. We are planning to move from 1 server to 3 in 1 datacenter and 3 more in another datacenter. However all of this is behind a forward proxy and nexus repo manager. I am runninginto some issues with the server 2 & 3. I was able to join them to the to the main server but I’m getting errors when I do a systemctl status.

I do have the proxy AND nexus repo manager configured so that k3s SHOULD be able to download docker images from nexus. However I’m not sure if I have it set up correctly.
Here is my k3s registries:

/etc/rancher/k3s/registries.yaml

mirrors:
  docker.io:
    endpoint:
      - http://nexus.example.com:9091
    tls:
      insecure_skip_verify: true
  ghcr.io:
    endpoint:
      - http://nexus.example.com:9092
    tls:
      insecure_skip_verify: true
  registry.redhat.io:
    endpoint:
      - http://nexus.example.com:9093
    tls:
      insecure_skip_verify: true

And then I also have my k3s service environment configured with the proxy JUST in case nexus fails.

 ~]$ sudo cat /etc/systemd/system/k3s.service.env
HTTP_PROXY=http://10.0.0.5:8080
HTTPS_PROXY=http://10.0.0.5:8080
NO_PROXY=127.0.0.1,localhost,.example.com,10.42.0.0/16,10.43.0.0/16 # Add your cluster's Pod and Service IP ranges

Also I don’t have ANY custom SSL certificates added to the k3s, JUST on traefik.

Here are the errors:

Sep 02 20:05:42 server02.example.com k3s[754194]: I0902 20:05:42.660396  754194 log.go:245] http: TLS handshake error from 10.1.148.98:29915: remote error: tls: bad certificate
Sep 02 20:05:57 server02.example.com k3s[754194]: I0902 20:05:57.669575  754194 log.go:245] http: TLS handshake error from 10.1.148.98:37006: remote error: tls: bad certificate
Sep 02 20:06:12 server02.example.com k3s[754194]: I0902 20:06:12.663560  754194 log.go:245] http: TLS handshake error from 10.1.148.98:16971: remote error: tls: bad certificate
Sep 02 20:06:27 server02.example.com k3s[754194]: I0902 20:06:27.664221  754194 log.go:245] http: TLS handshake error from 10.1.148.98:38023: remote error: tls: bad certificate
Sep 02 20:06:42 server02.example.com k3s[754194]: I0902 20:06:42.663246  754194 log.go:245] http: TLS handshake error from 10.1.148.98:55266: remote error: tls: bad certificate
Sep 02 20:06:57 server02.example.com k3s[754194]: I0902 20:06:57.669589  754194 log.go:245] http: TLS handshake error from 10.1.148.98:9662: remote error: tls: bad certificate
Sep 02 20:07:12 server02.example.com k3s[754194]: I0902 20:07:12.662554  754194 log.go:245] http: TLS handshake error from 10.1.148.98:1188: remote error: tls: bad certificate
Sep 02 20:07:27 server02.example.com k3s[754194]: I0902 20:07:27.659736  754194 log.go:245] http: TLS handshake error from 10.1.148.98:47421: remote error: tls: bad certificate
Sep 02 20:07:42 server02.example.com k3s[754194]: I0902 20:07:42.658798  754194 log.go:245] http: TLS handshake error from 10.1.148.98:37953: remote error: tls: bad certificate
Sep 02 20:07:57 server02.example.com k3s[754194]: I0902 20:07:57.660297  754194 log.go:245] http: TLS handshake error from 10.1.148.98:20757: remote error: tls: bad certificate

Sep 02 20:08:55 server02.example.com k3s[778434]: I0902 20:08:55.568980  778434 handler.go:288] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
Sep 02 20:08:55 server02.example.com k3s[778434]: time="2025-09-02T20:08:55-05:00" level=info msg="Started tunnel to 10.0.138.140:6443"
Sep 02 20:08:55 server02.example.com k3s[778434]: time="2025-09-02T20:08:55-05:00" level=info msg="Stopped tunnel to 127.0.0.1:6443"
Sep 02 20:08:55 server02.example.com k3s[778434]: time="2025-09-02T20:08:55-05:00" level=info msg="Connecting to proxy" url="wss://10.0.138.140:6443/v1-k3s/connect"
Sep 02 20:08:55 server02.example.com k3s[778434]: time="2025-09-02T20:08:55-05:00" level=info msg="Proxy done" err="context canceled" url="wss://127.0.0.1:6443/v1-k3s/connect"
Sep 02 20:08:55 server02.example.com k3s[778434]: time="2025-09-02T20:08:55-05:00" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF"
Sep 02 20:08:55 server02.example.com k3s[778434]: time="2025-09-02T20:08:55-05:00" level=info msg="Handling backend connection request [server02.example.com]"
Sep 02 20:08:55 server02.example.com k3s[778434]: time="2025-09-02T20:08:55-05:00" level=info msg="Remotedialer connected to proxy" url="wss://10.0.138.140:6443/v1-k3s/connect"
Sep 02 20:08:56 server02.example.com k3s[778434]: I0902 20:08:56.350793  778434 event.go:389] "Event occurred" object="server02.example.com" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="NodePasswordValidation>
Sep 02 20:08:57 server02.example.com k3s[778434]: I0902 20:08:57.663871  778434 log.go:245] http: TLS handshake error from 10.1.148.98:29426: remote error: tls: bad certificate

Sep 02 20:08:55 server02.example.com k3s[778434]: time="2025-09-02T20:08:55-05:00" level=info msg="Connecting to proxy" url="wss://10.0.138.140:6443/v1-k3s/connect"
Sep 02 20:08:55 server02.example.com k3s[778434]: time="2025-09-02T20:08:55-05:00" level=info msg="Proxy done" err="context canceled" url="wss://127.0.0.1:6443/v1-k3s/connect"
Sep 02 20:08:55 server02.example.com k3s[778434]: time="2025-09-02T20:08:55-05:00" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF"
Sep 02 20:08:55 server02.example.com k3s[778434]: time="2025-09-02T20:08:55-05:00" level=info msg="Handling backend connection request [server02.example.com]"
Sep 02 20:08:55 server02.example.com k3s[778434]: time="2025-09-02T20:08:55-05:00" level=info msg="Remotedialer connected to proxy" url="wss://10.0.138.140:6443/v1-k3s/connect"
Sep 02 20:08:56 server02.example.com k3s[778434]: I0902 20:08:56.350793  778434 event.go:389] "Event occurred" object="server02.example.com" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="NodePasswordValidation>
Sep 02 20:08:57 server02.example.com k3s[778434]: I0902 20:08:57.663871  778434 log.go:245] http: TLS handshake error from 10.1.148.98:29426: remote error: tls: bad certificate
Sep 02 20:09:01 server02.example.com k3s[778434]: I0902 20:09:01.335077  778434 event.go:389] "Event occurred" object="kube-system/traefik" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="UpdatedLoadBalancer" mes>
Sep 02 20:09:01 server02.example.com k3s[778434]: I0902 20:09:01.343773  778434 event.go:389] "Event occurred" object="kube-system/traefik" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="UpdatedLoadBalancer" mes>
Sep 02 20:09:12 server02.example.com k3s[778434]: I0902 20:09:12.668533  778434 log.go:245] http: TLS handshake error from 10.1.148.98:13663: remote error: tls: bad certificate
lines 1-30/30 (END)

Sep 02 20:08:55 server02.example.com k3s[778434]: time="2025-09-02T20:08:55-05:00" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF"
Sep 02 20:08:55 server02.example.com k3s[778434]: time="2025-09-02T20:08:55-05:00" level=info msg="Handling backend connection request [server02.example.com]"
Sep 02 20:08:55 server02.example.com k3s[778434]: time="2025-09-02T20:08:55-05:00" level=info msg="Remotedialer connected to proxy" url="wss://10.0.138.140:6443/v1-k3s/connect"
Sep 02 20:08:56 server02.example.com k3s[778434]: I0902 20:08:56.350793  778434 event.go:389] "Event occurred" object="server02.example.com" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="NodePasswordValidation>
Sep 02 20:08:57 server02.example.com k3s[778434]: I0902 20:08:57.663871  778434 log.go:245] http: TLS handshake error from 10.1.148.98:29426: remote error: tls: bad certificate
Sep 02 20:09:01 server02.example.com k3s[778434]: I0902 20:09:01.335077  778434 event.go:389] "Event occurred" object="kube-system/traefik" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="UpdatedLoadBalancer" mes>
Sep 02 20:09:01 server02.example.com k3s[778434]: I0902 20:09:01.343773  778434 event.go:389] "Event occurred" object="kube-system/traefik" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="UpdatedLoadBalancer" mes>
Sep 02 20:09:12 server02.example.com k3s[778434]: I0902 20:09:12.668533  778434 log.go:245] http: TLS handshake error from 10.1.148.98:13663: remote error: tls: bad certificate
Sep 02 20:09:19 server02.example.com k3s[778434]: I0902 20:09:19.572317  778434 handler.go:288] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
Sep 02 20:09:27 server02.example.com k3s[778434]: I0902 20:09:27.669220  778434 log.go:245] http: TLS handshake error from 10.1.148.98:64377: remote error: tls: bad certificate

Topic		Replies	Views
Unable to join k3s agent to to k3s server k3s, k3OS, and k3d	2	8611	September 18, 2021
K3s with etcd: nodes cannot connect to first node k3s, k3OS, and k3d	1	1248	October 15, 2020
First k3s cluster attempt on proxmox failed k3s, k3OS, and k3d	10	5252	December 30, 2021
K3s: x509: certificate is valid for 127.0.0.1, PublicIPv4, PublicIPv6 not Privatev4 k3s, k3OS, and k3d	2	2183	March 6, 2024
K3s dns resolution failure k3s, k3OS, and k3d	2	9639	December 19, 2022

Issues creating k3s cluster

Related topics