K3s communication

1

Rocky 9.5 host and I get the following running the command? Aside from physical failure which I highly doubt. What should I be looking at. I need a deep dive rather than high level try this command and see. we’ve ran commands and it has to be some sort of config setting somewhere on bootup. Please excuse my ignorance as I am thrown into the fire here with Rancher and K3s.

Thank you in advance.

kubectl get pods -A
E0128 19:51:21.921560 1380560 memcache.go:265] “Unhandled Error” err=“couldn’t get current server API group list: Get "https://127.0.0.1:7443/api?timeout=32s\”: dial tcp 127.0.0.1:7443: connect: connection refused"
E0128 19:51:21.925291 1380560 memcache.go:265] “Unhandled Error” err=“couldn’t get current server API group list: Get : dial tcp 127.0.0.1:7443: connect: connection refused”
E0128 19:51:21.928657 1380560 memcache.go:265] “Unhandled Error” err=“couldn’t get current server API group list: Get : dial tcp 127.0.0.1:7443: connect: connection refused”
E0128 19:51:21.932073 1380560 memcache.go:265] “Unhandled Error” err=“couldn’t get current server API group list: Get : dial tcp 127.0.0.1:7443: connect: connection refused”

2

@polpmpn1 Hi and welcome to the Forum :smile:
So is k3s running on the system? Is your KUBECONFIG pointing to the correct file etc?

3

@malcolmlewis
systemctl status k3s
● k3s.service - Lightweight Kubernetes
Loaded: loaded (/etc/systemd/system/k3s.service; enabled; preset: disabled)
Drop-In: /usr/lib/systemd/system/service.d
└─10-timeout-abort.conf
Active: activating (auto-restart) (Result: exit-code) since Tue 2025-01-28>
Docs: https://k3s.io
Process: 1423954 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled →
Process: 1423956 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, sta>
Process: 1423958 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0>
Process: 1423960 ExecStart=/opt/k3s/default/bin/k3s $MODE $ARGS (code=exite>
Main PID: 1423960 (code=exited, status=1/FAILURE)
Tasks: 74
Memory: 889.0M (peak: 1.5G swap: 220.0K swap peak: 692.0K)
CPU: 220ms
CGroup: /system.slice/k3s.service
├─3731 /var/lib/rancher/k3s/data/c3b3bbf39f70985bb27180c065eb5928b>
├─3732 /var/lib/rancher/k3s/data/c3b3bbf39f70985bb27180c065eb5928b>
├─3762 /var/lib/rancher/k3s/data/c3b3bbf39f70985bb27180c065eb5928b>
├─3850 /var/lib/rancher/k3s/data/c3b3bbf39f70985bb27180c065eb5928b>
└─3893 /var/lib/rancher/k3s/data/c3b3bbf39f70985bb27180c065eb5928b>

4

@polpmpn1 for example;

kubectl get pods -A

E0128 16:23:20.138058   25397 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:6443/api?timeout=32s\": dial tcp 127.0.0.1:6443: connect: connection refused"
....
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?

export KUBECONFIG=$(pwd)/k3s.yaml

kubectl get pods -A

NAMESPACE             NAME                                                READY   STATUS             RESTARTS        AGE
cattle-fleet-system   fleet-agent-7f9ccfb8b-cntdr                         1/1     Running            7 (2m7s ago)    324d
....
5

systemctl status k3s
● k3s.service - Lightweight Kubernetes
Loaded: loaded (/etc/systemd/system/k3s.service; enabled; preset: disabled)
Drop-In: /usr/lib/systemd/system/service.d
└─10-timeout-abort.conf
Active: activating (auto-restart) (Result: exit-code) since Tue 2025-01-28>
Docs: https://k3s.io

6

@polpmpn1 So not running…

systemctl status k3s
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
     Active: active (running) since Tue 2025-01-28 16:22:00 CST; 10min ago

Is the k3s.service modified?

7

Most likely and if you could describe what you mean? What should I check? Like I said I am thrown into this fire so I assume things have been modified but I dont know what I dont know.

8

@polpmpn1 check if /etc/systemd/system/k3s.service exists, how was k3s installed (check history command).

9

My background is sysadmin in datacenters from a windows\linux virtualization perspective heavy focus on VMware. There is no VMware here. Just straight hardware with Rocky 9.5. Now if this makes sense, my host is "'salted with configurations why I assume modifications…

10

Yes it is there.

hybrid-sleep.target
k3s.service
multi-user.target.wants
network-online.target.wants
nginx.service.d
remote-fs.target.wants
salt-master.service.d
salt-minion.service.d
sleep.target
sockets.target.wants
suspend.target
suspend-then-hibernate.target
sysinit.target.wants
systemd-homed.service.wants
systemd-journald.service.wants
SyxsenseResponder.service
timers.target.wants
user@.service.wants
vmtoolsd.service.requires

11

@polpmpn1 really need to see how k3s was installed on the system, I see in your output a systemd override. Note I only run openSUSE and SLE…

There should be a /etc/rancher/k3s/k3s.yaml with a server: entry?

12

dir
config.yaml config.yaml.d k3s.env k3s.yaml

13

@polpmpn1

This file… k3s.yaml, but also inspect the /etc/systemd/system/k3s.service file. You need to look at the contents of both. But seems to me if k3s isn’t running…

14

k3s.yaml server: https://127.0.0.1:7443

k3s.service

cat k3s.service
[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
Wants=network-online.target
After=network-online.target

[Install]
WantedBy=multi-user.target

[Service]
Type=notify
EnvironmentFile=-/etc/default/%N
EnvironmentFile=-/etc/sysconfig/%N
EnvironmentFile=-/etc/systemd/system/k3s.service.env
EnvironmentFile=-/etc/rancher/k3s/k3s.env
KillMode=process
Delegate=yes

Having non-zero Limit*s causes performance problems due to accounting overhead

in the kernel. We recommend using cgroups to do container-local accounting.

LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
ExecStartPre=/bin/sh -xc ‘! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service 2>/dev/null’
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/opt/k3s/default/bin/k3s $MODE $ARGS

15

@polpmpn1 those two lines
Having non-zero… and in the kernel… are commented out correct with #

What are the $MODE $ARGS variables set to?

16

How do I check those variables? Again please excuse my ignorance

17

Yes they are commented out sorry

18

@polpmpn1 maybe those values are set via the salt config? Or some config file that is sourced (a Rocky linux thing perhaps…).

19

You are correct. Ok. That answers that. So then im setup fine with my node, this is a salt issue then correct? im geographically disparate from the Salt server and dont know the who what where on that. Is salt part of Rancher, K3s?

20

@polpmpn1 I’m assuming that $MODE is server, then $ARGS for the cluster, ip address and the plethora of options for k3s that can be configured. I suspect it’s not populating hence the k3s service is not starting?

Salt is a separate tool for configuration, management of a system.