Hello - I’m running a Rancher RKE2 cluster which has rancher-monitoring helm chart installed.
I’ve installed Loki helm chart but when I try to add it as a Datasource into Grafana (that’s been deployed via ranching-monitoring), I always get the error ‘Unable to connect with Loki. Please check the server logs for more details.’
I’ve confirmed that DNS from the Grafana pod can successfully reach the Loki service.
I’ve confirmed that Loki service is up and responds to /ready URI
Can anyone suggest other things to check ?
I’m using rancher-monitoring helm version 105.1.1 and loki-stack-2.10.2
Specifically, for speed of testing, I just ran through the instructions on https://ranchergovernment.com/blog/enhancing-your-rancher-monitoring-experience-with-grafana-loki
To add some more detail, I’ve started looking at the Grafana logs
ie. kubectl logs -n cattle-monitoring-system -f rancher-monitoring-grafana-784dc4fc66-2hlbn grafana
and I can see the Loki healthcheck failing.
logger=tsdb.loki endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=checkHealth pluginId=loki dsName=Logging dsUID=PD3EF01B4A9C99103 uname=admin fromAlert=false t=2025-02-07T09:53:35.990652574Z level=info msg=“Prepared request to Loki” duration=32.201µs queriesLength=1 stage=prepareRequest runInParallel=false
logger=tsdb.loki endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=checkHealth pluginId=loki dsName=Logging dsUID=PD3EF01B4A9C99103 uname=admin fromAlert=false t=2025-02-07T09:53:35.994747732Z level=error msg=“Error received from Loki” duration=3.98997ms stage=databaseRequest statusCode=400 contentLength=65 start=1970-01-01T00:00:01Z end=1970-01-01T00:00:04Z step=1s query=vector(1)+vector(1) queryType=instant direction=backward maxLines=0 supportingQueryType=none lokiHost=loki.loki.svc.cluster.local:3100 lokiPath=/loki/api/v1/query status=error error=“parse error at line 1, col 1: syntax error: unexpected IDENTIFIER” statusSource=downstream
logger=tsdb.loki endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth t=2025-02-07T09:53:35.994849854Z level=error msg=“Loki health check failed” error=“error from loki: parse error at line 1, col 1: syntax error: unexpected IDENTIFIER”
logger=context userId=1 orgId=1 uname=admin t=2025-02-07T09:53:35.994915758Z level=info msg=“Request Completed” method=GET path=/api/datasources/uid/PD3EF01B4A9C99103/health status=400 remote_addr=172.24.100.124 time_ms=12 duration=12.158974ms size=106 referer=https://rancher.pub-mgmt.centralnicregistry.com/k8s/clusters/c-m-8p6wmbnm/api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-grafana:80/proxy/connections/datasources/edit/PD3EF01B4A9C99103 handler=/api/datasources/uid/:uid/health status_source=server
For anyone that should stumble across this and be scratching their head like me - I finally figured out the issue.
It was due to a mismatch in Loki / Grafana versions. Rancher monitoring helm chart was installing a fairly new version of Grafana, where as the helm chart for Loki was installing a fairly old version.
I’ve pinned Loki version in the values.yaml
loki:
image:
tag: 2.9.11
I would recommend doing it differently:
- first: use the loki chart from grafana ( which is at 6.x something) , the loki-stack-chart is deprecated
- consider using a standalone grafana for all custom work ( the version in rancher-monitoirng is quite old already)