# Precise logging to Elasticsearch
**URL:** https://forums.suse.com/t/precise-logging-to-elasticsearch/17991
**Category:** SUSE Rancher Prime
**Created:** [July 20, 2020, 1:30pm UTC](https://forums.suse.com/t/precise-logging-to-elasticsearch/17991 "2020-07-20T13:30:02Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@Til\_bardaga](https://forums.suse.com/u/Til_bardaga)
#### Post date: [July 20, 2020, 1:30pm UTC](https://forums.suse.com/t/precise-logging-to-elasticsearch/17991/1 "2020-07-20T13:30:02Z")
**Background:**
I setup a Rancher 2.x cluster with RKE.
OS: CentOS 7
I am able to access the Rancher UI and deploy containers in my cluster.
I deployed an ELK stack on a separate server VM and I am trying to configure logging for my Rancher environment.
Elasticsearch, Logstash, Kibana versions: v7.8
[https://rancher.com/docs/rancher/v2.x/en/cluster-admin/tools/logging/](https://rancher.com/docs/rancher/v2.x/en/cluster-admin/tools/logging/)
[https://rancher.com/docs/rancher/v2.x/en/cluster-admin/tools/logging/elasticsearch/](https://rancher.com/docs/rancher/v2.x/en/cluster-admin/tools/logging/elasticsearch/)
**Requirements:**
I am trying to configure logging for Rancher Projects. I want my container logs to end up in Elasticsearch.
In order to do this we need 1 program to send the logs and 1 program to parse the logs or just 1 program that does both.
**Problems:**
Rancher 2.x forces fluentd to be used to send logs directly to Elasticsearch. It doesn’t allow us to send the logs to Logstash.
How can I parse the log messages before they are indexed into Elasticsearch.
If I send the log messages to Elasticsearch I am getting all of these fields and most of them are not that important for people to see:
> @timestamp
> \_id
> \_index
> \_score
> \_type
> my-test-field
> docker.container\_id
> kubernetes.container\_image
> kubernetes.container\_image\_id
> kubernetes.container\_name
> kubernetes.host
> kubernetes.labels.pod-template-hash
> kubernetes.labels.workload\_user\_cattle\_io/workloadselector
> kubernetes.master\_url
> kubernetes.namespace\_id
> kubernetes.namespace\_labels.cattle\_io/creator
> kubernetes.namespace\_labels.field\_cattle\_io/projectId
> kubernetes.namespace\_name
> kubernetes.pod\_id
> kubernetes.pod\_name
> log
> log\_type
> projectID
> stream
> tag
I would like to limit these default fields and only include a couple of them.
I would like to be able to parse my log messages so that additional fields can be generated in Elasticsearch based on the log messages. I usually do this with Logstash Grok patterns, but with Rancher this doesn’t seem possible.
What is the equivalent configuration for doing this in Rancher ?
There is a “Edit as a file” option for fluentd, but from what I can see only the output should be written there. Is it possible to write a full fluentd configuration with input, filter, parser, output plugins ?
Why are we not allowed to send the logs to Logstash for parsing before indexing ?
---
### Author:  [@owensj](https://forums.suse.com/u/owensj)
#### Post date: [August 9, 2021, 6:39pm UTC](https://forums.suse.com/t/precise-logging-to-elasticsearch/17991/2 "2021-08-09T18:39:26Z")
Did you ever find the solution to this problem?
---
### Author:  [@zaakiy](https://forums.suse.com/u/zaakiy)
#### Post date: [February 8, 2022, 8:14am UTC](https://forums.suse.com/t/precise-logging-to-elasticsearch/17991/3 "2022-02-08T08:14:48Z")
@Til_bardaga@owensj
You can now send to Logstash as syslog input. ~~See [Rancher Docs: Outputs and ClusterOutputs](https://rancher.com/docs/rancher/v2.5/en/logging/custom-resource-config/outputs/#output-to-syslog)~~
OK I’m an idiot, it’s not yet available.
Update: I’m going to try to send to Kafka, then pull from Kafka into Logstash. I’ve done it before, just not on Rancher. [Rancher Docs: Migrating to Rancher v2.5 Logging - Kafka](https://rancher.com/docs/rancher/v2.6/en/logging/migrating/#kafka)