# Using custom Docker log drivers with Rancher

**URL:** <https://forums.suse.com/t/using-custom-docker-log-drivers-with-rancher/2705>\
**Category:** Rancher 1.x\
**Created:** [May 6, 2016, 10:17am UTC](https://forums.suse.com/t/using-custom-docker-log-drivers-with-rancher/2705 "2016-05-06T10:17:34Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![alecghica](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/alecghica/32/1290_2.png) [@alecghica](https://forums.suse.com/u/alecghica)\
**Post date:** [May 6, 2016, 10:17am UTC](https://forums.suse.com/t/using-custom-docker-log-drivers-with-rancher/2705/1 "2016-05-06T10:17:34Z")

</div>

Right now, with Rancher 1.0, it is possible to use custom Docker log drivers with Rancher like described in this article: [http://blog.yjwong.name/2016/03/23/using-custom-docker-log-drivers-with-rancher](http://blog.yjwong.name/2016/03/23/using-custom-docker-log-drivers-with-rancher)

The problem is that when using custom log driver, like GELF, the logs from Rancher UI stop working and a error message is displayed:

- `Invalid Date Invalid Daterunning logs job: configured logging reader does not support reading`

It is possible to make both logs work in the same time?

---

<div class="post-metadata">

**Author:** ![johnrengelman](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/johnrengelman/32/134_2.png) [@johnrengelman](https://forums.suse.com/u/johnrengelman)\
**Post date:** [May 6, 2016, 8:23pm UTC](https://forums.suse.com/t/using-custom-docker-log-drivers-with-rancher/2705/2 "2016-05-06T20:23:41Z")

</div>

Somewhat kind of on this topic -  
I was hoping to be able to use a log driver and point it at another service that was running in Rancher, but that doesn’t work (probably for obvious reasons). I would be awesome though 🙂

---

<div class="post-metadata">

**Author:** ![demarant](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/demarant/32/497_2.png) [@demarant](https://forums.suse.com/u/demarant)\
**Post date:** [May 7, 2016, 9:40am UTC](https://forums.suse.com/t/using-custom-docker-log-drivers-with-rancher/2705/3 "2016-05-07T09:40:22Z")

</div>

there was a similar question in the forum here (still unanswered)

> [@Using log\_driver with Gelf instead of logspunk](http://forums.suse.com/t/using-log-driver-with-gelf-instead-of-logspunk/1852):
>
> Hi, I found this post [Docker Logging Driver](http://forums.suse.com/t/docker-logging-driver/125) but its closed, so I couldnt continue on there but its a related issue… It seems that using log\_driver with Gelf would be a good way to substitute the requirement of Logspout, wouldnt it? Has anyone attempted this? Where are rancher itnernal containers sending their logs? is there a way to configure these to also use a log driver? (maybe a cluster/env wide log driver option?) On a semi-related note, is it possible/planned to be able to change cata…

@alecghica maybe using rancher-compose directly (instead of docker-compose) would make it work?  
I am quoting “rancher-compose should already work with it” from @ibuildthecloud

> [@Docker Logging Driver](http://forums.suse.com/t/docker-logging-driver/125/2):
>
> You’re in luck, we already do! Rancher Labs has contributed a bunch to the log driver (syslog in 1.6 and rollover in 1.8) so we added support for those before docker-compose did. I don’t think there is support in the UI, but rancher-compose should already work with it. Please let me know if it doesn’t work.

hey rancher folks … could you please cast some light on this topic?

---

<div class="post-metadata">

**Author:** ![vincent](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/vincent/32/7156_2.png) [@vincent](https://forums.suse.com/u/vincent)\
**Post date:** [May 9, 2016, 5:47am UTC](https://forums.suse.com/t/using-custom-docker-log-drivers-with-rancher/2705/4 "2016-05-09T05:47:54Z")

</div>

I don’t know anything about gelf, but this sounds like the logging driver has no integration to read data back from it. Does `docker logs` on the host show the same thing? The only info we have to show comes from asking the docker daemon.

---

<div class="post-metadata">

**Author:** ![demarant](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/demarant/32/497_2.png) [@demarant](https://forums.suse.com/u/demarant)\
**Post date:** [May 9, 2016, 1:20pm UTC](https://forums.suse.com/t/using-custom-docker-log-drivers-with-rancher/2705/5 "2016-05-09T13:20:49Z")

</div>

@vincent:

the docker logs command on the host is also not able to show the logs.

docker-compose.yml

```
hello:
  image: hello-world
  log_driver: "gelf"
  log_opt:
    gelf-address: "udp://logcentral.eea.europa.eu:12201"

```

I run with pure docker-compose up -d and I get the following

**output from docker logs**

```
[marinis@dev-fra-02 gelf-test]$ docker logs gelftest_hello_1
"logs" command is supported only for "json-file" and "journald" logging drivers (got: gelf)

```

**output from docker-compose logs**

```
[marinis@dev-fra-02 gelf-test]$ docker-compose logs
Attaching to gelftest_hello_1
hello_1 | Error attaching: configured logging reader does not support reading
gelftest_hello_1 exited with code 0

```

**output from Rancher log console**

`Invalid Date Invalid Daterunning logs job: configured logging reader does not support reading`

We are on CentOS 7 kernel 3.10.0-327.13.1.el7.x86\_64  
Docker 1.10.3  
Rancher 1.0.1

so the question is: How can we centralise logs and at the same time been able to see them in the rancher UI log console? Is it supposed to work somehow?  
Any help appriciated

---

<div class="post-metadata">

**Author:** ![vincent](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/vincent/32/7156_2.png) [@vincent](https://forums.suse.com/u/vincent)\
**Post date:** [May 9, 2016, 4:49pm UTC](https://forums.suse.com/t/using-custom-docker-log-drivers-with-rancher/2705/6 "2016-05-09T16:49:35Z")

</div>

As the error says, the only 2 drivers that implement reading back log entries ( [https://github.com/docker/docker/blob/master/daemon/logger/jsonfilelog/read.go](https://github.com/docker/docker/blob/master/daemon/logger/jsonfilelog/read.go) ) are docker json and systemd journal. I can’t show you information we can’t get from Docker.

---

<div class="post-metadata">

**Author:** ![demarant](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/demarant/32/497_2.png) [@demarant](https://forums.suse.com/u/demarant)\
**Post date:** [May 9, 2016, 7:47pm UTC](https://forums.suse.com/t/using-custom-docker-log-drivers-with-rancher/2705/7 "2016-05-09T19:47:25Z")

</div>

@vincent  
thanks, it is clear now the limitation is at the docker daemon level. nothing about Rancher.

So I guess we should probably try journald and from journald to graylog ([as this post suggests](https://graylogadmin.wordpress.com/journald-to-graylog/)). Let’s see how that works.

---

<div class="post-metadata">

**Author:** ![demarant](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/demarant/32/497_2.png) [@demarant](https://forums.suse.com/u/demarant)\
**Post date:** [May 10, 2016, 1:56pm UTC](https://forums.suse.com/t/using-custom-docker-log-drivers-with-rancher/2705/9 "2016-05-10T13:56:09Z")

</div>

so I tested with journald and it works fine 😃

# TL;DR

choosing journald as log plugin for docker will make the logs appearing in all tools (docker logs, docker-compose logs, journalctl on host, syslog on host and be able to forward to graylog2 server)

# Details on how to set it up

this is what I did, so if others are looking at the same solution

**enabled forward to rsyslog from journald**  
`vi nano /etc/systemd/journald.conf`  
added the line:  
`ForwardToSyslog=yes`

**made syslog forward to external graylog2 server**  
(or any other centralised log management)  
Added the following in `/etc/rsyslog.conf`

```
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList # run asynchronously
$ActionResumeRetryCount -1 # infinite retries if host is down
*.* @@logcentral.eea.europa.eu:1514;RSYSLOG_SyslogProtocol23Format

```

restarted the rsyslog service.

**docker-compose example logging via journald**  
created a sample app with journald logging

```
hello:
  image: hello-world
  log_driver: "journald"

```

and when started you can see the logs both with `docker-compose logs`, `docker logs` or within Rancher UI.

You can also see them with `journalctl CONTAINER_NAME=<container_name>`

The only issue I found is that in graylog2 there is no info on from which container the logs comes from. One way to achieve this is to use the tag option as which will add some identification to each log line:

```
 log_opt:
    tag: "{{.ImageName}}\{{.Name}}\{{.ID}}"

```

BUT the tag option for journald has been added to docker 1.11, and Rancher only supports docker 1.10.x right now. Of course docker 1.11 will soon be supported looks like  
[https://github.com/rancher/rancher/issues/4491](https://github.com/rancher/rancher/issues/4491)

---

<div class="post-metadata">

**Author:** ![blaggacao](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/blaggacao/32/531_2.png) [@blaggacao](https://forums.suse.com/u/blaggacao)\
**Post date:** [August 2, 2016, 4:00am UTC](https://forums.suse.com/t/using-custom-docker-log-drivers-with-rancher/2705/10 "2016-08-02T04:00:04Z")

</div>

This feels somewhat “half the way” as it needs special provisioning for the host. By this we turn to configuration management systems again. Schedulers set out to deprecate them.

I’d rather see rancher to allow consuming logs from other services through a configurable plugin based architecture. As docker [logging option `tag`](https://docs.docker.com/engine/admin/logging/log_tags/) has a default or is otherwise configurable AND always known to rancher, the log identity can be easily detected in most systems.

---

<div class="post-metadata">

**Author:** ![kbroughton](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/kbroughton/32/1225_2.png) [@kbroughton](https://forums.suse.com/u/kbroughton)\
**Post date:** [January 25, 2017, 5:06pm UTC](https://forums.suse.com/t/using-custom-docker-log-drivers-with-rancher/2705/11 "2017-01-25T17:06:28Z")

</div>

I would like to see the setting available in rancher-compose.yml not just the docker-compose.yml as these responses suggest.

For splunk we have to add 5 lines per container. In a 20 container stack, that’s a lot of hand editing. Allowing over-ride with precedence docker-compose.yml beats rancher-compose.yml beats /etc/defaults/docker would be nice.