Add new Logging driver "fluentd" #12876

tagomoris · 2015-04-29T15:35:08Z

This patch provides fluentd logging plugin for docker, which is mentioned at #12540.

How to confirm behavior of this patch/new logging driver.

Setup dev docker container
Build repository with this patch by make BINDDIR=. binary
Copy binary and execute it in debug mode docker -dD
Attach from external environment by docker exec -it kickass_heisenberg /bin/bash
Build/execute another docker container from other console, built on Dockerfile and configuration file below:

# sh
$ docker run --net=container:kickass_heisenberg tagomoris/docker-fluentd
# Dockerfile
FROM kiyoto/fluentd:0.10.56-2.1.1
MAINTAINER [email protected]
RUN mkdir /etc/fluent
ADD fluent.conf /etc/fluent/
ENTRYPOINT ["/usr/local/bin/fluentd", "-c", "/etc/fluent/fluent.conf"]
# fluent.conf
<source>
  type forward
</source>
<match docker.**>
  type stdout
</match>

Execute hello-world container on attached environment: docker run --log-driver=fluentd hello-world
Got logs

2015-04-20 09:32:27 +0000 [info]: starting fluentd-0.10.56
2015-04-20 09:32:27 +0000 [info]: reading config file path="/etc/fluent/fluent.conf"
2015-04-20 09:32:27 +0000 [info]: gem 'fluentd' version '0.10.56'
2015-04-20 09:32:27 +0000 [info]: using configuration file: <ROOT>
  <source>
    type forward
  </source>
  <match docker.**>
    type stdout
  </match>
</ROOT>
2015-04-20 09:32:27 +0000 [info]: adding source type="forward"
2015-04-20 09:32:27 +0000 [info]: adding match pattern="docker.**" type="stdout"
2015-04-20 09:32:27 +0000 [info]: listening fluent socket on 0.0.0.0:24224
2015-04-20 09:45:09 +0000 docker.483cb7e2cc42.stdout: {"source":"stdout","log":"Hello from Docker."}
2015-04-20 09:45:09 +0000 docker.483cb7e2cc42.stdout: {"source":"stdout","log":"This message shows that your installation appears to be working correctly."}
2015-04-20 09:45:09 +0000 docker.483cb7e2cc42.stdout: {"log":"","source":"stdout"}
2015-04-20 09:45:09 +0000 docker.483cb7e2cc42.stdout: {"source":"stdout","log":"To generate this message, Docker took the following steps:"}
2015-04-20 09:45:09 +0000 docker.483cb7e2cc42.stdout: {"source":"stdout","log":" 1. The Docker client contacted the Docker daemon."}
2015-04-20 09:45:09 +0000 docker.483cb7e2cc42.stdout: {"source":"stdout","log":" 2. The Docker daemon pulled the \"hello-world\" image from the Docker Hub."}
2015-04-20 09:45:09 +0000 docker.483cb7e2cc42.stdout: {"source":"stdout","log":"    (Assuming it was not already locally available.)"}
2015-04-20 09:45:09 +0000 docker.483cb7e2cc42.stdout: {"source":"stdout","log":" 3. The Docker daemon created a new container from that image which runs the"}
2015-04-20 09:45:09 +0000 docker.483cb7e2cc42.stdout: {"source":"stdout","log":"    executable that produces the output you are currently reading."}
2015-04-20 09:45:09 +0000 docker.483cb7e2cc42.stdout: {"source":"stdout","log":" 4. The Docker daemon streamed that output to the Docker client, which sent it"}
2015-04-20 09:45:09 +0000 docker.483cb7e2cc42.stdout: {"source":"stdout","log":"    to your terminal."}
2015-04-20 09:45:09 +0000 docker.483cb7e2cc42.stdout: {"source":"stdout","log":""}
2015-04-20 09:45:09 +0000 docker.483cb7e2cc42.stdout: {"source":"stdout","log":"To try something more ambitious, you can run an Ubuntu container with:"}
2015-04-20 09:45:09 +0000 docker.483cb7e2cc42.stdout: {"source":"stdout","log":" $ docker run -it ubuntu bash"}
2015-04-20 09:45:09 +0000 docker.483cb7e2cc42.stdout: {"source":"stdout","log":""}
2015-04-20 09:45:09 +0000 docker.483cb7e2cc42.stdout: {"source":"stdout","log":"For more examples and ideas, visit:"}
2015-04-20 09:45:09 +0000 docker.483cb7e2cc42.stdout: {"source":"stdout","log":" http://docs.docker.com/userguide/"}

LK4D4 · 2015-04-29T16:07:36Z

@tagomoris Overall looks cool. But I'd be much more eager to merge if for msgpack serialization were used something simpler then ugorji/go. Our logging structure is defined and we can generate serializer with msgp for example, but we need something like PostRaw([]byte) from fluentd library.

tagomoris · 2015-04-29T16:35:58Z

Hmm, I have 2 options:

rewrite fluent-logger-golang with simpler msgpack serializer library (but it runs slower than ugorji/go): github.com/vmihailenco/msgpack
send log events to Fluentd as JSON over TCP: that has less efficiency (in both CPU usage and network traffic)

Using JSON is much more simpler and easy to implement, but it requires more CPU usage if container generates large amount of logs.
@LK4D4 thoughts?

LK4D4 · 2015-04-29T16:42:23Z

@tagomoris what about PostRaw as I said and generated serializator in docker? we can disable ugorji/go with tag if we want. I don't see any particular problem with rewriting fluent-logger-golang only for our type of messages, which will be even faster then ugorji, but I want use community-provided solutions :)

Basically PostRaw is https://github.com/t-k/fluent-logger-golang/blob/master/fluent/fluent.go#L106 (body of else).

tagomoris · 2015-04-30T01:20:08Z

@LK4D4 make sense.
But Fluentd's all events are structured data (like JSON object), with tag & time, transferred in JSON/MessagePack. Fluentd doesn't have raw data in any way. It's biggest feature of Fluentd as a log collecting service.

tagomoris · 2015-04-30T01:25:37Z

Now I'm going to rewrite this patch with JSON serialization, without ugorji/go.

LK4D4 · 2015-04-30T02:49:32Z

@tagomoris So, whole problem, that we don't want to expose PostRaw from t-k/fluent-logger-golang because it "isn't right"?
Why we can't send to TCP msgpack serialized data? As I see t-k/fluent-logger-golang doing just this.
I'm not in particular happy with json too, I think we can find compromise.

tagomoris · 2015-04-30T03:30:51Z

There are no problem to send TCP msgpack serialized data, and to add methods like PostRaw on t-k/fluent-logger-golang (or its fork). Fluentd's will not break any compatibilities of its protocol.

Now I'm concerning about serializer, especially for msgpack.
@LK4D4 are you serious to write msgpack serializer in docker source tree only for this purpose?

I think that it's better to implement this feature based on JSON serialization at first, because JSON serializer is already used by jsonfilelog. And then we can re-fix this feature with msgpack serializer if someone complain its speed/efficiency.

LK4D4 · 2015-04-30T04:08:28Z

@tagomoris It is pretty easy to generate serializer if we know structure(and we know), and it will be superfast. Makes sense to make logs as fast as we can, it is most loaded part of docker. And generating serializer is very cheap: https://github.com/tinylib/msgp

tagomoris · 2015-04-30T04:29:02Z

@LK4D4 Ah, i got you mentioned finally!
OK, I'll add PostRaw -like method and then fix this patch without ugorji/go. Please wait a litte...

tagomoris · 2015-05-05T05:46:28Z

Updated patch w/ fluent-logger-golang v0.6.0, which depends on msgp and serializer code by msgp.
We (Fluentd community) decides to change serializer library from ugorji/go/code to msgp for its lightweightness. (And now fluent-logger-golang is hosted on fluent organization.)

Now fluent-logger-golang has also PostRaw method, but there's no difference between using PostRaw w/ pre-serialized data and just using PostWithTime.

@LK4D4 Could you review once more? (only dependencies are changed)

fw42 · 2015-05-07T18:51:07Z

2015-04-20 09:45:09 +0000 docker.483cb7e2cc42.stdout: ...

Would it be possible to also include the container name (not just container id) in the tag?

LK4D4 · 2015-05-07T18:56:00Z

hack/vendor.sh

+
+clone git github.com/fluent/fluent-logger-golang v0.6.0
+
+# get Go tip's archive/tar, for xattr support and improved performance


We removed this stuff.

Oh, i got a mistake on rebase operation of this file. I'll rebase this again and push it.

tagomoris · 2015-05-08T06:17:55Z

@fw42 yes, tag can contain any strings, including container name. But too long/complex tags are not good for users in general.
How about to add container_id and container_name attributes in log data to show log sources to users? (like journald logging driver does)

fw42 · 2015-05-08T12:32:36Z

How about to add (...) container_name attributes in log data to show log sources to users?

Hmm you mean the application itself should include the container name in it's own logs (in every single line)? That seems like an anti-pattern to me. I don't want my application to (have to) know that it is even running inside a container.

fw42 · 2015-05-08T13:46:12Z

@tagomoris: To clarify the use-case, I think it would be nice to be able to use different fluentd output plugins based on the container name. If the container name were part of the fluentd tag, that would be pretty easy. Do you have another idea?

tagomoris · 2015-05-09T10:27:13Z

@fw42

Hmm you mean the application itself should include the container name in it's own logs (in every single line)?

No. Logging driver get container name and id when initialized, like journald logger. I pushed a commit to do so.
Application doesn't have to take care about it. (I paste example log w/ container_id and container_name below)

2015-05-09 10:17:21 +0000 docker.a4da785a4379.stdout: {"source":"stdout","log":"Hello from Docker.","container_id":"a4da785a4379b10d3d2789084d788099c7322b6ac055526937badbadf8df83a3","container_name":"/modest_franklin"}

I think it would be nice to be able to use different fluentd output plugins based on the container name. If the container name were part of the fluentd tag, that would be pretty easy.

Hmm, it seems true for me too.
But at the same time, Fluentd's tags are likely used to format file paths on filesystems. Docker container name includes / char, so using container name in tags will make many troubles on filesystems.
I have no idea how we should handle it. Should we remove /? or replace with any other chars?

FYI: There're some plugins like rewrite-tag-filter to route logs by values of field specified by configuration. So users can use different output plugins for different containers by using these plugins, in ways less easier than just using tags.

tagomoris · 2015-05-09T23:17:49Z

It will be configurable with --log-opt option to select ID or Name for tags (and tag prefix, now it is a constant docker).

thaJeztah · 2015-05-10T02:49:45Z

@tagomoris oh, almost forgot; there's another PR that implements changing the "tag" (to use, e.g. the container-name) for the syslog driver.

It hasn't been approved yet, and needs a rebase (because the --log-opt PR (#12422) was merged)

You can find that PR here; #12668. I think it would make sense to make sure both drivers use the same approach. (Not taking a position here which approach that would be)

tmatsuo · 2015-06-24T16:59:00Z

docs/reference/logging/fluentd.md

+
+    # docker run -it -p 24224:24224 -v /path/to/conf/test.conf:/fluentd/etc -e FLUENTD_CONF=test.conf fluent/fluentd:latest
+
+Then, start any containers you want, with `fluentd` logging driver:


Remove the comma after Then?

tmatsuo · 2015-06-24T17:07:11Z

Looks better!
put some more comments

tagomoris · 2015-06-24T18:12:54Z

Add a commit and pushed.

tmatsuo · 2015-06-24T18:22:30Z

Thanks @tagomoris
The document looks great to me (I'm a novice on docker, but I can understand the document easily) 👍

@LK4D4 et al. Maybe it's your turn to do the docs review.

moxiegirl · 2015-06-25T11:24:53Z

docs/reference/run.md

@@ -932,6 +932,16 @@ driver to a GELF remote server at `192.168.0.42` on port `12201`

 The `gelf-tag` option specifies a tag for easy container identification.

+#### Logging driver: fluentd


This has some missing tagging and an awkward construction in the second option.

--> https://gist.github.com/moxiegirl/62fd3d0a52d6fb1d4853

Logging driver: fluentd

Fluentd logging driver for Docker. Writes log messages to fluentd (forward input). The docker logs
command is not available for this logging driver.

You can use the --log-opt NAME=VALUE flag to specify these additional Fluentd logging driver options.

fluentd-address: specify host:port to connect [localhost:24224]

fluentd-tag: specify tag for fluentd message,

When specifying a fluentd-tag value, you can use the following markup tags:

{{.ID}}: short container id (12 characters)

{{.FullID}}: full container id

{{.Name}}: container name

For example, to specify both additional options:

docker run --log-driver=fluentd --log-opt fluentd-address=localhost:24224 --log-opt fluentd-tag=docker.{{.Name}}

If container cannot connect to the Fluentd daemon on the specified address, the container stops
immediately.

moxiegirl · 2015-06-26T01:01:24Z

@tagomoris Apologies for stepping on your PR. I wanted to test the new organization and I knew your PR was in docs/review and almost ready to merge.

Signed-off-by: TAGOMORI Satoshi <[email protected]>

tagomoris · 2015-06-26T02:10:15Z

@moxiegirl @LK4D4 rebased again, on master branch, which already has document file for this driver.

moxiegirl · 2015-06-26T03:28:45Z

Thanks @tagomoris for being patient. LGTM

moxiegirl · 2015-06-26T03:29:30Z

@LK4D4 ready for merge at will

Add new Logging driver "fluentd"

moxiegirl · 2015-06-26T16:12:43Z

@LK4D4 ty. @tagomoris Thank you for this contribution. We really appreciate your work.

repeatedly · 2015-06-26T16:22:17Z

👍

edsiper · 2015-06-26T16:27:06Z

👍

thaJeztah · 2015-06-26T20:20:17Z

Yes, merged! Thanks so much @tagomoris sorry this took so long

tmatsuo · 2015-06-26T20:35:36Z

Great work @tagomoris (moris-san) 💯

tagomoris · 2015-06-27T04:26:55Z

Thank you so much! I'm very looking forward the release of next version!

kzk · 2015-06-27T14:51:03Z

Great job @tagomoris and Docker team! 👍

graemej · 2015-06-27T15:29:58Z

Thanks @tagomoris 😄 we've been looking forward to this!

- Add fluentd logging driver to zsh completion moby#12876 - Add inspect --type flag to zsh completion moby#13187 - Respect -H option in zsh completion moby#13195 - Fix number of argument limit for pause and unpause in zsh completion Signed-off-by: Steve Durrheimer <[email protected]>

GordonTheTurtle added the status/0-triage label Apr 29, 2015

tagomoris mentioned this pull request Apr 29, 2015

Proposal: Add Logging driver for fluentd #12540

Closed

tagomoris force-pushed the logger-driver-fluentd branch 2 times, most recently from cdf1f68 to 05df86d Compare April 29, 2015 15:50

icecrime added status/1-design-review and removed status/0-triage labels Apr 29, 2015

tagomoris force-pushed the logger-driver-fluentd branch from 05df86d to 8726745 Compare May 5, 2015 05:20

LK4D4 reviewed May 7, 2015
View reviewed changes

tagomoris force-pushed the logger-driver-fluentd branch from 8726745 to ec5ba89 Compare May 8, 2015 06:37

tmatsuo reviewed Jun 24, 2015
View reviewed changes

moxiegirl reviewed Jun 25, 2015
View reviewed changes

moxiegirl mentioned this pull request Jun 25, 2015

Breaking logging driver material out of run #14182

Merged

tagomoris added 2 commits June 26, 2015 11:03

Add new logging driver: fluentd

361a582

Signed-off-by: TAGOMORI Satoshi <[email protected]>

Update English expressions by review comments

78f4a9d

Signed-off-by: TAGOMORI Satoshi <[email protected]>

tagomoris force-pushed the logger-driver-fluentd branch from 779dab9 to 78f4a9d Compare June 26, 2015 02:09

moxiegirl added status/4-merge and removed status/3-docs-review labels Jun 26, 2015

LK4D4 added a commit that referenced this pull request Jun 26, 2015

Merge pull request #12876 from tagomoris/logger-driver-fluentd

b96f109

Add new Logging driver "fluentd"

LK4D4 merged commit b96f109 into moby:master Jun 26, 2015

thaJeztah added the impact/changelog label Jul 25, 2015


		clone git github.com/fluent/fluent-logger-golang v0.6.0

		# get Go tip's archive/tar, for xattr support and improved performance


		# docker run -it -p 24224:24224 -v /path/to/conf/test.conf:/fluentd/etc -e FLUENTD_CONF=test.conf fluent/fluentd:latest

		Then, start any containers you want, with `fluentd` logging driver:

		@@ -932,6 +932,16 @@ driver to a GELF remote server at `192.168.0.42` on port `12201`

		The `gelf-tag` option specifies a tag for easy container identification.

		#### Logging driver: fluentd

Add new Logging driver "fluentd" #12876

Add new Logging driver "fluentd" #12876

Conversation

tagomoris commented Apr 29, 2015

LK4D4 commented Apr 29, 2015

tagomoris commented Apr 29, 2015

LK4D4 commented Apr 29, 2015

tagomoris commented Apr 30, 2015

tagomoris commented Apr 30, 2015

LK4D4 commented Apr 30, 2015

tagomoris commented Apr 30, 2015

LK4D4 commented Apr 30, 2015

tagomoris commented Apr 30, 2015

tagomoris commented May 5, 2015

fw42 commented May 7, 2015

LK4D4 May 7, 2015

Choose a reason for hiding this comment

tagomoris May 8, 2015

Choose a reason for hiding this comment

tagomoris commented May 8, 2015

fw42 commented May 8, 2015

fw42 commented May 8, 2015

tagomoris commented May 9, 2015

tagomoris commented May 9, 2015

thaJeztah commented May 10, 2015

tmatsuo Jun 24, 2015

Choose a reason for hiding this comment

tmatsuo commented Jun 24, 2015

tagomoris commented Jun 24, 2015

tmatsuo commented Jun 24, 2015

moxiegirl Jun 25, 2015

Choose a reason for hiding this comment

Logging driver: fluentd

moxiegirl commented Jun 26, 2015

tagomoris commented Jun 26, 2015

moxiegirl commented Jun 26, 2015

moxiegirl commented Jun 26, 2015

moxiegirl commented Jun 26, 2015

repeatedly commented Jun 26, 2015

edsiper commented Jun 26, 2015

thaJeztah commented Jun 26, 2015

tmatsuo commented Jun 26, 2015

tagomoris commented Jun 27, 2015

kzk commented Jun 27, 2015

graemej commented Jun 27, 2015