Fluentd meetup #2

Log everyting in JSON.
Treasuare Data, Inc.
Sadayuki Furuhashi

Self-introduction
> Sadayuki Furuhashi
twitter: @frsyuki

> Original author of Fluentd
> Treasure Data, Inc.
Software Architect; Founder

> open-source
MessagePack - efficient serialization format

0. Why logging?

1. Why Fluentd? - Design of Fluentd
> Extensibility
> Unified log format
> Simplicity

2. Who uses Fluentd?

3. Future of Fluentd

0. Why logging?

> Error notifications
> Performance monitoring
> User segment analysis
> Funnel analysis
> Heatmap analysis
> Market prediction
etc...

0. Why logging? - Error notifications

Error!

0. Why logging? - Performance monitor

0. Why logging? - User segment analysis

0. Why logging? - Funnel analysis

-28%!
-27%!

0. Why logging? - Heatmap analysis

0. Why logging? - Market prediction

Alerting
Nagios

Analysis
MongoDB
MySQL
Hadoop
log utilization
Archiving
Amazon S3

Access logs Alerting
Apache Nagios

App logs Analysis
Frontend log sources MongoDB
Backend MySQL

System logs Hadoop
syslogd log utilization
Archiving
Databases Amazon S3

Apache Nagios

App logs Analysis
Frontend MongoDB
Backend MySQL

System logs Hadoop
syslogd
Archiving
Databases Amazon S3

Access logs rsync servers
Alerting
Apache Nagios

App logs Analysis
Frontend MongoDB
Backend MySQL
bash scripts

System logs Hadoop
syslogd
Archiving
Databases Amazon S3
perl scripts

Problems...
No unified method to collect logs
> Too many bash/perl scripts
Fragile for changes
Less reliable
> Mixed log formats
Old-fashioned “Human-readable” text logs
Not ready to analyze
> High latency
must wait a day for log rotation

Apache Nagios

App logs Analysis
Frontend MongoDB
Backend MySQL

System logs Hadoop
syslogd
Archiving
filter / buffer / routing
Databases Amazon S3

Input Plugins Output Plugins

Buffer Plugins
Filter Plugins


2012-02-04 01:33:51
JSON format
myapp.buylog {
“user”: ”me”,
“path”: “/buyItem”,
“price”: 150,
“referer”: “/landing”
}


time
tag

2012-02-04 01:33:51
JSON format
myapp.buylog {
“user”: ”me”,
“path”: “/buyItem”,
“price”: 150,
“referer”: “/landing”
}
record

Why Fluentd?
> Extensibility - Plugin architecture
collect logs from various systems
forward logs to various systems
> Unified log format - JSON format
modern “Machine-readable” log format
immediately ready to analyze
> Reliable - HA configuration
> Easy to install - RPM/deb packages
deploy instantly to everywhere

Comparision with other log collectors:
> Scribe
Less extensible
No unified log format
No longer developped?

> Flume
Less simple
No unified log format
Little information about Flume-NG

NHN Japan COOKPAD NAVER

Crocos

http://www.quora.com/Who-uses-Fluentd-in-production

Future of Fluentd
> <filter>
> <match> in <source>
> <label>
> MessagePack for Ruby v5
> td-agent-lite
> Pub/Sub & Monitoring API
> New process model & Live restart
> Backward compatibility

<source> <match **>
type tail type forward
path /var/log/httpd.log host log.server
format apache </match>
tag not_filtered.apache
</source>
Mysterious tag
<match not_filetered.**>
type rewrite
remove_prefix not_filtered tag operations
<rule>
key status
pattern ^500$
ignore true
</rule>
</match>

Before

<source> <match **>
tag apache
</source>

<filter **> Filter plugins!
type rewrite
<rule>
key status
pattern ^500$
ignore true
</rule>
</match>

After (v11)

<source> <match **>
tag apache

<filter **>
type rewrite
<rule>
key status
pattern ^500$
<filter>/<match> in
ignore true <source>
</rule>
</match>
</source>

After (v11)

<source> <match **>
tag apache </match>
</source>

I want to add flowcounter here...

Before

<source> <store>
tag apache </store>
</source> </match>

<match flow.traffic>
type forward
host traffic.server
</match> Nested!
<match **>
type copy
<store>
type flowcounter
tag flow.traffic
</store>

Before

<source> <match **>
tag apache </match>
</source>

<filter **>
type copy
<match>
type flowcounter
tag flow.traffic
<match>
type forward
host traffic.server
</match>
</match> Filtering pipeline
</match>

After (v11)

<source> # copy & label & forward
type forward <filter **>
</source> type copy
<match>
<filter **> type forward
type copy label alert
<match> host alerting.server
type file </match>
path /mnt/local_archive </filter>
</match>
</filter> # copy & label & forward
<filter **>
<label alert> type copy
<match **> <match>
... type forward
</match> label analysis
</label> host analysis.server
</match>
<label analysis> </filter>
...
</label>
After (v11)

MessagePack for Ruby v5

(tweets/sec)

40000

30000

20000

10000

0
Serialize Deserialize

msgpack v5 msgpack v4 yajl json

td-agent-lite

> in_tail + out_forward in “single” binary
statically linked ruby binary + scripts tied with the binary

New process model & Live restart
Old multiprocess model
detached
process

fork()
Supervisor Engine

detached
all data pass through process
the central process

New multiprocess model
detached
process

Process
Supervisor Engine
Manager

detached
process
direct communication

New multiprocess model
detached
process

Process
Supervisor Engine
Manager

detached
Live restart process

Process
Engine
Manager

Backward compatibility

Fluentd v11 includes 2 namespaces:
> Fluentd:: new code base
> Fluent:: old code base + wrapper classes

Checkout the repository for details:
> http://github.com/frsyuki/fluentd-v11

Conculution

Fluentd makes logging better
> Plugin architecture
> JSON format
> HA configuration
> RPM/deb package

Fluentd is under active development
Fluentd is suppored by many committers

contact: sales@treasure-data.com

ログ収集/解析に使っているツール

Fluentdを導入するにあたっての障壁

Fluentd meetup #2

Recommended

More Related Content

What's hot (20)

Similar to Fluentd meetup #2 (20)

More from Treasure Data, Inc. (20)

Recently uploaded (20)

Fluentd meetup #2