[B! fluentd] tksmdã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

tksmd id:tksmd

fluentdã«é–¢ã™ã‚‹tksmdã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (9)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

Fluentdã§ãƒãƒƒãƒ•ã‚¡ã¤ãOutputPluginã‚’ä½¿ã†ã¨ãã®ãƒ‡ãƒ•ã‚©ãƒ«ãƒˆå€¤ - ãŸã”ã‚‚ã‚Šã™ãƒ¡ãƒ¢
ãªã‚“ã‹è‡ªåˆ†ã§ docs.fluentd.org ã¸ã®patchã‚’æ›¸ã„ã¦ã¦æ··ä¹±ã—ã¦ããŸã®ã§ã¾ã¨ã‚ã‚‹ã€‚ã‚³ãƒ¼ãƒ‰ã‚’èªã‚“ã§ã‚‚é–¢ä¿‚ã™ã‚‹è¨å®šå€¤ãŒã„ãã¤ã‚‚ã®ãƒ¢ã‚¸ãƒ¥ãƒ¼ãƒ«ã«åˆ†æ•£ã—ã¦ãŠã‚Šã€å®Œå…¨ã«æŠŠæ¡ã™ã‚‹ã“ã¨ãŒå›°é›£ã§ã‚ã‚‹ã€‚å…·ä½“çš„ã«ã¯ã€ã“ã®çµ„åˆã›ã‚’è¨˜æ†¶ã ã‘ã§ç”ãˆã‚‰ã‚Œã‚‹ fluentd ã‚³ãƒŸãƒƒã‚¿ã¯ãŠãã‚‰ãä¸€äººã‚‚ã„ãªã„ã€‚ æ¦‚è¦ å¯¾è±¡ã¯ BufferedOutput ãŠã‚ˆã³ TimeSlicedOutput ã‚’ç¶™æ‰¿ã—ã¦ã„ã‚‹ output plugin ã®å…¨ã¦*1ã€‚out_forward, out_exec ã‚„ out_exec_filter ã‚‚å«ã¾ã‚Œã‚‹ã€‚ åŸºæœ¬çš„ã«ã¯ã„ãã¤ã‹ã®è¨å®šã«ã‚ˆã‚Š flush ã‚’ã™ã‚‹ã‚¿ã‚¤ãƒŸãƒ³ã‚°ã‚’åˆ¶å¾¡ã™ã‚‹ãƒ‘ãƒ©ãƒ¡ãƒ¼ã‚¿ä¸€å¼ã€ãŠã‚ˆã³flushå¯¾è±¡ã¨ãªã‚‹ãƒ‡ãƒ¼ã‚¿ã®ãƒãƒ£ãƒ³ã‚¯ã‚’æºœã‚ã¦ãŠãé‡ã®ä¸Šé™ã‚’æ±ºã‚ã‚‹ã“ã¨ã¨ãªã‚‹ã€‚fluentd ã‚’ã†ã£ã‹ã‚Šè©¦ã—ãŸã¨ãã«ã€Œã‚¢ã‚¤ã‚¨ã‚¨ã‚¨ã€fluent-cat ã—ã¦ã¿ãŸã‚“ã ã‘ã©ã€è¨å®šã—ãŸã¨
tksmd 2015/01/29
â€œbuffer_chunk_limitâ€

fluentd
ãƒªãƒ³ã‚¯
PostgreSQLã®ãƒã‚°ã‚’Fluentdã§åŽé›†ã™ã‚‹ - Tech-Sketch
PostgreSQLã®é‹ç”¨ç®¡ç†ã«ãƒã‚°ã®æƒ…å ±ã¯ä¸å¯æ¬ ã§ã™ãŒã€PostgreSQLãŒå‡ºåŠ›ã™ã‚‹ãƒã‚°ã¯ç›´æŽ¥å–ã‚Šæ‰±ã†ã®ã«é©ã—ãŸå½¢å¼ã¨ã¯è¨€ãˆã¾ã›ã‚“ã€‚ã“ã“ã§ã¯Fluentdã‚’ä½¿ã£ã¦é‹ç”¨ç®¡ç†ã«æ´»ç”¨ã—ã‚„ã™ã„å½¢ã«PostgreSQLã®ãƒã‚°ã‚’åŠ å·¥ã™ã‚‹æ–¹æ³•ã‚’ç´¹ä»‹ã—ã¾ã™ã€‚ PostgreSQLã®é‹ç”¨ç®¡ç†ã¨ãƒã‚° ãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹ã®é‹ç”¨ç®¡ç†ã«ã¯ æ»æ´»ç›£è¦–ã€ãƒªã‚½ãƒ¼ã‚¹ç›£è¦– æ€§èƒ½åˆ†æžã€ãƒãƒ¥ãƒ¼ãƒ‹ãƒ³ã‚° ãƒãƒƒã‚¯ã‚¢ãƒƒãƒ—ã€ãƒªã‚¹ãƒˆã‚¢ ã¨ã„ã£ãŸã‚‚ã®ãŒè€ƒãˆã‚‰ã‚Œã¾ã™ãŒã€ç¾çŠ¶ã‚’æŠŠæ¡ã—ãªã‘ã‚Œã°ä½•ã‚‚å¯¾ç–ã§ããªã„ã®ã§ã€ã¾ãšã¯å¿…è¦ãªç›£è¦–ã‚’æ£ã—ãè¡Œã†ã“ã¨ãŒé‡è¦ã§ã™ã€‚PostgreSQLã§ç›£è¦–ã«ä½¿ãˆã‚‹æ©Ÿèƒ½ã«ã¯ç¨¼åƒçµ±è¨ˆæƒ…å ±ã¨ãƒã‚°ãŒã‚ã‚Šã¾ã™ãŒã€ä»Šå›žã¯ãƒã‚°ã«æ³¨ç›®ã—ã¦ã¿ã¾ã™ã€‚ PostgreSQLã®ãƒã‚°ã‹ã‚‰ã‚ã‹ã‚‹ã“ã¨ PostgreSQLãŒå‡ºåŠ›ã™ã‚‹ãƒã‚°ã‹ã‚‰ã¯ä»¥ä¸‹ã®æ§˜ãªã“ã¨ãŒã‚ã‹ã‚Šã¾ã™ã€‚ ä½•ã‚‰ã‹ã®ç•°å¸¸(ã‚¨ãƒ©ãƒ¼)ãŒç™ºç”Ÿã—ãŸã“ã¨ å‡¦ç†ã•ã‚ŒãŸSQL ãƒã‚§ãƒƒã‚¯ãƒã‚¤ãƒ³ãƒˆã€è‡ªå‹•VA
tksmd 2014/07/28
PostgreSQL

fluentd
ãƒªãƒ³ã‚¯
PostgreSQLã®ãƒã‚°ã‚’fluentdã§å›žåŽã™ã‚‹è¨å®š - still deeper
PostgreSQLã®ãƒã‚°ã‚’fluentdçµŒç”±ã§å›žåŽã™ã‚‹ã‚ˆã†ã«ã—ãŸã®ã§è¨å®šã‚’æ™’ã—ã¦ãŠãã¾ã™ã€‚ã»ã¼åŒã˜è¨å®šã‚’ä½¿ã„ã¾ã‚ã›ã‚‹ã¯ãšãªã®ã§ã€fluentd & postgresã®çµ„ã¿åˆã‚ã›ã‚’ä½¿ã£ã¦ã„ã‚‹äººã¯ã©ã†ãžã€‚ PostgreSQLå´ postgresql.conf postgresã®ãƒã‚°ã®è¨å®šã¯ã“ã‚“ãªæ„Ÿã˜ã€‚ # csv logã‚’å‡ºåŠ› logging_collector = on log_destination = 'csv log,stderr' # 1æ—¥ã§ãƒãƒ¼ãƒ†ãƒ¼ã‚·ãƒ§ãƒ³ log_rotation_age = 1440 # /var/log/pgsql/postgres-%Y%m%d.(log|csv)ã«å‡ºåŠ› log_directory = '/var/log/pgsql/' log_filename = 'postgres-%Y%m%d.log' # modeã‚’644ã« log_file_mode
tksmd 2014/06/01
kibana

Postgres

fluentd
ãƒªãƒ³ã‚¯
Fluentd meetup dive into fluent plugin (outdated)
Fluentd meetup in Japan. I talked about "Dive into Fluent plugin". Some contents are outdated. See this slide: http://www.slideshare.net/repeatedly/dive-into-fluentd-plugin-v012Read less
tksmd 2014/05/27
plugin deep dive

fluentd
ãƒªãƒ³ã‚¯
#fluentd ã®ãŸã‚ã®ãƒ—ãƒ©ã‚°ã‚¤ãƒ³ã‚’ã‚¤ãƒã‹ã‚‰æ›¸ãæ‰‹é †(bundlerç‰ˆ) - tagomorisã®ãƒ¡ãƒ¢ç½®ãå ´
å‰ã«è‡ªåˆ†ã§æ›¸ã„ãŸ fluentdã®ãŸã‚ã®ãƒ—ãƒ©ã‚°ã‚¤ãƒ³ã‚’ã‚¤ãƒã‹ã‚‰æ›¸ãæ‰‹é † - tagomorisã®ãƒ¡ãƒ¢ç½®ãå ´ ã¯ãŸã„ã¸ã‚“é‡å®ã—ã¦ã„ãŸã®ã ãŒã€æ›¸ã„ãŸã™ã“ã—å¾Œã«ãªã£ã¦å®Ÿã¯ç¾åœ¨ã™ã§ã« bundle gem ã‚³ãƒžãƒ³ãƒ‰ã‚’ä½¿ã†ã‚„ã‚Šã‹ãŸãŒè‰¯ã•ãã†ã ã¨ã„ã†ã“ã¨ãŒã‚ã‹ã£ã¦ã—ã¾ã£ãŸãŒã°ãŸã°ãŸã—ã¦ã¦ç§»è¡Œã—ã¦ãªã‹ã£ãŸã€‚ ã§ã€ã¾ãŸã²ã¨ã¤ãƒ—ãƒ©ã‚°ã‚¤ãƒ³ã‚’æ›¸ãã“ã¨ã«ã—ãŸã®ã§ã¤ã„ã§ã« bundle ã‚’ä½¿ã£ãŸæ‰‹é †ã‚’ã–ã£ãã‚Šã¾ã¨ã‚ã¦ãŠãã€‚ä»¥ä¸‹ã®ã‚¨ãƒ³ãƒˆãƒªã‚’ãŸã„ã¸ã‚“å‚è€ƒã«ã•ã›ã¦ã‚‚ã‚‰ã£ãŸã€‚ T-POINTã‚’å–å¾—ã™ã‚‹ã‚¹ã‚¯ãƒªãƒ—ãƒˆã‚’Gistã‹ã‚‰ç§»å‹•, Bundlerã‚’ä½¿ã£ãŸGemä½œæˆãƒ¡ãƒ¢ (è‡ªåˆ†ç”¨) - ãŸã ã®ã«ã£ã(2012-02-18) æº–å‚™ã¨ãƒ‡ã‚£ãƒ¬ã‚¯ãƒˆãƒªãƒ„ãƒªãƒ¼ã®ä½œæˆ bundler ã¯å¿…è¦ãªã®ã§ã€ãªã«ã¯ãªãã¨ã‚‚å…¥ã‚Œã¦ãŠã“ã†ã€‚ gem install bundler ãã—ã¦ãƒ—ãƒ©ã‚°ã‚¤ãƒ³ç”¨ãƒ‡ã‚£ãƒ¬ã‚¯ãƒˆãƒªãƒ„ãƒªãƒ¼ã‚’ä½œæˆã™ã‚‹ã€‚ä»Šå›žã¯ DataCount
tksmd 2014/05/13
how to write plugin

fluentd
ãƒªãƒ³ã‚¯
è¶…åˆç´šï¼Fluentdã®ãƒ—ãƒ©ã‚°ã‚¤ãƒ³ã‚’æ›¸ããŸããªã£ãŸæ™‚ã®ä¸‹åœ°ã¥ãã‚Š Â· ã•ã‚ˆãªã‚‰ã‚¤ãƒ³ã‚¿ãƒ¼ãƒãƒƒãƒˆ
January 5, 2013 fluentdã‚’ã¾ã¨ã‚‚ã«å‹•ã‹ã—ãŸã“ã¨ãªã„ã‘ã© ãƒ—ãƒ©ã‚°ã‚¤ãƒ³æ›¸ã„ãŸã‚‰ã‚ã‹ã‚‹ã®ã§ã¯ï¼ï¼ï¼Ÿ ã¨æ€ã„ç«‹ã£ã¦å–ã‚Šæ•¢ãˆãšä¸‹åœ°ã ã‘ã¤ãã£ãŸã®ã§ãƒ¡ãƒ¢ã€‚ å‚è€ƒ fluentdã®ãŸã‚ã®ãƒ—ãƒ©ã‚°ã‚¤ãƒ³ã‚’ã‚¤ãƒã‹ã‚‰æ›¸ãæ‰‹é †(bundlerç‰ˆ) / tagomorisã®ãƒ¡ãƒ¢ç½®ãå ´ Writing plugins / fluentd fluent-plugin-imkayac / fujiwara fluent-plugin-r18 / studio3104 åŸºæœ¬ã¯ã‚‚ã‚Šã™å…ˆç”Ÿã®æ‰‹é †ã«ãã£ã¦è¡Œãˆã°å•é¡Œãªã—ã€‚ å‚è€ƒã«ã‚ªãƒ•ã‚£ã‚·ãƒ£ãƒ«ã®ãƒ‰ã‚ãƒ¥ãƒ¡ãƒ³ãƒˆã¨ @fujiwaraã•ã‚“ã®ã‚·ãƒ³ãƒ—ãƒ«ãªãƒ—ãƒ©ã‚°ã‚¤ãƒ³ @studio3104ã•ã‚“ã®ä¸‹åœ°ã‚’è¦‹ãªãŒã‚‰æ›¸ãã¨ãªãŠã‚ˆã—ã€‚ ã¾ãŸã€rake testã¾ã§é€šã—ãŸã‚‚ã®ã‚’ https://github.com/kenjiskywalker/fluent-plugin-hoge ã“
tksmd 2014/05/13
how to write plugin

fluentd
ãƒªãƒ³ã‚¯
Fluentdã§parserç”¨ã®æ£è¦è¡¨ç¾ã‚’æ›¸ããƒ»è©¦ã™ - ãŸã”ã‚‚ã‚Šã™ãƒ¡ãƒ¢
ã“ã‚“ãªã‚¨ãƒ³ãƒˆãƒªã‚’ç›®ã«ã—ãŸã®ã§ã€ãªã‚“ã‹æ›¸ã“ã†ã‹ãªã¨æ€ã£ãŸã€‚ fluentdã®format(æ£è¦è¡¨ç¾)ã®ä½œã‚Šæ–¹ã«ã¤ã„ã¦è©¦è¡ŒéŒ¯èª¤ä¸ #fluentd - Glide Note - ã‚°ãƒ©ã‚¤ãƒ‰ãƒŽãƒ¼ãƒˆ Fluentd ã® in_tail ã‚„æ‹™ä½œ fluent-plugin-parser ã§ã¯ãƒã‚°ã®parseç”¨ã®æ£è¦è¡¨ç¾ã‚’æŒ‡å®šã™ã‚‹ã“ã¨ã«ãªã‚‹ãŒã€ç¢ºã‹ã«ã“ã‚Œã‚’è¨å®šã—ã¦ãƒã‚°ã‚’æµã—ã¦è©¦ã—ã¦ç›´ã—ã¦ã€ã¨ã„ã†ã®ã¯ã„ã•ã•ã‹åŠ¹çŽ‡ãŒæ‚ªã„ã€‚ã®ã§ã€ç°¡å˜ã«è©¦ã™æ–¹æ³•ã‚’æ›¸ã„ã¦ã¿ã‚‹ã€‚ ãªãŠ irb ã‚’ä½¿ã†æ‰‹é †ã€‚ruby 1.9ç³»ãŒã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã•ã‚Œã¦ã„ã‚Œã°ãŸã¶ã‚“å…¥ã£ã¦ã„ã‚‹ã‘ã‚Œã© td-agent ã ã¨ã¡ã‚‡ã£ã¨ã©ã†ã ã£ã‘ã€‚é©å½“ã«ãŒã‚“ã°ã£ã¦ã„ãŸã ããŸã„ã€‚ ã¨ã‚Šã‚ãˆãšä»¥ä¸‹ã®ã‚ˆã†ãªãƒã‚°ã‚’parseã—ãŸã„ã¨ã™ã‚‹ã€‚ Jul 16 00:45:47 BJA login[58879]: USER_PROCESS: 58879 ttys002irbã‚’èµ·å‹•
tksmd 2012/11/28
parser ã®æ£è¦è¡¨ç¾ã®ãƒ‡ãƒãƒƒã‚°æ–¹æ³•

fluentd
ãƒªãƒ³ã‚¯
#fluentd fluentdï¼ˆtd-agentï¼‰æ§‹ç¯‰æ™‚ã«æ°—ã‚’ã¤ã‘ã‚‹ãƒã‚¤ãƒ³ãƒˆã«ã¤ã„ã¦[2012/3/26 è¿½è¨˜ã‚ã‚Š] - oranie's blog
çµæ§‹æ—¢å‡ºãªå†…å®¹ã§ã™ãŒã€æ€ã„ã£ãã‚Šãƒãƒžã£ãŸã®ã§æ”¹ã‚ã¦ã€‚ â—fluentdâ†’fluentdã®é€šä¿¡æ™‚ã«fowardã‚’ä½¿ãŠã†ã¨ã—ãŸã‚‰ã‚¨ãƒ©ãƒ¼ã«ãªã‚‹ã€‚ â€»è¿½è¨˜ VMwareã®å ´åˆã¯ã“ã£ã¡ã‚’å‚è€ƒã«ã€‚fluentdã˜ã‚ƒãªãã¦VMwareã®å•é¡Œã¨ã„ã†ãƒ‘ã‚¿ãƒ¼ãƒ³ã‚‚ã‚ã‚Šã€‚ http://d.hatena.ne.jp/oranie/20121107/1352298768 ä¸é–“fluentdã‹ã‚‰æœ€çµ‚çš„ã«é›†ç´„ã™ã‚‹fluentdã«fowardã§ãƒã‚°ã‚’é€ã‚ã†ã¨ã™ã‚‹ã¨ã€å§‹ã‚ã¯ã¡ã‚ƒã‚“ã¨é€ã‚Œã‚‹ã®ã«ã€ã¡ã‚‡ã£ã¨ã™ã‚‹ã¨ãƒã‚°ã«ã‚¨ãƒ©ãƒ¼ãŒå‡ºã¦ã„ãŸã€‚ã‚‚ã¡ã‚ã‚“é›†ç´„å…ˆã®æ–¹ã§ã¯ãƒã‚°å—ä¿¡å‡ºæ¥ãªããªã£ã¦ã„ã‚‹ã€‚ failed to flush the buffer, retrying. error="no nodes are available" ã‚‚ã£ã¨å…·ä½“çš„ã«ã¯ã“ã‚“ãªã®ã€‚ ã§ã€ç†ç”±ã¯ @oranie æ»æ´»ç›£è¦–ã«UDPã®ã»ã†ä½¿ã†ã‚‰ã—ã„ã®ã§ãã£ã¡ã‚‚é€šã•ãªã„
tksmd 2012/10/12
forward æ™‚ã®ãƒãƒ¼ãƒˆã«ã¤ã„ã¦

fluentd
ãƒªãƒ³ã‚¯
http://dl.dropbox.com/u/224433/fluentd_casual_1/index.html
tksmd 2012/10/02
Fluentd ã®æ´»ç”¨æ–¹æ³•ãªã©

fluentd

Archetecture
ãƒªãƒ³ã‚¯
1