[B! fluentd][Fluentd] hondallicaã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

hondallica id:hondallica

fluentdã¨Fluentdã«é–¢ã™ã‚‹hondallicaã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (8)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

Fluentdã‹ã‚‰Amazon Elasticsearch Serviceã¸ãƒã‚°è»¢é€ã™ã‚‹æ™‚ã®æ³¨æ„ç‚¹ - æœ¬æ—¥ã‚‚ä¹™
æ˜¨å¹´10æœˆã«Amazon Elasticsearch Service(Amazon ES)ãŒãƒªãƒªãƒ¼ã‚¹ã•ã‚Œã¾ã—ãŸã€‚ ä»Šã¾ã§ã¯EC2ã‚¤ãƒ³ã‚¹ã‚¿ãƒ³ã‚¹ã«Elasticsearch ã‚’ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã—ã¦é‹ç”¨ã—ã¦ã„ã¾ã—ãŸãŒã€AWSå´ã§ãƒžãƒãƒ¼ã‚¸ãƒ‰ã—ã¦ãã‚Œã‚‹ã¨ã„ã†ã“ã¨ã§ã¨ã¦ã‚‚ä¾¿åˆ©ãã†ã ãªã¨æ€ã„ä½¿ã£ã¦ã¿ã¾ã—ãŸã€‚ å‹•ä½œç’°å¢ƒã¯ä»¥ä¸‹ã®é€šã‚Šã§ã™ã€‚ Fluentd 0.10.58 fluent-plugin-elasticsearch 0.9.0 Amazon Elasticsearch Service r3.xlarge æ§‹æˆã¨ã—ã¦ã¯ã€å„Webã‚µãƒ¼ãƒã‹ã‚‰Apacheã®ã‚¢ã‚¯ã‚»ã‚¹ãƒã‚°ã‚’Fleuntdã‚µãƒ¼ãƒã«é›†ç´„ã—ã¦Amazon ESã¸è»¢é€ã™ã‚‹ã‚ˆã†ã«ã—ã¦ã„ã¾ã—ãŸã€‚ ã—ã‹ã—ã€Fluentdã‹ã‚‰Amazon ESã¸è»¢é€ã™ã‚‹éš›ã«é€”ä¸ã§ã‚¨ãƒ©ãƒ¼ãŒå‡ºã¦ã—ã¾ã„ã€è»¢é€ã§ããªããªã‚‹ã“ã¨ãŒã‚ã‚Šã¾ã—ãŸã€‚ 2015-12-14 07:00:55 +
hondallica 2016/01/07
fluentd

aws

elasticsearch
ãƒªãƒ³ã‚¯
fluentdã§æœ¬ç•ªç’°å¢ƒã‚’å†ç¾ã™ã‚‹ - Qiita
toyama0919/fluent-plugin-http_shadowã¨ã„ã†Shadow Proxyã£ã½ã„ã“ã¨ã‚’ç°¡å˜ã«ã‚„ã‚‹ãƒ—ãƒ©ã‚°ã‚¤ãƒ³ã‚’ä½œã‚Šã¾ã—ãŸã€‚ productionç’°å¢ƒã§åŠå¹´ãã‚‰ã„å‹•ã‹ã—ã¦ãŸã®ã§ãƒ¡ãƒ¢ã—ã¨ãã¾ã™ã€‚ ã€ŒFluentd Meetup 2015 å¤ã€ã§å®Ÿéš›ã®ãƒ¦ãƒ¼ã‚¹ã‚±ãƒ¼ã‚¹ã‚’ç™ºè¡¨ã—ã¾ã—ãŸã€‚ Shadow Proxyã‚µãƒ¼ãƒã¨ã¯ Shadow Proxyã‚µãƒ¼ãƒã«ã¤ã„ã¦ã¯ä»¥ä¸‹ãŒã‚ã‹ã‚Šã‚„ã™ã„ã§ã™ã€‚ æ°—è»½ãªMySQLãƒãƒ¼ã‚¸ãƒ§ãƒ³ã‚¢ãƒƒãƒ— - ã¾ã‚ç•‘ Goè¨€èªžã‚’å«ã‚€è¤‡æ•°ç¨®é¡žã®è¨€èªžã«ã‚ˆã‚Šå®Ÿè£…ã•ã‚ŒãŸã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ã®ãƒ™ãƒ³ãƒãƒžãƒ¼ã‚¯ - Qiita å®Ÿè£…ã¨ã—ã¦ã¯ä»¥ä¸‹ã®ã‚ˆã†ãªã‚‚ã®ãŒå…¬é–‹ã•ã‚Œã¦ã„ã¾ã™ã€‚ cookpad/kage lestrrat/p5-Geest kentaro/delta æœ¬ç•ªã®ãƒªã‚¯ã‚¨ã‚¹ãƒˆã‚’ãã®ã¾ã¾ãƒãƒƒã‚¯ã‚¨ãƒ³ãƒ‰ã«ã‚ã‚‹ã‚µãƒ¼ãƒãƒ¼ã«è¤‡è£½ã—ã¦é€ä¿¡ã™ã‚‹ã®ã§ã™ãŒã€ã‚¢ãƒ—ãƒªã‚±ãƒ¼ã‚·ãƒ§ãƒ³ã®è¦æ¨¡ãŒå¤§ãããªã‚Œã°ãªã‚‹ã»
hondallica 2015/05/21
fluentd

test

performance
ãƒªãƒ³ã‚¯
Fluentdãƒ—ãƒ©ã‚°ã‚¤ãƒ³ in_exec, out_execã®å‹•ä½œã‚’ç¢ºèªã™ã‚‹ â€“ OpenGroove
Fluentdã®çµ„ã¿è¾¼ã¿ãƒ—ãƒ©ã‚°ã‚¤ãƒ³ã«ã€ in_exec / out_exec ã¨ã„ã†ã®ãŒã‚ã‚‹ã€‚ in_exec ä»»æ„ã®ã‚³ãƒžãƒ³ãƒ‰/ãƒ—ãƒã‚°ãƒ©ãƒ ã®å®Ÿè¡Œçµæžœã‚’ã‚¤ãƒ™ãƒ³ãƒˆã®ã‚½ãƒ¼ã‚¹ã¨ã™ã‚‹ out_exec ã‚¤ãƒ™ãƒ³ãƒˆã®Msgã‚’ãƒ•ã‚¡ã‚¤ãƒ«ã¨ã—ã¦ä»»æ„ã®ã‚³ãƒžãƒ³ãƒ‰/ãƒ—ãƒã‚°ãƒ©ãƒ ã«æ¸¡ã™ã€‚ãã®ãƒ•ã‚¡ã‚¤ãƒ«åãŒå¼•æ•°ã¨ãªã‚‹ã€‚ ã‚¤ãƒ³ã‚¿ãƒ¼ãƒãƒ«ã‚’æŒ‡å®šã—ã€ãã®é–“éš”ã«ãŠã„ã¦ãƒ—ãƒã‚°ãƒ©ãƒ ãŒå®Ÿè¡Œã•ã‚Œã‚‹ã€‚ â€¦ã¨èžã„ã¦ã‚‚åˆ†ã‹ã£ãŸã‚ˆã†ãªåˆ†ã‹ã‚‰ãªã„ã‚ˆã†ãªå¾®å¦™ãªå¿ƒå¢ƒãªã®ã§ã€ã¨ã«ã‹ãå‹•ã‹ã™ã€‚å®Ÿè¡Œç’°å¢ƒã¯AWSä¸Šã®CentOS6.5 1å°ã€‚td-agentã¯ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«æ¸ˆã¿ã€‚ in_execã®å‹•ä½œã‚’ç¢ºèªã—ã¦ã¿ã‚‹ã€‚ã“ã‚“ãªã‚¹ã‚¯ãƒªãƒ—ãƒˆã‚’ç”¨æ„ã€‚ random_number.py #!/usr/bin/env python # -*- coding: utf-8 -*- import random for i in range(0, 30): print random.
hondallica 2014/11/05
fluentd
ãƒªãƒ³ã‚¯
Fluentd v0.12ã§ã®Filterã¨Label - Go ahead!
Fluentdï¼Œæœ€è¿‘ã ã¨æµ·å¤–ã§ã‚‚éœ²å‡ºãŒå¢—ãˆã¦ãã¦ã„ã¦ï¼Œè»½é‡ãƒ»æŸ”è»Ÿãƒ»ãƒãƒã‚¹ãƒˆã¨ã„ã†æ‰€ã§ï¼Œ æ–°è¦ã®ä»–ï¼Œæ—¢å˜ã®ãƒã‚°ã‚³ãƒ¬ã‚¯ã‚¿ã®ãƒªãƒ—ãƒ¬ãƒ¼ã‚¹å«ã‚ï¼ŒæŽ¡ç”¨ã™ã‚‹æ‰€ãŒå¢—ãˆã¦ãŸã‚Šã—ã¾ã™ï¼Ž ã‚ˆã‚Šæ”¹å–„ã™ã‚‹ãŸã‚è‰²ã€…ã¨ãƒ¦ãƒ¼ã‚¶ã«ãƒ’ã‚¢ãƒªãƒ³ã‚°ã—ãŸçµæžœï¼Œã€Œãƒ•ã‚£ãƒ«ã‚¿æ©Ÿèƒ½ãŒæ¬²ã—ã„ã€ã¨ã„ã†ã®ãŒä¸€ç•ªå¤šã„æ„è¦‹ã§ã—ãŸï¼Ž Fluentdã¯å…ƒã€…Treasure Dataã¸ãƒãƒã‚¹ãƒˆã«ãƒ‡ãƒ¼ã‚¿ã‚’è»¢é€ã™ã‚‹ãŸã‚ã®ãƒŸãƒ‰ãƒ«ã‚¦ã‚§ã‚¢ã§ï¼Œã€ŒETLã¨ã‹ã¯Treasure Dataã§ã€ ã¨ã„ã†ã®ã‚‚ã‚ã‚Šï¼Œçµ„ã¿è¾¼ã¿ã§ãƒ•ã‚£ãƒ«ã‚¿æ©Ÿèƒ½ã¯ã‚ã‚Šã¾ã›ã‚“ã§ã—ãŸï¼Ž ä»Šç¾åœ¨ã®Outputãƒ—ãƒ©ã‚°ã‚¤ãƒ³ã«ã‚ˆã‚‹ãƒ•ã‚£ãƒ«ã‚¿å®Ÿè£…ã¯ï¼Œã‚¿ã‚°ã®æ›¸ãæ›ãˆãŒå¿…è¦ã ã£ãŸã‚Šã—ã¦å°‘ã—æ…£ã‚ŒãŒå¿…è¦ã§ï¼Œåˆå¿ƒè€…ã«ã¯ã¡ã¨é›£ã—ã„ï¼Ž ã¨ã„ã†ã“ã¨ã§ï¼Œã‚ˆã‚Šç°¡å˜ã«åŠ¹çŽ‡ã‚ˆããƒ‡ãƒ¼ã‚¿ã‚¹ãƒˆãƒªãƒ¼ãƒ ã‚’æ‰±ãˆã‚‹ãƒ•ã‚£ãƒ«ã‚¿æ©Ÿèƒ½ã‚’å…¥ã‚Œã‚‹ã“ã¨ã«ã—ã¾ã—ãŸï¼ å‰ç½®ããŒé•·ããªã‚Šã¾ã—ãŸãŒï¼Œæ¬¡ã®ãƒãƒ¼ã‚¸ãƒ§ãƒ³ã§ã‚ã‚‹v0.12ã§ã¯Filterã¨Labelã®å°Žå…¥ãŒç›®çŽ‰æ©Ÿèƒ½ã«ãªã‚Š
hondallica 2014/08/25
Fluentd
ãƒªãƒ³ã‚¯
fluentdã®åŠ¹æžœçš„ãªæ´»ç”¨ä¾‹ã¨å®‰å®šé‹ç”¨ã®ãƒã‚¤ãƒ³ãƒˆ
fluentdã®åŠ¹æžœçš„ãªæ´»ç”¨ä¾‹ã¨å®‰å®šé‹ç”¨ã®ãƒã‚¤ãƒ³ãƒˆï¼šä»Šã•ã‚‰èžã‘ãªã„fluentdï½žã‚¯ãƒ©ã‚¦ãƒ‰æ™‚ä»£ã®ãƒã‚°ç®¡ç†å…¥é–€ï¼ˆ3ï¼‰ï¼ˆ1/3 ãƒšãƒ¼ã‚¸ï¼‰ åŠ¹çŽ‡è‰¯ãã€æ„å‘³ã®ã‚ã‚‹ãƒã‚°ç®¡ç†ã‚’å®Ÿç¾ã™ã‚‹ãƒ„ãƒ¼ãƒ«ã¨ã—ã¦æ³¨ç›®ã•ã‚Œã¦ã„ã‚‹ã€Œfluentdã€ã€‚æœ€çµ‚å›žã§ã¯ã€å®Ÿéš›ã®åˆ©ç”¨ã‚·ãƒ¼ãƒ³ã‚’æƒ³å®šã—ã€ã‚ˆã‚ŠåŠ¹æžœçš„ãªfluentdã®åˆ©ç”¨æ³•ã‚’ç´¹ä»‹ã—ã¾ã™ã€‚ ç¬¬1å›žã€ç¬¬2å›žã§fluentdã‚’ä½¿ã£ã¦åŸºæœ¬çš„ãªãƒã‚°ç®¡ç†ãŒå®Ÿç¾ã§ãã‚‹ã‚ˆã†ã«ãªã£ãŸã®ã§ã¯ãªã„ã§ã—ã‚‡ã†ã‹ã€‚fluentdã¯ãƒ—ãƒ©ã‚°ã‚¤ãƒ³ã®çµ„ã¿åˆã‚ã›ã«ã‚ˆã‚Šæ›´ã«åŠ¹æžœã‚’ç™ºæ®ã—ã¾ã™ã€‚æœ€çµ‚å›žã§ã¯ã€å®Ÿéš›ã®åˆ©ç”¨ã‚·ãƒ¼ãƒ³ã‚’æƒ³å®šã—ã€ã‚ˆã‚ŠåŠ¹æžœçš„ãªåˆ©ç”¨æ³•ã‚’ç´¹ä»‹ã—ã¾ã™ã€‚ fluentdã®å…·ä½“çš„ãªæ´»ç”¨ä¾‹ å®Ÿéš›ã®åˆ©ç”¨ã‚·ãƒ¼ãƒ³ã‚’æƒ³å®šã—ãŸ2ã¤ã®fluentdã®æ´»ç”¨ä¾‹ã‚’ç´¹ä»‹ã—ã¾ã™ã€‚ å¤§é‡ã®ãƒã‚°ã‚’åˆ†æžã—ã€ã€Œæ„å‘³ã®ã‚ã‚‹æƒ…å ±ã€ã¨ã—ã¦ç®¡ç†ã™ã‚‹ ã‚¿ã‚°ãƒ‡ãƒ¼ã‚¿ã‚’åŠ¹æžœçš„ã«æ´»ç”¨ã—ã€ãƒã‚°ãƒ‡ãƒ¼ã‚¿ã®ãƒ•ã‚£ãƒ«ã‚¿ãƒªãƒ³ã‚°ç®¡ç†ã‚’è¡Œã† 1.å¤§é‡ã®ãƒã‚°ã‚’åˆ†æžã—ã€ã€Œæ„å‘³ã®ã‚
hondallica 2014/04/04
fluentd
ãƒªãƒ³ã‚¯
JBATã€ãƒã‚°åŽé›†åŸºç›¤ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ã€ŒFluentdã€ã®Windowsç‰ˆé–‹ç™ºã‚’æŽ¨é€²
JBCCãƒ›ãƒ¼ãƒ«ãƒ‡ã‚£ãƒ³ã‚°ã‚¹ã®äº‹æ¥ä¼šç¤¾ã§JBã‚°ãƒ«ãƒ¼ãƒ—ã®ã‚ªãƒªã‚¸ãƒŠãƒ«ã‚½ãƒªãƒ¥ãƒ¼ã‚·ãƒ§ãƒ³é–‹ç™ºä¼šç¤¾ã§ã‚ã‚‹JBã‚¢ãƒ‰ãƒãƒ³ã‚¹ãƒˆãƒ»ãƒ†ã‚¯ãƒŽãƒã‚¸ãƒ¼ï¼ˆä»¥ä¸‹ã€JBATï¼‰ã¯ã€ç±³å›½ã®Treasure Data,IncãŒé–‹ç™ºã—ã€ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ã¨ã—ã¦å…¬é–‹ã—ã¦ã„ã‚‹ãƒã‚°åŽé›†åŸºç›¤ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ã€ŒFluentdï¼ˆãƒ•ãƒ«ãƒ¼ã‚¨ãƒ³ãƒˆãƒ‡ã‚£ï¼‰ã€ã®ã‚³ãƒŸãƒ¥ãƒ‹ãƒ†ã‚£ã«å‚åŠ ã—ã€Windowsç‰ˆã®é–‹ç™ºã‚’æŽ¨é€²ã—ã¦ã„ã‚‹ã“ã¨ã‚’ç™ºè¡¨ã—ãŸã€‚ Fluentdã¯ã€ãƒã‚°åŽé›†åŸºç›¤ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ã¨ã—ã¦å›½å†…å¤–ã®Webç³»ä¼æ¥ã‚„ã‚ªãƒ³ãƒ©ã‚¤ãƒ³ã‚²ãƒ¼ãƒ ä¼šç¤¾ãªã©ã§æ€¥é€Ÿã«å°Žå…¥ãŒé€²ã‚“ã§ãŠã‚Šã€ãƒ€ã‚¦ãƒ³ãƒãƒ¼ãƒ‰æ•°ã¯92,000ä»¶ä»¥ä¸Šã«ã®ã¼ã‚‹ã¨ã„ã†ã€‚ JBATã¯ã€Windowsç’°å¢ƒã§å‹•ä½œã™ã‚‹Fluentdã®é–‹ç™ºã‚’é€²ã‚ã€ãã®æˆæžœã‚’ã‚³ãƒŸãƒ¥ãƒ‹ãƒ†ã‚£ã«æä¾›ã€‚ã“ã‚Œã«ã‚ˆã‚Šã€ã“ã‚Œã¾ã§ä»¥ä¸Šã«å¤šãã®ä¼æ¥ã§Fluentdã®æ´»ç”¨æ©Ÿä¼šã‚’åºƒã’ã‚‹ã€‚ã¾ãŸã€Fluentdã®ç‰¹é•·ã§ã‚ã‚‹ãƒ‡ãƒ¼ã‚¿ç™ºç”Ÿã®éƒ½åº¦ã€ãƒ‡ãƒ¼ã‚¿ã‚’åŽé›†ã™ã‚‹ä»•çµ„ã¿ã‚’JBATã‚ª
hondallica 2014/03/20
fluentd

windows
ãƒªãƒ³ã‚¯
fluentdã®outputãƒ—ãƒ©ã‚°ã‚¤ãƒ³ã§ãƒ–ãƒãƒƒã‚¯ã™ã‚‹ã‚‚ã®ã¯BufferedOutputã‚’ä½¿ã†ã¹ã - still deeper
tl;dr ã‚¿ã‚¤ãƒˆãƒ«ã®ã¾ã¾ å‰ç½®ã fluentdã‚¯ãƒ©ã‚¹ã‚¿ã®ã‚ã‚‹ãƒŽãƒ¼ãƒ‰ã«ã ã‘ã€ãã®ãƒŽãƒ¼ãƒ‰ã«é€ä¿¡ã—ã¦ã„ã‚‹out_forwardãŒdetachã•ã‚Œç¶šã‘ã‚‹ã¨ã„ã†ç—‡çŠ¶ãŒå‡ºã¾ã—ãŸã€‚ èª¿æŸ»ã—ãŸã¨ã“ã‚ã€å¤–éƒ¨ã¸ã®é€šçŸ¥ç”¨ã«è¿½åŠ ã—ãŸhipchatãƒ—ãƒ©ã‚°ã‚¤ãƒ³ã‚’è¿½åŠ ã—ãŸã¨ã“ã‚ã§ç—‡çŠ¶ãŒç™ºç”Ÿã™ã‚‹ã‚ˆã†ã§ã™ã€‚ åŽŸå› BufferedOutputãƒ—ãƒ©ã‚°ã‚¤ãƒ³ã§ã® write ãƒ¡ã‚½ãƒƒãƒ‰ã§ã®ã‚¹ã‚¿ãƒƒã‚¯ãƒˆãƒ¬ãƒ¼ã‚¹ã¯ã“ã‚“ãªé¢¨ã«ãªã‚Šã¾ã™ã€‚ /home/choplin/git/fluentd/lib/fluent/buffer.rb:296:in `write_chunk' /home/choplin/git/fluentd/lib/fluent/buffer.rb:276:in `pop' /home/choplin/git/fluentd/lib/fluent/output.rb:309:in `try_flush' /home/cho
hondallica 2014/01/20
fluentd
ãƒªãƒ³ã‚¯
dstatã‚’fluentd + GrowthForecastã§å¯è¦–åŒ–ã™ã‚‹ - Qiita
dstatã®çµæžœã‚’fluentdçµŒç”±ã§GrowthForecastã«æŠ•ã’ã‚‹æ–¹æ³•ã¯ã€ããã‚‹ã¨ä½•ä»¶ã‹ãƒ–ãƒã‚°è¨˜äº‹ãŒã§ã¦ãã¾ã™ã€‚ ãŸã ã€ã©ã‚Œã‚‚dstatã‚’æ”¹é€ ã—ãŸã‚Šã€ãƒ—ãƒ©ã‚°ã‚¤ãƒ³ã‚’æ”¹é€ ã—ãŸã‚Šã—ã¦ã¦ã€ã‚ã¾ã‚Šä½¿ã„å‹æ‰‹ãŒã‚ˆãã‚ã‚Šã¾ã›ã‚“ã€‚ ã¨ã„ã†ã‚ã‘ã§ã€gemã§ã„ã‚Œã‚Œã‚‹ãƒ—ãƒ©ã‚°ã‚¤ãƒ³ã ã‘ã‚’ä½¿ã£ã¦ã€å¯è¦–åŒ–ã§ãã‚‹è¨å®šã‚’æŽ¢ã—ã¦ã¿ã¾ã—ãŸã€‚ æº–å‚™ å¿…è¦ãªgemã‚’ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã™ã‚‹ã€‚ <source> type dstat tag dstat option -cm delay 3 </source> # GrowthForecastç”¨ã«ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã‚’åŠ å·¥ã™ã‚‹ <match dstat> type copy # CPUé–¢é€£ã®çµæžœã ã‘ã‚’å–ã‚Šå‡ºã™ <store> type map tag "perf.cpu" time time record record['dstat']['total cpu usage'] </store> #
hondallica 2012/11/12
fluentd

GrowthForecast
ãƒªãƒ³ã‚¯
1