ãFluentd Meetup 2015 å¤ãã§çºè¡¨ããå 容ã§ãã
åã¯ä»äºã§ã¯2種é¡ã®ãã°è§£æåºç¤ãè¦ã¦ãã¾ãã 1ã¤ç®ã¯ã©ã¡ããã¨ããã¨ã¨ã³ã¸ãã¢ããã®è§£æåºç¤ã§ãµã¼ãã¹å´ã®ã¨ã³ã¸ãã¢ãShib, ShibUIãéãã¦å¥½ãã«ã¯ã¨ãªãæãããã¨ãã§ãã¾ãããã ãtableãcreateãããdropãããinsertãããã¯ã§ãã¾ãããselectã®ã¿ã§ãããã¼ã¿ã®æ´æ°ä½æ¥ã¯å¥écronã®hive batchã§è¡ãã¾ãããã¼ã¿ã¯Fluentdçµç±ã§åãµã¼ãã¹ã®ãµã¼ãã¼ããåéãã¾ãããã£ã¡ã®ã·ã¹ãã ã¯å¤ããããã£ã¦åã¯å¼ãç¶ãã ã ããªã®ã§è¦ã¦ããã¨ã¯ãã£ã¦ããããªã«ãããã¨ã¯ç¡ãã§ãããèªãããã¨ãå°ãªãã§ãã 2ã¤ç®ã¯ç´1å¹´åã«åãä¸ããæ§ç¯ããã·ã¹ãã ã§ãã©ã³ãã¼ããã®ã·ã¹ãã ã«ãªã£ã¦ã¾ããåã®ãã¼ã å ã®ã¨ã³ã¸ãã¢ã ããrawãã¼ã¿ã触ã£ããæ´æ°ããããããã¨ãã§ãã¦ããã©ã³ãã¼ã¯ã¬ãã¼ããéãã¦å å·¥ããããã¼ã¿ãè¦ãå½¢ã«ãªãã¾ãããªã®ã§1ã¤ç®ã®ã·ã¹
TODO: å¿ è¦ãªãå³ãè¶³ã ä»ã«æ¸ããæ¹ãè¯ãPros/Consã®ãªã¯ã¨ã¹ãããã£ããè¿½è¨ å é¨ã®ã¤ãã³ãã¹ããªã¼ã ã®æ±ã Pros: Inputãã¹ã±ã¼ã©ãã«ã«å®è£ ããããï¼ãã¼ã¿ã¹ããªã¼ã ãæ£å¸¸æ/ã¨ã©ã¼æã§åãæ¿ãããã Cons: ã¨ã©ã¼ãã³ããªã³ã°ãããããã³ã°ã¢ãã«ãããè¤éã«ãªãããã 以ä¸é·ã ã¨çç±æ¸ãã¾ãï¼ Fluentdã¯ã¤ãã³ãã¹ããªã¼ã ãå¹çè¯ãï¼ã¾ãããã¹ãã«æ±ããã¨ãç®çã«è¨è¨ããã¦ãã¾ãï¼ãã®ããï¼ç¬èªã®è»¢éãããã³ã«(forwardãã©ã°ã¤ã³)ãå®è£ ãã¦ãã¾ããï¼å é¨ã®ã¤ãã³ãã®ãã³ããªã³ã°ãããã«æ²¿ãããã«ãªã£ã¦ãã¾ãï¼ãã ï¼ããã«ãã£ã¦ç¸æ§ã®æªãæä½ã¨ããããã¾ãï¼ Fluentdã¯ãããã¡æ©è½ãæä¾ãã¦ããï¼ããã«ãã£ã¦è»¢éã®å¹çåã¨ã¨ã©ã¼æã®ãã¼ã¿ãã¹ãé²ãè¨è¨ã«ãªã£ã¦ãã¾ãï¼ãï¼ãã¾ãã«ãæ¸ãè¾¼ã¿å ãé ããªã©ã®åé¡ãããã¨ï¼ãããã¡ã®å¶éãè¶ ãã¦
ãã§ã«ã¢ãã¦ã³ã¹æ¸ã¿ã§ããï¼Treasure Agent(td-agent) 2.2.0ãåºãã¾ãã td-agent 2.2.0 - Google Groups ããã±ã¼ã¸å ã®æ´æ°ã¯ä»¥ä¸ã®éãã§ãï¼ fluentd v0.12.7 fluent-plugin-s3 v0.5.7 fluent-plugin-mongo v0.7.8 td v0.11.8.2 fluentd-ui v0.3.18 ruby 2.1.5 json 1.8.2 cool.io 1.3.0 Amazon Linux support fluentd v0.12 ãã®ãã¼ã¸ã§ã³ããï¼fluentdãv0.10ã§ã¯ãªãv0.12ç³»ã使ãããããã«ãªãã¾ããï¼ãã®çµæï¼Filter/Labelï¼forwardãã©ã°ã¤ã³ã§ã®at-least-onceãªã©ã使ããããã«ãªãã¾ãï¼ ãããã®æ©è½ã«é¢ãã¦ã¯ï¼v0.12ã®ãªãªã¼ã¹
Fluentd ã¨ããã½ããã¦ã§ã¢ããããæ¥æ¬å½å ã§ã¯ãããã話é¡ã«ãªã£ã¦ããããä½ãã§ããã®ããä½ã«ä½¿ãã¨å¬ããã®ããä½ã«ä½¿ããã®ããã¨ããç¹ã«ã¤ãã¦è©³ç´°ãããç¥ããªãã¨ãã人ãããããã¾ã å¤ããã¨ã§ãããã ãªã®ã§ãç°¡åã«ã¾ã¨ããã http://fluentd.org/ ãªã以ä¸ã®åå¥é ç®ãã¨ã«æ¸ãã¦ãããããã®æåã«ã¾ã¨ããç½®ãã¦ããã®ã§å¿ãã人ã¯ããã ãèªãã¨ãããã¤ã³ã¹ãã¼ã«ãè¨å®ã«ã¤ãã¦ã¯å°å ¥é¨åã«ã¤ãã¦ã¯æ¥æ¬èªã®è¨äºã¯ããå¤ãããã®ã§ã触ããªãã æ¦è¦ ã§ããã㨠ãã°ã®åé ã»ã³ãµãã¼ã¿çã®åé æ±ç¨ãã¼ã¿å¦çããã»ããµã¨ãã¦ é »åºã¦ã¼ã¹ã±ã¼ã¹ ãã°ã®åé ãã¼ã¿ã®éç´ ç°¡åãªãªã¢ã«ã¿ã¤ã éè¨ ã½ããã¦ã§ã¢ã¨ãã¦ã®ç¹å¾´ ã³ã¢ ãã©ã°ã¤ã³ å®å®æ§ æ§è½ éçºä½å¶ ã³ãã¥ãã㣠ã¶ã£ã¡ããã©ããªã®ï¼ ã¾ã¨ã ç¾æç¹ã§ãè¤æ°ã®å ´æã«åæ£ãããã¼ã¿ã常ã«å¢ãç¶ãããã¼ã¿ã®å®å ¨ãªè»¢
fluentdã®ã»ãã«ããã¤ããªã·ãªã¢ã©ã¤ã¼ã¼ã·ã§ã³ãã©ã¼ããããMessagePackãã®éçºãªã©ã§ç¥ããã夿©æ°ã ããå¦çæä»£ãããã®æè¡åã®é«ãã«ã¯å®è©ããããæ³¨ç®ããç¶ãã¦ããã¹ã¼ãã¼ã¨ã³ã¸ãã¢ã§ãããã ä»åãfluentdã®ã¦ã¼ã¶ã¼ã§ãããã夿©æ°ã¨ã¯æ§ç¥ã®ä»²ã§ãããã°ãªã¼ éçºæ¬é¨ ãªã¼ãã¼ã®æ£®ç°æ³å¹³æ°ãã¤ã³ã¿ãã¥ã¢ã¼ã¨ãªããfluentdã«ã¾ã¤ãããããã¯ãããã¬ã¸ã£ã¼ãã¼ã¿ã§ã®éçºããªã¼ãã³ã½ã¼ã¹ã¸ã®æ³ããªã©ãè¨ãã¦ãããæ¬ç¨¿ã§ã¯ããã®æ¨¡æ§ããä¼ãããªãããâã¨ã³ã¸ãã¢ã»å¤æ©è²ä¹âã®é åã«è¿«ã£ã¦ã¿ããã fluentd v11ã®æ³¨ç®ãã¤ã³ã 森ç°ãã¾ãã¯ãã°ãªã¼ã§ã大å¤ãä¸è©±ã«ãªã£ã¦ããfluentdã«ã¤ãã¦ãããããèããã¦ãã ãããéçºä¸ã®æ°ãã¼ã¸ã§ã³ï¼v11ï¼ã§ã¯ãããªã大ããªå¤æ´ãæ©è½è¿½å ãããã¨ä¼ºã£ã¦ãã¾ãããæ³¨ç®ãã¤ã³ããããã¤ãæãã¦ãããã¾ããã ãã£ã«ã¿
Fluentdã¯ãRuby製ã®ãã°ã³ã¬ã¯ã¿ã ãã³ã¼ãã¯å ¬éããã¦ããã æ§ã ãªãã°ãæ§é åãã¦ä¸å 管çãããã¨ãã§ããåéã¨è§£æã¸ã®ãã¼ãã«ã大ããä¸ãã¦ãããã ã¤ã³ã¹ãã¼ã«ããã©ã°ã¤ã³éçºãç°¡åãæ¥æ¬èªã®è³æãå¤ãã ãã®è³æãæ§ã ãããããã©ã°ã¤ã³ãè¦ããªããããæè¯ã ã¨æããå¿ è¦ãªæ å ±ãããã¾ã¨ã¾ã£ã¦ãããå¿ èªã¨ãããã Big Dataå ¥éã«è¦ããããFluentdå ¥é from Keisuke Takahashi ãã¼ã¿ã®ç¢ºå®ãªè»¢éãå®ç¾ãããããã¡æ©è½ã«ã¤ãã¦ã¯ãæ± ç°å¤§è¼ããã®ããã°ã詳ããããã¦ãFluentdã¯ãã¼ã¿ãåéãã¦ãããããä¿åã¯ãã¦ãããªãã æ°¸ç¶åã«ã¯ãã¼ã¿ãã¼ã¹ãå¿ è¦ã ã ããã§ãRiakã Basho社ãã¹ãã³ãµã¼ãããErlangè£½åæ£åKVSããããOSSã ããå¥ç´ã«ãã£ã¦åç¨ãµã¼ãã¹ãåããããã ãããã¾ãã¨ãã¸ç«ã¡ã¾ãã£ã¦ã¦
OSSã®ãã°åé管çãã¼ã«Fluentdãç¨ãã¦ãã°ãçµ±å管çãã¦ããå ´åã®æ¸å¿µç¹ã¨ãã¦ããã°ã®åéæ¼ããèãããã¾ãã Fluentdã§ã¯ããããã¡æ©è½ãæ´»ç¨ãããã¨ã§ãã°ãåéæ¼ããããã¨ãªã確å®ã«åéãããã¨ãã§ãã¾ãã ãã®ãããã¡æ©è½ã®ã¡ã«ããºã ãçè§£ãã¹ãå使¤è¨¼ããçµæãç´¹ä»ãã¾ãã対象ã¨ããFluentdã®ãã¼ã¸ã§ã³ã¯0.10.30ã§ãã Fluentdã¨ã¯ Rubyå®è£ ã®OSSã®ãã°åé管çãã¼ã«ã§ãã Fluentdã¯ãInputãBufferãOutputã®3ã¤ã®ã³ã³ãã¼ãã³ãã§å®ç¾ããã¦ãã¾ãã æ§ã ãªå ´æãããã°ãåéãJSONå½¢å¼ã«å¤æã(Input)ãèç©(Buffer)ãæ§ã ãªåºåå ã«ãã¼ã¿åºå(Output)ãã¾ãã ä¾ã¨ãã¦ããããµã¼ã(server01)ã®Apacheã®ã¢ã¯ã»ã¹ãã°ãå¥ã®ãµã¼ã(server02)å ã«ãã¡ã¤ã«ã¨ãã¦åºåããå ´å
ã¾ãã両è ã¯ããªãæ§è³ªã®ç°ãªããããã¯ããªã®ã§ã以ä¸ã®æ¯è¼ã¯çéãã çéãã§ãããã¨ãåæã«ãã¹ããªã¼ã ãã¼ã¿å¦çãã©ãããã©ã¼ã ã¨ãã¦ã®ä¸¡è ãæ¯ã¹ã¦ã¿ãã åºæ¬æ å ± fluentd http://fluentd.org/ ä»ãã¨ããããã°ã³ã¬ã¯ã¿ã¼/ã¤ãã³ãã¢ã°ãªã²ã¼ã¿ã¼ãRubyã§å®è£ ããã¦ããã軽éé«éã RPCåºç¤ã§ã¯ãªãããã®ä¸ã®ã¬ã¤ã¤ã¼ã«ä½ç½®ãããããã¯ãã Storm http://storm-project.net/ 忣RPCåºç¤ãã¹ããªã¼ã ãã¼ã¿çMapReduce風ãã¬ã¼ã ã¯ã¼ã¯ãJava+Clojureã§å®è£ ããã¦ããã æ¦è¦ã«ã¤ãã¦ã¯ãä¸è¨ã®ã¹ã©ã¤ããã¨ã¦ãããããããã£ãã Twitterã®ãªã¢ã«ã¿ã¤ã 忣å¦çã·ã¹ãã ãStormãå ¥é ã¹ããªã¼ã ãã¼ã¿å¦çã§ä½ãããã®ãã«ã¤ã㦠ã¹ããªã¼ã ãã¼ã¿å¦çã®ãã¼ãºã«ã¤ãã¦ãèªåãçè§£ãã¦ããç¯å²ã§ã®ç°¡åãªèª¬æã å ¸
JAWS DAYS 2013ã¨ããã¤ãã³ãããã£ã¦ï¼2æ¥ç®ã«Treasure Dataã¨ãã¦çºè¡¨ãã¦ãã¾ããï¼å®ã¯1æ¥ç®ã«ã¯å¼ç¤¾CTOãããã«ãã£ã¹ã«ãã·ã§ã³ã«ç»å£ããããã¦ã¾ããï¼ Slideshare Ustream Video streaming by Ustream æåã¯frsyukiãç»å£äºå®ã«ä¸ãã£ã¦ãããã ãã©ï¼ä»å½¼ã¯ã¢ã¡ãªã«ã¨ãããã¨ã§ä»£æã§å½¹å²ãåã£ã¦ããï¼ã¨ããã®ãçºè¡¨ã®çµç·¯ï¼ ãªããã©ãã«ãâDevâã¨âOpsâã¨ããè¶ å¤§éæãªãããã§ãAWSã§âDevâã£ã¦ä½çºè¡¨ããã°â¦ãã¨ããç¶æ ã ã£ãã®ã§ï¼ AWSä¸ã§å±éãã¦ããTreasure Dataãµã¼ãã¹ã®ä»çµã¿ã«ã¤ãã¦ã¤ãã¤ãã¨è©±ãã¾ããï¼ AWSã¹ã£ãããªæãã§ã¯ãªã㦠Treasure Dataã®ãµã¼ãã¹å 容ã¨ç®ç ã©ãAWSã®ãããã¯ãã使ã£ã¦ããã®ã ãªããã®AWSã®ãµã¼ãã¹ã使ããªãã®ã ã¨ããã®ã大ã¾ã
Distributed Stream Processing on Fluentd / #fluentdAI-enhanced description This document discusses NHN Japan's use of Fluentd for log collection, conversion, and analysis. It summarizes their architecture which involves collecting logs from servers using Scribeline, delivering them to Fluentd processes, converting the logs to a structured format using Fluentd plugins, writing the structured data t
id:tagomoris ããã«ã声ããããã ãã¾ãã¦ãFluentd Casual Talks ã«ã¦ãfluentdã§Webãµã¤ãéç¨ã楽ã«ãããã¨ããã¿ã¤ãã«ã§çºè¡¨ããã¦ããã ãã¾ããã çºè¡¨è³æã¯ãã¡ã 主å¬è ã® id:tagomoris ãããä¼å ´ãæä¾ãã¦ããã ãã DeNA æ§ãããããæºåããã¦ãã ãã£ã id:riywo ããã¯ããå¤ãã®æ¹ã ãåå ãã¦ãã ãã£ã100å以ä¸ã®çæ§ããããã¨ããããã¾ããï¼æ¥½ããã£ãã§ãã çºè¡¨ã§ã¯ããåå¹´ã»ã© Fluentd ãéç¨ãã¦æ¥ãçµé¨ãã話ããã¾ããããçºè¡¨å ã§è§¦ããªãã£ããã¨ã§å¤§äº(?)ãªãã¨ãããã¾ãã®ã§ä»¥ä¸ã«è£éºãããã¤ãæ¸ãã¦ããã¾ãã MongoDB ã«ãã°ãæºããããªãæ¹ããããã 太ç°ãã (@kzk_mover) ã®çºè¡¨å ã§ã触ãããã¦ãã¾ããããæ°åä¸ä»¶ç¨åº¦ã«ãã¦ãããã»ããããã®ã§ã¯â¦â¦ã¨ãã宿ã§ãã çºè¡¨å
Fluentd Casual Talksã«åå ããfluentdã®ä½¿ãã©ãããçè§£ã§ãã¦ã æèãé«ããã¡ã«æ©éfluentdã®å°å ¥ããSubsonic+nginxã®ã¢ã¯ã»ã¹ãã°ãMongoDBã«å ¥ãã¦ã¿ãã®ã§ã¡ã¢ã å°å ¥ç°å¢ã¯webãµã¼ãCentOS 6ããã°åéãµã¼ããCentOS 5ã§ãfluentdã®ãã¼ã¸ã§ã³ã¯ td-agent-1.1.5.1-0.x86_64ã«ãªãã¾ãããããããããVPSã§ç¨¼åããã¦ãã¾ãã nginxã®å°å ¥ã¯ä»¥åæ¸ããä¸è¨ã®ã¨ã³ããªãåèã«ãã¦ä¸ããã ãããVPS 2G+Subsonic+Nginxã§èªåç¨ã¹ããªã¼ãã³ã°ãµã¼ããæ§ç¯ - Glide Note - ã°ã©ã¤ããã¼ã ä»åè¡ã£ã使¥ã®æµã fluentdãåãã¦è§¦ãã®ã§ãä»åã¯ä¸è¨ã®ããã«é ã追ã£ã¦æ¤è¨¼ã¨å°å ¥ãè¡ãã¾ããã fluentdã®å°å ¥ ãµã¼ã1å°ã§webãµã¼ãã¨ãã°åéãµã¼ããåãã
fluentd ã®ã½ã¼ã¹ãèªã¿å§ãã¾ãããåãªãè¶£å³ã§ãã çµç·¯: fluentd ã® exec buffered output plugin ã試ãã¦ã¿ããflush_interval ãçããã¦ã USR1 ã·ã°ãã«ãéã£ã¦ããããã¡ã flush ããã command ãå®è¡ãããªãã¦æ©ãã§ãããTimeSlicedOutput ã¯ã¹ã©ã¤ã¹ã®æéãçµéããªãã¨ãããã¡ããåºã¦ããªãã¨æãã¦ããããã½ã¼ã¹ãèªãã§ã¡ããã¨çè§£ãããã¨æã£ããããã§ãã ã¨ã¯ãã plugin/out_exec.rb ããããªãèªãã§ããã£ã±ãããããªãã£ãã®ã§ãé ã追ã£ã¦èªãã§ã¿ããã¨ã«ãã¾ãã ã¾ã㯠bin/fluentd ãå®è¡ãã㨠fluend ãèµ·åããã¨ããã¾ã§ã bin/fluend 㯠fluend/command/fluentd.rb ãå®è¡ãã¦ãã ã fluent/command
ã¯ã¬ã¸ããã«ã¼ãç¾éåè©æ¬ºãæ¥ç人ãæããå£ã³ãæ å ±ã åã¯è¡ã£ã¦ãªããã§ããTwitterãUstreamãã¹ã©ã¤ããããã°ãªã©ãè¦ãéãã ãã¶çãä¸ãã£ãããã§ããã¼ãåèªèº«ãä»äºã§ä½¿ãäºå®ã¯ä»ã®ã¨ããç¡ããã§ãããã°è§£æé¢é£ã®ä»äºããã¦ãããã¨ãããã®ã§ç´ æ¯ããã¦ã¿ããã¨æãã¾ãã ç°å¢ã¯VirtualBoxä¸ã®CenOS 5.7(x86_64)ã使ãã¾ããã fluentdã¯Ruby 1.9ä¸ã§åããã§ããCentOS 5.7ã«å ¥ã£ã¦ããã®ã¯Ruby 1.8.5ã§ããRuby 1.9ã®ã¤ã³ã¹ãã¼ã«ããå§ããã¨ã¯ã¾ããããªã®ã§yumã§ã¤ã³ã¹ãã¼ã«ã§ããtd-agentã使ãã¾ããtd-agentã¯fluentdã®å®å®çããã±ã¼ã¸ã¨ããä½ç½®ä»ãã®ããã§ãã 試ããã®ã¯ä¸è¨ï¼ã¤ã§ãã fluent-catã§ãã°ãéã Apacheã®ã¢ã¯ã»ã¹ãã°ãåé Apacheã®ã¢ã¯ã»ã¹ãã°ãMong
ã©ã³ãã³ã°
ã©ã³ãã³ã°
ãç¥ãã
ãªãªã¼ã¹ãé害æ å ±ãªã©ã®ãµã¼ãã¹ã®ãç¥ãã
ææ°ã®äººæ°ã¨ã³ããªã¼ã®é ä¿¡
å¦çãå®è¡ä¸ã§ã
j次ã®ããã¯ãã¼ã¯
kåã®ããã¯ãã¼ã¯
lãã¨ã§èªã
eã³ã¡ã³ãä¸è¦§ãéã
oãã¼ã¸ãéã
{{#tags}}- {{label}}
{{/tags}}