[B! fluentd] Konboiã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

Konboi id:Konboi

fluentdã«é–¢ã™ã‚‹Konboiã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (4)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

fluent-logger-golang ã®å®Ÿæˆ¦çš„ãªä½¿ã„ã‹ãŸã¾ã¨ã‚ - é…’æ—¥è¨˜ ã¯ã¦ãªæ”¯åº—
OSSç´¹ä»‹ã‚¢ãƒ‰ãƒ™ãƒ³ãƒˆã‚«ãƒ¬ãƒ³ãƒ€ãƒ¼ ã®14æ—¥ç›®ã®è¨˜äº‹ã§ã™ã€‚ Fluentd ã® å…¬å¼ Go ç‰ˆ Logger ã§ã‚ã‚‹ fluent-logger-golang ã¯ã“ã®ã‚ˆã†ã«ä½¿ã†ã®ãŒã‚ˆã•ãã†ã€ã¨ã„ã†ä½¿ã„æ–¹ã‚’ã¾ã¨ã‚ã¦ã¿ã¾ã—ãŸã€‚ å…ƒã€…ç¤¾å†…ã§æ›¸ã„ã¦ãŠã„ãŸãƒ‰ã‚ãƒ¥ãƒ¡ãƒ³ãƒˆã‚’ç·¨é›†ã—ãŸã‚‚ã®ã§ã™ã€‚ github.com å‰æã®ãƒ¦ãƒ¼ã‚¹ã‚±ãƒ¼ã‚¹ Webã‚¢ãƒ—ãƒªã‚±ãƒ¼ã‚·ãƒ§ãƒ³(APIã‚µãƒ¼ãƒ) ã‚’ Go ã§æ›¸ã„ã¦ã„ã¦ã€ãã“ã‹ã‚‰ä½•ã‚‰ã‹ã®ãƒã‚°ã‚’ Fluentd ã«é€ä¿¡ã—ãŸã„ã€‚ config ã®ãŠå‹§ã‚ã‚ªãƒ—ã‚·ãƒ§ãƒ³ Timeout : Connect ã«å¯¾ã™ã‚‹ã‚¿ã‚¤ãƒ ã‚¢ã‚¦ãƒˆã€‚ãƒ‡ãƒ•ã‚©ãƒ«ãƒˆ3ç§’ãªã®ã§ãã®ã¾ã¾ã§ã‚ˆã•ãã† WriteTimeout : æ›¸ãè¾¼ã¿ã®ã‚¿ã‚¤ãƒ ã‚¢ã‚¦ãƒˆã€‚ãƒ‡ãƒ•ã‚©ãƒ«ãƒˆã ã¨ãšã£ã¨å¾…ã£ã¦ã—ã¾ã†ã®ã§ 3 ç§’ã¨ã‹? BufferLimit : ãƒ‡ãƒ•ã‚©ãƒ«ãƒˆ 8MB ã“ã‚Œã‚’è¶…ãˆã‚‹ã¨æ¨ã¦ã‚‰ã‚Œã¦ã—ã¾ã†ã€‚é€ã‚‹æµé‡ã«ã‚ˆã£ã¦èª¿æ•´ãŒå¿…è¦ MaxRetry
Konboi 2017/12/14
çŸ¥è¦‹ã

fluentd

golang
ãƒªãƒ³ã‚¯
NLBã§fluentdã®forwardãƒ‘ã‚±ãƒƒãƒˆã‚’åˆ†æ•£ã•ã›ã¦ã¿ãŸ - Qiita
AWSã®æ–°ã—ã„ãƒãƒ¼ãƒ‰ãƒãƒ©ãƒ³ã‚µã§ã‚ã‚‹NLB(Network Load Balancer)ã‚’ä½¿ã£ã¦fluentdã®forwardãƒ‘ã‚±ãƒƒãƒˆã‚’åˆ†æ•£ã—ã¦ã¿ãŸã®ã§ã€ãƒ¬ãƒãƒ¼ãƒˆã‚’ã¾ã¨ã‚ã¦ãŠãã€‚ NLBè‡ªä½“ã«ã¤ã„ã¦ã¯ã€ã‚¯ãƒ©ã‚¹ãƒ¡ã‚½ãƒƒãƒ‰ã®ãƒ–ãƒã‚°ç‰ã§ç´¹ä»‹ã•ã‚Œã¦ã„ã‚‹ã®ã§ãã¡ã‚‰ã‚’å‚ç…§ã™ã‚‹ã®ãŒåˆ†ã‹ã‚Šæ˜“ã„ã€‚ é™çš„ãªIPã‚’æŒã¤ãƒãƒ¼ãƒ‰ãƒãƒ©ãƒ³ã‚µãƒ¼Network Load Balancerï¼ˆNLBï¼‰ãŒç™ºè¡¨ã•ã‚Œã¾ã—ãŸï¼ è©¦ã—ã¦ã‚ã‹ã£ãŸ NLB ã®ç´°ã‹ã„ãŠä½œæ³• ã–ã£ãã‚Šè¨€ã†ã¨ã€TCPãƒ—ãƒãƒˆã‚³ãƒ«ã‚’å¯¾è±¡ã«ã—ãŸALBã£ã¦æ„Ÿã˜ã€‚ ã‚¿ãƒ¼ã‚²ãƒƒãƒˆã‚°ãƒ«ãƒ¼ãƒ—ã¯ãƒãƒ¼ãƒˆãƒ¬ãƒ™ãƒ«ã§è¨å®šã§ãã‚‹ã®ã§ã€ã‚³ãƒ³ãƒ†ãƒŠç’°å¢ƒã¨ç›¸æ€§ãŒè‰¯ã„ã€‚ ãƒãƒ¼ãƒˆé•ã„ã§è¤‡æ•°fluentdãŒç«‹ã£ã¦ã„ã¦ã‚‚åŒã˜ã‚°ãƒ«ãƒ¼ãƒ—ã¨ã—ã¦ã¾ã¨ã‚ã¦åˆ†æ•£ã§ãã‚‹ã€‚ åˆ©ç”¨ã¾ã§ã®æµã‚Œ ã‚¿ãƒ¼ã‚²ãƒƒãƒˆã‚°ãƒ«ãƒ¼ãƒ—ã‚’ä½œæˆã—ã€TCPãƒ¬ãƒ™ãƒ«ã§ã‚³ãƒã‚¯ã‚·ãƒ§ãƒ³ãŒè²¼ã‚Œã‚‹ã‹ã®ãƒ˜ãƒ«ã‚¹ãƒã‚§ãƒƒã‚¯ã®è¨å®šã‚’ã™ã‚‹ ã‚¤ãƒ³ã‚¹ã‚¿ãƒ³ã‚¹ã‚‚ã—ãã¯å¯¾è±¡IPã¨ã€ãƒãƒ¼ãƒˆã®çµ„ã‚’
Konboi 2017/10/03
aws

nlb

fluentd
ãƒªãƒ³ã‚¯
ã‚¹ã‚±ãƒ¼ãƒ©ãƒ–ãƒ« tail -f | grep on AWS - KAYAC Engineers' Blog
ã“ã‚“ã«ã¡ã¯ã€‚çµ„é•·ã“ã¨ @fujiwara (id:sfujiwara) ã§ã™ã€‚ æ—©é€Ÿã§ã™ãŒçš†ã•ã‚“ã€ãƒã‚°ã¯è¦‹ã¦ã„ã¾ã™ã‹? å½“ç„¶è¦‹ã¦ã„ã¾ã™ã‚ˆãã€‚tail -f ã§è¿½ã„ã‹ã‘ã‚‹ã¨ã‹å¤§å¥½ãã§ã™ã‚ˆãã€‚ã¨ã„ã†å‰æã§è©±ã‚’é€²ã‚ã¾ã—ã‚‡ã†ã€‚ ã“ã®è¨˜äº‹ã¯ Tech KAYAC Advent Calendar 2016 ã®24æ—¥ç›®ã®è¨˜äº‹ã§ã™ã€‚ TL;DR å¤šç¨®å¤šæ§˜ãªãƒ›ã‚¹ãƒˆã‹ã‚‰ç™ºç”Ÿã™ã‚‹ãƒã‚°ã®æœ«å°¾ã‚’ tail -f ã§è¿½ã„ã‹ã‘ãŸã„ fluentd + ãƒãƒ¼ã‚«ãƒ«ãƒ•ã‚¡ã‚¤ãƒ«ã§è¡Œã†ã®ã¯ã¡ã‚‡ã£ã¨é¢å€’ Amazon Kinesis + Lambda ã‚’ä½¿ã£ã¦ã‚¹ã‚±ãƒ¼ãƒ©ãƒ–ãƒ«ãªä»•çµ„ã¿ã‚’è€ƒãˆã¦ã¿ãŸ kinesis-tailf ã¨ã„ã†ãƒ„ãƒ¼ãƒ«ã‚‚ä½œã£ãŸ æ—¢å˜ã®ä»•çµ„ã¿ã¨å•é¡Œç‚¹ tech blog.kayac.com å¼Šç¤¾ã® Lobi ã¨ã‚½ãƒ¼ã‚·ãƒ£ãƒ«ã‚²ãƒ¼ãƒ ã§ã¯ã€ã“ã®è¨˜äº‹ã§ç´¹ä»‹ã•ã‚ŒãŸã‚ˆã†ãª Fluentd ã‚’æ´»ç”¨ã—ãŸãƒã‚°é›†ç´„ã®ä»•çµ„ã¿ã‚’æ•´å‚™ã—ã¦ã„ã¾ã™ã€‚ã“ã®
Konboi 2016/12/27
fluentd

AWS Lambda

kinesis
ãƒªãƒ³ã‚¯
S3ã¨Fluentdã‚’ç”¨ã„ãŸåŠ¹çŽ‡çš„ãªãƒã‚°ç®¡ç† | SmartNewsé–‹ç™ºè€…ãƒ–ãƒã‚°
ã‚´ã‚¯ãƒã®å¤§å¹³ã§ã™ã€‚ ç§ã«ã¨ã£ã¦ä¸€ç•ªå¤§äº‹ã§æ›¿ãˆã®åŠ¹ã‹ãªã„ãƒŸãƒ¥ãƒ¼ã‚¸ã‚·ãƒ£ãƒ³ã¯ã•ã ã¾ã•ã—ã•ã‚“ã§ã™ãŒã€ç§ã«ã¨ã£ã¦ã‚¯ãƒ©ã‚¦ãƒ‰ã‚³ãƒ³ãƒ”ãƒ¥ãƒ¼ãƒ†ã‚£ãƒ³ã‚°ã®ã‚µãƒ¼ãƒ“ã‚¹ã®ä¸ã§ä¸€ç•ªå¤§äº‹ã§æ›¿ãˆãŒåŠ¹ã‹ãªã„ã‚µãƒ¼ãƒ“ã‚¹ã¯S3ã§ã™ã€‚ å¤šç¨®å¤šæ§˜ãªAPIã‚’ç”¨ã„ã¦æŸ”è»Ÿã«ãƒ•ã‚¡ã‚¤ãƒ«ã®æ“ä½œãŒå‡ºæ¥ã‚‹äº‹ã‚„ã€â€99.999999999ï¼…â€ã¨è¬³ã‚ã‚Œã¦ã„ã‚‹é«˜ã„è€éšœå®³æ€§ã€S3ã«ã‚ã‚‹ãƒ‡ãƒ¼ã‚¿ã‚’Elastic MapReduceã‚„Redshiftãªã©ã‚’ç”¨ã„ã¦æ‰‹è»½ã«ãƒ‡ãƒ¼ã‚¿è§£æžã‚’è¡Œãˆã‚‹åŸºç›¤ãŒæä¾›ã•ã‚Œã¦ã„ã‚‹ã“ã¨ãªã©ã€ã‚ã¾ã‚Šã«ä¾¿åˆ©ã™ãŽã¦S3ã®ä»£æ›¿ã¨ãªã‚‹ã‚µãƒ¼ãƒ“ã‚¹ã‚’æŽ¢ã—ã ã™ã®ãŒé›£ã—ã„çŠ¶æ…‹ã§ã™ã€‚ ã‚‚ã¡ã‚ã‚“å¤šãã®AWSãƒ¦ãƒ¼ã‚¶ãƒ¼ãŒåŒã˜ã‚ˆã†ã«S3ã®ä¾¿åˆ©ã•ã‚’äº«å—ã—ã¦ã„ã‚‹ã¨æ€ã„ã¾ã™ã—ã€ã‚¤ãƒ³ã‚¿ãƒ¼ãƒãƒƒãƒˆä¸Šã§ã‚‚å¤šãã®ãƒ–ãƒã‚°ç‰ã§ãã®é…åŠ›ãŒèªžã‚‰ã‚Œã¦ã„ã¾ã™ã€‚ãã®ä¸ã§æœ¬è¨˜äº‹ã¯æ—¢ã«å˜åœ¨ã™ã‚‹è¨˜äº‹ã¨ä¼¼ãŸã‚ˆã†ãªå†…å®¹ã‚’æ›¸ã„ã¦ã—ã¾ã†ã‹ã‚‚ã—ã‚Œã¾ã›ã‚“ãŒã€å¼Šç¤¾ãªã‚Šã®S3ã®ä½¿ã„æ–¹ã«ã¤ã„ã¦ã”ç´¹ä»‹ã—ãŸã„ã¨æ€ã„ã¾ã™ã€‚ ãªãŠ
Konboi 2013/09/02
s3

fluentd

ãƒã‚°
ãƒªãƒ³ã‚¯
1