[B! Scribe] shiumachiã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

shiumachi id:shiumachi

Scribeã«é–¢ã™ã‚‹shiumachiã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (7)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

scribedã‚’ç°¡å˜ã«èµ·å‹•/åœæ¢ã§ãã¦ãƒã‚°å‡ºåŠ›ã‚’ä¾¿åˆ©ã«ã§ãã‚‹init scriptæ›¸ã„ãŸ - ãŸã”ã‚‚ã‚Šã™ãƒ¡ãƒ¢
æŽ¨è¨ˆå…¨å›½20äººãã‚‰ã„ã¯ã„ã‚‹ã¨æ€ã‚ã‚Œã‚‹scribedç®¡ç†è€…ã«ã¯ã‚ã‹ã£ã¦ã‚‚ã‚‰ãˆã‚‹ã¨æ€ã†ã‘ã©scribedã®èµ·å‹•/åœæ¢ã®åˆ¶å¾¡ã¯ã„ã‚ã„ã‚ã¨é¢å€’ãã•ã„ã—ãƒã‚°ã‚‚daemontoolsã¨ã‹ã«ãŠé¡˜ã„ã—ãªã„é™ã‚Šã¯ãªã‚“ã‹ã†ã–ã„ã€‚ãã—ã¦æ›´ã«ãã®ã†ã¡ã®10äººãã‚‰ã„ã«ã¯ã‚ã‹ã£ã¦ã‚‚ã‚‰ãˆã‚‹ã¨æ€ã†ãŒscribedã‹ã‚‰HDFSã«æ›¸ãè¾¼ã¿ã‚’ã—ã‚ˆã†ã¨æ€ã†ã¨èµ·å‹•ãƒ¦ãƒ¼ã‚¶ã‚‚rootä»¥å¤–ã«ã—ãªã„ã¨ã„ã‘ãªã„ã‚“ã ã‘ã©åœæ¢ã‚³ãƒžãƒ³ãƒ‰ã¯rootã§å®Ÿè¡Œã—ãªã„ã¨ã„ã‘ãªãã¦ã†ã‚ã‚ã‚ã‚ã‚ã‚ã‚ã‚ã‚ã‚ã‚ã‚ã‚ã‚ã‚ã‚ã‚ã‚ã‚ã‚ã‚ã‚ã‚ã‚ã‚ã€‚ ã¨ã‹ã„ã†ã‚¹ãƒˆãƒ¬ã‚¹ãŒé™ç•Œã«é”ã—ãŸã®ã§init scriptã‚’æ›¸ã„ãŸã€‚ tagomoris/scribed_launcher Â· GitHub ã“ã‚Œã¾ã§ èµ·å‹• $ su - hadoop $ /usr/bin/nohup scribed /etc/scribed.conf | /usr/local/apache2/bin/rota
shiumachi 2011/10/27
Scribe
ãƒªãƒ³ã‚¯
Hadoop at Twitter (Hadoop Summit 2010)
Kevin Weil presented on Hadoop at Twitter. He discussed Twitter's data lifecycle including data input via Scribe and Crane, storage in HDFS and HBase, analysis using Pig and Oink, and data products like Birdbrain. He described how tools like Scribe, Crane, Elephant Bird, Pig, and HBase were developed and used at Twitter to handle large volumes of log and tabular data at petabyte scale.Read less
shiumachi 2010/06/30
Hadoop

twitter

Scribe

HBase

Pig

Crane

ElephantBird
ãƒªãƒ³ã‚¯
Scribeã§ãƒã‚°ã®é›†ç´„ãƒ»åŽé›†ã€è¿½è¨˜ã€‘ - ãƒ•ã‚¿ãªã—ã‚«ãƒ³ãƒ…ãƒ¡
ã‚ã¨ãŒãã§ãƒã‚°ã‚’å–ã‚‹éš›ã®è² è·ãŒé«˜ã„ã¨ã„ã†ã“ã¨ã‚’æ›¸ã„ãŸã®ã§ã™ãŒã€ã‚‚ã†ä¸€ã¤ã®Webã‚µãƒ¼ãƒ(lighttpd)ã§ã‚‚ãƒã‚°ã‚’å–ã‚‹ã‚ˆã†ã«ã—ãŸã¨ã“ã‚ã€ã‚ã¾ã‚Šã«ã‚‚ãƒ¬ã‚¹ãƒãƒ³ã‚¹ãŒé…ããªã£ã¦ã—ã¾ã£ãŸã®ã§ã€Pythonã§æ›¸ãç›´ã—ã¦ã¿ã¾ã—ãŸã€‚ã¨ã„ã£ã¦ã‚‚ã‚µãƒ³ãƒ—ãƒ«ãƒ•ã‚¡ã‚¤ãƒ«ã‚’ã¡ã‚‡ã£ã¨ã„ã˜ã£ãŸã ã‘ãƒ»ãƒ»ãƒ»ã€‚ ## /usr/local/bin/scribe_httpd2 #!/usr/bin/python import sys from scribe import scribe from thrift.transport import TTransport, TSocket from thrift.protocol import TBinaryProtocol if len(sys.argv) == 2: category = sys.argv[1] host = '127.0.0.1' port = 1463 elif l
shiumachi 2010/05/28
Scribe
ãƒªãƒ³ã‚¯
Scribeã§ãƒã‚°ã®é›†ç´„ãƒ»åŽé›†ã€å¾Œç·¨ã€‘ - ãƒ•ã‚¿ãªã—ã‚«ãƒ³ãƒ…ãƒ¡
ã„ã‚ˆã„ã‚ˆWebã‚µãƒ¼ãƒã®ãƒã‚°ã‚’Scribeã‚’é€šã—ã¦å‡¦ç†ã—ã¦ã¿ã¾ã™ã€‚examplesãƒ‡ã‚£ãƒ¬ã‚¯ãƒˆãƒªã«å…¥ã£ã¦ã„ãŸã€scribe_catã¨scribe_ctrlã¯ãã®ã¾ã¾ä½¿ãˆãã†ãªã®ã§ã“ã‚Œã‚’åˆ©ç”¨ã—ã¦ã¿ã¾ã™ã€‚ # cp examples/{scribe_cat,scribe_ctrl} /usr/local/bin/ ã¾ãšã€ãƒã‚°ã‚µãƒ¼ãƒ(ãƒã‚°ã‚’é›†ç´„ã™ã‚‹ã‚µãƒ¼ãƒ)ã®è¨å®šã‚’ã—ã¾ã™ã€‚ã“ã®ã‚µãƒ¼ãƒã«å„Webã‚µãƒ¼ãƒã®ãƒã‚°ãŒæ›¸ãè¾¼ã¾ã‚Œã¦ã„ãã¾ã™ã€‚è¨å®šãƒ•ã‚¡ã‚¤ãƒ«ã¯/usr/local/etc/scribeã®ä¸‹ã«ç½®ãã‚‚ã®ã¨ã—ã¾ã™ã€‚ # mkdir /usr/local/etc/scribe # mkdir /var/log/scribe # touch /usr/local/etc/scribe/scribed.conf ãƒã‚°ã‚µãƒ¼ãƒã®è¨å®šãƒ•ã‚¡ã‚¤ãƒ«(scribed.conf)ã®å†…å®¹ã¯ä¸‹è¨˜ã«ãªã‚Šã¾ã™ã€‚ port=1463 # å¾…
shiumachi 2010/05/28
Scribe
ãƒªãƒ³ã‚¯
Scribeã§ãƒã‚°ã®é›†ç´„ãƒ»åŽé›†ã€ä¸ç·¨ã€‘ - ãƒ•ã‚¿ãªã—ã‚«ãƒ³ãƒ…ãƒ¡
å‰ç·¨ã§ã¯Scribeã®ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã¨å‹•ä½œç¢ºèªã¾ã§ãŠã“ãªã„ã¾ã—ãŸã€‚å®Ÿéš›ã«Webã‚µãƒ¼ãƒã®ãƒã‚°ã‚’Scribeã§å‡¦ç†ã—ã¦ã„ãã‚ã‘ã§ã™ãŒã€ãã®å‰ã«ãƒãƒƒãƒˆãƒ¯ãƒ¼ã‚¯ã‚’çµŒç”±ã—ã¦æ£å¸¸ã«ãƒã‚°ã®å—ã‘æ¸¡ã—ãŒã§ãã‚‹ã‹ã©ã†ã‹ã‚’èª¿ã¹ã¦ã„ãã¾ã™ã€‚ åˆ¥ã®ã‚µãƒ¼ãƒã«ã‚‚Scribeã‚’ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã—ã¦è©¦ã—ãŸã»ã†ãŒã„ã„ã®ã§ã™ãŒã€ä»Šå›žã¯ãƒãƒ¼ã‚«ãƒ«å†…ã«ãƒãƒ¼ãƒˆã‚’å¤‰ãˆã¦äºŒã¤ã®scribedã‚’èµ·å‹•ã—ã¦æ¤œè¨¼ã—ã¾ã™ã€‚ ## ãƒã‚°ã‚µãƒ¼ãƒå´ã¨ä»®å®š ### ## ä»Šå›žã¯è¨å®šã‚µãƒ³ãƒ—ãƒ«ã®centralã¨ã„ã†ã®ã‚’ä½¿ã£ã¦ã¿ã‚‹ $ mkdir /tmp/scribetest $ /usr/local/bin/scribed examples/example2central.conf ## Webã‚µãƒ¼ãƒå´ã¨ä»®å®š ### ## åˆ¥ã®ã‚·ã‚§ãƒ«ã‹ã‚‰åŒã˜ã‚ˆã†ã«è¨å®šãƒ•ã‚¡ã‚¤ãƒ«ã‚’å¤‰ãˆã¦ç«‹ã¡ä¸Šã’ã‚‹ $ mkdir /tmp/scribetest2 $ /usr/local/bin/sc
shiumachi 2010/05/28
Scribe
ãƒªãƒ³ã‚¯
Scribeã§ãƒã‚°ã®é›†ç´„ãƒ»åŽé›†ã€å‰ç·¨ã€‘ - ãƒ•ã‚¿ãªã—ã‚«ãƒ³ãƒ…ãƒ¡
å‰å›žã®è¨˜äº‹ã§é™çš„ãƒ•ã‚¡ã‚¤ãƒ«ã‚’åˆ¥ã‚µãƒ¼ãƒã‹ã‚‰é…ä¿¡ã™ã‚‹ã“ã¨ã«ã—ã¾ã—ãŸãŒã€ä»Šåº¦ã¯ã‚¢ã‚¯ã‚»ã‚¹ãƒã‚°ãŒã‚µãƒ¼ãƒæ¯Žã«åˆ†æ•£ã—ã¦ã—ã¾ã†ã®ã§ã€å°‘ã€…ã‚„ã£ã‹ã„ã ãªã¨æ€ã„ã¾ã—ãŸã€‚ å‡ºæ¥ã‚‹ã ã‘æ£ç¢ºã§ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ã«è¿‘ã„å½¢ã§ãƒã‚°ã‚’åŽé›†ã§ãã‚Œã°ã„ã„ãªã¨ã„ã†æ„Ÿã˜ã§ã™ã€‚ ã¾ãšã€è€ƒãˆã¦ã„ãŸã®ã¯ä¸€å®šæœŸé–“æ¯Žã«ä¸€ã¤ã®ã‚µãƒ¼ãƒã«ãƒã‚°ã‚’åŽé›†ã—ã¦ã„ãæ–¹æ³•ã§ã™ã€‚ã›ã£ã‹ãlsyncdã‚‚å…¥ã‚Œã¦ã„ã‚‹ã®ã§ã€rsyncã§åŽé›†ã™ã‚‹ã®ãŒä¸€ç•ªç°¡å˜ãã†ã§ã™ã€‚ã—ã‹ã—ã€lsyncdã§ãã®ã¾ã¾ãƒã‚°ãƒ•ã‚¡ã‚¤ãƒ«ã‚’ãƒŸãƒ©ãƒ¼ã—ã¦ã—ã¾ã†ã¨ã€ãƒã‚°ãŒæ›¸ãè¾¼ã¾ã‚Œã‚‹åº¦ã«é€ä¿¡ã—ã¡ã‚ƒã†ã‚ã‘ã§ã€‚ã€‚ã€‚ã“ã®è¾ºã‚Šã¯ã©ã†ã«ã‹å‡ºæ¥ã‚‹ã¨ã¯æ€ã„ã¾ã™ãŒã€ã¿ã‚“ãªã¯ã©ã†ã‚„ã£ã¦ã„ã‚‹ã®ã‹ã¡ã‚‡ã£ã¨èª¿ã¹ã¦ã¿ã¾ã—ãŸã€‚ ã¨ã‚Šã‚ãˆãšã€æ¤œç´¢ã—ã¦ã¿ãŸã‚‰ä¸‹è¨˜ã®ãƒšãƒ¼ã‚¸ãŒè¦‹ã¤ã‹ã‚Šã¾ã—ãŸã€‚ ãƒã‚°é›†ç´„ãƒ»åŽé›†ã«ã¤ã„ã¦ã€syslog - é›†ç´„ã€‘ - ãƒ—ãƒã‚°ãƒ©ãƒž ç¦é‡ ä¼¸å¤ªæœ— ã€œåŸºæœ¬ã¸å¸°ã‚ã†ã€œ ã©ã†ã‚„ã‚‰ã€ã€Œé›†ç´„ã€ã¨ã€ŒåŽé›†ã€ã§ã¯æ„å‘³åˆã„ãŒé•ã†æ¨¡æ§˜ã€‚ãŸã—ã‹ã«ã
shiumachi 2010/05/28
Scribe
ãƒªãƒ³ã‚¯
GitHub - facebookarchive/scribe: Scribe is a server for aggregating log data streamed in real time from a large number of servers.
Archived Repo ============= This is an archived project and is no longer supported or updated by Facebook. Please do not file issues or pull-requests against this repo. If you wish to continue to develop this code yourself, we recommend you fork it. ------------- Introduction ============ Scribe is a server for aggregating log data that's streamed in real time from clients. It is designed to be sc
shiumachi 2010/05/28
Scribe

Hadoop

log

facebook
ãƒªãƒ³ã‚¯
1