[B! hdfs] muddydixonã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

muddydixon id:muddydixon

hdfsã«é–¢ã™ã‚‹muddydixonã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (16)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

ã€ŒHadoopã®æ™‚ä»£ã¯çµ‚ã‚ã£ãŸã€ã®æ„å‘³ã‚’æ£ã—ãç†è§£ã™ã‚‹ - ç§‘å¦ã¨éžç§‘å¦ã®è¿·å®®
Hadoopã®æ™‚ä»£ã¯çµ‚ã‚ã£ãŸã€ã¨ã„ã†è¨€èª¬ã‚’ãŸã¾ã«è¦‹ã‹ã‘ã‚‹ã‚ˆã†ã«ãªã‚Šã¾ã—ãŸã€‚ ã‚‚ã¡ã‚ã‚“çµ‚ã‚ã£ã¦ãªã©ã„ã¾ã›ã‚“ã€‚ã—ã‹ã—ã€Hadoopã¨ãã®å–ã‚Šå·»ãç’°å¢ƒãŒå¤‰åŒ–ã—ãŸã®ã¯äº‹å®Ÿã§ã™ã€‚ æœ¬è¨˜äº‹ã§ã¯ã€ã“ã®å¤‰åŒ–ãŒä½•ãªã®ã‹ã‚’æ˜Žã‚‰ã‹ã«ã—ã€ãã®ä¸Šã§ã€ãªãœHadoopã®æ™‚ä»£ã¯çµ‚ã‚ã£ãŸã¨ã„ã†ä¸»å¼µãŒå®Ÿæ…‹ã‚’æ£ã—ãè¡¨ã—ã¦ã„ãªã„ã®ã‹ã‚’èª¬æ˜Žã—ã¦ã„ãã¾ã™ã€‚ DISCLAIMER ç§ã¯Hadoopã‚’ä¸å¿ƒã¨ã—ãŸãƒ‡ãƒ¼ã‚¿åŸºç›¤ã‚’å–ã‚Šæ‰±ã†ãƒ™ãƒ³ãƒ€ãƒ¼ã€Clouderaã®ç¤¾å“¡ã§ã™ã€‚ ä¸ç«‹çš„ã«æ›¸ãã‚ˆã†åŠªã‚ã¾ã™ãŒã€æ‰€å±žçµ„ç¹”ã«ã‚ˆã£ã¦ç™ºç”Ÿã™ã‚‹ãƒã‚¤ã‚¢ã‚¹ã®å®Œå…¨ãªæŽ’é™¤ã‚’ä¿è¨¼ã™ã‚‹ã“ã¨ã¯ã§ãã¾ã›ã‚“ã€‚ ä»¥ä¸Šã‚’ã”äº†æ‰¿ã®ä¸Šã€èªã¿é€²ã‚ã¦ãã ã•ã„ã€‚ è¦ç´„ ãƒ‡ãƒ¼ã‚¿åŸºç›¤ã¯ã€Hadoopã®ç™»å ´ã«ã‚ˆã‚Šéžå¸¸ã«å®‰ä¾¡ã¨ãªã‚Šã€ä»Šã¾ã§ã§ã¯ä¸å¯èƒ½ã ã£ãŸå¤§é‡ã®ãƒ‡ãƒ¼ã‚¿ã‚’å–ã‚Šæ‰±ãˆã‚‹ã‚ˆã†ã«ãªã‚Šã¾ã—ãŸã€‚ Hadoopã¯ã€NoSQLãƒ–ãƒ¼ãƒ ã®ä¸ã€å‡¦ç†ã‚¨ãƒ³ã‚¸ãƒ³ã§ã‚ã‚‹MapReduceã¨ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ã§ã‚ã‚‹HDFSãŒ
muddydixon 2017/07/10
hadoop

mapreduce

history

storage

hdfs

spark
ãƒªãƒ³ã‚¯
ä»Šæ—¥ã‹ã‚‰ã¯ã˜ã‚ã‚‹ãŠæ‰‹è»½ Hive ãƒ‡ãƒ¼ã‚¿ç§»è¡Œ - mixi engineer blog
ã“ã‚“ã«ã¡ã¯ï¼Ž æ˜¨å¹´çŸ¥äººã®ã‚ªãƒ¼ã‚±ã‚¹ãƒˆãƒ©æ¼”å¥ä¼šã§è´ã„ãŸãƒ¡ãƒ³ãƒ‡ãƒ«ã‚¹ã‚¾ãƒ¼ãƒ³äº¤éŸ¿æ›²ç¬¬ï¼”ç•ªã€Œã‚¤ã‚¿ãƒªã‚¢ã€ãŒå¤§å¥½ããªï¼ŒæŠ€è¡“éƒ¨ã®çŸ³å·æœ‰ã§ã™ï¼Ž ãã‚“ãªã€Œã‚¤ã‚¿ãƒªã‚¢ã€å¤§å¥½ããªç§ã§ã™ãŒ Hive ã®ãƒ‡ãƒ¼ã‚¿ç§»è¡Œã‚‚å¤§å¥½ãã¨ã„ã†ä½“ã§ï¼Œ ä»Šå›žã®è¨˜äº‹ã§ã¯ã‚ªãƒ³ãƒ—ãƒ¬ãƒŸã‚¹ã§ Hive ã‚’é ‘å¼µã£ã¦é‹ç”¨ã—ã¦ã„ã‚‹æ–¹ãŸã¡ã«å‘ã‘ã¦ï¼Œã©ã†ã—ã¦ã‚‚ã¨ã„ã†ã¨ãã® Hive ã®ãƒ‡ãƒ¼ã‚¿ç§»è¡Œæ–¹æ³•ã‚’ç´¹ä»‹ã—ãŸã„ã¨æ€ã„ã¾ã™ï¼Ž å¼Šç¤¾ã§ã¯æœ€è¿‘ï¼ŒCloudera's Distribution including Apache Hadoop 3 update 2 (CDH3u2) ã‹ã‚‰ CDH 4.1.2 ã«ãƒãƒ¼ã‚¸ãƒ§ãƒ³ã‚¢ãƒƒãƒ—ã‚’è¡Œãªã£ã¦ã„ã¾ã™ï¼Ž ã“ã®ã‚ˆã†ã«ã‚ªãƒ³ãƒ—ãƒ¬ãƒŸã‚¹ã§ Hive ã‚’é‹ç”¨ã—ã¦ã„ã‚‹æ–¹ãŸã¡ã®ä¸ã«ã¯ï¼Œã™ã§ã«æŒã£ã¦ã„ã‚‹ãƒ‡ãƒ¼ã‚¿ã‚’ã†ã¾ãç§»è¡Œã—ãŸã„ã¨ã„ã†æ–¹ã‚‚å¤šã„ã‹ã¨æ€ã„ã¾ã™ï¼Ž Hive ã«ã¯ç¾çŠ¶ MySQL ã® mysqldump ã®ã‚ˆã†ãªæ©Ÿèƒ½ãŒæä¾›ã•ã‚Œã¦ã„ãªã„ã®ã§ãƒ
muddydixon 2014/07/25
hdfs

hive

hadoop
ãƒªãƒ³ã‚¯
Fluentdï¼‹WebHDFSã®æ›¸ãè¾¼ã¿å•é¡Œ | å¤–é“çˆ¶ã®åŒ
ä»¥å‰ã«æ™’ã—ãŸFluentdã‹ã‚‰WebHDFSã«å¯¾ã—ã¦ãƒã‚°ã‚’æµã—è¾¼ã‚€ãƒ•ãƒãƒ¼ã®éƒ¨åˆ†ã‚’ã€ å°‘ã—ã‚ãƒ„ç›®ã®ç’°å¢ƒã«ãƒ–ã£ã“ã‚“ã§é‹ç”¨ã—ãŸã‚‰è‰²ã€…å•é¡Œç‚¹ãŒã§ã¦ããŸã®ã§è¨˜ã—ã¦ãŠãã¾ã™ã€‚ ã©ã¡ã‚‰ã‹ã¨ã„ã†ã¨Fluentdã¨ã„ã†ã‚ˆã‚Šã¯HDFSã‚ˆã‚Šã®è©±ã«ãªã‚Šã¾ã™ã€‚ HDFSãƒ•ã‚¡ã‚¤ãƒ«ã®CREATEã‚¨ãƒ©ãƒ¼ è¤‡æ•°ã®Fluentd Collectorã‹ã‚‰HDFSã®ï¼‘ãƒ•ã‚¡ã‚¤ãƒ«ã¸å…±é€šã—ã¦æ›¸ãè¾¼ã‚€ã¨ã„ã†ã‚¹ã‚¿ã‚¤ãƒ«ã‚’ã¨ã£ã¦ã¿ã¾ã—ãŸã€‚ã“ã‚“ãªæ„Ÿã˜ã«ï¼‘åˆ†æ¯Žã®HDFSãƒ•ã‚¡ã‚¤ãƒ«ã¨ã—ã¦ã€‚ Collector-01 â”€â” Collector-02 â”€â”¤ Collector-03 â”€â”¼â”€> HDFS %Y%m%d-%H%M.log Collector-04 â”€â”¤ ...
muddydixon 2014/07/25
fluentd

hadoop

hdfs

troubleshooting
ãƒªãƒ³ã‚¯
ã€14-B-2ã€‘ã‚°ãƒªãƒ¼ã‚’æ”¯ãˆã‚‹ãƒ‡ãƒ¼ã‚¿åˆ†æžåŸºç›¤ã®éŽåŽ»ã¨ç¾åœ¨ï¼ˆæ©‹æœ¬æ³°ä¸€ã€”ã‚°ãƒªãƒ¼ã€•ï¼‰
ã‚°ãƒªãƒ¼ã§ã¯ãƒ¦ãƒ¼ã‚¶ã«å–œã‚“ã§ã‚‚ã‚‰ãˆã‚‹ã‚µãƒ¼ãƒ“ã‚¹ã‚’æä¾›ã™ã‚‹ãŸã‚ã®ç¶™ç¶šçš„ãªæ”¹å–„ã‚’é‡è¦–ã—ã¦ãŠã‚Šã€å‰µæ¥æœŸã‚ˆã‚Šãƒã‚°ãƒ‡ãƒ¼ã‚¿ã®åˆ†æžåŸºç›¤ã®é–‹ç™ºãƒ»é‹ç”¨ã«æ³¨åŠ›ã—ã¦ã¾ã„ã‚Šã¾ã—ãŸã€‚æ˜¨å¹´ã‚ˆã‚Šã€å¾“æ¥ã®è‡ªç¤¾é–‹ç™ºã®è§£æžåŸºç›¤ã«åŠ ãˆã€Hadoopã‚„fluentdãªã©ã‚’æœ¬æ ¼çš„ã«é‹ç”¨é–‹å§‹ã—ã€è§£æžåŸºç›¤ã®ã•ã‚‰ãªã‚‹å¼·åŒ–ã‚’å®Ÿæ–½ã—ã¦ãŠã‚Šã¾ã™ã€‚ã‚µãƒ¼ãƒ“ã‚¹ã®æˆé•·ã‚’æ”¯ãˆã‚‹ãƒ‡ãƒ¼ã‚¿åˆ†æžåŸºç›¤ã®æ§‹ç¯‰ãƒ»é‹ç”¨ãƒ»æ´»ç”¨æ–¹æ³•ã«ã¤ã„ã¦è‡ªç¤¾ã®äº‹ä¾‹ã‚’ãƒ™ãƒ¼ã‚¹ã«ãŠè©±ã—ã¾ã™ã€‚
muddydixon 2014/02/20
presto

data

hadoop

yarn

hdfs

spark
ãƒªãƒ³ã‚¯
Cloudera Standard ã®ã”æ¡ˆå†… ~ ç„¡å„Ÿç‰ˆ å¤§å¹…æ©Ÿèƒ½å¼·åŒ–ã®ãŠçŸ¥ã‚‰ã› | Cloudera Japan
Cloudera ã¯ã€ã‚¯ãƒ©ã‚¦ãƒ‰ã€ãƒ‡ãƒ¼ã‚¿ã‚»ãƒ³ã‚¿ãƒ¼ã€ã‚¨ãƒƒã‚¸ã‚’å•ã‚ãšã€ã‚ã‚‰ã‚†ã‚‹å ´æ‰€ã®ãƒ‡ãƒ¼ã‚¿ã« AI ã®åŠ›ã‚’ã‚‚ãŸã‚‰ã™å”¯ä¸€ã®ãƒã‚¤ãƒ–ãƒªãƒƒãƒ‰ãƒ‡ãƒ¼ã‚¿ AI ãƒ—ãƒ©ãƒƒãƒˆãƒ•ã‚©ãƒ¼ãƒ ä¼æ¥ã§ã™ã€‚ ã‚ã‚‰ã‚†ã‚‹å ´é¢ã§ AI ã®æ´»ç”¨ã‚’åŠ é€Ÿ ãƒ—ãƒ©ã‚¤ãƒ™ãƒ¼ãƒˆ AI ã‚’åˆ©ç”¨ã™ã‚‹ã“ã¨ã§ã€é–‹ç™ºã‹ã‚‰æŽ¨è«–ã«è‡³ã‚‹ã¾ã§ã€ã‚¨ãƒ³ã‚¿ãƒ¼ãƒ—ãƒ©ã‚¤ã‚º AI ãŠã‚ˆã³ã‚¨ãƒ¼ã‚¸ã‚§ãƒ³ãƒˆåž‹ AI ã®ã‚¤ãƒŽãƒ™ãƒ¼ã‚·ãƒ§ãƒ³ã‚’ã‚ã‚‰ã‚†ã‚‹å ´é¢ã§åŠ é€Ÿã§ãã¾ã™ã€‚
muddydixon 2013/06/05
ãŠãŠã‰

search

hadoop

cloudera
ãƒªãƒ³ã‚¯
Rã§HDFSä¸Šã®ãƒ‡ãƒ¼ã‚¿ã‚’èªã¿æ›¸ãã™ã‚‹ãƒ‘ãƒƒã‚±ãƒ¼ã‚¸ã‚’å…¬é–‹ã—ã¾ã—ãŸ - yokkunsã®æ—¥è¨˜
Rã§Hadoopã‚’ä½¿ã†ãƒ‘ãƒƒã‚±ãƒ¼ã‚¸ã¯ã€RHadoopã¨ã‹RHIPEã¨ã‹ã‚ã‚Šã¾ã™ãŒã€ å˜ç´”ã«HDFSä¸Šã®ãƒ‡ãƒ¼ã‚¿ã‚’èªã¿æ›¸ãå‡ºæ¥ã‚‹ã ã‘ã®ã‚·ãƒ³ãƒ—ãƒ«ãªã‚‚ã®ãŒæ¬²ã—ã‹ã£ãŸã®ã§ãƒ‘ãƒƒã‚±ãƒ¼ã‚¸ã‚’ä½œã£ã¦å…¬é–‹ã—ã¦ã¿ã¾ã—ãŸã€‚ http://crantastic.org/packages/rHadoopClient Javaã¨ã‹Pigã§ä¸€æ¬¡é›†è¨ˆã—ãŸãƒ‡ãƒ¼ã‚¿ã‚’Rã§èªã¿è¾¼ã‚€æ™‚ã«ä½¿ã†ã‚¤ãƒ¡ãƒ¼ã‚¸ã§ã™ã€‚ æº–å‚™ install.packages("rHadoopClient") library(rHadoopClient) HDFSä¸Šã®ãƒ‡ãƒ¼ã‚¿ã‚’èªã¿è¾¼ã‚€ ./tmp/iris ã«ã‚¢ãƒƒãƒ—ã—ãŸirisã®ãƒ‡ãƒ¼ã‚¿ã‚’èªã¿è¾¼ã‚€ data.hdfs <- read.hdfs("tmp/iris") names(data.hdfs) <- names(iris) head(data.hdfs) ## Sepal.Length Sepal.Width Pe
muddydixon 2013/03/31
r

hdfs
ãƒªãƒ³ã‚¯
RCFileï¼ŒParquetï¼ŒORCFile
ã“ã®2ãƒ¶æœˆã§ï¼ŒCloudera/Twitterï¼ŒHortonworks ã‹ã‚‰ãã‚Œãžã‚Œåˆ¥ã®åˆ—æŒ‡å‘ãƒ•ã‚¡ã‚¤ãƒ«ãƒ•ã‚©ãƒ¼ãƒžãƒƒãƒˆãŒå…¬é–‹ã•ã‚Œã¾ã—ãŸï¼ŽParquetÂ ã¨ ORCFile ã§ã™ï¼Ž ã“ã®è¨˜äº‹ã§ã¯ï¼Œã¾ãš RCFile ã®å¾©ç¿’ã‚’ã—ã¦ï¼Œãã®å¾Œ Parquet ã¨ ORCFile ãã‚Œãžã‚Œã®å…±é€šç‚¹ã¨é•ã„ã‚’ãŠãŠã¾ã‹ã«è¦‹ã¦ã„ã“ã†ã¨æ€ã„ã¾ã™ï¼Žã‚³ãƒ¼ãƒ‰ãƒ¬ãƒ™ãƒ«ã®è©³ç´°ãªé•ã„ã«ã¤ã„ã¦ã¯ï¼Œæ¬¡å›žä»¥é™ã§è¦‹ã¦ã„ãã¾ã™ï¼Ž RCFile ã®å¾©ç¿’ RCFile ã¯ã€€Record Columnar File ã®ç•¥ã§ï¼ŒHive ã‹ã‚‰åˆ©ç”¨ã§ãã‚‹ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ãƒ•ã‚©ãƒ¼ãƒžãƒƒãƒˆã§ã™ï¼Žç‰¹ã«ï¼ŒHDFS ã‚„ S3 ã¨ã„ã£ãŸåˆ†æ•£ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ä¸Šã§ãƒ‘ãƒ•ã‚©ãƒ¼ãƒžãƒ³ã‚¹ãŒã§ã‚‹ã‚ˆã†ã«è¨è¨ˆã•ã‚Œã¦ã„ã¾ã™ï¼Ž HDFS/S3 ã¨ã„ã£ãŸã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ã§ã¯ï¼ŒåŸºæœ¬çš„ã«ãƒ‡ãƒ¼ã‚¿ã‚’è¨ˆç®—æ©Ÿé–“ã§åŒã˜è² è·ã«ãªã‚‹ã‚ˆã†ã«ãƒ‡ãƒ¼ã‚¿ã‚’åˆ†æ•£é…ç½®ã—ã¾ã™ï¼Žã“ã®ãŸã‚ï¼Œå¾“æ¥ã®åˆ—æŒ‡å‘ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ãƒ•ã‚©ãƒ¼ãƒžãƒƒãƒˆã®ã‚ˆã†ã«é©å½“ã«åˆ—æ¯Žã«
muddydixon 2013/03/19
hadoop

hdfs

rcfile

parquet

orcfile
ãƒªãƒ³ã‚¯
Fluentdï¼‹WebHDFSï¼†DataNodeåŠæ»ã§èµ·ããŸå•é¡Œ | å¤–é“çˆ¶ã®åŒ
Fluentd Collectorã‹ã‚‰HDFSã«æ›¸ãè¾¼ã‚€ã®ã« fluent-plugin-webhdfs ã‚’åˆ©ç”¨ã—ã¦ã„ã¾ã™ãŒã€ DataNodeãŒï¼‘å°å¤‰æ»ã—ãŸéš›ã«è‰²ã€…ãŠã‹ã—ããªã£ãŸã®ã§æ›¸ãç•™ã‚ã¦ãŠãã¾ã™ã€‚ åŽŸå› ç‰¹å®šã¨è§£æ±ºæ–¹æ³•ã®ç¢ºç«‹ã¯ã§ãã¦ã„ã¾ã›ã‚“ï¼ã‚ã—ã‹ã‚‰ãšã€‚ ç›´æŽ¥ã®åŽŸå› ã¯SLAVEã‚µãƒ¼ãƒï¼ˆDataNodeï¼‰ãŒä¸é€”åŠç«¯ã«è½ã¡ãŸã“ã¨ ï¼‘å°ã®SLAVEã‚µãƒ¼ãƒã«ç•°å¸¸ãŒç™ºç”Ÿã—ãŸã“ã¨ãŒç›´æŽ¥ã®åŽŸå› ã§ã‚ã‚Šã€çŠ¶æ…‹ã¨ã—ã¦ã¯ SLAVEã‚µãƒ¼ãƒãŒKernel Panic!! ãƒ›ã‚¹ãƒˆã¸ã®Pingã¯é€šã‚‹ å„ç¨®ãƒ‡ãƒ¼ãƒ¢ãƒ³ã¸ã®TCPæŽ¥ç¶šã¯ç¢ºç«‹ã§ãã‚‹ å„ç¨®ãƒ‡ãƒ¼ãƒ¢ãƒ³ã¯ä¸€åˆ‡ãŠè¿”äº‹ã‚’ã—ã¦ãã‚Œãªã„ è©¦ã—ãŸã®ãŒDataNodeã§ãªã„ã®ãŒå¿ƒè‹¦ã—ã„ã§ã™ãŒã€å¾©æ—§å‰ã«ç¢ºèªã§ããŸã®ã¯SSHæŽ¥ç¶šã§ã€ ssh -p22 host ã¯ç„¡å¿œç”ã§ã€telnet host 22 ã¯ãƒªã‚¯ã‚¨ã‚¹ãƒˆå¾…ã¡çŠ¶æ…‹ã«ãªã‚‹åŠæ»çŠ¶æ…‹ã§ã—ãŸã€‚ ã“ã®çŠ¶æ…‹ãŒã€Fluentdã¾ãŸã¯
muddydixon 2013/03/07
fluentd

hdfs

hadoop

troubleshooting
ãƒªãƒ³ã‚¯
Fluentd and WebHDFS
This document discusses Fluentd and its webhdfs output plugin. It explains how the webhdfs plugin was created in 30 minutes by leveraging existing Ruby gems for WebHDFS operations and output formatting. The document concludes that output plugins can reuse code from mixins and that developing shared mixins allows plugins to incorporate common features more easily.
muddydixon 2012/11/08
fluentd

presentation

hadoop

hdfs
ãƒªãƒ³ã‚¯
Fluentdã®æ‰€æ„Ÿ ãã®ï¼‘ | å¤–é“çˆ¶ã®åŒ
Agent ãƒã‚°ã®é‡ã‚„Fluentdï¼†CPUã®æ€§èƒ½ã‚’è€ƒãˆã‚‹ã¨ã€è² è·çš„ã«ã¯ï¼‘ã‚µãƒ¼ãƒï¼‘Agentã§ååˆ†è¶³ã‚Šã‚‹ã®ã§ã€ã‚¹ãƒ†ãƒ¼ã‚¿ã‚¹æ¤œçŸ¥ãªã©ã®ç›£è¦–ã ã‘ã—ã£ã‹ã‚Šã—ã¦ãŠã‘ã°OKã¨è€ƒãˆã¾ã™ã€‚ãªã®ã§ä¾‹ãˆã°WEBã‚µãƒ¼ãƒã«æ™®é€šã«ï¼‘Agentå…¥ã‚Œã¦ãã‚ŒãŒæ•°ç™¾ãƒ»æ•°åƒå°ã«ãªã‚‹ã“ã¨ã‚’æƒ³å®šã—ã¾ã™ã€‚ Collector è¤‡æ•°å°ç”¨æ„ã—ã€Agentã‹ã‚‰roundrobinã§é€ä¿¡ã™ã‚‹ã“ã¨ã§å‡ä¸€åŒ–ã—ã¾ã™ã€‚Collectorãƒ€ã‚¦ãƒ³æ™‚ã‚„å¾©æ—§æ™‚ã¯ã€ãƒã‚°ã®ãƒã‚¹ãƒˆç„¡ã—ã«ã™ã¿ã‚„ã‹ã«roundrobinã‹ã‚‰å¤–ã‚ŒãŸã‚Šå¾©æ´»ã™ã‚‹ã“ã¨ã‚’ç¢ºèªæ¸ˆã¿ã§ã™ã€‚å°æ•°ãŒå¢—ãˆã™ãŽãŸæ™‚ã®æ‡¸å¿µç‚¹ã¯ã€HDFSã«å¯¾ã™ã‚‹ï¼‘ãƒ•ã‚¡ã‚¤ãƒ«ã¸ã®APPENDæ•°ãŒå¢—ãˆã‚‹ã“ã¨ã§ã™ãŒã€ã“ã“ã¾ã§ã®è©¦é¨“ã‚’è¦‹ã‚‹é™ã‚Šã¯ãŠãã‚‰ãã‹ãªã‚Šã®æ•°ã¾ã§å¤§ä¸ˆå¤«ã§ã™ã—ã€ä»®ã«HDFSã¸ã®æ›¸ãè¾¼ã¿ãŒå•é¡Œã«ãªã‚‹å ´åˆã¯Agent -> Collectorã®é¸æŠžæ¡ä»¶ã‚„ã€æ›¸ãè¾¼ã¿ãƒ•ã‚¡ã‚¤ãƒ«ãƒ‘ã‚¹ã§å·¥å¤«ã™ã‚Œã°å¤§ä¸ˆå¤«ã§ã™ã€‚ ã¨ã¯ã„ãˆã€APP
muddydixon 2012/09/28
fluentd

hdfs

hadoop

log
ãƒªãƒ³ã‚¯
Faunus
Faunus : Graph Computing with HadoopFaunus is a Hadoop-based graph processing framework for analyzing massive-scale graphs. A breadth-first version of the graph traversal language Gremlin operates on graphs stored in the distributed graph database Titan, in any Rexster-fronted graph database, or in HDFS via various text and binary formats. In addition, Faunus provides the following features: Scala
muddydixon 2012/09/23
graph

hadoop

hdfs

analytics

mining
ãƒªãƒ³ã‚¯
å‰¯æ¥Cloudï½œè‡ªç”±ãªãƒ“ã‚¸ãƒã‚¹ã‚’æ±‚ã‚ã¦
å‰¯æ¥å¸‚å ´ã®æ€¥æˆé•·ã«ä¼´ã„ã€å¤šãã®äººã€…ãŒæ–°ãŸãªåŽç›Šæºã‚’æ±‚ã‚ã¦å‰¯æ¥ã‚„ãƒãƒƒãƒˆãƒ“ã‚¸ãƒã‚¹ã«å‚å…¥ã—ã¦ã„ã¾ã™ã€‚ã—ã‹ã—ã€ãã®ä¸ã«ã¯ä¸é€æ˜Žãªæƒ…å ±ã‚„èª¤è§£ã‚’æ‹›ãå®£ä¼ã€ãƒªã‚¹ã‚¯ã®é«˜ã„æ¡ˆä»¶ã‚‚å°‘ãªãã‚ã‚Šã¾ã›ã‚“ã€‚å½“ãƒ–ãƒã‚°ã§ã¯ã€ã“ã†ã—ãŸç’°å¢ƒã®ä¸ã§æ£ã—ã„é¸æŠžã‚’è¡Œã†ãŸã‚ã®çŸ¥è˜ã¨æ¤œè¨¼ã‚’æä¾›ã™ã‚‹ã“ã¨ã‚’ç›®çš„ã¨ã—ã¦ã„ã¾ã™ã€‚ ãƒãƒƒãƒˆä¸Šã®å‰¯æ¥ã‚’è©³ç´°ã«æ¤œè¨¼ã—ã€ä¿¡é ¼ã§ãã‚‹æƒ…å ±ã‚’æ•´ç†ãƒ»æä¾›ã™ã‚‹ã“ã¨ã§ã€ã¿ãªã•ã¾ãŒå®‰å…¨ã‹ã¤åŠ¹æžœçš„ã«å‰¯æ¥ã«å–ã‚Šçµ„ã‚ã‚‹ã‚ˆã†æ”¯æ´ã—ã¾ã™ã€‚
muddydixon 2012/09/19
hdfs

name
ãƒªãƒ³ã‚¯
Fluentdã®æ‰€æ„Ÿ ãã®ï¼’ | å¤–é“çˆ¶ã®åŒ
YARNã«ã¤ã„ã¦æ›¸ããŸã„æ°—æŒã¡ã‚’æŠ‘ãˆã¦å‰å›žã®ç¶šãã§ã™ã€‚ ã¾ã fluentdã‚’ãŒã£ã¤ã‚Šå°Žå…¥ã—ã¦ã„ã‚‹ã‚ã‘ã˜ã‚ƒãªãã€ç‹¬è‡ªãƒ—ãƒ©ã‚°ã‚¤ãƒ³ã‚„æ§‹æˆã«ã¤ã„ã¦ã¯æ”¹è‰¯ã®ä½™åœ°ãŒã‚ã‚‹ã‹ã‚‚ãªã®ã§ã€ã¾ãå‚è€ƒç¨‹åº¦ã«è¦‹ã¦ãŠã„ã¦ãã ã•ã„ã€‚ ãƒ‡ãƒ¼ã‚¿ãƒ•ãƒãƒ¼ APã‚µãƒ¼ãƒã®ãƒã‚°æ›¸ãè¾¼ã¿ã‹ã‚‰ã€HDFSã«æ›¸ãè¾¼ã‚€ã¾ã§ã®å‡¦ç†å†…å®¹ã§ã™ã€‚ãƒ‘ã‚¹ã¨ã‹ã¯é©å½“ã«æ›¸ãæ›ãˆã¦ã¾ã™ã€‚ãªãœã“ã‚“ãªãƒ•ãƒãƒ¼ãªã®ã‹ã¯å¾Œè¿°ã€‚ã‚ã‚“ã©ã„ã®ã§ãƒ†ã‚ã‚¹ãƒˆã§ã€‚ (WEB/AP) â—†ãƒã‚°ã‚’å›ºå®šãƒ‘ã‚¹ã«è¿½è¨˜ (ex: date >> /tmp/service_plathome_logtype.log ã®ã‚ˆã†ãª) â”œâ”€â–¡cronã«ã‚ˆã‚‹å¼·åˆ¶logrotate (10åˆ†ï½ž1æ™‚é–“ç¨‹åº¦ãŠã) â”‚ (Fluentd Agent) â—‡IN æ”¹é€ tailãƒ—ãƒ©ã‚°ã‚¤ãƒ³ã§ãƒã‚°å–å¾— (pathã«*ã®ä½¿ç”¨å¯ã€basenameã‚’tagã«ã§ãã‚‹) â”‚ â—†OUT copy â”œâ”€â—†flowcounter (unit h
muddydixon 2012/08/30
fluentd

hdfs
ãƒªãƒ³ã‚¯
cloudera/hoop - GitHub
----------------------------------------------------------------------------- Hoop, Hadoop HDFS over HTTP Hoop is a server that provides a REST HTTP gateway to HDFS with full read & write capabilities. Hoop is distributed under Apache License 2.0. Hoop can be used to transfer data between clusters running different versions of Hadoop (overcoming RPC versioning issues), for example using Hadoop Dis
muddydixon 2012/04/25
hadoop

hoop

hdfs
ãƒªãƒ³ã‚¯
Hoop (HDFS over HTTP) ã‚’è©¦ã—ã¦ã¿ãŸ - ãŸã”ã‚‚ã‚Šã™ãƒ¡ãƒ¢
ãƒãƒƒãƒå‡¦ç†ã«ã‹ã‘ã‚‹å¯¾è±¡ãƒ•ã‚¡ã‚¤ãƒ«ã®æŠ½å‡ºã«ã¯ä¸èº«ã‚’è¦‹ã‚‹å¿…è¦ãŒã‚ã£ã¦ã€headã‚„tailã§ã„ã„ã‚“ã ã‘ã©hadoop fsã‚³ãƒžãƒ³ãƒ‰ã§ã¯ãã†ã„ã†ã‚ªãƒšãƒ¬ãƒ¼ã‚·ãƒ§ãƒ³ãŒã§ããªã„*1ã®ã§ FUSE hdfs ã§Linuxã«HDFSã‚’mountã—ã¦ã‚„ã£ã¦ã„ã‚‹ã€‚ ãŒã€ã“ã‚ŒãŒã¾ãŸãƒ“ãƒ«ãƒ‰ã™ã‚‹ã®ã«ã²ã¨è‹¦åŠ´ã ã£ãŸã‚Šmountå…ƒã¨å…ˆãŒå¯†çµåˆã«ãªã£ã¡ã‚ƒã£ãŸã‚Šé…ã‹ã£ãŸã‚Šã„ã¤ã¾ã§ãƒ¡ãƒ³ãƒ†ã•ã‚Œã‚‹ã‚‚ã‚“ã‹ã¨æ€ã‚ã‚ŒãŸã‚Šã™ã‚‹ã—å¤–éƒ¨ã‚³ãƒžãƒ³ãƒ‰ã‚’èµ·å‹•ã—ã¦çµæžœã‚’èªã‚“ã§ã‚´ãƒ‹ãƒ§ã‚´ãƒ‹ãƒ§ã—ãŸã‚Šã™ã‚‹ã®ãŒé¢å€’ãªã®ã§ã©ã†ã«ã‹ã—ãŸã„ãªãƒ¼ã¨æ€ã£ã¦ã„ãŸã‚‰ã€Clouderaã‹ã‚‰ Hoop ãªã‚‹ãƒ—ãƒãƒ€ã‚¯ãƒˆãŒå°‘ã—å‰ã«ãƒªãƒªãƒ¼ã‚¹ã•ã‚ŒãŸã€‚ Hoop - Hoop, Hadoop HDFS over HTTP - Documentation Sets 0.1.0-SNAPSHOT HTTP REST APIã‚’çµŒç”±ã—ã¦HDFSã®å†…å®¹ã«ã‚¢ã‚¯ã‚»ã‚¹ã§ãã‚‹ã—ã€ã‚ªãƒšãƒ¬ãƒ¼ã‚·ãƒ§ãƒ³ã‚‚ã§ãã‚‹ã€‚GETã®
muddydixon 2012/04/25
hoop

hadoop

hdfs
ãƒªãƒ³ã‚¯
agilecatcloud.com
muddydixon 2010/06/22
hdfs

hadoop
ãƒªãƒ³ã‚¯
1