[B! MapReduce] yuroyoroã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

yuroyoro id:yuroyoro

MapReduceã«é–¢ã™ã‚‹yuroyoroã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (5)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

Hadoopã‚’ä½¿ã„ã“ãªã™(1)
ã¾ãšã€ 1 ã®å…¥åŠ›ãƒ•ã‚¡ã‚¤ãƒ«ã‚’åˆ†å‰²ã™ã‚‹æ–¹æ³•ã¯ã€InputFormatã‚¯ãƒ©ã‚¹ã®ã€getSplitsé–¢æ•°ã‚’ä¸Šæ›¸ãã™ã‚‹ã“ã¨ã§ã€ã‚«ã‚¹ã‚¿ãƒžã‚¤ã‚ºã§ãã¾ã™ã€‚ ã¾ãŸã€ 3 ã®InputSplitã‹ã‚‰ã€Keyã¨Valueã‚’æŠ½å‡ºã™ã‚‹å‡¦ç†ã‚‚ã€InputFormatã‚¯ãƒ©ã‚¹ã‚’é€šã˜ã¦ã‚«ã‚¹ã‚¿ãƒžã‚¤ã‚ºã§ãã¾ã™ã€‚ InputFormatã®getRecordReaderé–¢æ•°ã‚’é€šã˜ã¦ã€RecordReaderã‚¯ãƒ©ã‚¹ã‚’ç”Ÿæˆã™ã‚‹ã®ã§ã™ãŒã€ã“ã‚Œã«ä»»æ„ã®RecordReaderã‚¯ãƒ©ã‚¹ã‚’æŒ‡å®šã™ã‚Œã°OKã§ã™ã€‚ 2 ã®Mapå‡¦ç†ã§ã™ãŒã€ãƒ¦ãƒ¼ã‚¶ãŒæŒ‡å®šã—ãŸMapperã‚¯ãƒ©ã‚¹ã®å‡¦ç†ã‚’å®Ÿè¡Œã—ã¾ã™ã€‚ Mapperã‚¯ãƒ©ã‚¹ã¯ã€MapRunnerã‚¯ãƒ©ã‚¹ã‚’é€šã˜ã¦ã€åˆæœŸåŒ–å‡¦ç†ã€mapé–¢æ•°ã‚’ç¹°ã‚Šè¿”ã™éŽç¨‹ã€çµ‚äº†å‡¦ç†ã¨ã„ã£ãŸä¸€é€£ã®æµã‚Œã‚’å®Ÿè¡Œã—ã¾ã™ã€‚ MapRunnerã‚¯ãƒ©ã‚¹ã‚’ã‚«ã‚¹ã‚¿ãƒžã‚¤ã‚ºã™ã‚Œã°ã€ã“ã†ã—ãŸæµã‚Œã‚’åˆ¶å¾¡ã™ã‚‹ã“ã¨ãŒã§ãã¾ã™ã€‚ 0.20.0ã‹ã‚‰ã®æ–°ã—ã„MapRed
yuroyoro 2010/02/04
MapReduce

hadoop
ãƒªãƒ³ã‚¯
ã‚¯ãƒƒã‚¯ãƒ‘ãƒƒãƒ‰ã®åˆ†æ•£å‡¦ç†LTã§GridGainã‚’ãƒ‡ãƒ¢ã—ã¾ã—ãŸ - kaisehã®ãƒ–ãƒã‚°
20æ—¥ã«ã€ã‚¯ãƒƒã‚¯ãƒ‘ãƒƒãƒ‰ã§é–‹å‚¬ã•ã‚ŒãŸã€Œtechlifeãƒ©ã‚¤ãƒˆãƒ‹ãƒ³ã‚°ãƒˆãƒ¼ã‚¯ã€ã«å‚åŠ ã—ã¦ãã¾ã—ãŸã€‚ ä»Šå›žã¯ã€Œåˆ†æ•£å‡¦ç†ã€ã¨ã„ã†ãƒ†ãƒ¼ãƒžã§LTãŒè¡Œã‚ã‚Œã¾ã—ãŸã€‚åƒ•ã¯ã€Javaã®ã‚°ãƒªãƒƒãƒ‰ã‚³ãƒ³ãƒ”ãƒ¥ãƒ¼ãƒ†ã‚£ãƒ³ã‚°ãƒ•ãƒ¬ãƒ¼ãƒ ãƒ¯ãƒ¼ã‚¯GridGainã‚’Scalaã‹ã‚‰ä½¿ã†ãƒ‡ãƒ¢ã‚’ã—ã¾ã—ãŸã€‚ ï¼ˆè¿½è¨˜: å‚åŠ è€…å…¨å“¡ã®ç™ºè¡¨å†…å®¹ãŒYouTubeã§å…¬é–‹ã•ã‚Œã¦ã„ã¾ã™ï¼‰ GridGainã¯ã€Hadoopã¨åŒæ§˜ã«MapReduceã®æ¦‚å¿µã«åŸºã¥ããƒ•ãƒ¬ãƒ¼ãƒ ãƒ¯ãƒ¼ã‚¯ã§ã™ãŒã€ã‚ãã¾ã§ã‚‚è¨ˆç®—å‡¦ç†ã®åˆ†æ•£ã«ãƒ•ã‚©ãƒ¼ã‚«ã‚¹ã—ã¦ã„ã‚‹ã‚ˆã†ã§ã™ã€‚ã‚ˆã£ã¦ã€GridGainã«ã¯HDFSç›¸å½“ã®ãƒ•ã‚¡ã‚¤ãƒ«ã‚·ã‚¹ãƒ†ãƒ ãŒã‚ã‚Šã¾ã›ã‚“ãŒã€ãã®åˆ†ã‚·ãƒ³ãƒ—ãƒ«ã§ã€éžå¸¸ã«ç°¡å˜ã«ä½¿ç”¨ã§ãã¾ã™ã€‚ ä»¥ä¸‹ã®ã‚ˆã†ã«ã€åˆ†æ•£ã•ã›ãŸã„ãƒ¡ã‚½ãƒƒãƒ‰ã«@Gridifyã¨ã„ã†ã‚¢ãƒŽãƒ†ãƒ¼ã‚·ãƒ§ãƒ³ã‚’ä»˜ã‘ã¦ãŠãã€-javaagent VMã‚ªãƒ—ã‚·ãƒ§ãƒ³ã§AspectJã®weaverç‰ã‚’æŒ‡å®šã—ã¦å®Ÿè¡Œã™ã‚‹ã¨ã€ãƒ¡ã‚½ãƒƒãƒ‰ã‚’ã‚¨ãƒ³ãƒãƒ³ã‚¹ã—ã¦ã‚°ãƒªãƒƒãƒ‰
yuroyoro 2009/11/26
Shadoopã¨ã„ã„ã€æœ€è¿‘ã¯åˆ†æ•£å‡¦ç†ã§Scalaã‚’ä½¿ã†ã®ãŒã‚¢ãƒ„ã„ã®ã‹ãª?

MapReduce

hadoop

scala
ãƒªãƒ³ã‚¯
ã‚¯ãƒƒã‚¯ãƒ‘ãƒƒãƒ‰ã¨Hadoop - ã‚¯ãƒƒã‚¯ãƒ‘ãƒƒãƒ‰é–‹ç™ºè€…ãƒ–ãƒã‚°
ã¯ã˜ã‚ã¾ã—ã¦ã€‚ä»Šå¹´ã®5æœˆã«å…¥ç¤¾ã—ãŸå‹é–“@ã•ãŒã™ãƒãƒ¼ãƒ ã§ã™ã€‚ å…¥ç¤¾ã—ã¦ã‹ã‚‰ã¯ã€ãªã‹ãªã‹å¤§å¤‰ãªã“ã¨ã‚‚å¤šã„ã§ã™ãŒã€æœ€è¿‘ã¯ãŠé…’å¥½ããŒé›†ã¾ã£ã¦æœˆæ›œã‹ã‚‰é£²ã¿åˆã† ã€Œå‹é–“ä¼šã€ãªã‚‹ã‚‚ã®ã‚‚ç™ºè¶³ã—ã¦ã€ä»•äº‹é¢ã§ã‚‚ä»•äº‹ä»¥å¤–ã®é¢ã§ã‚‚å¯†åº¦ã®é«˜ã„æ¯Žæ—¥ã‚’éŽã”ã—ã¦ã„ã¾ã™ï¼ ã•ã¦ã€åƒ•ã¯ã€Œã•ãŒã™ã€ãƒãƒ¼ãƒ æ‰€å±žã¨ã„ã†ã“ã¨ã§ã€æ™®æ®µã¯ãƒ¬ã‚·ãƒ”ã‚’ã€Œã•ãŒã™ã€ãƒ¦ãƒ¼ã‚¶ã®æº€è¶³åº¦ã‚’ä¸Šã’ã‚‹ãŸã‚ã«ã€ ã‚¯ãƒƒã‚¯ãƒ‘ãƒƒãƒ‰ã®æ¤œç´¢ã¾ã‚ã‚Šã«ã¤ã„ã¦ã€ã„ã‚ã„ã‚ãªé–‹ç™ºã‚’è¡Œã£ã¦ã„ã¾ã™ã€‚ ä¸€æ–¹ã§ã€ãƒ¦ãƒ¼ã‚¶ã®ã€Œã•ãŒã™æ¬²æ±‚ã€ã«ã¤ã„ã¦æ·±ãçŸ¥ã‚‹ãŸã‚ã«ã€å¤§è¦æ¨¡ãªãƒ‡ãƒ¼ã‚¿è§£æžã‚’è¡Œã„ã€æ¬²æ±‚ã®åˆ†æžã‚’è¡Œã†æ©Ÿä¼šã‚‚å¢—ãˆã¦ãã¾ã—ãŸã€‚ ã¨ã“ã‚ãŒã€ã‚¯ãƒƒã‚¯ãƒ‘ãƒƒãƒ‰ã®ãƒã‚°ã¯è†¨å¤§ãªæ•°ãŒã‚ã‚‹ã®ã§ã€ä¸€å£ã®ãƒ‡ãƒ¼ã‚¿è§£æžã¨è¨€ã£ã¦ã‚‚é€šå¸¸ã®ãƒãƒƒãƒå‡¦ç†ã ã¨é–“ã«åˆã‚ãªã„ãŸã‚ã€ åˆ†æ•£å‡¦ç†ç’°å¢ƒã®å¿…è¦æ€§ãŒé«˜ã¾ã£ã¦ãã¾ã—ãŸã€‚ ãã“ã§ã€ã¾ãšã¯æ‰‹è»½ã«è©¦ã›ã‚‹åˆ†æ•£å‡¦ç†ã®çŽ‹é“ã¨ã„ã†ã“ã¨ã§ã€æœ€è¿‘ã§ã¯Hadoopã‚’ä½¿ã£ãŸãƒ‡ãƒ¼ã‚¿è§£æžç’°å¢ƒã‚’æ•´å‚™ã—ã¦ã„ã¾ã™ã€‚
yuroyoro 2009/09/16
hadoop

MapReduce
ãƒªãƒ³ã‚¯
ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã‚ãƒ¥ãƒ¼ã‚’ä½¿ã£ã¦åˆ†æ•£MapReduceã‚’å®Ÿè£…ã™ã‚‹ 2009-02-16 - ãã—ã ã®ã¯ã¦ãª
ã•ã¦ã€JMSã§ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã‚ãƒ¥ãƒ¼ã‚‚ä½¿ãˆã‚‹ã‚ˆã†ã«ãªã£ã¦ã€Hadoopã§MapReduceã‚‚è©¦ã—ã¦ã¿ãŸã€‚ãã†ã™ã‚‹ã¨ã‚„ã‚ŠãŸããªã‚‹ã®ãŒãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã‚ãƒ¥ãƒ¼ã‚’ä½¿ã£ãŸåˆ†æ•£MapReduceã®å®Ÿè£…ã§ã™ãã€‚ã¨ã„ã†ã“ã¨ã§ã€JMSã‚’ä½¿ã£ã¦ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã‚ãƒ¥ãƒ¼ã«ã‚ˆã‚‹åˆ†æ•£MapReduceã‚’ã‚„ã£ã¦ã¿ã¾ã—ãŸã€‚å®Ÿéš›ã«ã¯ãƒãƒ¼ã‚«ãƒ«ã§ã—ã‹å‹•ã‹ãªã„ã®ã§ã™ãŒã€ã‚³ãƒ³ã‚»ãƒ—ãƒˆãƒ¢ãƒ‡ãƒ«ã¨ã„ã†ã“ã¨ã§ã€‚ ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã‚ãƒ¥ãƒ¼ã§éŠã³ãŸã„ã®ã§JMSã‚’è©¦ã™ Hadoopã§ã®MapReduceã‚’æ°—è»½ã«è©¦ã™ã‚µãƒ³ãƒ—ãƒ« Hadoopã‚µãƒ³ãƒ—ãƒ«ã§ä½œã£ãŸã®ã¨åŒã˜ã‚ˆã†ã«ã€ã‚¯ãƒ©ã‚¹ãŒJavaãƒ•ã‚¡ã‚¤ãƒ«ä¸ã§importã•ã‚Œã¦ã„ã‚‹å›žæ•°ã‚’æ•°ãˆã¦ã¿ã¾ã™ã€‚ è€ƒãˆæ–¹ã¨ã—ã¦ã€ã¡ã‚‡ã£ã¨å¼·å¼•ã§ã™ãŒã€Googleã‚„Hadoopã®MapReduceã¯åˆ†æ•£ãƒ•ã‚¡ã‚¤ãƒ«ã‚·ã‚¹ãƒ†ãƒ ä»˜ããƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã‚ãƒ¥ãƒ¼ã¨ã„ãˆã¾ã™ã€‚ã‘ã©å°è¦æ¨¡ã«ã¤ã¤ã¾ã—ãã‚„ã‚‹åˆ†ã«ã¯åˆ†æ•£ãƒ•ã‚¡ã‚¤ãƒ«ã‚·ã‚¹ãƒ†ãƒ ã¯å¿…è¦ãªã„ã®ã§ã€Mapã¨Reduceã‚’åˆ†æ•£ã™ã‚‹ã“ã¨ã
yuroyoro 2009/02/18
hadoop

MapReduce

java
ãƒªãƒ³ã‚¯
Googleã®MapReduceã¯åƒ•ãŸã¡ã«å¿…è¦ã‹ï¼Ÿ - ãã—ã ã®ã¯ã¦ãª
ã¨ã„ã†ã“ã¨ã§ã€Google MapReduceã®å®Ÿè£…ã§ã‚ã‚‹Hadoopã‚’ä½¿ã£ãŸMapReduceã¨ã€JMSã‚’ä½¿ã£ãŸMapReduceã‚’ã‚„ã£ã¦ã¿ã¾ã—ãŸã€‚ ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã‚ãƒ¥ãƒ¼ã‚’ä½¿ã£ã¦åˆ†æ•£MapReduceã‚’å®Ÿè£…ã™ã‚‹ Hadoopã§ã®MapReduceã‚’æ°—è»½ã«è©¦ã™ã‚µãƒ³ãƒ—ãƒ« ã“ã‚Œä½•ã®ãŸã‚ã«ã‚„ã£ãŸã‹ã¨ã„ã†ã¨ã€ãã“ã‚‰ã«ã‚ã‚‹ã‚ˆã†ãªæ•°åå°è¦æ¨¡ã®ã‚µãƒ¼ãƒãƒ¼ã‚’å‰æã¨ã—ãŸã¨ãã«ã€Hadoopã®æœ‰åŠ¹æ€§ã€ã²ã„ã¦ã¯ãã®å…ƒã«ãªã£ã¦ã‚‹Google MapReduceã®æœ‰åŠ¹æ€§ã«ã¤ã„ã¦ç–‘å•ã«æ€ã£ãŸã‹ã‚‰ã§ã™ã€‚ãã“ã§ã€ã¡ã‚‡ã£ã¨è©¦ã—ã¦ã¿ãŸã€ã¨ã€‚ ã“ã“ã§ã€ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã‚ãƒ¥ãƒ¼ã‚’ä½¿ã£ãŸå ´åˆã«1ç§’ã§ã§ãã¦ãŸå‡¦ç†ãŒã€Hadoopã‚’ä½¿ã†ã¨ã‚¹ã‚¿ãƒ³ãƒ‰ã‚¢ãƒãƒ³ãƒ¢ãƒ¼ãƒ‰ã§ã‚‚40ç§’è¿‘ãã‹ã‹ã‚Šã¾ã—ãŸã€‚æ“¬ä¼¼åˆ†æ•£ãƒ¢ãƒ¼ãƒ‰ã ã¨4åˆ†è¿‘ãã§ã™ã€‚ ã„ãã‚‰Hadoopã®å®Ÿè£…ãŒã²ã©ã„ã¨ã—ã¦ã‚‚ã€ã“ã‚Œã¯ã‚ã‚“ã¾ã‚Šã§ã™ã€‚ Googleã§ã®å®Ÿè£…ã¯ã‚‚ã£ã¨åŠ¹çŽ‡çš„ãªã‚‚ã®ã«ãªã£ã¦ã„ã‚‹ã¨æ€ã„ã¾ã™ãŒã€ã
yuroyoro 2009/02/18
google

MapReduce

hadoop

èª•ç”Ÿæ—¥ãŠã‚ã§ã¨ã†
ãƒªãƒ³ã‚¯
1