[B! mapreduce] HISAMATSUã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

HISAMATSU id:HISAMATSU

mapreduceã«é–¢ã™ã‚‹HISAMATSUã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (7)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

æ¤œç´¢ã‚¯ã‚¨ãƒªãƒã‚°ã‹ã‚‰ã®ã‚¹ãƒšãƒ«è¨‚æ£è¾žæ›¸ã®è‡ªå‹•ç”Ÿæˆ - mixi engineer blog
å…ˆæœˆãƒãƒ¯ã‚¤ã«è¡Œã£ã¦ãã¦ã‚ªãƒ«ã‚ªãƒ«ãª ï¼ˆãƒãƒ¯ã‚¤èªžã§ 'æ¥½ã—ã„' ã¨ã„ã†æ„å‘³ï¼‰ æ°—åˆ†ã® takahi-i ã§ã™ã€‚æœ€è¿‘ãƒã‚°ãƒ‡ãƒ¼ã‚¿ã®æœ‰åŠ¹æ´»ç”¨ãŒè©±é¡Œã«ãªã£ã¦ã„ã¾ã™ãŒã€æ¤œç´¢ã‚¨ãƒ³ã‚¸ãƒ³ãŒå‡ºåŠ›ã™ã‚‹æ¤œç´¢ã‚¯ã‚¨ãƒªãƒã‚°ã‚’ä½¿ç”¨ã—ã¦ã©ã‚“ãªã“ã¨ãŒã§ãã‚‹ã®ã‹ã«ã¤ã„ã¦ç´¹ä»‹ã•ã›ã¦ã„ãŸã ãã¾ã™ã€‚ æ¤œç´¢ã‚¯ã‚¨ãƒªãƒã‚° æ¤œç´¢ã‚¯ã‚¨ãƒªãƒã‚° (ä»¥ä¸‹æ¤œç´¢ãƒã‚°) ã¯æ¤œç´¢ã‚¨ãƒ³ã‚¸ãƒ³ã‚’ä½¿ç”¨ã™ã‚‹ãƒ¦ãƒ¼ã‚¶ã‹ã‚‰ç™ºè¡Œã•ã‚ŒãŸæ¤œç´¢ã®å±¥æ´ã‚’ä¿å˜ã—ãŸãƒ•ã‚¡ã‚¤ãƒ«ã§ã™ã€‚æ¤œç´¢ãƒã‚°ã®ãƒ•ã‚©ãƒ¼ãƒžãƒƒãƒˆã¯ä½¿ç”¨ã™ã‚‹æ¤œç´¢ã‚¨ãƒ³ã‚¸ãƒ³ã‚„ Web ã‚µãƒ¼ãƒã«ã‚ˆã£ã¦ç•°ãªã‚Šã¾ã™ã€‚ã•ã‚‰ã«ã¾ãŸæ¤œç´¢ãƒã‚°ãŒå«ã‚€æƒ…å ±ã«ã‚‚å·®ç•°ãŒã‚ã‚‹ã“ã¨ãŒè€ƒãˆã‚‰ã‚Œã¾ã™ãŒã€æœ¬ç¨¿ã§ã¯æ¤œç´¢ãƒã‚°ã¯è§£æžã‚’è¡Œã†ä¸Šã§é‡è¦ãªä¸‰ã¤ã®è¦ç´ ã‚’å«ã‚€ã¨ä»®å®šã—ã¾ã™ã€‚ä¸‰ã¤ã®è¦ç´ ã¨ã¯ãƒ¦ãƒ¼ã‚¶ ID ï¼ˆã‚‚ã—ãã¯ IP ã‚¢ãƒ‰ãƒ¬ã‚¹ï¼‰ã€ã‚¯ã‚¨ãƒªæ–‡ã€ãã—ã¦ã‚¯ã‚¨ãƒªãŒæ¤œç´¢ã‚¨ãƒ³ã‚¸ãƒ³ã«å‡¦ç†ã•ã‚ŒãŸæ™‚é–“ã§ã™ã€‚ä»¥ä¸‹æ¤œç´¢ãƒã‚°ã®ä¸€ä¾‹ã‚’è¼‰ã›ã¾ã™ã€‚ ãƒ¦ãƒ¼ã‚¶ID ã‚¯ã‚¨ãƒªæ–‡ ã‚¯ã‚¨ãƒªç™ºè¡Œæ™‚ 438904 Su
HISAMATSU 2009/12/15
ã€ŒHadoop ã‚’è¨ˆç®—æ©Ÿ å››å°ã«ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã—ã¦ä½¿ç”¨ã—ãŸéš›ã€ä¸‰ã‚®ã‚¬ãƒã‚¤ãƒˆã®æ¤œç´¢ãƒã‚°ã‚’å‡¦ç†ã™ã‚‹ã®ã«ã‹ã‹ã£ãŸæ™‚é–“ã¯äºŒåˆ†ç¨‹åº¦ã§ã—ãŸã€‚ã€3GB ç¨‹åº¦ãƒ‡ãƒ¼ã‚¿ã‚’å¯¾è±¡ã«ã—ã¦ã‚‚ï¼Œååˆ†æ„å‘³ãŒã‚ã‚‹ã¿ãŸã„ï¼Ž

mapreduce
ãƒªãƒ³ã‚¯
å„ªè‰¯ä¼æ¥ã¯ãªãœHadoopã«èµ°ã‚‹ã®ã‹
ã¡ãªã¿ã«ã€ã“ã®åˆ†æžã®ãŸã‚ã«å¿…è¦ã¨ã•ã‚Œã‚‹MapReduceã®ã‚³ãƒ¼ãƒ‰ã§ã‚ã‚‹ãŒã€ãã®ã‚µã‚¤ã‚ºã¯ã‚ãšã‹20ã‚¹ãƒ†ãƒƒãƒ—ã ã¨ã„ã†ã€‚Yahoo!ã®ãƒ—ãƒ¬ã‚¼ãƒ³ãƒ†ãƒ¼ã‚¿ãƒ¼ã§ã‚ã‚‹ã€ã‚¨ãƒªãƒƒã‚¯ãƒ»ãƒãƒ«ãƒ‡ã‚·ãƒ¥ãƒã‚¤ãƒ©ãƒ¼æ°ã«ã‚ˆã‚‹ã¨ã€ãŸã¨ãˆçµŒé¨“ã®æµ…ã„ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ã§ã‚ã£ã¦ã‚‚ã€MapReduceã«ã‚ˆã‚‹ãƒ—ãƒã‚°ãƒ©ãƒŸãƒ³ã‚°ã¯å¯èƒ½ã§ã‚ã‚‹ã¨ã•ã‚Œã‚‹ã€‚ ã¾ãŸã€VISAã®ã‚¸ãƒ§ãƒ¼ãƒ»ã‚«ãƒ‹ãƒ³ã‚¬ãƒ æ°ã‹ã‚‰ã‚‚ã€è²´é‡ãªãƒ‡ãƒ¼ã‚¿ãŒæä¾›ã•ã‚Œã¦ã„ãŸã®ã§ä»¥ä¸‹ã«ç´¹ä»‹ã™ã‚‹ã€‚åŒç¤¾ã§ã¯ã€1æ—¥ã«1å„„ãƒˆãƒ©ãƒ³ã‚¶ã‚¯ã‚·ãƒ§ãƒ³ãŒç™ºç”Ÿã™ã‚‹ãŸã‚ã€2å¹´é–“ã§700å„„å¼·ã®ãƒˆãƒ©ãƒ³ã‚¶ã‚¯ã‚·ãƒ§ãƒ³ãƒã‚°ãŒè“„ç©ã•ã‚Œã€ãã®ãƒ‡ãƒ¼ã‚¿é‡ã¯36ãƒ†ãƒ©ãƒã‚¤ãƒˆã«è‡³ã‚‹ã¨ã„ã†ã€‚ã“ã†ã—ãŸã‚¹ã‚±ãƒ¼ãƒ«ã®ãƒ‡ãƒ¼ã‚¿ã‚’ã€å¾“æ¥ã®RDBã‚’ç”¨ã„ã¦åˆ†æžã™ã‚‹ã«ã¯ã€ç´„1ã‚«æœˆã®æ™‚é–“ãŒå¿…è¦ã¨ã•ã‚Œã¦ããŸãŒã€Hadoopã‚’ç”¨ã„ã‚‹ã“ã¨ã§13åˆ†ã«çŸç¸®ã•ã‚ŒãŸã¨ã„ã†ã€‚ ã“ã‚Œã¾ã§ã¯ã€Yahoo!ã«ã—ã‚VISAã«ã—ã‚ã€è†¨å¤§ãªãƒ‡ãƒ¼ã‚¿ã‚’RDBã«æŠ¼ã—è¾¼ã‚€ã»ã‹ã«æ–¹æ³•ã¯ãªãã€ãã®åˆ†æžã«æ•°åæ—¥ã‚’è¦ã™ã‚‹
HISAMATSU 2009/10/18
mapreduce

hadoop
ãƒªãƒ³ã‚¯
Rubyã§Hadoopã‚’ãƒ©ãƒƒãƒ—ã€åˆ†æ•£å‡¦ç†ãƒ„ãƒ¼ãƒ«ã‚ãƒƒãƒˆãŒç™»å ´ ï¼ ï¼ IT
2009/05/12 ç±³æ–°èžç¤¾å¤§æ‰‹ã®ãƒ‹ãƒ¥ãƒ¼ãƒ¨ãƒ¼ã‚¯ãƒ»ã‚¿ã‚¤ãƒ ã‚ºã¯5æœˆ11æ—¥ã€Rubyã«ã‚ˆã‚‹å¤§è¦æ¨¡åˆ†æ•£å‡¦ç†ã®ãƒ„ãƒ¼ãƒ«ã‚ãƒƒãƒˆã€ŒMap/Reduce Toolkitã€ï¼ˆMRToolkitï¼‰ã‚’GPLv3ã®ä¸‹ã«ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ã§å…¬é–‹ã—ãŸã¨ç™ºè¡¨ã—ãŸã€‚MRToolkitã¯ã€ã™ã§ã«ç¨¼åƒã—ã¦ã„ã‚‹ã‚¯ãƒ©ã‚¹ã‚¿ä¸Šã®Hadoopã¨åˆã‚ã›ã¦ä½¿ã†ã“ã¨ã§Rubyã§å®¹æ˜“ã«Map/Reduceå‡¦ç†ã‚’è¨˜è¿°ã™ã‚‹ã“ã¨ãŒã§ãã‚‹ä¸€ç¨®ã®ãƒ©ãƒƒãƒ‘ãƒ¼ã€‚å‡¦ç†è‡ªä½“ã¯HadoopãŒè¡Œã†ã€‚ã™ã§ã«Hadoopã‚’ä½¿ã£ã¦ã„ã‚‹ãƒ¦ãƒ¼ã‚¶ãƒ¼ã§ã‚ã‚Œã°ã€ä¸å°è¦æ¨¡ã®ãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆã«å¯¾ã—ã¦ã€ã™ãã«MRToolkitã‚’é©ç”¨å¯èƒ½ã¨ã—ã¦ã„ã‚‹ã€‚ ãƒ‡ãƒ•ã‚©ãƒ«ãƒˆã§æœ‰ç”¨ãªMapã€Reduceã®å‡¦ç†ãƒ¢ã‚¸ãƒ¥ãƒ¼ãƒ«ãŒå«ã¾ã‚Œã¦ã„ã¦ã€æ•°è¡Œã®Rubyã‚¹ã‚¯ãƒªãƒ—ãƒˆã‚’æ›¸ãã ã‘ã§ã€ä¾‹ãˆã°è†¨å¤§ãªApacheã®ãƒã‚°ã‹ã‚‰IPã‚¢ãƒ‰ãƒ¬ã‚¹åˆ¥ã®é–²è¦§å±¥æ´ã‚’ã¾ã¨ã‚ã‚‹ã¨ã„ã£ãŸå‡¦ç†ãŒå¯èƒ½ã¨ã„ã†ã€‚ç‹¬è‡ªã«Mapã‚„Reduceã®å‡¦ç†ã‚’å®šç¾©ã™ã‚‹ã“ã¨ã‚‚
HISAMATSU 2009/05/19
research

hadoop

mapreduce
ãƒªãƒ³ã‚¯
ã¸ã€œãŸã®ã‚ã‚‚:Hadoop ã¨ RDBMS ã®æ€§èƒ½ã‚’æ¯”è¼ƒã—ã¦ã¿ãŸã€‚ã¨ã„ã†è«–æ–‡ã€‚ã®æ„Ÿæƒ³ã€‚ã®ç¿»è¨³ã€‚ - livedoor Blogï¼ˆãƒ–ãƒã‚°ï¼‰
2009å¹´04æœˆ19æ—¥ Hadoop ã¨ RDBMS ã®æ€§èƒ½ã‚’æ¯”è¼ƒã—ã¦ã¿ãŸã€‚ã¨ã„ã†è«–æ–‡ã€‚ã®æ„Ÿæƒ³ã€‚ã®ç¿»è¨³ã€‚ Stonebraker, DeWitt, et al. compare MapReduce to DBMS Map-Reduce (Hadoop)ã¨ Parallel DBMS (Vertica ã¨ DBMS-X) ã®æ€§èƒ½ã‚’æ¯”è¼ƒã—ãŸã¨ã“ã‚ã€ãŠãŠã‚€ã DB ã®æ–¹ãŒæ€§èƒ½ãŒã‚ˆãã€ã‚±ãƒ¼ã‚¹ã«ã‚ˆã£ã¦ã¯ 6.5å€ã‚‚æ€§èƒ½å·®ãŒã‚ã‚‹ã¨ã„ã†å®Ÿé¨“çµæžœãŒã€SIGMOD09 ã§ç™ºè¡¨ã•ã‚ŒãŸã‚‰ã—ã„ã€‚åŽŸè«–æ–‡ã¯ã“ã“ã‹ã‚‰ PDF ã§ãƒ€ã‚¦ãƒ³ãƒãƒ¼ãƒ‰ã§ãã‚‹ã€‚ ä»¥ä¸‹ã¯ DBMS2 ã¨ã„ã†ãƒ–ãƒã‚°ã«æ›¸ã‹ã‚ŒãŸã€ã“ã®è«–æ–‡ã‚’èªã‚“ã äººã®æ„Ÿæƒ³ã®ç¿»è¨³ã€‚ Stonebraker, DeWitt, et al. MapReduce ã¨ DBMS ã®æ¯”è¼ƒ 5äººã®å…±è‘—è€…(ä¸»ç†è€…ã¯ Andy Pavlo ã£ã½ã„)ã¨ã¨ã‚‚ã«ã€Map-Reduce å«Œã„ã§çŸ¥ã‚‰ã‚Œ
HISAMATSU 2009/04/22
ã“ã®è«–æ–‡ã¯ã‹ãªã‚Šå‚è€ƒã«ãªã‚Šãã†ï¼Ž

mapreduce

research

database
ãƒªãƒ³ã‚¯
Amazon Elastic MapReduceã‚’ä½¿ã£ã¦ã¿ãŸ - moratorium
Amazon Elastic MapReduceã‚’ä½¿ã£ã¦ã¿ãŸ 2009-04-03 (Fri) 3:06 Amazon EC2 é€£æ—¥ã®EC2ãƒã‚¿ã§ã™ã€‚æœ¬æ—¥ã€Amazonã‹ã‚‰Elastic MapReduceã¨ã„ã†ã‚µãƒ¼ãƒ“ã‚¹ãŒãƒªãƒªãƒ¼ã‚¹ã•ã‚Œã¾ã—ãŸã€‚å¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿å‡¦ç†æŠ€è¡“ãŒä¸€æ°—ã«æ°‘é–“ã®æ‰‹ã«ä¸‹ã‚Šã¦ãã‚‹ã€ã¾ã•ã«é©å‘½çš„ãªã‚µãƒ¼ãƒ“ã‚¹ã ã¨æ€ã„ã¾ã™ã€‚ Amazon Elastic MapReduce Amazon ElasticMapReduce ç´¹ä»‹ãƒ“ãƒ‡ã‚ª With Hadoop, Amazon Adds A Web-Scale Data Processing Engine To Its Cloud Computer by techcrunch.com Elastic MapReduceã¯ã€Googleã®åŸºç›¤æŠ€è¡“ã®ä¸€ã¤ã§ã‚ã‚‹MapReduceã‚’æ™‚é–“å˜ä½èª²é‡‘ã§å®Ÿè¡Œã§ãã‚‹ã‚µãƒ¼ãƒ“ã‚¹ã§ã™ã€‚MapReduceã«ã¤ã„ã¦ã¯ä»¥
HISAMATSU 2009/04/03
mapreduce ã¯ä»Šå¹´ã®å’è«–ã®ãƒã‚¿ã«ã™ã‚‹

amazon

mapreduce

research
ãƒªãƒ³ã‚¯
Big Data Platform - Amazon EMR - AWS
Amazon EMR Serverless is a new option in Amazon EMR that makes it easy and cost-effective for data engineers and analysts to run applications built using open source big data frameworks such as Apache Spark, Hive or Presto, without having to tune, operate, optimize, secure or manage clusters.
HISAMATSU 2009/04/03
amazon

mapreduce
ãƒªãƒ³ã‚¯
MOONGIFT: Â» Googleã®ãƒ‡ãƒ¼ã‚¿å‡¦ç†åˆ†æ•£ã‚·ã‚¹ãƒ†ãƒ MapReduceã®ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹å®Ÿè£…ã€ŒSkynetã€:ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ã‚’æ¯Žæ—¥ç´¹ä»‹
Googleã§ã¯ãã®è¶…å·¨å¤§ãªã‚³ãƒ³ãƒ”ãƒ¥ãƒ¼ã‚¿ãƒãƒƒãƒˆãƒ¯ãƒ¼ã‚¯ã‚’ä½¿ã£ã¦ã€ãƒ‡ãƒ¼ã‚¿å‡¦ç†ãŒåˆ†æ•£åŒ–ã•ã‚Œã¦ã„ã‚‹ã€‚ã“ã‚Œã«ã‚ˆã‚Šã€å¤§é‡ã®ãƒ‡ãƒ¼ã‚¿ã‚’çž¬æ™‚ã«å‡¦ç†ã™ã‚‹ã“ã¨ãŒå¯èƒ½ã«ãªã£ã¦ã„ã‚‹ã€‚ã“ã®åˆ†æ•£å‡¦ç†ã‚·ã‚¹ãƒ†ãƒ ã¯MapReduceã¨å‘¼ã°ã‚Œã¦ãŠã‚Šã€Googleã®åŸºç›¤ã‚’æ”¯ãˆã‚‹ã‚³ã‚¢æŠ€è¡“ã®ä¸€ã¤ã ã€‚ å‡¦ç†çŠ¶æ…‹ã‚’ç¢ºèªã™ã‚‹ã‚³ãƒ³ã‚½ãƒ¼ãƒ« ã”ãå°è¦æ¨¡ãªã‚·ã‚¹ãƒ†ãƒ ã§ã‚ã‚Œã°ãƒ‹ãƒ¼ã‚ºã¯ç™ºç”Ÿã—ãªã„ã‹ã‚‚çŸ¥ã‚Œãªã„ãŒã€æ•°åå°ã€æ•°ç™¾å°ã®ã‚³ãƒ³ãƒ”ãƒ¥ãƒ¼ã‚¿ã‚’çµã³ã¤ã‘ã‚‹ä¸Šã§åˆ†æ•£åŒ–å‡¦ç†ã¯æ¬ ã‹ã›ãªã„æŠ€è¡“ã ã€‚ãã“ã§MapReduceã‚’ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹å®Ÿè£…ã—ãŸã“ã¡ã‚‰ã‚’ç´¹ä»‹ã—ã‚ˆã†ã€‚ ä»Šå›žç´¹ä»‹ã™ã‚‹ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ãƒ»ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ã¯Skynetã€Rubyã§å®Ÿè£…ã•ã‚ŒãŸMapReduceã®ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹å®Ÿè£…ã ã€‚ Skynetã¯å¤šæ•°ã®ãƒ¯ãƒ¼ã‚«ãƒ¼ã‚’ç«‹ã¡ä¸Šã’ã€ãã‚Œã‚‰ãŒäº’ã„ã«ç›£è¦–ã—åˆã†ã“ã¨ã§éšœå®³ç™ºç”Ÿæ™‚ã«ã‚‚æŸ”è»Ÿã«ã‚¿ã‚¹ã‚¯ã®å—ã‘æ¸¡ã—ãŒå¯èƒ½ã«ãªã£ã¦ã„ã‚‹ã€‚å˜ä¸€éšœå®³ç‚¹ã¯ãªãã€ãƒžã‚¹ã‚¿ã‚µãƒ¼ãƒã¨ã„ã†ä½ç½®ã¥ã‘ã®ã‚‚ã®ã§ã™ã‚‰ä»–ã®
HISAMATSU 2008/06/21
rubyç‰ˆã® mapreduce

ruby

mapreduce
ãƒªãƒ³ã‚¯
1