[B! mapreduce] y_uukiã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

y_uuki id:y_uuki

mapreduceã«é–¢ã™ã‚‹y_uukiã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (7)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

ã€ŒHadoopã®æ™‚ä»£ã¯çµ‚ã‚ã£ãŸã€ã®æ„å‘³ã‚’æ£ã—ãç†è§£ã™ã‚‹ - ç§‘å¦ã¨éžç§‘å¦ã®è¿·å®®
Hadoopã®æ™‚ä»£ã¯çµ‚ã‚ã£ãŸã€ã¨ã„ã†è¨€èª¬ã‚’ãŸã¾ã«è¦‹ã‹ã‘ã‚‹ã‚ˆã†ã«ãªã‚Šã¾ã—ãŸã€‚ ã‚‚ã¡ã‚ã‚“çµ‚ã‚ã£ã¦ãªã©ã„ã¾ã›ã‚“ã€‚ã—ã‹ã—ã€Hadoopã¨ãã®å–ã‚Šå·»ãç’°å¢ƒãŒå¤‰åŒ–ã—ãŸã®ã¯äº‹å®Ÿã§ã™ã€‚ æœ¬è¨˜äº‹ã§ã¯ã€ã“ã®å¤‰åŒ–ãŒä½•ãªã®ã‹ã‚’æ˜Žã‚‰ã‹ã«ã—ã€ãã®ä¸Šã§ã€ãªãœHadoopã®æ™‚ä»£ã¯çµ‚ã‚ã£ãŸã¨ã„ã†ä¸»å¼µãŒå®Ÿæ…‹ã‚’æ£ã—ãè¡¨ã—ã¦ã„ãªã„ã®ã‹ã‚’èª¬æ˜Žã—ã¦ã„ãã¾ã™ã€‚ DISCLAIMER ç§ã¯Hadoopã‚’ä¸å¿ƒã¨ã—ãŸãƒ‡ãƒ¼ã‚¿åŸºç›¤ã‚’å–ã‚Šæ‰±ã†ãƒ™ãƒ³ãƒ€ãƒ¼ã€Clouderaã®ç¤¾å“¡ã§ã™ã€‚ ä¸ç«‹çš„ã«æ›¸ãã‚ˆã†åŠªã‚ã¾ã™ãŒã€æ‰€å±žçµ„ç¹”ã«ã‚ˆã£ã¦ç™ºç”Ÿã™ã‚‹ãƒã‚¤ã‚¢ã‚¹ã®å®Œå…¨ãªæŽ’é™¤ã‚’ä¿è¨¼ã™ã‚‹ã“ã¨ã¯ã§ãã¾ã›ã‚“ã€‚ ä»¥ä¸Šã‚’ã”äº†æ‰¿ã®ä¸Šã€èªã¿é€²ã‚ã¦ãã ã•ã„ã€‚ è¦ç´„ ãƒ‡ãƒ¼ã‚¿åŸºç›¤ã¯ã€Hadoopã®ç™»å ´ã«ã‚ˆã‚Šéžå¸¸ã«å®‰ä¾¡ã¨ãªã‚Šã€ä»Šã¾ã§ã§ã¯ä¸å¯èƒ½ã ã£ãŸå¤§é‡ã®ãƒ‡ãƒ¼ã‚¿ã‚’å–ã‚Šæ‰±ãˆã‚‹ã‚ˆã†ã«ãªã‚Šã¾ã—ãŸã€‚ Hadoopã¯ã€NoSQLãƒ–ãƒ¼ãƒ ã®ä¸ã€å‡¦ç†ã‚¨ãƒ³ã‚¸ãƒ³ã§ã‚ã‚‹MapReduceã¨ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ã§ã‚ã‚‹HDFSãŒ
y_uuki 2017/07/10
ã‚ã‹ã‚Šã‚„ã™ã„

Hadoop

mapreduce
ãƒªãƒ³ã‚¯
MapReduce - munching through Big Data
How Google tackled the probl em of processing enormous amounts of data, and how you can do the same with Go. Itâ€™s been a while since the last post, and I have to apologize for the long wait. The last weeks have been quite busy, but I finally managed to complete another article. I hope youâ€™ll enjoy it. Map and ReduceThis is going to be a boring article about two boring functions, map() and reduce().
y_uuki 2016/12/18
golang

MapReduce
ãƒªãƒ³ã‚¯
http://tothestars.io/blog/2016/11/2/serverless-mapreduce
y_uuki 2016/11/06
serverless

mapreduce

ã‚ã¨ã§èªã‚€
ãƒªãƒ³ã‚¯
The Truth About MapReduce Performance on SSDs | USENIX
y_uuki 2016/05/09
mapreduce

ssd

performance
ãƒªãƒ³ã‚¯
Googleã€å¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿ã‚’ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ã«åˆ†æžã§ãã‚‹ã‚¯ãƒ©ã‚¦ãƒ‰ã‚µãƒ¼ãƒ“ã‚¹ã€ŒGoogle Cloud Dataflowã€ã‚’ç™ºè¡¨ã€‚ã€Œ1å¹´å‰ã‹ã‚‰MapReduceã¯ä½¿ã£ã¦ã„ãªã„ã€ã€‚Google I/O 2014
Googleã€å¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿ã‚’ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ã«åˆ†æžã§ãã‚‹ã‚¯ãƒ©ã‚¦ãƒ‰ã‚µãƒ¼ãƒ“ã‚¹ã€ŒGoogle Cloud Dataflowã€ã‚’ç™ºè¡¨ã€‚ã€Œ1å¹´å‰ã‹ã‚‰MapReduceã¯ä½¿ã£ã¦ã„ãªã„ã€ã€‚Google I/O 2014 å¤§è¦æ¨¡åˆ†æ•£å‡¦ç†ã®ãƒ•ãƒ¬ãƒ¼ãƒ ãƒ¯ãƒ¼ã‚¯ã¨ã—ã¦GoogleãŒé–‹ç™ºã—ã€Hadoopã«æŽ¡ç”¨ã•ã‚Œã¦åºƒãä½¿ã‚ã‚Œã¦ã„ã‚‹MapReduceã€‚ã—ã‹ã—Googleã¯ã‚‚ã†MapReduceã‚’ä½¿ã‚ãšã€ã‚ˆã‚Šå„ªã‚ŒãŸå‡¦ç†ç³»ã®ã€ŒGoogle Cloud Dataflowã€ã‚’ä½¿ã£ã¦ã„ã‚‹ã“ã¨ãŒã€Google I/O 2014ã®åŸºèª¿è¬›æ¼”ã§æ˜Žã‚‰ã‹ã«ã•ã‚Œã¾ã—ãŸã€‚ Googleã®ã‚·ãƒ‹ã‚¢ãƒã‚¤ã‚¹ãƒ—ãƒ¬ã‚¸ãƒ‡ãƒ³ãƒˆUrs HÃ¶lzleæ°ã¯ã€ã€Œã‚¨ã‚¯ã‚µãƒã‚¤ãƒˆã®ã‚¹ã‚±ãƒ¼ãƒ«ã¾ã§æ‰±ãˆã€ãƒ‘ã‚¤ãƒ—ãƒ©ã‚¤ãƒ³å‡¦ç†ã‚’è¨˜è¿°ã—ã‚„ã™ãæœ€é©åŒ–ã‚‚ã—ã¦ãã‚Œã‚‹ã€‚ãã‚Œã«ãƒãƒƒãƒã‚‚ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ åˆ†æžã‚‚åŒã˜ã‚³ãƒ¼ãƒ‰ã§è¨˜è¿°ã§ãã‚‹ã€ã¨ã€Cloud Dataflowã®ç‰¹é•·ã‚’èª¬æ˜Žã—ã¾ã™ã€‚ Google I/Oã®
y_uuki 2014/06/29
Google

mapreduce
ãƒªãƒ³ã‚¯
Facebookã€åˆ†æ•£SQLã‚¨ãƒ³ã‚¸ãƒ³ã€ŒPrestoã€å…¬é–‹ã€‚å¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿ã‚’MapReduce/Hiveã®10å€åŠ¹çŽ‡ã‚ˆãå‡¦ç†ã™ã‚‹ã¨
Facebookã¯ã€æ•°ãƒšã‚¿ãƒã‚¤ãƒˆç´šã®å¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿ã«å¯¾ã—ã¦ã‚‚ã€å¯¾è©±çš„ã«ã‚¢ãƒ‰ãƒ›ãƒƒã‚¯ãªå•ã„åˆã‚ã›ã‚’å¯èƒ½ã«ã™ã‚‹åˆ†æ•£SQLã‚¨ãƒ³ã‚¸ãƒ³ã€ŒPrestoã€ã‚’ã€ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ã§å…¬é–‹ã—ã¾ã—ãŸã€‚ Prestoã¯Facebookç¤¾å†…ã§å¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿ã®åˆ†æžã®ãŸã‚ã«é–‹ç™ºã•ã‚Œã€ã™ã§ã«åŒç¤¾ç¤¾å†…ä½¿ã‚ã‚Œã¦ã„ã‚‹ã‚‚ã®ã€‚ Facebookã¯Prestoã‚’é–‹ç™ºã—ãŸèƒŒæ™¯ã¨ã—ã¦ã€å¤§é‡ã®ãƒ‡ãƒ¼ã‚¿ã‚’Hadoop/HDFSãƒ™ãƒ¼ã‚¹ã§ä¿å˜ã—ãŸã‚‚ã®ã®ã€ãƒãƒƒãƒæŒ‡å‘ã®MapReduceã§ã¯ãªãã€ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ æ€§ã«å„ªã‚ŒãŸå‡¦ç†ãŒå¿…è¦ã«ãªã£ãŸãŸã‚ã ã¨ã€æ¬¡ã®ã‚ˆã†ã«èª¬æ˜Žã—ã¦ã„ã¾ã™ã€‚ Facebookâ€™s warehouse data is stored in a few large Hadoop/HDFS-based clusters. Hadoop MapReduce [2] and Hive are designed for large-scale, reliabl
y_uuki 2013/11/10
facebook

mapreduce

Hadoop
ãƒªãƒ³ã‚¯
MapReduceã¯ä»Šå¾Œã©ã†ãªã‚‹ã®ã‹ï¼Ÿ - æ€¥ãŒã°å›žã‚Œã€é¸ã¶ãªã‚‰è¿‘é“
2012å¹´ã®ç¾åœ¨ã€å‰²ã¨æ‚©ã‚“ã§ã„ã‚‹ã®ã§ãƒ¡ãƒ¢ã£ã¦ãŠãã€‚ å¹´åº¦æœ«ãã‚‰ã„ã«å†èª¿æŸ»ã®äºˆå®šã€‚ãƒ»ãƒ»ãªã®ã§æš«å®šã§ã™ã‚ˆã€‚ ã¾ãšå‰æã¨ã—ã¦ã€ç¾è¡Œã®Hadoopã®å®Ÿè¡Œãƒ•ãƒ¬ãƒ¼ãƒ ãƒ¯ãƒ¼ã‚¯ã§ã‚ã‚‹MapReduceã¯ã€å®Ÿè¡ŒåŠ¹çŽ‡ã¯æ±ºã—ã¦è‰¯ãã¯ãªã„ã§ã™ã€‚ã“ã®è¾ºãŒå‰²ã¨è¾›ã„ã€‚ ã¨ã¯ã„ãˆã€å¤§è¦æ¨¡ä¸¦åˆ—å‡¦ç†ã‚’ä¸€èˆ¬çš„ã«è¡Œã†ã¨ã„ã†è¦³ç‚¹ã§ã®å“è³ªã‚„å–ã‚Šå›žã—ã‚’è€ƒãˆãŸå ´åˆã€â€çµæžœã¨ã—ã¦â€éžå¸¸ã«ãƒãƒ©ãƒ³ã‚¹ãŒã¨ã‚Œã¦ãŠã‚Šã€æ™®åŠã—ã¦ã„ã‚‹ã€‚ãã®ä¸Šã§ã€ã“ã®MapReduceã§ã™ãŒã€ä»Šå¾Œã®è¦‹é€šã—ã«ã¤ã„ã¦ã¯ã€æ½®æµã¯ä»Šã®ã¨ã“ã‚äºŒã¤ã«å‰²ã‚Œã¦ã„ã‚‹ã‚ˆã†è¦‹ãˆã‚‹ã€‚ã®ã§ã€ãã®è¾ºã®ãƒ¡ãƒ¢ã€‚ â– YARN ä¸€ã¤ã®æ–¹å‘æ€§ã¯ã€ç¾åœ¨ã®Hadoop2.0ç³»ã§å®Ÿè£…ã•ã‚Œã¦ã„ã‚‹MapReduce2.0ã€ã¨ã„ã†ã‹ã€MapReduceã¨ã¯åˆ¥ã®å®Ÿè¡ŒåŸºç›¤ã‚’åˆ©ç”¨ã™ã‚‹ã¨ã„ã†æ–¹å‘ã§ã™ãã€‚ã™ãªã‚ã¡BSPã‚„ã€MPIã‚’åˆ©ç”¨ã™ã‚‹ã€‚è¦ã¯ã€ä»Šã¾ã§ã®ä¸¦åˆ—å‡¦ç†ã®æˆæžœã‚’ãã®ã¾ã¾åˆ©ç”¨ã—ã¾ã—ã‚‡ã†ã€ã¨ã„ã†æµã‚Œã«è¿‘ã„ã€‚ MapReduce
y_uuki 2012/10/09
mapreduce
ãƒªãƒ³ã‚¯
1