[B! MapReduce] koemuã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

koemu id:koemu

MapReduceã«é–¢ã™ã‚‹koemuã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (8)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

Apache Sparkã£ã¦ã©ã‚“ãªã‚‚ã®ã‹è¦‹ã¦ã¿ã‚‹ï¼ˆãã®ï¼‘ - å¤¢ã¨ã‚¬ãƒ©ã‚¯ã‚¿ã®é›†ç©å ´
ã“ã‚“ã«ã¡ã¯ã€‚ Kafkaã‚’è©¦ã—ã¦ã„ã‚‹æœ€ä¸ã§å¾®å¦™ã§ã™ãŒã€æœ€è¿‘ä½¿ãˆã‚‹ã®ã‹ãªãã€ã¨æƒ…å ±ã‚’é›†ã‚ã¦ã„ã‚‹ã®ãŒã€ŒApache Sparkã€ã§ã™ã€‚ MapReduceã¨åŒã˜ãåˆ†æ•£ä¸¦è¡Œå‡¦ç†ã‚’è¡Œã†åŸºç›¤ãªã®ã§ã™ãŒã€MapReduceã‚ˆã‚Šã‚‚æ•°åå€é€Ÿã„ã¨ã‹ã®æƒ…å ±ãŒã‚ã‚Šã¾ã™ã€‚ ãƒ»ãƒ»ãƒ»ã‚“ãªé˜¿å‘†ãªã€ã¨ã‚‚æ€ã£ãŸã®ã§ã™ãŒã€å†…éƒ¨ã§ä¿æŒã—ã¦ã„ã‚‹RDDã¨ã„ã†ä»•çµ„ã¿ãŒé¢ç™½ã„ã“ã¨ã‚‚ã‚ã‚Šã€ ã¨ã‚Šã‚ãˆãšè³‡æ–™ã‚„è«–æ–‡ã‚’èªã‚“ã§ã¿ã‚‹ã“ã¨ã«ã—ã¾ã—ãŸã€‚ ã¾ãšè¦‹ã¦ã¿ãŸè³‡æ–™ã¯ã€ŒOverview of Sparkã€ï¼ˆhttp://spark.incubator.apache.org/talks/overview.pdfï¼‰ã§ã™ã€‚ ã¨ã„ã†ã‚ã‘ã§ã€èªã‚“ã çµæžœã‚’ã¾ã¨ã‚ã¦ã¿ã¾ã™ã€‚ Sparkã¨ã¯ï¼Ÿ é«˜é€Ÿã§ã‚¤ãƒ³ã‚¿ãƒ©ã‚¯ãƒ†ã‚£ãƒ–ãªè¨€èªžçµ±åˆã‚¯ãƒ©ã‚¹ã‚¿ã‚³ãƒ³ãƒ”ãƒ¥ãƒ¼ãƒ†ã‚£ãƒ³ã‚°åŸºç›¤ Sparkãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆã®ã‚´ãƒ¼ãƒ«ã¯ï¼Ÿ ä»¥ä¸‹ã®2ã¤ã®è§£æžãƒ¦ãƒ¼ã‚¹ã‚±ãƒ¼ã‚¹ã«ã‚ˆã‚Šé©åˆã™ã‚‹ã‚ˆã†MapReduceã‚’æ‹¡å¼µ
koemu 2013/09/03
MapReduce

åˆ†æ•£å‡¦ç†

Apache Spark
ãƒªãƒ³ã‚¯
Apache Sparkâ„¢ - Unified Engine for large-scale data analytics
Apache Sparkâ„¢ is a multi-language engine for executing data engineering, data science, and machine learning on single-node machines or clusters.
koemu 2013/08/25
åˆ†æ•£å‡¦ç†

MapReduce

Apache Spark
ãƒªãƒ³ã‚¯
å¹³æˆï¼’ï¼‘å¹´åº¦ ç”£å¦é€£æºã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢å·¥å¦å®Ÿè·µäº‹æ¥å ±å‘Šæ›¸ã®å…¬è¡¨ã«ã¤ã„ã¦ï¼ˆMETI/çµŒæ¸ˆç”£æ¥çœï¼‰
é«˜ä¿¡é ¼çµ„è¾¼ã¿ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢é–‹ç™ºï¼ˆå§”è¨—å…ˆï¼šä¸€èˆ¬ç¤¾å›£æ³•äººJASPARï¼‰ å ±å‘Šæ›¸ï¼ˆPDFå½¢å¼ï¼š3,278KBï¼‰ ï¼ˆZIPå½¢å¼ï¼š2,993KBï¼‰ ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢å·¥å¦ã®å®Ÿè·µå¼·åŒ–ã«é–¢ã™ã‚‹èª¿æŸ»ç ”ç©¶ï¼ˆå§”è¨—å…ˆï¼šæ ªå¼ä¼šç¤¾ä¸‰è±ç·åˆç ”ç©¶æ‰€ï¼‰ å ±å‘Šæ›¸ï¼ˆPDFå½¢å¼ï¼š2,501KBï¼‰ ã‚¯ãƒ©ã‚¦ãƒ‰ã‚³ãƒ³ãƒ”ãƒ¥ãƒ¼ãƒ†ã‚£ãƒ³ã‚°æ™‚ä»£ã®Dependabilityã®è€ƒãˆæ–¹ãªã©ã«é–¢ã™ã‚‹ç±³å›½ã®å‹•å‘èª¿æŸ»ï¼ˆå§”è¨—å…ˆï¼šæ ªå¼ä¼šç¤¾ã‚¢ã‚¤ãƒ»ãƒ“ãƒ¼ãƒ»ãƒ†ã‚£ï¼‰ å ±å‘Šæ›¸ï¼ˆPDFå½¢å¼ï¼š4,583KBï¼‰ ï¼ˆZIPå½¢å¼ï¼š4,300KBï¼‰ ã‚¯ãƒ©ã‚¦ãƒ‰ãƒ»ã‚³ãƒ³ãƒ”ãƒ¥ãƒ¼ãƒ†ã‚£ãƒ³ã‚°ã«é–¢ã™ã‚‹å›½å†…å¤–ã®åˆ¶åº¦ãƒ»æŠ€è¡“å‹•å‘ç‰ã®èª¿æŸ»ç ”ç©¶ï¼ˆå§”è¨—å…ˆï¼šæ ªå¼ä¼šç¤¾é‡Žæ‘ç·åˆç ”ç©¶æ‰€ï¼‰ å ±å‘Šæ›¸ï¼ˆPDFå½¢å¼ï¼š2,050KBï¼‰ é«˜ä¿¡é ¼ã‚¯ãƒ©ã‚¦ãƒ‰å®Ÿç¾ç”¨ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢é–‹ç™ºï¼ˆåˆ†æ•£åˆ¶å¾¡å‡¦ç†æŠ€è¡“ç‰ã«ä¿‚ã‚‹ãƒ‡ãƒ¼ã‚¿ã‚»ãƒ³ã‚¿ãƒ¼é«˜ä¿¡é ¼åŒ–ã«å‘ã‘ãŸå®Ÿè¨¼äº‹æ¥ï¼‰ï¼ˆå§”è¨—å…ˆï¼šæ ªå¼ä¼šç¤¾ã‚¨ãƒŒãƒ»ãƒ†ã‚£ãƒ»ãƒ†ã‚£ãƒ»ãƒ‡ãƒ¼ã‚¿ï¼‰ ï¼ˆPDFå½¢å¼ï¼š9,606KBï¼‰ ï¼ˆZIPå½¢å¼ï¼š8,656
koemu 2010/09/29
ã‚¯ãƒ©ã‚¦ãƒ‰

é–‹ç™º

Hadoop

MapReduce

æ–‡æ›¸

ITã‚¤ãƒ³ãƒ•ãƒ©
ãƒªãƒ³ã‚¯
è‡ªç„¶è¨€èªžå‡¦ç†ã«ãŠã‘ã‚‹ MapReduce ã®ä½¿ã„æ–¹ã®ãƒãƒ¥ãƒ¼ãƒˆãƒªã‚¢ãƒ« - æ¦è”µé‡Žæ—¥è¨˜
ä»Šå¦æœŸã¯æ¯Žé€±è«–æ–‡ç´¹ä»‹ã™ã‚‹ãƒã‚¿ã‚’æŽ¢ã™ã®ã‚‚ç–²ã‚Œã‚‹ã®ã§ã€é©å®œè«–æ–‡ç´¹ä»‹ã‚’å…¥ã‚ŒãªãŒã‚‰ã€ Data-Intensive Text Processing With MapReduce (Synthesis Lectures on Human Language Techno logies) ä½œè€…: Jimmy Lin,Chris Dyerå‡ºç‰ˆç¤¾/ãƒ¡ãƒ¼ã‚«ãƒ¼: Morgan and Claypool Publishersç™ºå£²æ—¥: 2010/08/15ãƒ¡ãƒ‡ã‚£ã‚¢: ãƒšãƒ¼ãƒ‘ãƒ¼ãƒãƒƒã‚¯ ã‚¯ãƒªãƒƒã‚¯: 67å›žã“ã®å•†å“ã‚’å«ã‚€ãƒ–ãƒã‚° (6ä»¶) ã‚’è¦‹ã‚‹ã‚’ã—ã°ã‚‰ãèªã‚‚ã†ã‹ã¨æ€ã£ã¦ã„ã‚‹ã€‚ã¡ãªã¿ã« http://www.umiacs.umd.edu/~jimmylin/book.html ã‹ã‚‰å…¨æ–‡ã® PDF ãŒãƒ€ã‚¦ãƒ³ãƒãƒ¼ãƒ‰ã§ãã‚‹ã®ã§ã€ãã¡ã‚‰ã‚’ä½¿ç”¨äºˆå®šã€‚å°åˆ·ãƒ»è£½æœ¬ã—ã¦ãƒ‘ãƒ©ãƒ‘ãƒ©ã¨ã‚ãã£ã¦ã„ã‚‹ãŒã€ãƒ†ã‚ã‚¹ãƒˆçš„ã«ã¯å‰²ã¨ã„ã„æœ¬ã ã¨æ€ã†ã€‚ã¿ã‚“ãªãŒã“ãž
koemu 2010/04/21
MapReduce

è‡ªç„¶è¨€èªžå‡¦ç†
ãƒªãƒ³ã‚¯
Parallel::MapReduce
NAME Parallel::MapReduce - MapReduce Infrastructure, multithreaded SYNOPSIS ## THIS IS ALL STILL EXPERIMENTAL!! ## DO NOT USE FOR PRODUCTION!! ## LOOK AT THE ROADMAP AND FEEDBACK WHAT YOU FIND IMPORTANT!! use Parallel::MapReduce; my $mri = new Parallel::MapReduce (MemCacheds => [ '127.0.0.1:11211', .... ], Workers => [ '10.0.10.1', '10.0.10.2', ...]); my $A = {1 => 'this is something ', 2 => 'this
koemu 2008/07/21
Perl

MapReduce

ãƒ©ã‚¤ãƒ–ãƒ©ãƒª
ãƒªãƒ³ã‚¯
MapReduce:Simplified Data Processing on Large Clusters
MapReduce: Simplified Data Processing on Large Clusters
koemu 2008/02/12
MapReduce

é–‹ç™º
ãƒªãƒ³ã‚¯
YappoLogs: Googleã®MapReduceã‚’ã„ã¾ã•ã‚‰å¦„æƒ³ã—ãŸ
å‡¦ç†å†…å®¹ mapã«å¯¾ã—ã¦ã¯ä»»æ„ã®ãƒ‡ãƒ¼ã‚¿ãŒä¸Žãˆã‚‰ã‚Œã‚‹ã€‚ mapã¯keyã¨valueã‹ã‚‰ãªã‚‹å¤§é‡ã®ãƒ‡ãƒ¼ã‚¿ã‚’æˆ»ã™ã€‚ shuffleã«ã¦ã€å…¨ã¦ã®mapã®keyã‚’ã¾ã¨ã‚ã‚ã’ã¦ã€keyã”ã¨ã«reduce workerã«keyã¨value listã‚’æ¸¡ã™ã€‚ reduceã¯ã€å—ã‘å–ã£ãŸkey/value listã‚’å‡¦ç†ã™ã‚‹ã€‚ key/valueãªãƒ‡ãƒ¼ã‚¿ã«ç‰¹åŒ–ã—ãŸPlaggerã£ã¦ã“ã¨ã§é–“é•ã„ãªã„ã®ã‹ãªãï¼Ÿ mapperã§å¤§é‡ã®ãƒ‡ãƒ¼ã‚¿ã‹ã‚‰å¿…è¦ãªç‰©ã‚’ãƒ•ã‚£ãƒ«ã‚¿ãƒªãƒ³ã‚°(Subscription,Aggregator)ã—ã¦ã€reducerã§å®Ÿéš›ã®å‡¦ç†(Filter,Publish,Notify)ã‚’è¡Œã†ã¨ã„ã†Plaggerã¿ãŸã„ãªæ„Ÿã˜ã€‚ å…¨ã¦ã®mapã‚„reduceã«å¤§ã—ã¦ã®å„workerã®ä»•äº‹é‡ã¯å¹³å‡çš„ã«ãªã‚‹æ§˜ã«ãƒãƒ©ãƒ³ã‚¹è‰¯ãé…ç½®ã™ã‚‹ã€‚ å¤šåˆ†ã€mapã‚„reduceã®ç›´å‰ã§ã€ãã‚Œãžã‚Œã®jobã®å¤§ãã•ã‚’è¨ˆæ¸¬ã—ã¦manage
koemu 2008/02/10
(å…ˆã«ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ã—ã¦ã„ã‚‹äººãŒã„ã‚‹ã£ã™)

MapReduce

è² è·å¯¾ç–

é–‹ç™º
ãƒªãƒ³ã‚¯
OBB vs AABB - Radium Software Development
This domain may be for sale!
koemu 2007/10/29
CAPTCHAã®å¼·åº¦ã«ã¤ã„ã¦ã®è§£èª¬

èªè¨¼

ã‚»ã‚ãƒ¥ãƒªãƒ†ã‚£
ãƒªãƒ³ã‚¯
1