MRUnit Incubation Status - Apache Incubator
This page tracks the project status, incubator-wise. For more general project status, look on the project website. The MRUnit project graduated on 2012-05-16 2012-05-16 MRUnit graduates from the Incubator to become a TLP 2012-05-01 Version 0.9.0-incubating released. 2012-04-18 Dave Beech added as a new committer. 2012-04-14 Jarek Jarcec Cecho added as a new committer. 2012-04-11 New website create
Map Reduce 〜入門編：仕組ã¿ã®ç†è§£ã¨ã‚¢ãƒ«ã‚´ãƒªã‚ºãƒ デザイン〜
This document discusses using sliding windows to aggregate streaming data in MapReduce. It proposes buffering input tuples in mappers until a window is full, then emitting the aggregate. Combiners and reducers combine partial aggregates across windows. Window ranges are initialized and updated during merging to remove outdated data and handle late arrivals. This approach allows streaming aggregati
Hadoop 㧠Wikipedia ã®ãƒ†ã‚スト処ç†ã‚’900å€é«˜é€ŸåŒ– - æ¦è”µé‡Žæ—¥è¨˜
今月ä¸ã«å®Ÿé¨“ã®å®Ÿè£…ãŒçµ‚ã‚ã‚‹ãらã„ã§ãªã„ã¨æ¥æœˆã®æŠ•ç¨¿ã€†åˆ‡ã«é–“ã«åˆã‚ãªã„ã®ã§ã€ä»Šé€±ã‹ã‚‰ç ”究室ã®ã‚µãƒ¼ãƒã« Hadoop をインストールã—ã¦ã„る。 ç ”ç©¶å®¤ã«ã¯ã‚µãƒ¼ãƒãŒ20å°å¼±ã‚ã‚‹ã®ã ãŒã€ãã®ã†ã¡10å°å¼·ã‚’使ã†ã“ã¨ã«ã—ã¦è¨å®šã€‚ã“ã‚Œãらã„ã®è¦æ¨¡ã ã¨ã€Œå¤§è¦æ¨¡ã€ã¨è¨€ã†ã®ã¯æ†šã‚‰ã‚Œã‚‹ã‹ã‚‚ã—ã‚Œãªã„ãŒ(Yahoo! ã‚„ Google ã¨æ¯”ã¹ã¦ã€ã¨ã„ã†æ„味ã§ã€‚)ã€ä¸è¦æ¨¡ã€ãらã„ã«ã¯è¨€ã£ã¦ã‚‚ã„ã„ã ã‚ã†ã—ã€ãŸã¶ã‚“ã€å¤šãã®å¤§å¦ã‚„ä¼æ¥ã§ä½¿ãˆã‚‹å°æ•°ã‚‚ã“ã‚Œãらã„ã ã¨æ€ã†ã—ã€å¤§ä¼æ¥ã«ã„ãªã„ã¨ã§ããªã„ç ”ç©¶ã‚’ã™ã‚‹ã®ã‚‚大変価値ãŒã‚ã‚‹ãŒã€ä»–ã®äººãŸã¡ãŒã‚„ã‚‹æ°—ã«ãªã‚Œã°çœŸä¼¼ã§ãã‚‹ç ”ç©¶ã‚’ã™ã‚‹ã®ã‚‚(データやインフラå‹è² ã§ã¯ãªãアイデアå‹è² ã«ãªã‚‹ã®ã§è‹¦ã—ãã¯ã‚ã‚‹ã®ã ãŒ)é‡è¦ã ã¨è€ƒãˆã¦ã„る。 ãŸã¨ãˆã°ã€æ•°å°ã§ã‚‚分散環境ã®æ©æµãŒå—ã‘られるã€ã¨ã„ã†ã®ã¯PFI ãŒå‡ºã—㟠Hadoop ã®è§£æžè³‡æ–™ã§çŸ¥ã£ã¦ã„ãŸã®ã§ã€åˆã‚ã¦å°Žå…¥ã—ãŸã¨ãã¯å‚考ã«ãªã£ãŸã—ã€ã“ã†ã„ã†
nutrun » Blog Archive » EventMachine MapReduce
MapReduce is a parallel computation strategy useful for scaling large data set processing by distributing workload over multiple worker nodes. The distributed nature of MapReduce suggests network communication and, with that in mind, I thought I’d put together a demonstration employing EventMachine, a library which makes efficient network programming relatively simple in Ruby. Before going any fur
グーグルã«ã‚ˆã‚‹MapReduceサービス「BigQueryã€ãŒç™»å ´ã€‚SQLライクãªå‘½ä»¤ã§å¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿æ“作
「数兆件ã®ãƒ‡ãƒ¼ã‚¿ã‚‚対話的ã«ã€é«˜é€Ÿã«åˆ†æžã§ãã‚‹ã€ã€‚グーグルã¯5月19æ—¥ã«ã“ã®ã‚ˆã†ãªè¡¨ç¾ã§æ–°ã—ã„サービス「BigQueryã€ã®ç™»å ´ã‚’紹介ã™ã‚‹ã‚¨ãƒ³ãƒˆãƒªã‚’ã€ãƒ–ãƒã‚°ã«ãƒã‚¹ãƒˆã—ã¦ã„ã¾ã™ã€‚ グーグルãŒå…¬é–‹ã—ãŸBigQueryã¯ã€Hadoopやデータウェアãƒã‚¦ã‚¹ãªã©ã‚’用ã„ã¦å¤šãã®ä¼æ¥ãŒè¡ŒãŠã†ã¨ã—ã¦ã„る大è¦æ¨¡ãƒ‡ãƒ¼ã‚¿ï¼ˆã„ã‚ゆる「Big Dataã€ï¼‰ã®åˆ†æžã‚’ã€ã‚°ãƒ¼ã‚°ãƒ«ã®ã‚¯ãƒ©ã‚¦ãƒ‰ã§å¯èƒ½ã«ã—ã¾ã™ã€‚利用者ã¯Google Storage経由ã§å¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿ã‚’転é€ã—ã€SQLライクãªå‘½ä»¤ã«ã‚ˆã£ã¦æŠ½å‡ºã‚„分æžã‚’è¡Œã„ã¾ã™ã€‚ ã¾ã‚‹ã§ã‚°ãƒ¼ã‚°ãƒ«ãŒå¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿å‡¦ç†ã®MapReduceをホスティングã—ã€ãã®æ©Ÿèƒ½ã‚’サービスã¨ã—ã¦æä¾›ã™ã‚‹ã‚ˆã†ãªã‚‚ã®ãŒBigQueryã¨ã„ãˆã¾ã™ï¼ˆãŸã ã—公開ã•ã‚ŒãŸã€ŒBigQueryã€ã®èª¬æ˜Žã«ã¯ã€å†…部ã§MapReduceを利用ã—ã¦ã„ã‚‹ã®ã‹ã©ã†ã‹ã®è¨˜è¿°ã¯ãªã„ã®ãŸã‚ã€MapReduce「的ã€ãªã‚µãƒ¼ãƒ“スã¨è¡¨ç¾ã™ã¹ãã‹ã‚‚ã—ã‚Œ
Hadoopã‚’æ¥å‹™ã§ä½¿ã£ã¦ã¿ãŸè©± - クックパッド開発者ブãƒã‚°
8月ã«å…¥ç¤¾ã—ãŸä½ã€…木ã§ã™ã€‚ã“ã‚“ã«ã¡ã‚ï¼ å…¥ç¤¾ã—ã¦ã‹ã‚‰ã¯Hadoopを使ã†ã“ã¨ãŒå¤šãã€æ—¥ã€…ã€å¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿ã¨æ ¼é—˜ã—ã¦ã„ã¾ã™ã€‚大変ã§ã¯ã‚ã‚Šã¾ã™ãŒã€å€‹äººã§ã¯ãªã‹ãªã‹è§¦ã‚‹ã“ã¨ãŒå‡ºæ¥ãªã„よã†ãªå¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿ã‚’触れるã®ã¯æ¥½ã—ã„ã§ã™ã€‚ ã•ã¦ã€Hadoopã¯æœ€è¿‘色々ãªã¨ã“ã‚ã§ä½¿ã‚れ始ã‚ã¦ãã¦ã„ã‚‹ã¨æ€ã†ã‚“ã§ã™ãŒã€å®Ÿéš›ã«åˆ©ç”¨ã—ã¦ã¿ã¦å›°ã£ãŸäº‹ã‚„tipsãªã©ã€å®Ÿè·µçš„ãªæƒ…å ±ã¯ã¾ã ã‚ã¾ã‚Šå…¬é–‹ã•ã‚Œã¦ã„ã¾ã›ã‚“。ãã®è¾ºã®æƒ…å ±ã‚’ã¿ã‚“ãªæ±‚ã‚ã¦ã„ã‚‹ã¯ãšâ€¦ï¼ï¼ ãã“ã§ã€åƒ•ãŒå®Ÿéš›ã«è§¦ã£ã¦ã¿ã¦å›°ã£ãŸäº‹ã‚„Hadoopを使ã†ä¸Šã§ãƒã‚¤ãƒ³ãƒˆã ã¨æ€ã£ãŸã“ã¨ãªã©ã‚’社内勉強会ã§ç™ºè¡¨ã—ãŸã®ã§å…¬é–‹ã—ã¦ã¿ã¾ã™ã€‚Hadoopを使ã£ã¦ã„る（使ã„ãŸã„ã¨æ€ã£ã¦ã„る）方ã®å‚考ã«ãªã‚Œã°å¹¸ã„ã§ã™ã€‚ [slideshare id=2711363&doc=20091214techblog-091213183529-phpapp02] Hadoopã®åˆ©ç”¨ã¯ã¾ã ã¾ã 試行錯誤ã®é€£ç¶š
Hadoopã®æœ€æ–°å‹•å‘を「Hadoop World:NY 2009ã€ã®è³‡æ–™ã‹ã‚‰ï¼ˆå¾Œç·¨ï¼‰
Hadoopã¯ã€ã‚°ãƒ¼ã‚°ãƒ«ãŒå¤§è¦æ¨¡åˆ†æ•£ã‚·ã‚¹ãƒ†ãƒ ã®ãŸã‚ã«ç”¨ã„ã¦ã„ã‚‹MapReduceã¨ã„ã†æŠ€è¡“ã‚’ã€ã‚ªãƒ¼ãƒ—ンソースã¨ã—ã¦å®Ÿç¾ã™ã‚‹ãŸã‚ã«é–‹ç™ºã•ã‚ŒãŸJavaベースã®ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ã€‚クラウド対応ã®ã‚¢ãƒ—リケーションã§ã‚ã‚Šã€æ•°ãƒ†ãƒ©ãƒã‚¤ãƒˆã«ã‚‚ãŠã‚ˆã¶å¤§å®¹é‡ã®ãƒ‡ãƒ¼ã‚¿ã‚’高速ã‹ã¤ä½Žã‚³ã‚¹ãƒˆã«åˆ†æžã™ã‚‹æ–¹æ³•ã¨ã—ã¦æ³¨ç›®ã‚’集ã‚ã¦ã„ã¾ã™ã€‚ 後編ã§ã¯ã€10月2æ—¥ã«ãƒ‹ãƒ¥ãƒ¼ãƒ¨ãƒ¼ã‚¯ã§é–‹å‚¬ã•ã‚ŒãŸã€ŒHadoop World：NY 2009ã€ã®åˆå¾Œã®ã‚»ãƒƒã‚·ãƒ§ãƒ³ã®è³‡æ–™ã«ç›®ã‚’通ã—ã€èˆˆå‘³æ·±ã„ãƒã‚¤ãƒ³ãƒˆã‚’紹介ã—ã¾ã—ょã†ã€‚åˆå¾Œã¯3トラックã«åˆ†ã‹ã‚Œ30ã‚‚ã®ã‚»ãƒƒã‚·ãƒ§ãƒ³ãŒè¡Œã‚ã‚Œã¦ã„ã¾ã—ãŸã€‚ ã“ã®è¨˜äº‹ã¯ã€ŒHadoopã®æœ€æ–°å‹•å‘を「Hadoop World:NY 2009ã€ã®è³‡æ–™ã‹ã‚‰ï¼ˆå‰ç·¨ï¼‰ã€ã®ç¶šãã§ã™ã€‚ åˆå¾Œã®ã‚»ãƒƒã‚·ãƒ§ãƒ³è³‡æ–™ã‹ã‚‰ãƒã‚¤ãƒ©ã‚¤ãƒˆã‚’紹介 イェール大å¦ã®Azza Abouzeidæ°ã¨Kamil Bajda-Pawlikowskiæ°ã¯ã€Hadoopã¨ãƒ‘ラレル
Hadoopã®æœ€æ–°å‹•å‘を「Hadoop World:NY 2009ã€ã®è³‡æ–™ã‹ã‚‰ï¼ˆå‰ç·¨ï¼‰
Hadoopã¯ã€ã‚°ãƒ¼ã‚°ãƒ«ãŒå¤§è¦æ¨¡åˆ†æ•£ã‚·ã‚¹ãƒ†ãƒ ã®ãŸã‚ã«ç”¨ã„ã¦ã„ã‚‹MapReduceã¨ã„ã†æŠ€è¡“ã‚’ã€ã‚ªãƒ¼ãƒ—ンソースã¨ã—ã¦å®Ÿç¾ã™ã‚‹ãŸã‚ã«é–‹ç™ºã•ã‚ŒãŸJavaベースã®ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ã§ã™ã€‚開発ãŒå§‹ã¾ã£ãŸã®ã¯2005å¹´é ƒã§ã€å½“時Yahoo!ã«æ‰€å±žã—ç¾åœ¨ã¯Clouderaã«æ‰€å±žã™ã‚‹Doug Cuttingæ°ãŒä¸å¿ƒã¨ãªã£ã¦é€²ã‚ã¦ãã¾ã—ãŸã€‚ HadoopãŒå®Ÿç¾ã™ã‚‹MapReduce処ç†ã¨ã¯ã€ç°¡å˜ã«ã„ãˆã°å¤§é‡ã®ãƒ‡ãƒ¼ã‚¿ã‚’å°ã•ã分割ã—ã¦å¤šæ•°ã®ãƒŽãƒ¼ãƒ‰ã«å‰²ã‚Šå½“ã¦ï¼ˆMap処ç†ï¼‰ã€å„ノードã§å‡¦ç†ã‚’è¡Œã£ãŸã‚‰ãれを集約ã—ã¦çµæžœã‚’出ã™ï¼ˆReduce処ç†ï¼‰ã€ã¨ã„ã†åˆ†æ•£å‡¦ç†ã®æ–¹æ³•ã§ã™ã€‚数テラãƒã‚¤ãƒˆã«ã‚‚ãŠã‚ˆã¶å¤§å®¹é‡ã®ãƒ‡ãƒ¼ã‚¿ã‚’高速ã‹ã¤ä½Žã‚³ã‚¹ãƒˆã«åˆ†æ•£å‡¦ç†ã™ã‚‹æ–¹æ³•ã¨ã—ã¦æ³¨ç›®ã‚’集ã‚ã¦ã„ã¾ã™ã€‚ ニューヨークã§Hadoop WorldãŒé–‹å‚¬ã•ã‚Œã‚‹ ãã®Hadoopã®ã‚«ãƒ³ãƒ•ã‚¡ãƒ¬ãƒ³ã‚¹ã€ŒHadoop World：NY 2009ã€ãŒ10月2æ—¥ã«ãƒ‹ãƒ¥ãƒ¼ãƒ¨ãƒ¼ã‚¯ã§
優良ä¼æ¥ã¯ãªãœHadoopã«èµ°ã‚‹ã®ã‹
ã¡ãªã¿ã«ã€ã“ã®åˆ†æžã®ãŸã‚ã«å¿…è¦ã¨ã•ã‚Œã‚‹MapReduceã®ã‚³ãƒ¼ãƒ‰ã§ã‚ã‚‹ãŒã€ãã®ã‚µã‚¤ã‚ºã¯ã‚ãšã‹20ステップã ã¨ã„ã†ã€‚Yahoo!ã®ãƒ—レゼンテーターã§ã‚ã‚‹ã€ã‚¨ãƒªãƒƒã‚¯ãƒ»ãƒãƒ«ãƒ‡ã‚·ãƒ¥ãƒã‚¤ãƒ©ãƒ¼æ°ã«ã‚ˆã‚‹ã¨ã€ãŸã¨ãˆçµŒé¨“ã®æµ…ã„エンジニアã§ã‚ã£ã¦ã‚‚ã€MapReduceã«ã‚ˆã‚‹ãƒ—ãƒã‚°ãƒ©ãƒŸãƒ³ã‚°ã¯å¯èƒ½ã§ã‚ã‚‹ã¨ã•ã‚Œã‚‹ã€‚ ã¾ãŸã€VISAã®ã‚¸ãƒ§ãƒ¼ãƒ»ã‚«ãƒ‹ãƒ³ã‚¬ãƒ æ°ã‹ã‚‰ã‚‚ã€è²´é‡ãªãƒ‡ãƒ¼ã‚¿ãŒæä¾›ã•ã‚Œã¦ã„ãŸã®ã§ä»¥ä¸‹ã«ç´¹ä»‹ã™ã‚‹ã€‚åŒç¤¾ã§ã¯ã€1æ—¥ã«1億トランザクションãŒç™ºç”Ÿã™ã‚‹ãŸã‚ã€2å¹´é–“ã§700å„„å¼·ã®ãƒˆãƒ©ãƒ³ã‚¶ã‚¯ã‚·ãƒ§ãƒ³ãƒã‚°ãŒè“„ç©ã•ã‚Œã€ãã®ãƒ‡ãƒ¼ã‚¿é‡ã¯36テラãƒã‚¤ãƒˆã«è‡³ã‚‹ã¨ã„ã†ã€‚ã“ã†ã—ãŸã‚¹ã‚±ãƒ¼ãƒ«ã®ãƒ‡ãƒ¼ã‚¿ã‚’ã€å¾“æ¥ã®RDBを用ã„ã¦åˆ†æžã™ã‚‹ã«ã¯ã€ç´„1カ月ã®æ™‚é–“ãŒå¿…è¦ã¨ã•ã‚Œã¦ããŸãŒã€Hadoopを用ã„ã‚‹ã“ã¨ã§13分ã«çŸç¸®ã•ã‚ŒãŸã¨ã„ã†ã€‚ ã“ã‚Œã¾ã§ã¯ã€Yahoo!ã«ã—ã‚VISAã«ã—ã‚ã€è†¨å¤§ãªãƒ‡ãƒ¼ã‚¿ã‚’RDBã«æŠ¼ã—込むã»ã‹ã«æ–¹æ³•ã¯ãªãã€ãã®åˆ†æžã«æ•°å日をè¦ã™ã‚‹
2009年上期未è¸æœ¬ä½“ã«æŽ¡æŠžã•ã‚Œã¾ã—㟠- Cutting-Edge Programming
ã“ã“ã®ã¨ã“ã‚ã€æœªè¸ã®å ±å‘Šä¼šã«å‚åŠ ã—ãŸã‚Šã€ã„ã¾ã¾ã§æ‰±ã‚ãªã‹ã£ãŸã‚ˆã†ãªæŠ€è¡“ã®ã‚¨ãƒ³ãƒˆãƒªãŒå¢—ãˆãŸç†ç”±ã§ã™ã€‚ 未è¸æœ¬ä½“ã¨ã„ã†ã®ã¯ã€ç‹¬ç«‹è¡Œæ”¿æ³•äººæƒ…å ±å‡¦ç†æ©Ÿæ§‹(IPA)ãŒåŠå¹´ã”ã¨ã«å…¬å‹Ÿã—ã¦ã„ã‚‹ã‚‚ã®ã§ã€æœªè¸ã«æŽ¡æŠžã•ã‚Œã‚‹ã¨ãã®ãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆã¯æ¥ç•Œã®å…ˆé”(PM)ã®æŒ‡å°Žã‚’å—ã‘ãªãŒã‚‰ã€æ案ã—ãŸã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ã‚’開発ã™ã‚‹ã“ã¨ã§ãã®ãªã‹ã§ã‚‚一æ¡ã‚Šã®ã‚¹ãƒ¼ãƒ‘ークリエータを目指ã™ã“ã¨ã«ãªã‚Šã¾ã™ã€‚ ã§ã¯ä½•ã‚’開発ã™ã‚‹ã‹ã¨ã„ã†ã¨ã€ã‚‚ã†ã™ã§ã«æœªè¸ã®å…¬å¼ãƒšãƒ¼ã‚¸ã§å…¬é–‹ã•ã‚Œã¦ã„ã‚‹ã®ã§ã™ãŒã€ã€ŒMapReduce汎用化ã®ãŸã‚ã®DSL基盤・実行基盤ã®é–‹ç™ºã€ã¨ã„ã†ãƒ†ãƒ¼ãƒžã«ãªã£ã¦ã„ã¾ã™ã€‚ã“ã‚Œã ã‘ã ã¨ä½•ã‹ã‚ã‹ã‚‰ãªã„ã§ã™ãŒã€ç°¡å˜ã«èª¬æ˜Žã™ã‚‹ã¨ã€ MapReduceã‚’ã¿ã‚“ãªãŒç°¡å˜ã«ä½¿ãˆã‚‹ãŸã‚ã«ã€Hudson & Hadoopã«ã‚ˆã‚‹å®Ÿè¡Œç®¡ç†åŸºç›¤ã¨JRuby & Ruby DSLã«ã‚ˆã‚‹ãƒ—ãƒã‚°ãƒ©ãƒ フレームワークを開発ã—ã¾ã™ ã¨ã„ã†ã“ã¨ã§ã™ã€‚ ã“ã‚Œã‹ã‚‰ã¯ã€ã“ã®ãƒ–ãƒã‚°ã§
Steve Krenzel - 18. Distributed Computing: MapReduce and HashFold
This is a post about HashFold, a theoretical framework (for now) that is suited to performing the same types of distributed computation that MapReduce can perform, but improves upon MapReduce both in runtime and memory usage. This is quite a claim so I'm putting the idea out there for public scrutiny. If you're familiar with MapReduce, HashFold does away with sorting values and the concept of a co
「MapReduceã€ã‚’サãƒãƒ¼ãƒˆã—ã€é«˜é€Ÿãƒ»å®‰ä¾¡ã‚’実ç¾ã—ãŸDWH
「MapReduceã€ã‚’サãƒãƒ¼ãƒˆã—ã€é«˜é€Ÿãƒ»å®‰ä¾¡ã‚’実ç¾ã—ãŸDWH：180万ドルã§100テラãƒã‚¤ãƒˆã‚¯ãƒ©ã‚¹ã®DWHã‚’å®Ÿç¾ ç¾åœ¨ã»ã¨ã‚“ã©ã®ä¼æ¥ã§ã¯ã€ä¼æ¥å†…データã®æ€¥å¢—ã«é ã‚’æ‚©ã¾ã›ã¦ã„る。åŒæ™‚ã«ãƒ‡ãƒ¼ã‚¿ã‚’集約ã—ã€åˆ†æžã™ã‚‹é‡è¦æ€§ã‚‚増ã—ã¦ã„る。従ã£ã¦ã€æ€¥å¢—ã—ã¦ã„るデータを有効活用ã™ã‚‹ãŸã‚ã«ã€DWH（データウェアãƒã‚¦ã‚¹ï¼‰ã‚„BI（ビジãƒã‚¹ã‚¤ãƒ³ãƒ†ãƒªã‚¸ã‚§ãƒ³ã‚¹ï¼‰ãªã©ã®ãƒ„ールを導入ã™ã‚‹ä¼æ¥ã‚‚多ã„。ãŸã ã—ã€åˆ†æžåŸºã¨ãªã‚‹ãƒ‡ãƒ¼ã‚¿ãŒæ€¥å¢—ã—ã¦ã„ã‚‹ãŸã‚ã«ã€DWHã‚„BIã®å‡¦ç†é€Ÿåº¦ãŒé…ã„ã¨ã‚¯ã‚¨ãƒªã«å¯¾ã™ã‚‹å‡¦ç†ã«æ™‚é–“ãŒã‹ã‹ã‚Šã€å®Ÿç”¨é¢ã§å•é¡ŒãŒç”Ÿã˜ã¦ã„ã‚‹ã®ã‚‚事実ã 。ã“ã®ã“ã¨ã‹ã‚‰ã€å„DWHベンダã¯ã•ã¾ã–ã¾ãªæ‰‹æ³•ã§ã‚¯ã‚¨ãƒªã®å‡¦ç†é€Ÿåº¦å‘上を目指ã—ã¦ã„る。ãã®ä¸ã§ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ã®ä¸¦åˆ—処ç†ã ã‘ã§é«˜é€ŸDWHã‚’æä¾›ã—ã¦ã„ã‚‹ã®ãŒç±³ã‚°ãƒªãƒ¼ãƒ³ãƒ—ラムã 。今回ã¯ã€ç±³ã‚°ãƒªãƒ¼ãƒ³ãƒ—ラムã§ã‚¢ã‚¸ã‚¢ãƒ‘シフィック地域担当副社長を務ã‚ã‚‹ã‚ース・ãƒãƒƒã‚¸ï¼ˆKeith Budge）æ°ã«è©±ã‚’
Groovin' High:map reduceã§æ„Ÿã˜ã‚‹æœªæ¥ - livedoor Blog（ブãƒã‚°ï¼‰
「ã—らãªãƒ¼ã„ã€ã£ã¦äººã«ã€ä½•äººã‹ä¼šã£ãŸã®ã§ã€ã¡ã‚‡ã£ã¨è§£èª¬ã—ãªãŒã‚‰è©±ã‚’ã™ã™ã‚ã¾ã™ã€‚ ã„ã¤ã‚‚大é‡ã®ãƒ‡ãƒ¼ã‚¿ã‚’処ç†ã—ã¦ã„ã‚‹GoogleãŒã€ã€Œãƒãƒƒã‚¯ã‚¨ãƒ³ãƒ‰ã§ã“ã‚“ãªã“ã¨ã‚„ã£ã¦åŠ¹æžœã‚’上ã’ã¦ã¾ã™ã€‚ã€ã¨ã„ã†è«–文を2004å¹´ã«ç™ºè¡¨ã—ãŸã€‚ ãã“ã«æ›¸ã‹ã‚Œã¦ã„ã‚‹ã®ã¯ã€åˆ†æ•£å‡¦ç†ã®ãŸã‚ã®ãƒ‡ã‚¶ã‚¤ãƒ³ãƒ‘ターン「MapReduceã€ã€‚ 大é‡ãƒ‡ãƒ¼ã‚¿ã‚’ã„ãã¤ã‹ã®å˜ä½ã«åˆ†å‰²ã—ã¦ã€ã‚ーã¨å€¤ã«åˆ†å‰²ã™ã‚‹å‡¦ç†ï¼ˆMap）ã¨ã€ãれを集計ã™ã‚‹å‡¦ç†ï¼ˆReduce）ã«åˆ†ã‘る事ã§ã€å‡¦ç†å˜ä½åŒå£«ã®é–¢é€£æ€§ã‚’希薄化ã—ã€ãã®çµæžœåˆ†æ•£ä¸¦åˆ—処ç†ã‚’å¯èƒ½ã«ã™ã‚‹ã€‚ã¨ã„ã†ã‚‚ã®ã€‚ ã“ã®ãƒ‡ã‚¶ã‚¤ãƒ³ãƒ‘ターンを実装ã—ãŸã‚ªãƒ¼ãƒ—ンソース製å“ãŒã€ä¸–ç•Œä¸ã«ã†ã˜ã‚ƒã†ã˜ã‚ƒã‚ã‚‹ã‚“ã ã‘ã©ã€ä¸€ç•ªæœ‰åãªã®ãŒã€Javaã§å®Ÿè£…ã•ã‚ŒãŸã€ŒHadoopã€ã€‚ Javaã§å®Ÿè£…ã•ã‚Œã¦ã‚‹ã‹ã‚‰ã€Javaã§åˆ©ç”¨ã™ã‚Œã°ã„ã„ã‚“ã ã‘ã©ã€ã“ã„ã¤ã«ã¯streamingã¨ã„ã†æ©Ÿèƒ½ã‚‚æ載ã•ã‚Œã¦ã„ã¦ã€ä»–ã®è¨€èªžã§ã‚‚MapReduceを実装ã§ã
1å°ã®ãƒžã‚·ãƒ³ã§ä¸¦åˆ—実行ã™ã‚‹ãŸã‚ã®MapReduceフレームワークを作ã£ã¦ã¿ãŸ - ãã—ã ã®Hatena
ブックマークã®ã‚³ãƒ¡ãƒ³ãƒˆã« 1å°ã§ã‚‚Gbyteå˜ä½ã®ãƒ‡ãƒ¼ã‚¿ã‚’処ç†å¯èƒ½ãªãƒ—ãƒã‚°ãƒ©ãƒ ã‚’ç°¡å˜ã«æ›¸ã‘ã‚‹ã®ã¯å分ãªãƒ¡ãƒªãƒƒãƒˆã ã¨æ€ã†ã€‚ ã¨ã‚ã£ãŸã®ã§ã€ã¤ã„カッã¨ãªã£ã¦ä½œã£ã¦ã¿ãŸã€‚ サンプルã¯å‰ã¨åŒã˜ãã€ã‚¯ãƒ©ã‚¹ãŒJavaソースä¸ã§importã•ã‚Œã¦ã„る回数を数ãˆã‚‹ã‚‚ã®ã€‚ mapreduceメソッドをé©å½“ã«å‘¼ã³å‡ºã›ã°ãŠã£ã‘ーã§ã™ã€‚ ã“ã‚“ãªæ„Ÿã˜ã®çµæžœãƒ•ã‚¡ã‚¤ãƒ«ãŒç”Ÿæˆã•ã‚Œã¾ã—ãŸã€‚ java.io.FileOutputStream 1 java.io.FileReader 1 java.awt.Graphics 1 java.io.Reader 1 java.awt.event.* 3 java.io.BufferedReader 3 書ã„ã¦ã¿ãŸã‚‰å‹•ã„ãŸã¨ã„ã†ãƒ¬ãƒ™ãƒ«ãªã®ã§ã€å®Ÿéš›ã«ä½¿ã†ã«ã¯ã¡ã‚ƒã‚“ã¨ä¾‹å¤–処ç†ã¨ã‹ã‚’ã‚„ã£ã¦ãã ã•ã„。 ソースã¯ã“ã‚“ãªæ„Ÿã˜ import java.io.*; import java.lang.
メッセージã‚ューを使ã£ã¦åˆ†æ•£MapReduceを実装ã™ã‚‹ 2009-02-16 - ãã—ã ã®ã¯ã¦ãª
ã•ã¦ã€JMSã§ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã‚ューも使ãˆã‚‹ã‚ˆã†ã«ãªã£ã¦ã€Hadoopã§MapReduceも試ã—ã¦ã¿ãŸã€‚ãã†ã™ã‚‹ã¨ã‚„ã‚ŠãŸããªã‚‹ã®ãŒãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã‚ューを使ã£ãŸåˆ†æ•£MapReduceã®å®Ÿè£…ã§ã™ã。ã¨ã„ã†ã“ã¨ã§ã€JMSを使ã£ã¦ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã‚ューã«ã‚ˆã‚‹åˆ†æ•£MapReduceã‚’ã‚„ã£ã¦ã¿ã¾ã—ãŸã€‚実際ã«ã¯ãƒãƒ¼ã‚«ãƒ«ã§ã—ã‹å‹•ã‹ãªã„ã®ã§ã™ãŒã€ã‚³ãƒ³ã‚»ãƒ—トモデルã¨ã„ã†ã“ã¨ã§ã€‚ メッセージã‚ューã§éŠã³ãŸã„ã®ã§JMSを試㙠Hadoopã§ã®MapReduceを気軽ã«è©¦ã™ã‚µãƒ³ãƒ—ル Hadoopサンプルã§ä½œã£ãŸã®ã¨åŒã˜ã‚ˆã†ã«ã€ã‚¯ãƒ©ã‚¹ãŒJavaファイルä¸ã§importã•ã‚Œã¦ã„る回数を数ãˆã¦ã¿ã¾ã™ã€‚ 考ãˆæ–¹ã¨ã—ã¦ã€ã¡ã‚‡ã£ã¨å¼·å¼•ã§ã™ãŒã€Googleã‚„Hadoopã®MapReduceã¯åˆ†æ•£ãƒ•ã‚¡ã‚¤ãƒ«ã‚·ã‚¹ãƒ†ãƒ 付ãメッセージã‚ューã¨ã„ãˆã¾ã™ã€‚ã‘ã©å°è¦æ¨¡ã«ã¤ã¤ã¾ã—ãやる分ã«ã¯åˆ†æ•£ãƒ•ã‚¡ã‚¤ãƒ«ã‚·ã‚¹ãƒ†ãƒ ã¯å¿…è¦ãªã„ã®ã§ã€Mapã¨Reduceを分散ã™ã‚‹ã“ã¨ã
ランã‚ング
ランã‚ング
ãŠçŸ¥ã‚‰ã›
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
Jimmy Lin » Data-Intensive Text Processing with MapReduce
agilecatcloud.com
OBB vs AABB - Radium Software Development
twiwt.org - ã“ã®ã‚¦ã‚§ãƒ–サイトã¯è²©å£²ç”¨ã§ã™ï¼Â - twiwt リソースãŠã‚ˆã³æƒ…å ±
IBM Developer
{{#tags}}- {{label}}
{{/tags}}