ソーシャルアプリã§ã® Amazon Elastic MapReduce 活用事例
The document discusses Amazon Elastic MapReduce, which allows users to run Hadoop jobs on Amazon Web Services infrastructure. It describes how MapReduce works, provides an example of analyzing Apache log files using MapReduce, and discusses how to run and debug MapReduce jobs on Amazon Elastic MapReduce. Finally, it mentions how MapReduce jobs can be scheduled regularly using cron jobs to periodic
yebo blog: Googleã¯æ—¢ã«MapReduceを使ã£ã¦ã„ãªã„
2010/09/13 Googleã¯æ—¢ã«MapReduceを使ã£ã¦ã„ãªã„ Googleã®æ–°ã—ã„検索基盤「Caffeineã€ã§ã¯ã€MapReduceを使ã£ã¦ã„ãªã„ãã†ã [register]。新ã—ã„検索基盤ã¯GFS (Google File System) をオーãƒãƒ¼ãƒ›ãƒ¼ãƒ«ã—ãŸGFS2を活用ã—ã¦ãŠã‚Šã€åˆ†æ•£ãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹BigTableã«ç›´æŽ¥ã‚¤ãƒ³ãƒ‡ã‚¯ã‚·ãƒ³ã‚°ã‚’è¡Œã†ãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹é§†å‹•ã®æ–¹å¼ã«å¤‰ã‚ã£ã¦ã„る。ã“ã®æ¤œç´¢åŸºç›¤ã‚’「Colossusã€ã¨å‘¼ã‚“ã§ã„ã‚‹ãã†ã 。インデックスã®ä½œæˆæ™‚é–“ã‚’çŸç¸®ã™ã‚‹ã“ã¨ã§(MapReduceã«ã‚ˆã‚‹å‡¦ç†ãŒä¸è¦)ã€æ¤œç´¢ã®ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ 性を増ã™ã®ã ã‚ã†ã€‚Microsoftã€Facebookã€Yahoo!ãŒã¾ã MapReduceã®ã‚ªãƒ¼ãƒ—ンソース版ã¨ã‚‚ã„ãˆã‚‹ã€ŒHadoopã€ã‚’使ã£ã¦ã„ã‚‹ã“ã¨ã‚’考ãˆã‚‹ã¨Googleã¯å…ˆã«é€²ã‚“ã§ã„る。ã“ã®æˆæžœãŒã€ŒGoogle Instantã€ã«ç¹‹ãŒã£ã¦ã„ã‚‹ã®ã ã‚ã†
MapReduceã¨ãƒ‘ラレルRDBã§ãƒ™ãƒ³ãƒãƒžãƒ¼ã‚¯å¯¾æ±ºã€å‹è€…ã¯ãªã‚“ã¨RDBï¼
大é‡ã®ãƒ‡ãƒ¼ã‚¿ã‚’処ç†ã™ã‚‹æ‰‹æ³•ã¨ã—ã¦ç™»å ´ã—ãŸMapReduce。クラウドã«å¯¾å¿œã—ãŸåˆ†æ•£å‡¦ç†ã®å®šç•ªã¨ã—ã¦è©±é¡Œã«ä¸Šã‚‹ã“ã¨ãŒå¢—ãˆã¦ãã¾ã—ãŸã€‚ MapReduceã¯ã€å¤§é‡ã®ãƒ‡ãƒ¼ã‚¿ã‚’分割ã—ã€åˆ†å‰²ã—ãŸãƒ‡ãƒ¼ã‚¿ã‚’分散ã—ãŸãƒŽãƒ¼ãƒ‰ã«æŠ•ã’ã¦ãƒŽãƒ¼ãƒ‰ã”ã¨ã«å‡¦ç†ã‚’実行ã€çµæžœã‚’集約ã—ã¦æœ€çµ‚çš„ãªç”ãˆã‚’求ã‚ã‚‹ã€ã¨ã„ã£ãŸæ‰‹æ³•ã§ã™ã€‚ ã—ã‹ã—MapReduceãŒç™»å ´ã™ã‚‹ä»¥å‰ã‹ã‚‰å•†ç”¨ãƒ¬ãƒ™ãƒ«ã§ä½¿ã‚ã‚Œã¦ã„ãŸåˆ†æ•£å‡¦ç†æ‰‹æ³•ãŒã‚ã‚Šã¾ã™ã€‚データを分散ã—ãŸãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹ã«æ ¼ç´ã—処ç†ã‚’è¡Œã†ãƒ‘ラレル・リレーショナルデータベース（パラレルRDB）ãŒãã®1ã¤ã§ã™ã€‚ パラレルRDBã¯ã€ãƒ‡ãƒ¼ã‚¿ã‚’複数ã®ãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹ã«åˆ†æ•£ã—ã¦é…ç½®ã€ãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹ã”ã¨ã«å‡¦ç†ã‚’è¡Œã„ã€çµæžœã‚’求ã‚る手法ã§ã™ã€‚ä¸å¤®ã«å…±æœ‰ãƒ¡ãƒ¢ãƒªã‚’é…ç½®ã™ã‚‹ãªã©ã®æ–¹æ³•ã§åˆ†æ•£ã—ãŸãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹åŒå£«ã®é€£æºã‚’è¡Œã†ã“ã¨ãŒä¸€èˆ¬çš„ã§ã™ã€‚ ã§ã¯ãƒ‘ラレル・リレーショナルデータベースã¯MapReduceよりé…ã„ã®ã‹ï¼Ÿ 劣るã®ã‹ï¼Ÿ 両者
Amazon Elastic MapReduceã§PHPも使ãˆã‚‹ã‹è©¦ã—ã¦ã¿ãŸ - Stellaqua - ＴＯï¼ã®æŠ€è¡“日記
AmazonãŒEC2上ã§Hadoopã«ã‚ˆã‚‹MapReduceを実行ã§ãるサービスを出ã—ã¦ããŸã‚ˆã†ã§ã™ã。 →Amazon EMR - Amazon Web Services →http://kzk9.net/blog/2009/04/reviewing_amazon_elastic_map_reduce.html å‰å›žã®è¨˜äº‹ã§ã¯ã€EC2上ã§Hadoop用ã®ã‚¤ãƒ³ã‚¹ã‚¿ãƒ³ã‚¹ã‚’自分ã§å‹•ä½œã•ã›ã¦ã€ãƒ•ã‚¡ã‚¤ãƒ«ã®è»¢é€ã‚‚事å‰ã«ã‚„ã£ã¦â€¦ã¨è‰²ã€…下準備ãŒå¿…è¦ã ã£ãŸã‚“ã§ã™ãŒã€ã“ã®ã‚µãƒ¼ãƒ“スを使ãˆã°ãƒãƒ¼ã‚«ãƒ«ã®ç’°å¢ƒã‹ã‚‰ã„ããªã‚Šã‚³ãƒžãƒ³ãƒ‰ä¸€ç™ºã§MapReduceを実行ã™ã‚‹äº‹ãŒã§ãるよã†ã«ãªã‚Šã¾ã—ãŸã€‚ ãã‚“ãªã®ã‚’èžã„ãŸã‚‰ã‚„ã£ã¦ã¿ãŸããªã‚‹ã‚ˆãã€ã¨ã„ã†äº‹ã§ã€PHPã®Mapper/ReducerãŒä½¿ãˆã‚‹ã‹è©¦ã—ã¦ã¿ãŸã‹ã£ãŸã®ã§ã€æ—©é€Ÿä½¿ã£ã¦ã¿ã‚‹äº‹ã«ã—ã¾ã—ãŸã€‚ 多段階ã®MapReduceã¯é¢å€’ãªã®ã§ã€ä»Šå›žã¯ä»¥å‰ã®è¨˜äº‹ã§ä½¿ã£ãŸå˜èªžã‚«ã‚¦ãƒ³ãƒˆã®
#appengine ã†ãã£ã€ç§ã®MapReduceã€é…ã™ãŽï¼Ÿ - スティルãƒã‚¦ã‚¹ã®æ›¸åº«ã®æ›¸åº«
Fredã•ã‚“ã¸ã®åˆ¥ä»¶ãƒ¡ãƒ¼ãƒ«ä¸ã§ã€ŒMapper APIé…ã„ãªã〜ã€ã¨ã¡ã‚‡ã£ã¨ã‚°ãƒã‚’ã“ã¼ã—ãŸã‚‰ã€ã€Œãã‚“ãªã¯ãšã¯ãªã„ã€çš„ãªãƒªã‚¢ã‚¯ã‚·ãƒ§ãƒ³ã ã£ãŸã®ã§ã€å…ˆæ—¥ã®ãƒ†ã‚¹ãƒˆçµæžœã‚’ãŠçŸ¥ã‚‰ã›ã—ãŸã¨ã“ã‚ã€ã“ã‚“ãªè¿”事ãŒæ¥ã¾ã—ãŸï¼š You can adjust the processing_rate to be higher. The default of 100 is to prevent you from eating through quota too quickly. In mapreduce.yaml (Python) you add: - name: processing_rate default: {some large number} We missed this in the documentation. Thanks for point this out. We'll fix that. Fredãª
グーグルãŒHadoopã«MapReduce特許ã®åˆ©ç”¨ã‚’許å¯
大è¦æ¨¡åˆ†æ•£å‡¦ç†ã®æŠ€è¡“ã¨ã—ã¦çŸ¥ã‚‰ã‚Œã‚‹MapReduceã¯ã€ã‚°ãƒ¼ã‚°ãƒ«ãŒæ¤œç´¢ã‚¨ãƒ³ã‚¸ãƒ³ã®åŸºç›¤æŠ€è¡“ã¨ã—ã¦é–‹ç™ºã—ãŸã‚‚ã®ã€‚ãã—ã¦åŒç¤¾ã¯MapReduceã®ç‰¹è¨±ã‚’1月ã«å–å¾—ã—ã¦ã„ã¾ã—ãŸã€‚ グーグルãŒMapReduce特許をå–得。Hadoopã¸ã®å½±éŸ¿ã¯ï¼Ÿ ï¼ Publickey グーグルãŒç‰¹è¨±ã‚’ä¿æœ‰ã—ã¦ã„ã‚‹ã“ã¨ã§MapReduceã®ã‚ªãƒ¼ãƒ—ンソース実装ã§ã‚ã‚‹Hadoopã«å¯¾ã™ã‚‹å½±éŸ¿ãŒå¿ƒé…ã•ã‚Œã¦ã„ã¾ã—ãŸãŒã€Apache Foundationã®å¼è·å£«ãŒã‚°ãƒ¼ã‚°ãƒ«ã®å¼è·å›£ã‹ã‚‰ç‰¹è¨±ã®åˆ©ç”¨è¨±å¯ã‚’å¾—ãŸã“ã¨ã‚’明らã‹ã«ã—ã¾ã—ãŸã€‚ 予想通りã®å±•é–‹ã« 4月23日付ã‘ã§Yahoo!ã®Hadoopãƒãƒ¼ãƒ アーã‚テクトOwen O’Malleyæ°ãŒãƒ¡ãƒ¼ãƒªãƒ³ã‚°ãƒªã‚¹ãƒˆã«æŠ•ç¨¿ã—ãŸã€ŒRe: License for Google's patentã€ã¨ã„ã†ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã€‚ã“ã®ä¸ã§Apache Foundationã®å¼è·å£«ãŒã€ã‚°ãƒ¼ã‚°ãƒ«ã‹ã‚‰ã®ç‰¹è¨±åˆ©ç”¨è¨±å¯ã‚’ä¼
MapReduce - naoyaã®ã¯ã¦ãªãƒ€ã‚¤ã‚¢ãƒªãƒ¼
"MapReduce" 㯠Google ã®ãƒãƒƒã‚¯ã‚¨ãƒ³ãƒ‰ã§åˆ©ç”¨ã•ã‚Œã¦ã„る並列計算システムã§ã™ã€‚検索エンジンã®ã‚¤ãƒ³ãƒ‡ãƒƒã‚¯ã‚¹ä½œæˆã‚’ã¯ã˜ã‚ã¨ã™ã‚‹ã€å¤§è¦æ¨¡ãªå…¥åŠ›ãƒ‡ãƒ¼ã‚¿ã«å¯¾ã™ã‚‹ãƒãƒƒãƒå‡¦ç†ã‚’想定ã—ã¦ä½œã‚‰ã‚ŒãŸã‚·ã‚¹ãƒ†ãƒ ã§ã™ã€‚ MapReduce ã®é¢ç™½ã„ã¨ã“ã‚ã¯ã€map() 㨠reduce() ã¨ã„ã†äºŒã¤ã®é–¢æ•°ã®çµ„ã¿åˆã‚ã›ã‚’定義ã™ã‚‹ã ã‘ã§ã€å¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿ã«å¯¾ã™ã‚‹æ§˜ã€…ãªè¨ˆç®—å•é¡Œã‚’解決ã™ã‚‹ã“ã¨ãŒã§ãる点ã§ã™ã€‚ MapReduce ã®è¨ˆç®—モデル map() ã«ã¯ãã®è¨ˆç®—å•é¡Œã®ãƒ‡ãƒ¼ã‚¿ã¨ã—ã¦ã® key-value ペアãŒæ¬¡ã€…ã«æ¸¡ã£ã¦ãã¾ã™ã€‚map() ã§ã¯ key-value 値ã®ãƒšã‚¢ã‚’ç•°ãªã‚‹è¤‡æ•°ã® key-value ペアã«å¤‰æ›ã—ã¾ã™ã€‚reduce() ã«ã¯ã€map() ã§ä½œã£ãŸ key-value ペアをåŒä¸€ã® key ã§æŸããŸã‚‚ã®ãŒé †ç•ªã«æ¸¡ã£ã¦ãã¾ã™ã€‚ãã® key-values ペアを任æ„ã®å½¢å¼ã«å¤‰æ›ã™ã‚‹ã“ã¨
Amazon Elastic MapReduceを使ã£ã¦ã¿ãŸ - moratorium
Amazon Elastic MapReduceを使ã£ã¦ã¿ãŸ 2009-04-03 (Fri) 3:06 Amazon EC2 連日ã®EC2ãƒã‚¿ã§ã™ã€‚本日ã€Amazonã‹ã‚‰Elastic MapReduceã¨ã„ã†ã‚µãƒ¼ãƒ“スãŒãƒªãƒªãƒ¼ã‚¹ã•ã‚Œã¾ã—ãŸã€‚大è¦æ¨¡ãƒ‡ãƒ¼ã‚¿å‡¦ç†æŠ€è¡“ãŒä¸€æ°—ã«æ°‘é–“ã®æ‰‹ã«ä¸‹ã‚Šã¦ãã‚‹ã€ã¾ã•ã«é©å‘½çš„ãªã‚µãƒ¼ãƒ“スã ã¨æ€ã„ã¾ã™ã€‚ Amazon Elastic MapReduce Amazon ElasticMapReduce 紹介ビデオ With Hadoop, Amazon Adds A Web-Scale Data Processing Engine To Its Cloud Computer by techcrunch.com Elastic MapReduceã¯ã€Googleã®åŸºç›¤æŠ€è¡“ã®ä¸€ã¤ã§ã‚ã‚‹MapReduceを時間å˜ä½èª²é‡‘ã§å®Ÿè¡Œã§ãるサービスã§ã™ã€‚MapReduceã«ã¤ã„ã¦ã¯ä»¥
1
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
Google Code Archive - Long-term storage for Google Code Project Hosting.
連載: IBM Watson Workspace #鬼ã‚㋠アプリケーション開発: 第 7 回: IBM Watson Workspace 㧠AI を利用ã—ãŸã‚¢ãƒ—リ連æºã®å®Ÿç¾ #鬼ã‚㋠解説（å‰ç·¨ï¼‰
{{#tags}}- {{label}}
{{/tags}}