Google Code Archive - Long-term storage for Google Code Project Hosting.Code Archive Skip to content Google About Google Privacy Terms
クックパッドã¨Hadoop - クックパッド開発者ブãƒã‚°
ã¯ã˜ã‚ã¾ã—ã¦ã€‚今年ã®5月ã«å…¥ç¤¾ã—ãŸå‹é–“@ã•ãŒã™ãƒãƒ¼ãƒ ã§ã™ã€‚ 入社ã—ã¦ã‹ã‚‰ã¯ã€ãªã‹ãªã‹å¤§å¤‰ãªã“ã¨ã‚‚多ã„ã§ã™ãŒã€æœ€è¿‘ã¯ãŠé…’好ããŒé›†ã¾ã£ã¦æœˆæ›œã‹ã‚‰é£²ã¿åˆã† 「å‹é–“会ã€ãªã‚‹ã‚‚ã®ã‚‚発足ã—ã¦ã€ä»•äº‹é¢ã§ã‚‚仕事以外ã®é¢ã§ã‚‚密度ã®é«˜ã„毎日をéŽã”ã—ã¦ã„ã¾ã™ï¼ ã•ã¦ã€åƒ•ã¯ã€Œã•ãŒã™ã€ãƒãƒ¼ãƒ 所属ã¨ã„ã†ã“ã¨ã§ã€æ™®æ®µã¯ãƒ¬ã‚·ãƒ”を「ã•ãŒã™ã€ãƒ¦ãƒ¼ã‚¶ã®æº€è¶³åº¦ã‚’上ã’ã‚‹ãŸã‚ã«ã€ クックパッドã®æ¤œç´¢ã¾ã‚ã‚Šã«ã¤ã„ã¦ã€ã„ã‚ã„ã‚ãªé–‹ç™ºã‚’è¡Œã£ã¦ã„ã¾ã™ã€‚ 一方ã§ã€ãƒ¦ãƒ¼ã‚¶ã®ã€Œã•ãŒã™æ¬²æ±‚ã€ã«ã¤ã„ã¦æ·±ã知るãŸã‚ã«ã€å¤§è¦æ¨¡ãªãƒ‡ãƒ¼ã‚¿è§£æžã‚’è¡Œã„ã€æ¬²æ±‚ã®åˆ†æžã‚’è¡Œã†æ©Ÿä¼šã‚‚増ãˆã¦ãã¾ã—ãŸã€‚ ã¨ã“ã‚ãŒã€ã‚¯ãƒƒã‚¯ãƒ‘ッドã®ãƒã‚°ã¯è†¨å¤§ãªæ•°ãŒã‚ã‚‹ã®ã§ã€ä¸€å£ã®ãƒ‡ãƒ¼ã‚¿è§£æžã¨è¨€ã£ã¦ã‚‚通常ã®ãƒãƒƒãƒå‡¦ç†ã ã¨é–“ã«åˆã‚ãªã„ãŸã‚〠分散処ç†ç’°å¢ƒã®å¿…è¦æ€§ãŒé«˜ã¾ã£ã¦ãã¾ã—ãŸã€‚ ãã“ã§ã€ã¾ãšã¯æ‰‹è»½ã«è©¦ã›ã‚‹åˆ†æ•£å‡¦ç†ã®çŽ‹é“ã¨ã„ã†ã“ã¨ã§ã€æœ€è¿‘ã§ã¯Hadoopを使ã£ãŸãƒ‡ãƒ¼ã‚¿è§£æžç’°å¢ƒã‚’æ•´å‚™ã—ã¦ã„ã¾ã™ã€‚
AmazonEC2 - HADOOP2 - Apache Software Foundation
Running Hadoop on Amazon EC2 Amazon EC2 (Elastic Compute Cloud) is a computing service. One allocates a set of hosts, and runs one's application on them, then, when done, de-allocates the hosts. Billing is hourly per host. Thus EC2 permits one to deploy Hadoop on a cluster without having to own and operate that cluster, but rather renting it on an hourly basis. If you run Hadoop on EC2 you might c
ec2ã§hadoopã—ã¦ã¿ãŸã‚ˆ - id:lopnor
ã¨ã„ã†ã‹http://codezine.jp/article/detail/2841ã‚’ã¾ã‚“ã¾ã‚³ãƒ”ペã—ãŸã ã‘ãªã‚“ã§ã™ã‘ã©ã。 ec2ã®ã‚¢ã‚«ã‚¦ãƒ³ãƒˆæŒã£ã¦ã‚‹ã—使ã„ã¾ãã£ã¦ã‚‹ã‚ˆãƒ¼ã€ã¨ã„ã†å‰æã§ã€‚ 飛ã°ã—ãªãŒã‚‰èªã‚“ã§å®Ÿè¡Œã—ãŸã‚‰ã“ã‘ã¦æ°—ã¥ã„ãŸã‚“ã ã‘ã©ã€ ec2-add-keypair gsg-keypairã¯gsg-keypairã¨ã„ã†åå‰æ±ºã‚打ã¡ãªã®ã§ã¡ã‚ƒã‚“ã¨ã‚„ã‚Šã¾ã—ょã†ã€‚ EC2_KEYDIR=`dirname "$EC2_PRIVATE_KEY"`ã£ã¦ãªã£ã¦ã‚‹ã®ã§ãã“ã«id_rsa-gsg-keypairã¨ã„ã†åå‰ã§ä¿å˜ ./hadoop-ec2 launch-cluster test-cluster 2ã¨ã™ã‚‹ã¨ã¡ã‚ƒã‚“ã¨ç«‹ã¡ä¸ŠãŒã£ãŸ ./hadoop-ec2 login test-clusterã§ãƒã‚°ã‚¤ãƒ³ã€‚ ãŠã‚Œjava詳ã—ããªã„ã‚“ã§ã‚ˆãã‚ã‹ã‚“ãªã‹ã£ãŸã‚“ã ã‘㩠記事ã«æ›¸ã„ã¦ã‚ã‚‹ jar hado
blogeyeã®å®Ÿè£…ã«å¦ã¶ã€Amazon EC2/S3ã§ã®Hadoop活用術
ã¯ã˜ã‚ã« ã“ã‚“ã«ã¡ã¯ã€‚Hadoop連載 第4回ã¯å¤ªç”°ã•ã‚“ã«ä»£ã‚ã£ã¦å¤§å€‰ãŒæ‹…当ã—ã¾ã™ã€‚ ã“ã‚Œã¾ã§ã®é€£è¼‰ã§ã€Hadoopã«ã‚ˆã‚‹ãƒ‡ãƒ¼ã‚¿å‡¦ç†ã®æ¦‚ç•¥ã«ã¤ã„ã¦ã¯ç†è§£ã•ã‚Œã¦ã„ã‚‹ã¨æ€ã„ã¾ã™ã€‚今回ã¯Hadoopを利用ã—ãŸã‚·ã‚¹ãƒ†ãƒ ã®å®Ÿä¾‹ã¨ã„ã†ã“ã¨ã§ã€ãƒ–ãƒã‚°åˆ†æžã‚’è¡Œã†ã€Œblogeyeã€ã‚·ã‚¹ãƒ†ãƒ ã®æ¦‚ç•¥ã¨ã€ãã®ä¸ã§ã®Hadoop利用法を紹介ã—ã¾ã™ã€‚ ã¾ãŸã€blogeyeã¯AmazonãŒæä¾›ã—ã¦ã„ã‚‹EC2（レンタルサーãƒï¼‰ã€S3（ストレージ）をHadoopã¨çµ„ã¿åˆã‚ã›ã¦åˆ©ç”¨ã—ã¦ã„ã‚‹ã®ã§ã€ãã®è¾ºã‚Šã®å°Žå…¥æ–¹æ³•ã«ã¤ã„ã¦ã‚‚紹介ã—ã¾ã™ã€‚ ã“ã‚Œã¾ã§ã®é€£è¼‰ Hadoopã€hBaseã§æ§‹ç¯‰ã™ã‚‹å¤§è¦æ¨¡åˆ†æ•£ãƒ‡ãƒ¼ã‚¿å‡¦ç†ã‚·ã‚¹ãƒ†ãƒ Hadoopã®ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã¨ã‚µãƒ³ãƒ—ルプãƒã‚°ãƒ©ãƒ ã®å®Ÿè¡Œ 複数マシンã¸Hadoopをインストールã™ã‚‹ blogeyeã¨ã¯ 「blogeyeã€ï¼ˆãƒ–ãƒã‚°ã‚¢ã‚¤ï¼‰ã¯æ—¥æœ¬èªžã®ãƒ–ãƒã‚°ã‚’クãƒãƒ¼ãƒ«ã€ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ã«åˆ†æžã—ã¦ã€æµè¡Œèªžã¨æ€ã‚
hadoop-ec2ã§ã‚¢ã‚¯ã‚»ã‚¹è§£æžã—ã¦ã¿ãŸã‚ˆï¼
å‰å›žï¼ˆhttp://d.hatena.ne.jp/lopnor/20080831/1220183688）サンプルプãƒã‚°ãƒ©ãƒ ã‚’å‹•ã‹ã™ã¨ã“ã‚ã¾ã§è¡Œãã¾ã—ãŸã‘ã©ã€ä»Šå›žã¯ä¾‹ã®id:naoyaã®hadoop streamingã§ã‚¢ã‚¯ã‚»ã‚¹è§£æžï¼ˆhttp://d.hatena.ne.jp/naoya/20080513/1210684438）ã™ã‚‹ã®ã‚’ã‚„ã£ã¦ã¿ã¾ã™ã€‚ã¨ã„ã†ã‹ãªã‚“ã¨ã‹ãŸã©ã‚Šç€ã„ãŸã®ã§ã¾ã¨ã‚を書ãã¾ã™ã€‚ *ec2ã®ä½¿ã„æ–¹ id:rx7ã•ã‚“ãŒã¨ã¦ã‚‚ä¸å¯§ã«èª¬æ˜Žã•ã‚Œã¦ã„る資料（http://d.hatena.ne.jp/rx7/20080528/p1）ãŒã‚ã‚‹ã®ã§ã€ãã¡ã‚‰ã‚’èªã‚ã°å®Œç’§ã ã¨æ€ã„ã¾ã™ã€‚僕もã“ã‚Œã§ec2ãŒä½¿ãˆã‚‹ã‚ˆã†ã«ãªã‚Šã¾ã—ãŸã€‚ *hadoop-ec2ã®ä½¿ã„æ–¹ https://codezine.jp/article/detail/2841ãŒã‚¤ãƒ³ãƒˆãƒãƒ€ã‚¯ã‚·ãƒ§ãƒ³ã€‚http://d.hatena.n
楽天版MapReduce・Hadoopã¯Rubyを活用 ï¼ ï¼ IT
2008/12/01 楽天ã¯11月29æ—¥ã€æ±äº¬ãƒ»å“å·ã®æœ¬ç¤¾ã§é–‹å‚¬ã—ãŸæŠ€è¡“系イベント「楽天テクノãƒã‚¸ãƒ¼ã‚«ãƒ³ãƒ•ã‚¡ãƒ¬ãƒ³ã‚¹2008ã€ã«ãŠã„ã¦ã€è¿‘ã„å°†æ¥ã«åŒç¤¾ã®Eã‚³ãƒžãƒ¼ã‚¹ã‚µãƒ¼ãƒ“ã‚¹ã€Œæ¥½å¤©å¸‚å ´ã€ã‚’支ãˆã‚‹è¨ˆç”»ãŒã‚ã‚‹Rubyベースã®å¤§è¦æ¨¡åˆ†æ•£å‡¦ç†æŠ€è¡“「ROMAã€ï¼ˆãƒãƒ¼ãƒžï¼‰ã¨ã€Œfairyã€ï¼ˆãƒ•ã‚§ã‚¢ãƒªãƒ¼ï¼‰ã«ã¤ã„ã¦ã€ãã®æ¦‚è¦ã‚’明らã‹ã«ã—ãŸã€‚ レコメンデーションã®å‡¦ç†è‡ªä½“ã¯ã‚·ãƒ³ãƒ—ル æ¥½å¤©å¸‚å ´ã§ã¯ç¾åœ¨ã€2600万点ã®å•†å“ã‚’å–り扱ã„ã€4200万人ã®ä¼šå“¡ã«å¯¾ã—ã¦ã‚µãƒ¼ãƒ“スをæä¾›ã—ã¦ã„る。ã“ã®è¦æ¨¡ã®ä¼šå“¡æ•°ãƒ»å•†å“点数ã§ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³ï¼ˆå•†å“ã®æŽ¨è–¦ï¼‰ã‚’è¡Œã†ã®ã¯å®¹æ˜“ã§ã¯ãªã„。 ※記事åˆå‡ºæ™‚ã«æ¥½å¤©å¸‚å ´ã®ä¼šå“¡æ•°ã‚’4800万人ã¨ã—ã¦ã‚ã‚Šã¾ã—ãŸãŒã€ã“ã‚Œã¯æ¥½å¤©ã‚°ãƒ«ãƒ¼ãƒ—ã®ã‚µãƒ¼ãƒ“ス利用者全体ã®æ•°å—ã§ã—ãŸã€‚æ¥½å¤©å¸‚å ´ã®ä¼šå“¡æ•°ã¯æ£ã—ãã¯4200万人ã¨ã®ã“ã¨ã§ã™ã€‚ãŠè©«ã³ã—ã¦è¨‚æ£ã„ãŸã—ã¾ã™ã€‚ レコメンデーションã®ä»•çµ„ã¿ã¨ã—ã¦åŒç¤¾ã¯ã€ä¸€èˆ¬çš„ã§ã‚·ãƒ³ãƒ—ルãªã‚¢
InfoQ: Cascading -- Hadoop MapReduceå‘ã‘データ処ç†ç³»API
ã‚ãªãŸã«ã¨ã£ã¦é‡è¦ãªãƒˆãƒ”ックやåŒåƒšã®æœ€æ–°æƒ…å ±ã‚’å…¥æ‰‹ã—ã¾ã—ょã†æœ€æ–°ã®æ´žå¯Ÿã¨ãƒˆãƒ¬ãƒ³ãƒ‰ã«é–¢ã™ã‚‹æœ€æ–°æƒ…å ±ã‚’å³åº§ã«å—ã‘å–ã‚Šã¾ã—ょã†ã€‚ 継続的ãªå¦ç¿’ã®ãŸã‚ã«ã€ç„¡æ–™ã®ãƒªã‚½ãƒ¼ã‚¹ã«æ‰‹è»½ã«ã‚¢ã‚¯ã‚»ã‚¹ã—ã¾ã—ょã†ãƒŸãƒ‹ãƒ–ックã€ãƒˆãƒ©ãƒ³ã‚¹ã‚¯ãƒªãƒ—ト付ãå‹•ç”»ã€ãŠã‚ˆã³ãƒˆãƒ¬ãƒ¼ãƒ‹ãƒ³ã‚°æ•™æ。 記事をä¿å˜ã—ã¦ã€ã„ã¤ã§ã‚‚èªã‚€ã“ã¨ãŒã§ãã¾ã™è¨˜äº‹ã‚’ブックマークã—ã¦ã€æº–å‚™ãŒã§ããŸã‚‰ã„ã¤ã§ã‚‚èªã‚ã¾ã™ã€‚
Thread Base MapReduce - moratorium
Thread Base MapReduce 2007-01-09 (Tue) 0:29 Uncategorized 並列計算フレームワークを作ã£ã¦ã„る人を見ã¦ãŸã‚‰è‡ªåˆ†ã‚‚ãªã‚“ã‹ä½œã‚ŠãŸããªã£ã¦æ¥ãŸã®ã§ã€ã‚¹ãƒ¬ãƒƒãƒ‰ãƒ™ãƒ¼ã‚¹ã§Googleã®MapReduceを真似ã¦è¦‹ã¾ã—ãŸã€‚1マシン用ã®MapReduceã¨ã„ã£ãŸæ‰€ã§ã™ã‹ã。 以下ã«ã‚½ãƒ¼ã‚¹ã‚³ãƒ¼ãƒ‰ãŒæœ‰ã‚Šã¾ã™ã€‚é©å½“ã«ç…®ã‚‹ãªã‚Šç„¼ããªã‚Šã—ã¦ãã ã•ã„。 ソースコード ワードカウントãŒä»¥ä¸‹ã®ã‚ˆã†ãªã‚³ãƒ¼ãƒ‰ã§è¨˜è¿°ã§ãã¾ã™ã€‚ [code] class WordCounter : public Mapper { public: virtual void Map(const MapInput& input) { string text = input.value(); istringstream iss(text); string word; while
Simple MapReduce in Ruby
View more posts Here’s a simple version of the MapReduce framework presented in the now-famous Google paper by Dean and Ghemawat. My version of MapReduce is not intended as a usable high-performance framework, but rather as a learning tool. My goal is twofold: first, to learn to write algorithms in distributed/parallel MapReduce style. Second, to see how simply these concepts can be expressed in R
MapReduce
Jeffrey Dean and Sanjay Ghemawat of Google have written a paper about a method of processing large data sets they call MapReduce. Many will be familiar with the functional programming constructs of map and reduce. Map applies a function against each element of a list to get a transformed version of the list. For example, in Python, map(chr, [97,98,99]) transforms a list of three numbers into a l
Google Scalability Conference Trip Report: MapReduce, BigTable, and Other Distributed System Abstractions for Handling Large Datasets - Dare Obasanjo's weblog
Dare Obasanjo's weblog "You can buy cars but you can't buy respect in the hood" - Curtis Jackson Navigation for Google Scalability Conference Trip Report: MapReduce, BigTable, and Other Distributed System Abstractions for Handling Large Datasets - Dare Obasanjo's weblog Content Sidebar Footer These are my notes from the keynote session MapReduce, BigTable, and Other Distributed System Abstractions
JVM Under the Hood : Locking in IBM J9 VM
Welcome to the IBM TechXchange Community Connect and engage to get answers, discuss best practices, and continually learn more about IBM solutions. Become a member Register and get hands-on at a Dev Day event Register to get an exclusive access to the latest IBM tech and the experts who know how to get the most out of it. Register now Registration is Open! IBM TechXchange 2025 Join us October 6-9
Welcome to Apache™ Hadoop®!
A wide variety of companies and organizations use Hadoop for both research and production. Users are encouraged to add themselves to the Hadoop PoweredBy wiki page. 8 Aug 2018: Release 3.1.1 available This is the first stable release of Apache Hadoop 3.1 line. It contains 435 bug fixes, improvements and enhancements since 3.1.0 Users are encouraged to read the overview of major changes since 3.1.0
Why Should You Care About MapReduce?
This article explains MapReduce in enough detail that implementers can apply it to their own requriements. MapReduce is a distributed programming model intended for processing massive amounts of data in large clusters, developed by Jeffrey Dean and Sanjay Ghemawat at Google. What is MapReduce? MapReduce is a distributed programming model intended for processing massive amounts of data in large clu
ランã‚ング
ランã‚ング
メンテナンス
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
Lean & Mean Tokyo Cabinet Recipes
mrtoolkit - Google Code
MapReduce:Simplified Data Processing on Large Clusters
http://skynet.rubyforge.org/doc/index.html
http://www.databasecolumn.com/2008/01/mapreduce-a-major-step-back.html
{{#tags}}- {{label}}
{{/tags}}