Joins in Hadoop using CompositeInputFormat | Roberto Congiu Home Page
One of the first questions that a 'traditional' ETL engineer asks when learning hadoop is, "How do I do a join ?" For instance, how can we do in hadoop something like querying for the names of all employees who are in a California city: SELECT e.name, c.name from employees e INNER JOIN cities c on e.city_id = c.id AND c.state ='CA' If one dataset is big and the other is small enough that you're su
Mahoutã®ã‚µãƒ³ãƒ—ルプãƒã‚°ãƒ©ãƒŸãƒ³ã‚° - プãƒã‚°ãƒ©ãƒžã§ã‚ã‚ŠãŸã„
ã¡ã‚‡ã£ã¨ç©ºã„ãŸæ™‚é–“ã«Tasteã®å¾Œã‚’継ãオープンソースã®ãƒªã‚³ãƒ¡ãƒ³ãƒ‰ã‚¨ãƒ³ã‚¸ãƒ³ã®Mahoutを触ã£ã¦ã¿ã¾ã—ãŸã€‚ã¾ã 使ã„ã“ãªã›ã¦ã„ãªã„ã§ã™ãŒã€ã‹ãªã‚Šé¢ç™½ãã†ãªã‚¢ãƒ—リã§ã™ã€‚ 今回ã¯Livedoor Clipsã®ãƒ‡ãƒ¼ã‚¿ã‚’使ã£ã¦ã€URLã«å¯¾ã—ã¦ãŠå‹§ã‚ã®ã‚¿ã‚°ã¨ã‚¿ã‚°ã«å¯¾ã—ã¦ãŠå‹§ã‚ã®ã‚¿ã‚°ã‚’出ã—ã¦ã¿ã¾ã—ãŸã€‚ã‚¿ã‚°æƒ…å ±ã‚’ä½¿ã„ã‚„ã™ã„よã†ã«æ•´å½¢ã—ã¦ã„ã‚‹ã®ã§ã€è©³ã—ãã¯ã€Œlivedoor clipsã®ãƒ‡ãƒ¼ã‚¿ã‚’å°‘ã—ã ã‘眺ã‚ã¦ã¿ãŸã€‚ã€ã‚’å‚ç…§ã—ã¦ãã ã•ã„。 import java.io.File; import java.io.FileNotFoundException; import java.util.List; import org.apache.mahout.cf.taste.common.TasteException; import org.apache.mahout.cf.taste.impl.model.file
Log In - Apache Software Foundation
Powered by a free Atlassian Confluence Open Source Project License granted to Apache Software Foundation. Evaluate Confluence today. ÄŒeÅ¡tina Dansk Deutsch Eesti English (UK) English (US) Español Français Ãslenska Italiano Magyar Nederlands Norsk Polski Português Română SlovenÄina Suomi Svenska РуÑÑкий ä¸æ–‡ 日本語 í•œêµì–´ Powered by Atlassian Confluence 7.19.30 Printed by Atlassian Confluence 7.19.30 Report
Facebookã®æ–°ã—ã„リアルタイム解æžã‚·ã‚¹ãƒ†ãƒ ã¨ã¯ï¼Ÿ - nokunoã®æ—¥è¨˜
Facebookã®æ–°ã—ã„リアルタイム解æžã®ã‚·ã‚¹ãƒ†ãƒ ã§ã¯ã€HBaseã§1æ—¥200億件ã®ã‚¤ãƒ™ãƒ³ãƒˆã‚’処ç†ã—ã¦ã„ã‚‹ãã†ã§ã™ã€‚以下ã®è¨˜äº‹ã®ç¿»è¨³ã§ã™ã€‚High Scalability - High Scalability - Facebook’s New Realtime Analytics System: HBase to Process 20 Billion Events Per DayFacebookãŒã¾ãŸã‚„ã£ã¦ãã‚ŒãŸã€‚彼らã¯å·¨å¤§ãªãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ データã®ã‚¹ãƒˆãƒªãƒ¼ãƒ を処ç†ã™ã‚‹ã‚‚ã†1ã¤ã®ã‚·ã‚¹ãƒ†ãƒ を構築ã—ãŸã®ã 。以å‰ã«ã‚‚Facebookã¯ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ãªãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã‚·ã‚¹ãƒ†ãƒ ã‚’HBaseã§æ§‹ç¯‰ã—ã¦ã„ã‚‹(http://highscalability.com/blog/2010/11/16/facebooks-new-real-time-messaging-system-hbase-to-store-135.ht
『モãƒã‚²ãƒ¼ã®å¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿ãƒžã‚¤ãƒ‹ãƒ³ã‚°åŸºç›¤ã«ãŠã‘ã‚‹Hadoop活用ã€ï¼Hadoop Conference Japan 2011ï¼ #hcj2011
『モãƒã‚²ãƒ¼ã®å¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿ãƒžã‚¤ãƒ‹ãƒ³ã‚°åŸºç›¤ã«ãŠã‘ã‚‹Hadoop活用ã€ï¼Hadoop Conference Japan 2011ï¼ #hcj2011 2011/02/22 [登壇後エントリ] ：" 「モãƒã‚²ãƒ¼ã®å¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿ãƒžã‚¤ãƒ‹ãƒ³ã‚°åŸºç›¤ã«ãŠã‘ã‚‹Hadoop活用ã€ï¼Hadoop Conference Japan 2011 #hcj2011 ã§ç™»å£‡ã—ã¦ãã¾ã—㟠" http://d.hatena.ne.jp/hamadakoichi/20110222/p1Read less
Cloudera Blog
In an era where artificial intelligence (AI) is reshaping enterprises across the globe—be it in healthcare, finance, or manufacturing—it’s hard to overstate the transformation that AI has had on businesses, regardless of industry or size. At Cloudera, we recognize the urgent need for bold steps to harness this potential and dramatically accelerate the time to […] Read blog post
Runtime error - Meta Search
Error message : Directory is not found or not writable (DATA_DIR) Directory is not found or not writable (DIFF_DIR) Directory is not found or not writable (BACKUP_DIR) Directory is not found or not writable (CACHE_DIR) Site admin: whitestar Copyright © 2006-2023 whitestar. All Rights Reserved. Icons powered by famfamfam. PukiWiki 1.5.0 Copyright © 2001-2006 PukiWiki Developers Team. License is GPL
Runtime error - Meta Search
Error message : Directory is not found or not writable (DATA_DIR) Directory is not found or not writable (DIFF_DIR) Directory is not found or not writable (BACKUP_DIR) Directory is not found or not writable (CACHE_DIR) Site admin: whitestar Copyright © 2006-2023 whitestar. All Rights Reserved. Icons powered by famfamfam. PukiWiki 1.5.0 Copyright © 2001-2006 PukiWiki Developers Team. License is GPL
Hadoopを使ã„ã“ãªã™(1)
ã¾ãšã€ 1 ã®å…¥åŠ›ãƒ•ã‚¡ã‚¤ãƒ«ã‚’分割ã™ã‚‹æ–¹æ³•ã¯ã€InputFormatクラスã®ã€getSplits関数を上書ãã™ã‚‹ã“ã¨ã§ã€ã‚«ã‚¹ã‚¿ãƒžã‚¤ã‚ºã§ãã¾ã™ã€‚ ã¾ãŸã€ 3 ã®InputSplitã‹ã‚‰ã€Keyã¨Valueを抽出ã™ã‚‹å‡¦ç†ã‚‚ã€InputFormatクラスを通ã˜ã¦ã‚«ã‚¹ã‚¿ãƒžã‚¤ã‚ºã§ãã¾ã™ã€‚ InputFormatã®getRecordReader関数を通ã˜ã¦ã€RecordReaderクラスを生æˆã™ã‚‹ã®ã§ã™ãŒã€ã“ã‚Œã«ä»»æ„ã®RecordReaderクラスを指定ã™ã‚Œã°OKã§ã™ã€‚ 2 ã®Map処ç†ã§ã™ãŒã€ãƒ¦ãƒ¼ã‚¶ãŒæŒ‡å®šã—ãŸMapperクラスã®å‡¦ç†ã‚’実行ã—ã¾ã™ã€‚ Mapperクラスã¯ã€MapRunnerクラスを通ã˜ã¦ã€åˆæœŸåŒ–処ç†ã€map関数を繰り返ã™éŽç¨‹ã€çµ‚了処ç†ã¨ã„ã£ãŸä¸€é€£ã®æµã‚Œã‚’実行ã—ã¾ã™ã€‚ MapRunnerクラスをカスタマイズã™ã‚Œã°ã€ã“ã†ã—ãŸæµã‚Œã‚’制御ã™ã‚‹ã“ã¨ãŒã§ãã¾ã™ã€‚ 0.20.0ã‹ã‚‰ã®æ–°ã—ã„MapRed
MapReduce風ã®æ‰‹é †ã§ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ã‚¨ãƒ³ã‚¸ãƒ³ã‚’作る - ã±ã‚ã£ã¨ãƒ»ã™ãŸã˜ãŠ
先日ã€ã¡ã‚‡ã£ã¨æ°—分転æ›ãŒã—ãŸã‹ã£ãŸã®ã§ã€ å‰ã€…ã‹ã‚‰è©¦ãã†ã¨ã—ã¦ã„ãŸã€ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ã‚¨ãƒ³ã‚¸ãƒ³ã®å®Ÿè£…ã«æŒ‘戦ã—ã¦ã¿ã¾ã—㟠一応仕事ã®æ™‚é–“ãªã®ã§ã€ä¼šç¤¾çš„ã«æ„味ãŒã‚ã‚‹ã‚‚ã®ã‚’・・・ã¨ã„ã†ã“ã¨ã§ã€ è“„ç©ã•ã‚ŒãŸã¾ã¾ãŸã„ã—ã¦æ´»ç”¨ã•ã‚Œã¦ã„ãªã‹ã£ãŸã€ アクセスãƒã‚°ã‚’解æžã—ã¦ä½•ã‹ãƒ»ãƒ»ãƒ»ã¨æ€ã£ãŸã®ã§ã™ ãŸã ã€ãã®ã¾ã¾ä½œã£ã¦ã‚‚æ„味ãŒãªã„ã®ã§ã€ 後々MapReduceã«é©ç”¨ã§ãるよã†ã«ã€ æ„図的ã«å‡¦ç†ã‚’MapReduce風ã«æ›¸ã„ã¦ã¿ã¾ã—㟠çµæžœçš„ã«äºˆæƒ³ä»¥ä¸Šã®ä»£ç‰©ãŒã§ãã‚ãŒã£ã¦ã—ã¾ã„〠ç§è‡ªèº«ã³ã£ãã‚Šã—ãŸã‚ã‘ã§ã™ãŒã€ä»Šå›žã¯ãã®æ‰‹é †ã‚’メモ...φ(･ω･｀) ãªãŠã€ç´°ã‹ã„ç†å±ˆã¯ã“ã®è¾ºã§naoyaæ°ãŒè§£èª¬ã—ã¦ã¾ã™ã®ã§ã€ ãœã²ç¢ºèªã—ã¦ã¿ã¦ãã ã•ã„ WEB+DB PRESS Vol.57 作者: 今æ‘謙之,é è—¤æ£ä»,浜本階生,uupaa,増井俊之,大沢和å®,伊藤直也,æ‘瀬大輔,塙与志夫,ä¸å³¶æ‹“,ä¸å³¶è¡,角田直行,cho45,ã¯ã¾ã¡ã‚„
Hadoopã«ã‚ˆã‚‹Apacheã®ãƒã‚°è§£æžã®å®Ÿéš›
ã“ã‚“ã«ã¡ã¯ã€ãƒŸãƒ„ãƒãƒãƒ¯ãƒ¼ã‚¯ã‚¹ stoneã§ã™ã€‚ 今日ã¯ã€DECOLOGã§è¡Œã‚ã‚Œã¦ã„ã‚‹ã€Apacheã®ãƒã‚°è§£æžã«ã¤ã„ã¦ã€ ã”紹介ã—ã¦ã¿ã‚ˆã†ã‹ã¨æ€ã„ã¾ã™ã€‚ ç¾åœ¨ã€DECOLOGã§ã¯ã€ãƒªãƒãƒ¼ã‚¹ãƒ—ãƒã‚ã‚·ãŒ8å°ã‚ã£ã¦ã€ ãã®8å°ã®1æ—¥ã®Apacheã®ãƒã‚°ã¯ã€å…¨éƒ¨ã§ã€200Gãƒã‚¤ãƒˆä»¥ä¸Šã«ãªã£ã¦ã„ã¾ã™ã€‚ ã“れをã€13å°ã®Hadoopã®ã‚¹ãƒ¬ãƒ¼ãƒ–ノードã§è§£æžã‚’è¡Œã£ã¦ã„ã¾ã™ã€‚ 全体ã®æµã‚Œã¨ã—ã¦ã¯ã€ 1) リãƒãƒ¼ã‚¹ãƒ—ãƒã‚ã‚·ã‹ã‚‰HDFSã«ãƒã‚°ã‚’è»¢é€ 2) 解æžç”¨ã®ã‚µãƒ¼ãƒãƒ¼ã§ã€HDFSã«ãƒã‚°ã®è»¢é€ãŒçµ‚ã‚ã‚‹ã®ã‚’監視 3) ãƒã‚°ã®è»¢é€ãŒçµ‚ã‚ã£ãŸã‚‰ã€Hadoopã‚’èµ·å‹•ã€è§£æž 4) Hadoopã®è§£æžçµæžœã‚’データベースã«ä¿å˜ 以下ã§ã¯ã€å„々ã®ã‚¹ãƒ†ãƒƒãƒ—を個別ã«è¦‹ã¦è¡Œãã“ã¨ã«ã—ã¾ã™ã。 1. リãƒãƒ¼ã‚¹ãƒ—ãƒã‚ã‚·ã‹ã‚‰HDFSã«ãƒã‚°ã‚’è»¢é€ å½“åˆã€Hadoopã®ãƒ—ãƒã‚»ã‚¹ãŒç«‹ã¡ä¸ŠãŒã£ã¦ã„ãªã„ã¨ã€HDFSã«ã¯ã‚¢ã‚¯ã‚»ã‚¹ã§ããªã„〠ã¨æ€ã„込ん
Hadoop Streaming - naoyaã®ã¯ã¦ãªãƒ€ã‚¤ã‚¢ãƒªãƒ¼
id:naoya:20080511:1210506301 ã®ã‚¨ãƒ³ãƒˆãƒªã®ã‚³ãƒ¡ãƒ³ãƒˆæ¬„㧠kzk ã•ã‚“ã«æ•™ãˆã¦ã„ãŸã ã„㟠Hadoop Streaming を試ã—ã¦ã„ã¾ã™ã€‚ Hadoop ã¯ã‚ªãƒ¼ãƒ—ンソース㮠MapReduce + 分散ファイルシステムã§ã™ã€‚Java ã§ä½œã‚‰ã‚Œã¦ã„ã¾ã™ã€‚Yahoo! Inc ã®ãƒãƒƒã‚¯ã‚¨ãƒ³ãƒ‰ã‚„ã€Facebookã€Amazon.com ãªã©ã§ã‚‚利用ã•ã‚Œã¦ã„ã‚‹ã¨ã®ã“ã¨ã§ã™ã€‚詳ã—ã㯠http://codezine.jp/a/article/aid/2448.aspx (kzk ã•ã‚“ã«ã‚ˆã‚‹é€£è¼‰è¨˜äº‹)ã‚’å‚ç…§ã—ã¦ãã ã•ã„。 Hadoop Streaming 記事ã«ã‚‚ã‚ã‚Šã¾ã™é€šã‚Šã€Hadoop 拡張㮠Hadoop Streaming を使ã†ã¨æ¨™æº–入出力を介ã™ã‚‹ãƒ—ãƒã‚°ãƒ©ãƒ を記述ã™ã‚‹ã ã‘ã§ã€Hadoop ã«ã‚ˆã‚‹ MapReduce を利用ã™ã‚‹ã“ã¨ãŒã§ãã¾ã™ã€‚ã¤ã¾ã‚Šã€Java 以外
Amazon Elastic MapReduceを使ã£ã¦ã¿ãŸ - moratorium
Amazon Elastic MapReduceを使ã£ã¦ã¿ãŸ 2009-04-03 (Fri) 3:06 Amazon EC2 連日ã®EC2ãƒã‚¿ã§ã™ã€‚本日ã€Amazonã‹ã‚‰Elastic MapReduceã¨ã„ã†ã‚µãƒ¼ãƒ“スãŒãƒªãƒªãƒ¼ã‚¹ã•ã‚Œã¾ã—ãŸã€‚大è¦æ¨¡ãƒ‡ãƒ¼ã‚¿å‡¦ç†æŠ€è¡“ãŒä¸€æ°—ã«æ°‘é–“ã®æ‰‹ã«ä¸‹ã‚Šã¦ãã‚‹ã€ã¾ã•ã«é©å‘½çš„ãªã‚µãƒ¼ãƒ“スã ã¨æ€ã„ã¾ã™ã€‚ Amazon Elastic MapReduce Amazon ElasticMapReduce 紹介ビデオ With Hadoop, Amazon Adds A Web-Scale Data Processing Engine To Its Cloud Computer by techcrunch.com Elastic MapReduceã¯ã€Googleã®åŸºç›¤æŠ€è¡“ã®ä¸€ã¤ã§ã‚ã‚‹MapReduceを時間å˜ä½èª²é‡‘ã§å®Ÿè¡Œã§ãるサービスã§ã™ã€‚MapReduceã«ã¤ã„ã¦ã¯ä»¥
Cloudera を使ã£ã¦ CentOS ã« Hadoop on EC2 ãªç’°å¢ƒã‚’æ•´ãˆã‚‹ 第一回 - (ﾟ∀ﾟ)o彡 sasata299's blog
2009å¹´11月15æ—¥00:03 Hadoop Cloudera を使ã£ã¦ CentOS ã« Hadoop on EC2 ãªç’°å¢ƒã‚’æ•´ãˆã‚‹ 第一回 Hadoop ã®ã“ã¨ã‚’ã‚‚ã£ã¨çŸ¥ã‚‹ãŸã‚ã«ã€Cloudera を使ã£ã¦ Hadoop 環境を整ãˆã¦ã¿ã‚ˆã†ã¨æ€ã„ã¾ã™ã€‚Cloudera ã®ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã‚¬ã‚¤ãƒ‰ ã‚’å‚考ã«é€²ã‚ã¦ã¿ã¾ã™ã€‚OS 㯠CentOS 5.2 ã§ã™ã€‚ Cloudera's Distribution for Hadoop (CDH) ã¾ãšã€yum ã®ãƒªãƒã‚¸ãƒˆãƒªã« cloudera-stable.repos 㨠cloudera-testing.repos ã‚’è¿½åŠ ã—ã¾ã™ã€‚ã“れ㧠yum ã‹ã‚‰ Hadoop ã®ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ãŒå‡ºæ¥ã‚‹ã‚ˆã†ã«ãªã‚Šã¾ã™ã€‚便利ã§ã™ã〜。 ã¡ãªã¿ã«ã€cloudera-stable ã®æ–¹ã¯ Hadoop-0.18 ãŒã€cloudera-testing ã®æ–¹ã¯ Hadoo
HDFS ã®ä¿¡é ¼æ€§ï¼ˆåŽŸé¡Œï¼šHDFS Reliability） ‎(Pac Learner)‎
Not your computer? Use a private browsing window to sign in. Learn more about using Guest mode
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
IBM Developer
Demo for Hadoop User Environment (HUE) and Beeswax
特集 ： ECサイトã®ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰æŠ€è¡“を考ãˆã‚‹ - ZDNET Japan
Tatsuya Sasaki
hadoophn
{{#tags}}- {{label}}
{{/tags}}