Nosql at twitter_devoxx2010
Notas del editorSmall talk -- how many use twitter? How many tweeted today? How many checked today? Feel free to tweet during the talk, I won’t get offended. Who am I? I went to a couple universities, worked in a few places, now I work on the analytics infrastructure at Twitter. The NoSQL term is bad because it defines something by what it is not, conflating a number of different techs.
NTTデータã®Hadoopå ±å‘Šæ›¸ãŒã™ã”ã‹ã£ãŸ - 科å¦ã¨éžç§‘å¦ã®è¿·å®®
æ¥ç•Œãƒˆãƒƒãƒ— ã®ã‚¨ãƒ³ã‚¿ãƒ¼ãƒ—ライズ Hadoop ä¼æ¥ Cloudera ã«å…¥ç¤¾ã—ã¾ã—㟠http://www.cloudera.co.jp/ 今年ã®6月ã«ã€ã€Œå¹³æˆï¼’１年度 産å¦é€£æºã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢å·¥å¦å®Ÿè·µäº‹æ¥å ±å‘Šæ›¸ã€ã¨ã„ã†ãƒ‰ã‚ュメント群ãŒçµŒç”£çœã‹ã‚‰å…¬è¡¨ã•ã‚Œã¾ã—ãŸã€‚ ãã®ã†ã¡ã®ä¸€ã¤ã«ã€NTTデータã«å§”託ã•ã‚ŒãŸHadoopã«é–¢ã™ã‚‹å®Ÿè¨¼å®Ÿé¨“ã®å ±å‘Šæ›¸ãŒã‚ã‚Šã¾ã—ãŸã®ã§ã€ä»Šæ›´ãªãŒã‚‰èªã‚“ã§ã¿ã‚‹ã“ã¨ã«ã—ã¾ã—ãŸã€‚ Hadoop界隈ã®äººã¯ã‚‚ã†ã¿ã‚“ãªã¨ã£ãã«èªã‚“ã§ã‚‹ã®ã‹ã‚‚ã—ã‚Œã¾ã›ã‚“ã‘ã©ã€‚ http://www.meti.go.jp/policy/mono_info_service/joho/downloadfiles/2010software_research/clou_dist_software.pdf ã€Œé«˜ä¿¡é ¼ã‚¯ãƒ©ã‚¦ãƒ‰å®Ÿç¾ç”¨ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢é–‹ç™ºï¼ˆåˆ†æ•£åˆ¶å¾¡å‡¦ç†æŠ€è¡“ç‰ã«ä¿‚ã‚‹ãƒ‡ãƒ¼ã‚¿ã‚»ãƒ³ã‚¿ãƒ¼é«˜ä¿¡é ¼åŒ–ã«å‘ã‘ãŸå®Ÿè¨¼äº‹æ¥ï¼‰ã€ã¨ã„ã†
『Hadoop/Hiveを用ã„ãŸãƒã‚°è§£æžåŸºç›¤ã®æ§‹ç¯‰ã€
ã“ã‚“ã«ã¡ã¯ã€‚Amebaã®ãƒã‚°è§£æžåŸºç›¤ã‚’担当ã—ã¦ã„ã‚‹ICHIROã§ã™ã€‚ 今回ã¯æ§‹ç¯‰ãƒ»é‹ç”¨ä¸ã®ãƒã‚°è§£æžåŸºç›¤ã€ŒPatriotã€ã«ã¤ã„ã¦æ›¸ããŸã„ã¨æ€ã„ã¾ã™ã€‚ Webサービスをé‹å–¶ã—ã¦ã„ã‚‹ã¨æ—¥ã€…大é‡ã®ãƒã‚°ãƒ‡ãƒ¼ã‚¿ã‚„ユーザデータãŒè“„ç©ã•ã‚Œã¾ã™ã€‚ 今ã¾ã§ã¯PV（ページビュー）やUU（ユニークユーザ）ãªã©ã‚¢ã‚¯ã‚»ã‚¹ãƒã‚°ãªã©ã‹ã‚‰å–れるã€å¤§æž ã®æŒ‡æ¨™ã®ã¿ã‚’見るã“ã¨ãŒã»ã¨ã‚“ã©ã§ã—ãŸã€‚ ページビューã«åˆã‚ã›ã¦ã‚·ã‚¹ãƒ†ãƒ を増強ã™ã‚‹ãªã©ã€ã‚·ã‚¹ãƒ†ãƒ å´ã¨ã—ã¦ã¯å分ã¨ã‚‚言ãˆã‚‹æŒ‡æ¨™ã§ã™ãŒã€åºƒå‘Šã‚„課金サービスã¨ã„ã†è¦–点ã‹ã‚‰è¦‹ã‚‹ã¨å分ã¨ã¯è¨€ãˆã¾ã›ã‚“。 今ã¾ã§ã¯Ameba内ã®å€‹ã€…ã®ã‚µãƒ¼ãƒ“ス担当者ãŒå¿…è¦ã¨ã™ã‚‹æŒ‡æ¨™ã‚’出ã™ãŸã‚ã«ã‚¢ãƒ—リエンジニアやDBエンジニアã«éƒ½åº¦ä¾é ¼ã‚’ã™ã‚‹å½¢ã§ãƒ‡ãƒ¼ã‚¿ã‚’抽出ã—ã¦ã„ã¾ã—ãŸã€‚ 今後ã®èª²é‡‘サービスã®ç™ºå±•ãŒè¦‹è¾¼ã¾ã‚Œã€ã‚ˆã‚Šãƒ‡ãƒ¼ã‚¿åˆ†æžã®é‡è¦æ€§ãŒé«˜ã¾ã‚‹ã¨è€ƒãˆãŸç§ã¯ã€ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ã§ãªã„サービス担当者（主ã«ãƒ—ãƒãƒ‡ãƒ¥ãƒ¼ã‚µï¼‰ãŒã‚ˆã‚Šç°¡å˜
Flume User Guide
[email protected] version 0.9.0, June 29th 2010 Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. The system is centrally man
Hadoop and Pig at Twitter__HadoopSummit2010
1. Hadoop is used extensively at Twitter to handle large volumes of data from logs and other sources totaling 7TB per day. Tools like Scribe and Crane are used to input data and Elephant Bird and HBase for storage. 2. Pig is used for data analysis on these large datasets to perform tasks like counting, correlating, and researching trends in users and tweets. 3. The results of these analyses are us
Hadoop - Integration Patterns and Practices__HadoopSummit2010
Hadoop Summit 2010 - Application Track Hadoop - Integration Patterns and Practices Eric Sammer, ClouderaRead less
Rubyã§Hadoopをラップã€åˆ†æ•£å‡¦ç†ãƒ„ールã‚ットãŒç™»å ´ ï¼ ï¼ IT
2009/05/12 米新èžç¤¾å¤§æ‰‹ã®ãƒ‹ãƒ¥ãƒ¼ãƒ¨ãƒ¼ã‚¯ãƒ»ã‚¿ã‚¤ãƒ ズã¯5月11æ—¥ã€Rubyã«ã‚ˆã‚‹å¤§è¦æ¨¡åˆ†æ•£å‡¦ç†ã®ãƒ„ールã‚ット「Map/Reduce Toolkitã€ï¼ˆMRToolkit）をGPLv3ã®ä¸‹ã«ã‚ªãƒ¼ãƒ—ンソースã§å…¬é–‹ã—ãŸã¨ç™ºè¡¨ã—ãŸã€‚MRToolkitã¯ã€ã™ã§ã«ç¨¼åƒã—ã¦ã„るクラスタ上ã®Hadoopã¨åˆã‚ã›ã¦ä½¿ã†ã“ã¨ã§Rubyã§å®¹æ˜“ã«Map/Reduce処ç†ã‚’記述ã™ã‚‹ã“ã¨ãŒã§ãる一種ã®ãƒ©ãƒƒãƒ‘ー。処ç†è‡ªä½“ã¯HadoopãŒè¡Œã†ã€‚ã™ã§ã«Hadoopを使ã£ã¦ã„るユーザーã§ã‚ã‚Œã°ã€ä¸å°è¦æ¨¡ã®ãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆã«å¯¾ã—ã¦ã€ã™ãã«MRToolkitã‚’é©ç”¨å¯èƒ½ã¨ã—ã¦ã„る。 デフォルトã§æœ‰ç”¨ãªMapã€Reduceã®å‡¦ç†ãƒ¢ã‚¸ãƒ¥ãƒ¼ãƒ«ãŒå«ã¾ã‚Œã¦ã„ã¦ã€æ•°è¡Œã®Rubyスクリプトを書ãã ã‘ã§ã€ä¾‹ãˆã°è†¨å¤§ãªApacheã®ãƒã‚°ã‹ã‚‰IPアドレス別ã®é–²è¦§å±¥æ´ã‚’ã¾ã¨ã‚ã‚‹ã¨ã„ã£ãŸå‡¦ç†ãŒå¯èƒ½ã¨ã„ã†ã€‚独自ã«Mapã‚„Reduceã®å‡¦ç†ã‚’定義ã™ã‚‹ã“ã¨ã‚‚
本をèªã‚€ GREE Labsã§Hadoopã®è©±ã‚’èžã„ã¦ããŸ
GREEã•ã‚“ã§ä¸å®šæœŸã§ã‚„ã£ã¦ã‚‹ã€GREE Labsオープンソーステクノãƒã‚¸ãƒ¼å‹‰å¼·ä¼šã§ã€Hadoopã®è©±ã‚’èžã„ã¦ãã¾ã—ãŸã€‚Hadoopã¯ã€ã¤ã¾ã‚Šã¯Googleã®GFSã‚„MapReduceã®ã‚¯ãƒãƒ¼ãƒ³ã ãã†ã§ã€ã€ŒGoogleを支ãˆã‚‹æŠ€è¡“ã€ã«ãƒˆã‚メã„ãŸäººãªã‚‰å¿…見ã§ã™ã。 発表ã¯ã€æŠ€è¡“é¢ã‚’ç°¡æ½”ã«æŠ¼ã•ãˆãŸã†ãˆã§ã‚ã‹ã‚Šã‚„ã™ãã€ãã®ã†ãˆå®Ÿéš›ã®åˆ©ç”¨äº‹ä¾‹ã®è©±ã‚‚èžã‘ãŸã®ãŒé¢ç™½ã‹ã£ãŸã¨æ€ã„ã¾ã™ã€‚最近ã®Webç³»ã§ã¯ã€ã‚µãƒ¼ãƒ“スé¢ã§ã‚‚マãƒã‚¿ã‚¤ã‚ºé¢ã§ã‚‚ã€ãƒ‡ãƒ¼ã‚¿ãƒžã‚¤ãƒ‹ãƒ³ã‚°ã¨ã‹è¡Œå‹•ã‚¿ãƒ¼ã‚²ãƒ†ã‚£ãƒ³ã‚°ã¨ã‹ãŒã‚¢ãƒ„ã„ã‚“ã ãªãã¨æ€ã„ã¾ã—ãŸã€‚ プレゼン資料もust録画も公開ã•ã‚Œã¦ã„ã¾ã™ãŒã€ä»¥ä¸‹ã€è‡ªåˆ†ã®ãƒ¡ãƒ¢ã¨ã„ã†æ„味ã§è¨˜éŒ²ã—ã¦ãŠãã¾ã™ã€‚ Hadoopã«ã¤ã„ã¦ï¼ˆå¤ªç”°ä¸€æ¨¹ï¼‰ Preferred Infrastructureã®CTOã§ã€Sedueã®ä½œè€…。大é‡ã®ãƒ‡ãƒ¼ã‚¿ã®å‡¦ç†ãŒãƒ†ãƒ¼ãƒžã§ã€åŠåˆ†ã¯é…’ã§ã§ãã¦ã„ã‚‹w。ãã‚“ãªã“ã‚“ãªã§ã€ã¯ã¦ãƒ–検索ã§ã‚‚使ã‚ã‚Œã¦ã„
blogeyeã®å®Ÿè£…ã«å¦ã¶ã€Amazon EC2/S3ã§ã®Hadoop活用術
ã¯ã˜ã‚ã« ã“ã‚“ã«ã¡ã¯ã€‚Hadoop連載 第4回ã¯å¤ªç”°ã•ã‚“ã«ä»£ã‚ã£ã¦å¤§å€‰ãŒæ‹…当ã—ã¾ã™ã€‚ ã“ã‚Œã¾ã§ã®é€£è¼‰ã§ã€Hadoopã«ã‚ˆã‚‹ãƒ‡ãƒ¼ã‚¿å‡¦ç†ã®æ¦‚ç•¥ã«ã¤ã„ã¦ã¯ç†è§£ã•ã‚Œã¦ã„ã‚‹ã¨æ€ã„ã¾ã™ã€‚今回ã¯Hadoopを利用ã—ãŸã‚·ã‚¹ãƒ†ãƒ ã®å®Ÿä¾‹ã¨ã„ã†ã“ã¨ã§ã€ãƒ–ãƒã‚°åˆ†æžã‚’è¡Œã†ã€Œblogeyeã€ã‚·ã‚¹ãƒ†ãƒ ã®æ¦‚ç•¥ã¨ã€ãã®ä¸ã§ã®Hadoop利用法を紹介ã—ã¾ã™ã€‚ ã¾ãŸã€blogeyeã¯AmazonãŒæä¾›ã—ã¦ã„ã‚‹EC2（レンタルサーãƒï¼‰ã€S3（ストレージ）をHadoopã¨çµ„ã¿åˆã‚ã›ã¦åˆ©ç”¨ã—ã¦ã„ã‚‹ã®ã§ã€ãã®è¾ºã‚Šã®å°Žå…¥æ–¹æ³•ã«ã¤ã„ã¦ã‚‚紹介ã—ã¾ã™ã€‚ ã“ã‚Œã¾ã§ã®é€£è¼‰ Hadoopã€hBaseã§æ§‹ç¯‰ã™ã‚‹å¤§è¦æ¨¡åˆ†æ•£ãƒ‡ãƒ¼ã‚¿å‡¦ç†ã‚·ã‚¹ãƒ†ãƒ Hadoopã®ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã¨ã‚µãƒ³ãƒ—ルプãƒã‚°ãƒ©ãƒ ã®å®Ÿè¡Œ 複数マシンã¸Hadoopをインストールã™ã‚‹ blogeyeã¨ã¯ 「blogeyeã€ï¼ˆãƒ–ãƒã‚°ã‚¢ã‚¤ï¼‰ã¯æ—¥æœ¬èªžã®ãƒ–ãƒã‚°ã‚’クãƒãƒ¼ãƒ«ã€ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ã«åˆ†æžã—ã¦ã€æµè¡Œèªžã¨æ€ã‚
Hadoopã®è§£æžè³‡æ–™
オープンソース分散システム「Hadoopã€ã«é–¢ã™ã‚‹è§£æžè³‡æ–™ã‚’公開ã•ã›ã¦é ‚ã„ã¦ãŠã‚Šã¾ã™ã€‚ã“ã®èª¿æŸ»ã¯NTTãƒ¬ã‚¾ãƒŠãƒ³ãƒˆæ ªå¼ä¼šç¤¾æ§˜ã¨å…±åŒã§è¡Œã„ã¾ã—ãŸ(プレスリリース)。 Hadoop解æžè³‡æ–™(PDF), 最終更新: 2008/08/25, 公開: 2008/08/25 Hadoopã®å®Ÿéš›ã®ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«æ–¹æ³•ãªã©ã«ã¤ãã¾ã—ã¦ã¯ã€å¼Šç¤¾å¤ªç”°ã«ã‚ˆã‚‹ä»¥ä¸‹ã®è¨˜äº‹ã‚‚ã”å‚考下ã•ã„。 Hadoopã€hBaseã§æ§‹ç¯‰ã™ã‚‹å¤§è¦æ¨¡åˆ†æ•£ãƒ‡ãƒ¼ã‚¿å‡¦ç†ã‚·ã‚¹ãƒ†ãƒ Hadoopã®ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã¨ã‚µãƒ³ãƒ—ルプãƒã‚°ãƒ©ãƒ ã®å®Ÿè¡Œ 複数マシンã¸Hadoopをインストールã™ã‚‹
blogeyeã®å®Ÿè£…ã«å¦ã¶ã€Amazon EC2/S3ã§ã®Hadoop活用術：CodeZine
ã¯ã˜ã‚ã«ã€€ã“ã‚“ã«ã¡ã¯ã€‚Hadoop連載 第4回ã¯å¤ªç”°ã•ã‚“ã«ä»£ã‚ã£ã¦å¤§å€‰ãŒæ‹…当ã—ã¾ã™ã€‚　ã“ã‚Œã¾ã§ã®é€£è¼‰ã§ã€Hadoopã«ã‚ˆã‚‹ãƒ‡ãƒ¼ã‚¿å‡¦ç†ã®æ¦‚ç•¥ã«ã¤ã„ã¦ã¯ç†è§£ã•ã‚Œã¦ã„ã‚‹ã¨æ€ã„ã¾ã™ã€‚今回ã¯Hadoopを利用ã—ãŸã‚·ã‚¹ãƒ†ãƒ ã®å®Ÿä¾‹ã¨ã„ã†ã“ã¨ã§ã€ãƒ–ãƒã‚°åˆ†æžã‚’è¡Œã†ã€Œblogeyeã€ã‚·ã‚¹ãƒ†ãƒ ã®æ¦‚ç•¥ã¨ã€ãã®ä¸ã§ã®Hadoop利用法を紹介ã—ã¾ã™ã€‚　ã¾ãŸã€blogeyeã¯AmazonãŒæä¾›ã—ã¦ã„ã‚‹EC2（レンタルサーãƒï¼‰ã€S3（ストレージ）をHadoopã¨çµ„ã¿åˆã‚ã›ã¦åˆ©ç”¨ã—ã¦ã„ã‚‹ã®ã§ã€ãã®è¾ºã‚Šã®å°Žå…¥æ–¹æ³•ã«ã¤ã„ã¦ã‚‚紹介ã—ã¾ã™ã€‚ã“ã‚Œã¾ã§ã®é€£è¼‰Hadoopã€hBaseã§æ§‹ç¯‰ã™ã‚‹å¤§è¦æ¨¡åˆ†æ•£ãƒ‡ãƒ¼ã‚¿å‡¦ç†ã‚·ã‚¹ãƒ†ãƒ Hadoopã®ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã¨ã‚µãƒ³ãƒ—ルプãƒã‚°ãƒ©ãƒ ã®å®Ÿè¡Œè¤‡æ•°ãƒžã‚·ãƒ³ã¸Hadoopをインストールã™ã‚‹ blogeyeã¨ã¯ã€€ã€Œblogeyeã€ï¼ˆãƒ–ãƒã‚°ã‚¢ã‚¤ï¼‰ã¯æ—¥æœ¬èªžã®ãƒ–ãƒã‚°ã‚’クãƒãƒ¼ãƒ«ã€ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ã«åˆ†æžã—ã¦ã€æµè¡Œèªžã¨æ€ã‚れるもã®
FrontPage - Hadoop Wiki
Apache Hadoop Apache Hadoop is a framework for running applications on large cluster built of commodity hardware. The Hadoop framework transparently provides applications both reliability and data motion. Hadoop implements a computational paradigm named Map/Reduce, where the application is divided into many small fragments of work, each of which may be executed or re-executed on any node in the cl
Runtime error - Meta Search
Error message : Directory is not found or not writable (DATA_DIR) Directory is not found or not writable (DIFF_DIR) Directory is not found or not writable (BACKUP_DIR) Directory is not found or not writable (CACHE_DIR) Site admin: whitestar Copyright © 2006-2023 whitestar. All Rights Reserved. Icons powered by famfamfam. PukiWiki 1.5.0 Copyright © 2001-2006 PukiWiki Developers Team. License is GPL
MapReduceã¨Hadoopã®å°†æ¥ã«ã¤ã„ã¦ã€Yahooã®Doug Cuttingã«ã‚¤ãƒ³ã‚¿ãƒ“ュー
Spring Bootã«ã‚ˆã‚‹APIãƒãƒƒã‚¯ã‚¨ãƒ³ãƒ‰æ§‹ç¯‰å®Ÿè·µã‚¬ã‚¤ãƒ‰ 第2版 何åƒäººã‚‚ã®é–‹ç™ºè€…ãŒã€InfoQã®ãƒŸãƒ‹ãƒ–ック「Practical Guide to Building an API Back End with Spring Bootã€ã‹ã‚‰ã€Spring Bootを使ã£ãŸREST API構築ã®åŸºç¤Žã‚’å¦ã‚“ã 。ã“ã®æœ¬ã§ã¯ã€å‡ºç‰ˆæ™‚ã«æ–°ã—ãリリースã•ã‚ŒãŸãƒãƒ¼ã‚¸ãƒ§ãƒ³ã§ã‚ã‚‹ Spring Boot 2 を使用ã—ã¦ã„る。ã—ã‹ã—ã€Spring Boot3ãŒæœ€è¿‘リリースã•ã‚Œã€é‡è¦ãªå¤‰...
Welcome to Apache™ Hadoop®!
A wide variety of companies and organizations use Hadoop for both research and production. Users are encouraged to add themselves to the Hadoop PoweredBy wiki page. 8 Aug 2018: Release 3.1.1 available This is the first stable release of Apache Hadoop 3.1 line. It contains 435 bug fixes, improvements and enhancements since 3.1.0 Users are encouraged to read the overview of major changes since 3.1.0
Hadoop
13. $ env | grep JAVA JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home $ java -version java -versionjava version quot;1.5.0_07quot;Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_07-154)Java HotSpot(TM) Client VM (build 1.5.0_07-87, mixed mode, sharing) $ curl -O http://www.apache.org/dist/lucene/hadoop/stable/hadoop-0.13.0.tar.gz $ tar zxvf hadoop-0.13.
1
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
http://www.docstoc.com/docs/2996433/Hadoop-and-HBase-vs-RDBMS
http://news.tokyo.wakhok.ac.jp/x/modules/tinyd1/content/newtech2007/Amazon_EC2_Hadoop-2/siframes.html
{{#tags}}- {{label}}
{{/tags}}