Tokyo HBase Meetup - Realtime Big Data at Facebook with Hadoop and HBase (ja)
The document discusses Facebook's use of HBase to store messaging data. It provides an overview of HBase, including its data model, performance characteristics, and how it was a good fit for Facebook's needs due to its ability to handle large volumes of data, high write throughput, and efficient random access. It also describes some enhancements Facebook made to HBase to improve availability, stab
ç±³Yahoo!ã€ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ãªMapReduceフレームワーク「S4ã€å…¬é–‹ | gihyo.jp
2010å¹´11月4æ—¥ã€ç±³Yahoo! ã¯ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ã«MapReduceを処ç†ã™ã‚‹ã€ŒS4ã€ã‚’オープンソースã¨ã—ã¦å…¬é–‹ã—ã¾ã—ãŸã€‚ç±³Yahoo! ã§ã¯MapReduceã‚’è¡Œã†ãƒ•ãƒ¬ãƒ¼ãƒ ワークã¨ã—ã¦ã€ŒHadoopã€ã‚’利用ãŠã‚ˆã³é–‹ç™ºã—ã¦ã„ã‚‹ã“ã¨ã§æœ‰åã§ã™ã€‚ã—ã‹ã—ã€Hadoopã¯ãƒãƒƒãƒã®ã‚ˆã†ãªã„ã‚ゆるオフライン処ç†ã«å‘ã„ã¦ãŠã‚Šã€ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ã«å‡¦ç†ã™ã‚‹ã‚ˆã†ãªç›®çš„ã«ã¯é©ã—ã¦ã„ã¾ã›ã‚“。S4ã¯ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ã«MapReduceを処ç†ã™ã‚‹ç›®çš„ã§é–‹ç™ºã•ã‚Œã€ã™ã§ã«æ¤œç´¢åºƒå‘Šã®ãƒ‘ーソナライゼーションãªã©å®Ÿéš›ã®ã‚µãƒ¼ãƒ“スã§åˆ©ç”¨ã•ã‚Œã¦ã„るよã†ã§ã™ã€‚ S4ã¯Javaã§æ›¸ã‹ã‚Œã¦ãŠã‚Šã€Hadoopã¨ã¯ã‚³ãƒ¼ãƒ‰ãƒ™ãƒ¼ã‚¹ã§ã¯é–¢ä¿‚ã‚ã‚Šã¾ã›ã‚“。コードベースã§ã¯å¤§ããCommunication Layerã¨Core Classesã«åˆ†ã‹ã‚Œã¦ã„ã¾ã™ã€‚クラスタ管ç†ã«ã¯ZooKeeperã€ã‚·ãƒªã‚¢ãƒ©ã‚¤ã‚¼ãƒ¼ã‚·ãƒ§ãƒ³ã«ã¯Avroã¾ãŸã¯Kryoを利用ã—ã¦ã„ã¾ã™ã€‚
Hadoop Namenodeã®å†—長化ã«ã¤ã„ã¦ã€‚
最速é…ä¿¡ç ”ç©¶ä¼šå±±å´Žå¤§è¼” 制約ç†è«–åŠã³å¾…ã¡è¡Œåˆ—ç†è«–ã«ã‚ˆã‚‹æŠ€è¡“経営コンサルã¨ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢èµ·æ¥ç›¸è«‡ã‚„ã£ã¦ã¾ã™ @yamaz ãªã‚‹ã»ã©ï¼å€‹äººçš„ã«ã¯ç´ ã®HDFSをフãƒãƒ³ãƒˆé…ä¿¡ã«ã¯ä½¿ã„ãŸããªãã¨æ€ã£ã¦ãŸã®ã§ã€ãã‚Œãªã‚‰ç´å¾—ã§ã™ RT @kzk_mover: yamaz ã‚ã¨ç›¸å½“手を入れã¦Zookeeperを使ã£ãŸNameNodeã®å†—長化を独自ã§å®Ÿè£…ã—ã¦ã‚‹ã‚‰ã—ã„ã§ã™ã€‚ 2010-11-22 00:43:26 Tatsuya Kawano @tatsuya6502 .@yamaz @cocoatomo Facebook ã¯HDFS Name Nodeã®SPoF対ç–ã¨ã—㦠Avatar Node ã¨ã„ã†ãƒ›ãƒƒãƒˆã‚¹ã‚¿ãƒ³ãƒã‚¤ã®ã—ãã¿ã‚’開発・公開ã—ã€1500ノードã®æœ¬ç•ªç’°å¢ƒã§ä½¿ç”¨ã—ã¦ã„ã¾ã™ã€‚ http://j.mp/c0W60z #HBase 2010-11-22 08:50:24 Tatsuya Kawano @ta
rsyslog now supports Hadoop's HDFS - Rainer Gerhards
Rainer Gerhards Tech Enthusiast. Nature Lover. Computer Geek. rsyslog lead developer. I will be releasing rsyslog 5.7.1 today, a member of the v5-devel branch. With this version, omhdfs debuts. This is a specially-crafted output module to support Hadoop’s HDFS file system. The new module was a sponsored project and is useful for folks expecting enormous amounts of data or having high processing ti
Hadoopã«ã‚ˆã‚‹Apacheã®ãƒã‚°è§£æžã®å®Ÿéš›
ã“ã‚“ã«ã¡ã¯ã€ãƒŸãƒ„ãƒãƒãƒ¯ãƒ¼ã‚¯ã‚¹ stoneã§ã™ã€‚ 今日ã¯ã€DECOLOGã§è¡Œã‚ã‚Œã¦ã„ã‚‹ã€Apacheã®ãƒã‚°è§£æžã«ã¤ã„ã¦ã€ ã”紹介ã—ã¦ã¿ã‚ˆã†ã‹ã¨æ€ã„ã¾ã™ã€‚ ç¾åœ¨ã€DECOLOGã§ã¯ã€ãƒªãƒãƒ¼ã‚¹ãƒ—ãƒã‚ã‚·ãŒ8å°ã‚ã£ã¦ã€ ãã®8å°ã®1æ—¥ã®Apacheã®ãƒã‚°ã¯ã€å…¨éƒ¨ã§ã€200Gãƒã‚¤ãƒˆä»¥ä¸Šã«ãªã£ã¦ã„ã¾ã™ã€‚ ã“れをã€13å°ã®Hadoopã®ã‚¹ãƒ¬ãƒ¼ãƒ–ノードã§è§£æžã‚’è¡Œã£ã¦ã„ã¾ã™ã€‚ 全体ã®æµã‚Œã¨ã—ã¦ã¯ã€ 1) リãƒãƒ¼ã‚¹ãƒ—ãƒã‚ã‚·ã‹ã‚‰HDFSã«ãƒã‚°ã‚’è»¢é€ 2) 解æžç”¨ã®ã‚µãƒ¼ãƒãƒ¼ã§ã€HDFSã«ãƒã‚°ã®è»¢é€ãŒçµ‚ã‚ã‚‹ã®ã‚’監視 3) ãƒã‚°ã®è»¢é€ãŒçµ‚ã‚ã£ãŸã‚‰ã€Hadoopã‚’èµ·å‹•ã€è§£æž 4) Hadoopã®è§£æžçµæžœã‚’データベースã«ä¿å˜ 以下ã§ã¯ã€å„々ã®ã‚¹ãƒ†ãƒƒãƒ—を個別ã«è¦‹ã¦è¡Œãã“ã¨ã«ã—ã¾ã™ã。 1. リãƒãƒ¼ã‚¹ãƒ—ãƒã‚ã‚·ã‹ã‚‰HDFSã«ãƒã‚°ã‚’è»¢é€ å½“åˆã€Hadoopã®ãƒ—ãƒã‚»ã‚¹ãŒç«‹ã¡ä¸ŠãŒã£ã¦ã„ãªã„ã¨ã€HDFSã«ã¯ã‚¢ã‚¯ã‚»ã‚¹ã§ããªã„〠ã¨æ€ã„込ん
勉強会発表「プãƒã‚°ãƒ©ãƒžã®ãŸã‚ã®Hadoop入門〠- 科å¦ã¨éžç§‘å¦ã®è¿·å®®
id:kaigai ã®ä¸»å‚¬ã™ã‚‹å‹‰å¼·ä¼šã§ç™ºè¡¨ã—ã¦ãã¾ã—ãŸã€‚ Hadoop for programmerView more presentations from shiumachi. ç”ãˆã‚‰ã‚Œãªã‹ã£ãŸè³ªå• Shuffleフェーズã£ã¦ã€ã‚½ãƒ¼ãƒˆã‚’ã©ã†ã„ã†ä»•çµ„ã¿ã§ã‚„ã£ã¦ã‚‹ã®ï¼Ÿã€€ãƒ‡ãƒ¼ã‚¿å…¨éƒ¨ã‚’ãªã‚ã¦ã‚‹ã®ï¼Ÿ Partitionerã¨ã„ã†ã‚¯ãƒ©ã‚¹ã§ãƒ‡ãƒ¼ã‚¿ã‚’振り分ã‘ã¦ã‚‹ã€‚タスクã”ã¨ã¯ç‹¬ç«‹ã—ã¦ã‚‹ã®ã§ãƒ‡ãƒ¼ã‚¿ã‚’ã¾ãŸãŒã£ã¦ã‚¢ã‚¯ã‚»ã‚¹ã™ã‚‹ã“ã¨ã¯ãªã„ã¨æ€ã†ã€‚ã§ã‚‚ç´°ã‹ã„ã“ã¨ã¯ã¡ã‚‡ã£ã¨ã‚ã‹ã‚‰ãªã„。 Mapä¸ã«ãƒ‡ãƒ¼ã‚¿è¿½åŠ ã—ãŸã‚‰ã©ã†ãªã‚‹ã®ã‹ï¼Ÿ ã•ã™ãŒã«æ‰±ã†ãƒ‡ãƒ¼ã‚¿ã¯æœ€åˆã«æ±ºã‚ã¦ã„ã‚‹ã¨æ€ã†ãŒã€ã‚„ã£ãŸã“ã¨ãªã„ã®ã§ã‚ã‹ã‚‰ãªã„。 Streamingã£ã¦å…·ä½“çš„ã«ã©ã‚“ãªå‡¦ç†ã—ã¦ã‚‹ã®ï¼Ÿ jarファイルã¯æŠ•ã’ã¦ã‚‹ã‘ã©ã€å®Ÿè¡Œæ™‚ã«ä½¿ã†ã‚¹ã‚¯ãƒªãƒ—トã¯ã©ã†ã‚„ã£ã¦æŠ•ã’ã¦ã‚‹ã®ã‹ã‚ã‹ã‚‰ãªã„。 ã‚ã¨ã§èª¿ã¹ã¨ãã¾ã™ã€‚ 今ã®ä¸–ã®ä¸ã«å‡ºã¦ã‚‹Hadoop本ã£ã¦æ§‹ç¯‰ã¨ã‹é‹ç”¨ã®è©±
Hadoop＋Hive検証環境を構築ã—ã¦ã¿ã‚‹
Hadoop＋Hive検証環境を構築ã—ã¦ã¿ã‚‹ï¼šHive――RDB使ã„ã®ãŸã‚ã®Hadoopガイド（å‰ç·¨ï¼‰ï¼ˆ1/3 ページ） Hadoop Hiveã¯Hadoop上ã§SQLライクãªã‚¯ã‚¨ãƒªæ“作ãŒå¯èƒ½ãªDWHå‘ã‘ã®ãƒ—ãƒãƒ€ã‚¯ãƒˆã§ã™ã€‚SQLã«è¿‘ã„æ“作ãŒå¯èƒ½ãªãŸã‚ã€HBaseよりもデータベースã«æ…£ã‚Œè¦ªã—ã‚“ã ã¿ãªã•ã‚“ã«ã¯ä½¿ã„å‹æ‰‹ãŒã„ã„ã‹ã‚‚ã—ã‚Œã¾ã›ã‚“。本稿ã§ã¯ã“ã®Hiveã®ä½¿ã„æ–¹ã¨ãƒ¬ãƒ“ューを行ã£ã¦ã„ãã¾ã™ã€‚
hadoop - Open Blog - The New York Times
Oh, OSCON, we really do heart you here at NYTimes.com. You provide such great opportunities to mingle with developers and hear about all the new and exciting open-source technologies. This year, we were very excited to talk about our freshly open-sourced cloud management system, Nimbul. We also shared our stories about moving the NYTimes.com community platform to Amazon EC2. If you missed our pres
1
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
{{#tags}}- {{label}}
{{/tags}}