分散システムã«ã¤ã„ã¦èªžã‚‰ã›ã¦ãã‚ŒNTT Tech Conference #2 ã«ã¦è©±ã—ãŸè³‡æ–™ 時間ãŒè¶³ã‚Šãªã‹ã£ãŸã®ã§å…¨éƒ¨ã¯è©±ã›ãªã‹ã£ãŸã€‚Read less
Steve Krenzel - 6. MapReduce: Finding Friends
MapReduce is a framework originally developed at Google that allows for easy large scale distributed computing across a number of domains. Apache Hadoop is an open source implementation. I'll gloss over the details, but it comes down to defining two functions: a map function and a reduce function. The map function takes a value and outputs key:value pairs. For instance, if we define a map function
「Hadoopã®æ™‚代ã¯çµ‚ã‚ã£ãŸã€ã®æ„味をæ£ã—ãç†è§£ã™ã‚‹ - 科å¦ã¨éžç§‘å¦ã®è¿·å®®
Hadoopã®æ™‚代ã¯çµ‚ã‚ã£ãŸã€ã¨ã„ã†è¨€èª¬ã‚’ãŸã¾ã«è¦‹ã‹ã‘るよã†ã«ãªã‚Šã¾ã—ãŸã€‚ ã‚‚ã¡ã‚ん終ã‚ã£ã¦ãªã©ã„ã¾ã›ã‚“。ã—ã‹ã—ã€Hadoopã¨ãã®å–ã‚Šå·»ã環境ãŒå¤‰åŒ–ã—ãŸã®ã¯äº‹å®Ÿã§ã™ã€‚ 本記事ã§ã¯ã€ã“ã®å¤‰åŒ–ãŒä½•ãªã®ã‹ã‚’明らã‹ã«ã—ã€ãã®ä¸Šã§ã€ãªãœHadoopã®æ™‚代ã¯çµ‚ã‚ã£ãŸã¨ã„ã†ä¸»å¼µãŒå®Ÿæ…‹ã‚’æ£ã—ã表ã—ã¦ã„ãªã„ã®ã‹ã‚’説明ã—ã¦ã„ãã¾ã™ã€‚ DISCLAIMER ç§ã¯Hadoopã‚’ä¸å¿ƒã¨ã—ãŸãƒ‡ãƒ¼ã‚¿åŸºç›¤ã‚’å–り扱ã†ãƒ™ãƒ³ãƒ€ãƒ¼ã€Clouderaã®ç¤¾å“¡ã§ã™ã€‚ ä¸ç«‹çš„ã«æ›¸ãよã†åŠªã‚ã¾ã™ãŒã€æ‰€å±žçµ„ç¹”ã«ã‚ˆã£ã¦ç™ºç”Ÿã™ã‚‹ãƒã‚¤ã‚¢ã‚¹ã®å®Œå…¨ãªæŽ’除をä¿è¨¼ã™ã‚‹ã“ã¨ã¯ã§ãã¾ã›ã‚“。 以上をã”了承ã®ä¸Šã€èªã¿é€²ã‚ã¦ãã ã•ã„。 è¦ç´„ データ基盤ã¯ã€Hadoopã®ç™»å ´ã«ã‚ˆã‚Šéžå¸¸ã«å®‰ä¾¡ã¨ãªã‚Šã€ä»Šã¾ã§ã§ã¯ä¸å¯èƒ½ã ã£ãŸå¤§é‡ã®ãƒ‡ãƒ¼ã‚¿ã‚’å–り扱ãˆã‚‹ã‚ˆã†ã«ãªã‚Šã¾ã—ãŸã€‚ Hadoopã¯ã€NoSQLブームã®ä¸ã€å‡¦ç†ã‚¨ãƒ³ã‚¸ãƒ³ã§ã‚ã‚‹MapReduceã¨ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ã§ã‚ã‚‹HDFSãŒ
新人プãƒã‚°ãƒ©ãƒžã«çŸ¥ã£ã¦ã‚‚らã„ãŸã„RabbitMQåˆå¿ƒè€…ã®å…¥é–€ã®å…¥é–€ - Qiita
Deleted articles cannot be recovered. Draft of this article would be also deleted. Are you sure you want to delete this article? é¡Œåã¯ã“ã“オマージュã§ã™ã€‚ 背景ã¨ç›®çš„ 勤ã‚ã¦ã„る会社ã§ã¯ã€ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ãƒ³ã‚°ã®ãƒŸãƒ‰ãƒ«ã‚¦ã‚§ã‚¢ã¨ã—ã¦AMQPã¨ã„ã†ãƒ—ãƒãƒˆã‚³ãƒ«ã‚’利用ã—ã¦ã„ã‚‹RabbitMQを使用ã—ã¦ã„ã¾ã™ã€‚ RabbitMQã«ã¤ã„ã¦ã¯ã€å…¬å¼ã‚µã‚¤ãƒˆã®èª¬æ˜ŽãŒå……実ã—ã¦ã„ã‚‹ã®ã§ã€æ¥å‹™ã§ä½¿ç”¨ã‚’考ãˆã¦ã„る方㯠基本的ã«ã“ã¡ã‚‰ã‚’èªã‚€ã“ã¨ã‚’ãŠå‹§ã‚ã„ãŸã—ã¾ã™ã€‚ ãŸã ã€è‡ªåˆ†ã§èª¿ã¹ã¦ã„㦠公å¼ã‚µã‚¤ãƒˆã®å†…容ã¯è‹±èªž 日本語ã®ã‚µã‚¤ãƒˆã‚’調ã¹ã‚ˆã†ã¨ã—ã¦ã‚‚æƒ…å ±ãŒå°‘ãªã‹ã£ãŸã‚Šæ•£ã‚‰ã°ã£ã¦ã„ã‚‹ ãã‚‚ãもメッセージングã®ãƒŸãƒ‰ãƒ«ã‚¦ã‚§ã‚¢ãŒã©ã†ã„ã£ãŸã‚‚ã®ã§ã€ã©ã†ã„ã†éš›ã«å¿…è¦ã«ãªã‚‹ã‹ã®èª¬æ˜ŽãŒå°‘ãªã„ ã“ã¨ã§çµæ§‹æ®µå–り時間ãŒã‹ã‹
RabbitMQ ãƒãƒ¥ãƒ¼ãƒˆãƒªã‚¢ãƒ«ï¼–（RPC） - Qiita
RabbitMQã®ãƒãƒ¥ãƒ¼ãƒˆãƒªã‚¢ãƒ«ï¼– https://www.rabbitmq.com/tutorials/tutorial-six-python.html ã®ç¿»è¨³ã§ã™ã€‚ 翻訳ã®èª¤ã‚Šãªã©ã‚ã‚Œã°ã”指摘ãŠå¾…ã¡ã—ã¦ãŠã‚Šã¾ã™ã€‚ ###å‰ææ¡ä»¶ ã“ã®ãƒãƒ¥ãƒ¼ãƒˆãƒªã‚¢ãƒ«ã§ã¯ã€RabbitMQã®ãŒã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã•ã‚Œã€ãƒãƒ¼ã‚«ãƒ«ãƒ›ã‚¹ãƒˆã®æ¨™æº–ã®ãƒãƒ¼ãƒˆï¼ˆ5672）上ã§å®Ÿè¡Œã•ã‚Œã¦ã„ã‚‹å‰æã¨ã—ã¾ã™ã€‚別ã®ãƒ›ã‚¹ãƒˆã€ãƒãƒ¼ãƒˆã€ã¾ãŸã¯è³‡æ ¼æƒ…å ±ã‚’ä½¿ç”¨ã™ã‚‹å ´åˆã«ã¯ã€æŽ¥ç¶šè¨å®šã®èª¿æ•´ãŒå¿…è¦ã§ã™ã€‚ ###å•é¡ŒãŒç™ºç”Ÿã—ãŸå ´åˆ ã“ã®ãƒãƒ¥ãƒ¼ãƒˆãƒªã‚¢ãƒ«ã‚’通ã—ã¦å•é¡ŒãŒç™ºç”Ÿã—ãŸå ´åˆã€ãƒ¡ãƒ¼ãƒªãƒ³ã‚°ãƒªã‚¹ãƒˆã‚’通ã—ã¦ç§ãŸã¡ã«é€£çµ¡ã™ã‚‹ã“ã¨ãŒã§ãã¾ã™ã€‚ リモート・プãƒã‚·ãƒ¼ã‚¸ãƒ£ãƒ»ã‚³ãƒ¼ãƒ«ï¼ˆRPC） （pika 0.9.8 Python clientを使用） 第２ã®ãƒãƒ¥ãƒ¼ãƒˆãƒªã‚¢ãƒ«ã§ã¯ã€è¤‡æ•°ã®ãƒ¯ãƒ¼ã‚«ãƒ¼ã®é–“ã§æ™‚é–“ã®ã‹ã‹ã‚‹ã‚¿ã‚¹ã‚¯ã‚’分散ã™ã‚‹ãŸã‚ã«ãƒ¯ãƒ¼ã‚¯ã‚ューを使用ã™ã‚‹æ–¹æ³•ã‚’å¦ã³ã¾ã—ãŸ
SQL感覚ã§HiveQLを書ãã¨ç—›ã„ç›®ã«ã‚ã†ä¾‹ - still deeper
Hadoop Advent Calendar 2013 4日目ã®è¨˜äº‹ã§ã™ tl;dr explainã¨job historyã‚’èªã‚ 1 reducerã¯æ‚ª data skewã¯æ‚ª å‰æ›¸ã ã¿ã‚“ãªå¤§å¥½ãSQLã§Hadoop上ã§ã®å‡¦ç†ã‚’実行ã§ãã‚‹Hiveã«ã¯ã¿ãªã•ã‚“普段ã‹ã‚‰ãŠä¸–話ã«ãªã£ã¦ã„ã‚‹ã“ã¨ã§ã—ょã†ã€‚ã¡ã‚‡ã£ã¨èª¿ã¹ç‰©ã§ã‚°ã‚°ã‚‹åº¦ã«ç›®ã«å…¥ã‚‹æ„›ã‚‰ã„ã—ã„マスコットãŒã€è’ã‚“ã 心ã«æ¸…涼ãªé¢¨ã‚’ã¯ã“ã‚“ã§ãã‚Œã¾ã™ã€‚ ã§ã™ãŒHiveã®ã‚¯ã‚¨ãƒªè¨€èªžã¯SQLã§ã¯ãªãHiveQLã§ã™ã—ã€å®Ÿè¡Œã‚¨ãƒ³ã‚¸ãƒ³ã‚‚RDBã®ãã‚Œã¨ã¯å…¨ãç•°ãªã‚‹MapReduceã§ã™ã€‚SQLã®ã¤ã‚‚ã‚Šã§HiveQLを書ã„ã¦ã„ã‚‹ã¨åœ°é›·ã‚’è¸ã‚“ã§ã—ã¾ã†ã“ã¨ãŒã¾ã‚Œã«ã‚ˆãã‚ã‚Šã¾ã™ã€‚本エントリã§ã¯é™¥ã‚ŠãŒã¡ãªHiveQLã®è½ã¨ã—ç©´ã‚’2ã¤ç´¹ä»‹ã—ã¾ã™ã€‚ 例1 SELECT count(DISTINCT user_id) FROM access_log SQLã«æ…£ã‚ŒãŸæ–¹ã§ã‚ã‚Œ
LanguageManual - Apache Hive - Apache Software Foundation
This is the Hive Language Manual.  For other Hive documentation, see the Hive wiki's Home page. Commands and CLIs CommandsHive CLI (old)Beeline CLI (new)Variable SubstitutionHCatalog CLIFile FormatsAvro FilesORC FilesParquetCompressed Data StorageLZO CompressionData TypesData Definition StatementsDDL StatementsBucketed TablesStatistics (Analyze and Describe)IndexesArchivingData Manipulation Statem
1
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
Configuring the Mapping from Kerberos Principals to Short Names | 5.3.x | Cloudera Documentation
{{#tags}}- {{label}}
{{/tags}}