Hadoop利用者ãªã‚‰ãã£ã¨çŸ¥ã£ã¦ã‚‹ã€Hive/Prestoクエリ関数ã®æŒ™å‹•ã®é•ã„ - Qiita
HiveQLã§ã¯ã‚¹ãƒ”ードã«é›£ã‚’æ„Ÿã˜ã¦ã„ãŸãŸã‚ã€ç§ã‚‚Prestoを使ã„始ã‚ã¾ã—ãŸã€‚ MySQLã‚„Hiveã§ä½¿ã£ã¦ã„ãŸã‚¯ã‚¨ãƒªã‚’ç½®ãæ›ãˆã‚‹æ™‚ã«ãƒãƒžã£ãŸTipsã‚’ã¾ã¨ã‚ã¦ã„ãã¾ã™ã€‚ AWS Athenaã§Prestoを使ã£ã¦ã„る方も増ãˆã¦ã‚‹ã¨æ€ã†ã®ã§ã€Presto標準関数ã§ã®è¨˜è¿°ä¾‹ã‚‚æ‹¡å……ã—ã¦ã„ãã¾ã™ã€‚ Prestoã¨ã¯ Prestoã¯ã‚ªãƒ³ãƒ¡ãƒ¢ãƒªã§å‹•ã分散SQLエンジンã§ã€ãã®é€²åŒ–ã¯ç›®ã‚’見張る物ã§ã™ã€‚ 発表ã•ã‚ŒãŸå½“時ã¯è‰²ã€…ãªæˆç´„ãŒã‚り使ã†ã“ã¨ã‚’躊躇ã—ã¦ã„ã¾ã—ãŸãŒã€2015å¹´é ƒã‹ã‚‰ã¯ã‚‚ã†ä½¿ã‚ãªã„ç†ç”±ã¯ãªããªã‚Šã¾ã—ãŸã€‚ アドホックã«ä½¿ãˆã‚‹ã¨ã¦ã‚‚高速ãªSQLエンジンã§ã™ã®ã§ã€ãƒãƒƒãƒå‘ã‘ã®Hiveã®ã‚ˆã†ã«å®Ÿè¡Œçµæžœã‚’å¾…ã¤æ™‚é–“ã¯ã»ã¨ã‚“ã©ã‚ã‚Šã¾ã›ã‚“。 Hiveã§ã™ã¨1ã¤1ã¤ã®å®Ÿè¡Œã«æ™‚é–“ãŒæŽ›ã‹ã‚‹ã®ã§ã€ã‚¯ã‚¨ãƒªã«æ…£ã‚Œã¦ã„ãªã„æ–°å‚者ã«ã¯è¾›ã„物ãŒã‚ã‚Šã¾ã—ãŸã€‚ ã—ã‹ã—Prestoã§ã¯ã‚¤ãƒ³ã‚¿ãƒ©ã‚¯ãƒ†ã‚£ãƒ–ã«å®Ÿè¡Œã§ãã¾ã™ã®ã§ã€ãƒˆãƒ©ã‚¤
Apache Sparkã£ã¦ã©ã‚“ãªã‚‚ã®ã‹è¦‹ã¦ã¿ã‚‹ï¼ˆãã®ï¼‘ - 夢ã¨ã‚¬ãƒ©ã‚¯ã‚¿ã®é›†ç©å ´
ã“ã‚“ã«ã¡ã¯ã€‚ Kafkaを試ã—ã¦ã„る最ä¸ã§å¾®å¦™ã§ã™ãŒã€æœ€è¿‘使ãˆã‚‹ã®ã‹ãªãã€ã¨æƒ…å ±ã‚’é›†ã‚ã¦ã„ã‚‹ã®ãŒã€ŒApache Sparkã€ã§ã™ã€‚ MapReduceã¨åŒã˜ã分散並行処ç†ã‚’è¡Œã†åŸºç›¤ãªã®ã§ã™ãŒã€MapReduceよりも数åå€é€Ÿã„ã¨ã‹ã®æƒ…å ±ãŒã‚ã‚Šã¾ã™ã€‚ ・・・んãªé˜¿å‘†ãªã€ã¨ã‚‚æ€ã£ãŸã®ã§ã™ãŒã€å†…部ã§ä¿æŒã—ã¦ã„ã‚‹RDDã¨ã„ã†ä»•çµ„ã¿ãŒé¢ç™½ã„ã“ã¨ã‚‚ã‚り〠ã¨ã‚Šã‚ãˆãšè³‡æ–™ã‚„論文をèªã‚“ã§ã¿ã‚‹ã“ã¨ã«ã—ã¾ã—ãŸã€‚ ã¾ãšè¦‹ã¦ã¿ãŸè³‡æ–™ã¯ã€ŒOverview of Sparkã€ï¼ˆhttp://spark.incubator.apache.org/talks/overview.pdf）ã§ã™ã€‚ ã¨ã„ã†ã‚ã‘ã§ã€èªã‚“ã çµæžœã‚’ã¾ã¨ã‚ã¦ã¿ã¾ã™ã€‚ Sparkã¨ã¯ï¼Ÿ 高速ã§ã‚¤ãƒ³ã‚¿ãƒ©ã‚¯ãƒ†ã‚£ãƒ–ãªè¨€èªžçµ±åˆã‚¯ãƒ©ã‚¹ã‚¿ã‚³ãƒ³ãƒ”ューティング基盤 Sparkプãƒã‚¸ã‚§ã‚¯ãƒˆã®ã‚´ãƒ¼ãƒ«ã¯ï¼Ÿ 以下ã®2ã¤ã®è§£æžãƒ¦ãƒ¼ã‚¹ã‚±ãƒ¼ã‚¹ã«ã‚ˆã‚Šé©åˆã™ã‚‹ã‚ˆã†MapReduceã‚’æ‹¡å¼µ
Hiveクエリãƒãƒ¥ãƒ¼ãƒ‹ãƒ³ã‚°ãƒ¡ãƒ¢ - 科å¦ã¨éžç§‘å¦ã®è¿·å®®
Hadoop本2版を買ã£ãŸãã®ä¸€é€±é–“後ã«Deals of the day ã§åŠé¡ã‚»ãƒ¼ãƒ«ã‚„られã¦æ»ã«ãŸããªã£ãŸã®ã§è…¹ã„ã›ã«æ›¸ã ã¯ã˜ã‚ã« ã“ã“ã«æ›¸ã„ã¦ã‚ã‚‹ã®ã¯å…¨éƒ¨å‚考リンク・文献ã‹ã‚‰ã²ã£ã±ã£ã¦ããŸã ã‘ã§ã€ã»ã¨ã‚“ã©å…¨éƒ¨æ¤œè¨¼ã—ã¦ãªã„。間é•ã„ãŒã‚ã‚Œã°ãªã‚‹ã¹ãæ—©ã‚ã«æ›´æ–°ã™ã‚‹ã‚ˆã†åŠªåŠ›ã¯ã™ã‚‹ãŒã€éµœå‘‘ã¿ã«ã—ã¦ä½•ãŒèµ·ãã¦ã‚‚自己責任ã§ã€‚ Hive ã®ã‚¯ã‚¨ãƒªãƒãƒ¥ãƒ¼ãƒ‹ãƒ³ã‚°ã«é–¢ã™ã‚‹ãƒ¡ãƒ¢æ›¸ãã§ã‚る。以下ã®ã“ã¨ã¯ã€Œæ›¸ã„ã¦ã„ãªã„〠Hadoop自体ã®ãƒãƒ¥ãƒ¼ãƒ‹ãƒ³ã‚° Hive ã®ã‚¯ã‚¨ãƒªãƒãƒ¥ãƒ¼ãƒ‹ãƒ³ã‚°ä»¥å¤–ã®è©± 例ãˆã°ã€åœ§ç¸®ãƒ•ã‚¡ã‚¤ãƒ«ã‚’ Hive 上ã§æ‰±ã†ã«ã¯ã©ã†ã™ã‚‹ã‹ã¨ã‹ JOIN 一番左ã®ãƒ†ãƒ¼ãƒ–ルã«æœ€ã‚‚大ããªãƒ†ãƒ¼ãƒ–ルをæŒã£ã¦ãã‚‹ 一番左ã®ãƒ†ãƒ¼ãƒ–ルãŒMRã§ã„ã†å…¥åŠ›ãƒ‡ãƒ¼ã‚¿ã¨ã—ã¦æµã‚Œã‚‹ã€‚インナーテーブルã®ãƒ‡ãƒ¼ã‚¿ã¯ãƒ¡ãƒ¢ãƒªã«ä¿æŒã•ã‚Œã‚‹ã€‚ åŒä¸€ JOIN ã‚ー 通常㯠1 JOIN = 1 MR ジョブã ãŒã€åŒä¸€ã® JOIN ã‚ーを使ã£ã¦ã„
『Prestoã¨ã¯ä½•ã‹ï¼ŒPrestoã§ä½•ãŒã§ãã‚‹ã‹ã€ - トレジャーデータ（Treasure Data）ブãƒã‚°
トレジャーデータã¯ã‚¯ãƒ©ã‚¦ãƒ‰ã§ãƒ‡ãƒ¼ã‚¿ãƒžãƒãƒ¼ã‚¸ãƒ¡ãƒ³ãƒˆã‚µãƒ¼ãƒ“スをæä¾›ã—ã¦ã„ã¾ã™ã€‚ Hadoop Conference Japan 2014 以å‰ã«å‘ŠçŸ¥ã—ãŸHadoop Conference Japan 2014ã§ï¼Œå¼Šç¤¾Software Architectã®å¤æ©‹ãŒç™ºè¡¨ã—ã¾ã—ãŸã€‚ テーマã¯ï¼ŒFacebookãŒå…¬é–‹ã—ãŸæ–°ã—ã„分散処ç†åŸºç›¤ï¼ŒPresto。実ã¯FacebookãŒå½¼ã‚‰ã®è¶…大è¦æ¨¡ãªãƒ‡ãƒ¼ã‚¿ã‚»ãƒƒãƒˆã«å¯¾ã—ã¦ã‚¤ãƒ³ã‚¿ãƒ©ã‚¯ãƒ†ã‚£ãƒ–ã«çµæžœã‚’è¿”ã›ã‚‹ã‚ˆã†ã«ã¨é–‹ç™ºã•ã‚ŒãŸã‚‚ã®ã§ã™ã€‚開発ãŒå§‹ã¾ã£ã¦ã¾ã 2年も経ã£ã¦ãŠã‚Šã¾ã›ã‚“ãŒï¼Œä»Šã§ã¯ãƒˆãƒ¬ã‚¸ãƒ£ãƒ¼ãƒ‡ãƒ¼ã‚¿ã‚’åˆã‚ã¨ã—ã¦å¤šãã®ãƒãƒƒã‚«ãƒ¼é”ãŒã‚³ãƒŸãƒƒã‚¿ãƒ¼ã¨ã—ã¦å‚åŠ ã™ã‚‹æ´»ç™ºçš„ãªãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆã«æˆé•·ã—ã¦ã„ã¾ã™ã€‚ Prestoã¯Hiveã‚„Impalaã¨åŒã˜ã€ŒSQL Query Engineã€ã§ã‚り，特ã«æ•°ç™¾GBを超ãˆã‚‹å¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿ã«å¯¾ã—ã¦ã‚‚インタラクティブãªãƒ¬ã‚¹ãƒãƒ³ã‚¹ã‚’（コンマ0秒以下，é…ãã¦
カラムナストレージ - Yet Another HDIF?
Disclaimer: The opinions expressed here are my own and do not necessarily represent those of current or past employers.Twitter / Photos Disclaimer: The opinions expressed here are my own and do not necessarily represent those of current or past employers. Twitter / Photos Henry Robinsonã«ã‚ˆã‚‹ã€ã‚«ãƒ©ãƒ ナストレージã®è§£èª¬è¨˜äº‹ã‚’翻訳ã—ã¾ã—ãŸã€‚カラムナストレージã¯ã€Googleã§é–‹ç™ºã•ã‚ŒãŸãƒ‡ãƒ¼ã‚¿å‡¦ç†ãƒ„ールã§ã‚ã‚‹Dremelã«ä½¿ç”¨ã•ã‚Œã¦ã„るファイルフォーマットã§ã‚ã‚Šã€ClouderaãŒé–‹ç™ºã‚’進ã‚ã‚‹Impalaã§ã‚‚採用
Googleã®è™Žã®å「BigQueryã€ã‚’FluentdユーザーãŒä½¿ã‚ãªã„ç†ç”±ãŒãªããªã£ãŸç†ç”± #gcpja - Qiita
From Fluentd Meetupã«è¡Œã£ã¦ãã¾ã—㟠ã“れをèªã‚“ã 時ã€BigQueryã®æ¤œç´¢ã‚¹ãƒ”ードã«ã¤ã„ã¦ã¡ã‚‡ã£ã¨è£œè¶³ã—ãŸããªã£ãŸã€‚確ã‹ã«Fluentd Meetupã®ãƒ‡ãƒ¢ã§ã¯9億件を7秒程度ã§æ¤œç´¢ã—ã¦ã„ãŸãŒã€BigQueryã®çœŸã®å®ŸåŠ›ã¯ã“れより1〜2ケタ上ã ã‹ã‚‰ã 。ã¡ã‚‡ã£ã¨æ‰‹å…ƒã§å°‘ã—大ãã‚ã®ãƒ†ãƒ¼ãƒ–ルã§è©¦ã—ã¦ã¿ãŸã‚‰ã€120å„„è¡Œã®æ£è¦è¡¨ç¾ãƒžãƒƒãƒä»˜ã集計ãŒ5秒ã§å®Œäº†ã—ãŸã€‚è«–ã‚ˆã‚Šè¨¼æ‹ ã§ã€ãƒ‡ãƒ¢ãƒ“デオ（1分16秒）を作ã£ã¦ã¿ãŸï¼š From The Speed of Google BigQuery ã“ã‚Œã¯é€Ÿã™ãŽã‚‹ã€‚何ã‹ã®ã‚¤ãƒ³ãƒã‚ã§ã‚る（最åˆã«ãƒ‡ãƒ¢ã‚’見ãŸæ™‚ãã†æ€ã£ãŸï¼‰ã€‚æ£è¦è¡¨ç¾ã‚’ã„ã‚ã„ã‚変ãˆã¦ã¿ã¦ã‚‚スピードã¯å¤‰ã‚らãªã„。ã¤ã¾ã‚Šã€ã‚¤ãƒ³ãƒ‡ãƒƒã‚¯ã‚¹ã‚’事å‰æ§‹ç¯‰ã§ããªã„クエリã«å¯¾ã—ã¦ã“ã®ã‚¹ãƒ”ードãªã®ã§ã‚る。 ä¾¡æ ¼ã‚‚å®‰ã„。ã•ã™ãŒã«120å„„è¡Œã®ã‚¯ã‚¨ãƒªã¯1回ã§200円もã‹ã‹ã£ã¦æ°—軽ã«å®Ÿè¡Œã§ããªã•ãã†ã§ã‚ã‚‹ãŒã€1.2å„„
Hiveã®joinã®æœ€é©åŒ– - wyukawa's diary
å…ƒãƒã‚¿ã¯ã“ã¡ã‚‰ Join Optimization in Apache Hive Hiveã¯0.7ã‹ã‚‰joinãŒæœ€é©åŒ–ã•ã‚Œã¦ã„ã¾ã™ã€‚ã©ã®ã‚ˆã†ã«æœ€é©åŒ–ã•ã‚ŒãŸã®ã‹ä¸Šè¨˜ã®è³‡æ–™ã‚’ã²ã‚‚ã¨ã„ã¦ã¿ã¾ã™ã€‚ ã„ã¾ã¾ã§ã®join ã„ã¾ã¾ã§ã®joinã¯ã„ã‚ゆるソートマージジョインã§ã™ã€‚ mapフェーズã§ãƒ†ãƒ¼ãƒ–ルã®ãƒ‡ãƒ¼ã‚¿ã‚’èªã¿è¾¼ã‚“ã§joinã‚ーã€joinãƒãƒªãƒ¥ãƒ¼ã‚’出力ã—ã€shuffleフェーズã§ã‚½ãƒ¼ãƒˆã€reduceフェーズã§joinã¨ã„ã†æµã‚Œã§ã™ã€‚ ã“ã®å ´åˆshuffleフェーズã®ã‚½ãƒ¼ãƒˆå‡¦ç†ãŒãƒœãƒˆãƒ«ãƒãƒƒã‚¯ã¨ãªã£ã¦ã„ã¾ã—ãŸã€‚ ãã“ã§ç™»å ´ã™ã‚‹ã®ãŒMap Joinã§ã™ã€‚ joinã®ç‰‡æ–¹ã®ãƒ†ãƒ¼ãƒ–ルã®ã‚µã‚¤ã‚ºãŒãƒ¡ãƒ¢ãƒªã«åŽã¾ã‚‹ã»ã©å°ã•ã„ã®ã§ã‚ã‚Œã°ã€mapperã®ãƒ¡ãƒ¢ãƒªã«èªã¿è¾¼ã‚“ã§mapフェーズã ã‘ã§joinã—ã¾ã™ã€‚ ã“ã‚“ãªæ„Ÿã˜ã®æ§‹æ–‡ã§æ›¸ãã¾ã™ã€‚ select /*+mapjoin(a)*/ * from src1 x join
SIOS　ビッグデータ技術ブãƒã‚°: Treasure Dataã«ãŠã‘ã‚‹HiveQLã®Tips
ã“ã‚“ã«ã¡ã¯ã€ä»Šå›žã®ãƒ–ãƒã‚°æ‹…当 高橋ã§ã™ã€‚ 本題ã¨ã¯é€¸ã‚Œã¾ã™ãŒã€ãƒ“ッグデータã«é–¢é€£ã™ã‚‹ãƒˆãƒ¬ãƒ³ãƒ‰ã¨ã—ã¦ã€M2M(Machine to Machine)ã‚„IoT(Internet of Things)ã¨å‘¼ã°ã‚Œã‚‹æŠ€è¡“ãŒã‚ã‚Šã¾ã™ã€‚ SIOSビッグデータãƒãƒ¼ãƒ ã¨ã—ã¦ã‚‚ã€ã“れらã®æŠ€è¡“ã«ã‚ˆã£ã¦å¤§é‡ã«åŽé›†ã•ã‚Œã‚‹ãƒ‡ãƒ¼ã‚¿ã«ã¯æ³¨ç›®ã—ã¦ã„ã¾ã™ã€‚ ã“れらã®æŠ€è¡“を個人ã§å®Ÿç¾å¯èƒ½ãªãƒ—ãƒã‚°ãƒ©ãƒžãƒ–ルデãƒã‚¤ã‚¹ã¨ã—ã¦ã€Arduinoã‚„Raspberry PiãŒæ™®åŠã—ã¦ãã¦ã„ã¾ã™ã€‚ 特ã«ã€Arduinoã¯ã€æŽ¥è§¦ã‚»ãƒ³ã‚µã‚„赤外センサãªã©å„種センサを実装ã§ãã€ãªãŠã‹ã¤Bluetoothã‚„ZigBeeãªã©ã®é€šä¿¡ãƒ¢ã‚¸ãƒ¥ãƒ¼ãƒ«ã®å®Ÿè£…ã‚‚å¯èƒ½ã§ã™ã€‚ 例ãˆã°ã€è¤‡æ•°å°ã®Arduinoを組ã¿åˆã‚ã›ã¦è‡ªå®…内センサãƒãƒƒãƒˆãƒ¯ãƒ¼ã‚¯ã‚’構築ã—ã€æ—¥å¸¸ç”Ÿæ´»ã®è¦‹ãˆã‚‹åŒ–ãŒã§ããŸã‚‰æ¥½ã—ãã†ã§ã™ã。 ã“ã†ã—ãŸãƒ“ッグデータを生ã¿å‡ºã™æ§˜ã€…ãªã‚¢ã‚¤ãƒ‡ã‚¢ã‚’実ç¾ã™ã‚‹ãŸã‚ã«ã€ç§ãŸã¡ã‚‚日々ã€ãƒ“
スã‚ーマレスã«ã¤ã„ã¦ã¡ã‚‡ã£ã¨è€ƒãˆã¦ã¿ãŸ - As a Futurist...
ã“ã®ã‚¨ãƒ³ãƒˆãƒªã¯ãŸã¶ã‚“ã«ç…½ã‚Šè¦ç´ ã‚’å«ã‚ã¦ã„ã¾ã™ãŒã€æ„図的ãªã‚‚ã®ã§ã™ã€‚僕㯠NoSQL ã¯ç´ 晴らã—ã„ã¨æ€ã„ã¾ã™ã€‚ ã•ã¦ã€NoSQL ãªã‚“ã¦è¨€è‘‰ã«è¸Šã‚‰ã•ã‚Œã¦ã‚‹äººã¯ç½®ã„ã¨ã„ã¦ã€æœ€è¿‘ RDBMS 以外ã®ãƒ‡ãƒ¼ã‚¿ã‚¹ãƒˆã‚¢ã¨ã„ã†ã®ãŒè‰²ã€…ã§ã¦ãã¦ã¾ã™ã€‚今時点ã§è¦‹æ¸¡ã™é™ã‚Šã«ãŠã„ã¦ã¯ã€å®‰å®šæ€§ã€è€éšœå®³æ€§ã€ãƒ‘フォーマンスã€æƒ…å ±é‡ã€é–‹ç™ºè€…ã®æ…£ã‚Œã€å…¨ä½“ã®ãƒãƒ©ãƒ³ã‚¹ã§è¨€ãˆã° RDBMS ã«ã‹ãªã†ã‚‚ã®ã¯ãªã„ã‚ã‘ã§ã™ãŒã€ä»Šå¾Œã©ã†ãªã£ã¦ã„ãã‹ã¯ã¾ã分ã‹ã‚Šã¾ã›ã‚“。 一方ã§ã€RDBMS ãŒã©ã†ã—ã¦ã‚‚苦手ã¨ã™ã‚‹åˆ†é‡Žã¨ã„ã†ã®ã¯å˜åœ¨ã—ã¾ã™ã€‚例ãˆã° 1 サーãƒã«åŽã¾ã‚Šãらãªã„様ãªå¤§å®¹é‡ãƒ‡ãƒ¼ã‚¿ã«å¯¾ã™ã‚‹ãƒãƒƒãƒå‡¦ç†ã€ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ãªãƒ©ãƒ³ã‚ングã€ã‚¢ã‚¯ãƒ†ã‚£ãƒ“ティãªã©ã®ãƒ•ã‚£ãƒ¼ãƒ‰æƒ…å ±ã€ãã—ã¦æ§‹é€ 化ã•ã‚ŒãŸãƒ‡ãƒ¼ã‚¿ã®å–り扱ã„。何ã§ã‚‚ã‹ã‚“ã§ã‚‚ NoSQL ã«ç½®ãæ›ãˆã‚Œã°ã„ã„ãªã‚“ã¦è€ƒãˆã¯ç¾æ™‚点ã§ã¯åˆ°åº•å—ã‘入れãŒãŸã„ã§ã™ãŒã€ä¾‹ã¨ã—ã¦æŒ™ã’ãŸæ§˜ãªãƒ”ンãƒã‚¤ãƒ³ãƒˆãªéƒ¨åˆ†ã§ã¯ãã‚Œã«
Cloudera Blog
We are excited to announce the acquisition of Octopai, a leading data lineage and catalog platform that provides data discovery and governance for enterprises to enhance their data-driven decision making. Cloudera’s mission since its inception has been to empower organizations to transform all their data to deliver trusted, valuable, and predictive insights. With AI and […] Read blog post
Battle on: MapR, Cloudera pimp their Hadoop products – Old GigaOm
The fight for Hadoop dominance is officially on. The unveiling of Yahoo’s (s yhoo) Hadoop spinoff Hortonworks will undoubtedly be the talk of today’s Hadoop Summit, but it’s not the only game in town. In fact, while Hortonworks is busy answering questions about its product strategy, Cloudera and MapR will demonstrate new versions of their distributions overflowing with bells and whistles. I wrote
Amazon Redshiftã§ã¯ã˜ã‚るビッグデータ処ç†å…¥é–€ 記事一覧 | gihyo.jp
第5回Amazon Redshiftã®ã‚¢ãƒ¼ã‚テクãƒãƒ£ ～スケーリングã¨ãƒªã‚¹ãƒˆã‚¢ã‚’試ã—ã¦ã¿ã‚ˆã† 宮崎真，藤å·å¹¸ä¸€ 2013-06-10
SQL感覚ã§HiveQLを書ãã¨ç—›ã„ç›®ã«ã‚ã†ä¾‹ - still deeper
Hadoop Advent Calendar 2013 4日目ã®è¨˜äº‹ã§ã™ tl;dr explainã¨job historyã‚’èªã‚ 1 reducerã¯æ‚ª data skewã¯æ‚ª å‰æ›¸ã ã¿ã‚“ãªå¤§å¥½ãSQLã§Hadoop上ã§ã®å‡¦ç†ã‚’実行ã§ãã‚‹Hiveã«ã¯ã¿ãªã•ã‚“普段ã‹ã‚‰ãŠä¸–話ã«ãªã£ã¦ã„ã‚‹ã“ã¨ã§ã—ょã†ã€‚ã¡ã‚‡ã£ã¨èª¿ã¹ç‰©ã§ã‚°ã‚°ã‚‹åº¦ã«ç›®ã«å…¥ã‚‹æ„›ã‚‰ã„ã—ã„マスコットãŒã€è’ã‚“ã 心ã«æ¸…涼ãªé¢¨ã‚’ã¯ã“ã‚“ã§ãã‚Œã¾ã™ã€‚ ã§ã™ãŒHiveã®ã‚¯ã‚¨ãƒªè¨€èªžã¯SQLã§ã¯ãªãHiveQLã§ã™ã—ã€å®Ÿè¡Œã‚¨ãƒ³ã‚¸ãƒ³ã‚‚RDBã®ãã‚Œã¨ã¯å…¨ãç•°ãªã‚‹MapReduceã§ã™ã€‚SQLã®ã¤ã‚‚ã‚Šã§HiveQLを書ã„ã¦ã„ã‚‹ã¨åœ°é›·ã‚’è¸ã‚“ã§ã—ã¾ã†ã“ã¨ãŒã¾ã‚Œã«ã‚ˆãã‚ã‚Šã¾ã™ã€‚本エントリã§ã¯é™¥ã‚ŠãŒã¡ãªHiveQLã®è½ã¨ã—ç©´ã‚’2ã¤ç´¹ä»‹ã—ã¾ã™ã€‚ 例1 SELECT count(DISTINCT user_id) FROM access_log SQLã«æ…£ã‚ŒãŸæ–¹ã§ã‚ã‚Œ
SQLライクã«Hadoop Hiveを使ã„倒ã™!
パーティションを利用ã™ã‚‹ 今回ã¯å°‘ã—å‡ã£ãŸãƒ†ãƒ¼ãƒ–ルを定義をã—ã¦ã¿ã¾ã—ょã†ã€‚ 郵便番å·ãƒ‡ãƒ¼ã‚¿ã¯æ¯Žæœˆæ›´æ–°ã•ã‚Œã‚‹ã®ã§ã€ãƒ†ãƒ¼ãƒ–ル指定時ã«ãƒãƒ¼ã‚¸ãƒ§ãƒ³ã‚‚指定ã§ãるよã†ã«ã—ã¾ã™ã€‚ã“ã®ã‚ˆã†ãªå ´åˆã€Hiveã§ã¯ãƒ‘ーティションを使ã„ã¾ã™ã€‚ 以下ã«éƒµä¾¿ç•ªå·ã‚’ä¿å˜ã™ã‚‹ãƒ†ãƒ¼ãƒ–ル「zipã€ã‚’定義ã—ã¾ã™ãŒã€æ—¥ä»˜åž‹DATEã®ãƒ‘ーティションverã‚’è¨å®šã™ã‚‹ã‚ˆã†ã«ã—ã¾ã™ã€‚ hive> CREATE TABLE zip (zip STRING, pref INT, city STRING, town STRING) > PARTITIONED BY (ver DATE) > ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' > LINES TERMINATED BY '\n'; OK Time taken: 0.128 seconds
liris.org - ã“ã®ã‚¦ã‚§ãƒ–サイトã¯è²©å£²ç”¨ã§ã™ï¼Â - liris リソースãŠã‚ˆã³æƒ…å ±
This webpage was generated by the domain owner using Sedo Domain Parking. Disclaimer: Sedo maintains no relationship with third party advertisers. Reference to any specific service or trade mark is not controlled by Sedo nor does it constitute or imply its association, endorsement or recommendation.
Writing An Hadoop MapReduce Program In Python
In this tutorial I will describe how to write a simple MapReduce program for Hadoop in the Python programming language. Motivation What we want to do Prerequisites Python MapReduce Code Map step: mapper.py Reduce step: reducer.py Test your code (cat data | map | sort | reduce) Running the Python Code on Hadoop Download example input data Copy local example data to HDFS Run the MapReduce job Improv
ランã‚ング
ランã‚ング
ランã‚ング
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
The Best 4 Software Products for a Manufacturing Business - Sky Tree
crocos.jp
Insights - Enterprise Strategy Group
Grokking the Menagerie: An Introduction to the Hadoop Software Ecosystem by Blake Matheny | Articles | ebooks for software developers, programmers and designers
{{#tags}}- {{label}}
{{/tags}}