Apache Hive vs Apache Impala Query Performance Comparison - Cloudera Blog
The 100% open source and community driven innovation of Apache Hive 2.0 and LLAP (Long Last and Process) truly brings agile analytics to the next level. It enables customers to perform sub-second interactive queries without the need for additional SQL-based analytical tools, enabling rapid analytical iterations and providing significant time-to-value. TRY HIVE LLAP TODAY Read about how Hive with L
Tuning Apache Hive on Spark in CDH | 6.3.x | Cloudera Documentation
Minimum Required Role: Configurator (also provided by Cluster Administrator, Full Administrator) Hive on Spark provides better performance than Hive on MapReduce while offering the same features. Running Hive on Spark requires no changes to user queries. Specifically, user-defined functions (UDFs) are fully supported, and most performance-related configurations work with the same semantics. This t
Apache Spark @Scale: A 60 TB+ production use case
Facebook often uses analytics for data-driven decision making. Over the past few years, user and product growth has pushed our analytics engines to operate on data sets in the tens of terabytes for a single query. Some of our batch analytics is executed through the venerable Hive platform (contributed to Apache Hive by Facebook in 2009) and Corona, our custom MapReduce implementation. Facebook has
Tez on EMRを試ã—ã¦ã¿ãŸ
1. Developer Day Tez  on  EMRを試ã—ã¦ã¿ãŸ 1 G-â€â€‘‒1 ã‚¯ãƒ©ã‚¹ãƒ¡ã‚½ãƒƒãƒ‰æ ªå¼ä¼šç¤¾â€ƒã€€èƒ½ç™» Â è« â’¸  Classmethod,  Inc. 2015年年03⽉月29⽇日 #cmdevio2015G
Unit testing HQL - Apache Hive - Apache Software Foundation
Motivations Hive is widely applied as solution to numerous distinct problem types in the domain of big data. Quite clearly it often used for the ad hoc querying of large datasets. However it is also used to implement ETL type processes. Unlike ad hoc queries, the HQL written for ETLs has some distinct attributes: It is executed repeatedly, usually on a schedule. It is frequently a large, complex b
A perfect Hive query for a perfect meeting (Hadoop Summit 2014)
A perfect Hive query for a perfect meeting (Hadoop Summit 2014) During one of our epic parties, Martin Lorentzon (chairman of Spotify) agreed to help me to arrange a dinner for me and Timbuktu (my favourite Swedish rap and reggae artist), if I prove somehow that I am the biggest fan of Timbuktu in my home country. Because at Spotify we attack all problems using data-driven approaches, I decided to
Hiveを高速化ã™ã‚‹LLAP
2. About me • å°æ¾¤ ç¥ä¹Ÿ( @__john_smith__ ) • ãƒ¤ãƒ•ãƒ¼æ ªå¼ä¼šç¤¾ • データ&サイエンスソリューション統括本部 • データインフラ本部 開発2部 OSS開発 â†å¹³å¸¸æ™‚ã®è‡ªå·±ç´¹ä»‹ 3. アジェンダ Hiveã®å¾©ç¿’ – MR, Tez, Spark – Hiveã®é™ç•Œ LLAP(Live Long and Process)ã®ç´¹ä»‹ – LLAPã¨ã¯ – LLAPã®ç‰¹å¾´ã¨å‹•ä½œ – LLAP vs Spark SQL – 実行速度 LLAPã®ä»Šå¾Œ
MRv2/Tezã§ç°¡å˜ã«ã‚¯ã‚¨ãƒªã®ãƒ™ãƒ³ãƒã‚’ã¨ã£ãŸ - ãŸã”ã‚‚ã‚Šã™ãƒ¡ãƒ¢
Hiveã—ã‹ä½¿ã£ã¦ãªã„ã®ã§ä»¥ä¸‹ã®ã‚ªãƒ—ションをè¨å®šã™ã‚‹ã ã‘ã§ä½¿ãˆã‚‹ã€‚楽。 SET hive.execution.engine=tez; ãªãŠ HDP 2.1 with Hive 0.10, Tez 0.4 ã§ã®è©±ã§ã™ã€‚クラスタã®æ¦‚è¦ã¯ä»¥ä¸‹ã®é€šã‚Šã€‚ master x3 slave x20 Xeon(R) CPU E5-2630L v2 (6core 12Threads) x2 RAM 64GB HDD x12 シナリオ データã®æµã‚Œã¨ã—ã¦ã¯ä»¥ä¸‹ã®ã‚ˆã†ãªã‚·ãƒŠãƒªã‚ªã‚’想定ã™ã‚‹ã€‚ 外部ã‹ã‚‰éžåœ§ç¸®plain text tsvã§HDFS内ã®ãƒ•ã‚¡ã‚¤ãƒ«ã«ãƒ‡ãƒ¼ã‚¿ãŒæ›¸ã‹ã‚Œã‚‹ LOAD㧠hourly テーブルã«èªã¿è¾¼ã‚€ daily㧠INSERT ã«ã‚ˆã‚Š daily テーブルã«æ›¸ã込む ã“ã®ã¨ãファイルフォーマット変æ›ã‚„圧縮をåŒæ™‚ã«è¡Œã† hourly テーブルã®å¤‰æ›æ¸ˆã¿ãƒ‘ーティションãŠã‚ˆã³å…ƒã®ç”Ÿãƒ‡ãƒ¼ã‚¿ã¯å‰Šé™¤ã™ã‚‹
今日ã‹ã‚‰ã¯ã˜ã‚ã‚‹ãŠæ‰‹è»½ Hive データ移行 - mixi engineer blog
ã“ã‚“ã«ã¡ã¯ï¼Ž 昨年知人ã®ã‚ªãƒ¼ã‚±ã‚¹ãƒˆãƒ©æ¼”å¥ä¼šã§è´ã„ãŸãƒ¡ãƒ³ãƒ‡ãƒ«ã‚¹ã‚¾ãƒ¼ãƒ³äº¤éŸ¿æ›²ç¬¬ï¼”番「イタリアã€ãŒå¤§å¥½ããªï¼ŒæŠ€è¡“部ã®çŸ³å·æœ‰ã§ã™ï¼Ž ãã‚“ãªã€Œã‚¤ã‚¿ãƒªã‚¢ã€å¤§å¥½ããªç§ã§ã™ãŒ Hive ã®ãƒ‡ãƒ¼ã‚¿ç§»è¡Œã‚‚大好ãã¨ã„ã†ä½“ã§ï¼Œ 今回ã®è¨˜äº‹ã§ã¯ã‚ªãƒ³ãƒ—レミス㧠Hive ã‚’é ‘å¼µã£ã¦é‹ç”¨ã—ã¦ã„ã‚‹æ–¹ãŸã¡ã«å‘ã‘ã¦ï¼Œã©ã†ã—ã¦ã‚‚ã¨ã„ã†ã¨ãã® Hive ã®ãƒ‡ãƒ¼ã‚¿ç§»è¡Œæ–¹æ³•ã‚’紹介ã—ãŸã„ã¨æ€ã„ã¾ã™ï¼Ž 弊社ã§ã¯æœ€è¿‘，Cloudera's Distribution including Apache Hadoop 3 update 2 (CDH3u2) ã‹ã‚‰ CDH 4.1.2 ã«ãƒãƒ¼ã‚¸ãƒ§ãƒ³ã‚¢ãƒƒãƒ—ã‚’è¡Œãªã£ã¦ã„ã¾ã™ï¼Ž ã“ã®ã‚ˆã†ã«ã‚ªãƒ³ãƒ—レミス㧠Hive ã‚’é‹ç”¨ã—ã¦ã„ã‚‹æ–¹ãŸã¡ã®ä¸ã«ã¯ï¼Œã™ã§ã«æŒã£ã¦ã„るデータをã†ã¾ã移行ã—ãŸã„ã¨ã„ã†æ–¹ã‚‚多ã„ã‹ã¨æ€ã„ã¾ã™ï¼Ž Hive ã«ã¯ç¾çŠ¶ MySQL ã® mysqldump ã®ã‚ˆã†ãªæ©Ÿèƒ½ãŒæä¾›ã•ã‚Œã¦ã„ãªã„ã®ã§ãƒ
Hiveã®ãƒ‘ーティションã¨ã‚«ãƒ©ãƒ è¿½åŠ ã«ãƒãƒžã‚‹ - ãªãœã‹æ•°å¦è€…ã«ã¯ãƒ¯ã‚¤ãƒ³å¥½ããŒå¤šã„
ã¾ãšã¯æ™®é€šã«TSVå½¢å¼ã®ãƒ•ã‚¡ã‚¤ãƒ«ã‹ã‚‰ï¼ŒHiveã«LOAD DATAã—よã†ã¨ã—ã¾ã—ãŸï¼Ž #テスト用ã®ãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹ã‚’作る $ hive -e 'CREATE DATABASE test' OK Time taken: 3.668 seconds #テーブルãŒç©ºã§ã‚ã‚‹ã“ã¨ã®ç¢ºèª $ hive -database=test -e 'SHOW TABLES' Time taken: 2.944 seconds OK tab_name Time taken: 0.557 seconds # 3ã¤ã®ã‚«ãƒ©ãƒ ã‹ã‚‰ãªã‚‹ãƒ†ãƒ¼ãƒ–ãƒ«ã‚’ä½œæˆ $ hive -database=test -e "CREATE TABLE tmp_members (id INT, name STRING, country STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'" OK
Treasure Dataã§ã®å¤§å®¹é‡ãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ³ãƒãƒžãƒ¼ã‚¯ - Qiita
ã‚ãµã‚Œã‚‹ãƒ‡ãƒ¼ã‚¿ 会社ã§ã€Treasure Dataを使ã£ãŸåˆ†æžã‚·ã‚¹ãƒ†ãƒ を作ã£ã¦ã„ã‚‹ã€‚ã‚²ãƒ¼ãƒ æƒ…å ±ã‚’åŽé›†ã—ã¦ã€ãƒ¦ãƒ¼ã‚¶ãƒ¼ã®ä½“験å‘上ã«å½¹ç«‹ã¦ã‚‹ãŸã‚ã 。ãã®ãŸã‚ã€ãƒ¦ãƒ¼ã‚¶ãƒ¼ã®è¡Œå‹•ã‚’ç´°ã‹ã把æ¡ã™ã‚‹å¿…è¦ãŒã‚る。勢ã„データ容é‡ã¯å¢—ãˆã¦ã—ã¾ã†ã€‚åŠ ãˆã¦ã€ã‚ªãƒ³ãƒ©ã‚¤ãƒ³ã®ã‚²ãƒ¼ãƒ ã¯ã€ãƒ‘ッケージゲームã¨é•ã„売ã£ã¦çµ‚ã‚ã‚Šã§ã¯ãªãã€ãã®å¾Œä½•å¹´ã«ã‚‚渡ã£ã¦ã€ã‚µãƒ¼ãƒ“スをæä¾›ã™ã‚‹ã€‚ãã®ãŸã‚ã€ãƒ¦ãƒ¼ã‚¶ãƒ¼ã®è¡Œå‹•ãƒã‚°ã¯æ•°å„„件ã«é”ã™ã‚‹ã“ã¨ã‚‚çã—ããªã„。 Treasure Dataã§ã®ãƒã‚°åˆ†æž å…ˆã«æ›¸ã„ãŸãŒã€å¤§é‡ã®ãƒã‚°ã«å¯¾å¿œã™ã‚‹ãŸã‚ã€hadoopを利用ã—ãŸå•é¡Œè§£æ±ºãŒæ§˜ã€…ãªä¼æ¥ã‹ã‚‰æä¾›ã•ã‚Œå§‹ã‚ã¦ã„る。タイトルã«æœ‰ã‚‹Treasure Dataã‚‚ãã®ä¼æ¥ã®ä¸€ã¤ã 。ã“ã¡ã‚‰ã‹ã‚‰ã¯ã€ãƒã‚°ã‚’é€ã‚‹ã ã‘ã§hadoopã‚„hiveを用ã„ãŸåˆ†æžç’°å¢ƒã‚’æä¾›ã—ã¦ãれる。一方ã§ã€ã“ã¡ã‚‰ãŒåˆ†æžæ©Ÿæを用æ„ã™ã‚‹ã‚ã‘ã§ã¯ãªã„ãŸã‚ã€ã©ã®ãらã„ã®é€Ÿåº¦ã§åˆ†æžã§ãã‚‹ã‹ã‚ã‹ã‚‰ãªã„。特ã«è¤‡é›‘ãª
Hiveã¨Impalaã®ãŠã„ã—ã„ã¨ã“å–ã‚Š
1. Copyright © CELLANT Corp. All Rights Reserved. h t t p : / / w w w . c e l l a n t . j p / 1 1 Hiveã¨Impalaã®ãŠã„ã—ã„ã¨ã“å–ã‚Š 『プãƒã‚°ãƒ©ãƒŸãƒ³ã‚° Hiveã€ã€ŽHadoop 第3版〠刊行記念 Hadoopセミナー 2013年年7⽉月24⽇日 æ ªå¼ä¼šç¤¾ã‚»ãƒ©ãƒ³ R&D戦略略室 Â é ˆâ½¥ç”°å¹¸æ†² 3. Copyright © CELLANT Corp. All Rights Reserved. h t t p : / / w w w . c e l l a n t . j p / vï¶â€¯é ˆâ½¥ç”°å¹¸æ†²ï¼ˆ@sudabon) vï¶â€¯æ ªå¼ä¼šç¤¾ã‚»ãƒ©ãƒ³  R&D戦略略室  室⻑⾧長 vï¶â€¯çµŒæ´ lï¬â€¯1997〜～2004  NEC  ä¸å¤®ç ”究所  /  ãƒãƒƒãƒˆãƒ¯ãƒ¼ã‚¯ã®ç ”究 lï¬â€¯2005〜～2006  B
オライリーセミナー Hive入門 #oreilly0724
2. 自己紹介 •  嶋内　翔(ã—ã¾ã†ã¡ ã—ょã†) •  2011å¹´4月ã«Clouderaã®æœ€åˆã®æ—¥æœ¬äººç¤¾å“¡ã¨ã—ã¦å…¥ 社 •  テクニカルサãƒãƒ¼ãƒˆæ¥å‹™ã‚’メインã«ã€æ—¥æœ¬ã«ãŠã‘る技 è¡“ã«é–¢ä¿‚ã™ã‚‹æ¥å‹™å…¨èˆ¬ã‚’担当 2 3. Apache Hadoop Hadoopã¯ã€ 1.  å˜ä¸€ã‚µãƒ¼ãƒã§å‡¦ç†ã—ãã‚Œãªã„ã»ã©å¤§é‡ãªãƒ‡ãƒ¼ã‚¿ã‚’ 2.  コモディティサーãƒã‚’並ã¹ã‚‹ã“ã¨ã«ã‚ˆã£ã¦ 3.  分散処ç†åŠã³ä¿å˜ã™ã‚‹ãŸã‚ã® 4.  オープンソースソフトウェアã§ã™ 3 4. Hadoopシステムã®å…¨ä½“æ§‹æˆ 4 Hadoop 外部システムRDBMS Webサーãƒç‰ã® ãƒã‚°ã‚’生æˆã™ã‚‹ã‚µãƒ¼ãƒ æºå¸¯ç«¯æœ«ã® 通信ãƒã‚° APIアクセス ãƒã‚°åŽé›† テーブルã”㨠インãƒãƒ¼ãƒˆ 分散ãƒãƒƒãƒå‡¦ç† 機械å¦ç¿’ 外部システムAPIアクセス ユーザ DWH テーブルã”㨠エクスãƒãƒ¼ãƒˆ BIツール + JDBC/ODBC 検索 SQL 5. Ha
ã‚„ã£ãŸãƒ¼ï¼è¶…ã‚ュートãªHiveãƒã‚´ã‚’æ„›ã§ãªãŒã‚‰ãƒ–ラウジングã§ãã‚‹Chromeæ‹¡å¼µã§ããŸã‚ˆãƒ¼ï¼¼(^o^)ï¼ - ダメ人間オンライン
http://www.apache.org/ 事ã®ç™ºç«¯ @tagomorisã•ã‚“ãŒHiveãƒã‚´ãŒãƒ—リントã•ã‚ŒãŸTシャツをç€ã¦ãŸ ↓ 背ä¸å´ãŒç‹‚æ°—ã«æº€ã¡ã¦ã‚‹ã¨æ„Ÿã˜ãŸ ↓ kowai ↓ 夢ã«å‡ºã¦ã㟠↓ èµ·ããŸã‚‰Hiveãƒã‚´ãŒã¨ã¦ã‚‚æ„›ãŠã—ãæ„Ÿã˜ã‚‹ã‚ˆã†ã«ãªã£ã¦ãŸ ↓ ã‚‚ã£ã¨é•·ã„時間眺ã‚ã¦ã„ãŸã„ï¼ â†“ ã©ã®ãƒšãƒ¼ã‚¸è¦‹ã¦ã‚‚Hiveãƒã‚´ãŒç›®ã«å…¥ã‚‹ã‚ˆã†ã«ã™ã‚Œã°ãˆãˆã‚“ã‚„ï¼ï¼ @tagomorisã•ã‚“ãŒç€ã¦ãŸTシャツã®ãƒ‡ã‚¶ã‚¤ãƒ³ã¯ã“ã¡ã‚‰ Hadoop Conference Japan 2013 Winter ã«ã„ã£ã¦ããŸï¼†ã—ゃã¹ã£ã¦ã㟠- tagomorisã®ãƒ¡ãƒ¢ç½®ãå ´ ⊂二二二（ ＾ω＾）二⊃ dameninngenn/chrome-ext-bu-------n · GitHub ã‚„ã£ãŸãƒ¼ï¼ã‹ãƒ¼ã‚ーã„ーã„ーï¼ï¼ ãŸã 大é‡ã«ç™ºç”Ÿã•ã›ã™ãŽã‚‹ã¨CPUãŒã‚¦ã‚©ãƒ¼ãƒ¼ãƒ¼ã£ã¦ãªã£ã¦ãƒ•ã‚¡ãƒ³ãŒãƒ•ã‚¡ãƒ¼ãƒ¼ãƒ¼ãƒ¼ãƒ¼ã£ã¦ãªã‚‹ã€‚。 余談
Impala データサイエンティストã®ãŸã‚㮠高速大è¦æ¨¡åˆ†æ•£åŸºç›¤ #tokyowebmining
Tokyo Webmining #25 ã§ç™ºè¡¨ã—ãŸã¨ãã®è³‡æ–™ã§ã™ã€‚メインã¯ãƒ‡ãƒ¢ã®æ–¹ã§ã™Read less
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
GitHub - edwardcapriolo/hive_test: Unit test framework for hive and hive-service
HiveQLUnit
GitHub - kawaa/Beetest: A super simple utility for testing Apache Hive scripts locally for non-Java developers.
GitHub - maropu/hivemall-spark: A Hivemall wrapper for Spark
CDH4.2 Recursive directory searches in Hive partitioned external tables - Grokbase
{{#tags}}- {{label}}
{{/tags}}