A Benchmark Test on Presto, Spark Sql and Hive on Tez
Prestoã€Spark SQLã¨Hive on Tezã®æ€§èƒ½ã«é–¢ã—ã¦ã€æ•°ä¸‡ä»¶ã‹ã‚‰æ•°å億件ã¾ã§ã®ãƒ‡ãƒ¼ã‚¿ä¸Šã«ã€å¸¸ç”¨ã‚¯ã‚¨ãƒªãƒ‘ターンã®å®Ÿè¡Œã‚¹ãƒ”ードãªã©ã‚’検証ã—ã¦ã¿ãŸã€‚ We conducted a benchmark test on mainstream big data sql engines including Presto, Spark SQL, Hive on Tez. We focused on the performance over medium data (from tens of GB to 1 TB) which is the major case used in most services. Read less
コミッタãŒèªžã‚‹ã€Hadoopã‚„Sparkã€2016å¹´ã¯ã©ã†ãªã‚‹ï¼Ÿ | gihyo.jp
ã‚ã‘ã¾ã—ã¦ã€ãŠã‚ã§ã¨ã†ã”ã–ã„ã¾ã™ã€‚ 例年ã€Apache Hadoopã‚’ä¸å¿ƒã«ä¸¦åˆ—分散処ç†ãƒŸãƒ‰ãƒ«ã‚¦ã‚§ã‚¢ã®å‹•å‘や展望ã«ã¤ã„ã¦ã”紹介ã—ã¦ã„ã¾ã™ã€‚ 今年ã¯ã€NTTデータã«æ‰€å±žã™ã‚‹ é¯µå‚ æ˜Žï¼ˆApache Hadoopコミッタ） 岩崎 æ£å‰›ï¼ˆApache Hadoop開発者ã€HTraceコミッタ） é–¢ å …å¾ï¼ˆApache Yetusコミッタ） 猿田 浩輔（Apache Sparkコミッタ） ã®4åã«ã‚ˆã‚‹å¯¾è«‡ã‹ã‚‰ã¾ã¨ã‚ãŸã€Hadoopã¨Sparkã‚’ä¸å¿ƒã«2015å¹´ã®æŒ¯ã‚Šè¿”ã‚Šã¨2016å¹´ã®å±•æœ›ã‚’ã”紹介ã—ã¾ã™ã€‚ 対談ã®æ¨¡æ§˜ï¼ˆ1ï¼‰é¯µå‚ æ˜Žæ°ï¼ˆå·¦ï¼‰ã¨å²©å´Ž æ£å‰›æ° 2015年を振り返る 2015å¹´ã®æ–°æ˜¥ç‰¹åˆ¥ä¼ç”»ã§ã¯ã€ŒTezã‚„Sparkãªã©ã®æ–°ã—ã„処ç†åŸºç›¤ã®æ´»ç”¨ãŒé€²ã‚€ã“ã¨ã«ãªã‚‹ã€ã¨ç´¹ä»‹ã—ã¾ã—ãŸã€‚実際ã«æŒ¯ã‚Šè¿”ã£ã¦ã¿ã‚‹ã¨ã€Apache Sparkã¯ã‹ãªã‚Šç››ã‚Šä¸ŠãŒã‚Šã¾ã—ãŸã€‚ Apache Sparkã¯ã€å¹´åˆã«ãƒãƒ¼ã‚¸ãƒ§ãƒ³
Google BigQuery Benchmark — DoIT International :: Helping businesses adopt, manage and enhance cloud solutions from Google and Zendesk
By Vadim Solovey, Google Developer Expert.Recently, I came across an interesting benchmark of BigData systems based on "A Comparison of Approaches to Large-Scale Data Analysis" by Pavlo et al. (SIGMOD 2009). Based on the benchmark methodology, the APMLab guys from Berkeley University developed an open-source software that allows anyone to run this benchmark using public cloud (AWS in this case).Th
Hive on Tez ã‚’ CDH ã§å‹•ã‹ã™ - Qiita
Tezã¨ã¯ ã¾ãšã¯Tezã®èª¬æ˜Žã§ã™ã€‚ Tezã¯YARN上ã§å‹•ä½œã™ã‚‹ãƒãƒƒãƒå‡¦ç†ç”¨ã®ãƒ•ãƒ¬ãƒ¼ãƒ ワークã§ã€HortonworksãŒä¸»å°Žã—ã¦é–‹ç™ºã—ã¦ã„ã¾ã™ã€‚ Tezã«è¿‘ã„ä½ç½®ä»˜ã‘ã®ã‚‚ã®ã«ã¯ã€ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ 処ç†ç”¨ã®SliderãŒã‚ã‚Šã¾ã™ã€‚ Hortonworksã¯Sliderã®ä¸Šã§Stormã‚’å‹•ã‹ã™ã“ã¨ã‚’計画ã—ã¦ã„ã¾ã™ã€‚ Hive on Tezã¨ã¯ Hive on Tezã¯ã€ä¸Šè¨˜ã®Tezã®ä¸Šã§Hiveã‚’å‹•ã‹ã™ã‚‚ã®ã§ã™ã€‚ Hive 0.13ã‹ã‚‰Tez対応ã®ãƒ‘ッãƒãŒå…¥ã£ã¦ã„ã¦ã€å‹•ã‹ã™æº–å‚™ã¯ã§ãã¦ã„ã¾ã™ã€‚ ã—ã‹ã—ã€Clouderaã¯CDH 5.2ã§Hive 0.13ã«å¯¾å¿œã—ãŸã‚‚ã®ã®ã€Tezをパッケージングã—ã¾ã›ã‚“ã§ã—ãŸã€‚ ãã®ãŸã‚ã€ãã®ã¾ã¾ã®CDHã§ã¯Hive on TezãŒå‹•ãã¾ã›ã‚“。 Hive on Tezã‚’CDHã§å‹•ã‹ã™æ–¹æ³• Tezã¯ãƒ©ã‚¤ãƒ–ラリã§ã‚ã‚‹ãŸã‚ã€CDHã®Hive 0.13ã¨çµ„ã¿åˆã‚ã›ã‚Œã°å‹•ä½œã™
SQL on Hadoop 比較検証 ã€2014月11æ—¥ã«ãŠã‘る検証レãƒãƒ¼ãƒˆã€‘
Impala Meetup 2014/10/31 @Tokyo 講演資料 ã€æ³¨æ„äº‹é …ã€‘ 本資料ã§ç´¹ä»‹ã—ã¦ã„る検証çµæžœã¯2014年当時ã®ã‚‚ã®ã§ã™ã€‚当該ソフトウェアã¯æˆé•·ã‚„改善ãŒæ—©ãã€ç¾æ™‚点ã®ãƒãƒ¼ã‚¸ãƒ§ãƒ³ã§ã¯å¤§ããç•°ãªã‚‹æ©Ÿèƒ½ã‚„性能ã¨ãªã£ã¦ã„ã¾ã™ã€‚ SQL on Hadoopã®æœ€æ–°æƒ…å ±ã«åŸºã¥ãサービスやシステムインテグレーションã«ã”興味をãŠæŒã¡ã®æ–¹ã¯ã€NTTデータ 基盤システム事æ¥æœ¬éƒ¨ OSSプãƒãƒ•ã‚§ãƒƒã‚·ãƒ§ãƒŠãƒ«ã‚µãƒ¼ãƒ“ス（電åメール： hadoop [AT] kits.nttdata.co.jp） ã«ã”相談ãã ã•ã„。Read less
Elastic MapReduce Now Supports Hive 13 | Amazon Web Services
AWS News Blog Elastic MapReduce Now Supports Hive 13 I am pleased to announce that Elastic MapReduce now supports version 13 of Hive. Hive is a great tool for building and querying large data sets. It supports the ETL (Extract/Transform/Load) process with some powerful tools, and give you access to files stored on your EMR cluster in HDFS or in Amazon Simple Storage Service (Amazon S3). Programmat
Cloudera Blog
We are thrilled to announce the general availability of the Cloudera AI Inference service, powered by NVIDIA NIM microservices, part of the NVIDIA AI Enterprise platform, to accelerate generative AI deployments for enterprises. This service supports a range of optimized AI models, enabling seamless and scalable AI inference. Background The generative AI landscape is evolving […] Read blog post
Prestoã¨ã‹Ansibleã¨ã‹ãã®è¾ºã®è©±ã‚’軽ã書ã„ã¦ã¿ã‚‹ - wyukawa's diary
今日ã¯Prestoã¨ã‹Ansibleã¨ã‹ãã®è¾ºã®è©±ã‚’軽ã書ã„ã¦ã¿ã‚ˆã†ã¨æ€ã„ã¾ã™ã€‚çªã£è¾¼ã‚“ã 話ãŒå‡ºæ¥ã‚‹ã‚ã‘ã§ã¯ãªã„ã®ã§ã‚ã—ã‹ã‚‰ãšã€‚ 僕ã®ã¨ã“ã‚ã®ç’°å¢ƒã§ã¯Prestoを使ã£ã¦ã„ã¦ã€Prestoã¯DataNodeã‚„NodeManagerã¨åŒå±…ã—ã¦ã¾ã™ã€‚主ãªãƒ¦ãƒ¼ã‚¹ã‚±ãƒ¼ã‚¹ã¯ã‚¢ãƒ‰ãƒ›ãƒƒã‚¯ã‚¯ã‚¨ãƒªã®å®Ÿè¡Œã§ã™ã€‚ã¨ã‚るレãƒãƒ¼ãƒˆã‚’作りãŸã„ã£ã¦ãªã£ãŸã¨ãã«ãƒ‡ãƒ¼ã‚¿ã®ä¸èº«ã‚’ãƒã‚§ãƒƒã‚¯ã™ã‚‹ã®ã«ä½¿ã„ã¾ã™ã€‚従æ¥ã ã¨ã“ã‚ŒãŒHiveã ã£ãŸã®ã§ã™ãŒã€Hiveã ã¨MapReduceã«ãªã£ã¦é…ã„ã®ã§ï¼ˆãƒãƒ¼ã‚«ãƒ«ãƒ¢ãƒ¼ãƒ‰ã§æ¸ˆã‚€å ´åˆã‚‚ã‚ã‚‹ã‘ã©ï¼‰ã€ãã®ç‚¹Prestoã¯æ—©ãã¦ã„ã„ã§ã™ã€‚ãŸã ã“ã‚Œã¯åƒ•ã®ç’°å¢ƒãŒã‚¹ãƒ¢ãƒ¼ãƒ«ãƒ‡ãƒ¼ã‚¿ã ã‹ã‚‰ã£ã¦ã„ã†ã®ã‚‚ã‚ã£ã¦ã€åœ§ç¸®æ¸ˆã¿æ•°ç™¾GBã®ãƒ‡ãƒ¼ã‚¿ã«å¯¾ã—ã¦selectã‹ã‘ã‚‹ã¨ã‹ã ã¨Prestoã¨ã„ãˆã©ã‚‚é…ããªã‚‹ã¨æ€ã„ã¾ã™ã€‚ã‚ã¨ãªã«ã’ã«è‰¯ã„ã®ãŒPresto CLI経由ã ã¨ã‚«ãƒ©ãƒ åãŒè¡¨ç¤ºã•ã‚Œã‚‹ã®ã§ã©ã®ãƒ‡ãƒ¼ã‚¿ãŒã©ã®ã‚«ãƒ©ãƒ ãªã®ã‹ã™ã分ã‹
War of the Hadoop SQL engines. And the winner is ...? - Sonra
War of the Hadoop SQL engines. And the winner is …? You may have wondered why we were quiet over the last couple of weeks? Well, we locked ourselves into the basement and did some research and a couple of projects and PoCs on Hadoop, Big Data, and distributed processing frameworks in general. We were also looking at Clickstream data and Web Analytics solutions. Over the next couple of weeks we wil
hiveserverã®ãƒ¢ãƒ‹ã‚¿ãƒªãƒ³ã‚° - wyukawa's diary
hiveserver(not hiveserver2)ã®èª¿åãŒæ‚ªãOutOfMemoryãŒç™ºç”Ÿã™ã‚‹ã“ã¨ã‚‚ã‚ã‚‹ã®ã§ãƒ¡ãƒ¢ãƒªã‚’増やã—ã¤ã¤ãƒ¢ãƒ‹ã‚¿ãƒªãƒ³ã‚°ã‚‚強化ã—ã¦ã¿ã¦ã„る。 メモリを増やã—ã¦ã‹ã¤GCãƒã‚°ã‚’å–ã‚‹ãŸã‚ã«hive-env.shã«ä¸‹è¨˜ã‚’è¿½åŠ ã€‚ã“ã®ä¾‹ã§ã¯hiveserverã®ãƒ’ープを2GBã«ã—ã¦ã„る。 if [ "$SERVICE" = "hiveserver" ]; then export HADOOP_HEAPSIZE=2048 export HIVE_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps ..." fiãã—ã¦hadoop-env.shã§HIVE_OPTSã‚’å—ã‘ã¨ã‚‹ã‚ˆã†ã«ã™ã‚‹ã€‚ export HADOOP_OPTS="${HIVE_OPTS}"ãªã‚“ã§hadoop-env.shã‚’ã„ã˜ã‚‹å¿…è¦ãŒã‚ã‚‹ã‹ã£ã¦ã„ã‚‹ã¨h
Hivemall talk@Hadoop summit 2014, San Jose
This document discusses Hivemall, a scalable machine learning library for Apache Hive. It provides concise summaries of machine learning algorithms as user-defined functions that can run on large datasets in Hive. The document outlines the motivation for Hivemall, what algorithms it supports, how to use it to perform tasks like data preparation, training models, and prediction, and how it handles
Hiveã®ãƒ¡ã‚¿ã‚¹ãƒˆã‚¢ã«ã¤ã„ã¦ãŠã•ã‚‰ã„ – OpenGroove
Hiveã®ãƒ¡ã‚¿ã‚¹ãƒˆã‚¢ã¯ã€ä½•ã‹ã«ã¤ã‘ã¦ã€ãƒãƒžã‚‹ã€‚分ã‹ã£ã¦ã„ã‚‹ã¤ã‚‚ã‚Šã§åˆ†ã‹ã£ã¦ãªã‹ã£ãŸã“ã¨ã‚‚ã€æœªã ã«ã‚る。ã“ã®ãƒ–ãƒã‚°å†…ã‚„ä»–ã®å ´æ‰€ã®ã‚ã¡ã“ã¡ã«æ°—ã¥ã„ãŸã“ã¨ã‚’書ã„ã¦ã„ã‚‹ãŒã€æ•£ä¹±ã—ã¦ã„ã‚‹ã®ã§ã“ã“ã§ã¾ã¨ã‚ã¦ã¿ã‚‹ã€‚ 以下ã€éŽåŽ»è¨˜äº‹ã‹ã‚‰ã®å¼•ç”¨ã€‚ メタストアサービスã«ã¤ã„㦠Hiveã§ã¯ãƒ¡ã‚¿ã‚¹ãƒˆã‚¢ã‚µãƒ¼ãƒ“スã¨å‘¼ã°ã‚Œã‚‹ã‚‚ã®ã‚’利用ã™ã‚‹ã€‚メタストアサービスã¯ãƒ†ãƒ¼ãƒ–ル定義ã€ã“ã“ã§ã¯ã€Œãƒ†ãƒ¼ãƒ–ルã¨HDFS上ã®ãƒ‡ãƒ¼ã‚¿ã®å¯¾å¿œæƒ…å ±ã€ã‚’管ç†ã™ã‚‹ã€‚メタストア自体ã¯RDBMSを使用ã—ã¦ãã®æƒ…å ±ã‚’ç®¡ç†ã—ã¦ã„る。メタストアサービスã¯ä»¥ä¸‹ã®3種類ãŒã‚ã‚Šã€è¦ä»¶ã«ã‚ˆã£ã¦ä½¿ã„分ã‘る。 組ã¿è¾¼ã¿ãƒ¡ã‚¿ã‚¹ãƒˆã‚¢ ã‚‚ã£ã¨ã‚‚手軽ã«åˆ©ç”¨ã§ãる形態。Hiveã«ã¯Javaã§å®Ÿè£…ã•ã‚ŒãŸDerbyã¨ã„ã†DBãŒçµ„ã¿è¾¼ã¾ã‚Œã¦ãŠã‚Šã€ãã®DBをメタストアリãƒã‚¸ãƒˆãƒªã¨ã—ã¦åˆ©ç”¨ã™ã‚‹ã€‚hiveコマンド起動ã«ã‚ˆã‚ŠDerbyã‚‚èµ·å‹•ã™ã‚‹ã€‚åŒæ™‚ã«ä¸€äººã®ãƒ¦ãƒ¼ã‚¶ãƒ¼ã—ã‹Hiveを利用ã§ããªã„ã€
ãƒžãƒ¡ç ”ã‚¤ãƒ³ã‚¿ãƒ“ãƒ¥ãƒ¼ã‚·ãƒªãƒ¼ã‚ºç¬¬1回～産æ¥æŠ€è¡“ç·åˆç ”ç©¶æ‰€ã€€æ²¹äº•èª ç ”ç©¶å“¡ï¼šãƒžãƒ¼ã‚±ã‚¿ãƒ¼ã‚‚ã‚¯ã‚¨ãƒªè¨€èªžã‚’ã‚„ã‚Šã¾ã—ょã†ï¼ï½ž - ãƒžãƒ¼ã‚±ãƒ†ã‚£ãƒ³ã‚°ãƒ¡ãƒˆãƒªãƒƒã‚¯ã‚¹ç ”ç©¶æ‰€/MARKETING METRICS Lab. – コミュニケーションã®æœ€é©åŒ–を考ãˆã‚‹ã€‚ãƒžãƒ¼ã‚±ãƒ†ã‚£ãƒ³ã‚°ãƒ¡ãƒˆãƒªãƒƒã‚¯ã‚¹ç ”ç©¶æ‰€
ãƒžãƒ¼ã‚±ãƒ†ã‚£ãƒ³ã‚°ãƒ¡ãƒˆãƒªãƒƒã‚¯ã‚¹ç ”ç©¶æ‰€ã§ã¯ãƒ‡ãƒ¼ã‚¿åˆ†æžã«é–¢ã‚ã‚‹ç ”ç©¶è€…ã‚„å®Ÿå‹™å®¶ã®æ–¹ã€…ã‚’ãŠæ‹›ãã—インタビューシリーズã«ã‚‚å–り組むé‹ã³ã¨ãªã‚Šã¾ã—ãŸã€‚ 当サイトã«ãŠè¶Šã—下ã•ã‚‹èªè€…ã®çš†æ§˜ã®ãƒ‡ãƒ¼ã‚¿æ´»ç”¨ã«å°‘ã—ã§ã‚‚ãŠå½¹ã«ç«‹ã¦ã‚Œã°å¹¸ã„ã§ã™ã€‚記念ã™ã¹ã第1回目ã¯ç”£æ¥æŠ€è¡“ã®å¹…広ã„分野ã®ç ”究を行ã†å›½å†…最大級ã®å…¬çš„ç ”ç©¶æ©Ÿé–¢ã§ã‚る産æ¥æŠ€è¡“ç·åˆç ”究所ã®æ²¹äº•èª ç ”ç©¶å“¡ã‚’ãŠè¿Žãˆã—ã¦ãƒ“ッグデータを扱ã†åŸºç›¤æŠ€è¡“や機械å¦ç¿’ã«ã¤ã„ã¦ãŠè©±ã‚’伺ã„ã¾ã™ã€‚ 2003å¹´èŠæµ¦å·¥æ¥å¤§å¦å·¥å¦éƒ¨å’æ¥ã€‚奈良先端科å¦æŠ€è¡“大å¦é™¢å¤§å¦åšå£«å‰æœŸãƒ»å¾ŒæœŸèª²ç¨‹ã‚’ãã‚Œãžã‚ŒçŸç¸®ä¿®äº†ã—2009å¹´3月åšå£«ï¼ˆå·¥å¦ï¼‰ã‚’å–得。2003å¹´7月IPA未è¸ãƒ¦ãƒ¼ã‚¹ã€ã‚¹ãƒ¼ãƒ‘ークリエータèªå®šã€‚2009年日本å¦è¡“æŒ¯èˆˆä¼šç‰¹åˆ¥ç ”ç©¶å“¡ (PD å·¥å¦)ã€æ—©ç¨²ç”°å¤§å¦ITç ”ç©¶æ©Ÿæ§‹å®¢å“¡ç ”ç©¶å“¡ï¼ˆå…¼ä»»ï¼‰ã€‚2010å¹´4月より独立行政法人産æ¥æŠ€è¡“ç·åˆç ”究所 æƒ…å ±æŠ€è¡“ç ”ç©¶éƒ¨é–€ã‚µãƒ¼ãƒ“ã‚¹ã‚¦ã‚§ã‚¢ç ”ç©¶ã‚°ãƒ«ãƒ¼ãƒ—ç ”ç©¶å“¡ã€ç¾åœ¨ã«è‡³ã‚‹ã€‚ [
Cloudera Blog
We are thrilled to announce the general availability of the Cloudera AI Inference service, powered by NVIDIA NIM microservices, part of the NVIDIA AI Enterprise platform, to accelerate generative AI deployments for enterprises. This service supports a range of optimized AI models, enabling seamless and scalable AI inference. Background The generative AI landscape is evolving […] Read blog post
Yahoo Betting on Apache Hive, Tez, and YARN
Low-latency SQL queries, Business Intelligence (BI), and Data Discovery on Big Data are some of the hottest topics these days in the industry with a range of solutions coming to life lately to address them as either proprietary or open-source implementations on top of Hadoop. Some of the popular ones talked about in the Big Data communities are Hive, Presto, Impala, Shark, and Drill. Yahoo has tr
Cloudera Blog
We are excited to announce the acquisition of Octopai, a leading data lineage and catalog platform that provides data discovery and governance for enterprises to enhance their data-driven decision making. Cloudera’s mission since its inception has been to empower organizations to transform all their data to deliver trusted, valuable, and predictive insights. With AI and […] Read blog post
ãŠçŸ¥ã‚‰ã›
障害
ランã‚ング
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
stripe/herringbone · GitHub - Tools for working with parquet, impala, and hive
HPE Ezmeral: Uncut Blog | HPE Blogs, Discussions and Forums Community
HPE Ezmeral: Uncut Blog | HPE Blogs, Discussions and Forums Community
Cloudera Blog
{{#tags}}- {{label}}
{{/tags}}