Cloudera Blog
We are thrilled to announce the general availability of the Cloudera AI Inference service, powered by NVIDIA NIM microservices, part of the NVIDIA AI Enterprise platform, to accelerate generative AI deployments for enterprises. This service supports a range of optimized AI models, enabling seamless and scalable AI inference. Background The generative AI landscape is evolving […] Read blog post
Apache Spark @Scale: A 60 TB+ production use case
Facebook often uses analytics for data-driven decision making. Over the past few years, user and product growth has pushed our analytics engines to operate on data sets in the tens of terabytes for a single query. Some of our batch analytics is executed through the venerable Hive platform (contributed to Apache Hive by Facebook in 2009) and Corona, our custom MapReduce implementation. Facebook has
hive ã§è«–ç†å·® - ritchiekotzen's blog
hive 㯠where exists をサãƒãƒ¼ãƒˆã—ã¦ã„ãªã„。ã§ã€where not exists もサãƒãƒ¼ãƒˆã—ã¦ãªã„。（if not exists ã¯ã‚µãƒãƒ¼ãƒˆï¼‰ ã§ã€è«–ç†å·®ï¼ˆé›†åˆAã¨é›†åˆBãŒã‚ã‚‹ã¨ãã€Aã®ãƒ¡ãƒ³ãƒã§ã‹ã¤Bã®ãƒ¡ãƒ³ãƒã§ãªã„ã‚‚ã®ã‚’抽出）をã©ã†å®Ÿç¾ã™ã‚‹ã‹ï¼Ÿ 以下ã§ã§ããŸã€‚ select aid from ( select a.id as aid, b.id as bid from a left outer join b on (a.id = b.id) ) abjoin where abjoin.bid is NULL ã¡ãªã¿ã«ã€is NULL ã¯ã‚µãƒãƒ¼ãƒˆã§ã™ã€‚
Ruby scripting in Hive Query Language | GREE Engineering
ã“ã‚“ã«ã¡ã¯ã€‚Web Game事æ¥çµ±æ‹¬æœ¬éƒ¨ データ基盤ãƒãƒ¼ãƒ ã® lan ã§ã™ã€‚ Advent Calendar 3日目ã®ä»Šæ—¥ã¯ã€Hadoopã®ä¸Šã«ä¹—るデータウェアãƒã‚¦ã‚¹ã§ã‚ã‚‹Apache Hiveã«ã¤ã„ã¦ã€è©±ã‚’ã•ã›ã¦é ‚ããŸã„ã¨æ€ã„ã¾ã™ã€‚グリーã§ã¯ã€Hadoopをレãƒãƒ¼ãƒ†ã‚£ãƒ³ã‚°ã‚„大è¦æ¨¡ãƒãƒƒãƒå‡¦ç†ãªã©ã«ä½¿ã£ã¦ã„ã¾ã™ã€‚ç¾åœ¨ã¯ã€Apache Hive 0.12.0版を導入ã—ã¦ãŠã‚Šã¾ã™ã€‚OSSコミュニティã‹ã‚‰æœ€æ–°ã®æˆæžœã‚’ã§ãã‚‹ã ã‘æ—©ãå–り込んã§ã„ããŸã„ã¨è€ƒãˆã¦ã„ã‚‹ã‹ã‚‰ã§ã™ã€‚ レãƒãƒ¼ãƒˆãªã©ã®ãŸã‚ã«ç¤¾å†…用ã®UDFライブラリも作ã£ã¦ã„ã¾ã™ã€‚ã—ã‹ã—ã€ãã‚Œã§ã‚‚サãƒãƒ¼ãƒˆã—ã«ãã„シーンãŒã¾ã ã¾ã ã‚ã‚Šã¾ã™ã€‚ãã®ãŸã‚ã«ã€å°ã•ãªå°ã•ãªãƒãƒƒã‚ングをã—ã¦ã€HQL内ã«Rubyスクリプティングを利用ã§ãる機能を開発ã—ã¾ã—ãŸã€‚Redisã®Luaスクリプティングã®ã‚ˆã†ãªã‚‚ã®ã‚’イメージã™ã‚‹ã¨ã‚ã‹ã‚Šã‚„ã™ã„ã‹ã‚‚ã—ã‚Œã¾ã›ã‚“。 今回紹介ã™ã‚‹ã®
Hive/Hivemallを利用ã—ãŸåºƒå‘Šã‚¯ãƒªãƒƒã‚¯ã‚¹ãƒ«ãƒ¼çŽ‡(CTR)ã®æŽ¨å®š - Qiita
Hadoop Advent Calendar 2013 2013 12/25ã®Xmasエントリã§ã™ã€‚ 本記事ã§ã¯ç§ãŒé–‹ç™ºã—ã¦ã„ã‚‹Hadoop/Hive上ã§å‹•ä½œã™ã‚‹æ©Ÿæ¢°å¦ç¿’ライブラリã®Hivemallã«ã¤ã„ã¦ã€KDD Cup 2012, Track 2ã®ãƒ‡ãƒ¼ã‚¿ã‚»ãƒƒãƒˆã‚’用ã„ã¦åˆ©ç”¨æ–¹æ³•ã‚’解説ã—ã¾ã™ã€‚ https://github.com/myui/hivemall 基本的ã«ãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆã®Wikiサイトã«ã‚ã‚‹KDDCup 2012 track 2 CTR predictionã®èª¬æ˜Žã‚’ä¸å¯§ã«ã—ãŸã‚‚ã®ã§ã™ã€‚a9a binaryã‚„news20 binaryã®æ–¹ãŒã‚ˆã‚Šã‚·ãƒ³ãƒ—ルã®ä¾‹ã§ã™ã®ã§ã€ãã¡ã‚‰ã‚‚å‚考ã«ã—ã¦é ‚ã‘ã‚Œã°ã¨æ€ã„ã¾ã™ã€‚ KDD Cup 2012, Track 2ã®CTR推定タスク ã“ã®ã‚¿ã‚¹ã‚¯ã¯ä¸Žãˆã‚‰ã‚ŒãŸã‚»ãƒƒã‚·ãƒ§ãƒ³æƒ…å ±ï¼ˆãƒ¦ãƒ¼ã‚¶å±žæ€§ã¨åºƒå‘Šã®å±žæ€§ï¼‰ã‚’ã‚‚ã¨ã«ã€æ¤œç´¢ã‚¨ãƒ³ã‚¸ãƒ³ã®åºƒå‘Šã‚¯ãƒªãƒƒã‚¯çŽ‡(Click-Th
Hadoop + Hiveã‹ã‚‰UDFã§ãŠæ‰‹è»½ã«æ©Ÿæ¢°å¦ç¿’を回ã›ã‚‹ãƒ©ã‚¤ãƒ–ラリ"hivemall"ãŒãƒªãƒªãƒ¼ã‚¹ã•ã‚ŒãŸã¨ã®ã“㨠- 渋谷駅å‰ã§åƒãデータサイエンティストã®ãƒ–ãƒã‚°
TL上ã«ã‚‚ã®ã™ã”ーã刺激的ãªæƒ…å ±ãŒé£›ã³è¾¼ã‚“ã§ããŸã®ã§ã€æ€ã‚ãšã”紹介。 一部ã®æ–¹ã«ã¯ãŠä¼ãˆã—ã¦ãŠã‚Šã¾ã™ãŒã€hive/hadoop上ã§ã‚¹ã‚±ãƒ¼ãƒ©ãƒ–ルã«æ©Ÿæ¢°å¦ç¿’（今ã®ã¨ã“クラス分類ã€å›žå¸°åˆ†æžï¼‰ã‚’è¡Œã†ãƒ©ã‚¤ãƒ–ラリをオープンソース(LGPL)ã§ãƒªãƒªãƒ¼ã‚¹ã—ã¾ã—ãŸã€‚mahoutç‰ã‚ˆã‚Šæ‰±ã„ã‚„ã™ã„&スケーラブルãªã¯ãšã€‚ https://t.co/eBGmoW66V0— myui (@myui) 2013, 9月 30 ã¨ã„ã†ã“ã¨ã§ã€ã”本人ã®ãƒ„イートã«ã‚‚ã‚るよã†ã«GitHubã«ä¸€å¼å…¨ã¦ä¸ŠãŒã£ã¦ã„ã¾ã™ã€‚ライセンスもLPGLãªã®ã§è‡ªç”±ã«ä½¿ãˆã‚‹ã¨ã„ã†ç´ 晴らã—ã•ã€‚ã“ã‚Œè·å ´ã®åˆ†æžãƒžã‚·ãƒ³ã«ã‚‚cloneã—よã†ã‹ãªï¼ˆç¬‘）。 myui/hivemall Wikiページを見るã¨ã€ã“ã®ãƒ©ã‚¤ãƒ–ラリã®ä½¿ã„æ–¹ãŒæ›¸ã„ã¦ã‚ã‚‹ã‚“ã§ã™ãŒã€‚。。何ãŒå‡„ã„ã£ã¦ã€ä½•ã¨Hiveクエリを書ãè¦é ˜ã§æ©Ÿæ¢°å¦ç¿’ã¾ã‚ã‚Šã®æ¼”ç®—ã‚’å…¨ã¦å®Ÿè¡Œã§ãã¦ã—ã¾ã†ï¼ã¨ã„ã†ã¨ã“ã‚。ã¡ã‚‡ã£
1å°ã§HBase, Hive, Pig, HUE(æ—§Cloudera Desktop)を試ã—ã¦ã¿ã‚‹(CentOS + Cloudera)
å¿…è¦ãªã‚‚㮠・Linux(CentOS)+Cloudera版Hadoop環境CDH3(1å°) → 構築方法ã¯ã“ã¡ã‚‰ ・インターãƒãƒƒãƒˆæŽ¥ç¶š ・Cloudera社ã®HBase,Hive,Pig,HUEã®ãƒ‘ッケージ(yumã§ã‚¤ãƒ³ã‚¿ãƒ¼ãƒãƒƒãƒˆã‹ã‚‰ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«) 作æ¥æ‰‹é † 以下ã®ã€1. HBase, 2. Pig, 3. Hive, 4. HUE(æ—§Cloudera Desktop)ã®æ‰‹é †ã¯ã€ç‹¬ç«‹ã—ã¦è©¦ã™ã“ã¨ãŒã§ãã¾ã™ã—ã€ä¸€ã¤ã®ç’°å¢ƒã§ã¾ã¨ã‚ã¦è©¦ã™ã“ã¨ã‚‚ã§ãã¾ã™ã€‚(ãã‚Œãžã‚Œãƒ‡ãƒ¼ã‚¿ã®ç®¡ç†ã¯åˆ¥ã€…ã§ã™ã€‚) ã“ã“ã§ã¯ã€Cloudera社ã®CDH3リãƒã‚¸ãƒˆãƒªãŒå¿…è¦ã§ã™ã®ã§ã€æ—¢ã«è¨å®šã•ã‚Œã¦ã„ãªã„ã‹ç¢ºèªã—ã¾ã™ã€‚ ls /etc/yum.repos.d/ ã§ã€cloudera-cdh3.repoãŒå˜åœ¨ã™ã‚Œã°ã€OKã®ã¯ãšã§ã™ã€‚ç„¡ã„å ´åˆã¯ã€CDHã®hadoopã®ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«æ–¹æ³•ã‚’確èªã—ã¦ãã ã•ã„。 1. HBase CDH
CDH4 YARNã§Hive - marblejediary
å‹•ã‹ãªã„ã¨ã„ã†å™‚ãŒã‚ã£ãŸã®ã§è©¦ã—ã¦ã¿ã¾ã—ãŸã€‚別ãªã¨ã“ã‚ã§ã¯ã¾ã‚Šã¾ã—ãŸãŒã€å‹•ã„ãŸæ°—ãŒã—ã¾ã™ã€‚ãªã«ã’ã«å®Œå…¨åˆ†æ•£ç’°å¢ƒã§ã®Hiveã¯åˆã‚ã¦ã§ã—ãŸã€‚ãªã®ã¨ã€ç‰¹ã«Hiveã«ã¤ã„ã¦ã‚ã¾ã‚Šã¾ã˜ã‚ã«èª¿ã¹ã¦ã„ãªã„ã®ã§ã€ã“ã†ã„ã†è¨å®šæ£ã—ã„ã‹ã¯å®šã‹ã§ã¯ãªã„ã§ã™ã€‚metastoreã¯ãƒãƒ¼ã‚«ãƒ«ã®ã¾ã¾ã¨ã‹ã€‚ ã¨ã‚Šã‚ãˆãšã€exampleã§resource managerã«jobを投ã’られるã“ã¨ã‚’確èªã—ãŸçŠ¶æ…‹ã§ã€hiveã«ã‚¯ã‚¨ãƒªã‚’投ã’ã‚‹ã¨æ€’られã¾ã—ãŸã€‚ WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files. Logging initialized using config
CDH3u2ã§åˆã‚ã¦Hiveを使ã£ã¦ã¿ã‚‹ï¼ˆåˆå¿ƒè€…å‘ã‘？） - æ°—ã¾ãã‚Œãªå‚™å¿˜éŒ²(ä»®)
ã¨ã„ã†ã‚ã‘ã§ã€ CDH3u2ã§åˆã‚ã¦Hiveを使ãŠã†ã¨ã—ãŸã‚‰ã€ 環境ã¾ã‚り（ã¨ã„ã†ã‹ã»ã¨ã‚“ã©permission denied）㧠何度ã‹ã¯ã¾ã£ãŸã®ã§ã€æ•´ç†ã—ã¦ãŠã。 ãªãŠã€CDH3u2自体ã®ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã¯ã€ã€å¿˜ã‚ŒãŸã€‚ ã“ã®ãらã„ãªã‚‰ã°ã€ããã‚Œã°ãŸãã•ã‚“ヒットã™ã‚‹ã€‚ 以下ã€å‰ææ¡ä»¶ã¨ã—ã¦ã€hdfsユーザã§æ“作を行ã†ã‚‚ã®ã¨ã™ã‚‹ã€‚ CDH3u2ã®hiveをインストールã™ã‚Œã°ã€ hiveコマンドã®ãƒ‘スãªã©ã€è‡ªå‹•ã§é€šã—ã¦ãれるã®ã§ã€ 後ã¯ç’°å¢ƒã¾ã‚ã‚Šã‚’æ•´ç†ã™ã‚Œã°ã‚ˆã„。 ã¨ã‚Šã‚ãˆãšã€hiveコマンドを実行ã™ã‚‹ã¨ã€ä»¥ä¸‹ã®ã‚ˆã†ãªã‚¨ãƒ©ãƒ¼ãŒå‡ºåŠ›ã•ã‚Œã‚‹ã€‚ -bash-4.1$ hive Hive history file=/tmp/hdfs/hive_job_log_hdfs_201112070248_1529599074.txt Exception in thread "main" java.io.FileN
1
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
Cloudera Blog
GitHub - gree/hive-ruby-scripting
GitHub - myui/hivemall: Scalable machine learning library for Apache Hive/Spark/Pig
GitHub - recruitcojp/WebHive: ブラウザよりHiveQLを実行ã§ãã‚‹WEBアプリケーション
{{#tags}}- {{label}}
{{/tags}}