[B! hive] InoHiroã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

InoHiro id:InoHiro

hiveã«é–¢ã™ã‚‹InoHiroã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (31)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

Hiveã‚’å°Žå…¥ãƒ»é«˜é€ŸåŒ–ã—ãŸã„æ™‚ã«æœ€ä½Žé™å¿…è¦ãªçŸ¥è˜ - Qiita
ãŠç–²ã‚Œæ§˜ã§ã™ã€‚ãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿ã¨ã„ã†è¨€è‘‰ãŒæµè¡Œã‚Šã ã—ã¦ã‹ã‚‰å¹¾æ˜Ÿéœœã€çš†ã•ã‚“ã®ä¸ã§ã‚‚ãã‚ãã‚ç¤¾å†…ã«ãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿å‡¦ç†åŸºç›¤ã‚’ä½œã‚ŠãŸã„ã¨ã„ã†æ–¹ãŒã„ã‚‹ã®ã§ã¯ãªã„ã§ã—ã‚‡ã†ã‹ï¼Ÿ ã¨ã„ã†ã‚ã‘ã§SQL on Hadoopã§ã‚ˆãçŸ¥ã‚‰ã‚Œã‚‹Hiveã‚’ä½¿ã†ä¸Šã§æœ€ä½Žé™å¿…è¦ã ãªã¨æ€ã†çŸ¥è˜ã‚’ã¾ã¨ã‚ã¦ã¿ã¾ã—ãŸã€‚ Prestoã‚„AWS Athenaã€Amazon Redshift Spectrumã‚’ä½¿ã†å ´åˆã§ã‚‚è£å´ã«ã¯Hiveã®ãƒ†ãƒ¼ãƒ–ãƒ«ãŒå¿…è¦ãªå ´åˆãŒå¤šã„ã¨æ€ã„ã¾ã™ã®ã§ã€çŸ¥ã£ã¦ãŠã„ã¦æã¯ç„¡ã„ã‹ã¨æ€ã„ã¾ã™ã€‚ ##ãƒ†ãƒ¼ãƒ–ãƒ«ãƒ»ãƒ‡ãƒ¼ã‚¿è¨è¨ˆ ###ãƒ‘ãƒ¼ãƒ†ã‚£ã‚·ãƒ§ãƒ³ã‚’è¨å®šã™ã‚‹ Hiveã§ã¯ãƒ†ãƒ¼ãƒ–ãƒ«ã«ãƒ‘ãƒ¼ãƒ†ã‚£ã‚·ãƒ§ãƒ³ã‚’è¨å®šã™ã‚‹äº‹ã§ãƒ†ãƒ¼ãƒ–ãƒ«ã®æ¤œç´¢ãƒ»æ›´æ–°ç¯„å›²ã‚’é™å®šã™ã‚‹äº‹ãŒã§ãã¾ã™ã€‚é€†ã«è¨€ãˆã°ãƒ‘ãƒ¼ãƒ†ã‚£ã‚·ãƒ§ãƒ³ã‚’è¨å®šã—ãªã„å ´åˆã€å¸¸ã«ãƒ†ãƒ¼ãƒ–ãƒ«å…¨ä½“ã‚’æ¤œç´¢ãƒ»æ›´æ–°ã—ã¦ã—ã¾ã†ã¨ã„ã†äº‹ã§ã™ã€‚ãªã®ã§ãƒ‘ãƒ¼ãƒ†ã‚£ã‚·ãƒ§ãƒ³è¨å®šã¯åŸºæœ¬çš„ã«å¿…é ˆã§ã™ã€‚ ãƒ‘ãƒ¼ãƒ†ã‚£ã‚·ãƒ§ãƒ³ã¯ãƒ‡ãƒ¼ã‚¿ã‚’ã‚¤ãƒ³ãƒãƒ¼ãƒˆã™
InoHiro 2017/11/14
Hive

AWS-EMR

Hadoop
ãƒªãƒ³ã‚¯
hive-performance.md Â· GitHub
InoHiro 2017/11/14
Hive

Tez
ãƒªãƒ³ã‚¯
S3 ä¸Šã®å¤§é‡ãƒ‡ãƒ¼ã‚¿ã‚’ EMR ã™ã‚‹ã¨ãã¯ S3DistCp ã‚’ä½¿ã†ã¨æ—ã‚‹ â€“ I'm Sei.
CloudFront ã®ã‚¢ã‚¯ã‚»ã‚¹ãƒã‚°ã‚’ S3 ã«å‡ºåŠ›ã™ã‚‹ã‚ˆã†ã«è¨å®šã—ã¦ã„ã‚‹ã¨ã€æ—¥ã«æ—¥ã«å¤§é‡ã®ãƒã‚°ãƒ•ã‚¡ã‚¤ãƒ«ãŒæºœã¾ã£ã¦ã„ãã¾ã™ã€‚ ä¸ã«ã¯ 1 MB ç¨‹åº¦ã®ãƒ•ã‚¡ã‚¤ãƒ«ã‚‚ã‚ã‚Œã°ã€æ•°ç™¾ bytes ç¨‹åº¦ã®å°ã•ãªã‚‚ã®ã‚‚ã‚ã‚Šã¾ã™ã€‚ ä½•ã«ã›ã‚ˆã€ã¨ã«ã‹ãå¤§é‡ã«ã‚ã‚Šã¾ã™ã€‚ å°ã•ã„ãƒ•ã‚¡ã‚¤ãƒ«ã¯ã¾ã¨ã‚ã‚‹CloudFront ã®ãƒã‚°ã®ã‚ˆã†ã«å°ã•ã„ãƒ•ã‚¡ã‚¤ãƒ«ãŒå¤§é‡ã«ã‚ã‚‹ã‚ˆã†ãªã‚±ãƒ¼ã‚¹ã¯ Hadoop ãŒè‹¦æ‰‹ã¨ã™ã‚‹ãƒ‘ã‚¿ãƒ¼ãƒ³ã§ã€ãã®ã¾ã¾æ‰±ã†ã¨å¤§æŠµã‚ã¡ã‚ƒã‚“ã“æ™‚é–“ãŒã‹ã‹ã‚Šã¾ã™ã€‚ ãã†ã„ã†å ´åˆã¯ã€S3DistCp ã¨ã„ã†ä¾¿åˆ©ãƒ„ãƒ¼ãƒ«ã‚’ä½¿ã£ã¦ S3 ä¸Šã®ãƒã‚°ãƒ•ã‚¡ã‚¤ãƒ«ã‚’ã‚‚ã†ã™ã“ã—å¤§ãã„ã‚µã‚¤ã‚ºã«é€£çµã—ã¦ã€HDFS ã«ã‚³ãƒ”ãƒ¼ã™ã‚‹ã¨ãƒ‘ãƒ•ã‚©ãƒ¼ãƒžãƒ³ã‚¹ãŒã‚ãŒã‚‹ã“ã¨ãŒã‚ã‚Šã¾ã™ã€‚ ã¡ãªã¿ã«ã€EMR ã®ãƒ™ã‚¹ãƒˆãƒ—ãƒ©ã‚¯ãƒ†ã‚£ã‚¹ã«ã‚ˆã‚‹ã¨ã€Gzip ã§åœ§ç¸®ã•ã‚Œã¦ã„ã‚‹å ´åˆã¯ 1 ~ 2GBã€LZO ã§åœ§ç¸®ã•ã‚Œã¦ã„ã‚‹å ´åˆã¯ 2 ~ 4GB ãŒæœ€é©ãªãƒ•ã‚¡ã‚¤ãƒ«ã‚µã‚¤ã‚ºã‚‰ã—ã„ã§ã™
InoHiro 2017/11/14
Hadoop

Hive

AWS-EMR
ãƒªãƒ³ã‚¯
Hiveã§ãƒ†ã‚ã‚¹ãƒˆã®Fuzzy Search
å…ˆé€±ã¯PostgreSQLä¸Šã§ãƒ†ã‚ã‚¹ãƒˆã®Fuzzy Searchã‚’è©¦ã—ãŸã€‚ãã®ã¨ãã¯ fuzzystrmatch ã‚„ pg_trgm ã¨ã„ã£ãŸãƒ¢ã‚¸ãƒ¥ãƒ¼ãƒ«ãŒæ´»èºã—ã¦ã„ãŸã€‚ ã§ã¯ã€åŒã˜ã“ã¨ã‚’Hiveã§å®Ÿç¾ã™ã‚‹ã¨ã©ã†ãªã‚‹ã ã‚ã†ã€‚ ãƒ‡ãƒ¼ã‚¿ é©å½“ã«ãƒ†ãƒ¼ãƒ–ãƒ« sample ã‚’ã¤ãã£ã¦ãŠãï¼š hive> CREATE TABLE sample AS > SELECT 1 AS id, 'I live in Tokyo.' AS document > UNION ALL > SELECT 2 AS id, 'Are you happy?' AS document > ; hive> SELECT * FROM sample; OK sample.id sample.document 1 I live in Tokyo. 2 Are you happy? Time taken: 0.066 seconds,
InoHiro 2017/08/24
hive
ãƒªãƒ³ã‚¯
ãƒ‡ãƒ¼ã‚¿ã‚µã‚¤ã‚¨ãƒ³ãƒ†ã‚£ã‚¹ãƒˆã®ãŸã‚ã®HiveQLåˆ†æžã‚¯ã‚¨ãƒªãƒ†ãƒ³ãƒ—ãƒ¬ãƒ¼ãƒˆ ãã®ï¼– - ãƒˆãƒ¬ã‚¸ãƒ£ãƒ¼ãƒ‡ãƒ¼ã‚¿ï¼ˆTreasure Dataï¼‰ãƒ–ãƒã‚°
*ãƒˆãƒ¬ã‚¸ãƒ£ãƒ¼ãƒ‡ãƒ¼ã‚¿ã¯ãƒ‡ãƒ¼ã‚¿åŽé›†ã€ä¿å˜ã€åˆ†æžã®ãŸã‚ã®ã‚¨ãƒ³ãƒ‰ãƒ„ãƒ¼ã‚¨ãƒ³ãƒ‰ã§ã‚µãƒãƒ¼ãƒˆã•ã‚ŒãŸã‚¯ãƒ©ã‚¦ãƒ‰ã‚µãƒ¼ãƒ“ã‚¹ã§ã™ã€‚ ãƒ‡ãƒ¼ã‚¿ã‚µã‚¤ã‚¨ãƒ³ãƒ†ã‚£ã‚¹ãƒˆã®ãŸã‚ã®HiveQLåˆ†æžã‚¯ã‚¨ãƒªãƒ†ãƒ³ãƒ—ãƒ¬ãƒ¼ãƒˆã‚·ãƒªãƒ¼ã‚ºï¼š ãã®ï¼‘,Â ãã®ï¼’,Â ãã®ï¼“,Â ãã®ï¼”,Â ãã®ï¼•, ãã®ï¼– E. ãƒªãƒ†ãƒ³ã‚·ãƒ§ãƒ³åˆ†æž æœ€å¾Œã¯ãƒªãƒ†ãƒ³ã‚·ãƒ§ãƒ³ã«é–¢ã™ã‚‹ã‚¯ã‚¨ãƒªãƒ†ãƒ³ãƒ—ãƒ¬ãƒ¼ãƒˆã§ã™ã€‚ ãƒ‘ã‚¿ãƒ¼ãƒ³ E-1ï¼šç›´å¸°çŽ‡ â€» ã“ã“ã§ã¯åŒæ—¥å†…ã«1å›žã—ã‹ãƒã‚°ã‚¤ãƒ³ã—ãªã‹ã£ãŸãƒ¦ãƒ¼ã‚¶ãƒ¼ã‚’ã€Œç›´å¸°ã€ã¨è¦‹ãªã—ã¾ã™ã€‚ SELECT t2.d AS d, COUNT(1)/SUM(t2.cnt)*100 AS bounce_rate FROM ( SELECT TD_TIME_FORMAT(time, 'yyyy-MM-dd', 'JST') AS d, uid, COUNT(1) AS cnt FROM login GROUP BY TD_TIME_FORMAT(time, 'yyyy-M
InoHiro 2016/05/23
TD

hive
ãƒªãƒ³ã‚¯
TreasureDataã§æ©Ÿæ¢°å¦ç¿’ã—ã¦ã¿ã‚‹ | ADN LAB's Blog
ã“ã®è¨˜äº‹ã¯Treasure Data Advent CalendarÂ 11æ—¥ç›®ã®è¨˜äº‹ã§ã™ã€‚ ã“ã‚“ã«ã¡ã¯ã€ãƒ‡ãƒ¼ã‚¿ãƒžã‚¤ãƒ‹ãƒ³ã‚°ãƒãƒ¼ãƒ ã®y_kawasakiã§ã™ã€‚ï¼ˆæ¯Žå›žæ‰€å±žãŒå¤‰ã‚ã£ã¦ã¾ã™ï¼ï¼‰ æœ€è¿‘ã€å¤§é‡ã®ãƒ‡ãƒ¼ã‚¿ã‚’å‰ã«é€”æ–¹ã«ãã‚Œã¦ã„ã¾ã—ãŸã€‚TreasureDataï¼ˆä»¥ä¸‹TDï¼‰ã¨ã„ã†ã€æ¦å™¨ã‚’æ‰‹ã«å…¥ã‚Œã¦ã€ãŒã£ã¤ã‚Šã¨æˆ¦ã£ã¦ã„ã‚‹ã¨ã“ã‚ã§ã™ã€‚ TDã«ã¯Hivemallã¨ã„ã†ã€Hiveä¸Šã§å‹•ãscala ble machine learning libraryãŒå®Ÿè£…ã•ã‚Œã¦ã„ã¾ã™ã€‚Hivemallã®ä½¿ã„æ–¹ã¯githubã®wikiã«ã‚ˆãã¾ã¨ã‚ã‚‰ã‚Œã¦ã„ã‚‹ã®ã§ã™ãŒã€ãã‚Œã¯ã€ã‚ãã¾ã§ã€Hiveä¸Šã§å‹•ã‹ã™ã“ã¨ã‚’å‰æã«æ›¸ã‹ã‚Œã¦ã„ã¾ã™ã€‚ä»Šå›žã¯ã€TDä¸Šã§å‹•ã„ã¦ã„ã‚‹Hivemallã‚’ä½¿ã„ãŸã„ãŸã‚ã€ä¸€éƒ¨ã€å¤‰æ›´ã—ãŸã‚Šã™ã‚‹å¿…è¦ãŒã‚ã‚Šã¾ã™ã€‚ ã¨ã‚Šã‚ãˆãšã€æ‰‹ã‚’å‹•ã‹ã—ãŸã„ã¨ã„ã†ã“ã¨ã§ã€ã©ã“ã®ã”å®¶åºã«ã‚‚å¸¸å‚™ã—ã¦ã‚ã‚‹ã€ã‚ã‚„ã‚ã®ãƒ‡ãƒ¼ã‚¿ã‚’ä½¿ã„ãŸã„ã¨æ€ã„
InoHiro 2016/01/14
TD

Hive

Hivemall

machine_learning
ãƒªãƒ³ã‚¯
Cloudera Blog
We are thrilled to announce the general availability of the Cloudera AI Inference service, powered by NVIDIA NIM microservices, part of the NVIDIA AI Enterprise platform, to accelerate generative AI deployments for enterprises. This service supports a range of optimized AI models, enabling seamless and scala ble AI inference. Background The generative AI landscape is evolving [â€¦] Read blog post
InoHiro 2015/05/07
hive
ãƒªãƒ³ã‚¯
ãƒˆãƒ¬ã‚¸ãƒ£ãƒ¼ãƒ‡ãƒ¼ã‚¿UDFã§åŠ¹çŽ‡ã®è‰¯ã„ã‚¯ã‚¨ãƒªã‚’å®Ÿç¾ã™ã‚‹ - Date Function ç·¨ - - ãƒˆãƒ¬ã‚¸ãƒ£ãƒ¼ãƒ‡ãƒ¼ã‚¿ï¼ˆTreasure Dataï¼‰ãƒ–ãƒã‚°
ãƒˆãƒ¬ã‚¸ãƒ£ãƒ¼ãƒ‡ãƒ¼ã‚¿ã¯ã‚¯ãƒ©ã‚¦ãƒ‰ã§ãƒ‡ãƒ¼ã‚¿ãƒžãƒãƒ¼ã‚¸ãƒ¡ãƒ³ãƒˆã‚µãƒ¼ãƒ“ã‚¹ã‚’æä¾›ã—ã¦ã„ã¾ã™ã€‚ ãƒˆãƒ¬ã‚¸ãƒ£ãƒ¼ãƒ‡ãƒ¼ã‚¿ã‚µãƒ¼ãƒ“ã‚¹ã¯ï¼Œã‚¹ã‚¿ãƒ³ãƒ€ãƒ¼ãƒ‰ãƒ—ãƒ©ãƒ³ã¨ã—ã¦ãƒãƒƒãƒã‚¯ã‚¨ãƒªã§ã‚ã‚‹Hive, Pigã‚’ã‚µãƒãƒ¼ãƒˆã—ã¦ã„ã¾ã™ã€‚ã¾ãŸï¼Œãƒ—ãƒ¬ãƒŸã‚¢ãƒ ãƒ—ãƒ©ãƒ³ã¾ãŸã¯ã‚ªãƒ—ã‚·ãƒ§ãƒ³è¿½åŠ ã«ã‚ˆã£ã¦ã‚¢ãƒ‰ãƒ›ãƒƒã‚¯ã‚¯ã‚¨ãƒªã§ã‚ã‚‹TQAï¼ˆBackend: Prestoï¼‰ã‚’ã‚µãƒãƒ¼ãƒˆã—ã¦ã„ã¾ã™ã€‚ Treasure Data Hive UDF ç„¡å°½è”µã®åˆ—æŒ‡å‘ãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹ã§ã‚ã‚‹ãƒˆãƒ¬ã‚¸ãƒ£ãƒ¼ãƒ‡ãƒ¼ã‚¿ã®ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ã§ã™ãŒï¼Œç‰¹ã«æ™‚ç³»åˆ—ãƒ‡ãƒ¼ã‚¿ã«å¯¾ã—ã¦ã¯å¼·åŠ›ãªãƒ‘ãƒ•ã‚©ãƒ¼ãƒžãƒ³ã‚¹ã‚’ç™ºæ®ã—ã¾ã™ã€‚ä»Šå›žã¯ãã®ç†ç”±ã¨ï¼Œãƒ‘ãƒ•ã‚©ãƒ¼ãƒžãƒ³ã‚¹ã‚’å‡ºã™ãŸã‚ã®ã‚¯ã‚¨ãƒªã®è¨˜è¿°æ–¹æ³•ã‚’ã”ç´¹ä»‹ã—ã¾ã™ã€‚ TDS ãƒãƒƒã‚¯ã‚¨ãƒ³ãƒ‰ã‚¢ãƒ¼ã‚ãƒ†ã‚¯ãƒãƒ£æ¦‚ç•¥ ãƒˆãƒ¬ã‚¸ãƒ£ãƒ¼ãƒ‡ãƒ¼ã‚¿ã‚µãƒ¼ãƒ“ã‚¹ã§ã¯ï¼ŒRealtime Storage ã¨ Archive Storage ã®äºŒç¨®é¡žã®ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ã‚’åˆ©ç”¨ã—ã¦ã„ã¾ã™ã€‚Treasure Agent (Fluentd)
InoHiro 2014/12/08
TD

hive
ãƒªãƒ³ã‚¯
Hive QL Except clause
InoHiro 2014/10/24
hive
ãƒªãƒ³ã‚¯
1 ã¤ã® HiveQL ã§ã‚°ãƒ«ãƒ¼ãƒ—ã”ã¨ã®é›†ç´„çµæžœä¸Šä½ã®ã‚‚ã®ã‚’æŠ½å‡ºã™ã‚‹ - ã‚ã‚‰ã³ãæ—¥è¨˜
ã“ã®è¨˜äº‹ã¯ abicky.net ã® 1 ã¤ã® HiveQL ã§ã‚°ãƒ«ãƒ¼ãƒ—ã”ã¨ã®é›†ç´„çµæžœä¸Šä½ã®ã‚‚ã®ã‚’æŠ½å‡ºã™ã‚‹ ã«ç§»è¡Œã—ã¾ã—ãŸ
InoHiro 2014/10/01
Hive

TD
ãƒªãƒ³ã‚¯
Hiveã‚¯ã‚¨ãƒªãƒãƒ¥ãƒ¼ãƒ‹ãƒ³ã‚°ãƒ¡ãƒ¢ - ç§‘å¦ã¨éžç§‘å¦ã®è¿·å®®
Hadoopæœ¬2ç‰ˆã‚’è²·ã£ãŸãã®ä¸€é€±é–“å¾Œã«Deals of the day ã§åŠé¡ã‚»ãƒ¼ãƒ«ã‚„ã‚‰ã‚Œã¦æ»ã«ãŸããªã£ãŸã®ã§è…¹ã„ã›ã«æ›¸ã ã¯ã˜ã‚ã« ã“ã“ã«æ›¸ã„ã¦ã‚ã‚‹ã®ã¯å…¨éƒ¨å‚è€ƒãƒªãƒ³ã‚¯ãƒ»æ–‡çŒ®ã‹ã‚‰ã²ã£ã±ã£ã¦ããŸã ã‘ã§ã€ã»ã¨ã‚“ã©å…¨éƒ¨æ¤œè¨¼ã—ã¦ãªã„ã€‚é–“é•ã„ãŒã‚ã‚Œã°ãªã‚‹ã¹ãæ—©ã‚ã«æ›´æ–°ã™ã‚‹ã‚ˆã†åŠªåŠ›ã¯ã™ã‚‹ãŒã€éµœå‘‘ã¿ã«ã—ã¦ä½•ãŒèµ·ãã¦ã‚‚è‡ªå·±è²¬ä»»ã§ã€‚ Hive ã®ã‚¯ã‚¨ãƒªãƒãƒ¥ãƒ¼ãƒ‹ãƒ³ã‚°ã«é–¢ã™ã‚‹ãƒ¡ãƒ¢æ›¸ãã§ã‚ã‚‹ã€‚ä»¥ä¸‹ã®ã“ã¨ã¯ã€Œæ›¸ã„ã¦ã„ãªã„ã€ Hadoopè‡ªä½“ã®ãƒãƒ¥ãƒ¼ãƒ‹ãƒ³ã‚° Hive ã®ã‚¯ã‚¨ãƒªãƒãƒ¥ãƒ¼ãƒ‹ãƒ³ã‚°ä»¥å¤–ã®è©± ä¾‹ãˆã°ã€åœ§ç¸®ãƒ•ã‚¡ã‚¤ãƒ«ã‚’ Hive ä¸Šã§æ‰±ã†ã«ã¯ã©ã†ã™ã‚‹ã‹ã¨ã‹ JOIN ä¸€ç•ªå·¦ã®ãƒ†ãƒ¼ãƒ–ãƒ«ã«æœ€ã‚‚å¤§ããªãƒ†ãƒ¼ãƒ–ãƒ«ã‚’æŒã£ã¦ãã‚‹ ä¸€ç•ªå·¦ã®ãƒ†ãƒ¼ãƒ–ãƒ«ãŒMRã§ã„ã†å…¥åŠ›ãƒ‡ãƒ¼ã‚¿ã¨ã—ã¦æµã‚Œã‚‹ã€‚ã‚¤ãƒ³ãƒŠãƒ¼ãƒ†ãƒ¼ãƒ–ãƒ«ã®ãƒ‡ãƒ¼ã‚¿ã¯ãƒ¡ãƒ¢ãƒªã«ä¿æŒã•ã‚Œã‚‹ã€‚ åŒä¸€ JOIN ã‚ãƒ¼ é€šå¸¸ã¯ 1 JOIN = 1 MR ã‚¸ãƒ§ãƒ–ã ãŒã€åŒä¸€ã® JOIN ã‚ãƒ¼ã‚’ä½¿ã£ã¦ã„
InoHiro 2014/09/11
hive
ãƒªãƒ³ã‚¯
SIOSã€€ãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿æŠ€è¡“ãƒ–ãƒã‚°: Treasure Dataã«ãŠã‘ã‚‹HiveQLã®Tips
ã“ã‚“ã«ã¡ã¯ã€ä»Šå›žã®ãƒ–ãƒã‚°æ‹…å½“ é«˜æ©‹ã§ã™ã€‚ æœ¬é¡Œã¨ã¯é€¸ã‚Œã¾ã™ãŒã€ãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿ã«é–¢é€£ã™ã‚‹ãƒˆãƒ¬ãƒ³ãƒ‰ã¨ã—ã¦ã€M2M(Machine to Machine)ã‚„IoT(Internet of Things)ã¨å‘¼ã°ã‚Œã‚‹æŠ€è¡“ãŒã‚ã‚Šã¾ã™ã€‚ SIOSãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿ãƒãƒ¼ãƒ ã¨ã—ã¦ã‚‚ã€ã“ã‚Œã‚‰ã®æŠ€è¡“ã«ã‚ˆã£ã¦å¤§é‡ã«åŽé›†ã•ã‚Œã‚‹ãƒ‡ãƒ¼ã‚¿ã«ã¯æ³¨ç›®ã—ã¦ã„ã¾ã™ã€‚ ã“ã‚Œã‚‰ã®æŠ€è¡“ã‚’å€‹äººã§å®Ÿç¾å¯èƒ½ãªãƒ—ãƒã‚°ãƒ©ãƒžãƒ–ãƒ«ãƒ‡ãƒã‚¤ã‚¹ã¨ã—ã¦ã€Arduinoã‚„Raspberry PiãŒæ™®åŠã—ã¦ãã¦ã„ã¾ã™ã€‚ ç‰¹ã«ã€Arduinoã¯ã€æŽ¥è§¦ã‚»ãƒ³ã‚µã‚„èµ¤å¤–ã‚»ãƒ³ã‚µãªã©å„ç¨®ã‚»ãƒ³ã‚µã‚’å®Ÿè£…ã§ãã€ãªãŠã‹ã¤Bluetoothã‚„ZigBeeãªã©ã®é€šä¿¡ãƒ¢ã‚¸ãƒ¥ãƒ¼ãƒ«ã®å®Ÿè£…ã‚‚å¯èƒ½ã§ã™ã€‚ ä¾‹ãˆã°ã€è¤‡æ•°å°ã®Arduinoã‚’çµ„ã¿åˆã‚ã›ã¦è‡ªå®…å†…ã‚»ãƒ³ã‚µãƒãƒƒãƒˆãƒ¯ãƒ¼ã‚¯ã‚’æ§‹ç¯‰ã—ã€æ—¥å¸¸ç”Ÿæ´»ã®è¦‹ãˆã‚‹åŒ–ãŒã§ããŸã‚‰æ¥½ã—ãã†ã§ã™ãã€‚ ã“ã†ã—ãŸãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿ã‚’ç”Ÿã¿å‡ºã™æ§˜ã€…ãªã‚¢ã‚¤ãƒ‡ã‚¢ã‚’å®Ÿç¾ã™ã‚‹ãŸã‚ã«ã€ç§ãŸã¡ã‚‚æ—¥ã€…ã€ãƒ“
InoHiro 2014/09/11
hive
ãƒªãƒ³ã‚¯
LanguageManual UDF - Apache Hive - Apache Software Foundation
When hive.cache.expr.evaluation is set to true (which is the default) a UDF can give incorrect results if it is nested in another UDF or a Hive function. This bug affects releases 0.12.0, 0.13.0, and 0.13.1. Release 0.14.0 fixed the bug (HIVE-7314). The probl em relates to the UDF's implementation of the getDisplayString method, as discussed in the Hive user mailing list.
InoHiro 2014/07/18
hive
ãƒªãƒ³ã‚¯
HiveQL SELECTãƒ¡ãƒ¢(Hishidama's Apache Hive SELECT Memo)
ã€ŒSELECT *ã€ã¨ã€ŒSELECT é …ç›®ã€ã®é•ã„ Hiveã§ã¯ã€ŒSELECT *ã€ã¨ã€ŒSELECT é …ç›®æŒ‡å®šã€ã§ã¯æŒ™å‹•ãŒç•°ãªã‚‹ã€‚ createæ–‡ã§ãƒ†ãƒ¼ãƒ–ãƒ«ã‚’ä½œã£ãŸæ®µéšŽã§ã¯ã€ãƒ‡ãƒ¼ã‚¿ï¼ˆãƒ•ã‚¡ã‚¤ãƒ«ï¼‰ãŒå˜åœ¨ã—ã¦ã„ã¦ã‚‚ã€ãã®ä¸ã®æ•´åˆæ€§ã¯å•ã‚ã‚Œãªã„ã€‚ ã€Œselect * from ãƒ†ãƒ¼ãƒ–ãƒ«ã€ã§ã¯ã€Hiveè‡ªèº«ãŒãƒ•ã‚¡ã‚¤ãƒ«ã®ä¸èº«ã‚’é …ç›®å®šç¾©ã«åˆã‚ã›ã¦è§£é‡ˆã—ã€ãã®å€¤ã‚’è¡¨ç¤ºã™ã‚‹ã€‚[/2011-08-11] ä¸€æ–¹ã€ã€Œselect é …ç›® from ãƒ†ãƒ¼ãƒ–ãƒ«ã€ã§ã¯ã€MapReduceï¼ˆMapå‡¦ç†ï¼‰ã‚’å®Ÿè¡Œã—ã¦è§£é‡ˆã‚’è¡Œã„ã€çµæžœã‚’è¡¨ç¤ºã™ã‚‹ã€‚ ï¼ˆINSERTï½žSELECTã§ã¯ã€ŒSELECT *ã€ã§ã‚‚MapReduceãŒå®Ÿè¡Œã•ã‚Œã‚‹ã€‚[2011-08-11]ï¼‰ FROMã®ä½ç½® Hiveã§ã¯ã€FROMå¥ã‚’å…ˆé ã«æŒã£ã¦ãã‚‹ã“ã¨ãŒå‡ºæ¥ã‚‹ã€‚ select ï½ž from ãƒ†ãƒ¼ãƒ–ãƒ« where æ¡ä»¶; â†“â†‘ç‰ä¾¡ from ãƒ†ãƒ¼ãƒ–ãƒ« sele
InoHiro 2014/07/11
hive
ãƒªãƒ³ã‚¯
Using group by on multiple columns
Ask questions, find answers and collaborate at work with Stack Overflow for Teams. Explore Teams Collectivesâ„¢ on Stack Overflow Find centralized, trusted content and collaborate around the techno logies you use most. Learn more about Collectives
InoHiro 2014/07/11
sql

hive
ãƒªãƒ³ã‚¯
Hiveã®JOINã«ã¤ã„ã¦ â€“ OpenGroove
Hiveã®JOINã«ã¤ã„ã¦ã€SQLã¨ã®é•ã„ã‚’ãƒã‚¤ãƒ³ãƒˆã«é›‘è¨˜ãƒ¡ãƒ¢ã€‚ Hiveã§ã¯ INNER JOINã¯JOINã§ã„ã„ ã©ã“ã§èªã‚“ã ã‹ä»Šã¨ãªã£ã¦ã¯å®šã‹ã§ã¯ãªã„ãŒï¼ˆæ±—ï¼‰ã€ãã†ã„ã†ã“ã¨ã‚‰ã—ã„ã€‚ Hiveã§ã‚µãƒãƒ¼ãƒˆã•ã‚Œã¦ã„ã‚‹JOIN Only equality joins, outer joins, and left semi joins are supported in Hive. Hive does not support join conditions that are not equality conditions as it is very difficult to express such conditions as a map/reduce job. Also, more than two tables can be joined in Hive. ã¨ã„ã†ã“ã¨ãªã®ã§ã€Hiveã§ã‚µãƒãƒ¼
InoHiro 2014/07/10
hive

join
ãƒªãƒ³ã‚¯
ã»ã¼ã‚„ã‘ããHive Hacks â€“ OpenGroove
Hive Hacksã‚ã‚Œã“ã‚Œã€‚å†…å®¹ã¯ã»ã¼Oâ€™REILLY Hadoop Hacksã‹ã‚‰ã®å¼•ç”¨ãã®ã¾ã‚“ã¾ã€‚ãŸã ã®å€‹äººãƒ¡ãƒ¢ãªã®ã ãŒã€ãšã†ãšã†ã—ãå…¬é–‹ã•ã›ã¦ã‚‚ã‚‰ã„ã¾ã™ã€‚ã„ã‚ã‚“ãªã¨ã“ã‚ã«è¨˜éŒ²ã—ã¦ãŠã„ã¦ã‚‚ã™ãã«ã€Œã‚ã‚Œã€ã‚ã®ãƒ¡ãƒ¢ã©ã“ã‚„ã£ãŸã£ã‘ã€ã¨ãªã‚‹ã®ã§ã“ã“ã«æ›¸ãã®ãŒä¸€ç•ªãªã‚“ã ã‚ˆãã€‚æ›¸ã„ãŸã‹ã‚‰ã£ã¦ç†è§£ã§ãã‚‹ã‚ã‘ã§ã‚‚ãªã„ã‚“ã ãŒâ€¦ ï¼ˆåˆã‚ã«æ›¸ã„ã¦ãŠãã¨ã€ã“ã®æŠ•ç¨¿é•·ã„ã§ã™ï¼‰ åŸºæœ¬åŽŸå‰‡çš„ãªã“ã¨ã€‚ â—UPDATEã¯å›žé¿ã™ã‚‹ å‡¦ç†é€Ÿåº¦ãŒé…å»¶ã™ã‚‹ãŸã‚ã€UPDATEã‚’å¤šæ•°å«ã‚€ã‚ˆã†ãªSQLã‚’HiveSQLã«å¤‰æ›ã™ã‚‹ã“ã¨ã¯é¿ã‘ã‚‹ã¹ã â—MapReduceã‚¿ã‚¹ã‚¯ã®ã‚ªãƒ¼ãƒãƒ¼ãƒ˜ãƒƒãƒ‰ Hiveã¯ã€Œé«˜ã‚¹ãƒ«ãƒ¼ãƒ—ãƒƒãƒˆã‚’ç›®æŒ‡ã™å‡¦ç†ã«ã¯å‘ã„ã¦ã„ã‚‹ãŒã€ä½Žãƒ¬ãƒ³ãƒ†ãƒ³ã‚·ã‚’ç›®æŒ‡ã™å‡¦ç†ã«ã¯å‘ã„ã¦ã„ãªã„ã€ã¨ã„ã†MapReduceå‡¦ç†ã®ç‰¹å¾´ã‚’å¼•ãç¶™ã„ã§ã„ã‚‹ã€‚MapReduceã‚¿ã‚¹ã‚¯ã®ã‚ªãƒ¼ãƒãƒ¼ãƒ˜ãƒƒãƒ‰ãŒä»˜ãã¾ã¨ã†ã“ã¨ã‚’å¿µé ã«ãŠãã€‚ â—ä¸¦åˆ—åˆ†æ•£ãŒã§ããªã„å‡¦ç†
InoHiro 2014/07/08
hive
ãƒªãƒ³ã‚¯
å®Ÿãƒ‡ãƒ¼ã‚¿ã§è¦šãˆã‚‹ Treasure Client ã‚³ãƒžãƒ³ãƒ‰ãƒ©ã‚¤ãƒ³ãƒªãƒ•ã‚¡ãƒ³ã‚¹ ã€œ1.Data Importã€œ - doryokujin's blog
æœ¬ã‚·ãƒªãƒ¼ã‚ºã§ã¯Treasure Client ãƒ„ãƒ¼ãƒ«ã®ã‚³ãƒžãƒ³ãƒ‰ãƒ©ã‚¤ãƒ³ãƒªãƒ•ã‚¡ãƒ¬ãƒ³ã‚¹ã‚’ä»¥ä¸‹ã®5ã¤ã®ãƒ¬ã‚¤ãƒ¤ãƒ¼ã«åˆ†è§£ã—ï¼Œå„ã€…ã«ã¤ã„ã¦è§£èª¬ã—ã¦ã„ãã‚‚ã®ã¨ã—ã¾ã™ã€‚ No.ãƒ¬ã‚¤ãƒ¤ãƒ¼ã‚³ãƒžãƒ³ãƒ‰ 1. Data Import one-time import bulk import 2. Data Management db table 3. DataÂ Processing schema query job 4. DataÂ Scheduling sched result 5. Other help status server sample æœ¬æ—¥ã¯ 1. Data Import ã‚’ç´¹ä»‹ã—ã¾ã™ã€‚æœ¬ã‚·ãƒªãƒ¼ã‚ºã§ã¯å®Ÿéš›ã«ãƒ‡ãƒ¼ã‚¿ã‚’ä½¿ç”¨ã—ã¦ï¼Œãã®åˆ†æžã®æµã‚Œã«æ²¿ã„ãªãŒã‚‰ã‚³ãƒžãƒ³ãƒ‰ãƒ©ã‚¤ãƒ³ã‚’ç´¹ä»‹ã—ã¦ã„ãã¾ã™ã€‚ Treasure Platform ã®ç™»éŒ²ãŠã‚ˆã³ Client Tool ã®ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ« ãã‚Œã§ã¯æ—©é€Ÿã‚µã‚¤ãƒ³ã‚¢ãƒƒãƒ—ã‚’å§‹ã‚ã¾ã—
InoHiro 2014/07/08
ã‚„ã‚ã†

hive

TD
ãƒªãƒ³ã‚¯
SQLæ„Ÿè¦šã§HiveQLã‚’æ›¸ãã¨ç—›ã„ç›®ã«ã‚ã†ä¾‹ - still deeper
Hadoop Advent Calendar 2013 4æ—¥ç›®ã®è¨˜äº‹ã§ã™ tl;dr explainã¨job historyã‚’èªã‚ 1 reducerã¯æ‚ª data skewã¯æ‚ª å‰æ›¸ã ã¿ã‚“ãªå¤§å¥½ãSQLã§Hadoopä¸Šã§ã®å‡¦ç†ã‚’å®Ÿè¡Œã§ãã‚‹Hiveã«ã¯ã¿ãªã•ã‚“æ™®æ®µã‹ã‚‰ãŠä¸–è©±ã«ãªã£ã¦ã„ã‚‹ã“ã¨ã§ã—ã‚‡ã†ã€‚ã¡ã‚‡ã£ã¨èª¿ã¹ç‰©ã§ã‚°ã‚°ã‚‹åº¦ã«ç›®ã«å…¥ã‚‹æ„›ã‚‰ã„ã—ã„ãƒžã‚¹ã‚³ãƒƒãƒˆãŒã€è’ã‚“ã å¿ƒã«æ¸…æ¶¼ãªé¢¨ã‚’ã¯ã“ã‚“ã§ãã‚Œã¾ã™ã€‚ ã§ã™ãŒHiveã®ã‚¯ã‚¨ãƒªè¨€èªžã¯SQLã§ã¯ãªãHiveQLã§ã™ã—ã€å®Ÿè¡Œã‚¨ãƒ³ã‚¸ãƒ³ã‚‚RDBã®ãã‚Œã¨ã¯å…¨ãç•°ãªã‚‹MapReduceã§ã™ã€‚SQLã®ã¤ã‚‚ã‚Šã§HiveQLã‚’æ›¸ã„ã¦ã„ã‚‹ã¨åœ°é›·ã‚’è¸ã‚“ã§ã—ã¾ã†ã“ã¨ãŒã¾ã‚Œã«ã‚ˆãã‚ã‚Šã¾ã™ã€‚æœ¬ã‚¨ãƒ³ãƒˆãƒªã§ã¯é™¥ã‚ŠãŒã¡ãªHiveQLã®è½ã¨ã—ç©´ã‚’2ã¤ç´¹ä»‹ã—ã¾ã™ã€‚ ä¾‹1 SELECT count(DISTINCT user_id) FROM access_log SQLã«æ…£ã‚ŒãŸæ–¹ã§ã‚ã‚Œ
InoHiro 2013/12/05
sql

hive
ãƒªãƒ³ã‚¯
Presto: Free, Open-Source SQL Query Engine for any Data
Calling our Presto community speakers â€“ we want to hear from you! Fill out out community call for papers to speak at upcoming meetups andÂ conferences. What is Presto?Presto is an open source SQL query engine thatâ€™s fast, reliable, and efficient at scale. Use Presto to run interactive/ad hoc queries at sub-second performance for your high volume apps.
InoHiro 2013/11/08
å‡¦ç†ç”»é¢

sql

facebook

hive
ãƒªãƒ³ã‚¯
1 2 æ¬¡ã®ãƒšãƒ¼ã‚¸