[B! hive] nantanã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

nantan id:nantan

hiveã«é–¢ã™ã‚‹nantanã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (8)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

blog.katsuma.tv
å‰å›žã€JavaScriptã§Map Reduceã®ã‚³ãƒ¼ãƒ‰ãŒæ›¸ã‘ã‚‹Hadoop Streamingã«ã¤ã„ã¦ç´¹ä»‹ã—ã¾ã—ãŸã€‚ æ¨™æº–å…¥å‡ºåŠ›ã•ãˆã‚µãƒãƒ¼ãƒˆã•ã‚Œã¦ã‚ã‚Œã°ã€ä»»æ„ã®ã‚³ãƒ¼ãƒ‰ã§Map Reduuceã®å‡¦ç†ãŒæ›¸ã‘ã‚‹ã€ã¨ã„ã†ã‚‚ã®ã§ã—ãŸãŒã€ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ã¯ãã‚‚ãã‚‚é¢å€’ãã•ãŒã‚Šã€‚ã‚³ãƒ¼ãƒ‰ã‚‚æ›¸ãã®ã‚‚é¢å€’ã§ã™ã€‚ ã¨ã€ã„ã†ã‚ã‘ã§ã€ä»Šå›žã¯ã‚‚ã†ã‚³ãƒ¼ãƒ‰ã™ã‚‰æ›¸ã‹ãšã«SQLãƒ©ã‚¤ã‚¯ã§Map Reduceã§ãã‚‹Hiveã¨ã„ã†ãƒ—ãƒãƒ€ã‚¯ãƒˆã«ã¤ã„ã¦ã€ã¾ã¨ã‚ãŸã„ã¨æ€ã„ã¾ã™ã€‚ Hive Hiveã¨ã¯ã€ç°¡å˜ã«è¨€ã†ã¨Hadoopä¸Šã§å‹•ä½œã™ã‚‹RDBã®ã‚ˆã†ãªã‚‚ã®ã§ã™ã€‚ HDFSãªã©ã®åˆ†æ•£ãƒ•ã‚¡ã‚¤ãƒ«ã‚·ã‚¹ãƒ†ãƒ ä¸Šã«å˜åœ¨ã™ã‚‹ãƒ‡ãƒ¼ã‚¿ã«å¯¾ã—ã¦ã€HiveQLã¨ã„ã†SQLãƒ©ã‚¤ã‚¯ãªè¨€èªžã§æ“ä½œã§ãã¾ã™ã€‚ ã§ã€é¢ç™½ã„ã®ãŒHiveQLã®æ“ä½œã¯åŸºæœ¬çš„ã«Map Reduceã®ãƒ©ãƒƒãƒ‘ãƒ¼ã«ãªã£ã¦ã„ã‚‹ã“ã¨ã€‚ è¦ã™ã‚‹ã«ã€SELECTæ–‡å®Ÿè¡Œã™ã‚‹ã¨è£ã§Map&Reduceã®ã‚¿ã‚¹ã‚¯ãŒèµ°ã‚Šå‡º
nantan 2012/07/11
hadoop

hive

mapreduce
ãƒªãƒ³ã‚¯
Hadoop and Hive
- Facebook relies heavily on open source software like Apache Hadoop and Hive for its data warehousing and analytics needs. Hive provides an SQL-like interface to query large datasets stored in Hadoop's HDFS distributed file system using MapReduce. - Hive optimizes SQL queries by merging joins, sharing common operations between queries, and load balancing MapReduce jobs. It uses a metastore to man
nantan 2012/07/11
Facebook

hadoop

hive
ãƒªãƒ³ã‚¯
Amebaã‚µãƒ¼ãƒ“ã‚¹ã®ãƒã‚°è§£æžåŸºç›¤
This document summarizes the Ameba and Neutral Techno logy Group's Patriot and Stinger big data platforms. Patriot is based on Hadoop and Hive and allows analyzing log data through a web UI and Hue. It loads data from MySQL into HDFS and analyzes it with HiveQL. Stinger collects log data with Flume agents and stores it incrementally in HBase for real-time analytics of hourly and daily metrics throu
nantan 2012/07/02
hadoop

hive
ãƒªãƒ³ã‚¯
Hadoopã‚’ã‚ˆã‚Šä¾¿åˆ©ã«ä½¿ã†ï¼Hiveã§ã®MapReduceã¾ã¨ã‚ - Y's note
ç›®æ¬¡ Hiveã¨ã¯ Hiveã®è¨å®š HiveQLæ§‹æ–‡(DDL) DataBase/SCHEMAã®ä½œæˆ Database/SCHEMAã®å‰Šé™¤ Tableã®ä½œæˆ Tableã®å‰Šé™¤ Tableåå¤‰æ›´ Partitionä½œæˆ Partitionã®å‰Šé™¤ Columnã®å¤‰æ›´ Columnã®è¿½åŠ /ç½®ãæ›ãˆ Tableã®Propertyå¤‰æ›´ SerDe Propertyã®è¿½åŠ HiveQLæ§‹æ–‡(SQL) ãƒ†ãƒ¼ãƒ–ãƒ«ä¸€è¦§è¡¨ç¤º ãƒ†ãƒ¼ãƒ–ãƒ«ã®å†…å®¹ã‚’è¡¨ç¤º åŸºæœ¬çš„ãªSELECTæ–‡ WHEREå¥(æ¡ä»¶æŒ‡å®š) DISTINCT(é‡è¤‡å‰Šé™¤) ORDER BY / SORT BYå¥(ã‚½ãƒ¼ãƒˆ) GROUP BYå¥(ã‚°ãƒ«ãƒ¼ãƒ—åŒ–) HAVINGå¥(ã‚°ãƒ«ãƒ¼ãƒ—åŒ–å¾Œã®æ¡ä»¶) LIMITå¥ JOIN(ãƒ†ãƒ¼ãƒ–ãƒ«çµåˆ) æŠ½å‡ºã‚«ãƒ©ãƒ ã‚’æ£è¦è¡¨ç¾ã§æŒ‡å®š UNION(çµæžœã®çµåˆ) SUBQUERY LOAD(ãƒ‡ãƒ¼ã‚¿ã®èªã¿è¾¼ã¿) Hiveã‚’ä½¿ã† äº‹å‰æº–å‚™
nantan 2012/07/02
hadoop

hive
ãƒªãƒ³ã‚¯
Hadoopï¼‹Hiveæ¤œè¨¼ç’°å¢ƒã‚’æ§‹ç¯‰ã—ã¦ã¿ã‚‹
Hadoopï¼‹Hiveæ¤œè¨¼ç’°å¢ƒã‚’æ§‹ç¯‰ã—ã¦ã¿ã‚‹ï¼šHiveâ€•â€•RDBä½¿ã„ã®ãŸã‚ã®Hadoopã‚¬ã‚¤ãƒ‰ï¼ˆå‰ç·¨ï¼‰ï¼ˆ1/3 ãƒšãƒ¼ã‚¸ï¼‰ Hadoop Hiveã¯Hadoopä¸Šã§SQLãƒ©ã‚¤ã‚¯ãªã‚¯ã‚¨ãƒªæ“ä½œãŒå¯èƒ½ãªDWHå‘ã‘ã®ãƒ—ãƒãƒ€ã‚¯ãƒˆã§ã™ã€‚SQLã«è¿‘ã„æ“ä½œãŒå¯èƒ½ãªãŸã‚ã€HBaseã‚ˆã‚Šã‚‚ãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹ã«æ…£ã‚Œè¦ªã—ã‚“ã ã¿ãªã•ã‚“ã«ã¯ä½¿ã„å‹æ‰‹ãŒã„ã„ã‹ã‚‚ã—ã‚Œã¾ã›ã‚“ã€‚æœ¬ç¨¿ã§ã¯ã“ã®Hiveã®ä½¿ã„æ–¹ã¨ãƒ¬ãƒ“ãƒ¥ãƒ¼ã‚’è¡Œã£ã¦ã„ãã¾ã™ã€‚
nantan 2012/07/02
hadoop

hive
ãƒªãƒ³ã‚¯
SQLãƒ©ã‚¤ã‚¯ã«Hadoop Hiveã‚’ä½¿ã„å€’ã™!
ãƒ‘ãƒ¼ãƒ†ã‚£ã‚·ãƒ§ãƒ³ã‚’åˆ©ç”¨ã™ã‚‹ ä»Šå›žã¯å°‘ã—å‡ã£ãŸãƒ†ãƒ¼ãƒ–ãƒ«ã‚’å®šç¾©ã‚’ã—ã¦ã¿ã¾ã—ã‚‡ã†ã€‚ éƒµä¾¿ç•ªå·ãƒ‡ãƒ¼ã‚¿ã¯æ¯Žæœˆæ›´æ–°ã•ã‚Œã‚‹ã®ã§ã€ãƒ†ãƒ¼ãƒ–ãƒ«æŒ‡å®šæ™‚ã«ãƒãƒ¼ã‚¸ãƒ§ãƒ³ã‚‚æŒ‡å®šã§ãã‚‹ã‚ˆã†ã«ã—ã¾ã™ã€‚ã“ã®ã‚ˆã†ãªå ´åˆã€Hiveã§ã¯ãƒ‘ãƒ¼ãƒ†ã‚£ã‚·ãƒ§ãƒ³ã‚’ä½¿ã„ã¾ã™ã€‚ ä»¥ä¸‹ã«éƒµä¾¿ç•ªå·ã‚’ä¿å˜ã™ã‚‹ãƒ†ãƒ¼ãƒ–ãƒ«ã€Œzipã€ã‚’å®šç¾©ã—ã¾ã™ãŒã€æ—¥ä»˜åž‹DATEã®ãƒ‘ãƒ¼ãƒ†ã‚£ã‚·ãƒ§ãƒ³verã‚’è¨å®šã™ã‚‹ã‚ˆã†ã«ã—ã¾ã™ã€‚ hive> CREATE TABLE zip (zip STRING, pref INT, city STRING, town STRING) > PARTITIONED BY (ver DATE) > ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' > LINES TERMINATED BY '\n'; OK Time taken: 0.128 seconds
nantan 2012/07/02
hadoop

hive
ãƒªãƒ³ã‚¯
ä¸¦åˆ—åˆ†æ•£å‡¦ç†ã®å¸¸è˜ã‚’Hadoopãƒ•ã‚¡ãƒŸãƒªã‹ã‚‰å¦ã¶
ä¸¦åˆ—åˆ†æ•£å‡¦ç†ã®å¸¸è˜ã‚’Hadoopãƒ•ã‚¡ãƒŸãƒªã‹ã‚‰å¦ã¶ï¼šãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿å‡¦ç†ã®å¸¸è˜ã‚’Javaã§èº«ã«ã¤ã‘ã‚‹ï¼ˆ2ï¼‰ï¼ˆ3/3 ãƒšãƒ¼ã‚¸ï¼‰ 3ã¤ã®èª²é¡Œã«å¿œãˆã‚‹ã€ŒHadoopãƒ•ã‚¡ãƒŸãƒªã€ é–¢é€£ãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆã¯ã€HadoopãŒæŠ±ãˆã‚‹ä»¥ä¸‹ã®3ã¤ã®èª²é¡Œã«å¿œãˆãŸã‚‚ã®ã§ã™ã€‚ ã€èª²é¡Œ1ã€‘ã€Œä¸¦åˆ—åˆ†æ•£å‡¦ç†ã‚’ã‚‚ã£ã¨ç°¡å˜ã«æ›¸ããŸã„ã€ MapReduceã‚’Javaã§æ›¸ãå‡¦ç†ã¯ã‚ã‚‹ç¨®ã®ã²ã‚‰ã‚ããŒå¿…è¦ã¨ã‚‚ã„ã‚ã‚Œã¾ã™ã€‚ç‰¹ã«ã€SQLã§æ›¸ã‹ã‚Œã¦ã„ãŸå‡¦ç†ã‚’MapReduceã§æ›¸ãæ›¿ãˆã‚‹å ´åˆã€æ€è€ƒã®åˆ‡ã‚Šæ›¿ãˆãŒå¿…è¦ã§è‹¦åŠ´ã™ã‚‹ã“ã¨ã‚‚å¤šã„ã‚ˆã†ã§ã™ã€‚ãŸã¨ãˆæ…£ã‚Œã¦ã„ã¦ã‚‚ã€Hadoopã§ç°¡å˜ãªãƒ‡ãƒ¼ã‚¿åŠ å·¥ã—ãŸã„ã€ãã®ãŸã‚ã ã‘ã«å¤šãã®ã‚¯ãƒ©ã‚¹å®šç¾©ã‚’ã™ã‚‹ã®ã¯é¢å€’ã§ã™ã€‚ ã“ã®é›£æ˜“åº¦ã‚’ä¸‹ã’ã‚‹ãƒã‚¤ãƒ¬ãƒ™ãƒ«è¨€èªžã¨ã—ã¦ã€ŒHiveã€ã€ŒPigã€ãŒå˜åœ¨ã—ã¾ã™ã€‚ã©ã¡ã‚‰ã®è¨€èªžã‚‚SQLã‚„ã‚¹ãƒˆã‚¢ãƒ‰ãƒ—ãƒã‚·ã‚¸ãƒ£ã®ã‚ˆã†ãªã‚³ãƒ¼ãƒ‰ã‚’æ›¸ã„ã¦ã€MapReduceã‚’å‹•ã‹ã›ã¾ã™ã€‚å¯¾è©±åž‹ã®ã‚·ã‚§ãƒ«ãƒ¦ãƒ¼ãƒ†ã‚£ãƒªãƒ†ã‚£ã‚‚ã‚ã‚Š
nantan 2012/07/02
hadoop

hive

MapReduce
ãƒªãƒ³ã‚¯
Hiveãƒ¡ãƒ¢(Hishidama's Apache Hive Memo)
Apache Hive(ãƒã‚¤ãƒ´)ã¯ã€Hadoopï¼ˆHDFSï¼‰ä¸Šã®ãƒ•ã‚¡ã‚¤ãƒ«ã‚’SQLã«ä¼¼ãŸå‘½ä»¤ã§æ“ä½œï¼ˆMapReduceã‚’å®Ÿè¡Œï¼‰ã§ãã‚‹ãƒ„ãƒ¼ãƒ«ã€‚ å¤§å‰æ Hiveã¯RDBMSã®SQLã«ä¼¼ãŸæ§‹æ–‡ï¼ˆHiveQLã¨ã„ã†åç§°ï¼‰ã§ãƒ‡ãƒ¼ã‚¿æ“ä½œã‚’è¨˜è¿°ã§ãã‚‹ã€‚Oracleã®SQL*Plusã®ã‚ˆã†ãªå¯¾è©±åž‹ãƒ„ãƒ¼ãƒ«ã€‚ HDFSä¸Šã®ãƒ•ã‚¡ã‚¤ãƒ«ã‚’ãƒ†ãƒ¼ãƒ–ãƒ«ã«è¦‹ç«‹ã¦ã€SQLã§ãƒ†ãƒ¼ãƒ–ãƒ«ã‚’æ‰±ã†æ„Ÿè¦šã§ã€HiveQLã§ãƒ•ã‚¡ã‚¤ãƒ«ã‚’æ‰±ã†ã€‚ é‡è¦ãªã®ã¯ã€RDBMSã¨ã¯ç•°ãªã‚Šã€æ“ä½œå¯¾è±¡ã¯ã‚ãã¾ã§HDFSä¸Šã®ãƒ•ã‚¡ã‚¤ãƒ«ã§ã‚ã‚‹ã“ã¨ã€‚ HDFSã®ãƒ•ã‚¡ã‚¤ãƒ«ã¯ã€åŸºæœ¬çš„ã«æ›´æ–°ã¯å‡ºæ¥ãªã„ã€‚ä½•ã‹ãƒ‡ãƒ¼ã‚¿ã‚’å¤‰æ›´ã™ã‚‹ã¨ãã¯ã€å…¨ä»¶èªã¿è¾¼ã‚“ã§åŠ å·¥ã—ã¦æ–°ã—ã„ãƒ•ã‚¡ã‚¤ãƒ«ã‚’ä½œã‚‹ã®ãŒåŸºæœ¬çš„ãªè€ƒãˆæ–¹ã€‚ Hiveã®æ“ä½œå¯¾è±¡ã¯HDFSãªã®ã§ã€è€ƒãˆæ–¹ã¯åŒã˜ã€‚ã—ãŸãŒã£ã¦ã€Hiveã«ã¯ã€ŒINSERT INTOï¼ˆãƒ‡ãƒ¼ã‚¿è¿½åŠ ï¼‰ã€ã‚„ã€ŒUPDATEï¼ˆæ›´æ–°ï¼‰ã€ã€ŒDELETEï¼ˆå‰Šé™¤ï¼‰ã€ã¯åŸºæœ¬çš„ã«ã¯å˜åœ¨ã—
nantan 2012/06/25
apache

hadoop

hive
ãƒªãƒ³ã‚¯
1