[B! hadoop][hive] yukungã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

yukung id:yukung

hadoopã¨hiveã«é–¢ã™ã‚‹yukungã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (5)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

Prestoã¨ã‹Ansibleã¨ã‹ãã®è¾ºã®è©±ã‚’è»½ãæ›¸ã„ã¦ã¿ã‚‹ - wyukawa's diary
ä»Šæ—¥ã¯Prestoã¨ã‹Ansibleã¨ã‹ãã®è¾ºã®è©±ã‚’è»½ãæ›¸ã„ã¦ã¿ã‚ˆã†ã¨æ€ã„ã¾ã™ã€‚çªã£è¾¼ã‚“ã è©±ãŒå‡ºæ¥ã‚‹ã‚ã‘ã§ã¯ãªã„ã®ã§ã‚ã—ã‹ã‚‰ãšã€‚ åƒ•ã®ã¨ã“ã‚ã®ç’°å¢ƒã§ã¯Prestoã‚’ä½¿ã£ã¦ã„ã¦ã€Prestoã¯DataNodeã‚„NodeManagerã¨åŒå±…ã—ã¦ã¾ã™ã€‚ä¸»ãªãƒ¦ãƒ¼ã‚¹ã‚±ãƒ¼ã‚¹ã¯ã‚¢ãƒ‰ãƒ›ãƒƒã‚¯ã‚¯ã‚¨ãƒªã®å®Ÿè¡Œã§ã™ã€‚ã¨ã‚ã‚‹ãƒ¬ãƒãƒ¼ãƒˆã‚’ä½œã‚ŠãŸã„ã£ã¦ãªã£ãŸã¨ãã«ãƒ‡ãƒ¼ã‚¿ã®ä¸èº«ã‚’ãƒã‚§ãƒƒã‚¯ã™ã‚‹ã®ã«ä½¿ã„ã¾ã™ã€‚å¾“æ¥ã ã¨ã“ã‚ŒãŒHiveã ã£ãŸã®ã§ã™ãŒã€Hiveã ã¨MapReduceã«ãªã£ã¦é…ã„ã®ã§ï¼ˆãƒãƒ¼ã‚«ãƒ«ãƒ¢ãƒ¼ãƒ‰ã§æ¸ˆã‚€å ´åˆã‚‚ã‚ã‚‹ã‘ã©ï¼‰ã€ãã®ç‚¹Prestoã¯æ—©ãã¦ã„ã„ã§ã™ã€‚ãŸã ã“ã‚Œã¯åƒ•ã®ç’°å¢ƒãŒã‚¹ãƒ¢ãƒ¼ãƒ«ãƒ‡ãƒ¼ã‚¿ã ã‹ã‚‰ã£ã¦ã„ã†ã®ã‚‚ã‚ã£ã¦ã€åœ§ç¸®æ¸ˆã¿æ•°ç™¾GBã®ãƒ‡ãƒ¼ã‚¿ã«å¯¾ã—ã¦selectã‹ã‘ã‚‹ã¨ã‹ã ã¨Prestoã¨ã„ãˆã©ã‚‚é…ããªã‚‹ã¨æ€ã„ã¾ã™ã€‚ã‚ã¨ãªã«ã’ã«è‰¯ã„ã®ãŒPresto CLIçµŒç”±ã ã¨ã‚«ãƒ©ãƒ åãŒè¡¨ç¤ºã•ã‚Œã‚‹ã®ã§ã©ã®ãƒ‡ãƒ¼ã‚¿ãŒã©ã®ã‚«ãƒ©ãƒ ãªã®ã‹ã™ãåˆ†ã‹
yukung 2014/08/04
hive

hadoop

ansible

presto

hiveserver

hugedata
ãƒªãƒ³ã‚¯
Facebookã€Hadoopã®ã‚¹ã‚±ãƒ¼ãƒ©ãƒ“ãƒªãƒ†ã‚£å•é¡Œã‚’è§£æ±ºã™ã‚‹ã€ç‹¬è‡ªã®ã€ŒHadoop Coronaã€ã‚’ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ã§å…¬é–‹
Facebookã€Hadoopã®ã‚¹ã‚±ãƒ¼ãƒ©ãƒ“ãƒªãƒ†ã‚£å•é¡Œã‚’è§£æ±ºã™ã‚‹ã€ç‹¬è‡ªã®ã€ŒHadoop Coronaã€ã‚’ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ã§å…¬é–‹ Facebookã§ã¯ã€24æ™‚é–“ã”ã¨ã«0.5ãƒšã‚¿ãƒã‚¤ãƒˆã®ãƒ‡ãƒ¼ã‚¿ãŒç”Ÿæˆã•ã‚Œã€ãã‚Œã‚‰ã‚’åˆ†æžã™ã‚‹ãŸã‚ã«æ¯Žæ—¥6ä¸‡å›žä»¥ä¸Šã‚‚ã®Hiveã®ã‚¯ã‚¨ãƒªãŒå®Ÿè¡Œã•ã‚Œã¦ã„ã‚‹ãã†ã§ã™ã€‚ ã“ã†ã—ãŸå¤§è¦æ¨¡å‡¦ç†ã‚’è¡Œã†ãƒ—ãƒ©ãƒƒãƒˆãƒ•ã‚©ãƒ¼ãƒ ã¨ã—ã¦ä½¿ã‚ã‚Œã¦ã„ã‚‹ã®ãŒHadoopã€‚ã—ã‹ã—Facebookã¯Hadoop MapReduceã®ã‚¹ã‚±ãƒ¼ãƒ©ãƒ“ãƒªãƒ†ã‚£ã«é™ç•Œã‚’æ„Ÿã˜ã¦ãŠã‚Šã€ãã‚Œã‚’è§£æ±ºã™ã‚‹ãŸã‚ã®æ–°ã—ã„ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ã€ŒHadoop Coronaã€ã‚’é–‹ç™ºã€ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ã§å…¬é–‹ã—ã¾ã—ãŸã€‚ Facebookã®ãƒšãƒ¼ã‚¸ã€ŒUnder the Hood: Scheduling MapReduce jobs more efficiently with Coronaã€ã§ã¯ã€å¾“æ¥ã®Hadoop MapReduceã®ã©ã“ã«èª²é¡ŒãŒã‚ã£ãŸã®ã‹ã€4ã¤
yukung 2012/11/12
hadoop

mapreduce

hive

corona
ãƒªãƒ³ã‚¯
ã€ŽHadoop/Hiveã‚’ç”¨ã„ãŸãƒã‚°è§£æžåŸºç›¤ã®æ§‹ç¯‰ã€
ã“ã‚“ã«ã¡ã¯ã€‚Amebaã®ãƒã‚°è§£æžåŸºç›¤ã‚’æ‹…å½“ã—ã¦ã„ã‚‹ICHIROã§ã™ã€‚ ä»Šå›žã¯æ§‹ç¯‰ãƒ»é‹ç”¨ä¸ã®ãƒã‚°è§£æžåŸºç›¤ã€ŒPatriotã€ã«ã¤ã„ã¦æ›¸ããŸã„ã¨æ€ã„ã¾ã™ã€‚ Webã‚µãƒ¼ãƒ“ã‚¹ã‚’é‹å–¶ã—ã¦ã„ã‚‹ã¨æ—¥ã€…å¤§é‡ã®ãƒã‚°ãƒ‡ãƒ¼ã‚¿ã‚„ãƒ¦ãƒ¼ã‚¶ãƒ‡ãƒ¼ã‚¿ãŒè“„ç©ã•ã‚Œã¾ã™ã€‚ ä»Šã¾ã§ã¯PVï¼ˆãƒšãƒ¼ã‚¸ãƒ“ãƒ¥ãƒ¼ï¼‰ã‚„UUï¼ˆãƒ¦ãƒ‹ãƒ¼ã‚¯ãƒ¦ãƒ¼ã‚¶ï¼‰ãªã©ã‚¢ã‚¯ã‚»ã‚¹ãƒã‚°ãªã©ã‹ã‚‰å–ã‚Œã‚‹ã€å¤§æž ã®æŒ‡æ¨™ã®ã¿ã‚’è¦‹ã‚‹ã“ã¨ãŒã»ã¨ã‚“ã©ã§ã—ãŸã€‚ ãƒšãƒ¼ã‚¸ãƒ“ãƒ¥ãƒ¼ã«åˆã‚ã›ã¦ã‚·ã‚¹ãƒ†ãƒ ã‚’å¢—å¼·ã™ã‚‹ãªã©ã€ã‚·ã‚¹ãƒ†ãƒ å´ã¨ã—ã¦ã¯ååˆ†ã¨ã‚‚è¨€ãˆã‚‹æŒ‡æ¨™ã§ã™ãŒã€åºƒå‘Šã‚„èª²é‡‘ã‚µãƒ¼ãƒ“ã‚¹ã¨ã„ã†è¦–ç‚¹ã‹ã‚‰è¦‹ã‚‹ã¨ååˆ†ã¨ã¯è¨€ãˆã¾ã›ã‚“ã€‚ ä»Šã¾ã§ã¯Amebaå†…ã®å€‹ã€…ã®ã‚µãƒ¼ãƒ“ã‚¹æ‹…å½“è€…ãŒå¿…è¦ã¨ã™ã‚‹æŒ‡æ¨™ã‚’å‡ºã™ãŸã‚ã«ã‚¢ãƒ—ãƒªã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ã‚„DB ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ã«éƒ½åº¦ä¾é ¼ã‚’ã™ã‚‹å½¢ã§ãƒ‡ãƒ¼ã‚¿ã‚’æŠ½å‡ºã—ã¦ã„ã¾ã—ãŸã€‚ ä»Šå¾Œã®èª²é‡‘ã‚µãƒ¼ãƒ“ã‚¹ã®ç™ºå±•ãŒè¦‹è¾¼ã¾ã‚Œã€ã‚ˆã‚Šãƒ‡ãƒ¼ã‚¿åˆ†æžã®é‡è¦æ€§ãŒé«˜ã¾ã‚‹ã¨è€ƒãˆãŸç§ã¯ã€ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ã§ãªã„ã‚µãƒ¼ãƒ“ã‚¹æ‹…å½“è€…ï¼ˆä¸»ã«ãƒ—ãƒãƒ‡ãƒ¥ãƒ¼ã‚µï¼‰ãŒã‚ˆã‚Šç°¡å˜
yukung 2011/10/17
development

hadoop

hive

hugedata

logging
ãƒªãƒ³ã‚¯
Hadoopã§ã®MapReduceå®Ÿè¡Œæ–¹æ³• - wyukawa's diary
Hadoopã§ã®MapReduceå®Ÿè¡Œæ–¹æ³•ã¯ä»¥ä¸‹ã®5ã¤ã‚ã‚‹ã ã‚ã†ã€‚ Javaã§ã”ã‚Šã”ã‚Šå®Ÿè£… Asakusaã¤ã‹ã£ã¦å®Ÿè£… Hadoopã‚¹ãƒˆãƒªãƒ¼ãƒŸãƒ³ã‚°ã‚’ä½¿ã£ã¦å®Ÿè£… Pigä½¿ã† Hiveä½¿ã† åƒ•è‡ªèº«ã¯ã‚‚ã£ã±ã‚‰Hiveã ã€‚ Asakusaã¯å…¨ç„¶ã‚ã‹ã‚‰ãªã„ã®ã§ã“ã“ã§ã¯è§¦ã‚Œãªã„ãŒã€ãã‚Œä»¥å¤–ã¯è±¡æœ¬èªã‚“ã§ä½•ã¨ãªãé›°å›²æ°—ã¯ã¤ã‹ã‚ãŸã€‚ åˆ†é¡žçš„ã«è¨€ã†ã¨Pigã¨Hiveã¯æ‰‹ç¶šãçš„ã€å®£è¨€çš„ã¨ã„ã†é•ã„ã¯ã‚ã‚‹ãŒDSLã¨ã„ã†æ„å‘³ã§ã¯åŒã˜ã‚ˆã†ãªã‚‚ã®ãªã®ã§ã“ã“ã§ã¯ã‚ã¾ã‚Šè§¦ã‚Œãªã„ã€‚ ã¾ãšJavaã§ã”ã‚Šã”ã‚Šå®Ÿè£…ã™ã‚‹å ´åˆã ãŒã“ã‚Œã¯ãªã‹ãªã‹å¤§å¤‰ã ã€‚ ã¨ãã«çµåˆãŒå¤§å¤‰ã€‚çµåˆãŒã©ã‚“ãªæ„Ÿã˜ã‹ã¯è±¡æœ¬ã®8ç« ã«æ›¸ã„ã¦ã‚ã‚‹ã€‚ ãã‚Œä»¥å¤–ã®è³‡æ–™ã¨ã—ã¦ã¯ä»¥å‰@n3104ã•ã‚“ãŒ 5æœˆ11æ—¥ DevLOVE é»„è‰²ã„è±¡ä½¿ã„ãŒã€ç„é•·ã«å‡ºä¼šã†ã¾ã§ã€‚ã€œè±¡ã€é‚‚é€…ç·¨ã€œ(æ±äº¬éƒ½)ã§èª¬æ˜Žã—ã¦ã„ã‚‹è³‡æ–™ã¨ã‚µãƒ³ãƒ—ãƒ«ã‚³ãƒ¼ãƒ‰ãŒè‰¯ã„é¡Œæã«ãªã‚‹ã ã‚ã†ã€‚ã‚»ã‚«ãƒ³ãƒ€ãƒªã‚½ãƒ¼ãƒˆã‚„map side
yukung 2011/09/26
development

hadoop

mapreduce

hive
ãƒªãƒ³ã‚¯
Hiveã®ãƒ™ã‚¹ãƒˆãƒ—ãƒ©ã‚¯ãƒ†ã‚£ã‚¹(ã‹ã‚‚ã—ã‚Œãªã„ã“ã¨)ã‚’ã‚ã‚‚ã£ã¦ãŠã - wyukawa's diary
Hiveã®ç¾å ´ã«æ¥ã¦4ã‹æœˆçµŒã¡å›žã‚Šã®ã‚„ã‚Šæ–¹ã‚’è¦³å¯Ÿã—ãŸã‚Šã€ä»–ã®äº‹ä¾‹ã‚’èª¿ã¹ãŸã‚Šã—ã¦ã„ã‚‹ã†ã¡ã«Hiveã‚’ä½¿ç”¨ã™ã‚‹éš›ã®ãƒ‘ã‚¿ãƒ¼ãƒ³ã¨ã„ã†ã‹ãƒ™ã‚¹ãƒˆãƒ—ãƒ©ã‚¯ãƒ†ã‚£ã‚¹ãŒè¦‹ãˆã¦ããŸæ°—ãŒã™ã‚‹ã®ã§æ›¸ã„ã¦ã¿ã‚‹ã‚ˆã€‚ ãƒ¦ãƒ¼ã‚¹ã‚±ãƒ¼ã‚¹ã¯ãƒã‚°è§£æžã§ã™ã€‚ 1. ãƒ•ã‚¡ã‚¤ãƒ«ãƒ•ã‚©ãƒ¼ãƒžãƒƒãƒˆã¨åœ§ç¸® ãƒã‚°ã¯åœ§ç¸®çŽ‡é«˜ã„ã®ã§åœ§ç¸®ã™ã¹ã—ã€‚ã§ã‚‚ã‚¹ãƒ—ãƒªãƒƒãƒˆå¯èƒ½ã§ã‚ã‚‹ãŸã‚ã«ã¯ã£ã¦ã“ã¨ã§SequenceFileã§ãƒ–ãƒãƒƒã‚¯åœ§ç¸®ã¯ç¢ºå®šã€‚ åœ§ç¸®ã‚³ãƒ¼ãƒ‡ãƒƒã‚¯ã¯åœ§ç¸®çŽ‡ã‚’è€ƒãˆã¦gzipã€‚ ã¡ãªã¿ã«åœ§ç¸®çŽ‡ã¯ bzip2>gzip>LZO ã§ã‚¹ãƒ”ãƒ¼ãƒ‰ã¯ LZO>gzip>bzip2 ã ã€‚bzipã¯ã‚¹ãƒ—ãƒªãƒƒãƒˆå¯èƒ½ã§åœ§ç¸®çŽ‡ã‚‚é«˜ã„ã‚“ã ã‘ã©Hadoop 0.20ç³»ã§ã¯ã¾ã ä½¿ãˆãªã‹ã£ãŸã¨æ€ã†ã€‚Hadoop 0.20ç³»ã§ã‚‚ä½¿ãˆã‚‹ã‚‰ã—ã„ã§ã™ãŒé…ãã¦ä½¿ã„ç‰©ã«ãªã‚‰ãªã„ã‚ˆã†ã§ã™ã€‚ æœ€è¿‘ã¯snappyãªã‚“ã¦ã®ã‚‚å‡ºã¦ã¦CDH3 Update 1ã§ä½¿ãˆã‚‹ã‚ˆã†ã«ãªã‚Šã¾ã—ãŸãã€‚ snappyã¯åœ§ç¸®çŽ‡ã¯L
yukung 2011/09/26
development

hadoop

hive

tips
ãƒªãƒ³ã‚¯
1