[B! hadoop] dirablueã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

dirablue id:dirablue

hadoopã«é–¢ã™ã‚‹dirablueã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (9)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

Hadoopã®SQLå¯¾å¿œåˆ†æ•£ã‚¯ã‚¨ãƒªã‚¨ãƒ³ã‚¸ãƒ³ã€ŒCloudera Impalaã€ã€‚ClouderaãŒã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ã§å…¬é–‹
Hadoopã®SQLå¯¾å¿œåˆ†æ•£ã‚¯ã‚¨ãƒªã‚¨ãƒ³ã‚¸ãƒ³ã€ŒCloudera Impalaã€ã€‚ClouderaãŒã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ã§å…¬é–‹ Hadoopã®ãƒ‡ã‚£ã‚¹ãƒˆãƒªãƒ“ãƒ¥ãƒ¼ã‚·ãƒ§ãƒ³ãƒ™ãƒ³ãƒ€ã¨ã—ã¦çŸ¥ã‚‰ã‚Œã‚‹Clouderaã¯10æœˆ25æ—¥ã€SQLã«å¯¾å¿œã—ã€ãƒ‡ãƒ¼ã‚¿ã®åˆ†æžé€Ÿåº¦ã¯MapReduceã‚ˆã‚Šã‚‚ä½•å€ã‚‚é«˜é€Ÿã ã¨ã„ã†æ–°ã—ã„åˆ†æ•£ã‚¯ã‚¨ãƒªã‚¨ãƒ³ã‚¸ãƒ³ã€ŒCloudera Impalaã€ï¼ˆè£½å“åã€ŒCloudera Enterprise RTQã€ï¼‰ã‚’ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ã§å…¬é–‹ã—ã¾ã—ãŸã€‚ ã“ã‚Œã¾ã§Hadoopã§ã¯å†…éƒ¨ã§MapReduceã¨å‘¼ã°ã‚Œã‚‹å‡¦ç†ãŒç”¨ã„ã‚‰ã‚Œã¦ã„ã¾ã—ãŸãŒã€Impalaã§ã¯MapReduceã‚’ä½¿ã‚ãšã€ClouderaãŒ2å¹´ã‹ã‘ã¦é–‹ç™ºã—ãŸç‹¬è‡ªã®åˆ†æ•£ã‚¯ã‚¨ãƒªã‚¨ãƒ³ã‚¸ãƒ³ã‚’ç”¨ã„ã¦å‡¦ç†ã‚’è¡Œã„ã¾ã™ã€‚Hiveã®ä¸Šä½äº’æ›ã®SQLãŒåˆ©ç”¨ã§ãã€Hive/MapReduceã§æ•°åˆ†ã‹ã‹ã£ã¦ã„ãŸå¿œç”æ™‚é–“ã‚’æ•°ç§’ã«çŸç¸®ã™ã‚‹ã¨èª¬æ˜Žã•ã‚Œã¦ã„ã¾ã™ã€‚ ã‚°ãƒ¼ã‚°ãƒ«ã®Dremel
dirablue 2012/10/27
hadoop

mapreduce
ãƒªãƒ³ã‚¯
HADOOPã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ãƒˆãƒ©ã‚¤ä¸
dirablue 2012/10/23
hadoop
ãƒªãƒ³ã‚¯
Hadoop on MacOSX
MacOSXã«Hadoopã‚’æ“¬ä¼¼åˆ†æ•£ãƒ¢ãƒ¼ãƒ‰ã§å‹•ä½œã§ãã‚‹ã‚ˆã†ã«ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã—ãŸæ™‚ã®è¨˜éŒ²ã§ã™ã€‚ åˆ©ç”¨ã—ãŸç’°å¢ƒã¯æ¬¡ã®é€šã‚Šã§ã™ã€‚ MacOSX Snow LeopardHadoop 0.20.2 ã¾ãŸã€å‚è€ƒã«ã—ãŸãƒšãƒ¼ã‚¸ã¯æ¬¡ã®ãƒšãƒ¼ã‚¸ã§ã™ã€‚ Hadoop, HDFS and Hbase on Ubuntu & Macintosh LeopardHBase/Hadoop on Mac OS Xã“ã®ãƒšãƒ¼ã‚¸ã«ã¯HBaseã®ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã¾ã§è¨˜è¼‰ã•ã‚Œã¦ã„ã¾ã™ãŒã€ã¾ã ãã“ã¾ã§ã‚„ã£ã¦ã„ãªã„ã®ã§ã€æ©Ÿä¼šãŒã‚ã‚Œã°ã¨ã„ã†ã“ã¨ã§ã€‚ ãƒ€ã‚¦ãƒ³ãƒãƒ¼ãƒ‰ãƒ€ã‚¦ãƒ³ãƒãƒ¼ãƒ‰ãƒšãƒ¼ã‚¸ã‹ã‚‰ã€é©å½“ãªãƒŸãƒ©ãƒ¼ã‚µã‚¤ãƒˆã‚’é¸ã‚“ã§ãƒ€ã‚¦ãƒ³ãƒãƒ¼ãƒ‰ã€‚ /usr/local ã«å±•é–‹ã—ã¾ã™ã€‚ $ pwd /usr/local $ sudo tar zxvf /path/to/hadoop-0.20.2.tar.gz $ sudo ln -s hadoop-0.20.2 h
dirablue 2012/09/23
hadoop

mac

hive
ãƒªãƒ³ã‚¯
Runtime error - Meta Search
Error message : Directory is not found or not writable (DATA_DIR) Directory is not found or not writable (DIFF_DIR) Directory is not found or not writable (BACKUP_DIR) Directory is not found or not writable (CACHE_DIR) Site admin: whitestar Copyright Â© 2006-2023 whitestar. All Rights Reserved. Icons powered by famfa mfam. PukiWiki 1.5.0 Copyright Â© 2001-2006 PukiWiki Developers Team. License is GPL
dirablue 2012/09/23
pig

hive

hadoop

map reduce
ãƒªãƒ³ã‚¯
http://infra-engineer.com/hadoop/hadoop%E8%B1%A1%E6%9C%AC%E7%AC%AC2%E7%89%88-11%E7%AB%A0-pig-%E3%81%AE%E3%81%BE%E3%81%A8%E3%82%81/
dirablue 2012/06/19
pigã®ãƒãƒ¼ã‚«ãƒ«å®Ÿè¡Œ

pig

hadoop
ãƒªãƒ³ã‚¯
PigInstall â€“ ã‚¢ã‚¯ã‚»ãƒ³ã‚¹ã®ãŠã¾ã‘
dirablue 2012/06/19
pig

hadoop
ãƒªãƒ³ã‚¯
ZooKeeper: é¢å€’ãªåˆ†æ•£ã‚·ã‚¹ãƒ†ãƒ ã®ã‚³ãƒ¼ãƒ‡ã‚£ãƒãƒ¼ãƒˆã‚’æ”¯æ´ã™ã‚‹ã‚µãƒ¼ãƒ“ã‚¹
ZooKeeper: é¢å€’ãªåˆ†æ•£ã‚·ã‚¹ãƒ†ãƒ ã®ã‚³ãƒ¼ãƒ‡ã‚£ãƒãƒ¼ãƒˆã‚’æ”¯æ´ã™ã‚‹ã‚µãƒ¼ãƒ“ã‚¹ ZooKeeper ã¯ã€åˆ†æ•£ã‚¢ãƒ—ãƒªã‚±ãƒ¼ã‚·ãƒ§ãƒ³ã®ãŸã‚ã®ãƒ‘ãƒ•ã‚©ãƒ¼ãƒžãƒ³ã‚¹ã®é«˜ã„ã‚³ãƒ¼ãƒ‡ã‚£ãƒãƒ¼ã‚·ãƒ§ãƒ³ã‚µãƒ¼ãƒ“ã‚¹ã§ã™ã€‚ZooKeeper ã§ã¯ã€åå‰ä»˜ã‘ã€è¨å®šç®¡ç†ã€åŒæœŸåŒ–ã€å„ç¨®ã‚°ãƒ«ãƒ¼ãƒ—ã‚µãƒ¼ãƒ“ã‚¹ãªã©ã€åˆ©ç”¨é »åº¦ã®é«˜ã„ã•ã¾ã–ã¾ãªã‚µãƒ¼ãƒ“ã‚¹ãŒç”¨æ„ã•ã‚Œã¦ã„ã‚‹ã®ã§ã€ã“ã‚Œã‚‰ã®ã‚µãƒ¼ãƒ“ã‚¹ã‚’ã‚¼ãƒã‹ã‚‰ä½œæˆã™ã‚‹å¿…è¦ã¯ã‚ã‚Šã¾ã›ã‚“ã€‚ZooKeeper ã‚’ãã®ã¾ã¾ä½¿ãˆã°ã€åˆæ„ãƒ—ãƒãƒˆã‚³ãƒ«ã€ã‚°ãƒ«ãƒ¼ãƒ—ç®¡ç†ãƒ—ãƒãƒˆã‚³ãƒ«ã€ãƒªãƒ¼ãƒ€ãƒ¼é¸æŒ™ãƒ—ãƒãƒˆã‚³ãƒ«ã€ãŠã‚ˆã³ãƒ—ãƒ¬ã‚¼ãƒ³ã‚¹ãƒ—ãƒãƒˆã‚³ãƒ«ã‚’å®Ÿè£…ã§ãã¾ã™ã€‚ZooKeeper ã‚’åŸºã«ã€ç‹¬è‡ªã®ãƒ‹ãƒ¼ã‚ºã«å¯¾å¿œã—ãŸã‚µãƒ¼ãƒ“ã‚¹ã‚’æ§‹ç¯‰ã™ã‚‹ã“ã¨ã‚‚ã§ãã¾ã™ã€‚ ä»¥ä¸‹ã®ãƒ‰ã‚ãƒ¥ãƒ¡ãƒ³ãƒˆã§ã¯ã€ZooKeeper ã®ã‚³ãƒ³ã‚»ãƒ—ãƒˆã‚„ ZooKeeper ã‚’ä½¿ã†ãŸã‚ã®æ‰‹é †ã«ã¤ã„ã¦èª¬æ˜Žã—ã¦ã„ã¾ã™ã€‚ãƒ‰ã‚ãƒ¥ãƒ¡ãƒ³ãƒˆã‚’èªã‚“ã§ã‚ã‹ã‚‰ãªã„ã“ã¨ãŒã‚ã£ãŸã‚‰ã€ãƒ¡ãƒ¼ãƒªãƒ³ã‚°ãƒªã‚¹ãƒˆã«è³ªå•ã‚’æŠ•ç¨¿ã™ã‚‹ã‹ã€ã‚¢
dirablue 2011/11/22
åˆ†æ•£ãƒãƒƒã‚¯ æ—¥æœ¬èªžãƒ‰ã‚ãƒ¥ãƒ¡ãƒ³ãƒˆ

ZooKeeper

hadoop
ãƒªãƒ³ã‚¯
MapReduce - naoyaã®ã¯ã¦ãªãƒ€ã‚¤ã‚¢ãƒªãƒ¼
"MapReduce" ã¯ Google ã®ãƒãƒƒã‚¯ã‚¨ãƒ³ãƒ‰ã§åˆ©ç”¨ã•ã‚Œã¦ã„ã‚‹ä¸¦åˆ—è¨ˆç®—ã‚·ã‚¹ãƒ†ãƒ ã§ã™ã€‚æ¤œç´¢ã‚¨ãƒ³ã‚¸ãƒ³ã®ã‚¤ãƒ³ãƒ‡ãƒƒã‚¯ã‚¹ä½œæˆã‚’ã¯ã˜ã‚ã¨ã™ã‚‹ã€å¤§è¦æ¨¡ãªå…¥åŠ›ãƒ‡ãƒ¼ã‚¿ã«å¯¾ã™ã‚‹ãƒãƒƒãƒå‡¦ç†ã‚’æƒ³å®šã—ã¦ä½œã‚‰ã‚ŒãŸã‚·ã‚¹ãƒ†ãƒ ã§ã™ã€‚ MapReduce ã®é¢ç™½ã„ã¨ã“ã‚ã¯ã€map() ã¨ reduce() ã¨ã„ã†äºŒã¤ã®é–¢æ•°ã®çµ„ã¿åˆã‚ã›ã‚’å®šç¾©ã™ã‚‹ã ã‘ã§ã€å¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿ã«å¯¾ã™ã‚‹æ§˜ã€…ãªè¨ˆç®—å•é¡Œã‚’è§£æ±ºã™ã‚‹ã“ã¨ãŒã§ãã‚‹ç‚¹ã§ã™ã€‚ MapReduce ã®è¨ˆç®—ãƒ¢ãƒ‡ãƒ« map() ã«ã¯ãã®è¨ˆç®—å•é¡Œã®ãƒ‡ãƒ¼ã‚¿ã¨ã—ã¦ã® key-value ãƒšã‚¢ãŒæ¬¡ã€…ã«æ¸¡ã£ã¦ãã¾ã™ã€‚map() ã§ã¯ key-value å€¤ã®ãƒšã‚¢ã‚’ç•°ãªã‚‹è¤‡æ•°ã® key-value ãƒšã‚¢ã«å¤‰æ›ã—ã¾ã™ã€‚reduce() ã«ã¯ã€map() ã§ä½œã£ãŸ key-value ãƒšã‚¢ã‚’åŒä¸€ã® key ã§æŸããŸã‚‚ã®ãŒé †ç•ªã«æ¸¡ã£ã¦ãã¾ã™ã€‚ãã® key-values ãƒšã‚¢ã‚’ä»»æ„ã®å½¢å¼ã«å¤‰æ›ã™ã‚‹ã“ã¨
dirablue 2011/04/17
mapreduce

ã‚¢ãƒ«ã‚´ãƒªã‚ºãƒ

hadoop
ãƒªãƒ³ã‚¯
MapReduceã®Javaå®Ÿè£…Apache Hadoopã‚’ä½¿ã£ã¦ã¿ãŸ (1/3) - @IT
MapReduceã®Javaå®Ÿè£…Apache Hadoopã‚’ä½¿ã£ã¦ã¿ãŸï¼šã„ã¾å†æ³¨ç›®ã®åˆ†æ•£å‡¦ç†æŠ€è¡“ï¼ˆå¾Œç·¨ï¼‰ï¼ˆ1/3 ãƒšãƒ¼ã‚¸ï¼‰ æœ€è¿‘æ³¨ç›®ã‚’æµ´ã³ã¦ã„ã‚‹åˆ†æ•£å‡¦ç†æŠ€è¡“ã€ŒMapReduceã€ã®åˆ©ç‚¹ã‚’ã‚µãƒ³ãƒ—ãƒ«ã‹ã‚‰ã‚¢ãƒ«ã‚´ãƒªã‚ºãƒ ãƒ¬ãƒ™ãƒ«ã§ç†è§£ã—ã€æ˜”ã‹ã‚‰ã‚ã‚‹Javaé–¢é€£ã®åˆ†æ•£å‡¦ç†æŠ€è¡“ã‚’è¦‹ç›´ã™ç‰¹é›†ä¼ç”»ï¼ˆç·¨é›†éƒ¨ï¼‰ Apache Hadoopãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆã¨ã¯ä½•ã‹ï¼Ÿ æœ¬ç‰¹é›†ã§ã¯ã€ã„ã¾å†æ³¨ç›®ã®åˆ†æ•£å‡¦ç†æŠ€è¡“ã«ã¤ã„ã¦ã„ã‚ã„ã‚ã¨ç´¹ä»‹ã—ã¦ãã¾ã—ãŸã€‚å‰ç·¨ã®ã€ŒGoogleã®MapReduceã‚¢ãƒ«ã‚´ãƒªã‚ºãƒ ã‚’Javaã§ç†è§£ã™ã‚‹ã€ã§ã¯ã€åˆ†æ•£å‡¦ç†æŠ€è¡“ã§æœ€è¿‘æ³¨ç›®ã‚’æµ´ã³ã¦ã„ã‚‹ã‚‚ã®ã¨ã—ã¦ã€ã‚°ãƒ¼ã‚°ãƒ«ã®MapReduceã‚¢ãƒ«ã‚´ãƒªã‚ºãƒ ã‚’ç´¹ä»‹ã—ã€ä¸ç·¨ã®ã€Œã‚¤ãƒã‚¤ãƒãªåˆ†æ•£å‡¦ç†æŠ€è¡“ã¨ã‚¤ãƒžãƒ‰ã‚ã®Webã‚µãƒ¼ãƒ“ã‚¹ã€ã§ã¯ã€MapReduceã«è‡³ã‚‹ã¾ã§ã®ã•ã¾ã–ã¾ãªåˆ†æ•£å‡¦ç†æŠ€è¡“ã‚„Webã‚µãƒ¼ãƒ“ã‚¹ã«ã¤ã„ã¦ç´¹ä»‹ã—ã¾ã—ãŸã€‚ ãã®MapReduceã‚¢ãƒ«ã‚´ãƒªã‚ºãƒ ã‚’Javaã§å®Ÿè£…ã—ãŸã‚‚
dirablue 2010/01/02
hadoop

mapreduce
ãƒªãƒ³ã‚¯
1