[B! hadoop] a_bickyã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

a_bicky id:a_bicky

hadoopã«é–¢ã™ã‚‹a_bickyã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (36)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

Hadoopã®Writableã«é–¢ã™ã‚‹ãƒ¡ãƒ¢ | mwSoft
æ¦‚è¦ Hadoopã®Writableã¯Javaã®Serializeã‚ˆã‚Šã‚‚åŽ³å¯†ã˜ã‚ƒãªã„åˆ†ã€è»½ã„å®Ÿè£…ã«ãªã£ã¦ã‚‹ã€‚ãã®è¾ºã®å‹•ãã‚’å°‘ã—è¿½ã£ã¦ã¿ã‚‹ã€‚ ç”¨æ„ã•ã‚Œã¦ã„ã‚‹ã‚¯ãƒ©ã‚¹ åŸºæœ¬çš„ãªã¨ã“ã‚ã¨ã—ã¦ã€longã‚’æ‰±ã†LongWritableã€doubleã‚’æ‰±ã†DoubleWritableãªã©ã€ãƒ—ãƒªãƒŸãƒ†ã‚£ãƒ–ãªåž‹ã«å¯¾ã—ã¦ã¯ãã‚Œãžã‚Œç”¨æ„ã•ã‚Œã¦ã„ã‚‹ã€‚ ã‚ã¨ã¯æ–‡å—åˆ—ã‚’æ‰±ã†Textã€å€¤ãŒãªã„å ´åˆç”¨ã®NullWritableã€é…åˆ—ã‚’æ ¼ç´ã™ã‚‹ArrayWritableã€Mapã‚’æ ¼ç´ã™ã‚‹MapWritableãªã©ãŒå˜åœ¨ã™ã‚‹ã€‚ Text Textã¯Stringã‚’æ‰±ã†ã€‚ã‚·ãƒªã‚¢ãƒ©ã‚¤ã‚ºã—ã¦å‡ºåŠ›ã•ã‚Œã‚‹å€¤ã¯ã€ãã®æ–‡å—åˆ—ã®ãƒã‚¤ãƒˆé•·ãŒæœ€åˆã«æŒ‡å®šã•ã‚Œã€ãã®å¾Œã«byteé…åˆ—ãŒè¨˜è¿°ã•ã‚Œã‚‹å½¢ã«ãªã‚‹ã€‚ ãŸã¨ãˆã°ã€Œabcã€ã‚’ã‚·ãƒªã‚¢ãƒ©ã‚¤ã‚ºã™ã‚‹ã¨ä»¥ä¸‹ã®ã‚ˆã†ãªbyteé…åˆ—ãŒå‡ºåŠ›ã•ã‚Œã‚‹ã€‚ 3 97 98 99 é•·ã•ãŒ3ã§ã€ã‚ã¨ã¯a(97)ã€b(98)ã€c(99
a_bicky 2021/03/12
hadoop
ãƒªãƒ³ã‚¯
IBM SPSS Modeler - ODBC Configuration Best Practices and Troubleshooting
IBM Developer is your one-stop location for getting hands-on training and learning in-demand skills on relevant techno logies such as generative AI, data science, AI, and open source.
a_bicky 2019/02/25
yarn

hadoop
ãƒªãƒ³ã‚¯
Hiveã‚’å°Žå…¥ãƒ»é«˜é€ŸåŒ–ã—ãŸã„æ™‚ã«æœ€ä½Žé™å¿…è¦ãªçŸ¥è˜ - Qiita
ãŠç–²ã‚Œæ§˜ã§ã™ã€‚ãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿ã¨ã„ã†è¨€è‘‰ãŒæµè¡Œã‚Šã ã—ã¦ã‹ã‚‰å¹¾æ˜Ÿéœœã€çš†ã•ã‚“ã®ä¸ã§ã‚‚ãã‚ãã‚ç¤¾å†…ã«ãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿å‡¦ç†åŸºç›¤ã‚’ä½œã‚ŠãŸã„ã¨ã„ã†æ–¹ãŒã„ã‚‹ã®ã§ã¯ãªã„ã§ã—ã‚‡ã†ã‹ï¼Ÿ ã¨ã„ã†ã‚ã‘ã§SQL on Hadoopã§ã‚ˆãçŸ¥ã‚‰ã‚Œã‚‹Hiveã‚’ä½¿ã†ä¸Šã§æœ€ä½Žé™å¿…è¦ã ãªã¨æ€ã†çŸ¥è˜ã‚’ã¾ã¨ã‚ã¦ã¿ã¾ã—ãŸã€‚ Prestoã‚„AWS Athenaã€Amazon Redshift Spectrumã‚’ä½¿ã†å ´åˆã§ã‚‚è£å´ã«ã¯Hiveã®ãƒ†ãƒ¼ãƒ–ãƒ«ãŒå¿…è¦ãªå ´åˆãŒå¤šã„ã¨æ€ã„ã¾ã™ã®ã§ã€çŸ¥ã£ã¦ãŠã„ã¦æã¯ç„¡ã„ã‹ã¨æ€ã„ã¾ã™ã€‚ ##ãƒ†ãƒ¼ãƒ–ãƒ«ãƒ»ãƒ‡ãƒ¼ã‚¿è¨è¨ˆ ###ãƒ‘ãƒ¼ãƒ†ã‚£ã‚·ãƒ§ãƒ³ã‚’è¨å®šã™ã‚‹ Hiveã§ã¯ãƒ†ãƒ¼ãƒ–ãƒ«ã«ãƒ‘ãƒ¼ãƒ†ã‚£ã‚·ãƒ§ãƒ³ã‚’è¨å®šã™ã‚‹äº‹ã§ãƒ†ãƒ¼ãƒ–ãƒ«ã®æ¤œç´¢ãƒ»æ›´æ–°ç¯„å›²ã‚’é™å®šã™ã‚‹äº‹ãŒã§ãã¾ã™ã€‚é€†ã«è¨€ãˆã°ãƒ‘ãƒ¼ãƒ†ã‚£ã‚·ãƒ§ãƒ³ã‚’è¨å®šã—ãªã„å ´åˆã€å¸¸ã«ãƒ†ãƒ¼ãƒ–ãƒ«å…¨ä½“ã‚’æ¤œç´¢ãƒ»æ›´æ–°ã—ã¦ã—ã¾ã†ã¨ã„ã†äº‹ã§ã™ã€‚ãªã®ã§ãƒ‘ãƒ¼ãƒ†ã‚£ã‚·ãƒ§ãƒ³è¨å®šã¯åŸºæœ¬çš„ã«å¿…é ˆã§ã™ã€‚ ãƒ‘ãƒ¼ãƒ†ã‚£ã‚·ãƒ§ãƒ³ã¯ãƒ‡ãƒ¼ã‚¿ã‚’ã‚¤ãƒ³ãƒãƒ¼ãƒˆã™
a_bicky 2018/10/29
hive

hadoop
ãƒªãƒ³ã‚¯
Hiveã®ãƒ‘ãƒ•ã‚©ãƒ¼ãƒžãƒ³ã‚¹ãƒãƒ¥ãƒ¼ãƒ‹ãƒ³ã‚°ã§è©¦ã—ãŸï¼—ã¤ã®ã“ã¨ - Qiita
Spark, SQL on Hadoop etc. Advent Calendar 2014 - Qiita 10æ—¥ç›®ã®è¨˜äº‹ã§ã™ã€‚ ã¨ã‚ã‚‹ãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆã«ã¦ã€ãƒ‘ãƒ•ã‚©ãƒ¼ãƒžãƒ³ã‚¹ãƒãƒ¥ãƒ¼ãƒ‹ãƒ³ã‚°ã®ãŸã‚ã«å®Ÿæ–½ã—ãŸï¼—ã¤ã®ã“ã¨ã‚’ã¾ã¨ã‚ã¾ã—ãŸã€‚ ã“ã®å†…å®¹ã¯Cloudera World Tokyo 2014ã§ãŠè©±ã—ã•ã›ã¦ã„ãŸã ã„ãŸå†…å®¹ã‚’å†ç·¨ã—ãŸã‚‚ã®ã§ã™ã€‚ ç™»å£‡è³‡æ–™ - Hadoopã§ä½œã‚‹åºƒå‘Šåˆ†æžãƒ—ãƒ©ãƒƒãƒˆãƒ•ã‚©ãƒ¼ãƒ ç™»å£‡ã®æ§˜å - å›½å†…æœ€å¤§ç´šã®Hadoopé–¢é€£ã‚«ãƒ³ãƒ•ã‚¡ãƒ¬ãƒ³ã‚¹ã«ç™»å£‡ã—ã¦ãã¾ã—ãŸï¼ 1.YARNãŒåˆ©ç”¨å¯èƒ½ãªãƒªã‚½ãƒ¼ã‚¹ã®å¤‰æ›´ YARNã§ã¯MR1ã¨ç•°ãªã‚Šã‚¹ãƒãƒƒãƒˆã§ã¯ãªãã‚³ãƒ³ãƒ†ãƒŠã¨ã„ã†æ¦‚å¿µã§ãƒªã‚½ãƒ¼ã‚¹ãŒç®¡ç†ã•ã‚Œã¾ã™ã€‚ ä»¥ä¸‹ã®ãƒ‘ãƒ©ãƒ¡ãƒ¼ã‚¿ã§ãƒŽãƒ¼ãƒ‰ãƒžãƒãƒ¼ã‚¸ãƒ£ãŒã‚³ãƒ³ãƒ†ãƒŠã«åˆ©ç”¨å¯èƒ½ãªãƒ¡ãƒ¢ãƒªé‡ã€CPUæ•°ã‚’å¤‰æ›´ã—ã¾ã—ãŸã€‚ yarn.nodemanager.resource.memory-mb yarn.nodemanager.
a_bicky 2018/10/29
hive

hadoop
ãƒªãƒ³ã‚¯
Hive on Tez Performance Tuning - Determining Reducer Counts
How Does Tez determine the number of reducers? How can I control this for performance? In this article, I will attempt to answer this while executing and tuning an actual query to illustrate the concepts. Then I will provide a summary with a full explanation. if you wish, you can advance ahead to the summary. ------------- 0. Prep Work and Checklist We followed the Tez Memory Tuning steps as outli
a_bicky 2018/10/29
hadoop

hive
ãƒªãƒ³ã‚¯
HIVE Dynamic Partitioning tips â€“ Qubole Support Center
a_bicky 2018/10/29
hadoop

hive
ãƒªãƒ³ã‚¯
Cloudera | ãƒã‚¤ãƒ–ãƒªãƒƒãƒ‰ãƒ‡ãƒ¼ã‚¿ã‚«ãƒ³ãƒ‘ãƒ‹ãƒ¼
ãƒ‡ãƒ¼ã‚¿ã‚’ä¿¡é ¼ã—ã€AI ã‚’ä¿¡é ¼ã™ã‚‹ ä¿¡é ¼ã§ãã‚‹ãƒ‡ãƒ¼ã‚¿ã€ä¿¡é ¼ã§ãã‚‹ãƒ¢ãƒ‡ãƒ«ã€ä¿¡é ¼ã§ãã‚‹ AI ã‚’å®Ÿç¾ã™ã‚‹ãŸã‚ã«ã€ã“ã‚Œã»ã©å¤šãã®ã‚¯ãƒ©ã‚¦ãƒ‰ã®ã•ã¾ã–ã¾ãªãƒ‡ãƒ¼ã‚¿ã‚¿ã‚¤ãƒ—ã‚’ç®¡ç†ã§ãã€ã‚ªãƒ¼ãƒ—ãƒ³ãƒ‡ãƒ¼ã‚¿ã®ã‚¤ãƒŽãƒ™ãƒ¼ã‚·ãƒ§ãƒ³ã¨å¤§è¦æ¨¡å±•é–‹ã«å¯¾å¿œã§ãã‚‹ãƒ—ãƒ©ãƒƒãƒˆãƒ•ã‚©ãƒ¼ãƒ ã¯ä»–ã«ã‚ã‚Šã¾ã›ã‚“ã€‚
a_bicky 2013/05/13
MR2 ã¯ YARN ä¸Šã§å‹•ä½œã™ã‚‹ MapReduce ãƒ•ãƒ¬ãƒ¼ãƒ ãƒ¯ãƒ¼ã‚¯ã§ã‚ã‚Šä¸¡è€…ã¯ç•°ãªã‚‹ã‚‚ã®ã¨ãª

hadoop

yarn
ãƒªãƒ³ã‚¯
ï¼»4ï¼½æ©Ÿæ¢°å¦ç¿’ã«ãŠã‘ã‚‹Apache Mahoutã®å¯èƒ½æ€§ã¨èª²é¡Œ
ç¬¬2å›žã€ç¬¬3å›žã§ã¯ã€éŸ³å£°ãƒ‡ãƒ¼ã‚¿ã¨ã€ç”»åƒãƒ‡ãƒ¼ã‚¿ã«é–¢ã™ã‚‹ãƒ¡ãƒ‡ã‚£ã‚¢èªè˜æŠ€è¡“ã«ã¤ã„ã¦è§£èª¬ã—ã¦ãã¾ã—ãŸã€‚ä»Šå›žã¯ã€ãƒ¡ãƒ‡ã‚£ã‚¢èªè˜æŠ€è¡“ã«å…±é€šã™ã‚‹èª²é¡Œã§ã‚ã‚‹ã€æ©Ÿæ¢°å¦ç¿’ã®é«˜é€ŸåŒ–ã«ã¤ã„ã¦å–ã‚Šä¸Šã’ã¾ã™ã€‚ ã¾ãšã€æ©Ÿæ¢°å¦ç¿’ã¨ã¯ä½•ã‹ã¨ã„ã†ã“ã¨ã‹ã‚‰ç°¡å˜ã«èª¬æ˜Žã™ã‚‹ã¨ã€ã‚ã‚‹ç¨‹åº¦å¤§é‡ã®ãƒ‡ãƒ¼ã‚¿ã®ä¸ã‹ã‚‰è¦å‰‡æ€§ã‚„ãƒ‘ã‚¿ãƒ¼ãƒ³ã‚’è¦‹ã„ã ã™ã“ã¨ã§ã€ãƒ‡ãƒ¼ã‚¿ã®èªè˜ã‚„åˆ¤å®šãªã©ã«å½¹ç«‹ã¦ã‚‹å–ã‚Šçµ„ã¿ã®ã“ã¨ã‚’æŒ‡ã—ã¾ã™ã€‚ä¾‹ãˆã°ã€çš†ã•ã‚“ãŒä½¿ç”¨ã—ã¦ã„ã‚‹ãƒ‘ã‚½ã‚³ãƒ³ã‚„æƒ…å ±ã‚·ã‚¹ãƒ†ãƒ ã®å¤šãã«ã¯ã€ãƒ¡ãƒ¼ãƒ«ã‚’å—ä¿¡ã—ãŸã¨ãã«ã€ã‚¦ã‚¤ãƒ«ã‚¹ãŒå«ã¾ã‚Œã¦ã„ãªã„ã‹ã©ã†ã‹ã€ã‚¹ãƒ‘ãƒ ãƒ¡ãƒ¼ãƒ«ã§ãªã„ã‹ã©ã†ã‹ã€åˆ¤å®šã™ã‚‹ã‚½ãƒ•ãƒˆã‚¦ã‚¨ã‚¢ãŒå…¥ã£ã¦ã„ã‚‹ã¨æ€ã„ã¾ã™ã€‚ã“ã‚Œã‚‚ã€éŽåŽ»ã®å¤§é‡ãƒ‡ãƒ¼ã‚¿ã«åŸºã¥ã„ã¦ã€ã‚¦ã‚¤ãƒ«ã‚¹ãŒå…¥ã£ã¦ã„ãŸã‚Šã€ã‚¹ãƒ‘ãƒ ãƒ¡ãƒ¼ãƒ«ã ã£ãŸã‚Šã™ã‚‹å¯èƒ½æ€§ãŒé«˜ã„ãƒ‘ã‚¿ãƒ¼ãƒ³ã‚’æ©Ÿæ¢°å¦ç¿’ã™ã‚‹ã“ã¨ã§åˆ¤å®šã®ä»•çµ„ã¿ã‚’ä½œã‚Šã€ãã®ãƒã‚¸ãƒƒã‚¯ã‚’æ´»ç”¨ã—ã¦ã„ã‚‹ã®ã§ã™ã€‚ ãã—ã¦ãƒ¡ãƒ‡ã‚£ã‚¢å‡¦ç†ã‚·ã‚¹ãƒ†ãƒ ã«ã¯ã€éŸ³å£°ã€ç”»åƒã€ãƒ†ã‚ã‚¹ãƒˆãªã©ãƒ‡ãƒ¼ã‚¿ã®ç¨®é¡žã‚’å•ã‚
a_bicky 2012/11/01
mahout

hadoop

machine learning
ãƒªãƒ³ã‚¯
Hadoopã§ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ã‚·ã‚¹ãƒ†ãƒ ã‚’ä½œã‚ã† è¨˜äº‹ä¸€è¦§ | gihyo.jp
é‹å–¶å…ƒã®ãƒã‚´ Copyright Â© 2007-2024 All Rights Reserved by Gijutsu-Hyoron Co., Ltd. ãƒšãƒ¼ã‚¸å†…å®¹ã®å…¨éƒ¨ã‚ã‚‹ã„ã¯ä¸€éƒ¨ã‚’ç„¡æ–ã§åˆ©ç”¨ã™ã‚‹ã“ã¨ã‚’ç¦æ¢ã—ã¾ã™â ã€‚å€‹åˆ¥ã«ãƒ©ã‚¤ã‚»ãƒ³ã‚¹ãŒè¨å®šã•ã‚Œã¦ã„ã‚‹è¨˜äº‹ç‰ã¯ãã®ãƒ©ã‚¤ã‚»ãƒ³ã‚¹ã«å¾“ã„ã¾ã™ã€‚
a_bicky 2012/02/13
ä½•ã“ã‚ŒãŠã‚‚ã—ã‚ãã†ï¼

hadoop

recommend
ãƒªãƒ³ã‚¯
1å°æ§‹æˆã®Hadoopã‚’30åˆ†ã§è©¦ã—ã¦ã¿ã‚‹(Ubuntu + Cloudera)
(å‚è€ƒ) Clouderaç¤¾ã®Hadoopãƒ‘ãƒƒã‚±ãƒ¼ã‚¸ã®æƒ…å ± http://archive.cloudera.com/docs/ å¿…è¦ãªã‚‚ã® ãƒ»Ubuntuã‚„debianã®Linuxç’°å¢ƒ1å°(æ‰‹å…ƒã§ã¯Ubuntu Server 11.04/10.04/9.10/8.04, debian 5ã‚ãŸã‚Šã§è©¦ã—ã¦ã„ã¾ã™ãŒã€ä»–ãƒãƒ¼ã‚¸ãƒ§ãƒ³ã§ã‚‚å¤§ä¸ˆå¤«ã ã¨æ€ã„ã¾ã™) ãƒ»ã‚¤ãƒ³ã‚¿ãƒ¼ãƒãƒƒãƒˆæŽ¥ç¶š ãƒ»Sun(Oracle)ã®Javaãƒ‘ãƒƒã‚±ãƒ¼ã‚¸(aptã§ã‚¤ãƒ³ã‚¿ãƒ¼ãƒãƒƒãƒˆã‹ã‚‰ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«) ãƒ»Clouderaç¤¾ã®CDH3ã®Hadoopãƒ‘ãƒƒã‚±ãƒ¼ã‚¸(aptã§ã‚¤ãƒ³ã‚¿ãƒ¼ãƒãƒƒãƒˆã‹ã‚‰ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«) ä½œæ¥æ‰‹é † 1. ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«: Linuxç’°å¢ƒã«ã¦ã€rootã§ä½œæ¥ã—ã¾ã™ã€‚ sudo su 1-1. Sun(Oracle)ã®Javaã‚’å…¥ã‚Œã¾ã™ã€‚(Sun(Oracle)ã®ã‚‚ã®ãŒå¿…è¦ã§ã™ã€‚) â€» ã“ã“ã§ã€ubuntu 10ã‚„11ã®äººã¯/etc
a_bicky 2012/02/01
hadoop

ubuntu
ãƒªãƒ³ã‚¯
Hiveã‚¯ã‚¨ãƒªãƒãƒ¥ãƒ¼ãƒ‹ãƒ³ã‚°ãƒ¡ãƒ¢ - ç§‘å¦ã¨éžç§‘å¦ã®è¿·å®®
Hadoopæœ¬2ç‰ˆã‚’è²·ã£ãŸãã®ä¸€é€±é–“å¾Œã«Deals of the day ã§åŠé¡ã‚»ãƒ¼ãƒ«ã‚„ã‚‰ã‚Œã¦æ»ã«ãŸããªã£ãŸã®ã§è…¹ã„ã›ã«æ›¸ã ã¯ã˜ã‚ã« ã“ã“ã«æ›¸ã„ã¦ã‚ã‚‹ã®ã¯å…¨éƒ¨å‚è€ƒãƒªãƒ³ã‚¯ãƒ»æ–‡çŒ®ã‹ã‚‰ã²ã£ã±ã£ã¦ããŸã ã‘ã§ã€ã»ã¨ã‚“ã©å…¨éƒ¨æ¤œè¨¼ã—ã¦ãªã„ã€‚é–“é•ã„ãŒã‚ã‚Œã°ãªã‚‹ã¹ãæ—©ã‚ã«æ›´æ–°ã™ã‚‹ã‚ˆã†åŠªåŠ›ã¯ã™ã‚‹ãŒã€éµœå‘‘ã¿ã«ã—ã¦ä½•ãŒèµ·ãã¦ã‚‚è‡ªå·±è²¬ä»»ã§ã€‚ Hive ã®ã‚¯ã‚¨ãƒªãƒãƒ¥ãƒ¼ãƒ‹ãƒ³ã‚°ã«é–¢ã™ã‚‹ãƒ¡ãƒ¢æ›¸ãã§ã‚ã‚‹ã€‚ä»¥ä¸‹ã®ã“ã¨ã¯ã€Œæ›¸ã„ã¦ã„ãªã„ã€ Hadoopè‡ªä½“ã®ãƒãƒ¥ãƒ¼ãƒ‹ãƒ³ã‚° Hive ã®ã‚¯ã‚¨ãƒªãƒãƒ¥ãƒ¼ãƒ‹ãƒ³ã‚°ä»¥å¤–ã®è©± ä¾‹ãˆã°ã€åœ§ç¸®ãƒ•ã‚¡ã‚¤ãƒ«ã‚’ Hive ä¸Šã§æ‰±ã†ã«ã¯ã©ã†ã™ã‚‹ã‹ã¨ã‹ JOIN ä¸€ç•ªå·¦ã®ãƒ†ãƒ¼ãƒ–ãƒ«ã«æœ€ã‚‚å¤§ããªãƒ†ãƒ¼ãƒ–ãƒ«ã‚’æŒã£ã¦ãã‚‹ ä¸€ç•ªå·¦ã®ãƒ†ãƒ¼ãƒ–ãƒ«ãŒMRã§ã„ã†å…¥åŠ›ãƒ‡ãƒ¼ã‚¿ã¨ã—ã¦æµã‚Œã‚‹ã€‚ã‚¤ãƒ³ãƒŠãƒ¼ãƒ†ãƒ¼ãƒ–ãƒ«ã®ãƒ‡ãƒ¼ã‚¿ã¯ãƒ¡ãƒ¢ãƒªã«ä¿æŒã•ã‚Œã‚‹ã€‚ åŒä¸€ JOIN ã‚ãƒ¼ é€šå¸¸ã¯ 1 JOIN = 1 MR ã‚¸ãƒ§ãƒ–ã ãŒã€åŒä¸€ã® JOIN ã‚ãƒ¼ã‚’ä½¿ã£ã¦ã„
a_bicky 2012/01/30
hadoop

hive
ãƒªãƒ³ã‚¯
ã“ã®ãƒšãƒ¼ã‚¸ã‚’è¦‹ã‚‹ã«ã¯ã€ãƒã‚°ã‚¤ãƒ³ã¾ãŸã¯ç™»éŒ²ã—ã¦ãã ã•ã„
Facebookã§æŠ•ç¨¿ã‚„å†™çœŸãªã©ã‚’ãƒã‚§ãƒƒã‚¯ã§ãã¾ã™ã€‚
a_bicky 2011/09/12
AvatarNodeã«ã¤ã„ã¦ã¨ã‹

hadoop
ãƒªãƒ³ã‚¯
Hadoopã®å¯ç”¨æ€§ã«ã¤ã„ã¦ï¼ˆç§è¨³ï¼‰ - developerâ€™s delight
ã“ã®è¨˜äº‹ã¯Clouderaã®Blogã®è¨˜äº‹â€Hadoop Availability | Apache Hadoop for the Enterprise | Clouderaâ€ã®ç§çš„ç¿»è¨³ã§ã™ã€‚Hadoopã®å¯ç”¨æ€§ã«ã¤ã„ã¦ã¯èˆˆå‘³ã®ã‚ã‚‹æ–¹ã‚‚å¤šã„ã¨æ€ã„ã¾ã™ã®ã§ã€èªã‚€ã¤ã„ã§ã«è¨³ã—ã¦ã¿ã¾ã—ãŸã€‚å‹¢ã„ã§è¨³ã—ãŸã®ã§è³ªã«é–¢ã—ã¦ã¯è²¬ä»»ã‚’æŒã¦ã¾ã›ã‚“ã®ã§ã‚ˆã‚ã—ããŠé¡˜ã„ã—ã¾ã™ã€‚é–“é•ã„ãªã©ãŒã‚ã‚Šã¾ã—ãŸã‚‰ã”æŒ‡æ‘˜ã„ãŸã ã‘ã‚Œã°åŠ©ã‹ã‚Šã¾ã™ã€‚(id:kkawamura)Apache Hadoopã®ãƒ¡ãƒ¼ãƒªãƒ³ã‚°ãƒªã‚¹ãƒˆã§ã‚ˆãã‚ã‚‹è³ªå•ã¯ã€å¯ç”¨æ€§ã‚’ä¿ã¤ãŸã‚ã«ã©ã†ã™ã‚‹ã‹ï¼Ÿã¨ã„ã†ã‚‚ã®ã§ã™ã€‚ã“ã®è¨˜äº‹ã§ã¯ã€Hadoopã®ã‚³ãƒ³ãƒ†ã‚ã‚¹ãƒˆã§ã®å¯ç”¨æ€§ã«ã¤ã„ã¦è¦‹ã¦ã„ãã€é€²è¡Œä¸ã®é–‹ç™ºã®æ–¹å‘æ€§ã‚’ç¤ºã—ã¾ã™ã€‚èƒŒæ™¯Hadoopã®å¯ç”¨æ€§ã‚’è°è«–ã™ã‚‹ã¨ãã€äººã¯ã‚ˆãNameNodeãŒHDFSã«ãŠã„ã¦å˜ä¸€æ•…éšœç‚¹ã§ã‚ã‚‹ãŸã‚ã€NameNodeã‹ã‚‰è©±ã‚’ã¯ã˜ã‚ã¾ã™ã€‚ãã—ã¦Hadoo
a_bicky 2011/09/05
hadoop
ãƒªãƒ³ã‚¯
Hadoop AvatarNode High Availability
Our Use-Case The Hadoop Distributed File System's (HDFS) NameNode is a single point of falure. This has been a major stumbling block in using HDFS for a 24x7 type of deployment. It has been a topic of discussion among a wide circle of engineers. I am part of a team that is operating a cluster of 1200 nodes and a total size of 12 PB. This cluster is currently running hadoop 0.20. The NameNode is co
a_bicky 2011/09/05
AvatarNodeã¯NameNodeã‚’å†…åŒ…ã—ã¦ã„ã‚‹ã‹ã‚‰åˆ‡ã‚Šæ›¿ãˆãŒæ—©ã„ã‘ã©ã€DRBDã¯ãƒ¡ã‚¿ãƒ‡ãƒ¼ã‚¿ã®ã‚³ãƒ”ãƒ¼ã‚’æŒã£ã¦ã„ã‚‹ã ã‘ã ã‹ã‚‰éšœå®³ãŒèµ·ãã¦ã‹ã‚‰NameNodeã¨ã—ã¦ç¨¼å‹•ã•ã›ã‚‹ã¾ã§ã«æ™‚é–“ãŒã‹ã‹ã‚‹ã£ã¦ã“ã¨ï¼Ÿ

hadoop

avatarnode
ãƒªãƒ³ã‚¯
ä¼æ¥ã§ä½¿ã‚ã‚Œã‚‹Hadoop
Copyright Â© 2004-2024 Impress Corporation. An Impress Group Company. All rights reserved.
a_bicky 2011/09/05
hadoop
ãƒªãƒ³ã‚¯
ã‚¯ãƒƒã‚¯ãƒ‘ãƒƒãƒ‰ã¨Hadoop - ã‚¯ãƒƒã‚¯ãƒ‘ãƒƒãƒ‰é–‹ç™ºè€…ãƒ–ãƒã‚°
ã¯ã˜ã‚ã¾ã—ã¦ã€‚ä»Šå¹´ã®5æœˆã«å…¥ç¤¾ã—ãŸå‹é–“@ã•ãŒã™ãƒãƒ¼ãƒ ã§ã™ã€‚ å…¥ç¤¾ã—ã¦ã‹ã‚‰ã¯ã€ãªã‹ãªã‹å¤§å¤‰ãªã“ã¨ã‚‚å¤šã„ã§ã™ãŒã€æœ€è¿‘ã¯ãŠé…’å¥½ããŒé›†ã¾ã£ã¦æœˆæ›œã‹ã‚‰é£²ã¿åˆã† ã€Œå‹é–“ä¼šã€ãªã‚‹ã‚‚ã®ã‚‚ç™ºè¶³ã—ã¦ã€ä»•äº‹é¢ã§ã‚‚ä»•äº‹ä»¥å¤–ã®é¢ã§ã‚‚å¯†åº¦ã®é«˜ã„æ¯Žæ—¥ã‚’éŽã”ã—ã¦ã„ã¾ã™ï¼ ã•ã¦ã€åƒ•ã¯ã€Œã•ãŒã™ã€ãƒãƒ¼ãƒ æ‰€å±žã¨ã„ã†ã“ã¨ã§ã€æ™®æ®µã¯ãƒ¬ã‚·ãƒ”ã‚’ã€Œã•ãŒã™ã€ãƒ¦ãƒ¼ã‚¶ã®æº€è¶³åº¦ã‚’ä¸Šã’ã‚‹ãŸã‚ã«ã€ ã‚¯ãƒƒã‚¯ãƒ‘ãƒƒãƒ‰ã®æ¤œç´¢ã¾ã‚ã‚Šã«ã¤ã„ã¦ã€ã„ã‚ã„ã‚ãªé–‹ç™ºã‚’è¡Œã£ã¦ã„ã¾ã™ã€‚ ä¸€æ–¹ã§ã€ãƒ¦ãƒ¼ã‚¶ã®ã€Œã•ãŒã™æ¬²æ±‚ã€ã«ã¤ã„ã¦æ·±ãçŸ¥ã‚‹ãŸã‚ã«ã€å¤§è¦æ¨¡ãªãƒ‡ãƒ¼ã‚¿è§£æžã‚’è¡Œã„ã€æ¬²æ±‚ã®åˆ†æžã‚’è¡Œã†æ©Ÿä¼šã‚‚å¢—ãˆã¦ãã¾ã—ãŸã€‚ ã¨ã“ã‚ãŒã€ã‚¯ãƒƒã‚¯ãƒ‘ãƒƒãƒ‰ã®ãƒã‚°ã¯è†¨å¤§ãªæ•°ãŒã‚ã‚‹ã®ã§ã€ä¸€å£ã®ãƒ‡ãƒ¼ã‚¿è§£æžã¨è¨€ã£ã¦ã‚‚é€šå¸¸ã®ãƒãƒƒãƒå‡¦ç†ã ã¨é–“ã«åˆã‚ãªã„ãŸã‚ã€ åˆ†æ•£å‡¦ç†ç’°å¢ƒã®å¿…è¦æ€§ãŒé«˜ã¾ã£ã¦ãã¾ã—ãŸã€‚ ãã“ã§ã€ã¾ãšã¯æ‰‹è»½ã«è©¦ã›ã‚‹åˆ†æ•£å‡¦ç†ã®çŽ‹é“ã¨ã„ã†ã“ã¨ã§ã€æœ€è¿‘ã§ã¯Hadoopã‚’ä½¿ã£ãŸãƒ‡ãƒ¼ã‚¿è§£æžç’°å¢ƒã‚’æ•´å‚™ã—ã¦ã„ã¾ã™ã€‚
a_bicky 2011/09/05
hadoop
ãƒªãƒ³ã‚¯
HDFSã¯ãƒ•ã‚¡ã‚¤ãƒ«ã‚µãƒ¼ãƒãƒ¼ã«ä½¿ã†ã‚‚ã®ã˜ã‚ƒãªã„ã‚ˆ
Hadoopï¼ˆ0.22ä»¥å‰ã‚„1.0.0ï¼‰ã®HDFSã¯NameNodeãŒSPOFï¼ˆå˜ä¸€éšœå®³ç‚¹ï¼‰ãªã®ã§ã€ãƒ•ã‚¡ã‚¤ãƒ«ã‚µãƒ¼ãƒãƒ¼ã‚„ãƒ‡ãƒ¼ã‚¿ã®â€œãƒãƒƒã‚¯ã‚¢ãƒƒãƒ—â€ã«ã¯å‘ã‹ãªã„ã¨ã„ã†è©±ã§ã™ã€‚ HDFSä¸Šã«æ§‹ç¯‰ã•ã‚Œã¦ã„ã‚‹HBaseã®å ´åˆã€ï¼ˆHBaseè‡ªèº«ã«ã¯SPOFã¯ç„¡ã„ã‘ã‚Œã©ã‚‚ï¼‰HDFSã«SPOFãŒã‚ã‚‹ã®ã§ã€NameNodeã¯å†—é•·åŒ–ã—ãŸæ–¹ãŒè‰¯ã„ã‚ˆã†ã§ã™ã€‚ å†—é•·åŒ–ã«ã¯ä»¥ä¸‹ã®ã‚ˆã†ãªä»•çµ„ã¿ãŒè€ƒãˆã‚‰ã‚Œã¦ã„ã‚‹ã‚ˆã†ã§ã™ã€‚ ç¶šãã‚’èªã‚€
a_bicky 2011/09/05
hadoop

hdfs
ãƒªãƒ³ã‚¯
MountableHDFS - HADOOP2 - Apache Software Foundation
[machine1] ~ > df -kh /export/hdfs/ Filesystem Size Used Avail Use% Mounted on fuse 4.1P 642T 3.5P 21% /export/hdfs [machine1] ~ > ls /export/hdfs/ home tmp Trash user usr var These projects (enumerated below) allow HDFS to be mounted (on most flavors of Unix) as a standard file system using the mount command. Once mounted, the user can operate on an instance of hdfs using standard Unix utilities
a_bicky 2011/08/09
hadoop

hdfs
ãƒªãƒ³ã‚¯
Hadoop Streaming ã§å¤–éƒ¨ãƒ•ã‚¡ã‚¤ãƒ«ã‚’æ‰±ã†æ–¹æ³•ã®ã¾ã¨ã‚ - (ï¾Ÿâˆ€ï¾Ÿ)oå½¡ sasata299's blog
2010å¹´05æœˆ06æ—¥09:21 Hadoop Hadoop Streaming ã§å¤–éƒ¨ãƒ•ã‚¡ã‚¤ãƒ«ã‚’æ‰±ã†æ–¹æ³•ã®ã¾ã¨ã‚ Hadoop Streaming ã§ã®å¤–éƒ¨ãƒ•ã‚¡ã‚¤ãƒ«ã®æ‰±ã„ã‚‚ã ã„ã¶å›ºã¾ã£ã¦ãã¾ã—ãŸã€‚ç™ºè¡¨è³‡æ–™ã®ã‚¹ãƒ©ã‚¤ãƒ‰ã§ã¯ã“ã®è¾ºã®è©±ã‚’æ›¸ã„ãŸã“ã¨ãŒã‚ã‚‹ã‚“ã§ã™ãŒã€ãƒ–ãƒã‚°ã«ã¯æ›¸ã„ã¦ãªã‹ã£ãŸã®ã§ä¸€åº¦ãã£ã¡ã‚Šã¾ã¨ã‚ã¦ãŠã“ã†ã‹ãªãƒ¼ã£ã¨ã€‚ã¨ã„ã†ã‚ã‘ã§ä»Šå›žã¯ Hadoop Streaming ã§ã®å¤–éƒ¨ãƒ•ã‚¡ã‚¤ãƒ«ã®èªã¿è¾¼ã¿ã«ã¤ã„ã¦ã¾ã¨ã‚ã¾ã™ã‚ˆï¼(*ï¾ŸÐ”ï¾Ÿ)=3 ï¾‘ï¾Šï½° ãã‚‚ãã‚‚å¤–éƒ¨ãƒ•ã‚¡ã‚¤ãƒ«ã®èªã¿è¾¼ã¿ã¨è¨€ã£ã¦ã‚‚ã€ã“ã®äºŒã¤ã®ãƒ‘ã‚¿ãƒ¼ãƒ³ãŒã‚ã‚Šã¾ã™ã€‚ 1) å¤–éƒ¨ãƒ•ã‚¡ã‚¤ãƒ«ãŒ master ä¸Šã«ã‚ã‚‹ï¼ˆã¤ã¾ã‚Šãƒãƒ¼ã‚«ãƒ«ãƒ‡ã‚£ã‚¹ã‚¯ã«ã‚ã‚‹ï¼‰å ´åˆ 2) å¤–éƒ¨ãƒ•ã‚¡ã‚¤ãƒ«ãŒåˆ¥ãƒ•ã‚¡ã‚¤ãƒ«ã‚·ã‚¹ãƒ†ãƒ ï¼ˆS3 ã ã£ãŸã‚Š HDFS ã ã£ãŸã‚Šï¼‰ã«ã‚ã‚‹å ´åˆ å¤–éƒ¨ãƒ•ã‚¡ã‚¤ãƒ«ãŒ master ä¸Šã«ã‚ã‚‹å ´åˆ ã¾ãšã€ã“ã®å ´åˆã¯ã¨ã¦ã‚‚ç°¡å˜ã§ã™ã€‚ä¾‹ãˆã° hoge.txt ã¨
a_bicky 2011/08/06
hadoop

streaming
ãƒªãƒ³ã‚¯
Hadoopã‚’ä½¿ã„ã“ãªã™(1)
ã¾ãšã€ 1 ã®å…¥åŠ›ãƒ•ã‚¡ã‚¤ãƒ«ã‚’åˆ†å‰²ã™ã‚‹æ–¹æ³•ã¯ã€InputFormatã‚¯ãƒ©ã‚¹ã®ã€getSplitsé–¢æ•°ã‚’ä¸Šæ›¸ãã™ã‚‹ã“ã¨ã§ã€ã‚«ã‚¹ã‚¿ãƒžã‚¤ã‚ºã§ãã¾ã™ã€‚ ã¾ãŸã€ 3 ã®InputSplitã‹ã‚‰ã€Keyã¨Valueã‚’æŠ½å‡ºã™ã‚‹å‡¦ç†ã‚‚ã€InputFormatã‚¯ãƒ©ã‚¹ã‚’é€šã˜ã¦ã‚«ã‚¹ã‚¿ãƒžã‚¤ã‚ºã§ãã¾ã™ã€‚ InputFormatã®getRecordReaderé–¢æ•°ã‚’é€šã˜ã¦ã€RecordReaderã‚¯ãƒ©ã‚¹ã‚’ç”Ÿæˆã™ã‚‹ã®ã§ã™ãŒã€ã“ã‚Œã«ä»»æ„ã®RecordReaderã‚¯ãƒ©ã‚¹ã‚’æŒ‡å®šã™ã‚Œã°OKã§ã™ã€‚ 2 ã®Mapå‡¦ç†ã§ã™ãŒã€ãƒ¦ãƒ¼ã‚¶ãŒæŒ‡å®šã—ãŸMapperã‚¯ãƒ©ã‚¹ã®å‡¦ç†ã‚’å®Ÿè¡Œã—ã¾ã™ã€‚ Mapperã‚¯ãƒ©ã‚¹ã¯ã€MapRunnerã‚¯ãƒ©ã‚¹ã‚’é€šã˜ã¦ã€åˆæœŸåŒ–å‡¦ç†ã€mapé–¢æ•°ã‚’ç¹°ã‚Šè¿”ã™éŽç¨‹ã€çµ‚äº†å‡¦ç†ã¨ã„ã£ãŸä¸€é€£ã®æµã‚Œã‚’å®Ÿè¡Œã—ã¾ã™ã€‚ MapRunnerã‚¯ãƒ©ã‚¹ã‚’ã‚«ã‚¹ã‚¿ãƒžã‚¤ã‚ºã™ã‚Œã°ã€ã“ã†ã—ãŸæµã‚Œã‚’åˆ¶å¾¡ã™ã‚‹ã“ã¨ãŒã§ãã¾ã™ã€‚ 0.20.0ã‹ã‚‰ã®æ–°ã—ã„MapRed
a_bicky 2011/07/30
hadoop
ãƒªãƒ³ã‚¯
1 2 æ¬¡ã®ãƒšãƒ¼ã‚¸