[B! Hadoop] otani0083ã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

otani0083 id:otani0083

Hadoopã«é–¢ã™ã‚‹otani0083ã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (6)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

ç¬¬1å›žã€€ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ã‚·ã‚¹ãƒ†ãƒ ã¨é›†åˆçŸ¥ | gihyo.jp
ã¯ã˜ã‚ã« ä»Šå›žã‹ã‚‰9å›žã«æ¸¡ã‚Šã€Hadoopã‚’ä½¿ã£ãŸãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ã‚·ã‚¹ãƒ†ãƒ ã®å®Ÿè£…ã«ã¤ã„ã¦ç´¹ä»‹ã•ã›ã¦ã„ãŸã ãã“ã¨ã«ãªã‚Šã¾ã—ãŸã€‚ ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ã‚·ã‚¹ãƒ†ãƒ ã‚’æ§‹ç¯‰ã—ãŸæ–¹ã¯å°‘ãªã„ã¨æ€ã„ã¾ã™ãŒã€ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ã®ã‚µãƒ¼ãƒ“ã‚¹ã«è§¦ã‚Œã¦ã„ã‚‹æ–¹ã¯å¤šã„ã¨æ€ã„ã¾ã™ã€‚ä»Šå›žã®é€£è¼‰ã§ã€èªè€…ã®çš†æ§˜ã«ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ã‚·ã‚¹ãƒ†ãƒ ã®å¯èƒ½æ€§ã¨ãã®å®Ÿè£…ã®é¢ç™½ã•ã‚’ãŠä¼ãˆã§ãã‚Œã°ã¨æ€ã„ã¾ã™ã€‚ã‚ˆã‚ã—ããŠé¡˜ã„ç”³ã—ä¸Šã’ã¾ã™ã€‚ é€£è¼‰ã®äºˆå®šã¯æ¬¡ã®é€šã‚Šã§ã™ã€‚ ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ã‚·ã‚¹ãƒ†ãƒ ã¨é›†åˆçŸ¥ï¼ˆä»Šå›žï¼‰ ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ã‚·ã‚¹ãƒ†ãƒ ã®å®Ÿè£…ã¨èª²é¡Œ å”èª¿ãƒ•ã‚£ãƒ«ã‚¿ãƒªãƒ³ã‚°ï¼ˆå‰ãƒ»å¾Œç·¨ï¼‰ ã‚³ãƒ³ãƒ†ãƒ³ãƒ„ãƒ™ãƒ¼ã‚¹ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ï¼ˆå‰ãƒ»å¾Œç·¨ï¼‰ ä»Šå›žã®è¨˜äº‹ã®ãƒã‚¤ãƒ³ãƒˆã¯ä»¥ä¸‹ã®é€šã‚Šã§ã™ã€‚ ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ã‚·ã‚¹ãƒ†ãƒ ã®ç›®çš„ã¯æ°—ä»˜ãã¨é©šãã‚’ä¸Žãˆã‚‹ã“ã¨ ç†æƒ³ã®ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ã¯ã‚½ãƒ ãƒªã‚¨ã®ãŠè–¦ã‚ ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ã‚·ã‚¹ãƒ†ãƒ ã«å¿…è¦ãªã®ã¯å—œå¥½ã¨å°‚é–€æ€§ ã§ã¯ã€æ—©é€Ÿã¯ã˜ã‚ã¾ã—ã‚‡ã†ã€‚ ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ã‚·ã‚¹ãƒ†ãƒ ã¨ã¯ï¼Ÿ ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ã‚·ã‚¹ãƒ†ãƒ ã¯æƒ…å ±ãƒ•ã‚£ãƒ«ã‚¿ãƒªãƒ³ã‚°ã®ä¸€ç¨®ã§ã€å¤§é‡ã®
otani0083 2011/11/01
Hadoop

recommend
ãƒªãƒ³ã‚¯
Hadoop ç¬¬2ç‰ˆ
ã€Œè±¡æœ¬ã€ã®åå‰ã§è¦ªã—ã¾ã‚Œã‚‹ã€ŽHadoopã€ã®æ”¹è¨‚ç‰ˆã§ã™ã€‚ Hadoopã®åŸºç¤Žã‹ã‚‰å¿œç”¨ã¾ã§ã‚’åŒ…æ‹¬çš„ã«è§£èª¬ã™ã‚‹åˆç‰ˆã®å……å®Ÿã‚’ãã®ã¾ã¾ã«ã€Pigã€HBaseã€Hiveã€Sqoopã€Avroãªã©ã‚µãƒ–ãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆã‚„ã‚»ã‚ãƒ¥ãƒªãƒ†ã‚£ã«ã¤ã„ã¦ã‚‚å¤§å¹…ã«åŠ ç†ã€ã‚ˆã‚Šå®Ÿå‹™ã«å¯¾å¿œã§ãã‚‹å†…å®¹ã«ãªã‚Šã¾ã—ãŸã€‚Hadoop 0.1ç³»ã«åŠ ãˆã€0.2ç³»ã«é–¢ã™ã‚‹æƒ…å ±ã‚‚ä½µè¨˜ã€‚æ—¥æœ¬èªžç‰ˆã§ã¯ã€ŒNTTãƒ‡ãƒ¼ã‚¿ã®å®Ÿè¨¼äº‹æ¥ã«ãŠã‘ã‚‹Hadoopæ´»ç”¨ã®ãƒã‚¤ãƒ³ãƒˆã€ã‚’ä»˜éŒ²ã¨ã—ã¦æŽ²è¼‰ã—ã¦ã„ã¾ã™ã€‚Hadoopã®å…¨ä½“åƒã‚’ç¶²ç¾…ã—ã€ã‹ã¤å®Ÿè·µçš„ãªãƒˆãƒ”ãƒƒã‚¯ã‚’ç››ã‚Šè¾¼ã‚“ã æœ¬æ›¸ã¯ã€Hadoopã«é–¢å¿ƒã‚ã‚‹ã™ã¹ã¦ã®äººã«å¿…æºã®ä¸€å†Šã§ã™ã€‚ åºæ–‡ è¨³è€…ã¾ãˆãŒã ã¯ã˜ã‚ã« 1ç« ã€€Hadoopäº‹å§‹ã‚ 1.1ã€€ãƒ‡ãƒ¼ã‚¿ï¼ 1.2ã€€ãƒ‡ãƒ¼ã‚¿ã®ä¿ç®¡ã¨åˆ†æž 1.3ã€€ä»–ã®ã‚·ã‚¹ãƒ†ãƒ ã¨ã®æ¯”è¼ƒ 1.3.1ã€€RDBMS 1.3.2ã€€ã‚°ãƒªãƒƒãƒ‰ã‚³ãƒ³ãƒ”ãƒ¥ãƒ¼ãƒ†ã‚£ãƒ³ã‚° 1.3.3ã€€ãƒœãƒ©ãƒ³ãƒ†ã‚£ã‚¢ã‚³ãƒ³ãƒ”ãƒ¥ãƒ¼ãƒ†ã‚£ãƒ³ã‚° 1.4
otani0083 2011/07/12
Hadoop
ãƒªãƒ³ã‚¯
ã„ã¾ã•ã‚‰èžã‘ãªã„Hadoopã¨ãƒ†ã‚ã‚¹ãƒˆãƒžã‚¤ãƒ‹ãƒ³ã‚°å…¥é–€
ãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿æ™‚ä»£ã®æ•‘ä¸–ä¸»ã€ŒHadoopã€ã¨ã¯ ã€ŒApache Hadoopã€ã¯ä»Šã€æœ€ã‚‚æ³¨ç›®ã‚’é›†ã‚ã¦ã„ã‚‹æŠ€è¡“ã®1ã¤ã§ã™ã€‚Hadoopã¨ã¯ã€å¤§é‡ã®ãƒ‡ãƒ¼ã‚¿ã‚’æ‰‹è»½ã«è¤‡æ•°ã®ãƒžã‚·ãƒ³ã«åˆ†æ•£ã—ã¦å‡¦ç†ã§ãã‚‹ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ã®ãƒ—ãƒ©ãƒƒãƒˆãƒ•ã‚©ãƒ¼ãƒ ã§ã™ã€‚ Hadoopã‚’æ´»ç”¨ã—ã¦ã„ã‚‹ä¼æ¥ã¯å¹´ã€…å¢—ãˆç¶šã‘ã¦ã„ã¦ã€ä¸å¯æ¬ ãªæŠ€è¡“ã«ãªã‚Šã¤ã¤ã‚ã‚‹ã¨ã„ãˆã‚‹ã§ã—ã‚‡ã†ã€‚ æœ¬é€£è¼‰ã§ã¯ã€Hadoopã¨ã¯ä½•ã‹ã€HadoopãŒã©ã†æ´»ç”¨ã§ãã‚‹ã®ã‹ã¨ã„ã†ã“ã¨ã‚’ã€ã€Œãƒ†ã‚ã‚¹ãƒˆãƒžã‚¤ãƒ‹ãƒ³ã‚°ã€ã«ç„¦ç‚¹ã‚’å½“ã¦ã¦è§£èª¬ã—ã¦ã„ããŸã„ã¨æ€ã„ã¾ã™ã€‚ é‡ã„å‡¦ç†ã‚’è¤‡æ•°ã®ãƒžã‚·ãƒ³ã«åˆ†æ•£ã•ã›ã‚‹ è¤‡æ•°ã®ãƒžã‚·ãƒ³ã«å‡¦ç†ã‚’åˆ†æ•£ã•ã›ã‚‹ã«ã¯ã€ãƒ—ãƒã‚»ã‚¹åŒå£«ã®é€šä¿¡ã€ç›£è¦–ã€éšœå®³æ™‚ã®å¯¾å¿œãªã©ã‚’è€ƒãˆãªã‘ã‚Œã°ãªã‚‰ãšã€ãƒ—ãƒã‚°ãƒ©ãƒžã«ã¨ã£ã¦ãƒãƒ¼ãƒ‰ãƒ«ãŒé«˜ã„ã‚‚ã®ã§ã™ã€‚ã—ã‹ã—ã€Hadoopã¯ãã†ã„ã£ãŸé¢å€’ãã•ã„å‡¦ç†ã‚’ä¸€æ‰‹ã«å¼•ãå—ã‘ã¦ãã‚Œã€ãƒ—ãƒã‚°ãƒ©ãƒžã¯ã€ã‚„ã‚ŠãŸã„å‡¦ç†ã ã‘ã«é›†ä¸ã§ãã¾ã™ã€‚ ä¾‹ãˆã°ã€Hadoopã‚’ä½¿ã†ã¨ã€1
otani0083 2011/06/22
programming

Hadoop

è‡ªç„¶è¨€èªžå‡¦ç†
ãƒªãƒ³ã‚¯
Hadoopã‚’ç”¨ã„ãŸå¤§è¦æ¨¡ãƒã‚°è§£æž
this document is for #tokyowebmining held at 2011/04/10Read less
otani0083 2011/04/11
Hadoop
ãƒªãƒ³ã‚¯
Hadoopã‚’ä½¿ã„ã“ãªã™(1)
ã¾ãšã€ 1 ã®å…¥åŠ›ãƒ•ã‚¡ã‚¤ãƒ«ã‚’åˆ†å‰²ã™ã‚‹æ–¹æ³•ã¯ã€InputFormatã‚¯ãƒ©ã‚¹ã®ã€getSplitsé–¢æ•°ã‚’ä¸Šæ›¸ãã™ã‚‹ã“ã¨ã§ã€ã‚«ã‚¹ã‚¿ãƒžã‚¤ã‚ºã§ãã¾ã™ã€‚ ã¾ãŸã€ 3 ã®InputSplitã‹ã‚‰ã€Keyã¨Valueã‚’æŠ½å‡ºã™ã‚‹å‡¦ç†ã‚‚ã€InputFormatã‚¯ãƒ©ã‚¹ã‚’é€šã˜ã¦ã‚«ã‚¹ã‚¿ãƒžã‚¤ã‚ºã§ãã¾ã™ã€‚ InputFormatã®getRecordReaderé–¢æ•°ã‚’é€šã˜ã¦ã€RecordReaderã‚¯ãƒ©ã‚¹ã‚’ç”Ÿæˆã™ã‚‹ã®ã§ã™ãŒã€ã“ã‚Œã«ä»»æ„ã®RecordReaderã‚¯ãƒ©ã‚¹ã‚’æŒ‡å®šã™ã‚Œã°OKã§ã™ã€‚ 2 ã®Mapå‡¦ç†ã§ã™ãŒã€ãƒ¦ãƒ¼ã‚¶ãŒæŒ‡å®šã—ãŸMapperã‚¯ãƒ©ã‚¹ã®å‡¦ç†ã‚’å®Ÿè¡Œã—ã¾ã™ã€‚ Mapperã‚¯ãƒ©ã‚¹ã¯ã€MapRunnerã‚¯ãƒ©ã‚¹ã‚’é€šã˜ã¦ã€åˆæœŸåŒ–å‡¦ç†ã€mapé–¢æ•°ã‚’ç¹°ã‚Šè¿”ã™éŽç¨‹ã€çµ‚äº†å‡¦ç†ã¨ã„ã£ãŸä¸€é€£ã®æµã‚Œã‚’å®Ÿè¡Œã—ã¾ã™ã€‚ MapRunnerã‚¯ãƒ©ã‚¹ã‚’ã‚«ã‚¹ã‚¿ãƒžã‚¤ã‚ºã™ã‚Œã°ã€ã“ã†ã—ãŸæµã‚Œã‚’åˆ¶å¾¡ã™ã‚‹ã“ã¨ãŒã§ãã¾ã™ã€‚ 0.20.0ã‹ã‚‰ã®æ–°ã—ã„MapRed
otani0083 2010/01/27
programming

Hadoop
ãƒªãƒ³ã‚¯
æ¤œç´¢ã‚¯ã‚¨ãƒªãƒã‚°ã‹ã‚‰ã®ã‚¹ãƒšãƒ«è¨‚æ£è¾žæ›¸ã®è‡ªå‹•ç”Ÿæˆ - mixi engineer blog
å…ˆæœˆãƒãƒ¯ã‚¤ã«è¡Œã£ã¦ãã¦ã‚ªãƒ«ã‚ªãƒ«ãª ï¼ˆãƒãƒ¯ã‚¤èªžã§ 'æ¥½ã—ã„' ã¨ã„ã†æ„å‘³ï¼‰ æ°—åˆ†ã® takahi-i ã§ã™ã€‚æœ€è¿‘ãƒã‚°ãƒ‡ãƒ¼ã‚¿ã®æœ‰åŠ¹æ´»ç”¨ãŒè©±é¡Œã«ãªã£ã¦ã„ã¾ã™ãŒã€æ¤œç´¢ã‚¨ãƒ³ã‚¸ãƒ³ãŒå‡ºåŠ›ã™ã‚‹æ¤œç´¢ã‚¯ã‚¨ãƒªãƒã‚°ã‚’ä½¿ç”¨ã—ã¦ã©ã‚“ãªã“ã¨ãŒã§ãã‚‹ã®ã‹ã«ã¤ã„ã¦ç´¹ä»‹ã•ã›ã¦ã„ãŸã ãã¾ã™ã€‚ æ¤œç´¢ã‚¯ã‚¨ãƒªãƒã‚° æ¤œç´¢ã‚¯ã‚¨ãƒªãƒã‚° (ä»¥ä¸‹æ¤œç´¢ãƒã‚°) ã¯æ¤œç´¢ã‚¨ãƒ³ã‚¸ãƒ³ã‚’ä½¿ç”¨ã™ã‚‹ãƒ¦ãƒ¼ã‚¶ã‹ã‚‰ç™ºè¡Œã•ã‚ŒãŸæ¤œç´¢ã®å±¥æ´ã‚’ä¿å˜ã—ãŸãƒ•ã‚¡ã‚¤ãƒ«ã§ã™ã€‚æ¤œç´¢ãƒã‚°ã®ãƒ•ã‚©ãƒ¼ãƒžãƒƒãƒˆã¯ä½¿ç”¨ã™ã‚‹æ¤œç´¢ã‚¨ãƒ³ã‚¸ãƒ³ã‚„ Web ã‚µãƒ¼ãƒã«ã‚ˆã£ã¦ç•°ãªã‚Šã¾ã™ã€‚ã•ã‚‰ã«ã¾ãŸæ¤œç´¢ãƒã‚°ãŒå«ã‚€æƒ…å ±ã«ã‚‚å·®ç•°ãŒã‚ã‚‹ã“ã¨ãŒè€ƒãˆã‚‰ã‚Œã¾ã™ãŒã€æœ¬ç¨¿ã§ã¯æ¤œç´¢ãƒã‚°ã¯è§£æžã‚’è¡Œã†ä¸Šã§é‡è¦ãªä¸‰ã¤ã®è¦ç´ ã‚’å«ã‚€ã¨ä»®å®šã—ã¾ã™ã€‚ä¸‰ã¤ã®è¦ç´ ã¨ã¯ãƒ¦ãƒ¼ã‚¶ ID ï¼ˆã‚‚ã—ãã¯ IP ã‚¢ãƒ‰ãƒ¬ã‚¹ï¼‰ã€ã‚¯ã‚¨ãƒªæ–‡ã€ãã—ã¦ã‚¯ã‚¨ãƒªãŒæ¤œç´¢ã‚¨ãƒ³ã‚¸ãƒ³ã«å‡¦ç†ã•ã‚ŒãŸæ™‚é–“ã§ã™ã€‚ä»¥ä¸‹æ¤œç´¢ãƒã‚°ã®ä¸€ä¾‹ã‚’è¼‰ã›ã¾ã™ã€‚ ãƒ¦ãƒ¼ã‚¶ID ã‚¯ã‚¨ãƒªæ–‡ ã‚¯ã‚¨ãƒªç™ºè¡Œæ™‚ 438904 Su
otani0083 2009/12/09
æ¤œç´¢

Hadoop
ãƒªãƒ³ã‚¯
1