[B! hadoop] superbrothersã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

superbrothers id:superbrothers

hadoopã«é–¢ã™ã‚‹superbrothersã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (5)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

HadoopãŒã‚¹ã‚±ãƒ¼ãƒ«ãƒ»ã‚¢ã‚¦ãƒˆã™ã‚‹ä»•çµ„ã¿
å‰å›žã®è¨˜äº‹ã§ã¯ã€HadoopãŒè†¨å¤§ãªãƒ‡ãƒ¼ã‚¿ã‚’ãƒãƒƒãƒå‡¦ç†ã™ã‚‹ãŸã‚ã®ã€Œã‚¤ãƒ³ãƒ•ãƒ©ã€ã¨ã—ã¦ã®æ€§è³ªã‚’å‚™ãˆã¦ã„ã‚‹ã“ã¨ã€æƒ…å ±çˆ†ç™ºæ™‚ä»£ã®æ–°ãŸãªã‚¤ãƒ³ãƒ•ãƒ©ã¨ã—ã¦æ™®åŠã—ã¤ã¤ã‚ã‚‹ã“ã¨ã‚’èª¬æ˜Žã—ã¾ã—ãŸã€‚ãã®ä¸ã§ã€æƒ…å ±çˆ†ç™ºæ™‚ä»£ã«å¿…è¦ã¨ã•ã‚Œã‚‹ã‚¤ãƒ³ãƒ•ãƒ©ã¯ã€ã€Œã‚¹ã‚±ãƒ¼ãƒ«ãƒ»ã‚¢ã‚¦ãƒˆãŒå¯èƒ½ã§ã‚ã‚‹ã“ã¨ã€ã¨ã„ã†æ¡ä»¶ã‚’å‚™ãˆã¦ã„ãªã‘ã‚Œã°ãªã‚‰ãªã„ã“ã¨ã‚’ç¤ºã—ã¾ã—ãŸã€‚ã‚µãƒ¼ãƒãƒ¼ã®å°æ•°ã‚’å¢—ã‚„ã™ã“ã¨ã§å®¹æ˜“ã«ã‚·ã‚¹ãƒ†ãƒ ã®å‡¦ç†æ€§èƒ½ãŒå¢—ã‚„ã›ã‚‹ã“ã¨ã¯ã€Hadoopã®é‡è¦ãªç‰¹å¾´ã§ã™ã€‚ä»Šå›žã¯ã€HadoopãŒã©ã®ã‚ˆã†ã«ã—ã¦ã‚¹ã‚±ãƒ¼ãƒ«ãƒ»ã‚¢ã‚¦ãƒˆã‚’å¯èƒ½ã«ã—ã¦ã„ã‚‹ã‹ã‚’ã€ã€Œåˆ†æ•£ãƒ•ã‚¡ã‚¤ãƒ«ãƒ»ã‚·ã‚¹ãƒ†ãƒ ã€ã¨ã€ŒMapReduceãƒ•ãƒ¬ãƒ¼ãƒ ãƒ¯ãƒ¼ã‚¯ã€ã®2ã¤ã®è¦³ç‚¹ã‹ã‚‰è§£èª¬ã—ã¾ã™ã€‚ ã‚¹ã‚±ãƒ¼ãƒ«ãƒ»ã‚¢ã‚¦ãƒˆã¨ã¯ï¼Ÿ ã‚³ãƒ³ãƒ”ãƒ¥ãƒ¼ã‚¿ãƒ»ã‚·ã‚¹ãƒ†ãƒ ã‚’æ–°ãŸã«æ§‹ç¯‰ã™ã‚‹ã¨ãã¯ã€å¿…ãšã€Œé‹ç”¨ã€ã®ã“ã¨ã‚’è€ƒãˆã¦ãŠãå¿…è¦ãŒã‚ã‚Šã¾ã™ã€‚ã‚·ã‚¹ãƒ†ãƒ ãƒ»ãƒˆãƒ©ãƒ–ãƒ«ãŒç™ºç”Ÿã—ãŸå ´åˆã®å¯¾å¿œç–ã‚’æ±ºã‚ãŸã‚Šã€å°†æ¥ã®ä»•æ§˜å¤‰æ›´ã«å‚™ãˆã¦ãƒ—ãƒã‚°ãƒ©ãƒ ã«æ‹¡å¼µæ€§ã‚’æŒãŸã›ãŸã‚Š
superbrothers 2010/06/20
hadoop

ã‚ã¨ã§èªã‚€
ãƒªãƒ³ã‚¯
BASH:Born again AudioSwitcHã€€ Hadoop Streaming ã‚’è©¦ã—ã¦ã¿ã‚‹
å¤§å¦ã® Debian GNU/Linux ç’°å¢ƒã« Hadoop ã‚’ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã—ãŸã®ã§æ—©é€Ÿè©¦ã—ã¾ã—ãŸã€‚ æ¨™æº–å…¥å‡ºåŠ›ã‚’åˆ©ç”¨ã™ã‚‹ Hadoop Streaming ã‚’ä½¿ã†ã¨ã©ã‚“ãªè¨€èªžã§ã‚‚è©¦ã›ã¾ã™ã€‚ Perl ã‚„ Python ã¯ãƒãƒƒãƒˆã«ä¾‹ãŒæ²¢å±±ã‚ã‚‹ã®ã§ Gauche ã§ã‚„ã£ã¦ã¿ã¾ã—ãŸã€‚ ã‚ˆãã‚ã‚‹å˜èªžæ•°ã‚’æ•°ãˆã‚‹ãƒ¢ãƒŽã§ã™ã€‚ 1.ã‚¹ã‚¯ãƒªãƒ—ãƒˆã‚’å®Ÿè¡Œã§ãã‚‹ã‚ˆã†ã«ã™ã‚‹ å„ã‚¹ãƒ¬ãƒ¼ãƒ–ã«ã‚¹ã‚¯ãƒªãƒ—ãƒˆã®ã‚¤ãƒ³ã‚¿ãƒ—ãƒªã‚¿ã‚’ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã—ã¦ãŠãã¾ã™ã€‚ $ sudo apt-get install gauche2.Mapperã‚’ç”¨æ„ã™ã‚‹ ã‚¹ãƒšãƒ¼ã‚¹åŒºåˆ‡ã‚Šã§å˜èªžã‚’æŠœãå‡ºã—ã€Œå˜èªž 1ã€ã¨ã„ã†ã®ã‚’å‡ºåŠ›ã—ã¾ã™ã€‚ $ vim mapper.scm #!/usr/bin/env gosh ;; -*- coding: utf-8 -*- (use srfi-13) (fold (lambda (str j) (if (not (strin
superbrothers 2010/05/10
hadoop
ãƒªãƒ³ã‚¯
Engineer event in Kyoto 10-04-17
This Ruby script takes a log file as input with entries separated by commas, parses each line into a unique ID and count, increments or sets the count for each ID in a hash, and finally prints out the number of unique IDs in the log file.Read less
superbrothers 2010/04/28
ruby

hadoop

cookpad

slideshare
ãƒªãƒ³ã‚¯
Hadoopã‚’ä½¿ã„ã“ãªã™(1)
ã¾ãšã€ 1 ã®å…¥åŠ›ãƒ•ã‚¡ã‚¤ãƒ«ã‚’åˆ†å‰²ã™ã‚‹æ–¹æ³•ã¯ã€InputFormatã‚¯ãƒ©ã‚¹ã®ã€getSplitsé–¢æ•°ã‚’ä¸Šæ›¸ãã™ã‚‹ã“ã¨ã§ã€ã‚«ã‚¹ã‚¿ãƒžã‚¤ã‚ºã§ãã¾ã™ã€‚ ã¾ãŸã€ 3 ã®InputSplitã‹ã‚‰ã€Keyã¨Valueã‚’æŠ½å‡ºã™ã‚‹å‡¦ç†ã‚‚ã€InputFormatã‚¯ãƒ©ã‚¹ã‚’é€šã˜ã¦ã‚«ã‚¹ã‚¿ãƒžã‚¤ã‚ºã§ãã¾ã™ã€‚ InputFormatã®getRecordReaderé–¢æ•°ã‚’é€šã˜ã¦ã€RecordReaderã‚¯ãƒ©ã‚¹ã‚’ç”Ÿæˆã™ã‚‹ã®ã§ã™ãŒã€ã“ã‚Œã«ä»»æ„ã®RecordReaderã‚¯ãƒ©ã‚¹ã‚’æŒ‡å®šã™ã‚Œã°OKã§ã™ã€‚ 2 ã®Mapå‡¦ç†ã§ã™ãŒã€ãƒ¦ãƒ¼ã‚¶ãŒæŒ‡å®šã—ãŸMapperã‚¯ãƒ©ã‚¹ã®å‡¦ç†ã‚’å®Ÿè¡Œã—ã¾ã™ã€‚ Mapperã‚¯ãƒ©ã‚¹ã¯ã€MapRunnerã‚¯ãƒ©ã‚¹ã‚’é€šã˜ã¦ã€åˆæœŸåŒ–å‡¦ç†ã€mapé–¢æ•°ã‚’ç¹°ã‚Šè¿”ã™éŽç¨‹ã€çµ‚äº†å‡¦ç†ã¨ã„ã£ãŸä¸€é€£ã®æµã‚Œã‚’å®Ÿè¡Œã—ã¾ã™ã€‚ MapRunnerã‚¯ãƒ©ã‚¹ã‚’ã‚«ã‚¹ã‚¿ãƒžã‚¤ã‚ºã™ã‚Œã°ã€ã“ã†ã—ãŸæµã‚Œã‚’åˆ¶å¾¡ã™ã‚‹ã“ã¨ãŒã§ãã¾ã™ã€‚ 0.20.0ã‹ã‚‰ã®æ–°ã—ã„MapRed
superbrothers 2010/03/18
yahoo

hadoop

ã‚ã¨ã§...
ãƒªãƒ³ã‚¯
Hadoopã§ã€ã‹ã‚“ãŸã‚“åˆ†æ•£å‡¦ç† (Yahoo! JAPAN Tech Blog)
ãƒ¤ãƒ•ãƒ¼æ ªå¼ä¼šç¤¾ã¯ã€2023å¹´10æœˆ1æ—¥ã«LINEãƒ¤ãƒ•ãƒ¼æ ªå¼ä¼šç¤¾ã«ãªã‚Šã¾ã—ãŸã€‚LINEãƒ¤ãƒ•ãƒ¼æ ªå¼ä¼šç¤¾ã®æ–°ã—ã„ãƒ–ãƒã‚°ã¯ã“ã¡ã‚‰ã§ã™ã€‚LINEãƒ¤ãƒ•ãƒ¼ Tech Blog ã“ã‚“ã«ã¡ã¯ã€åœ°åŸŸã‚µãƒ¼ãƒ“ã‚¹äº‹æ¥éƒ¨ã®å‰ç”°ä¸€æ˜Ÿã§ã™ã€‚ ä»Šå›žã¯ã€Hadoopã«ã¤ã„ã¦ã€Yahoo! JAPANã§ã®å®Ÿéš›ã®ä½¿ç”¨ä¾‹ã‚’äº¤ãˆãªãŒã‚‰æ›¸ããŸã„ã¨æ€ã„ã¾ã™ã€‚Hadoopã¨ã¯ã€å¤§é‡ã®ãƒ‡ãƒ¼ã‚¿ã‚’æ‰‹è»½ã«è¤‡æ•°ã®ãƒžã‚·ãƒ³ã«åˆ†æ•£ã—ã¦å‡¦ç†ã§ãã‚‹ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ã®ãƒ—ãƒ©ãƒƒãƒˆãƒ•ã‚©ãƒ¼ãƒ ã§ã™ã€‚ è¤‡æ•°ã®ãƒžã‚·ãƒ³ã¸ã®åˆ†æ•£å‡¦ç†ã¯ã€ãƒ—ãƒã‚»ã‚¹é–“é€šä¿¡ã‚„ã€éšœå®³æ™‚ã¸ã®å¯¾å¿œãªã©ã‚’è€ƒãˆãªã‘ã‚Œã°ãªã‚‰ãšã€ãƒ—ãƒã‚°ãƒ©ãƒžã«ã¨ã£ã¦æ•·å±…ãŒé«˜ã„ã‚‚ã®ã§ã™ãŒã€ Hadoopã¯ãã†ã„ã£ãŸé¢å€’ãã•ã„åˆ†æ•£å‡¦ç†ã‚’ä¸€æ‰‹ã«å¼•ãå—ã‘ã¦ãã‚Œã¾ã™ã€‚ 1å°ã§ã¯å‡¦ç†ã«ã‹ãªã‚Šæ™‚é–“ãŒã‹ã‹ã‚‹ã‚ˆã†ãªå¤§é‡ã®ãƒ‡ãƒ¼ã‚¿ã‚‚ã€è¤‡æ•°ãƒžã‚·ãƒ³ã«åˆ†æ•£ã•ã›ã‚‹ã“ã¨ã§ã€é©šãã¹ãã‚¹ãƒ”ãƒ¼ãƒ‰ã§å‡¦ç†ã‚’è¡Œã†ã“ã¨ãŒã§ãã¾ã™ã€‚ ä¾‹ãˆã°ã€ä»Šã¾ã§1å°ã§ã‚„ã£ã¦ã„ãŸã€ã‚ã‚‹ãƒã‚°é›†è¨ˆå‡¦ç†
superbrothers 2009/03/03
yahoo

hadoop
ãƒªãƒ³ã‚¯
1