[B! Hadoop] pokutunaã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

pokutuna id:pokutuna

Hadoopã«é–¢ã™ã‚‹pokutunaã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (5)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

Hadoopã‚’ç”¨ã„ãŸå¤§è¦æ¨¡ãƒã‚°è§£æž
this document is for #tokyowebmining held at 2011/04/10Read less
pokutuna 2011/04/12
Hadoop

DataMining
ãƒªãƒ³ã‚¯
Scala ã§ Hadoop ã‚’è©¦ã—ã¦ã¿ã‚‹ | Atsushi TATSUMA Web Page
ã¯ã˜ã‚ã« Scala ã§ Hadoop ç”¨ã® jar ãƒ•ã‚¡ã‚¤ãƒ«ã‚’ä½œã£ã¦ã¿ã¾ã—ãŸã€‚ä½œã£ã¦ã¿ãŸã®ã¯ã‚ˆãã‚ã‚‹å˜èªžã‚’æ•°ãˆã‚‹ãƒ—ãƒã‚°ãƒ©ãƒ ã§ã™ã€‚ ç’°å¢ƒã¯ã€Debian GNU/Linux ã« Cloudera ç¤¾ã® Hadoop ã‚’ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã—ãŸã‚‚ã®ã§ã™ã€‚ æº–å‚™ Scala ã§ä½œã£ãŸ jar ãƒ•ã‚¡ã‚¤ãƒ«ã‚’å®Ÿè¡Œã™ã‚‹ãŸã‚ã«ã€scala-library.jar ã‚’ Hadoop ã®ãƒ©ã‚¤ãƒ–ãƒ©ãƒªã«é…ç½®ã—ã¾ã™ã€‚ ã“ã®ä½œæ¥ã¯ã€ãƒžã‚¹ã‚¿ãƒ¼ã¨ã‚¹ãƒ¬ãƒ¼ãƒ–ã®å…¨ãƒžã‚·ãƒ³ã§è¡Œã†å¿…è¦ãŒã‚ã‚Šã¾ã™ã€‚ [bash] $ cd /usr/lib/hadoop/lib/ $ sudo ln -s /usr/local/lib/scala/lib/scala-library.jar scala-library.jar [/bash] ãƒ©ã‚¤ãƒ–ãƒ©ãƒªã®è¿½åŠ ã‚’åæ˜ ã•ã›ã‚‹ãŸã‚ tasktracker ã¨ jobtracker ã®å†èµ·å‹•ãŒå¿…è¦ã«ãª
pokutuna 2011/01/30
Hadoop

Scala
ãƒªãƒ³ã‚¯
ãã‚ãã‚Hadoopã«ã¤ã„ã¦ã²ã¨ã“ã¨è¨€ã£ã¦ãŠãã‹ - nokunoã®æ—¥è¨˜
ã‚‚ã†ã“ã®æ‰‹ã®è©±é¡Œã¯å‡ºå°½ãã—ãŸæ„ŸãŒã‚ã‚Šã¾ã™ãŒã€æœ€è¿‘Hadoopã«ã¤ã„ã¦è€ƒãˆã‚‹ã“ã¨ãŒå¤šã„ã®ã§ã€ã‚¨ãƒ³ãƒˆãƒªã«ã—ã¦ã¿ã¾ã™ã€‚ãªãŠã€ã“ã“ã§ã¯ãƒ™ãƒ¼ã‚·ãƒƒã‚¯ãªMapReduce+HDFSã®ã“ã¨ã‚’Hadoopã¨å‘¼ã¶ã“ã¨ã«ã—ã¾ã™ã€‚ Hadoopã¨ã¯Hadoopã¨ã¯è¨€ã‚ãšã¨çŸ¥ã‚ŒãŸGoogleã®MapReduce/GFSã®ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ã®ã‚¯ãƒãƒ¼ãƒ³ã§ã™ã€‚MapReduceã§ã¯ãƒ—ãƒã‚°ãƒ©ãƒžã¯Mapã¨Reduceã¨ã„ã†2ã¤ã®é–¢æ•°ã‚’æ›¸ãã ã‘ã§ã€ä¸¦åˆ—åˆ†æ•£å‡¦ç†ã‚’ã™ã‚‹ã“ã¨ãŒã§ãã¾ã™ã€‚ã“ã‚Œã¯(1) ãƒ‡ãƒ¼ã‚¿ã‚’å®Ÿéš›ã«æŒã¤ãƒžã‚·ãƒ³ã«ãƒ—ãƒã‚°ãƒ©ãƒ ã‚’é…å¸ƒã™ã‚‹ (2) Mapã¨Reduceã‚’ã¤ãªãShuffleãƒ•ã‚§ãƒ¼ã‚ºã§ã‚ãƒ¼ã‚’ã‚°ãƒ«ãƒ¼ãƒ—åŒ–ã—ã¦ã‚½ãƒ¼ãƒˆã™ã‚‹ã€(3) éšœå®³æ™‚ã®ãƒ•ã‚§ãƒ¼ãƒ«ã‚ªãƒ¼ãƒãƒ¼ã‚„ãƒ¬ãƒ—ãƒªã‚±ãƒ¼ã‚·ãƒ§ãƒ³ã€ã¨ã„ã£ãŸå‡¦ç†ã‚’ãƒ•ãƒ¬ãƒ¼ãƒ ãƒ¯ãƒ¼ã‚¯å´ãŒå—ã‘æŒã¤ã“ã¨ã«ã‚ˆã£ã¦ã€ãƒ—ãƒã‚°ãƒ©ãƒžå´ã®è² æ‹…ã‚’æ¸›ã‚‰ã™ã‚‚ã®ã§ã™ã€‚GFSã«å¯¾å¿œã™ã‚‹HDFSã«ã¯ãƒ•ã‚¡ã‚¤ãƒ«ã‚’ã‚¯ãƒ©ã‚¹ã‚¿ã«åˆ†æ•£ã—ã¦ä¿å˜
pokutuna 2011/01/03
Hadoop
ãƒªãƒ³ã‚¯
sbt ã‚’ä½¿ã£ã¦ Scala ã§ Hadoop MapReduce å®Ÿè£… - ãªã‚“ã¨ãªããª Developer ã®ãƒ¡ãƒ¢
sbt ã‚’ä½¿ã£ã¦ MapReduce ã‚’ Scala ã§å®Ÿè£…ã—ã¦ã¿ã¾ã—ãŸã€‚ ä½¿ç”¨ã—ãŸç’°å¢ƒã¯ä»¥ä¸‹ã®é€šã‚Šã€‚ï¼ˆsbtã®ç’°å¢ƒã¯ id:fits:20100810 ã¨åŒã˜ï¼‰ Hadoop 0.21.0 ã‚µãƒ³ãƒ—ãƒ«ã‚½ãƒ¼ã‚¹ã¯ http://github.com/fits/try_samples/tree/master/blog/20101010/ ãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆè¨å®š ã¯ã˜ã‚ã«ç’°å¢ƒå¤‰æ•° HADOOP_HOME ã« Hadoop ãƒ«ãƒ¼ãƒˆãƒ‡ã‚£ãƒ¬ã‚¯ãƒˆãƒªã¸ã®ãƒ‘ã‚¹ã‚’è¨å®šã—ã¦ãŠãã¾ã™ã€‚ æ¬¡ã« sbt ã§ãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆã‚’ä½œæˆã€ãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆãƒ•ã‚¡ã‚¤ãƒ«ã‚’ä»¥ä¸‹ã®ã‚ˆã†ã«ä½œæˆã—ã€Hadoop é–¢é€£ã® JAR ãƒ•ã‚¡ã‚¤ãƒ«ã‚’å‚ç…§ã—ã¦ãƒ“ãƒ«ãƒ‰ã§ãã‚‹ã‚ˆã†ã«ã—ã¾ã™ã€‚ project/build/MoneyCounterProject.scala import sbt._ class MoneyCounterProject(info: ProjectI
pokutuna 2010/10/28
Hadoop

Scala

Sbt
ãƒªãƒ³ã‚¯
NTTãƒ‡ãƒ¼ã‚¿ã®Hadoopå ±å‘Šæ›¸ãŒã™ã”ã‹ã£ãŸ - ç§‘å¦ã¨éžç§‘å¦ã®è¿·å®®
æ¥ç•Œãƒˆãƒƒãƒ— ã®ã‚¨ãƒ³ã‚¿ãƒ¼ãƒ—ãƒ©ã‚¤ã‚º Hadoop ä¼æ¥ Cloudera ã«å…¥ç¤¾ã—ã¾ã—ãŸ http://www.cloudera.co.jp/ ä»Šå¹´ã®6æœˆã«ã€ã€Œå¹³æˆï¼’ï¼‘å¹´åº¦ ç”£å¦é€£æºã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢å·¥å¦å®Ÿè·µäº‹æ¥å ±å‘Šæ›¸ã€ã¨ã„ã†ãƒ‰ã‚ãƒ¥ãƒ¡ãƒ³ãƒˆç¾¤ãŒçµŒç”£çœã‹ã‚‰å…¬è¡¨ã•ã‚Œã¾ã—ãŸã€‚ ãã®ã†ã¡ã®ä¸€ã¤ã«ã€NTTãƒ‡ãƒ¼ã‚¿ã«å§”è¨—ã•ã‚ŒãŸHadoopã«é–¢ã™ã‚‹å®Ÿè¨¼å®Ÿé¨“ã®å ±å‘Šæ›¸ãŒã‚ã‚Šã¾ã—ãŸã®ã§ã€ä»Šæ›´ãªãŒã‚‰èªã‚“ã§ã¿ã‚‹ã“ã¨ã«ã—ã¾ã—ãŸã€‚ Hadoopç•Œéšˆã®äººã¯ã‚‚ã†ã¿ã‚“ãªã¨ã£ãã«èªã‚“ã§ã‚‹ã®ã‹ã‚‚ã—ã‚Œã¾ã›ã‚“ã‘ã©ã€‚ http://www.meti.go.jp/policy/mono_info_service/joho/downloadfiles/2010software_research/clou_dist_software.pdf ã€Œé«˜ä¿¡é ¼ã‚¯ãƒ©ã‚¦ãƒ‰å®Ÿç¾ç”¨ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢é–‹ç™ºï¼ˆåˆ†æ•£åˆ¶å¾¡å‡¦ç†æŠ€è¡“ç‰ã«ä¿‚ã‚‹ãƒ‡ãƒ¼ã‚¿ã‚»ãƒ³ã‚¿ãƒ¼é«˜ä¿¡é ¼åŒ–ã«å‘ã‘ãŸå®Ÿè¨¼äº‹æ¥ï¼‰ã€ã¨ã„ã†
pokutuna 2010/09/28
Hadoop

Algorithm
ãƒªãƒ³ã‚¯
1