[B! distributed] kanbayashiã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

kanbayashi id:kanbayashi

distributedã«é–¢ã™ã‚‹kanbayashiã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (76)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

S4
S4 - Presentation Transcript S4: the open-source distributed stream computing platformNachi Ueno è‡ªå·±ç´¹ä»‹ æ—¥æœ¬ã®ã©ã“ã‹ã®ç ”ç©¶æ‰€ã¨å‘¼ã°ã‚Œã¦ã„ã‚‹çµ„ç¹”ã§æ—¥ã€…åƒã„ã¦ã„ã¾ã™ æœ€è¿‘ã¯ã€Hadoop,OpenStack,ã‚¹ã‚±ãƒ¼ãƒ«ã™ã‚‹RDBMSã¨ã‹ã«èˆˆå‘³ãŒã‚ã‚Šã¾ã™ SimCloudã¨ã„ã†ã‚¯ãƒ©ã‚¦ãƒ‰ç®¡ç†ãƒ„ãƒ¼ãƒ«ã‚’ä½œã£ã¦ã¾ã™ã€€->ã€€OSSå…¬é–‹äºˆå®šã€€ http://wiki.openstack.org/simcloud Twitter @nati OpenStackeré”ã¨@ç¯‰åœ° S4 (Simple Scala ble Streaming System) ã€€http://s4.io/ S4 is a general-purpose, distributed, scala ble, partially fault-tole
kanbayashi 2013/06/30
hadoop

distributed

s4

stream

map reduce

mapreduce

slide share

slideshare
ãƒªãƒ³ã‚¯
ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ åˆ†æ•£å‡¦ç†ã®å¸¸è˜ã‚’Apache S4ã§èº«ã«ã¤ã‘ã‚‹
ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ åˆ†æ•£å‡¦ç†ã®å¸¸è˜ã‚’Apache S4ã§èº«ã«ã¤ã‘ã‚‹ï¼šãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿å‡¦ç†ã®å¸¸è˜ã‚’Javaã§èº«ã«ã¤ã‘ã‚‹ï¼ˆ6ï¼‰ï¼ˆ1/2 ãƒšãƒ¼ã‚¸ï¼‰ Hadoopã‚’ã¯ã˜ã‚ã€Javaè¨€èªžã‚’ä½¿ã£ã¦æ§‹ç¯‰ã•ã‚Œã‚‹ã“ã¨ãŒå¤šã„ã€Œãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿ã€å‡¦ç†ã®ãŸã‚ã®ãƒ•ãƒ¬ãƒ¼ãƒ ãƒ¯ãƒ¼ã‚¯ï¼ãƒ©ã‚¤ãƒ–ãƒ©ãƒªã‚’ç´¹ä»‹ã—ãªãŒã‚‰ã€å¤§é‡ãƒ‡ãƒ¼ã‚¿ã‚’æ´»ç”¨ã™ã‚‹ãŸã‚ã®æŠ€è¡“ã®å¸¸è˜ã‚’èº«ã«ä»˜ã‘ã¦ã„ãé€£è¼‰ Hadoopã®å¼±ç‚¹ã€Œãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ åˆ†æ•£å‡¦ç†ã€ã¨ã¯ ã€Œãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿ã€å‡¦ç†ã®ãŸã‚ã«Hadoopã‚’ç”¨ã„ã‚‹ã¨ã€ã€Œè¤‡æ•°ã®ãƒžã‚·ãƒ³ã«å¤§é‡ãƒ‡ãƒ¼ã‚¿å‡¦ç†ã‚’åˆ†æ•£ã—ã¦é£›èºçš„ã«æ€§èƒ½ã‚’å‘ä¸Šã™ã‚‹ã€ã“ã¨ãŒå®¹æ˜“ã«ã§ãã¾ã™ã€‚ ã¨ã“ã‚ãŒHadoopã®å¼±ç‚¹ã¨ã—ã¦ã€ãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿ã‚’ã„ã£ãŸã‚“è“„ç©ã—ã€ãƒãƒƒãƒã§ä¸€æ‹¬å‡¦ç†ã™ã‚‹å½¢æ…‹ã§å‡¦ç†ã™ã‚‹ã®ã§ã€å‡¦ç†ãƒ‡ãƒ¼ã‚¿ãŒç™ºç”Ÿã—ã¦ã‹ã‚‰ã€ãã‚Œã«å¯¾ã™ã‚‹å‡¦ç†çµæžœãŒå¾—ã‚‰ã‚Œã‚‹ã¾ã§ã€å¿…ãšã‚¿ã‚¤ãƒ ãƒ©ã‚°ãŒç™ºç”Ÿã—ã¾ã™ã€‚ã“ã®ãŸã‚ã€ã‚¯ãƒ¬ã‚¸ãƒƒãƒˆã‚«ãƒ¼ãƒ‰ã®ä¸æ£ã‚¢ã‚¯ã‚»ã‚¹æ¤œçŸ¥ã€ã‚»ãƒ³ã‚µãƒ‡ãƒ¼ã‚¿ãªã©ã§ã®ç•°å¸¸å€¤æ¤œå‡ºã®ã‚ˆã†ãªãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ãª
kanbayashi 2013/06/30
stream

ã‚¹ãƒˆãƒªãƒ¼ãƒ

s4

apache s4

distributed

hadoop

map reduce

mapreduce
ãƒªãƒ³ã‚¯
ã‚¹ãƒˆãƒªãƒ¼ãƒ ãƒ‡ãƒ¼ã‚¿åˆ†æ•£å‡¦ç†åŸºç›¤Storm
2012å¹´12æœˆ10æ—¥ NTTãƒ‡ãƒ¼ã‚¿ ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹DAY 2012 è¬›æ¼”è³‡æ–™ ã€Žã‚¹ãƒˆãƒªãƒ¼ãƒ ãƒ‡ãƒ¼ã‚¿åˆ†æ•£å‡¦ç†åŸºç›¤ Stormã€ NTTãƒ‡ãƒ¼ã‚¿ åŸºç›¤ã‚·ã‚¹ãƒ†ãƒ äº‹æ¥æœ¬éƒ¨ OSSãƒ—ãƒãƒ•ã‚§ãƒƒã‚·ãƒ§ãƒŠãƒ«ã‚µãƒ¼ãƒ“ã‚¹ å²©å´Ž æ£å‰› http://oss.nttdata.co.jp/hadoop/Read less
kanbayashi 2013/06/13
slideshare

slide share

storm

twitter

distributed

stream

stream data

parallel

ntt
ãƒªãƒ³ã‚¯
Twitterã®ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ åˆ†æ•£å‡¦ç†ã‚·ã‚¹ãƒ†ãƒ ã€ŒStormã€å…¥é–€
éˆ´æœ¨ è²´å…¸ï¼‹æœ¨æ‘ å®—å¤ªéƒŽ Twitterã®Nathan Marzæ°ãŒé–‹ç™ºã—ã¦ã„ã‚‹ ã€ŒStormã€ ã€‚ 2011å¹´9æœˆã«ã€ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ã¨ã—ã¦å…¬é–‹ã•ã‚Œã¾ã—ãŸãŒã€ã¾ã ã¾ã æ—¥æœ¬ã§å¾—ã‚‰ã‚Œã‚‹æƒ…å ±ã¯å°‘ãªãã€ è©³ç´°ã¯ä¸æ˜Žãªã¨ã“ã‚ã‚‚å¤šã„ã§ã™ã€‚ ãã“ã§ã€æ—¥æœ¬åˆï¼ˆãŸã¶ã‚“ï¼‰ã®ã€ŒStormã€ã‚»ãƒƒã‚·ãƒ§ãƒ³ã‚’è¡Œã„ã¾ã™ã€‚ ã€ŒStormã€ã¯ã€CEPï¼ˆComplex Event Processingï¼è¤‡åˆã‚¤ãƒ™ãƒ³ãƒˆå‡¦ç†ï¼‰ã«å±žã™ã‚‹ãƒ—ãƒãƒ€ã‚¯ãƒˆã§ã‚ã‚Šã€ åˆ†æ•£ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ å‡¦ç†ã‚’è¡Œã†ãŸã‚ã®åŸºæœ¬ã‚»ãƒƒãƒˆã‚’æä¾›ã—ã¦ã„ã¾ã™ã€‚ ä»Šå›žã®ã‚»ãƒƒã‚·ãƒ§ãƒ³ã§ã¯ã€ãã®æ¦‚å¿µã‚„ç‰¹æ€§ãªã©ã«ã¤ã„ã¦è§£èª¬ã—ã¦ã¿ã¾ã™ã€‚Read less
kanbayashi 2013/06/13
slideshare

slide share

storm

twitter

distributed

stream

stream data

parallel
ãƒªãƒ³ã‚¯
Home
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert
kanbayashi 2013/06/12
storm

tutorial

distributed

stream
ãƒªãƒ³ã‚¯
http://storm-project.net/
kanbayashi 2013/06/12
storm

distributed

hadoop

map reduce

stream

stream data

online
ãƒªãƒ³ã‚¯
XcalableMP: a directive-based language extension for scalable extension for scalable and performance-aware parallel programming programming
kanbayashi 2013/06/12
xcalablemp

distributed

openmp

slide
ãƒªãƒ³ã‚¯
Tera Termãƒžã‚¯ãƒæ´»ç”¨å…¥é–€ï¼ˆ1ï¼‰ï¼šå„ç¨®ãƒã‚°ã‚¤ãƒ³ã‚’è‡ªå‹•åŒ–ã™ã‚‹ | OSDN Magazine
Windowså‘ã‘ã®ã‚¿ãƒ¼ãƒŸãƒŠãƒ«ã‚¨ãƒŸãƒ¥ãƒ¬ãƒ¼ã‚¿ã€ŒTera Termã€(ãƒ†ãƒ©ã‚¿ãƒ¼ãƒ )ã«ã¯å¼·åŠ›ãªãƒžã‚¯ãƒæ©Ÿèƒ½ãŒæè¼‰ã•ã‚Œã¦ãŠã‚Šã€ã“ã‚Œã‚’åˆ©ç”¨ã™ã‚‹ã“ã¨ã§ãƒãƒƒãƒˆãƒ¯ãƒ¼ã‚¯çµŒç”±ã§ãƒžã‚·ãƒ³ã‚’è‡ªå‹•æ“ä½œã—ãŸã‚Šã€ç®¡ç†æƒ…å ±ã‚’å–å¾—ã™ã‚‹ã€ã¨ã„ã£ãŸã“ã¨ãŒå¯èƒ½ã ã€‚æœ¬è¨˜äº‹ã§ã¯ã„ãã¤ã‹ã®ã‚µãƒ³ãƒ—ãƒ«ã¨ã¨ã‚‚ã«Tera Termãƒžã‚¯ãƒã®åŸºæœ¬çš„ãªä½¿ã„æ–¹ã‚’ç´¹ä»‹ã™ã‚‹ã€‚ Tera TermãŠã‚ˆã³Tera Termãƒžã‚¯ãƒã®åŸºæœ¬ Tera Termã®ãƒžã‚¯ãƒæ©Ÿèƒ½ã‚’ç´¹ä»‹ã™ã‚‹å‰ã«ã€ç°¡å˜ã«Tera Termã«ã¤ã„ã¦ç´¹ä»‹ã‚’ã—ã¦ãŠã“ã†ã€‚Tera Termã¯ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ã§é–‹ç™ºã•ã‚Œã¦ã„ã‚‹Windowså‘ã‘ã®ã‚¿ãƒ¼ãƒŸãƒŠãƒ«ã‚¨ãƒŸãƒ¥ãƒ¬ãƒ¼ã‚¿ï¼ˆç«¯æœ«ã‚¨ãƒŸãƒ¥ãƒ¬ãƒ¼ã‚¿ï¼‰ã ã€‚ã‹ã¤ã¦ã¯Telnetã‚¯ãƒ©ã‚¤ã‚¢ãƒ³ãƒˆã‚„COMãƒãƒ¼ãƒˆçµŒç”±ã§ã®ã‚·ãƒªã‚¢ãƒ«æŽ¥ç¶šæ©Ÿèƒ½ã‚’å‚™ãˆãŸç„¡å„Ÿã®ã‚¿ãƒ¼ãƒŸãƒŠãƒ«ã‚¨ãƒŸãƒ¥ãƒ¬ãƒ¼ã‚¿ã¨ã—ã¦æœ‰åã ã£ãŸãŒã€1998å¹´ã«ã„ã£ãŸã‚“é–‹ç™ºãŒä¸æ–ã•ã‚ŒãŸå¾Œã«ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹åŒ–ã•ã‚Œã€ç¾åœ¨ã§ã¯SSHã‚¯ãƒ©ã‚¤ã‚¢ãƒ³ãƒˆæ©Ÿèƒ½ã‚„C
kanbayashi 2012/01/21
teraterm

ssh

distributed

remote

putty

client
ãƒªãƒ³ã‚¯
Apache ZooKeeper
Welcome to Apache ZooKeeperâ„¢ Apache ZooKeeper is an effort to develop and maintain an open-source server which enables highly reliable distributed coordination. What is ZooKeeper? ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another b
kanbayashi 2011/11/04
apache

process management

distributed

framework

synchronization

jubatus

pfi

cluster
ãƒªãƒ³ã‚¯
Jubatusã‚’å…¬é–‹ã—ã¾ã—ãŸ - Preferred Networks Research & Development
å…ˆæ—¥ã€NTTã¨å…±åŒç ”ç©¶é–‹ç™ºã—ãŸJubatusã‚’å…¬é–‹ã—ã¾ã—ãŸã€‚ OSSã§å…¬é–‹ã•ã‚Œã¦ã¾ã™ã®ã§ã€èˆˆå‘³ãŒã‚ã‚‹æ–¹ã¯ä½¿ã£ã¦ã¿ã‚‹ãªã‚Šã€ã‚½ãƒ¼ã‚¹ã‚³ãƒ¼ãƒ‰ã‚’çœºã‚ã‚‹ãªã‚Šã€ã§ãã‚Œã°ãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆã«å‚åŠ ã—ã¦ã„ãŸã ã‘ãŸã‚‰ã¨æ€ã„ã¾ã™ã€‚ Jubatusï¼ˆãƒ¦ãƒã‚¿ã‚¹ï¼‰ã¯ã€å¤§è¦æ¨¡åˆ†æ•£ä¸Šã§ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ã§æ©Ÿæ¢°å¦ç¿’ã‚’è¡Œã†ãŸã‚ã®ãƒ•ãƒ¬ãƒ¼ãƒ ãƒ¯ãƒ¼ã‚¯ã§ã™ã€‚ ã“ã®ãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆã¯å…ƒã€…ã€æ©Ÿæ¢°å¦ç¿’ã‚„ãƒ‡ãƒ¼ã‚¿è§£æžãŒå¥½ããªPFIã¨ã€ãƒãƒƒãƒˆãƒ¯ãƒ¼ã‚¯ã‚„ã‚·ã‚¹ãƒ†ãƒ é‹ç”¨å®Ÿç¸¾ãŒè±Šå¯Œã§æŠ€è¡“åŠ›ãŒã‚ã‚‹NTTæƒ…å ±æµé€šãƒ—ãƒ©ãƒƒãƒˆãƒ•ã‚©ãƒ¼ãƒ ç ”ç©¶æ‰€ã¨çµ„ã‚“ã§ä½•ã‹ã§ããªã„ã‹ã¨ã„ã†ã“ã¨ã§å§‹ã‚ã¾ã—ãŸã€‚ ã€Œå¤§è¦æ¨¡åˆ†æ•£ã€ï¼‹ã€Œãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ã€ï¼‹ã€Œæ·±ã„è§£æžã€ã¨ã„ã†ä¸‰ã¤ã®ç‰¹å¾´ã‚’æŒã£ãŸä¸Šã§ãƒ‡ãƒ¼ã‚¿ã‚’åˆ†æžã™ã‚‹ãŸã‚ã«ã©ã®ã‚ˆã†ãªã‚¢ãƒ¼ã‚ãƒ†ã‚¯ãƒãƒ£ãŒè€ƒãˆã‚‰ã‚Œã¦ã€ãã®ä¸Šã§ã©ã®ã‚ˆã†ãªè§£æžãŒã§ãã‚‹ã®ã‹ã¨ã„ã†ã®ã‚’æ—¥ã€…è©¦è¡ŒéŒ¯èª¤ã—ã¦ä½œã£ã¦ã„ã¾ã™ã€‚ ä»Šå›žã¯ãƒªãƒªãƒ¼ã‚¹ç¬¬ï¼‘æ®µã¨ã„ã†ã“ã¨ã§ã€æ‰‹æ³•ã¨ã—ã¦ã¯åˆ†é¡žï¼ˆæ•™å¸«æœ‰å¦ç¿’ã®å¤šã‚¯ãƒ©ã‚¹åˆ†é¡žï¼‰ã«çµžã£ã¦OSS
kanbayashi 2011/11/04
haddop

pfi

distributed

data mining

dwh

db

jubatus

machine learning

æ©Ÿæ¢°å¦ç¿’
ãƒªãƒ³ã‚¯
Google Groups
î€Content unavailable Click here to try again. If you've seen this page more than once, try switching accounts.
kanbayashi 2011/05/22
nagoya

å‹‰å¼·ä¼š

*event

distributed

hadoop
ãƒªãƒ³ã‚¯
Apache Hadoop MapReduce (3) â€” MapReduce ã‚’å¤šæ®µã§å‹•ã‹ã™ : MYH
MapReduce ã§å°‘ã—è¤‡é›‘ãªã“ã¨ã‚’ã—ã‚ˆã†ã¨æ€ã†ã¨ã€MapReduce ã‚’è¤‡æ•°å›žé©ç”¨ã—ãªã‘ã‚Œã°ãªã‚‰ãªã„ã“ã¨ãŒã‚ã‚Šã¾ã™ã€‚ã“ã‚Œã«ã¯ã„ãã¤ã‹ã®æ–¹æ³•ãŒã‚ã‚Šã¾ã™ã€‚ ã‚‚ã—ã€Map+ Reduce Map* (ï¼‘å›žä»¥ä¸Šã® Map, Reduce, ï¼å›žä»¥ä¸Šã® Map) ã¨ã„ã†å½¢ã§è¨˜è¿°ã™ã‚‹ã“ã¨ãŒå‡ºæ¥ã‚‹ãªã‚‰ã°ã€ChainMapper, ChainReducer ã¨ã„ã†ã‚¯ãƒ©ã‚¹ãŒç”¨æ„ã•ã‚Œã¦ãŠã‚Šã€ã“ã‚Œã‚’ Mapper, Reducer ã¨ã—ã¦åˆ©ç”¨ã™ã‚‹ã®ãŒç°¡å˜ã§ã™ã€‚ ã‚‚ã—ã€ã‚·ãƒ¼ã‚±ãƒ³ã‚·ãƒ£ãƒ«ã« MapReduce ã‚’é©ç”¨ã™ã‚‹ã®ã§ã‚ã‚Œã°ã€Job ã®çµ‚äº†ã‚’å¾…ã£ã¦ã‹ã‚‰æ¬¡ã® Job ã‚’æŠ•å…¥ã™ã‚Œã°æ¸ˆã¿ã¾ã™ã€‚ ChainMapper / ChainReducer ã“ã®ã‚¯ãƒ©ã‚¹ã¯æ—¢å˜ã® Mapper ã‚’ç°¡å˜ã«ã¤ãªã’ã‚‰ã‚Œã‚‹ã“ã¨ã‚’ç›®çš„ã¨ã—ã¦ã„ã¾ã™ã€‚ ä»®ã« ChainMapper, ChainReducer ã¨ã„ã†ã‚¯ãƒ©ã‚¹ãŒãŒãªã‹ã£ãŸã¨ã—ã¾
kanbayashi 2011/05/22
hadoop

mapreduce

distributed

amazon emr

elastic map reduce

å¤šæ®µ

chain
ãƒªãƒ³ã‚¯
Amazon Elastic Map Reduce (EMR) beyond basics -- Sujee Maniyam
Created_at :Jan 2011 Amazon Elastic Map Reduce (EMR) Beyond Basics The code for all this is at GitHub:Â https://github.com/sujee/amazon-emr-beyond-basics Environment I run these commands from an linux EC2 instance.Â It doesn't have to be a 'powerful' instance, as it doesn't do much work.Â So an M1.SMALL type is fine.Â The following needs to be installed ec2 tools We use these tools to launch
kanbayashi 2011/05/21
amazon

amazon aws

emr

amazon emr

elastic map reduce

synchronization

ruby

cloud

distributed
ãƒªãƒ³ã‚¯
GNU ParallelãŒã™ã”ã™ãŽã¦ç”Ÿãã‚‹ã®ãŒã¤ã‚‰ã„
çš†ã•ã‚“ä»Šæ—¥ã‚‚ãŸãã•ã‚“ã®ã‚µãƒ¼ãƒã‚’ç›¸æ‰‹ã«ã•ã‚Œã¦ã„ã‚‹ã“ã¨ã‹ã¨æ€ã„ã¾ã™ãŒã€ã„ãã¤ã‹ã®ã‚µãƒ¼ãƒã«ã‚¢ã‚¯ã‚»ã‚¹ã—ã¦ 1 ç§’é–“ã®çµ±è¨ˆæƒ…å ±(ä¾‹ãˆã°vmstat 1 2)ã‚’é›†ã‚ã¦ãƒ‘ãƒƒã¨è¡¨ç¤ºã—ãŸã„æ™‚ã£ã¦ã©ã†ã‚„ã£ã¦ã¾ã™ã‹ãï¼Ÿshell script ã‚’å¦ã³ã¯ã˜ã‚ãŸã°ã£ã‹ã‚Šã®åƒ•ã¯ã“ã‚“ãªæ„Ÿã˜ã§æ›¸ã„ã¦ã¾ã—ãŸã€‚ $ for i in host1 host2 host3; do ssh $i "vmstat 1 2 | tail -1"; done 0 0 0 329004 210836 14275360 0 0 0 2424 1410 1828 0 0 100 0 0 0 0 0 3716112 587704 25921684 0 0 0 488 1643 2026 0 0 100 0 0 1 0 0 555440 265560 14015548 0 0 0 4204 1534 2392 1 0 99 0 0 vmstatã¨
kanbayashi 2011/05/15
parallel

gnu parallel

*hackhack

distributed

svengali
ãƒªãƒ³ã‚¯
Amazon Elastic MapReduce â€“ madoverload
Amazon Web Servicesã«ã¯ã€Amazon Elastic MapReduce(EMR)ã¨ã„ã†ã‚µãƒ¼ãƒ“ã‚¹ã‚‚ã‚ã‚Šã€Hadoopã‚¸ãƒ§ãƒ–ã‚’ç°¡å˜ã«èµ°ã‚‰ã›ã‚‹ã“ã¨ãŒã§ãã¾ã™ã€‚Hadoopãƒˆãƒ¬ãƒ¼ãƒ‹ãƒ³ã‚°ã§ã¯ã€CDHã‚’ä½¿ã£ã¦æ¼”ç¿’ã‚’è¡Œãªã„ã¾ã—ãŸãŒã€ã‚·ãƒ³ã‚°ãƒ«ãƒŽãƒ¼ãƒ‰ã ã£ãŸãŸã‚åˆ†æ•£å‡¦ç†ã®é†é†å‘³ãŒãªã‹ãªã‹å‘³ã‚ãˆã¾ã›ã‚“ã€‚ å®¶ã«è¤‡æ•°ãƒŽãƒ¼ãƒ‰ã®ãƒžã‚·ãƒ³ãŒãªãã¦ã‚‚ç°¡å˜ã«è©¦ã™ã“ã¨ãŒã§ãã‚‹EMRã‚’è©¦ã—ã¦ã¿ã¾ã—ãŸã€‚ æº–å‚™ EMRã®åˆ©ç”¨å‰ã«ã„ãã¤ã‹ã®æº–å‚™ã‚’ã—ã¾ã™ã€‚è©³ç´°ã¯ã€Getting Started Guide ã‚’å‚ç…§ã—ã¦ãã ã•ã„ã€‚ EMRã®åˆ©ç”¨é–‹å§‹ EMRã‚’åˆ©ç”¨ã™ã‚‹å‰ã«ã€EMRã®åˆ©ç”¨ç”³è«‹ã‚’è¡Œã†å¿…è¦ãŒã‚ã‚Šã¾ã™ã€‚ã“ã‚Œã¯ã€ãƒžãƒãƒ¼ã‚¸ãƒ¡ãƒ³ãƒˆã‚³ãƒ³ã‚½ãƒ¼ãƒ«ã§Elastic MapReduceã‚¿ãƒ–ã‚’ã‚¯ãƒªãƒƒã‚¯ã—ãŸéš›ã«ã€åˆ©ç”¨ç”³è«‹æ¸ˆã¿ã§ãªã‘ã‚Œã°ç”³è«‹ãƒšãƒ¼ã‚¸ã¸ã®ãƒªãƒ³ã‚¯ãŒå‡ºã‚‹ã®ã§ã€ãã“ã‹ã‚‰ç”³è«‹ã™ã‚Œã°OKã§ã™ã€‚ EMRã‚³ãƒžãƒ³ãƒ‰ãƒ©ã‚¤ãƒ³ã‚¯ãƒ©ã‚¤ã‚¢ãƒ³ãƒˆã®ãƒ€ã‚¦ãƒ³
kanbayashi 2011/05/03
elastic map reduce

ruby

mapreduce

cui

command line

distributed
ãƒªãƒ³ã‚¯
Amazon Elastic MapReduceã«ä»Šã•ã‚‰å…¥é–€ã—ã¦ã¿ãŸ - nokunoã®æ—¥è¨˜
ã¨ã„ã†ã‚ã‘ã§å†¬ä¼‘ã¿æœ€å¾Œã®è‡ªç”±ç ”ç©¶ã¨ã—ã¦ã€Amazon Elastic MapReduceï¼ˆEMRï¼‰ã‚’ä½¿ã£ã¦ã¿ã¾ã—ãŸã€‚ä»Šãªã‚‰å…¬å¼ãƒšãƒ¼ã‚¸ã‚‚ã»ã¼æ—¥æœ¬èªžåŒ–ã—ã¦ã„ã¦æ¥½ãƒãƒ³ã§ã™ã€‚Amazon Web Services (æ—¥æœ¬èªž) Amazon Elastic MapReduceã¨ã¯Amazon EMRã¯ã€Amazonã®ã‚¤ãƒ³ãƒ•ãƒ©ä¸Šã§å‹•ä½œã™ã‚‹ä»®æƒ³ã‚µãƒ¼ãƒãƒ¼ã‚’ä½¿ã£ãŸHadoopã‚¯ãƒ©ã‚¹ã‚¿ã‚’æ™‚é–“å˜ä¾¡ã§è²¸ã—å‡ºã™ã‚µãƒ¼ãƒ“ã‚¹ã§ã™ã€‚å°‘ã€…ã‚ã‹ã‚Šã«ãã„ã®ã§ã€Amazon Web Service(AWS)ã®é–¢é€£ã™ã‚‹è£½å“ç¾¤ã«ã¤ã„ã¦æ•´ç†ã—ã¦ãŠãã¾ã™ã€‚EC2 (Elastic Compute Cloud) EC2ã¯ã€ä»®æƒ³ãƒžã‚·ãƒ³ã‚’æ™‚é–“å˜ä¾¡ã§è²¸ã—å‡ºã™ã‚µãƒ¼ãƒ“ã‚¹ã§ã™ã€‚ EMRã‚’ä½¿ã‚ãšã«ã€EC2ã«è‡ªå‰ã§Hadoopã‚’ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã—ã¦ä½¿ã†ã‚„ã‚Šæ–¹ã‚‚ã‚ã‚Šã¾ã™ï¼ˆEMRãŒå‡ºæ¥ã‚‹ä»¥å‰ã¯ãã‚Œã—ã‹ãªã‹ã£ãŸï¼‰ã€‚ EMRã‚’ä½¿ã†å ´åˆã§ã‚‚ã€ãƒãƒƒã‚¯ã‚°ãƒ©ã‚¦ãƒ³ãƒ‰ã§ã¯è‡ªå‹•çš„ã«
kanbayashi 2011/04/10
nokuno

elastic map reduce

mapreduce

aws

amazon ec2

ec2

distributed
ãƒªãƒ³ã‚¯
Hadoop Streaming
Hadoop streaming is a utility that comes with the Hadoop distribution. The utility allows you to create and run map/reduce jobs with any executable or script as the mapper and/or the reducer. For example: $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/hadoop-streaming.jar \ -input myInputDirs \ -output myOutputDir \ -mapper /bin/cat \ -reducer /bin/wc In the above example, both the mapper and the reduce
kanbayashi 2011/04/09
hadoop

mapreduce

amazon ec2

elastic map reduce

aws

distributed
ãƒªãƒ³ã‚¯
MapReduce/Amazon Elastic MapReduceã‚’ä½¿ã£ã¦ã¿ã‚‹ï¼ˆãƒãƒƒãƒç·¨ï¼‰ - ä¿ºã®åŸºåœ°
è¨å®šãƒ•ã‚¡ã‚¤ãƒ«ã‚’ä½œã‚‹ ä½œè€…ã®è¶£å‘³ãªã®ã‹ã€ã‚„ãŸã‚‰jsonã«ã“ã ã‚ã‚‹ã®ãŒã“ã®ãƒ—ãƒã‚°ãƒ©ãƒ ã®ã‚ˆã†ã§ã€è¨å®šãƒ•ã‚¡ã‚¤ãƒ«ã‚‚jsonã§æ›¸ãã¾ã™ãƒ»ãƒ»ãƒ»ãƒ»ãƒ»ãƒ»è¨å®šãƒ•ã‚¡ã‚¤ãƒ«ãªã‚“ã¦ç´ ç›´ã«Rubyã‚³ãƒ¼ãƒ‰ã§æ›¸ã‘ã°ã„ã„ã ã‚ã†ã«ã€å‹•çš„è¨€èªžãªã‚“ã ã‹ã‚‰ æ–‡å¥ã‚’è¨€ã‚ãšè¨å®šãƒ•ã‚¡ã‚¤ãƒ«ã‚’æ›¸ãã¾ã™è¨å®šãƒ•ã‚¡ã‚¤ãƒ«ã¯ credentials.json ã¨ã„ã†åå‰ã«ã—ã¦ãƒ„ãƒ¼ãƒ«ã®ãƒ•ã‚©ãƒ«ãƒ€ç›´ä¸‹ã«èµ·ãã¾ã™ã€‚ å†…å®¹ã¯ { "access-id": "HOGEHOGEHOGE", "private-key": "PIYOHOGEkaljdlf/akajlfkjali1jkdfj", "key-pair": "hoge", "key-pair-file": "D:\elastic-mapreduce-ruby\cert-HOGEHOGHEOHODIDKLdlk.pem", "region": "us-east-1" } ã¨ã„ã†æ„Ÿã˜ã«ãªã‚Šã¾ã™ã€‚æœ€åˆã®ãƒ¤ãƒ„ã¨
kanbayashi 2011/04/08
hadoop

AEM

maprecude

amazon elastic mapreduce

distributed
ãƒªãƒ³ã‚¯
koshigoe.jp
kanbayashi 2011/03/16
gnu parallel

parallel

ssh

shell script

distributed

data processing

GXP
ãƒªãƒ³ã‚¯
GNU parallel - ã“ã—ã”ã‡
http://w.koshigoe.jp/study/?%5Bsystem%5D+GNU+parallel+%BB%C8%CD%D1%CE%E3 æœ€è¿‘ã€GNU paralle ã¨ã„ã†ã‚‚ã®ã‚’çŸ¥ã£ãŸã®ã§è»½ããƒ¡ãƒ¢ã€‚ xargs ã‚„ cat | bash ã®ä»£ã‚ã‚Šã«ä½¿ãˆã‚‹ã‚‚ã® ä¸¦åˆ—ã«å‡¦ç†ã‚’å®Ÿè¡Œã•ã›ã‚‰ã‚Œã‚‹(æ¨™æº–ã ã¨å‡ºåŠ›ã¯å‡¦ç†å˜ä½ã«ã¾ã¨ã‚ã‚‰ã‚Œã‚‹) è¤‡æ•°ã®ãƒ›ã‚¹ãƒˆã«å‡¦ç†ã‚’å®Ÿè¡Œã•ã›ã‚‰ã‚Œã‚‹ ã¨ã„ã†ã‚ãŸã‚ŠãŒä»£è¡¨çš„ãªç‰¹å¾´ã§ã—ã‚‡ã†ã‹ã€‚ for ã‚„ while ã®ä»£ã‚ã‚Šã«ä½¿ã£ã¦ãƒ«ãƒ¼ãƒ—ã‚’æ›¸ã inotifywait ã¨é€£æºã—ã¦ãƒ‡ã‚£ãƒ¬ã‚¯ãƒˆãƒªã¸ã®ãƒ•ã‚¡ã‚¤ãƒ«è¿½åŠ ãªã©ã‚’ãƒˆãƒªã‚¬ã«å‡¦ç†ã‚’èµ·å‹•ã•ã›ã‚‹ tail -f ãªã©ã§ãƒ•ã‚¡ã‚¤ãƒ«å…¥åŠ›ã‚’ç¶™ç¶šèªã¿è¾¼ã¿ã—ã¦ã‚¸ãƒ§ãƒ–ã‚ãƒ¥ãƒ¼ã¨ã—ã¦åˆ©ç”¨ã™ã‚‹ ã¨ã„ã†ç‰¹å¾´çš„ãªç”¨é€”ã‚‚ã‚ã‚‹æ§˜ã§ã™ã€‚ find ã® + ã®æ§˜ã«ã€Œã¨ã‚Œã‚‹ã ã‘ã®å¼•æ•°ã‚’ã¨ã‚‹ã€äº‹ã‚‚ã§ãã‚‹(æœ€å¤§æ•°ã‚’æŒ‡å®šã™ã‚‹ã“ã¨ã‚‚ã§ãã‚‹) å‡ºåŠ›ã‚’å¼•æ•°ã®é †ç•ªé€šã‚Šã«ä¸¦ã¹ã‚‰ã‚Œ
kanbayashi 2011/03/16
gnu parallel

parallel

ssh

shell script

distributed

data processing

GXP
ãƒªãƒ³ã‚¯
1 2 3 4 æ¬¡ã®ãƒšãƒ¼ã‚¸