[B! Hoop] shiumachiã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

shiumachi id:shiumachi

Hoopã«é–¢ã™ã‚‹shiumachiã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (7)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

WebHDFSã¨HttpFsã«ã¤ã„ã¦ã®ç°¡å˜ãªãƒ™ãƒ³ãƒãƒžãƒ¼ã‚¯ - ãŸã”ã‚‚ã‚Šã™ãƒ¡ãƒ¢
å‰ã®ã‚¨ãƒ³ãƒˆãƒªã®ç¶šãã€‚å®Ÿç”¨ä¸Šã©ã†ãªã®ã£ã¦ã“ã¨ã§ã€ã¨ã‚Šã‚ãˆãšç°¡å˜ã«ãƒ™ãƒ³ãƒãƒžãƒ¼ã‚¯çš„ãªè² è·èµ°è¡Œã‚’ã—ã¦ã¿ãŸã€‚ å®Ÿè¡Œç’°å¢ƒã¯ä»¥ä¸‹ã®é€šã‚Šã€‚ãªãŠ HttpFs Server ã¯NameNodeä¸Šã«ç«‹ã¦ã¦ã‚ã‚‹ã€‚ CDH4b2 + HttpFs/WebHDFS NameNode x1 2CPU 8GB Memory DataNode x4 2CPU 8GB Memory 2TB HDD ãƒ™ãƒ³ãƒãƒžãƒ¼ã‚¯ã‚¹ã‚¯ãƒªãƒ—ãƒˆå®Ÿè¡ŒãƒŽãƒ¼ãƒ‰ã‚‚NameNodeã¨åŒã‚¹ãƒšãƒƒã‚¯ ã¾ãŸãƒ™ãƒ³ãƒãƒžãƒ¼ã‚¯ã«ä½¿ç”¨ã—ãŸã‚³ãƒ¼ãƒ‰ã¯ã“ã¡ã‚‰ã®Rubyã‚¹ã‚¯ãƒªãƒ—ãƒˆã§ã€ruby 1.9.3 ã§å‹•ã‹ã—ãŸã€‚ç°¡å˜ã«å‡¦ç†å†…å®¹ã‚’ã¾ã¨ã‚ã‚‹ã¨ä»¥ä¸‹ã®ã‚ˆã†ãªæ„Ÿã˜ã€‚ ç‰¹å®šã®1ãƒ•ã‚¡ã‚¤ãƒ«ã«å¯¾ã—ã¦appendã‚’ç¹°ã‚Šè¿”ã™ appendã®ã‚µã‚¤ã‚ºã¯1å›žã‚ãŸã‚Š10MB æ™‚é–“ã‚ãŸã‚Šã§appendã«æˆåŠŸã—ãŸå›žæ•°ã‚’è¨˜éŒ² ä¸Šè¨˜å‡¦ç†ã‚’è¡Œã†ãƒ—ãƒã‚»ã‚¹ã‚’1ã€œ3ä¸¦åˆ—ã§1æ™‚é–“èµ°è¡Œ HttpFsã€WebHDFS ã«ã¤ã„ã¦ã
shiumachi 2012/06/07
webhdfs

Hoop

Hadoop
ãƒªãƒ³ã‚¯
CDH4b2ã«ãŠã‘ã‚‹WebHDFSã¨HttpFsã«ã¤ã„ã¦ - ãŸã”ã‚‚ã‚Šã™ãƒ¡ãƒ¢
CDH4b2ã§WebHDFSã¨HttpFsã«ã¤ã„ã¦ã„ã‚ã„ã‚è©¦ã—ã¦ã„ã‚‹ã®ã§ã€åˆ†ã‹ã£ã¦ã„ã‚‹å†…å®¹ã‚’ã¾ã¨ã‚ã¦ã¿ã‚‹ã€‚ãªãŠæ³¨æ„ç‚¹ã ãŒã€å„ã€…ä»¥ä¸‹ã®ã‚ˆã†ãªçŠ¶æ³ã§ã‚ã‚‹ã“ã¨ã«æ³¨æ„ã€‚ WebHDFS ãŸã¶ã‚“Hadoop 1.0ãƒ™ãƒ¼ã‚¹ HttpFs ãŸã¶ã‚“Hadoop 0.23ãƒ™ãƒ¼ã‚¹(ã‚ã‚‹ã„ã¯ 2.0 alpha ãƒ™ãƒ¼ã‚¹) ã§Hoopã¨ã¯åˆ¥ç‰© HttpFsã¯ã‚³ãƒ¼ãƒ‰ã¨ã—ã¦ã¯Clouderaã‚ªãƒªã‚¸ãƒŠãƒ«ã®Hoopã®ã‚‚ã®ãŒãƒ™ãƒ¼ã‚¹ã«ãªã£ã¦ã„ã‚‹ã‚“ã ã‚ã†ã—ã€ã‚¢ãƒ¼ã‚ãƒ†ã‚¯ãƒãƒ£è‡ªä½“ã‚‚åŒã˜(è©³ã—ãã¯ä»¥å‰ã®ã‚¨ãƒ³ãƒˆãƒªã‚’å‚ç…§ã®ã“ã¨)ã ã‘ã©ã€ç´°ã‹ã„ã¨ã“ã‚ãŒã‚ã¡ã“ã¡ç•°ãªã£ã¦ã„ã‚‹ã®ã§æ³¨æ„ãŒå¿…è¦ã€‚ ã¾ãŸCDH4ãƒ™ãƒ¼ã‚¹ãªã®ã§CDH3ãƒ™ãƒ¼ã‚¹ã®ã‚‚ã®ã¨ã¯ç‰¹ã«æ€§èƒ½ç‰¹æ€§ãŒç•°ãªã‚‹å¯èƒ½æ€§ãŒé«˜ã„ã€ãŒã€æ€§èƒ½ç‰¹æ€§ã«ã¤ã„ã¦ã¯ã¾ãŸåˆ¥ã«ã€‚ HTTP API WebHDFSã®APIã¯å¹´åˆã«ç¢ºèªã—ãŸã¨ãã‹ã‚‰å¤‰ã‚ã£ã¦ã„ãªã„ã€‚ HttpFsã®HTTP APIã¯Hoopã¨ã—ã¦Cloud
shiumachi 2012/06/07
Hadoop

Hoop

webhdfs
ãƒªãƒ³ã‚¯
Hoop(httpfs)ã¨webhdfsã®é•ã„ - ãŸã”ã‚‚ã‚Šã™ãƒ¡ãƒ¢
Hadoop 1.0.0ãŒãƒªãƒªãƒ¼ã‚¹ã•ã‚ŒãŸã€‚ã¾ã‚ä¸èº«ã®ã»ã¨ã‚“ã©ã¯ãŸã ã® 0.20.x å®‰å®šæ¿ãƒªãƒªãƒ¼ã‚¹ãªã®ã§ç‰¹åˆ¥ã«è¨€ã†ã“ã¨ã¯ãªã„ã‚“ã ã‘ã©ã€‚è©³ã—ãã¯ä»¥ä¸‹ã®blogã‚’èªã‚€ã®ãŒã‚ˆã‚ã—ã„ã€‚ hadoopã®ãƒãƒ¼ã‚¸ãƒ§ãƒ³è¡¨è¨˜ã«ã¤ã„ã¦ - ç§‘å¦ã¨éžç§‘å¦ã®è¿·å®® ãŸã ã—ã²ã¨ã¤ã ã‘ã³ã£ãã‚Šã—ãŸã®ã¯ã€webhdfsãªã‚‹æ©Ÿèƒ½ãŒå…¥ã£ã¦ããŸã“ã¨ã€‚(ã“ã®blogã§ã‚ˆãè©±é¡Œã«ã—ã¦ã„ã‚‹)Hoopã¨ä¸¦ã‚“ã§ãã‚“ãªã‚ˆã†ãªã‚‚ã®ãŒã‚ã‚‹ã“ã¨è‡ªä½“ã¯çŸ¥ã£ã¦ãŸã‘ã©ã‚ã‚“ã¾ã‚Šèˆˆå‘³ãªã‹ã£ãŸã®ã ãŒã€Apache Hadoopã®ãƒ‘ãƒƒã‚±ãƒ¼ã‚¸ã«(Hoopã‚ˆã‚Šå…ˆã«)å…¥ã£ãŸã¨ãªã‚‹ã¨ã¡ã‚‡ã£ã¨æ³¨ç›®ã›ã–ã‚‹ã‚’ãˆãªã„ã€‚ ãŒã€httpfs(Hoop)ã¨webhdfsã˜ã‚ƒåå‰ã‚‚ä¼¼ã¦ã¦è¶…ã¾ãŽã‚‰ã‚ã—ã„ã€‚ã„ã£ãŸã„ä½•ãŒãªã‚“ãªã®ã€‚ ãªãŠè‡ªåˆ†ã¯WebHDFSã¯APIãƒªãƒ•ã‚¡ãƒ¬ãƒ³ã‚¹ã‚’èªã‚“ã ã ã‘ã§ã€å®Ÿéš›ã«ã¯ã‚«ã‚±ãƒ©ã‚‚è§¦ã£ã¦ã„ãªã„ã€‚ãã®çŠ¶æ³ã§ã®ç†è§£ã«ã‚ˆã‚‹å†…å®¹ãªã®ã§ã€æ³¨æ„ã—ã¦èªã‚“ã§ãã ã•ã„ã€‚ å…ˆã«çµè«–
shiumachi 2012/01/02
è¨è¨ˆæ€æƒ³ã®é•ã„ãŒã‚ˆãã‚ã‹ã‚‹ãªã‚ã€‚ã¨ã“ã‚ã§ãƒžã‚¸ã§è‹±èªžã§ãƒ–ãƒã‚°æ›¸ã„ã¦ã»ã—ã„ã‚“ã§ã™ãŒ

Hadoop

Hoop
ãƒªãƒ³ã‚¯
HDFSã®ãƒ•ã‚¡ã‚¤ãƒ«ã‚ªãƒšãƒ¬ãƒ¼ã‚·ãƒ§ãƒ³å„ç¨® #hadoopAC11jp - ãŸã”ã‚‚ã‚Šã™ãƒ¡ãƒ¢
ã“ã®è¨˜äº‹ã¯ hadoopã‚¢ãƒ‰ãƒ™ãƒ³ãƒˆã‚«ãƒ¬ãƒ³ãƒ€ãƒ¼ ã®14æ—¥ç›®ã®è¨˜äº‹ã§ã™ã€‚ ã¿ãªã•ã‚“HDFSä½¿ã£ã¦ã¾ã™ã‹ã€‚ä½¿ã£ã¦ã¾ã™ã‚ˆãã€‚æœ€è¿‘ã¯gluster fsãªã©ã®é¸æŠžè‚¢ã‚‚å‡ºã¦ãã¦ã„ã¾ã™ãŒã€ã¾ã ç¬¬ä¸€ã®é¸æŠžè‚¢ã¯HDFSã¨ã„ã†äººãŒã»ã¨ã‚“ã©ã ã¨æ€ã„ã¾ã™ã€‚ ã§ã€HDFSã®ãƒ•ã‚¡ã‚¤ãƒ«æ“ä½œã‚’ã©ã†ã—ã‚ˆã†ã‹æ‚©ã¿ã¾ã™ã‚ˆãã€‚ã‚ã‚“ã©ãã•ã„ã§ã™ã€‚ã„ãã¤ã‹ã‚ã‚‹ã®ã§æ¯”è¼ƒã—ã¦ã¿ã¾ã—ã‚‡ã†ã€‚ hadoop fsã‚³ãƒžãƒ³ãƒ‰ ã¿ã‚“ãªå¤§å¥½ãhadoop fsã€‚æ—¥ã«100å›žãã‚‰ã„ã¯å©ãã¾ã™ãã€‚åŸºæœ¬çš„ã«ã¯ã¿ã‚“ãªã“ã‚Œã‚’ä½¿ã†ã§ã—ã‚‡ã†ã‹ã€‚ãŸã ã—ä»¥ä¸‹ã„ãã¤ã‹ã®ã€ŒãŸã‚‹ã„ãªãƒ¼ã€ã¨ã„ã†ç‚¹ãŒã‚ã‚Šã¾ã™ã€‚ HadoopãŒã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ãƒ»è¨å®šã•ã‚Œã¦ã„ãªã„ã¨å‹•ã‹ãªã„ ã‚ãŸã‚Šã¾ãˆãªã‚“ã§ã™ã‘ã©æ„å¤–ã«ã‚ã‚“ã©ãã•ã„ã§ã™ãã€‚ Hadoopã®å‡¦ç†å¯¾è±¡ã¨ãªã‚‹ãƒ•ã‚¡ã‚¤ãƒ«ã¯ã‚‚ã¡ã‚ã‚“Hadoopã‚¯ãƒ©ã‚¹ã‚¿ã€Œä»¥å¤–ã€ã®ã‚µãƒ¼ãƒã‹ã‚‰å‡ºã¦ãã‚‹ã‚ã‘ã§ã€ãã“ã‹ã‚‰ã‚‚ç›´æŽ¥ãƒ•ã‚¡ã‚¤ãƒ«ã‚’HDFSã«çªã£è¾¼ã¿ãŸã„ãªãƒ¼ã¨ã„ã†ã®ã¯å‰²
shiumachi 2011/12/14
"æœ€è¿‘ Hoop ã¨ã„ã†ãƒ„ãƒ¼ãƒ«ãŒå‡ºã¦ãã¾ã—ãŸã€‚Clouderaã‹ã‚‰å…¬é–‹ã•ã‚ŒãŸã‚‚ã®ã§ã™ãŒã€Hadoopæœ¬ä½“ã¸ã®ãƒžãƒ¼ã‚¸ãŒé€²ã‚ã‚‰ã‚ŒãŸã‚ˆã†ã§ã€0.23.1(ãŠã‚ˆã³0.24)ã®ãƒ„ãƒªãƒ¼ã«ãŠã„ã¦ãƒžãƒ¼ã‚¸ã•ã‚ŒãŸã‚ˆã†ã§ã™"

Hadoop

Hoop
ãƒªãƒ³ã‚¯
Hoopã®æ€§èƒ½ã‚’ç¢ºèªã—ã¦ã¿ãŸã‚‰ã‚‚ã†libhdfsã¨ã‹ã‚ªãƒ¯ã‚³ãƒ³ã§Hoopä½¿ãˆã£ã¦çµæžœã«ãªã£ãŸ - ãŸã”ã‚‚ã‚Šã™ãƒ¡ãƒ¢
å‰ã«æ›¸ã„ãŸ ã‚¨ãƒ³ãƒˆãƒª ã®é€šã‚ŠHoopãŒæœ‰æœ›ãªæ„Ÿã˜ã ã£ãŸã‚“ã ã‘ã©ã©ã®ãã‚‰ã„ã®æ€§èƒ½ãŒå‡ºã‚‹ã®ã‹è¦‹ã¦ã¿ãªã„ã“ã¨ã«ã¯æœ¬ç•ªæŠ•å…¥ã—ã¦æ€§èƒ½å‡ºã¾ã›ã‚“ã§ã—ãŸä¹™ã€ã¨ã„ã†ã“ã¨ã«ãªã‚Šã‹ããªã„ã€‚ã®ã§è¦‹ã¦ã¿ãŸã€‚ ãªãŠæ¤œè¨¼ã«é–¢ä¿‚ã™ã‚‹ç’°å¢ƒã¨ã—ã¦ã¯ä»¥ä¸‹ã®é€šã‚Šã€‚ã¡ãªã¿ã«å‰ã¯CDH3u1ã§è©¦ã—ã¦ãŸã‘ã©ã€ä»Šå›žã¯CDH3u2 (JDK6u29) on CentOS5ã€‚ãƒ¡ãƒ¢ãƒªãŒå•é¡Œã«ãªã‚‹ã“ã¨ã¯å…¨ããªã„ã®ã§å…¨ã¦çœç•¥ã€‚ ãƒ™ãƒ³ãƒç”¨ã‚µãƒ¼ãƒ Intel Xeon 4Core HT 2.4GHz ãƒ‡ãƒ¼ã‚¿ä¸ç¶™ã‚µãƒ¼ãƒ (deliver) Intel Xeon 4Core HT 2.4GHz Hadoop NameNode (+JobTracker) (namenode) Intel Xeon 1Core HT 3.16GHz Hadoop DataNode + TaskTracker x9 Intel Xeon 1Core HT 3.16GHz Ho
shiumachi 2011/11/02
ç´ æ™´ã‚‰ã—ã„

Hoop

Hadoop
ãƒªãƒ³ã‚¯
Hoop (HDFS over HTTP) ã‚’è©¦ã—ã¦ã¿ãŸ - ãŸã”ã‚‚ã‚Šã™ãƒ¡ãƒ¢
ãƒãƒƒãƒå‡¦ç†ã«ã‹ã‘ã‚‹å¯¾è±¡ãƒ•ã‚¡ã‚¤ãƒ«ã®æŠ½å‡ºã«ã¯ä¸èº«ã‚’è¦‹ã‚‹å¿…è¦ãŒã‚ã£ã¦ã€headã‚„tailã§ã„ã„ã‚“ã ã‘ã©hadoop fsã‚³ãƒžãƒ³ãƒ‰ã§ã¯ãã†ã„ã†ã‚ªãƒšãƒ¬ãƒ¼ã‚·ãƒ§ãƒ³ãŒã§ããªã„*1ã®ã§ FUSE hdfs ã§Linuxã«HDFSã‚’mountã—ã¦ã‚„ã£ã¦ã„ã‚‹ã€‚ ãŒã€ã“ã‚ŒãŒã¾ãŸãƒ“ãƒ«ãƒ‰ã™ã‚‹ã®ã«ã²ã¨è‹¦åŠ´ã ã£ãŸã‚Šmountå…ƒã¨å…ˆãŒå¯†çµåˆã«ãªã£ã¡ã‚ƒã£ãŸã‚Šé…ã‹ã£ãŸã‚Šã„ã¤ã¾ã§ãƒ¡ãƒ³ãƒ†ã•ã‚Œã‚‹ã‚‚ã‚“ã‹ã¨æ€ã‚ã‚ŒãŸã‚Šã™ã‚‹ã—å¤–éƒ¨ã‚³ãƒžãƒ³ãƒ‰ã‚’èµ·å‹•ã—ã¦çµæžœã‚’èªã‚“ã§ã‚´ãƒ‹ãƒ§ã‚´ãƒ‹ãƒ§ã—ãŸã‚Šã™ã‚‹ã®ãŒé¢å€’ãªã®ã§ã©ã†ã«ã‹ã—ãŸã„ãªãƒ¼ã¨æ€ã£ã¦ã„ãŸã‚‰ã€Clouderaã‹ã‚‰ Hoop ãªã‚‹ãƒ—ãƒãƒ€ã‚¯ãƒˆãŒå°‘ã—å‰ã«ãƒªãƒªãƒ¼ã‚¹ã•ã‚ŒãŸã€‚ Hoop - Hoop, Hadoop HDFS over HTTP - Documentation Sets 0.1.0-SNAPSHOT HTTP REST APIã‚’çµŒç”±ã—ã¦HDFSã®å†…å®¹ã«ã‚¢ã‚¯ã‚»ã‚¹ã§ãã‚‹ã—ã€ã‚ªãƒšãƒ¬ãƒ¼ã‚·ãƒ§ãƒ³ã‚‚ã§ãã‚‹ã€‚GETã®
shiumachi 2011/09/07
Hadoop

Hoop
ãƒªãƒ³ã‚¯
Cloudera Blog
In an era where artificial intelligence (AI) is reshaping enterprises across the globeâ€”be it in healthcare, finance, or manufacturingâ€”itâ€™s hard to overstate the transf ormation that AI has had on businesses, regardless of industry or size. At Cloudera, we recognize the urgent need for bold steps to harness this potential and dramatically accelerate the time to [â€¦] Read blog post
shiumachi 2011/07/21
HDFS Proxyã‚’å†è¨è¨ˆã—ãŸClouderaã®æ–°ã—ã„OSSã€Hoopã®ãƒªãƒªãƒ¼ã‚¹ã§ã™ï¼REST API ã®æä¾›ã€ç•°ãªã‚‹ãƒãƒ¼ã‚¸ãƒ§ãƒ³ã® Hadoop é–“ã§ã®ãƒ‡ãƒ¼ã‚¿è»¢é€ã€ãƒ•ã‚¡ã‚¤ã‚¢ã‚¦ã‚©ãƒ¼ãƒ«è¶Šã—ã®HDFSã‚¢ã‚¯ã‚»ã‚¹ãªã©ã€æ©Ÿèƒ½ã¯æº€è¼‰ã§ã™ï¼ã‚‚ã¡ã‚ã‚“ã‚½ãƒ¼ã‚¹ã¯å…¬é–‹

hadoop

Cloudera

Hoop
ãƒªãƒ³ã‚¯
1