[B! hdfs] aoe-tkã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

aoe-tk id:aoe-tk

hdfsã«é–¢ã™ã‚‹aoe-tkã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (6)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

Cloudera Impala ãŒãƒªãƒªãƒ¼ã‚¹ã•ã‚Œã¾ã—ãŸ - ç§‘å¦ã¨éžç§‘å¦ã®è¿·å®®
(2012/10/25 15:48 è¿½è¨˜) Cloudera å…¬å¼ãƒ–ãƒã‚°ã§ Impala ã«ã¤ã„ã¦ã®ç´¹ä»‹è¨˜äº‹ã‚’æŽ²è¼‰ã—ã¾ã—ãŸã€‚ã“ã®ãƒ–ãƒã‚°è¨˜äº‹ã®å®Œå…¨ä¸Šä½äº’æ›ãªã®ã§ãã¡ã‚‰ã®è¨˜äº‹ã‚’ã”å‚ç…§ãã ã•ã„ã€‚ Cloudera Impalaï¼šApache Hadoopã§å®Ÿç¾ã™ã‚‹ã€çœŸã®ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ã‚¯ã‚¨ãƒª | Hadoopã¨ãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿ã‚½ãƒªãƒ¥ãƒ¼ã‚·ãƒ§ãƒ³ã®ãƒªãƒ¼ãƒ‡ã‚£ãƒ³ã‚°ã‚«ãƒ³ãƒ‘ãƒ‹ãƒ¼ | Cloudera Japan Cloudera ã‹ã‚‰ã€ãƒ‡ãƒ¼ã‚¿ã‚µã‚¤ã‚¨ãƒ³ãƒ†ã‚£ã‚¹ãƒˆã®ãŸã‚ã®ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ã‚¯ã‚¨ãƒªã‚¨ãƒ³ã‚¸ãƒ³ã€ŒImpalaã€ãŒãƒªãƒªãƒ¼ã‚¹ã•ã‚Œã¾ã—ãŸã€‚Hive ã¨å®Œå…¨äº’æ›ã®ã‚¯ã‚¨ãƒªè¨€èªžã§ã€Hive ã‚ˆã‚Š10å€ä»¥ä¸Šé€Ÿãã‚¯ã‚¨ãƒªã‚’å‡¦ç†ã§ãã¾ã™ã€‚ æ¦‚è¦åŠã³ãƒ€ã‚¦ãƒ³ãƒãƒ¼ãƒ‰ã¯ã“ã¡ã‚‰ã‹ã‚‰! http://www.cloudera.com/content/cloudera/en/products/cloudera-enterprise-core/clouder
aoe-tk 2012/10/25
HDFSã€HBaseã«æ ¼ç´ã•ã‚ŒãŸãƒ‡ãƒ¼ã‚¿ã«å¯¾ã—ã¦MapReduceã‚’ä½¿ã‚ãšé«˜é€Ÿã«å‡¦ç†ã™ã‚‹ã‚¯ã‚¨ãƒªã‚¨ãƒ³ã‚¸ãƒ³ã§ã™ã‹ã€‚

cloudera

hdfs

hbase
ãƒªãƒ³ã‚¯
HDFSã®ãƒ•ã‚¡ã‚¤ãƒ«æ›¸ãè¾¼ã¿éƒ¨åˆ†ã®ã‚½ãƒ¼ã‚¹ã‚’èªã‚“ã§ã¿ãŸ Part2 - wyukawa's diary
å‰å›žã¯ã“ã¡ã‚‰ HDFSã®ãƒ•ã‚¡ã‚¤ãƒ«æ›¸ãè¾¼ã¿éƒ¨åˆ†ã®ã‚½ãƒ¼ã‚¹ã‚’èªã‚“ã§ã¿ãŸ - wyukawaâ€™s blog ResponseProcessorã‚¹ãƒ¬ãƒƒãƒ‰ã‚’startã™ã‚‹å‰ã«nextBlockOutputStreamãƒ¡ã‚½ãƒƒãƒ‰ã‚’å‘¼ã³å‡ºã—ãƒ‡ãƒ¼ã‚¿ãƒŽãƒ¼ãƒ‰ã¨æŽ¥ç¶šã—ã¾ã™ã€‚ // get new block from namenode. if (blockStream == null) { LOG.debug("Allocating new block"); nodes = nextBlockOutputStream(src); this.setName("DataStreamer for file " + src + " block " + block); response = new ResponseProcessor(nodes); response.start(); } nextBlockOutputStr
aoe-tk 2012/02/23
æ›¸ãè¾¼ã¿ä¸ã«dfs.client.block.write.retriesä»¥ä¸Šã®å°æ•°ã®DNãŒä¸€åº¦ã«è½ã¡ãŸã‚Šã—ãŸã‚‰æ›¸ãè¾¼ã¿ãŒå¤±æ•—ã™ã‚‹å¯èƒ½æ€§ãŒã‚ã‚‹ã¨ã„ã†ã“ã¨ã§ã™ã‹ãã€‚

hadoop

hdfs
ãƒªãƒ³ã‚¯
Gluster
Gluster is a free and open source software scala ble network filesystem.
aoe-tk 2012/01/19
åˆ†æ•£å‡¦ç†æŠ€è¡“

hdfs
ãƒªãƒ³ã‚¯
Hdfsã‚½ãƒ¼ã‚¹ã‚³ãƒ¼ãƒ‰ãƒªãƒ¼ãƒ‡ã‚£ãƒ³ã‚°ç¬¬ï¼’å›ž
5. Append/Hflush/Read DesignHDFS-265ã® appendDesign3.pdfAppendã‚’å®Ÿè£…ã™ã‚‹ãŸã‚ã«ã®ãƒ‡ã‚¶ã‚¤ãƒ³è¨è¨ˆHDFSã®æ›¸ãè¾¼ã¿æ™‚ã€èªã¿è¾¼ã¿æ™‚ã€ã‚¨ãƒ©ãƒ¼å‡¦ç†ã®æ™‚ã®æŒ¯ã‚‹èˆžã„ãŒç´°ã‹ãæ›¸ã‹ã‚Œã¦ã„ã‚‹Hadoop 0.21ã“ã‚Œã‚’èªã‚€ã“ã¨ã§ã€HDFSã®ä¸€è²«æ€§ãƒ¢ãƒ‡ãƒ«ã€æ•…éšœæ™‚ã®æŒ¯ã‚‹èˆžã„ãŒå…·ä½“çš„ã«ã‚ã‹ã‚‹ä»Šå›žã¯2ç« ã‚’ãƒ¡ã‚¤ãƒ³ã« 6. ãƒ–ãƒãƒƒã‚¯ã®çŠ¶æ…‹ç®¡ç†DataNodeã§ã®ãƒ–ãƒãƒƒã‚¯ã‚’ãƒ¬ãƒ—ãƒªã‚«ã¨å‘¼ã³ã€NameNodeã§ã®ãƒ–ãƒãƒƒã‚¯ã¯ãƒ–ãƒãƒƒã‚¯ã¨å‘¼ã¶NameNode{Finename, numReplicas, block-ids}{block-id, datanodes}ãƒ–ãƒãƒƒã‚¯ã®çŠ¶æ…‹DataNodesblock-idblock-idblock-idãƒ¬ãƒ—ãƒªã‚«ã®çŠ¶æ…‹ãƒ¬ãƒ—ãƒªã‚«ã®çŠ¶æ…‹ãƒ¬ãƒ—ãƒªã‚«ã®çŠ¶æ…‹
aoe-tk 2010/12/18
hadoop

hdfs
ãƒªãƒ³ã‚¯
Hdfsã‚½ãƒ¼ã‚¹ã‚³ãƒ¼ãƒ‰ãƒªãƒ¼ãƒ‡ã‚£ãƒ³ã‚°ç¬¬ä¸€å›ž
4. HDFSã‚½ãƒ¼ã‚¹ã‚³ãƒ¼ãƒ‰æ¦‚è¦cdsrc/hdfs/org/apache/hadoop/hdfs/wc -l *.java */*.java */*/*.java36690 totalhdfsDistributedFileSystem.javaprotocoltoolsserverdatanodenamenode 5. ä»Šå›žã®æ¦‚è¦Hadoopã® FileSystemå‘¨ã‚Šã®ã‚¹ã‚¿ãƒƒã‚¯0.21ä»¥é™ã§å°Žå…¥ã•ã‚Œã‚‹FileContextã«ã¤ã„ã¦FileSystem API ã®ä½¿ç”¨æ³•ã‹ã‚‰ã‚½ãƒ¼ã‚¹ã‚’è¦‹ã‚‹FileSystemFSDataInputStreamFSDataOutputStreamHDFSå´ã®å®Ÿè£…ã‚’è¦‹ã‚‹DistributedFileSystem
aoe-tk 2010/12/18
hadoop

hdfs
ãƒªãƒ³ã‚¯
HDFSã¯ãƒ•ã‚¡ã‚¤ãƒ«ã‚µãƒ¼ãƒãƒ¼ã«ä½¿ã†ã‚‚ã®ã˜ã‚ƒãªã„ã‚ˆ
Hadoopï¼ˆ0.22ä»¥å‰ã‚„1.0.0ï¼‰ã®HDFSã¯NameNodeãŒSPOFï¼ˆå˜ä¸€éšœå®³ç‚¹ï¼‰ãªã®ã§ã€ãƒ•ã‚¡ã‚¤ãƒ«ã‚µãƒ¼ãƒãƒ¼ã‚„ãƒ‡ãƒ¼ã‚¿ã®â€œãƒãƒƒã‚¯ã‚¢ãƒƒãƒ—â€ã«ã¯å‘ã‹ãªã„ã¨ã„ã†è©±ã§ã™ã€‚ HDFSä¸Šã«æ§‹ç¯‰ã•ã‚Œã¦ã„ã‚‹HBaseã®å ´åˆã€ï¼ˆHBaseè‡ªèº«ã«ã¯SPOFã¯ç„¡ã„ã‘ã‚Œã©ã‚‚ï¼‰HDFSã«SPOFãŒã‚ã‚‹ã®ã§ã€NameNodeã¯å†—é•·åŒ–ã—ãŸæ–¹ãŒè‰¯ã„ã‚ˆã†ã§ã™ã€‚ å†—é•·åŒ–ã«ã¯ä»¥ä¸‹ã®ã‚ˆã†ãªä»•çµ„ã¿ãŒè€ƒãˆã‚‰ã‚Œã¦ã„ã‚‹ã‚ˆã†ã§ã™ã€‚ ç¶šãã‚’èªã‚€
aoe-tk 2010/12/11
hadoop

hdfs

hbase
ãƒªãƒ³ã‚¯
1