[B! HDFS][hadoop] oza_x86ã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

oza_x86 id:oza_x86

HDFSã¨hadoopã«é–¢ã™ã‚‹oza_x86ã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (8)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

RCFileï¼ŒParquetï¼ŒORCFile
ã“ã®2ãƒ¶æœˆã§ï¼ŒCloudera/Twitterï¼ŒHortonworks ã‹ã‚‰ãã‚Œãžã‚Œåˆ¥ã®åˆ—æŒ‡å‘ãƒ•ã‚¡ã‚¤ãƒ«ãƒ•ã‚©ãƒ¼ãƒžãƒƒãƒˆãŒå…¬é–‹ã•ã‚Œã¾ã—ãŸï¼ŽParquetÂ ã¨ ORCFile ã§ã™ï¼Ž ã“ã®è¨˜äº‹ã§ã¯ï¼Œã¾ãš RCFile ã®å¾©ç¿’ã‚’ã—ã¦ï¼Œãã®å¾Œ Parquet ã¨ ORCFile ãã‚Œãžã‚Œã®å…±é€šç‚¹ã¨é•ã„ã‚’ãŠãŠã¾ã‹ã«è¦‹ã¦ã„ã“ã†ã¨æ€ã„ã¾ã™ï¼Žã‚³ãƒ¼ãƒ‰ãƒ¬ãƒ™ãƒ«ã®è©³ç´°ãªé•ã„ã«ã¤ã„ã¦ã¯ï¼Œæ¬¡å›žä»¥é™ã§è¦‹ã¦ã„ãã¾ã™ï¼Ž RCFile ã®å¾©ç¿’ RCFile ã¯ã€€Record Columnar File ã®ç•¥ã§ï¼ŒHive ã‹ã‚‰åˆ©ç”¨ã§ãã‚‹ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ãƒ•ã‚©ãƒ¼ãƒžãƒƒãƒˆã§ã™ï¼Žç‰¹ã«ï¼ŒHDFS ã‚„ S3 ã¨ã„ã£ãŸåˆ†æ•£ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ä¸Šã§ãƒ‘ãƒ•ã‚©ãƒ¼ãƒžãƒ³ã‚¹ãŒã§ã‚‹ã‚ˆã†ã«è¨è¨ˆã•ã‚Œã¦ã„ã¾ã™ï¼Ž HDFS/S3 ã¨ã„ã£ãŸã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ã§ã¯ï¼ŒåŸºæœ¬çš„ã«ãƒ‡ãƒ¼ã‚¿ã‚’è¨ˆç®—æ©Ÿé–“ã§åŒã˜è² è·ã«ãªã‚‹ã‚ˆã†ã«ãƒ‡ãƒ¼ã‚¿ã‚’åˆ†æ•£é…ç½®ã—ã¾ã™ï¼Žã“ã®ãŸã‚ï¼Œå¾“æ¥ã®åˆ—æŒ‡å‘ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ãƒ•ã‚©ãƒ¼ãƒžãƒƒãƒˆã®ã‚ˆã†ã«é©å½“ã«åˆ—æ¯Žã«
oza_x86 2013/03/18
HDFS

hadoop

Columnar Storage
ãƒªãƒ³ã‚¯
How to convert non-HA NameNode to QJM HA on CDH4 | å¤–é“çˆ¶ã®åŒ
CDH3ã‹ã‚‰å§‹ã‚ã¦CDH4.1ã¾ã§ã‚¢ãƒƒãƒ—ã‚°ãƒ¬ãƒ¼ãƒ‰ã—ã¦åˆ©ç”¨ã—ç¶šã‘ã¦ã„ã¾ã™ãŒã€ã“ã®éŽç¨‹ã§NameNodeã®æ§‹æˆã¯å¤‰æ›´ã›ãšã«é‹ç”¨ã—ã¦ãã¾ã—ãŸã€‚ å½“ç„¶ã€CDH4ã‹ã‚‰ã®å…¬å¼HAæ§‹æˆã«é–¢å¿ƒã¯ã‚ã£ãŸã®ã§ã™ãŒã€è¤‡æ•°ã®æ›´æ–°ã‚’åŒæ™‚ã«è¡Œã†ã¨å±ãªã„ã¨ã‹ã€è‹±èªžãƒžãƒ‹ãƒ¥ã‚¢ãƒ«èªã‚€ã®ã‚ã‚“ã©ãã›ãƒ¼æ„Ÿã‹ã‚‰ãƒŸãƒ‰ãƒ«ãƒ¬ãƒ³ã‚¸ã«è·é›¢ã‚’ç½®ã„ã¦ã„ãŸã¨ã“ã‚ã«ã€ã‚‚ã‚Šã™å…ˆç”ŸãŒãƒˆãƒ©ã‚¤ã—ã¦ãã‚Œã¦ã€ä¹—ã‚‹ã—ã‹ãªã„ãƒ“ãƒƒã‚°ã‚¦ã‚§ãƒ¼ãƒ–åˆ°æ¥ã€‚å¾…ã¤ã¹ãã‚‚ã®ã¯ä»–äººã®æ¤œè¨¼ã¨ã¯ã‚ˆãè¨€ã£ãŸã‚‚ã®ã§ã™ãªãã€‚ ã‚¿ã‚¤ãƒˆãƒ«ã¯NameNodeæ§‹æˆã®åˆ‡ã‚Šæ›¿ãˆã€ã¨ãªã£ã¦ã„ã¾ã™ãŒã€ã“ã‚Œã‹ã‚‰æ–°ã—ãQJM HAã§çµ„ã‚€äººã«ã‚‚å½¹ã«ç«‹ã¤å†…å®¹ã¨ãªã£ã¦ã„ã¾ã™ã‚†ãˆã€ç§ãŒè¡€ååã‚’åã„ã¦ã¾ã¨ã‚ãŸæƒ…å ±ã‚’æ˜¯éžã”è¦§ãã ã•ã„ã¾ã›ã€‚ ãƒªãƒ³ã‚¯ CDH4.1ã«ãŠã‘ã‚‹ã‚¯ã‚©ãƒ¼ãƒ©ãƒ ãƒ™ãƒ¼ã‚¹ã‚¸ãƒ£ãƒ¼ãƒŠãƒªãƒ³ã‚° Quorum-Journal Design CDH4.1ã‚ªãƒ¼ãƒãƒ¼ãƒ“ãƒ¥ãƒ¼ Software Configuration for Qu
oza_x86 2013/01/16
NameNode

Hadoop

HDFS
ãƒªãƒ³ã‚¯
[Hadoop]è¤‡æ•°ãƒ‡ã‚£ã‚¹ã‚¯ã‚’ä½¿ã£ã¦åŠ¹çŽ‡ã®è‰¯ã„å‡¦ç†
Hadoop ã§ã¯ä¸€ã¤ã®ãƒŽãƒ¼ãƒ‰ã‚ãŸã‚Šè¤‡æ•°ãƒ‡ã‚£ã‚¹ã‚¯ã‚’ä½¿ã†ã“ã¨ãŒã§ãã¾ã™ãŒï¼Œãƒ‡ã‚£ã‚¹ã‚¯ã‚’å¢—ã‚„ã™ã“ã¨ã«ã‚ˆã£ã¦ã©ã‚Œãã‚‰ã„æ€§èƒ½ãŒå‘ä¸Šã™ã‚‹ã‹èª¿ã¹ã¾ã—ãŸï¼Ž HDFSã§ä½¿ç”¨ã™ã‚‹ãƒ‡ã‚£ã‚¹ã‚¯ã‚’dfs.data.dirã«ã‚³ãƒ³ãƒžåŒºåˆ‡ã‚Šã§è¨˜å…¥ã™ã‚‹ã“ã¨ã§è¤‡æ•°ä½¿ãˆã¾ã™ï¼Ž <property> <name>dfs.data.dir</name> <value>/data/local/${user.name}/hadoop/dfs/data, /data/local2/${user.name}/hadoop/dfs/data</value> </property> ã—ã‹ã—ï¼Œã“ã‚Œã ã‘ã§ã¯ã¾ã ãƒ€ãƒ¡ã§ï¼Œmapã‚¿ã‚¹ã‚¯ï¼Œreduceã‚¿ã‚¹ã‚¯ãŒä¸é–“ãƒ‡ãƒ¼ã‚¿ã‚’æ›¸ãè¾¼ã‚€ãƒ‡ã‚£ã‚¹ã‚¯ã‚‚è¤‡æ•°æŒ‡å®šã—ãªã—ã¨Hadoopã®ã‚¸ãƒ§ãƒ–ã§è¤‡æ•°ãƒ‡ã‚£ã‚¹ã‚¯ã‚’åŠ¹çŽ‡è‰¯ãä½¿ãˆã¾ã›ã‚“ï¼Žmapred.local.dir ã§è¨å®šå¯èƒ½ã§ã™ï¼Ž <property> <name>mapre
oza_x86 2012/06/08
Hadoop

HDFS
ãƒªãƒ³ã‚¯
HDFS Federation++
oza_x86 2012/05/30
Hadoop

HDFS
ãƒªãƒ³ã‚¯
[HDFS-3077] Quorum-based protocol for reading and writing edit logs - ASF JIRA
Currently, one of the weak points of the HA design is that it relies on shared storage such as an NFS filer for the shared edit log. One alternative that has been proposed is to depend on BookKeeper, a ZooKeeper subproject which provides a highly available replicated edit log on commodity hardware. This JIRA is to implement another alternative, based on a quorum commit protocol, integrated more ti
oza_x86 2012/05/20
Hadoop

HDFS

Jira
ãƒªãƒ³ã‚¯
Hadoop and Solid State Drives
Is there a story for the Hadoop Storage Stack (HDFS+HBase) on Solid State Drive (SSD)? This is a question that I have been asked by quite a few people in the last two days, mostly by people at the OpenComputeSummit. This piece discusses the possible use cases of using SSD with Hadoop or HBase. Use Case Currently, there are two primary use cases for HDFS: data warehousing using map-reduce and a key
oza_x86 2012/05/09
Hadoop

HBase

HDFS
ãƒªãƒ³ã‚¯
Facebook has the world's largest Hadoop cluster!
It is not a secret anymore! The Datawarehouse Hadoop cluster at Facebook has become the largest known Hadoop storage cluster in the world. Here are some of the details about this single HDFS cluster:21 PB of storage in a single HDFS cluster2000 machines12 TB per machine (a few machines have 24 TB each)1200 machines with 8 cores each + 800 machines with 16 cores each32 GB of RAM per machine15 map-r
oza_x86 2011/09/02
HDFS

Hadoop

facebook
ãƒªãƒ³ã‚¯
HUG Nov 2010: HDFS Raid - Facebook
HDFS RAID is a contrib project in Hadoop that implements RAID techniques to reduce replication of data in HDFS and tolerate block failures. At Facebook, HDFS RAID is saving about 5PB of storage space by using XOR parity with gradual deployment across tables. The key components are the RaidNode daemon that generates parity files, the BlockFixer that reconstructs missing or corrupt blocks, and the D
oza_x86 2010/12/07
Storage

facebook

Hadoop

HDFS
ãƒªãƒ³ã‚¯
1