Introduction to HDFS Erasure Coding in Apache Hadoop - Cloudera Blog
Erasure coding, a new feature in HDFS, can reduce storage overhead by approximately 50% compared to replication while maintaining the same durability guarantees. This post explains how it works. HDFS by default replicates each block three times. Replication provides a simple and robust form of redundancy to shield against most failure scenarios. It also eases scheduling compute tasks on locally st
RCFile，Parquet，ORCFile
ã“ã®2ヶ月ã§ï¼ŒCloudera/Twitter，Hortonworks ã‹ã‚‰ãã‚Œãžã‚Œåˆ¥ã®åˆ—指å‘ファイルフォーマットãŒå…¬é–‹ã•ã‚Œã¾ã—ãŸï¼ŽParquet 㨠ORCFile ã§ã™ï¼Ž ã“ã®è¨˜äº‹ã§ã¯ï¼Œã¾ãš RCFile ã®å¾©ç¿’ã‚’ã—ã¦ï¼Œãã®å¾Œ Parquet 㨠ORCFile ãã‚Œãžã‚Œã®å…±é€šç‚¹ã¨é•ã„ã‚’ãŠãŠã¾ã‹ã«è¦‹ã¦ã„ã“ã†ã¨æ€ã„ã¾ã™ï¼Žã‚³ãƒ¼ãƒ‰ãƒ¬ãƒ™ãƒ«ã®è©³ç´°ãªé•ã„ã«ã¤ã„ã¦ã¯ï¼Œæ¬¡å›žä»¥é™ã§è¦‹ã¦ã„ãã¾ã™ï¼Ž RCFile ã®å¾©ç¿’ RCFile ã¯ã€€Record Columnar File ã®ç•¥ã§ï¼ŒHive ã‹ã‚‰åˆ©ç”¨ã§ãるストレージフォーマットã§ã™ï¼Žç‰¹ã«ï¼ŒHDFS ã‚„ S3 ã¨ã„ã£ãŸåˆ†æ•£ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ä¸Šã§ãƒ‘フォーマンスãŒã§ã‚‹ã‚ˆã†ã«è¨è¨ˆã•ã‚Œã¦ã„ã¾ã™ï¼Ž HDFS/S3 ã¨ã„ã£ãŸã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ã§ã¯ï¼ŒåŸºæœ¬çš„ã«ãƒ‡ãƒ¼ã‚¿ã‚’計算機間ã§åŒã˜è² è·ã«ãªã‚‹ã‚ˆã†ã«ãƒ‡ãƒ¼ã‚¿ã‚’分散é…ç½®ã—ã¾ã™ï¼Žã“ã®ãŸã‚，従æ¥ã®åˆ—指å‘ストレージフォーマットã®ã‚ˆã†ã«é©å½“ã«åˆ—毎ã«
How to convert non-HA NameNode to QJM HA on CDH4 | 外é“父ã®åŒ
CDH3ã‹ã‚‰å§‹ã‚ã¦CDH4.1ã¾ã§ã‚¢ãƒƒãƒ—グレードã—ã¦åˆ©ç”¨ã—続ã‘ã¦ã„ã¾ã™ãŒã€ã“ã®éŽç¨‹ã§NameNodeã®æ§‹æˆã¯å¤‰æ›´ã›ãšã«é‹ç”¨ã—ã¦ãã¾ã—ãŸã€‚ 当然ã€CDH4ã‹ã‚‰ã®å…¬å¼HA構æˆã«é–¢å¿ƒã¯ã‚ã£ãŸã®ã§ã™ãŒã€è¤‡æ•°ã®æ›´æ–°ã‚’åŒæ™‚ã«è¡Œã†ã¨å±ãªã„ã¨ã‹ã€è‹±èªžãƒžãƒ‹ãƒ¥ã‚¢ãƒ«èªã‚€ã®ã‚ã‚“ã©ãã›ãƒ¼æ„Ÿã‹ã‚‰ãƒŸãƒ‰ãƒ«ãƒ¬ãƒ³ã‚¸ã«è·é›¢ã‚’ç½®ã„ã¦ã„ãŸã¨ã“ã‚ã«ã€ã‚‚ã‚Šã™å…ˆç”ŸãŒãƒˆãƒ©ã‚¤ã—ã¦ãã‚Œã¦ã€ä¹—ã‚‹ã—ã‹ãªã„ビッグウェーブ到æ¥ã€‚å¾…ã¤ã¹ãã‚‚ã®ã¯ä»–人ã®æ¤œè¨¼ã¨ã¯ã‚ˆã言ã£ãŸã‚‚ã®ã§ã™ãªã。 タイトルã¯NameNode構æˆã®åˆ‡ã‚Šæ›¿ãˆã€ã¨ãªã£ã¦ã„ã¾ã™ãŒã€ã“ã‚Œã‹ã‚‰æ–°ã—ãQJM HAã§çµ„む人ã«ã‚‚å½¹ã«ç«‹ã¤å†…容ã¨ãªã£ã¦ã„ã¾ã™ã‚†ãˆã€ç§ãŒè¡€ååã‚’åã„ã¦ã¾ã¨ã‚ãŸæƒ…å ±ã‚’æ˜¯éžã”覧ãã ã•ã„ã¾ã›ã€‚ リンク CDH4.1ã«ãŠã‘るクォーラムベースジャーナリング Quorum-Journal Design CDH4.1オーãƒãƒ¼ãƒ“ュー Software Configuration for Qu
HBase File Locality in HDFS
One of the more ambiguous things in Hadoop is block replication: it happens automatically and you should not have to worry about it. HBase relies on it 100% to provide the data safety as it stores its files into the distributed file system. While that works completely transparent, one of the more advanced questions asked though is how does this affect performance? This usually arises when the user
[Hadoop]複数ディスクを使ã£ã¦åŠ¹çŽ‡ã®è‰¯ã„処ç†
Hadoop ã§ã¯ä¸€ã¤ã®ãƒŽãƒ¼ãƒ‰ã‚ãŸã‚Šè¤‡æ•°ãƒ‡ã‚£ã‚¹ã‚¯ã‚’使ã†ã“ã¨ãŒã§ãã¾ã™ãŒï¼Œãƒ‡ã‚£ã‚¹ã‚¯ã‚’増やã™ã“ã¨ã«ã‚ˆã£ã¦ã©ã‚Œãらã„性能ãŒå‘上ã™ã‚‹ã‹èª¿ã¹ã¾ã—ãŸï¼Ž HDFSã§ä½¿ç”¨ã™ã‚‹ãƒ‡ã‚£ã‚¹ã‚¯ã‚’dfs.data.dirã«ã‚³ãƒ³ãƒžåŒºåˆ‡ã‚Šã§è¨˜å…¥ã™ã‚‹ã“ã¨ã§è¤‡æ•°ä½¿ãˆã¾ã™ï¼Ž <property> <name>dfs.data.dir</name> <value>/data/local/${user.name}/hadoop/dfs/data, /data/local2/${user.name}/hadoop/dfs/data</value> </property> ã—ã‹ã—，ã“ã‚Œã ã‘ã§ã¯ã¾ã ダメã§ï¼Œmapタスク，reduceタスクãŒä¸é–“データを書ã込むディスクも複数指定ã—ãªã—ã¨Hadoopã®ã‚¸ãƒ§ãƒ–ã§è¤‡æ•°ãƒ‡ã‚£ã‚¹ã‚¯ã‚’効率良ã使ãˆã¾ã›ã‚“．mapred.local.dir ã§è¨å®šå¯èƒ½ã§ã™ï¼Ž <property> <name>mapre
[HDFS-3077] Quorum-based protocol for reading and writing edit logs - ASF JIRA
Currently, one of the weak points of the HA design is that it relies on shared storage such as an NFS filer for the shared edit log. One alternative that has been proposed is to depend on BookKeeper, a ZooKeeper subproject which provides a highly available replicated edit log on commodity hardware. This JIRA is to implement another alternative, based on a quorum commit protocol, integrated more ti
Hadoop and Solid State Drives
Is there a story for the Hadoop Storage Stack (HDFS+HBase) on Solid State Drive (SSD)? This is a question that I have been asked by quite a few people in the last two days, mostly by people at the OpenComputeSummit. This piece discusses the possible use cases of using SSD with Hadoop or HBase. Use Case Currently, there are two primary use cases for HDFS: data warehousing using map-reduce and a key
Cloudera Blog
In an era where artificial intelligence (AI) is reshaping enterprises across the globe—be it in healthcare, finance, or manufacturing—it’s hard to overstate the transformation that AI has had on businesses, regardless of industry or size. At Cloudera, we recognize the urgent need for bold steps to harness this potential and dramatically accelerate the time to […] Read blog post
Facebook has the world's largest Hadoop cluster!
It is not a secret anymore! The Datawarehouse Hadoop cluster at Facebook has become the largest known Hadoop storage cluster in the world. Here are some of the details about this single HDFS cluster:21 PB of storage in a single HDFS cluster2000 machines12 TB per machine (a few machines have 24 TB each)1200 machines with 8 cores each + 800 machines with 16 cores each32 GB of RAM per machine15 map-r
HUG Nov 2010: HDFS Raid - Facebook
HDFS RAID is a contrib project in Hadoop that implements RAID techniques to reduce replication of data in HDFS and tolerate block failures. At Facebook, HDFS RAID is saving about 5PB of storage space by using XOR parity with gradual deployment across tables. The key components are the RaidNode daemon that generates parity files, the BlockFixer that reconstructs missing or corrupt blocks, and the D
1
ランã‚ング
ランã‚ング
ランã‚ング
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
GitHub - GiraffaFS/giraffa: Giraffa FileSystem (Slack: giraffa-fs.slack.com)
Konstantin V. Shvachko: Publications
HDFS Federation++
{{#tags}}- {{label}}
{{/tags}}