HUG Nov 2010: HDFS Raid - Facebook
•Download as PPTX, PDF•
18 likes•6,411 views
HDFS RAID is a contrib project in Hadoop that implements RAID techniques to reduce replication of data in HDFS and tolerate block failures. At Facebook, HDFS RAID is saving about 5PB of storage space by using XOR parity with gradual deployment across tables. The key components are the RaidNode daemon that generates parity files, the BlockFixer that reconstructs missing or corrupt blocks, and the DistributedRaidFileSystem that allows reads from corrupt files by reconstructing missing blocks on the fly.
![Erasure CodeErasureCodeabstraction for erasure code implementations public void encode(int[] message, int[] parity); public void decode(int[] data, int[] erasedLocations,int[] erasedValues);Current implementationsXOR CodeReed Solomon CodeEncoder/Decoder – uses ErasureCode to integrate with RAID framework](https://image.slidesharecdn.com/hdfsraid-1116-101203173514-phpapp02/85/HUG-Nov-2010-HDFS-Raid-Facebook-16-320.jpg)
HUG Nov 2010: HDFS Raid - Facebook
- 1. HDFS RAIDDhrubaBorthakur ([email protected])Rodrigo Schmidt ([email protected])RamkumarVadali ([email protected])Scott Chen ([email protected])Patrick Kling ([email protected])
- 2. AgendaWhat is RAIDRAID at FacebookAnatomy of RAIDHow to DeployQuestions
- 3. What Is RAIDContrib project in MAPREDUCEDefault HDFS replication is 3Too much at PetaByte scaleRAID helps save space in HDFSReduce replication of “source” dataData safety using “parity” data
- 4. Tolerates 2 missing blocks, Storage cost 3x 123456789101234567891012345678910Tolerates 4 missing blocks, Storage cost 1.4x 12345678910Source fileP1P2P3P4Parity fileReed-Solomon Erasure Codes
- 5. RAID at FacebookReduces disk usage in the warehouseCurrently saving about 5PB with XOR RAIDGradual deploymentStarted with few tablesNow used with all tablesReed Solomon RAID under way
- 6. Saving 5PB at Facebook
- 7. Anatomy of RAIDServer-side:RaidNodeBlockFixerBlock placement policyClient-side:DistributedRaidFileSystemRaid Shell
- 8. Anatomy of RAIDDataNodesNameNode Obtain missing blocks
- 9. Get files to raidRaidNode Create parity files
- 10. Fix missing blocks
- 11. Recover files while readingJobTrackerRaid File System
- 12. RaidNodeDaemon that scans filesystemPolicy file used to provide file patternsGenerate parity filesSingle threadMap-Reduce jobReduces replication of source fileOne thread to purge outdated parity filesIf the source gets deletedOne thread to HAR parity filesTo reduce inode count
- 13. Block FixerReconstructs missing/corrupt blocksRetrieves a list of corrupt files from NameNodeSource blocks are reconstructed by “decoding”Parity blocks are reconstructed by “encoding”
- 14. Block FixerBonus: Parity HARsOne HAR block => multiple parity blocksReconstructs all necessary blocks
- 16. Erasure CodeErasureCodeabstraction for erasure code implementations public void encode(int[] message, int[] parity); public void decode(int[] data, int[] erasedLocations,int[] erasedValues);Current implementationsXOR CodeReed Solomon CodeEncoder/Decoder – uses ErasureCode to integrate with RAID framework
- 17. Block PlacementReplication = 3, Tolerates any 2 errors123456789101234567891012345678910Dependent BlocksReplication = 1, Parity Length = 4, Tolerates any 4 errors12345678910P1P2P3P4Dependent Blocks
- 18. Block PlacementRaid introduces new dependency between blocks in source and parity filesDefault block placement is bad for RAIDSource/Parity blocks can be on a single node/rackParity blocks could co-locate with source blocksRaid Block PolicySource files: After RAIDing, disperse blocksParity files: Control placement of parity blocks to avoid source blocks and other parity blocks
- 19. DistributedRaidFileSystemA filter file system implementationAllows clients to read “corrupt” source filesCatches BlockMissingException, ChecksumExceptionRecreates missing blocks on the fly by using parityDoes not fix the missing blocksOnly allows the reads to succeed
- 20. RaidShellAdministrator toolRecover blocksReconstruct missing blocksSend reconstructed block to a data nodeRaid FSCK Report corrupt files that cannot be fixed by raidHandy tool as a last resort to fix blocks
- 21. DeploymentSingle configuration file “raid.xml”Specifies file patterns to RAIDIn HDFS config fileSpecify raid.xml locationSpecify location of parity files (default: /raid)Specify FileSystem, BlockPlacementPolicyStarting RaidNodestart-raidnode.sh, stop-raidnode.shhttp://wiki.apache.org/hadoop/HDFS-RAID
- 23. LimitationsRAID needs file with 3 or more blocksOtherwise parity blocks negate space savingNeed to HAR small source filesReplication of 1 reduces locality for MR jobsReplication of 2 is not too badIts very difficult to manage block placement of Parity HAR blocks
- 24. File Stats