Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-265

Revisit append

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.21.0
    • 0.21.0
    • None
    • None

    Description

      HADOOP-1700 and related issues have put a lot of efforts to provide the first implementation of append. However, append is such a complex feature. It turns out that there are issues that were initially seemed trivial but needs a careful design. This jira revisits append, aiming for a design and implementation supporting a semantics that are acceptable to its users.

      Attachments

        1. a.sh
          1 kB
          Tsz-wo Sze
        2. appendDesign.pdf
          408 kB
          Konstantin Shvachko
        3. appendDesign.pdf
          580 kB
          Hairong Kuang
        4. appendDesign1.pdf
          624 kB
          Hairong Kuang
        5. appendDesign2.pdf
          868 kB
          Hairong Kuang
        6. appendDesign3.pdf
          869 kB
          Hairong Kuang
        7. AppendSpec.pdf
          48 kB
          Hairong Kuang
        8. AppendTestPlan.html
          64 kB
          Konstantin I Boudnik
        9. AppendTestPlan.html
          65 kB
          Konstantin I Boudnik
        10. AppendTestPlan.html
          65 kB
          Konstantin I Boudnik
        11. AppendTestPlan.html
          65 kB
          Konstantin I Boudnik
        12. AppendTestPlan.html
          65 kB
          Konstantin I Boudnik
        13. AppendTestPlan.html
          63 kB
          Konstantin I Boudnik
        14. AppendTestPlan.html
          62 kB
          Konstantin I Boudnik
        15. AppendTestPlan.html
          62 kB
          Konstantin I Boudnik
        16. AppendTestPlan.html
          55 kB
          Konstantin I Boudnik
        17. TestPlanAppend.html
          50 kB
          Konstantin I Boudnik

        Issue Links

          incorporates

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-728 Create a comprehensive functional test for append

          • Major - Major loss of function.
          • Closed
          is related to

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-6239 Command-line for append

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HDFS-744 Support hsync in HDFS

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-1831 HDFS equivalent of HADOOP-7223 changes to handle FileContext createFlag combinations

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-588 Fix TestFiDataTransferProtocol and TestAppend2 failures in append branch.

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-1060 Append/flush should support concurrent "tailer" use case

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-195 Need to handle access token expiration when re-establishing the pipeline for dfs write

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HDFS-1606 Provide a stronger data guarantee in the write pipeline

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-4359 Access Token: Support for data access authorization checking on DataNodes

          • Major - Major loss of function.
          • Closed

          Test - A new unit, integration or system test. HDFS-714 Create fault injection test for the new pipeline close

          • Major - Major loss of function.
          • Closed

          Test - A new unit, integration or system test. HDFS-451 Test DataTransferProtocol with fault injection

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          supercedes

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-1700 Append to files in HDFS

          • Major - Major loss of function.
          • Closed
          1.
          		 Factor out BlockInfo from BlocksMap Sub-task Closed Konstantin Shvachko
          2.
          		 Redesign DataNode volumeMap to include all types of Replicas Sub-task Resolved Hairong Kuang
          3.
          		 Introduce BlockInfoUnderConstruction to reflect block replica states while writing. Sub-task Resolved Konstantin Shvachko
          4.
          		 Create new tests for Append's hflush Sub-task Resolved Konstantin I Boudnik
          5.
          		 Create new tests for lease recovery Sub-task Closed Konstantin I Boudnik
          6.
          		 Create new tests for block recovery Sub-task Closed Hairong Kuang
          7.
          		 Create new tests for pipeline Sub-task Closed Konstantin I Boudnik
          8.
          		 Create stress tests for append feature Sub-task Resolved Unassigned
          9.
          		 Support hflush at DFSClient Sub-task Resolved Hairong Kuang
          10.
          		 DataNode uses ReplicaBeingWritten to support dfs writes/hflush Sub-task Resolved Hairong Kuang
          11.
          		 Break FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and append Sub-task Resolved Hairong Kuang
          12.
          		 Add a "rbw" sub directory to DataNode data directory Sub-task Resolved Hairong Kuang
          13.
          		 DataNode restarts may introduce corrupt/duplicated/lost replicas when handling detached replicas Sub-task Resolved Hairong Kuang
          14.
          		 Add a test for NameNode.getBlockLocations(..) to check read from un-closed file. Sub-task Resolved Tsz-wo Sze
          15.
          		 Introduce block committing logic during new block allocation and file close. Sub-task Resolved Konstantin Shvachko
          16.
          		 When opening a file for read, make the file length avaliable to client. Sub-task Resolved Tsz-wo Sze
          17.
          		 Extend Block report to include under-construction replicas Sub-task Resolved Konstantin Shvachko
          18.
          		 Datanode should serve up to visible length of a replica for read requests Sub-task Resolved Tsz-wo Sze
          19.
          		 Change block write protocol to support pipeline recovery Sub-task Resolved Hairong Kuang
          20.
          		 Create new functional test for a block report. Sub-task Closed Konstantin I Boudnik
          21.
          		 Allow client to get a new generation stamp from NameNode Sub-task Resolved Hairong Kuang
          22.
          		 Block report processing for append Sub-task Resolved Konstantin Shvachko
          23.
          		 Create functional tests for new design of the block report Sub-task Closed Konstantin I Boudnik
          24.
          		 Support replica recovery initialization in datanode Sub-task Resolved Tsz-wo Sze
          25.
          		 Client support pipeline recovery Sub-task Resolved Hairong Kuang
          26.
          		 Support replica update in datanode Sub-task Resolved Tsz-wo Sze
          27.
          		 SafeMode should count only complete blocks. Sub-task Resolved Konstantin Shvachko
          28.
          		 Support pipeline close and close recovery Sub-task Resolved Hairong Kuang
          29.
          		 Lease recovery, concurrency support. Sub-task Resolved Konstantin Shvachko
          30.
          		 Replace BlockInfo.isUnderConstruction() with isComplete() Sub-task Resolved Konstantin Shvachko
          31.
          		 Simplify the codes in the replica related classes Sub-task Resolved Unassigned
          32.
          		 Remove unused legacy protocol methods. Sub-task Resolved Konstantin Shvachko
          33.
          		 Block recovery for primary data-node Sub-task Resolved Konstantin Shvachko
          34.
          		 DFSClient cannot read all the available bytes Sub-task Resolved Tsz-wo Sze
          35.
          		 Remove deprecated methods from InterDatanodeProtocol. Sub-task Closed Konstantin Shvachko
          36.
          		 Data-node upgrade problem Sub-task Resolved Hairong Kuang
          37.
          		 Unnecessary info message from DFSClient Sub-task Resolved Hairong Kuang
          38.
          		 DFSIO for append Sub-task Closed Konstantin Shvachko
          39.
          		 TestFileAppend2 sometimes hangs Sub-task Resolved Hairong Kuang
          40.
          		 TestFileAppend3#TC7 sometimes hangs Sub-task Closed Hairong Kuang
          41.
          		 NPE in FSDataset.updateReplicaUnderRecovery(..) Sub-task Closed Konstantin Shvachko
          42.
          		 Create block recovery tests that handle errors Sub-task Closed Hairong Kuang

          Activity

            People

              hairong Hairong Kuang
              hairong Hairong Kuang
              Votes:
              5 Vote for this issue
              Watchers:
              54 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: