Uploaded image for project: 'hops'
  1. HOPS-711

1 datanode cluster - repeated append/close - DFSClient thinks datanode is faulty

    Details

    • Type: Bug
    • Status: To Do (View workflow)
    • Priority: Medium
    • Resolution: Unresolved
    • Affects versions: None
    • Fix versions: None
    • Labels:
      None
    • Sprint:

      Description

      Environment - vm with 1 datanode
      The following test fails:

      Configuration conf = new Configuration();
      conf.set("fs.defaultFS", "hdfs://10.0.2.15:8020");
      UserGroupInformation ugi = UserGroupInformation.createRemoteUser("vagrant");
      DistributedFileSystem fs = (DistributedFileSystem) FileSystem.get(conf);
          
      String filePath = "/test/file";
      fs.delete(new Path(filePath), true);
      fs.create(new Path(filePath)).close();
      
      //first append multiple writes to the out stream    
      FSDataOutputStream out = fs.append(new Path(filePath));
      out.write(1);
      out.write(2);
      out.write(3);
      out.close();
      
      //second append    
      out = fs.append(new Path(filePath));
      out.write(1);
      //test fails here due to "dfs.client.block.write.replace-datanode-on-failure.policy"
      //setting this policy to NEVER makes the test pass
      out.close();
          
      fs.close();
      "
      
      This test will pass:
      "
      Configuration conf = new Configuration();
      conf.set("fs.defaultFS", "hdfs://10.0.2.15:8020");
      //FIX?! Workaround?
      conf.set("dfs.client.block.write.replace-datanode-on-failure.policy", "NEVER");
      UserGroupInformation ugi = UserGroupInformation.createRemoteUser("vagrant");
      DistributedFileSystem fs = (DistributedFileSystem) FileSystem.get(conf);
          
      String filePath = "/test/file";
      fs.delete(new Path(filePath), true);
      fs.create(new Path(filePath)).close();
      
      //first append multiple writes to the out stream    
      FSDataOutputStream out = fs.append(new Path(filePath));
      out.write(1);
      out.write(2);
      out.write(3);
      out.close();
      
      //second append    
      out = fs.append(new Path(filePath));
      out.write(1);
      out.close();
          
      fs.close();
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              aaor Alex Ormenisan
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: