java—无法使用distcp将一个hdfs数据复制到另一个hdfs位置

xkftehaa  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(664)

我正在尝试将一个hdfs数据复制到另一个hdfs位置。
我可以使用“distcp”命令实现同样的功能

  1. hadoop distcp hdfs://mySrcip:8020/copyDev/* hdfs://myDestip:8020/copyTest

但是我想用javaapi尝试同样的方法。经过长时间的搜索,找到一个代码并执行。但它并没有把我的src文件复制到目的地。

  1. public class TouchFile {
  2. /**
  3. * @param args
  4. * @throws Exception
  5. */
  6. public static void main(String[] args) throws Exception {
  7. // TODO Auto-generated method stub
  8. //create configuration object
  9. Configuration config = new Configuration();
  10. config.set("fs.defaultFS", "hdfs://mySrcip:8020/");
  11. config.set("hadoop.job.ugi", "hdfs");
  12. /*
  13. * Distcp
  14. */
  15. String sourceNameNode = "hdfs://mySrcip:8020/copyDev";
  16. String destNameNode = "hdfs://myDestip:8020/copyTest";
  17. String fileList = "myfile.txt";
  18. distFileCopy(config,sourceNameNode,destNameNode,fileList);
  19. }
  20. /**
  21. * Copies files from one cloud to another using Hadoop's distributed copy features. Uses
  22. * input to build DISTCP configuration settings.
  23. *
  24. * param config Hadoop configuration
  25. * param sourceNameNode full HDFS path to parent source directory
  26. * param destNameNode full HDFS path to parent destination directory
  27. * param fileList Comma separated string of file names in sourceNameNode to be copied to destNameNode
  28. * returns Elapsed time in milliseconds to copy files
  29. */
  30. public static long distFileCopy( Configuration config, String sourceNameNode, String destNameNode, String fileList ) throws Exception {
  31. System.out.println("In dist copy");
  32. StringTokenizer tokenizer = new StringTokenizer(fileList,",");
  33. ArrayList<String> list = new ArrayList<>();
  34. while ( tokenizer.hasMoreTokens() ){
  35. String file = sourceNameNode + "/" + tokenizer.nextToken();
  36. list.add( file );
  37. }
  38. String[] args = new String[list.size() + 1];
  39. int count = 0;
  40. for ( String filename : list ){
  41. args[count++] = filename;
  42. }
  43. args[count] = destNameNode;
  44. System.out.println("args------>"+Arrays.toString(args));
  45. long st = System.currentTimeMillis();
  46. DistCp distCp=new DistCp(config,null);
  47. distCp.run(args);
  48. return System.currentTimeMillis() - st;
  49. }
  50. }

我做错什么了吗。请建议

ccgok5k5

ccgok5k51#

是的,问题解决了。
这是许可问题。
目标群集应授予用户权限。

相关问题