hadoop—如何使用JavaAPI在hdfs中移动或复制文件

ttcibm8c  于 2021-06-02  发布在  Hadoop
关注(0)|答案(2)|浏览(426)

我想在同一个hdfs中复制文件,就像从hdfs://:9000/user/a.txt到hdfs://:9000/用户/123/
我可以用javaapi来实现吗?谢谢

t8e9dugd

t8e9dugd1#

fileutil提供了一种复制文件的方法。

Configuration configuration = new Configuration();
configuration.set("fs.defaultFS", "hdfs://abc:9000");
FileSystem filesystem = FileSystem.get(configuration);
FileUtil.copy(filesystem, new Path("src/path"), filesystem, new Path("dst/path"), false, configuration);

如果您需要将它复制到另一个集群,只需创建一个新的 Configuration 设置和新建 FileSystem .

t98cgbkg

t98cgbkg2#

If you want to move files from directory it is little bit tricky below code done same task for me !!
val conf = new org.apache.hadoop.conf.Configuration()
    val src:Path = new org.apache.hadoop.fs.Path(hdfsDirectory)
    val fs = FileSystem.get(src.toUri,conf)
    val srcPath: Path = new Path("hdfs://sourcePath/")
    val srcFs =FileSystem.get(srcPath.toUri,conf)
    val dstPath:Path =new Path("hdfs://targetPath/")
    val dstFs =FileSystem.get(dstPath.toUri,conf)
    val exists = fs.exists(new org.apache.hadoop.fs.Path(hdfsDirectory))
    val status:Array[FileStatus] = fs.listStatus(new Path(hdfsDirectory))
    if (status.length>0) {
      status.foreach(x => {
        println("My files: " + x.getPath)
        FileUtil.copy(srcFs, x.getPath, dstFs, dstPath, true, conf)
        println("Files moved !!" +x.getPath)
      }
      )}
    else{
      println("No Files Found !!")
    }

相关问题