java—无法使用distcp将一个hdfs数据复制到另一个hdfs位置

xkftehaa  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(621)

我正在尝试将一个hdfs数据复制到另一个hdfs位置。
我可以使用“distcp”命令实现同样的功能

hadoop distcp hdfs://mySrcip:8020/copyDev/* hdfs://myDestip:8020/copyTest

但是我想用javaapi尝试同样的方法。经过长时间的搜索,找到一个代码并执行。但它并没有把我的src文件复制到目的地。

public class TouchFile {

/**
 * @param args
 * @throws Exception 
 */
public static void main(String[] args) throws Exception {
    // TODO Auto-generated method stub
    //create configuration object
    Configuration config = new Configuration();
    config.set("fs.defaultFS", "hdfs://mySrcip:8020/");
    config.set("hadoop.job.ugi", "hdfs");
    /*
     * Distcp
     */
    String sourceNameNode = "hdfs://mySrcip:8020/copyDev";
    String destNameNode = "hdfs://myDestip:8020/copyTest";
    String fileList = "myfile.txt";
    distFileCopy(config,sourceNameNode,destNameNode,fileList);
}
/**
 * Copies files from one cloud to another using Hadoop's distributed copy features. Uses
 * input to build DISTCP configuration settings. 
 *
 * param config Hadoop configuration
 * param sourceNameNode full HDFS path to parent source directory
 * param destNameNode full HDFS path to parent destination directory
 * param fileList Comma separated string of file names in sourceNameNode to be copied to destNameNode
 * returns Elapsed time in milliseconds to copy files
 */
public static long distFileCopy( Configuration config, String sourceNameNode, String destNameNode, String fileList ) throws Exception {
        System.out.println("In dist copy");

    StringTokenizer tokenizer = new StringTokenizer(fileList,",");
    ArrayList<String> list = new ArrayList<>();

    while ( tokenizer.hasMoreTokens() ){
        String file = sourceNameNode + "/" + tokenizer.nextToken();
        list.add( file );
    }

    String[] args = new String[list.size() + 1];
    int count = 0;
    for ( String filename : list ){
        args[count++] = filename;
    }

    args[count] = destNameNode;

    System.out.println("args------>"+Arrays.toString(args));
    long st = System.currentTimeMillis();        
    DistCp distCp=new DistCp(config,null);
    distCp.run(args);   
    return System.currentTimeMillis() - st;

}

}

我做错什么了吗。请建议

ccgok5k5

ccgok5k51#

是的,问题解决了。
这是许可问题。
目标群集应授予用户权限。

相关问题