我正在尝试将一个hdfs数据复制到另一个hdfs位置。
我可以使用“distcp”命令实现同样的功能
hadoop distcp hdfs://mySrcip:8020/copyDev/* hdfs://myDestip:8020/copyTest
但是我想用javaapi尝试同样的方法。经过长时间的搜索,找到一个代码并执行。但它并没有把我的src文件复制到目的地。
public class TouchFile {
/**
* @param args
* @throws Exception
*/
public static void main(String[] args) throws Exception {
// TODO Auto-generated method stub
//create configuration object
Configuration config = new Configuration();
config.set("fs.defaultFS", "hdfs://mySrcip:8020/");
config.set("hadoop.job.ugi", "hdfs");
/*
* Distcp
*/
String sourceNameNode = "hdfs://mySrcip:8020/copyDev";
String destNameNode = "hdfs://myDestip:8020/copyTest";
String fileList = "myfile.txt";
distFileCopy(config,sourceNameNode,destNameNode,fileList);
}
/**
* Copies files from one cloud to another using Hadoop's distributed copy features. Uses
* input to build DISTCP configuration settings.
*
* param config Hadoop configuration
* param sourceNameNode full HDFS path to parent source directory
* param destNameNode full HDFS path to parent destination directory
* param fileList Comma separated string of file names in sourceNameNode to be copied to destNameNode
* returns Elapsed time in milliseconds to copy files
*/
public static long distFileCopy( Configuration config, String sourceNameNode, String destNameNode, String fileList ) throws Exception {
System.out.println("In dist copy");
StringTokenizer tokenizer = new StringTokenizer(fileList,",");
ArrayList<String> list = new ArrayList<>();
while ( tokenizer.hasMoreTokens() ){
String file = sourceNameNode + "/" + tokenizer.nextToken();
list.add( file );
}
String[] args = new String[list.size() + 1];
int count = 0;
for ( String filename : list ){
args[count++] = filename;
}
args[count] = destNameNode;
System.out.println("args------>"+Arrays.toString(args));
long st = System.currentTimeMillis();
DistCp distCp=new DistCp(config,null);
distCp.run(args);
return System.currentTimeMillis() - st;
}
}
我做错什么了吗。请建议
1条答案
按热度按时间ccgok5k51#
是的,问题解决了。
这是许可问题。
目标群集应授予用户权限。