我试图复制文件从hdfs到s3使用 distcp 通过执行以下命令
hadoop distcp -fs.s3a.access.key=AccessKey -fs.s3a.secret.key=SecrerKey \s3n://testbdr/test2 hdfs://hostname:portnumber/tmp/test
但我得到以下错误：
17/09/05 02:59:30 ERROR tools.DistCp: Invalid arguments:java.lang.IllegalArgumentException: Both source file listing and source paths presentat org.apache.hadoop.tools.OptionsParser.parseSourceAndTargetPaths(OptionsParser.java:341)at org.apache.hadoop.tools.OptionsParser.parse(OptionsParser.java:89)at org.apache.hadoop.tools.DistCp.run(DistCp.java:112)at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)at org.apache.hadoop.tools.DistCp.main(DistCp.java:436)Invalid arguments: Both source file listing and source paths presentusage: distcp OPTIONS [source_path...] <target_path>OPTIONS-append Reuse existing data in target files andappend new data to them if possible-async Should distcp execution be blocking
2条答案
按热度按时间fquxozlt1#
要传递配置参数，必须使用
-D```hadoop distcp -Dfs.s3a.access.key=AccessKey -Dfs.s3a.secret.key=SecrerKey
s3n://testbdr/test2 hdfs://hostname:portnumber/tmp/test
lrl1mhuk2#
旧命令
hadoop distcp-dfs.s3a.access.key=accesskey-dfs.s3a.secret.key=secretkey\s3n://testbdr/test2hdfs://hostname：端口号/tmp/测试
修正指令
hadoop distcp-dfs.s3n.awsaccesskeyid=accesskey-dfs.s3n.awssecretaccesskey=secretkey\s3n://testbdr/test2hdfs://hostname：端口号/tmp/测试