使用distcp将数据从cloudera集群复制到googlecloudhdfs集群

b4qexyjb  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(453)

我正在使用cloudera quickstart虚拟机。我昨天开始玩谷歌云平台。我正在尝试将cloudera hdfs中的数据复制到1。谷歌云存储(gs://bucket\u name/)2。google云hdfs集群(使用hdfs://google_cluster_namenode:8020/)
我设置了服务帐户身份验证,并按照本文的说明配置了cloudera core-site.xml

hadoop fs -cp hdfs://quickstart.cloudera:8020/path_to_copy/ gs://bucket_name/

很好用。但是,我无法使用distcp复制到google云存储。我得到以下错误。我知道这不是uri问题。我还缺什么吗?

Error: java.io.IOException: File copy failed: hdfs://quickstart.cloudera:8020/path_to_copy/file --> gs://bucket_name/file
at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:284)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:252)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 
Caused by: java.io.IOException: Couldn't run retriable-command: Copying hdfs://quickstart.cloudera:8020/path_to_copy/file to gs://bucket_name/file
at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:280)
... 10 more 
Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: gs://bucket_name.distcp.tmp.attempt_1461777569169_0002_m_000001_2
at org.apache.hadoop.fs.Path.initialize(Path.java:206)
at org.apache.hadoop.fs.Path.<init>(Path.java:116)
at org.apache.hadoop.fs.Path.<init>(Path.java:94)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.getTmpFile(RetriableFileCopyCommand.java:233)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:107)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:100)
at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
... 11 more

我无法让distcp连接到googlecloudhdfs namenode;我得到“重试连接到服务器”。我找不到任何文档来配置cloudera hdfs集群和google cloud hdfs集群之间的连接。我假设服务帐户auth也应该与googlehdfs一起工作。是否有一个参考文档我可以用来设置集群之间的拷贝?我还缺少其他身份验证设置吗?

rryofs0p

rryofs0p1#

结果我不得不修改防火墙规则,以允许来自我运行distcp的ip的tcp/http。检查gcp计算示例上的网络防火墙。

相关问题