在hadoop2.x中从spring运行distcp作业

p8ekf7hl  于 2021-06-04  发布在  Hadoop
关注(0)|答案(1)|浏览(376)

我已经在我的项目中使用了springdatahadoop,并且能够在hadoop1.x中运行distcp作业。最近我们已经升级到Hadoop2.x,为此我将SpringDataHadoop升级到了2.0.4。大部分的东西仍在工作,但我遇到了一些问题与distcp。看起来spring数据hadoop是这样调用distcp的

Class<org.apache.hadoop.tools.DistCp> cl = org.apache.hadoop.tools.DistCp.class;
Class<?> argClass = ClassUtils.resolveClassName("org.apache.hadoop.tools.DistCp$Arguments",
                cl.getClassLoader());

https://github.com/spring-projects/spring-hadoop/blob/2.0.4.release/spring-hadoop-core/src/main/java/org/springframework/data/hadoop/fs/distcp.java#l274-l275型
请注意,spring代码正在查找 Arguments 中的内部类 Distcp 类,但这个内部类在较新的distcp代码中似乎不再存在。当我运行作业时,我得到这个错误

Caused by: java.lang.IllegalStateException: Cannot run distCp impersonated as 'null'
        at org.springframework.data.hadoop.fs.DistCp.copy(DistCp.java:268) ~[spring-data-hadoop-core-2.0.4.RELEASE.jar:2.0.4.RELEASE]
        at org.springframework.data.hadoop.fs.DistCp.copy(DistCp.java:216) ~[spring-data-hadoop-core-2.0.4.RELEASE.jar:2.0.4.RELEASE]
        at org.springframework.data.hadoop.fs.DistCp.copy(DistCp.java:152) ~[spring-data-hadoop-core-2.0.4.RELEASE.jar:2.0.4.RELEASE]
        at com.att.hadoop.hdfspub.source.hdfs.HdfsFileCopier.copyFolder(HdfsFileCopier.java:104) ~[classes/:na]
        ... 45 common frames omitted
Caused by: java.lang.IllegalArgumentException: Cannot find class [org.apache.hadoop.tools.DistCp$Arguments]
        at org.springframework.util.ClassUtils.resolveClassName(ClassUtils.java:286) ~[spring-core-4.1.4.RELEASE.jar:4.1.4.RELEASE]
        at org.springframework.data.hadoop.fs.DistCp.invokeCopy(DistCp.java:275) ~[spring-data-hadoop-core-2.0.4.RELEASE.jar:2.0.4.RELEASE]
        at org.springframework.data.hadoop.fs.DistCp.copy(DistCp.java:265) ~[spring-data-hadoop-core-2.0.4.RELEASE.jar:2.0.4.RELEASE]
        ... 48 common frames omitted
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.tools.DistCp$Arguments

在pom.xml中,我包含了spring数据hadoop2.0.4.release和hadoopdistcp.2.2.0。

rur96b6h

rur96b6h1#

这在最近的2.1rc1版本中已经得到了解决,我们还没有考虑将其向后移植到2.0.x分支。如果您想尝试使用2.1.0.rc1版本,您需要在构建中包含spring io repo-请参阅项目页面的“快速入门”部分-http://projects.spring.io/spring-hadoop/

相关问题