我正试图通过oozie运行hadoop作业。作业将数据上传到aws中的dynamodb。因此,我使用 AmazonDynamoDBClient
. 我在异径管中遇到以下例外情况:
2016-06-14 10:30:52,997 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoSuchMethodError: com.fasterxml.jackson.core.JsonFactory.requiresPropertyOrdering()Z
at com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:458)
at com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:379)
at com.amazonaws.util.json.Jackson.<clinit>(Jackson.java:32)
at com.amazonaws.internal.config.InternalConfig.loadfrom(InternalConfig.java:233)
at com.amazonaws.internal.config.InternalConfig.load(InternalConfig.java:251)
at com.amazonaws.internal.config.InternalConfig$Factory.<clinit>(InternalConfig.java:308)
at com.amazonaws.util.VersionInfoUtils.userAgent(VersionInfoUtils.java:139)
at com.amazonaws.util.VersionInfoUtils.initializeUserAgent(VersionInfoUtils.java:134)
at com.amazonaws.util.VersionInfoUtils.getUserAgent(VersionInfoUtils.java:95)
at com.amazonaws.ClientConfiguration.<clinit>(ClientConfiguration.java:42)
at com.amazonaws.PredefinedClientConfigurations.dynamoDefault(PredefinedClientConfigurations.java:38)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.<init>(AmazonDynamoDBClient.java:292)
at com.mypackage.UploadDataToDynamoDBMR$DataUploaderReducer.setup(UploadDataToDynamoDBMR.java:396)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:168)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
我使用了一个fat jar来打包所有依赖项,并将jar复制到oozie的 lib
目录。
我还使用pom中的依赖关系管理将fasterxml-jackson依赖关系绑定到2.4.1(由aws dynamodb sdk使用)。但是,当执行发生在reducer上时,fasterxml jackson的其他版本会首先出现在类路径上(我相信是这样)。
我还从dynamodb和aws sdk中排除了jackson依赖性。
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-dynamodb</artifactId>
<version>1.10.11</version>
<exclusions>
<exclusion>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-core</artifactId>
<version>1.10.11</version>
<exclusions>
<exclusion>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>
如何确保我的jar是mappers和reducer中类路径上的第一个jar?我尝试了此页上的建议,并将以下属性添加到作业的配置xml中:
<property>
<name>oozie.launcher.mapreduce.user.classpath.first</name>
<value>true</value>
</property>
但这没有帮助。
有什么建议吗?
1条答案
按热度按时间oyjwcjzk1#
您是否已将jar复制到lib workflow.xml旁边的lib文件夹或sharelib中?
检查hadoop发行版使用的jackson版本,并尝试在任何地方使用该版本的jackson。另外,可能值得检查的是,类路径上没有其他jackson jar。从异常情况看,hadoop似乎试图调用一个方法:
com.fasterxml.jackson.core.jsonfactory.requirePropertyOrdering
这个方法是在jacksonversion2.3中引入的,所以可能有一个更老的jackson版本在那里。