：初始化失败，无法创建inputinitializermanager

6g8kf2rb  于 2021-06-02  发布在  Hadoop
关注(0)|答案(0)|浏览(795)

我正在使用来自amazonwebservices的emr服务，并试图在我构建的外部表上运行count查询。表的数据存储在mongodb中，表是hive中的一个外部表。我试图运行的查询是

  1. select user_id, count (*) from myTable group by user_id;

我可以从mytable中查询select*，但不能执行任何其他查询。当我尝试获取此错误时：

  1. ----------------------------------------------------------------------------------------------
  2.         VERTICES      MODE        STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED
  3. ----------------------------------------------------------------------------------------------
  4. Map 1            container        FAILED     -1          0        0       -1       0       0
  5. Reducer 2        container        KILLED      1          0        0        1       0       0
  6. ----------------------------------------------------------------------------------------------
  7. VERTICES: 00/02  [>>--------------------------] 0%    ELAPSED TIME: 0.03 s
  8. ----------------------------------------------------------------------------------------------
  9. Status: Failed
  10. Vertex failed, vertexName=Map 1, vertexId=vertex_1476467351971_0008_2_00, diagnostics=[Vertex vertex_1476467351971_0008_2_00 [Map 1] killed/failed due to:INIT_FAILURE, Fail to create InputInitializerManager, org.apache.tez.dag.api.TezReflectionException: Unable to instantiate class with 1 arguments: org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator
  11.         at org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:70)
  12.         at org.apache.tez.common.ReflectionUtils.createClazzInstance(ReflectionUtils.java:89)
  13.         at org.apache.tez.dag.app.dag.RootInputInitializerManager$1.run(RootInputInitializerManager.java:151)
  14.         at org.apache.tez.dag.app.dag.RootInputInitializerManager$1.run(RootInputInitializerManager.java:148)
  15.         at java.security.AccessController.doPrivileged(Native Method)
  16.         at javax.security.auth.Subject.doAs(Subject.java:422)
  17.         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
  18.         at org.apache.tez.dag.app.dag.RootInputInitializerManager.createInitializer(RootInputInitializerManager.java:148)
  19.         at org.apache.tez.dag.app.dag.RootInputInitializerManager.runInputInitializers(RootInputInitializerManager.java:121)
  20.         at org.apache.tez.dag.app.dag.impl.VertexImpl.setupInputInitializerManager(VertexImpl.java:3986)
  21.         at org.apache.tez.dag.app.dag.impl.VertexImpl.access$3100(VertexImpl.java:204)
  22.         at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.handleInitEvent(VertexImpl.java:2818)
  23.         at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2765)
  24.         at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2747)
  25.         at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
  26.         at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
  27.         at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
  28.         at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
  29.         at org.apache.tez.state.StateMachineTez.doTransition(StateMachineTez.java:59)
  30.         at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1888)
  31.         at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:203)
  32.         at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2242)
  33.         at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2228)
  34.         at org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
  35.         at org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:114)
  36.         at java.lang.Thread.run(Thread.java:745)
  37. Caused by: java.lang.reflect.InvocationTargetException
  38.         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  39.         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  40.         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  41.         at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  42.         at org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:68)
  43.         ... 25 more
  44. Caused by: java.lang.RuntimeException: Failed to load plan: hdfs://ip-172-31-33-88.ec2.internal:8020/tmp/hive/hadoop/8c1eca9a-84ba-4d79-b39c-e633f1c6a646/hive_2016-10-14_18-57-38_728_4862537458701624850-1/hadoop/_tez_scratch_dir/157c0e27-0bfa-4acf-b0e9-b2fed595a8de/map.xml: org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find class: com.mongodb.hadoop.hive.input.HiveMongoInputFormat
  45. Serialization trace:
  46. inputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc)
  47. aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork)
  48.         at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:451)
  49.         at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:298)
  50.         at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.<init>(HiveSplitGenerator.java:131)
  51.         ... 30 more
  52. Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find class: com.mongodb.hadoop.hive.input.HiveMongoInputFormat
  53. Serialization trace:
  54. inputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc)
  55. aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork)
  56.         at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:156)
  57.         at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)
  58.         at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)
  59.         at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClass(SerializationUtilities.java:180)
  60.         at org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:326)
  61.         at org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:314)
  62.         at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:759)
  63.         at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObjectOrNull(SerializationUtilities.java:198)
  64.         at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:132)
  65.         at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
  66.         at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
  67.         at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:175)
  68.         at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:161)
  69.         at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)
  70.         at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
  71.         at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:213)
  72.         at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
  73.         at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
  74.         at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:686)
  75.         at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:205)
  76.         at org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializeObjectByKryo(SerializationUtilities.java:583)
  77.         at org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializePlan(SerializationUtilities.java:492)
  78.         at org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializePlan(SerializationUtilities.java:469)
  79.         at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:411)
  80.         ... 32 more
  81. Caused by: java.lang.ClassNotFoundException: com.mongodb.hadoop.hive.input.HiveMongoInputFormat
  82.         at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
  83.         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  84.         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
  85.         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  86.         at java.lang.Class.forName0(Native Method)
  87.         at java.lang.Class.forName(Class.java:348)
  88.         at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:154)
  89.         ... 55 more
  90. ]
  91. Vertex killed, vertexName=Reducer 2, vertexId=vertex_1476467351971_0008_2_01, diagnostics=[Vertex received Kill in NEW state., Vertex vertex_1476467351971_0008_2_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]
  92. DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1
  93. FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1476467351971_0008_2_00, diagnostics=[Vertex vertex_1476467351971_0008_2_00 [Map 1] killed/failed due to:INIT_FAILURE, Fail to create InputInitializerManager, org.apache.tez.dag.api.TezReflectionException: Unable to instantiate class with 1 arguments: org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator
  94.         at org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:70)
  95.         at org.apache.tez.common.ReflectionUtils.createClazzInstance(ReflectionUtils.java:89)
  96.         at org.apache.tez.dag.app.dag.RootInputInitializerManager$1.run(RootInputInitializerManager.java:151)
  97.         at org.apache.tez.dag.app.dag.RootInputInitializerManager$1.run(RootInputInitializerManager.java:148)
  98.         at java.security.AccessController.doPrivileged(Native Method)
  99.         at javax.security.auth.Subject.doAs(Subject.java:422)
  100.         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
  101.         at org.apache.tez.dag.app.dag.RootInputInitializerManager.createInitializer(RootInputInitializerManager.java:148)
  102.         at org.apache.tez.dag.app.dag.RootInputInitializerManager.runInputInitializers(RootInputInitializerManager.java:121)
  103.         at org.apache.tez.dag.app.dag.impl.VertexImpl.setupInputInitializerManager(VertexImpl.java:3986)
  104.         at org.apache.tez.dag.app.dag.impl.VertexImpl.access$3100(VertexImpl.java:204)
  105.         at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.handleInitEvent(VertexImpl.java:2818)
  106.         at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2765)
  107.         at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2747)
  108.         at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
  109.         at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
  110.         at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
  111.         at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
  112.         at org.apache.tez.state.StateMachineTez.doTransition(StateMachineTez.java:59)
  113.         at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1888)
  114.         at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:203)
  115.         at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2242)
  116.         at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2228)
  117.         at org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
  118.         at org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:114)
  119.         at java.lang.Thread.run(Thread.java:745)
  120. Caused by: java.lang.reflect.InvocationTargetException
  121.         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  122.         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  123.         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  124.         at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  125.         at org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:68)
  126.         ... 25 more
  127. Caused by: java.lang.RuntimeException: Failed to load plan: hdfs://ip-172-31-33-88.ec2.internal:8020/tmp/hive/hadoop/8c1eca9a-84ba-4d79-b39c-e633f1c6a646/hive_2016-10-14_18-57-38_728_4862537458701624850-1/hadoop/_tez_scratch_dir/157c0e27-0bfa-4acf-b0e9-b2fed595a8de/map.xml: org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find class: com.mongodb.hadoop.hive.input.HiveMongoInputFormat
  128. Serialization trace:
  129. inputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc)
  130. aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork)
  131.         at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:451)
  132.         at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:298)
  133.         at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.<init>(HiveSplitGenerator.java:131)
  134.         ... 30 more
  135. Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find class: com.mongodb.hadoop.hive.input.HiveMongoInputFormat
  136. Serialization trace:
  137. inputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc)
  138. aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork)
  139.         at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:156)
  140.         at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)
  141.         at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)
  142.         at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClass(SerializationUtilities.java:180)
  143.         at org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:326)
  144.         at org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:314)
  145.         at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:759)
  146.         at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObjectOrNull(SerializationUtilities.java:198)
  147.         at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:132)
  148.         at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
  149.         at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
  150.         at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:175)
  151.         at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:161)
  152.         at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)
  153.         at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
  154.         at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:213)
  155.         at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
  156.         at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
  157.         at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:686)
  158.         at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:205)
  159.         at org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializeObjectByKryo(SerializationUtilities.java:583)
  160.         at org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializePlan(SerializationUtilities.java:492)
  161.         at org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializePlan(SerializationUtilities.java:469)
  162.         at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:411)
  163.         ... 32 more
  164. Caused by: java.lang.ClassNotFoundException: com.mongodb.hadoop.hive.input.HiveMongoInputFormat
  165.         at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
  166.         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  167.         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
  168.         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  169.         at java.lang.Class.forName0(Native Method)
  170.         at java.lang.Class.forName(Class.java:348)
  171.         at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:154)
  172.         ... 55 more
  173. ]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1476467351971_0008_2_01, diagnostics=[Vertex received Kill in NEW state., Vertex vertex_1476467351971_0008_2_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1
  174. hive>

我使用的是aws中的单节点emr集群，配置为“核心hadoop”，但是使用三节点集群时出现了相同的错误。我已经将mongo hadoop核心、mongo hadoop hive、mongo java驱动程序添加到主节点。。这些是我用的jar：

  1. mongo-hadoop-hive-2.0.1.jar
  2. mongo-java-driver-3.3.0.jar 
  3. remotecontent?filepath=org%2Fmongodb%2Fmongo-hadoop%2Fmongo-hadoop-core%2F2.0.1%2Fmongo-hadoop-core-2.0.1.jar

这些jar是从maven下载的。
以前在stackoverflow上也曾提出过类似的问题，通过为apachetez0.8.4添加src并构建它就解决了这个问题。不过，至少根据我们选择的emr配置，apachetez0.8.4已经在集群中了。
我们使用的是Hadoop2.7.2和Hive2.1.0。

hadoopHivemongodbamazon-ec2emr

暂无答案！

目前还没有任何答案，快来回答吧！

我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com