hadoop java.io.InvalidClassException:我的意思是说，如果你有一个很好的应用程序，你就可以使用它,无法创建示例

cl25kdpy  于 2022-12-11  发布在  Hadoop
关注(0)|答案(1)|浏览(441)

我是Spark的新手，我的spark master在GCP Kubernetes标准集群上。我正在使用gremlin客户端来触发spark作业，在我的本地机器配置上，它工作正常，作业完成，但在Kubernetes上，我看到下面的错误。

Gremlin查询和错误

  1. ==>tinkerpop.hadoop activated
  2. gremlin> :plugin use tinkerpop.spark
  3. ==>tinkerpop.spark activated
  4. gremlin> graph = GraphFactory.open('/opt/janusgraph-0.6.2/conf/hadoop-graph/read-hbase-standalone-cluster-stg-bigtable-spark.properties')
  5. ==>hadoopgraph[hbaseinputformat->nulloutputformat]
  6. gremlin> 
  7. gremlin> g = graph.traversal().withComputer(SparkGraphComputer)
  8. ==>graphtraversalsource[hadoopgraph[hbaseinputformat->nulloutputformat], sparkgraphcomputer]
  9. gremlin> g.V().count()
  10. 14:01:48 WARN  org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer  - class org.apache.hadoop.mapreduce.lib.output.NullOutputFormat does not implement PersistResultGraphAware and thus, persistence options are unknown -- assuming all options are possible
  11. 14:02:03 WARN  org.apache.spark.scheduler.TaskSetManager  - Lost task 6.0 in stage 0.0 (TID 6, XX.XX.XX.XX, executor 0): java.io.InvalidClassException: org.apache.tinkerpop.gremlin.hadoop.structure.HadoopConfiguration; unable to create instance
  12.         at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2198)
  13.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
  14.         at java.base/java.io.ObjectInputStream.readArray(ObjectInputStream.java:2117)
  15.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1658)
  16.         at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
  17.         at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
  18.         at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
  19.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
  20.         at java.base/java.io.ObjectInputStream.readArray(ObjectInputStream.java:2117)
  21.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1658)
  22.         at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
  23.         at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
  24.         at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
  25.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
  26.         at java.base/java.io.ObjectInputStream.readArray(ObjectInputStream.java:2117)
  27.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1658)
  28.         at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
  29.         at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
  30.         at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
  31.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
  32.         at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
  33.         at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
  34.         at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
  35.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
  36.         at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
  37.         at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
  38.         at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
  39.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
  40.         at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:488)
  41.         at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:446)
  42.         at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:488)
  43.         at jdk.internal.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
  44.         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  45.         at java.base/java.lang.reflect.Method.invoke(Method.java:566)
  46.         at java.base/java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1046)
  47.         at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2340)
  48.         at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
  49.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
  50.         at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
  51.         at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
  52.         at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
  53.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
  54.         at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
  55.         at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
  56.         at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
  57.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
  58.         at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:488)
  59.         at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:446)
  60.         at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:488)
  61.         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  62.         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  63.         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  64.         at java.base/java.lang.reflect.Method.invoke(Method.java:566)
  65.         at java.base/java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1046)
  66.         at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2340)
  67.         at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
  68.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
  69.         at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
  70.         at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
  71.         at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
  72.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
  73.         at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
  74.         at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
  75.         at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
  76.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
  77.         at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:488)
  78.         at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:446)
  79.         at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
  80.         at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
  81.         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:85)
  82.         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
  83.         at org.apache.spark.scheduler.Task.run(Task.scala:127)
  84.         at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
  85.         at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
  86.         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
  87.         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  88.         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  89.         at java.base/java.lang.Thread.run(Thread.java:829)
  90. Caused by: java.lang.reflect.InvocationTargetException
  91.         at jdk.internal.reflect.GeneratedSerializationConstructorAccessor270.newInstance(Unknown Source)
  92.         at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
  93.         at java.base/java.io.ObjectStreamClass.newInstance(ObjectStreamClass.java:963)
  94.         at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2195)
  95.         ... 77 more
  96. Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.commons.configuration2.interpol.ConfigurationInterpolator
  97.         at org.apache.commons.configuration2.AbstractConfiguration.installDefaultInterpolator(AbstractConfiguration.java:455)
  98.         at org.apache.commons.configuration2.AbstractConfiguration.<init>(AbstractConfiguration.java:132)
  99.         ... 81 more
  101. 14:02:03 ERROR org.apache.spark.scheduler.TaskSetManager  - Task 8 in stage 0.0 failed 4 times; aborting job
  102. org.apache.spark.SparkException: Job aborted due to stage failure: Task 8 in stage 0.0 failed 4 times, most recent failure: Lost task 8.3 in stage 0.0 (TID 32, 10.12.11.7, executor 1): java.io.InvalidClassException: org.apache.tinkerpop.gremlin.hadoop.structure.HadoopConfiguration; unable to create instance
  103.         at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2198)
  104.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
  105.         at java.base/java.io.ObjectInputStream.readArray(ObjectInputStream.java:2117)
  106.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1658)
  107.         at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
  108.         at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
  109.         at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
  110.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
  111.         at java.base/java.io.ObjectInputStream.readArray(ObjectInputStream.java:2117)
  112.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1658)
  113.         at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
  114.         at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
  115.         at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
  116.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
  117.         at java.base/java.io.ObjectInputStream.readArray(ObjectInputStream.java:2117)
  118.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1658)
  119.         at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
  120.         at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
  121.         at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
  122.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
  123.         at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
  124.         at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
  125.         at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
  126.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
  127.         at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
  128.         at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
  129.         at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
  130.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
  131.         at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:488)
  132.         at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:446)
  133.         at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:488)
  134.         at jdk.internal.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
  135.         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  136.         at java.base/java.lang.reflect.Method.invoke(Method.java:566)
  137.         at java.base/java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1046)
  138.         at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2340)
  139.         at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
  140.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
  141.         at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
  142.         at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
  143.         at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
  144.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
  145.         at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
  146.         at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
  147.         at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
  148.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
  149.         at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:488)
  150.         at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:446)
  151.         at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:488)
  152.         at jdk.internal.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
  153.         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  154.         at java.base/java.lang.reflect.Method.invoke(Method.java:566)
  155.         at java.base/java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1046)
  156.         at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2340)
  157.         at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
  158.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
  159.         at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
  160.         at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
  161.         at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
  162.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
  163.         at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
  164.         at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
  165.         at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
  166.         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
  167.         at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:488)
  168.         at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:446)
  169.         at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
  170.         at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
  171.         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:85)
  172.         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
  173.         at org.apache.spark.scheduler.Task.run(Task.scala:127)
  174.         at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
  175.         at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
  176.         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
  177.         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  178.         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  179.         at java.base/java.lang.Thread.run(Thread.java:829)
  180. Caused by: java.lang.reflect.InvocationTargetException
  181.         at jdk.internal.reflect.GeneratedSerializationConstructorAccessor238.newInstance(Unknown Source)
  182.         at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
  183.         at java.base/java.io.ObjectStreamClass.newInstance(ObjectStreamClass.java:963)
  184.         at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2195)
  185.         ... 76 more
  186. Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.commons.configuration2.interpol.ConfigurationInterpolator
  187.         at org.apache.commons.configuration2.AbstractConfiguration.installDefaultInterpolator(AbstractConfiguration.java:455)
  188.         at org.apache.commons.configuration2.AbstractConfiguration.<init>(AbstractConfiguration.java:132)
  189.         ... 80 more

配置文件--〉read-hbase-standalone-cluster-stg-bigtable-spark.properties

  1. #
  2. # Licensed under the Apache License, Version 2.0 (the "License");
  3. # you may not use this file except in compliance with the License.
  4. # You may obtain a copy of the License at
  5. #
  6. #      http://www.apache.org/licenses/LICENSE-2.0
  7. #
  8. # Unless required by applicable law or agreed to in writing, software
  9. # distributed under the License is distributed on an "AS IS" BASIS,
  10. # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  11. # See the License for the specific language governing permissions and
  12. # limitations under the License.
  14. #
  15. # Hadoop Graph Configuration
  16. #
  17. gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph
  18. gremlin.hadoop.graphReader=org.janusgraph.hadoop.formats.hbase.HBaseInputFormat
  19. gremlin.hadoop.graphWriter=org.apache.hadoop.mapreduce.lib.output.NullOutputFormat
  21. gremlin.hadoop.jarsInDistributedCache=true
  22. gremlin.hadoop.inputLocation=none
  23. gremlin.hadoop.outputLocation=output
  24. gremlin.spark.persistContext=true
  26. #
  27. # JanusGraph HBase InputFormat configuration
  28. #
  29. #janusgraphmr.ioformat.conf.storage.backend=hbase
  30. #janusgraphmr.ioformat.conf.storage.hostname=localhost
  31. #janusgraphmr.ioformat.conf.storage.port=8586
  32. #janusgraphmr.ioformat.conf.storage.hbase.table=janusgraph
  33. janusgraphmr.ioformat.conf.storage.backend=hbase
  34. janusgraphmr.ioformat.conf.storage.hbase.ext.hbase.client.connection.impl=com.google.cloud.bigtable.hbase2_x.BigtableConnection
  35. janusgraphmr.ioformat.conf.storage.hbase.ext.google.bigtable.project.id=**********
  36. janusgraphmr.ioformat.conf.storage.hbase.ext.google.bigtable.instance.id=*********
  37. janusgraphmr.ioformat.conf.storage.hbase.table=**********
  38. janusgraphmr.ioformat.conf.storage.hbase.ext.hbase.regionsizecalculator.enable=false
  40. # This defines the indexing backend configuration used while writing data to JanusGraph.
  41. janusgraphmr.ioformat.conf.index.search.backend=elasticsearch
  42. janusgraphmr.ioformat.conf.index.search.hostname=**********
  44. #
  45. # SparkGraphComputer Configuration
  46. #
  47. spark.master=spark://spark-master:7077
  48. spark.executor.memory=1g
  49. spark.executor.extraClassPath=/opt/janusgraph-0.6.2/lib/*
  50. spark.serializer=org.apache.spark.serializer.KryoSerializer
  51. spark.kryo.registrator=org.janusgraph.hadoop.serialize.JanusGraphKryoRegistrator```
hadoop

1条答案

按热度按时间
jfgube3f

jfgube3f1#

参考gremlin参考配置文件中提到的https://li-boxuan.medium.com/configure-janusgraph-0-6-0-for-spark-64bd7628fb77 blog. spark.executor.extraClassPath=/opt/janusgraph-0.6.2/lib/*，这些库应该同时存在于主节点和工作节点中。

赞(0）分享  回复(0）举报  2022-12-11
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com