从EclipseIDE提交远程spark作业,出现paranamer错误

ht4b089n  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(371)

我正在使用eclipseide使用yarn客户机模式向集群提交远程spark作业。
但是在eclipseide中得到下面的错误消息。

  1. java.lang.NoClassDefFoundError: com/thoughtworks/paranamer/BytecodeReadingParanamer
  2. com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.<init>(BeanIntrospector.scala:40)
  3. com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.<clinit>(BeanIntrospector.scala)
  4. com.fasterxml.jackson.module.scala.introspect.ScalaPropertiesCollector.<init>(ScalaPropertiesCollector.scala:22)
  5. com.fasterxml.jackson.module.scala.introspect.ScalaClassIntrospector$.constructPropertyCollector(ScalaClassIntrospector.scala:24)
  6. com.fasterxml.jackson.databind.introspect.BasicClassIntrospector.collectProperties(BasicClassIntrospector.java:142)
  7. com.fasterxml.jackson.databind.introspect.BasicClassIntrospector.forSerialization(BasicClassIntrospector.java:68)
  8. com.fasterxml.jackson.databind.introspect.BasicClassIntrospector.forSerialization(BasicClassIntrospector.java:11)
  9. com.fasterxml.jackson.databind.SerializationConfig.introspect(SerializationConfig.java:490)
  10. com.fasterxml.jackson.databind.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:133)
  11. com.fasterxml.jackson.databind.SerializerProvider._createUntypedSerializer(SerializerProvider.java:873)
  12. com.fasterxml.jackson.databind.SerializerProvider._createAndCacheUntypedSerializer(SerializerProvider.java:833)
  13. com.fasterxml.jackson.databind.SerializerProvider.findValueSerializer(SerializerProvider.java:387)
  14. com.fasterxml.jackson.databind.SerializerProvider.findTypedValueSerializer(SerializerProvider.java:478)
  15. com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:97)
  16. com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:2718)
  17. com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2210)
  18. org.apache.spark.rdd.RDDOperationScope.toJson(RDDOperationScope.scala:51)
  19. org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:144)
  20. org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
  21. org.apache.spark.SparkContext.withScope(SparkContext.scala:714)
  22. org.apache.spark.SparkContext.textFile(SparkContext.scala:830)
  23. org.apache.spark.api.java.JavaSparkContext.textFile(JavaSparkContext.scala:181)
  24. com.nuevora.core.spark.commons.CommonFunctions.getJavaRDDFromFile(CommonFunctions.java:61)
  25. com.nuevora.core.spark.UpdateDataset.modifyInputDataset(UpdateDataset.java:103)
  26. com.nuevora.controllers.FormsValidatorServlet.service(FormsValidatorServlet.java:3070)
  27. javax.servlet.http.HttpServlet.service(HttpServlet.java:722)

执行器日志没有帮助。但job的最终状态是“成功”
需要帮助解决此错误吗

jv2fixgn

jv2fixgn1#

我通过添加cloudera发行版附带的paranamer jar解决了这个问题

相关问题