在spark 2.4(hdinsight)中使用delta lake源时出错

hjqgdpho 于 2021-05-27 发布在 Spark

关注(0)|答案(1)|浏览(640)

在下面的错误中，相同的代码在databricks中工作，但在hdinsight中不工作。我已经在类路径中添加了delta库和hadoopazure库。

io.delta:delta-core_2.11:0.5.0,org.apache.hadoop:hadoop-azure:3.1.3

ERROR ApplicationMaster [Driver]: User class threw exception: com.google.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: com/fasterxml/jackson/module/scala/experimental/ScalaObjectMapper$class
com.google.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: com/fasterxml/jackson/module/scala/experimental/ScalaObjectMapper$class
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2049)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3953)
    at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4873)
    at org.apache.spark.sql.delta.DeltaLog$.apply(DeltaLog.scala:740)
    at org.apache.spark.sql.delta.DeltaLog$.forTable(DeltaLog.scala:712)
    at org.apache.spark.sql.delta.sources.DeltaDataSource.createRelation(DeltaDataSource.scala:169)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:318)
    at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)
    at io.delta.tables.DeltaTable$.forPath(DeltaTable.scala:635)

apache-spark delta-lake azure-hdinsight

来源：https://stackoverflow.com/questions/63081874/error-while-using-the-delta-lake-source-in-spark-2-4-hdinsight

1条答案

按热度按时间

nwlqm0z11#

使用hdinsight打包的jackson json库的版本与spark、deltalake使用的版本之间存在冲突
有两种方法可以解决这个问题
将jackson json 2.6.7版本依赖项打包到应用程序中（maven shade插件或scala程序集）
或
如果您使用的是jupyter笔记本，请设置以下spark配置

{"conf":
 {"spark.jars.packages": "io.delta:delta-core_2.11:0.5.0", 
    "spark.driver.extraClassPath":
     "${PATH}/jackson-module-scala_2.11-2.6.7.1.jar;${PATH}/jackson-annotations-2.6.7.jar;
      ${PATH}/jackson-core-2.6.7.jar;
      ${PATH}/jackson-databind-2.6.7.1.jar;
      ${PATH}/jackson-module-paranamer-2.6.7.jar",
   "spark.executor.extraClassPath":
     "${PATH}/jackson-module-scala_2.11-2.6.7.1.jar;${PATH}/jackson-annotations-2.6.7.jar;
      ${PATH}/jackson-core-2.6.7.jar;${PATH}/jackson-databind-2.6.7.1.jar;
      ${PATH}/jackson-module-paranamer-2.6.7.jar",
   "spark.driver.userClassPathFirst":true}}

赞(0）回复(0）举报 2021-05-27

我来回答

在spark 2.4(hdinsight)中使用delta lake源时出错

1条答案

相关问题

热门标签

最新问答