如何在spark 2.4.4中使用delta lake

l2osamch  于 2021-05-24  发布在  Spark
关注(0)|答案(0)|浏览(623)

我使用的是spark 2.4.4,当我进入pyspark shell时,我指定delta lake和jackson包如下:

pyspark --packages io.delta:delta-core_2.11:0.6.1,com.fasterxml.jackson.module:jackson-module-scala_2.11:2.6.7.1 --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"

但随后出现以下错误:
data.write.format(“delta”).save(“/tmp/delta table”)traceback(上次调用):file“”,第1行,在文件“/usr/hdp/current/spark2 client/python/pyspark/sql/readwriter.py”中,第738行,在save self.jwrite.save(path)file“/usr/hdp/current/spark2 client/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway”中,第1257行,在调用文件“/usr/hdp/current/spark2 client/python/pyspark/sql/utils.py”的第63行,在deco return f(*a,**kw)file“/usr/hdp/current/spark2 client/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py”的第328行,在get\u return\u value py4j.protocol.py4jjavaerror:调用o91.save时出错:java.lang.nosuchmethoderror:com.fasterxml.jackson.module.scala.experimental.scalaobjectmapper.com$fasterxml$jackson$module$scala$experimental$scalaobjectmapper$setter$com$fasterxml$jackson$module$scala$experimental$scalaobjectmapper$$map$eq(ljava/lang/class;)v位于com.fasterxml.jackson.module.scala.experimental.scalaobjectmapper$class.$init$(scalaobjectmapper。scala:331)在org.apache.spark.sql.delta.util.jsonutils$$anon$1上。scala:27)在org.apache.spark.sql.delta.util.jsonutils$(jsonutils。scala:27)位于org.apache.spark.sql.delta.util.jsonutils$(jsonutils.scala)org.apache.spark.sql.delta.deltaoperations$write$$anonfun$1.apply(deltaoperations。scala:58)在org.apache.spark.sql.delta.deltaoperations$write$$anonfun$1.apply(deltaoperations。scala:58)在scala.option.map(option。scala:146)在org.apache.spark.sql.delta.deltaoperations$write上。scala:58)在org.apache.spark.sql.delta.commands.writeintodelta$$anonfun$运行$1.apply(writeintodelta)。scala:66)在org.apache.spark.sql.delta.commands.writeintodelta$$anonfun$运行$1.apply(writeintodelta)。scala:64)在org.apache.spark.sql.delta.deltalog.withnewtransaction(deltalog。scala:188)在org.apache.spark.sql.delta.commands.writeintodelta.run(writeintodelta。scala:64)在org.apache.spark.sql.delta.sources.deltadatasource.createrelation(deltadatasource。scala:134)在org.apache.spark.sql.execution.datasources.saveintodatasourcecommand.run(saveintodatasourcecommand。scala:45)在org.apache.spark.sql.execution.command.executedcommandexec.sideeffectresult$lzycompute(命令。scala:70)在org.apache.spark.sql.execution.command.executedcommandexec.sideeffectresult(commands。scala:68)在org.apache.spark.sql.execution.command.executecommandexec.doexecute(commands。scala:86)在org.apache.spark.sql.execution.sparkplan$$anonfun$执行$1.apply(sparkplan。scala:131)在org.apache.spark.sql.execution.sparkplan$$anonfun$execute$1.apply(sparkplan。scala:127)在org.apache.spark.sql.execution.sparkplan$$anonfun$executequery$1.apply(sparkplan。scala:155)在org.apache.spark.rdd.rddoperationscope$.withscope(rddoperationscope。scala:151)在org.apache.spark.sql.execution.sparkplan.executequery(sparkplan。scala:152)在org.apache.spark.sql.execution.sparkplan.execute(sparkplan。scala:127)在org.apache.spark.sql.execution.queryexecution.tordd$lzycompute(queryexecution。scala:80)在org.apache.spark.sql.execution.queryexecution.tordd(查询执行)。scala:80)在org.apache.spark.sql.dataframewriter$$anonfun$runcommand$1.apply(dataframewriter。scala:676)在org.apache.spark.sql.dataframewriter$$anonfun$runcommand$1.apply(dataframewriter。scala:676)在org.apache.spark.sql.execution.sqlexecution$$anonfun$withnewexecutionid$1.apply(sqlexecution)。scala:78)位于org.apache.spark.sql.execution.sqlexecution$.withsqlconfpropagated(sqlexecution)。scala:125)位于org.apache.spark.sql.execution.sqlexecution$.withnewexecutionid(sqlexecution)。scala:73)在org.apache.spark.sql.dataframewriter.runcommand(dataframewriter。scala:676)位于org.apache.spark.sql.dataframewriter.savetov1source(dataframewriter。scala:285)位于org.apache.spark.sql.dataframewriter.save(dataframewriter。scala:271)位于org.apache.spark.sql.dataframewriter.save(dataframewriter。scala:229)在sun.reflect.nativemethodaccessorimpl.invoke0(本机方法)位于sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl)。java:62)在sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl。java:43)在java.lang.reflect.method.invoke(方法。java:498)在py4j.reflection.methodinvoker.invoke(methodinvoker。java:244)在py4j.reflection.reflectionengine.invoke(reflectionengine。java:357)在py4j.gateway.invoke(gateway。java:282)在py4j.commands.abstractcommand.invokemethod(abstractcommand。java:132)在py4j.commands.callcommand.execute(callcommand。java:79)在py4j.gatewayconnection.run(网关连接。java:238)在java.lang.thread.run(线程。java:748)

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题