databricks笔记本中的cosmos db写入问题

6qftjkof  于 2021-05-27  发布在  Spark
关注(0)|答案(1)|浏览(470)

根据databricks文件-https://docs.databricks.com/data/data-sources/azure/cosmosdb-connector.html，我下载了最新的azure-cosmosdb-spark库（azure-cosmosdb-spark_2.4.0_2.11-2.1.2-uber.jar）并放在dbfs的libraries位置。
当试图将Dataframe中的数据写入cosmos容器时，我得到以下错误，任何帮助都将不胜感激。
我的databricks运行时版本是：7.0（包括apachespark3.0.0和scala 2.12）
从笔记本导入：

  1. import java.time.LocalDateTime
  2. import java.time.format.DateTimeFormatter
  3. import org.apache.spark.sql.{DataFrame, SaveMode, SparkSession}
  4. import org.apache.spark.sql.types.{StructField, _}
  5. import org.apache.spark.sql.functions._
  6. import com.microsoft.azure.cosmosdb.spark.schema._
  7. import com.microsoft.azure.cosmosdb.spark.CosmosDBSpark
  8. import com.microsoft.azure.cosmosdb.spark.config._
  10. val dtcCANWrite = Config(Map(
  11.   "Endpoint" -> "NOT DISPLAYED",
  12.   "Masterkey" -> "NOT DISPLAYED",
  13.   "Database" -> "NOT DISPLAYED",
  14.   "Collection" -> "NOT DISPLAYED",
  15.   "preferredRegions" -> "NOT DISPLAYED",
  16.   "Upsert" -> "true"
  17. ))
  19. distinctCANDF.write.mode(SaveMode.Append).cosmosDB(dtcCANWrite)

错误：

  1. at com.microsoft.azure.cosmosdb.spark.config.CosmosDBConfigBuilder.<init>(CosmosDBConfigBuilder.scala:31)
  2.     at com.microsoft.azure.cosmosdb.spark.config.Config$.apply(Config.scala:259)
  3.     at com.microsoft.azure.cosmosdb.spark.config.Config$.apply(Config.scala:240)
  4.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command-3649834446724317:7)
  5.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command-3649834446724317:69)
  6.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command-3649834446724317:71)
  7.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command-3649834446724317:73)
  8.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command-3649834446724317:75)
  9.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command-3649834446724317:77)
  10.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command-3649834446724317:79)
  11.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command-3649834446724317:81)
  12.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command-3649834446724317:83)
  13.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command-3649834446724317:85)
  14.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command-3649834446724317:87)
  15.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw$$iw$$iw$$iw.<init>(command-3649834446724317:89)
  16.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw$$iw$$iw.<init>(command-3649834446724317:91)
  17.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw$$iw.<init>(command-3649834446724317:93)
  18.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw.<init>(command-3649834446724317:95)
  19.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw.<init>(command-3649834446724317:97)
  20.     at line6d80624d7a774601af6eb962eb59453253.$read.<init>(command-3649834446724317:99)
  21.     at line6d80624d7a774601af6eb962eb59453253.$read$.<init>(command-3649834446724317:103)
  22.     at line6d80624d7a774601af6eb962eb59453253.$read$.<clinit>(command-3649834446724317)
  23.     at line6d80624d7a774601af6eb962eb59453253.$eval$.$print$lzycompute(<notebook>:7)
  24.     at line6d80624d7a774601af6eb962eb59453253.$eval$.$print(<notebook>:6)
  25.     at line6d80624d7a774601af6eb962eb59453253.$eval.$print(<notebook>)
  26.     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  27.     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  28.     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  29.     at java.lang.reflect.Method.invoke(Method.java:498)
  30.     at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:745)
  31.     at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1021)
  32.     at scala.tools.nsc.interpreter.IMain.$anonfun$interpret$1(IMain.scala:574)
  33.     at scala.reflect.internal.util.ScalaClassLoader.asContext(ScalaClassLoader.scala:41)
  34.     at scala.reflect.internal.util.ScalaClassLoader.asContext$(ScalaClassLoader.scala:37)
  35.     at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:41)
  36.     at scala.tools.nsc.interpreter.IMain.loadAndRunReq$1(IMain.scala:573)
  37.     at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:600)
  38.     at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:570)
  39.     at com.databricks.backend.daemon.driver.DriverILoop.execute(DriverILoop.scala:215)
  40.     at com.databricks.backend.daemon.driver.ScalaDriverLocal.$anonfun$repl$1(ScalaDriverLocal.scala:202)
  41.     at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
  42.     at com.databricks.backend.daemon.driver.DriverLocal$TrapExitInternal$.trapExit(DriverLocal.scala:714)
  43.     at com.databricks.backend.daemon.driver.DriverLocal$TrapExit$.apply(DriverLocal.scala:667)
  44.     at com.databricks.backend.daemon.driver.ScalaDriverLocal.repl(ScalaDriverLocal.scala:202)
  45.     at com.databricks.backend.daemon.driver.DriverLocal.$anonfun$execute$10(DriverLocal.scala:396)
  46.     at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:238)
  47.     at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
  48.     at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:233)
  49.     at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:230)
  50.     at com.databricks.backend.daemon.driver.DriverLocal.withAttributionContext(DriverLocal.scala:49)
  51.     at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:275)
  52.     at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:268)
  53.     at com.databricks.backend.daemon.driver.DriverLocal.withAttributionTags(DriverLocal.scala:49)
  54.     at com.databricks.backend.daemon.driver.DriverLocal.execute(DriverLocal.scala:373)
  55.     at com.databricks.backend.daemon.driver.DriverWrapper.$anonfun$tryExecutingCommand$1(DriverWrapper.scala:653)
  56.     at scala.util.Try$.apply(Try.scala:213)
  57.     at com.databricks.backend.daemon.driver.DriverWrapper.tryExecutingCommand(DriverWrapper.scala:645)
  58.     at com.databricks.backend.daemon.driver.DriverWrapper.getCommandOutputAndError(DriverWrapper.scala:486)
  59.     at com.databricks.backend.daemon.driver.DriverWrapper.executeCommand(DriverWrapper.scala:598)
  60.     at com.databricks.backend.daemon.driver.DriverWrapper.runInnerLoop(DriverWrapper.scala:391)
  61.     at com.databricks.backend.daemon.driver.DriverWrapper.runInner(DriverWrapper.scala:337)
  62.     at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:219)
  63.     at java.lang.Thread.run(Thread.java:748)
  64. Caused by: java.lang.ClassNotFoundException: scala.Product$class
  65.     at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
  66.     at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
  67.     at com.databricks.backend.daemon.driver.ClassLoaders$LibraryClassLoader.loadClass(ClassLoaders.scala:151)
  68.     at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
  69.     at com.microsoft.azure.cosmosdb.spark.config.CosmosDBConfigBuilder.<init>(CosmosDBConfigBuilder.scala:31)
  70.     at com.microsoft.azure.cosmosdb.spark.config.Config$.apply(Config.scala:259)
  71.     at com.microsoft.azure.cosmosdb.spark.config.Config$.apply(Config.scala:240)
  72.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command-3649834446724317:7)
  73.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command-3649834446724317:69)
  74.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command-3649834446724317:71)
  75.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command-3649834446724317:73)
  76.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command-3649834446724317:75)
  77.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command-3649834446724317:77)
  78.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command-3649834446724317:79)
  79.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command-3649834446724317:81)
  80.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command-3649834446724317:83)
  81.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command-3649834446724317:85)
  82.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command-3649834446724317:87)
  83.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw$$iw$$iw$$iw.<init>(command-3649834446724317:89)
  84.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw$$iw$$iw.<init>(command-3649834446724317:91)
  85.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw$$iw.<init>(command-3649834446724317:93)
  86.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw$$iw.<init>(command-3649834446724317:95)
  87.     at line6d80624d7a774601af6eb962eb59453253.$read$$iw.<init>(command-3649834446724317:97)
  88.     at line6d80624d7a774601af6eb962eb59453253.$read.<init>(command-3649834446724317:99)
  89.     at line6d80624d7a774601af6eb962eb59453253.$read$.<init>(command-3649834446724317:103)
  90.     at line6d80624d7a774601af6eb962eb59453253.$read$.<clinit>(command-3649834446724317)
  91.     at line6d80624d7a774601af6eb962eb59453253.$eval$.$print$lzycompute(<notebook>:7)
  92.     at line6d80624d7a774601af6eb962eb59453253.$eval$.$print(<notebook>:6)
  93.     at line6d80624d7a774601af6eb962eb59453253.$eval.$print(<notebook>)
  94.     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  95.     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  96.     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  97.     at java.lang.reflect.Method.invoke(Method.java:498)
  98.     at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:745)
  99.     at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1021)
  100.     at scala.tools.nsc.interpreter.IMain.$anonfun$interpret$1(IMain.scala:574)
  101.     at scala.reflect.internal.util.ScalaClassLoader.asContext(ScalaClassLoader.scala:41)
  102.     at scala.reflect.internal.util.ScalaClassLoader.asContext$(ScalaClassLoader.scala:37)
  103.     at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:41)
  104.     at scala.tools.nsc.interpreter.IMain.loadAndRunReq$1(IMain.scala:573)
  105.     at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:600)
  106.     at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:570)
  107.     at com.databricks.backend.daemon.driver.DriverILoop.execute(DriverILoop.scala:215)
  108.     at com.databricks.backend.daemon.driver.ScalaDriverLocal.$anonfun$repl$1(ScalaDriverLocal.scala:202)
  109.     at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
  110.     at com.databricks.backend.daemon.driver.DriverLocal$TrapExitInternal$.trapExit(DriverLocal.scala:714)
  111.     at com.databricks.backend.daemon.driver.DriverLocal$TrapExit$.apply(DriverLocal.scala:667)
  112.     at com.databricks.backend.daemon.driver.ScalaDriverLocal.repl(ScalaDriverLocal.scala:202)
  113.     at com.databricks.backend.daemon.driver.DriverLocal.$anonfun$execute$10(DriverLocal.scala:396)
  114.     at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:238)
  115.     at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
  116.     at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:233)
  117.     at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:230)
  118.     at com.databricks.backend.daemon.driver.DriverLocal.withAttributionContext(DriverLocal.scala:49)
  119.     at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:275)
  120.     at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:268)
  121.     at com.databricks.backend.daemon.driver.DriverLocal.withAttributionTags(DriverLocal.scala:49)
  122.     at com.databricks.backend.daemon.driver.DriverLocal.execute(DriverLocal.scala:373)
  123.     at com.databricks.backend.daemon.driver.DriverWrapper.$anonfun$tryExecutingCommand$1(DriverWrapper.scala:653)
  124.     at scala.util.Try$.apply(Try.scala:213)
  125.     at com.databricks.backend.daemon.driver.DriverWrapper.tryExecutingCommand(DriverWrapper.scala:645)
  126.     at com.databricks.backend.daemon.driver.DriverWrapper.getCommandOutputAndError(DriverWrapper.scala:486)
  127.     at com.databricks.backend.daemon.driver.DriverWrapper.executeCommand(DriverWrapper.scala:598)
  128.     at com.databricks.backend.daemon.driver.DriverWrapper.runInnerLoop(DriverWrapper.scala:391)
  129.     at com.databricks.backend.daemon.driver.DriverWrapper.runInner(DriverWrapper.scala:337)
  130.     at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:219)
  131.     at java.lang.Thread.run(Thread.java:748)
apache-sparkapache-spark-sqlazure-cosmosdb

1条答案

按热度按时间
wn9m85ua

wn9m85ua1#

谢谢雷恩·拉尔的建议。所以问题来自版本冲突。
解决方案的版本被降级。在这种情况下，您可以从spark 3.0.0、scala 2.12降级到spark 2.4.4、scala 2.11。

赞(0）分享  回复(0）举报  2021-05-27
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com