错误：bulkcopytosqldb不是org.apache.spark.sql.dataframewriter的成员

inb24sb2 于 2021-05-24 发布在 Spark

关注(0)|答案(1)|浏览(566)

使用以下链接将数据从databricks加载到sql db，我得到以下错误： command-3227900948916301:23: error: value bulkCopyToSqlDB is not a member of org.apache.spark.sql.DataFrameWriter[org.apache.spark.sql.Row] df.write.mode(SaveMode.Overwrite).bulkCopyToSqlDB(bulkCopyConfig) https://github.com/azure/azure-sqldb-spark/blob/fa1cf19ed797648a20d9b7f474d7c2cd88829ada/samples/scripts/bulkcopysample.scala
我的代码如下：

val bulkCopyConfig  = Config(Map(
  "url"          -> url,
  "databaseName" -> databaseName,
  "dbTable"      -> "dbo.xxxx",
  "user"         -> user,
  "password"     -> password,
  "connectTimeout" -> "120",
  "bulkCopyBatchSize" -> "100000",
  "bulkCopyTableLock" -> "true",
  "bulkCopyTimeout"   -> "0",
  "truncate" -> "true"
//  "queryTimeout"   -> "5"
))
//~
df. write.mode(SaveMode.Overwrite).bulkCopyToSqlDB(bulkCopyConfig)
Any thoughts on why I'm getting the error?

scala apache-spark pyspark databricks azure-databricks

来源：https://stackoverflow.com/questions/64212208/error-bulkcopytosqldb-is-not-a-member-of-org-apache-spark-sql-dataframewriter

1条答案

按热度按时间

pbpqsu0x1#

您需要有正确的导入，才能使用其他函数扩展Dataframe：

import com.microsoft.azure.sqldb.spark.connect._

下面是我得到的（它失败了，因为我没有活动的sql db，但它找到了函数）：

scala> import com.microsoft.azure.sqldb.spark.config.Config
import com.microsoft.azure.sqldb.spark.config.Config
scala> import com.microsoft.azure.sqldb.spark.connect._
import com.microsoft.azure.sqldb.spark.connect._
scala> val bulkCopyConfig = Config(Map(
     |   "url"               -> "mysqlserver.database.windows.net",
     |   "databaseName"      -> "MyDatabase",
     |   "user"              -> "username",
     |   "password"          -> "*********",
     |   "databaseName"      -> "MyDatabase",
     |   "dbTable"           -> "dbo.Clients",
     |   "bulkCopyBatchSize" -> "2500",
     |   "bulkCopyTableLock" -> "true",
     |   "bulkCopyTimeout"   -> "600"
     | ))
bulkCopyConfig: com.microsoft.azure.sqldb.spark.config.Config = com.microsoft.azure.sqldb.spark.config.ConfigBuilder$$anon$1@754443e1
scala> val df = spark.range(1,10)
df: org.apache.spark.sql.Dataset[Long] = [id: bigint]
scala> df.bulkCopyToSqlDB(bulkCopyConfig)
20/10/06 16:24:38 ERROR DataFrameFunctions: Connection cannot be established to the database
com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user 'username'. ClientConnectionId:b35aa6bb-24ec-4727-98b6-074cae4e6a32
...

展开查看全部

赞(0）回复(0）举报 2021-05-25

我来回答

错误：bulkcopytosqldb不是org.apache.spark.sql.dataframewriter的成员

1条答案

相关问题

热门标签

最新问答