scala—如何将包含结构的Dataframe写入cassandratable

vatpfxk5  于 2021-06-10  发布在  Cassandra
关注(0)|答案(1)|浏览(271)

我正在尝试将Dataframe数据写入cassandra表。如何将dataframe中的结构Map到cassandra表中的列。
Dataframe模式

root
   |-- _1: struct (nullable = true)
    |    |-- _1: string (nullable = true)
    |    |-- _2: integer (nullable = false)
    |    |-- _3: integer (nullable = false)
    |    |-- _4: integer (nullable = false)
    |-- _2: double (nullable = false)

cassandratable的描述(…我正在尝试使用tuple来保存结构数据)

create table knmi_r (idnkey tuple<text,int,int,int>, logval int, primary key (idnkey) ) ;

下面是我保存到cassandra表的代码

resultdf.printSchema()
              resultdf.write
                      .format("org.apache.spark.sql.cassandra")
                      .options(Map("table" -> "knmi_r", "keyspace" -> "dbks1"))
                      .save()

然而,我得到下面的错误。有人能告诉我怎么过去吗。

Exception in thread "main" java.util.NoSuchElementException: Columns not found in table dbks1.knmi_r: _1, _2
                at com.datastax.spark.connector.SomeColumns.selectFrom(ColumnSelector.scala:44)
                at com.datastax.spark.connector.writer.TableWriter$.apply(TableWriter.scala:385)
                at com.datastax.spark.connector.RDDFunctions.saveToCassandra(RDDFunctions.scala:35)
                at org.apache.spark.sql.cassandra.CassandraSourceRelation.insert(CassandraSourceRelation.scala:76)
                at org.apache.spark.sql.cassandra.DefaultSource.createRelation(DefaultSource.scala:90)
                at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
                at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
                at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
                at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
                at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
                at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
                at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
                at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
                at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
                at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
                at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
                at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
                at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)
                at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)
                at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
                at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
                at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
                at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:668)
                at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:276)
                at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:270)
                at nl.rug.sc.SparkExample.testExample(SparkExample.scala:286)
                at nl.rug.sc.app.SparkBootcamp$class.run(SparkBootcamp.scala:19)
                at nl.rug.sc.app.SparkLocalMain$.run(SparkLocalMain.scala:6)
                at nl.rug.sc.app.SparkLocalMain$.delayedEndpoint$nl$rug$sc$app$SparkLocalMain$1(SparkLocalMain.scala:18)
                at nl.rug.sc.app.SparkLocalMain$delayedInit$body.apply(SparkLocalMain.scala:6)
                at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
                at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
                at scala.App$$anonfun$main$1.apply(App.scala:76)
                at scala.App$$anonfun$main$1.apply(App.scala:76)
                at scala.collection.immutable.List.foreach(List.scala:392)
                at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
                at scala.App$class.main(App.scala:76)
                at nl.rug.sc.app.SparkLocalMain$.main(SparkLocalMain.scala:6)
                at nl.rug.sc.app.SparkLocalMain.main(SparkLocalMain.scala)
cwxwcias

cwxwcias1#

错误是由于架构名称和cassandra列名之间不匹配。我只是给他们重新命名,结果成功了

val newNames = Seq("idnkey", "logval")

                val resultdf1 = resultdf.toDF(newNames: _*)

                resultdf1.printSchema

现在dataframe的模式与cassandra表匹配

root
 |-- idnkey: struct (nullable = true)
 |    |-- _1: string (nullable = true)
 |    |-- _2: integer (nullable = false)
 |    |-- _3: integer (nullable = false)
 |    |-- _4: integer (nullable = false)
 |-- logval: double (nullable = false)

相关问题