Spark Cassandra连接器问题

jgovgodb  于 2022-11-05  发布在  Cassandra
关注(0)|答案(1)|浏览(228)

I am trying to integrate Cassandra with Spark and facing the below issue.

Issue:

com.datastax.spark.connector.util.ConfigCheck$ConnectorConfigurationException: Invalid Config Variables Only known spark.cassandra.* variables are allowed when using the Spark Cassandra Connector. spark.cassandra.keyspace is not a valid Spark Cassandra Connector variable. Possible matches: spark.cassandra.sql.keyspace spark.cassandra.output.batch.grouping.key at com.datastax.spark.connector.util.ConfigCheck$.checkConfig(ConfigCheck.scala:50) at com.datastax.spark.connector.cql.CassandraConnectorConf$.apply(CassandraConnectorConf.scala:253) at org.apache.spark.sql.cassandra.CassandraSourceRelation$.apply(CassandraSourceRelation.scala:263) at org.apache.spark.sql.cassandra.CassandraCatalog.org$apache$spark$sql$cassandra$CassandraCatalog$$buildRelation(CasandraCatalog.scala:41) at org.apache.spark.sql.cassandra.CassandraCatalog$$anon$1.load(CassandraCatalog.scala:26) at org.apache.spark.sql.cassandra.CassandraCatalog$$anon$1.load(CassandraCatalog.scala:23)
Please find the below versions of spark Cassandra and connector I am using.
Spark : 1.6.0 Cassandra : 2.1.17 Connector Used : spark-cassandra-connector_2.10-1.6.0-M1.jar
Below is the code snippet I am using to connect Cassandra from spark.

val conf: org.apache.spark.SparkConf = new SparkConf(true) \
            .setAppName("Spark Cassandra") \
            .set"spark.cassandra.connection.host", "abc.efg.lkh") \
            .set("spark.cassandra.auth.username", "xyz") \
            .set("spark.cassandra.auth.password", "1234") \
            .set("spark.cassandra.keyspace","abcded")

val sc = new SparkContext("local[*]", "Spark Cassandra",conf)

val csc = new CassandraSQLContext(sc)
csc.setKeyspace("abcded")

val my_df = csc.sql("select * from table")

Here when I try to create DF, I am getting above posted error. I tried without passing schema in conf but it is trying to access in default schema where mentioned user doesn't have access.
Already a JIRA was opened and closed. https://datastax-oss.atlassian.net/browse/SPARKC-102
yet I am getting this issue. Please let me know whether I need to use lastest connector to resolve this issue.
Thanks in advance.

sdnqo3pr

sdnqo3pr1#

重要信息在您发布的错误消息中[格式为可读性]:
无效的配置变量
在使用Spark Cassandra连接器时,只允许使用已知的spark.cassandra.* 变量。
不是有效的Spark Cassandra连接器变量。
可能的匹配项:spark.cassandra.sql.keyspace
spark.cassandra.keyspace不是连接器的可用属性。可用属性的完整列表可在此处找到:https://github.com/datastax/spark-cassandra-connector/blob/master/doc/reference.md
使用建议的spark.cassandra.sql.keyspace可能会有一些运气;否则,您可能只需要为使用连接器执行的每个Cassandra交互显式指定密钥空间。

相关问题