我遇到的问题是pyspark无法加载 com.datastax.spark:spark-cassandra-connector_2.11:2.5.1
当单元测试(或pytesting)时。当在普通代码中使用包时,它会很好地加载库,但在unittest中,永远不会加载库,并且会发生错误。
错误:
java.lang.ClassNotFoundException: Failed to find data source: org.apache.spark.sql.cassandra. Please find packages at http://spark.apache.org/third-party-projects.html
代码段:
conf = SparkConf()
conf.setAppName(XXX)
conf.setMaster(XXX)
conf.set("spark.jars.packages", "com.datastax.spark:spark-cassandra-connector_2.11:2.5.1")
sc = SparkContext(conf=conf)
spark = SparkSession(sc)
df = spark.read.format("org.apache.spark.sql.cassandra") \
.option("keyspace", "system") \
.option("table", "local") \
.option("spark.cassandra.auth.username", XXX) \
.option("spark.cassandra.auth.password", XXX) \
.option("spark.cassandra.connection.host", XXX) \
.load()
有什么解决办法吗?Cassandra?
暂无答案!
目前还没有任何答案,快来回答吧!