我是新的scala和sbt构建文件。在介绍性教程中,通过sbt spark软件包插件将spark依赖项添加到scala项目应该是直截了当的,但我发现以下错误:
[error] (run-main-0) java.lang.NoClassDefFoundError: org/apache/spark/SparkContext
请提供资源,以了解更多关于什么可能是驱动错误,因为我想了解过程更彻底。
代码:
trait SparkSessionWrapper {
lazy val spark: SparkSession = {
SparkSession
.builder()
.master("local")
.appName("spark citation graph")
.getOrCreate()
}
val sc = spark.sparkContext
}
import org.apache.spark.graphx.GraphLoader
object Test extends SparkSessionWrapper {
def main(args: Array[String]) {
println("Testing, testing, testing, testing...")
var filePath = "Desktop/citations.txt"
val citeGraph = GraphLoader.edgeListFile(sc, filepath)
println(citeGraph.vertices.take(1))
}
}
插件.sbt
resolvers += "bintray-spark-packages" at "https://dl.bintray.com/spark-packages/maven/"
addSbtPlugin("org.spark-packages" % "sbt-spark-package" % "0.2.6")
build.sbt—正在工作。为什么librarydependencies运行/工作?
spName := "yewno/citation_graph"
version := "0.1"
scalaVersion := "2.11.12"
sparkVersion := "2.2.0"
sparkComponents ++= Seq("core", "sql", "graphx")
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.2.0",
"org.apache.spark" %% "spark-sql" % "2.2.0",
"org.apache.spark" %% "spark-graphx" % "2.2.0"
)
build.sbt—不工作。希望它能正确编译和运行
spName := "yewno/citation_graph"
version := "0.1"
scalaVersion := "2.11.12"
sparkVersion := "2.2.0"
sparkComponents ++= Seq("core", "sql", "graphx")
额外的解释+资源链接,以了解更多关于sbt构建过程,jar文件,以及任何其他可以帮助我加快速度的东西!
1条答案
按热度按时间6rqinv9w1#
sbt spark包插件提供了
provided
范围:我们可以通过运行
show libraryDependencies
来自sbt:run in Compile := Defaults.runTask(fullClasspath in Compile, mainClass in (Compile, run), runner in (Compile, run)).evaluated