当我在intellij中创建第一个基于sparkscala的项目时，我的sbt设置工作正常。sbt如下
name := "sample1"
version := "0.1"
//running in cluster
//scalaVersion := "2.11.0"
//val sparkVersion = "2.1.1"
//running in local
scalaVersion := "2.11.12"
val sparkVersion = "2.3.2"
//sbt when running in cluster using spark-submit
/*
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % sparkVersion % "provided",
"org.apache.spark" %% "spark-sql" % sparkVersion % "provided",
"org.apache.spark" %% "spark-mllib" % sparkVersion % "provided",
"org.apache.spark" %% "spark-streaming" % sparkVersion % "provided"
)
* /
//sbt when running locally
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % sparkVersion,
"org.apache.spark" %% "spark-sql" % sparkVersion,
"org.apache.spark" %% "spark-mllib" % sparkVersion,
"org.apache.spark" %% "spark-streaming" % sparkVersion
)但是当我用新的sbt设置创建一个新项目时，我得到了一个错误。试图更改版本，但没有帮助。新项目的新sbt是
name := "ranger_policy"
version := "0.1"
scalaVersion := "2.11.11"
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core_2.11" % "2.2.0",
"org.apache.spark" % "spark-sql_2.11" % "2.2.0",
"org.apache.spark" % "spark-mllib_2.10" % "1.1.0"
)我的错误是
[error] stack trace is suppressed; run 'last update' for the full output
[error] stack trace is suppressed; run 'last ssExtractDependencies' for the full output
[error] (update) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/2.6.5/hadoop-hdfs-2.6.5.jar: wrong checksum: C:\Users\vidushi.jaiswal\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\apache\hadoop\hadoop-hdfs\2.6.5\hadoop-hdfs-2.6.5.jar (expected SHA-1 8bd0f95e29b9ba7960b4239a7f3706b37b183652 in C:\Users\vidushi.jaiswal\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\apache\hadoop\hadoop-hdfs\2.6.5\.hadoop-hdfs-2.6.5.jar__sha1, got 495045c7fe5110559fa09d7f2381cd3483189a7)
[error] (ssExtractDependencies) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/2.6.5/hadoop-hdfs-2.6.5.jar: wrong checksum: C:\Users\vidushi.jaiswal\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\apache\hadoop\hadoop-hdfs\2.6.5\hadoop-hdfs-2.6.5.jar (expected SHA-1 8bd0f95e29b9ba7960b4239a7f3706b37b183652 in C:\Users\vidushi.jaiswal\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\apache\hadoop\hadoop-hdfs\2.6.5\.hadoop-hdfs-2.6.5.jar__sha1, got 495045c7fe5110559fa09d7f2381cd3483189a7)
[error] Total time: 247 s (04:07), completed Jun 7, 2020 5:35:48 PM
[info] shutting down sbt server
1条答案
按热度按时间fae0ux8s1#
我在通过sbt导入sparksql包时也遇到了同样的问题。中有多个sparksql文件夹，即sparksql2.12和sparksql2.11。当我删除它们并刷新build.sbt时，问题得到了解决。
在这个路径中c:\users\vidushi.jaiswal\appdata\local\coursier\cache\v1\https\repo1.maven.org\maven2\org\apache\hadoop\hadoop hdfs
同一个包可能有多个不同版本的文件夹。全部删除并刷新build.sbt。所有依赖项都将新下载。