我有一个非常奇怪的依赖错误:
我有一个简单的scala代码:
val spark: SparkSession = SparkSession.builder()
.master("local[1]")
.appName("HDFStoAWSExample")
.getOrCreate()
spark.sparkContext
.hadoopConfiguration.set("fs.s3a.access.key", "ACCESS_KEY")
spark.sparkContext
.hadoopConfiguration.set("fs.s3a.secret.key", "SECRET_KEY")
spark.sparkContext
.hadoopConfiguration.set("fs.s3a.endpoint", "s3.amazonaws.com")
spark.sparkContext.hadoopConfiguration.set("fs.s3a.path.style.access", "true")
val hdfsCSV = spark.read.option("header",true).csv("hdfs://localhost:19000/testCSV.csv")
hdfsCSV.show()
hdfsCSV.write.parquet("s3a://test/parquet/abcCSV")
使用这个简单的sbt文件:
name := "spark-amazon-s3-parquet"
scalaVersion := "2.12.12"
val sparkVersion = "3.0.1"
libraryDependencies += "log4j" % "log4j" % "1.2.17"
libraryDependencies += "org.apache.spark" %% "spark-core" % sparkVersion
libraryDependencies += "org.apache.spark" %% "spark-sql" % sparkVersion
libraryDependencies += "org.apache.hadoop" % "hadoop-aws" % "3.3.0"
libraryDependencies += "org.apache.hadoop" % "hadoop-common" % "3.3.0"
updateOptions := updateOptions.value.withCachedResolution(true)
现在,当我试图编写拼图时,它抱怨缺少类或方法,比如org/apache/hadoop/tracing/spanreceiverhost(最后是完整的堆栈跟踪)
我曾经尝试过使用hadoopcommon和aws的2.7.3版本,但是s3抱怨了400个错误的请求(与之前的代码相同,只是在sbt中更改了common和aws的版本)
有人知道wtf正在使用hadoopcommon和hadoopaws吗?
满栈:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/tracing/SpanReceiverHost
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:634)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
at org.apache.spark.sql.execution.streaming.FileStreamSink$.hasMetadata(FileStreamSink.scala:46)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:366)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:297)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:286)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:286)
at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:723)
at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:553)
at HDFStoAWSExample$.delayedEndpoint$HDFStoAWSExample$1(HDFStoAWSExample.scala:16)
at HDFStoAWSExample$delayedInit$body.apply(HDFStoAWSExample.scala:3)
at scala.Function0.apply$mcV$sp(Function0.scala:39)
at scala.Function0.apply$mcV$sp$(Function0.scala:39)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
at scala.App.$anonfun$main$1$adapted(App.scala:80)
at scala.collection.immutable.List.foreach(List.scala:431)
at scala.App.main(App.scala:80)
at scala.App.main$(App.scala:78)
at HDFStoAWSExample$.main(HDFStoAWSExample.scala:3)
at HDFStoAWSExample.main(HDFStoAWSExample.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.tracing.SpanReceiverHost
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 28 more
ps:我的hadoop配置没有问题,我可以读写它
1条答案
按热度按时间ulmd4ohb1#
如前所述,您可能需要提供
hadoop-client
作为一种依赖。