部署时出错:未找到类org.apache.hadoop.fs.LocalFileSystem

yftpprvb  于 2022-11-01  发布在  Hadoop
关注(0)|答案(1)|浏览(248)

我正在尝试用Scala/sbt编写一个parquet文件。我的代码在我的计算机上运行良好,但在使用Jenkins部署到服务器上时总是失败。我遇到以下错误:

  1. java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.LocalFileSystem not found
  2. at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2688)
  3. at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3431)
  4. at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466)
  5. at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
  6. at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
  7. at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
  8. at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
  9. at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:288)
  10. at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:524)
  11. at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
  12. at org.apache.parquet.hadoop.util.HadoopOutputFile.fromPath(HadoopOutputFile.java:58)
  13. at org.apache.parquet.hadoop.ParquetWriter$Builder.build(ParquetWriter.java:677)
  14. at com.github.mjakubowski84.parquet4s.ParquetWriter$.internalWriter(ParquetWriter.scala:129)
  15. at com.github.mjakubowski84.parquet4s.ParquetWriterImpl.<init>(ParquetWriter.scala:186)
  16. at com.github.mjakubowski84.parquet4s.ParquetWriter$BuilderImpl.build(ParquetWriter.scala:111)
  17. at com.github.mjakubowski84.parquet4s.ParquetWriter$BuilderImpl.writeAndClose(ParquetWriter.scala:113)
  18. at ParquetExport$.$anonfun$tryExport$1(ParquetExport.scala:307)
  19. at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
  20. at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:658)
  21. at scala.util.Success.$anonfun$map$1(Try.scala:255)
  22. at scala.util.Success.map(Try.scala:213)
  23. at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
  24. at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
  25. at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
  26. at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
  27. at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
  28. at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
  29. at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
  30. at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
  31. at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
  32. Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.LocalFileSystem not found
  33. at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2592)
  34. at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2686)
  35. ... 29 more

我首先试着用Spark:

  1. libraryDependencies += "org.apache.spark" %% "spark-core" % "3.3.0"
  2. libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.3.0"

然后更改为代码以处理以下内容:

  1. libraryDependencies ++= Seq(
  2. "com.github.mjakubowski84" %% "parquet4s-core" % "2.6.0",
  3. "org.apache.hadoop" % "hadoop-client" % "2.10.2"
  4. )

但仍然遇到相同的错误。
设置Hadoop配置没有帮助:

  1. val hadoopConfig = new Configuration()
  2. hadoopConfig.set("fs.hdfs.impl", classOf[org.apache.hadoop.hdfs.DistributedFileSystem].getname)
  3. hadoopConfig.set("fs.hdfs.impl", classOf[org.apache.hadoop.fs.LocalFileSystem].getname)

更改ClassLoader也不会产生任何影响:Thread.currentThread.setContextClassLoader(getClass.getClassLoader)
在本地一切正常,但在服务器上就不行了。有什么想法吗?

tvz2xvvm

tvz2xvvm1#

我已经面临过同样的问题。
如果您查看日志:

  1. Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.LocalFileSystem not found
  2. at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2592)

您知道它找不到您为Hadoop配置设置的类LocalFileSystem的名称。
请尝试更改此设置:

  1. val hadoopConfig = new Configuration()
  2. hadoopConfig.set("fs.hdfs.impl", classOf[org.apache.hadoop.hdfs.DistributedFileSystem].getname)
  3. hadoopConfig.set("fs.hdfs.impl", classOf[org.apache.hadoop.fs.LocalFileSystem].getname)

收件人:

  1. val hadoopConfig = new Configuration()
  2. hadoopConfig.set("fs.hdfs.impl", "org.apache.hadoop.hdfs.DistributedFileSystem")
  3. hadoopConfig.set("fs.file.impl", "org.apache.hadoop.fs.LocalFileSystem")
展开查看全部

相关问题