来自s3 error org/jets/service/s3serviceexception的hadoop输入

zxlwwiss  于 2021-06-02  发布在  Hadoop
关注(0)|答案(1)|浏览(298)

我正在pc上运行cloudera虚拟机。我正在尝试运行一个简单的map reduce程序,该程序将从s3获取输入,我得到以下异常:

Exception in thread "main" java.lang.NoClassDefFoundError: org/jets3t/service/S3ServiceException at org.apache.hadoop.fs.s3native.NativeS3FileSystem.createDefaultStore(NativeS3FileSystem.java:242)
at org.apache.hadoop.fs.s3native.NativeS3FileSystem.initialize(NativeS3FileSystem.java:232)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2596)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:518)
at PopularHadoopMain.run(PopularHadoopMain.java:30)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at PopularHadoopMain.main(PopularHadoopMain.java:14)
Caused by: java.lang.ClassNotFoundException: org.jets3t.service.S3ServiceException
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 13 more

我在安装了vm的eclipse上运行它,还添加了一行代码(在导入hadoop核心jar之后):

job.getConfiguration().set("fs.s3n.impl", org.apache.hadoop.fs.s3native.NativeS3FileSystem.class.getName());

我已经读到了,看来我需要设置 HADOOP_CLASSPATH 但我试过了,没用(我可能已经知道了)有人知道吗?

相关问题