包括hadoop1.2.1中的第三方jar

cedebl8k  于 2021-06-04  发布在  Hadoop
关注(0)|答案(0)|浏览(253)

我有一个自定义(mahout)类,我正试图通过hadoop运行它。我遵循了这里的说明(以及一条将jar放在hdfs中的注解),但是我不断得到以下错误

  1. ClassNotFoundException: org.apache.mahout.common.AbstractJob

我的设置如下:

  1. REP_ROOT=/home/user/.m2/repository
  2. MAHOUT_CORE=$REP_ROOT/org/apache/mahout/mahout-core/0.9/mahout-core-0.9.jar
  3. MAHOUT_CLI=$REP_ROOT/org/apache/mahout/commons/commons-cli/2.0-mahout/commons-cli-2.0-mahout.jar
  4. MAHOUT_INTEGRATION=$REP_ROOT/org/apache/mahout/mahout-integration/0.9/mahout-integration-0.9.jar
  5. LANG3=$REP_ROOT/org/apache/commons/commons-lang3/3.1/commons-lang3-3.1.jar
  6. GOOGLE_LIST=$REP_ROOT/com/google/guava/guava/14.0.1/guava-14.0.1.jar
  7. export LIBJARS=$MAHOUT_CORE,$MAHOUT_CLI,$MAHOUT_INTEGRATION,$LANG3,$GOOGLE_LIST
  8. export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$MAHOUT_CORE:$MAHOUT_CLI:$MAHOUT_INTEGRATION:$LANG3:$GOOGLE_LIST
  9. hadoop jar /home/user/bin/CustomJar.jar com.company.Mahout.LDAJob \
  10. -libjars ${LIBJARS} \
  11. -DbaseLocation=/user/hadoop/test \
  12. -Dinput=input \
  13. -Doutput=output

缺少的类(abstractjob)位于included-in-mahout\u核心中。ldajob开始于:

  1. public class LDAJob extends AbstractJob{
  2. public static void main(String args[]) throws Exception {
  3. Configuration conf = new Configuration();
  4. ToolRunner.run(conf,new LDAJob(), args);
  5. }
  6. public int run(String[] args) throws Exception {
  7. Configuration conf = super.getConf();
  8. String baseFileLocation = conf.get("baseLocation");
  9. .....
  10. }

错误详细信息

  1. 14/02/26 01:29:44 INFO util.NativeCodeLoader: Loaded the native-hadoop library
  2. 14/02/26 01:29:44 WARN snappy.LoadSnappy: Snappy native library not loaded
  3. 14/02/26 01:29:47 INFO mapred.JobClient: Running job: job_201402260118_0004
  4. 14/02/26 01:29:48 INFO mapred.JobClient: map 0% reduce 0%
  5. 14/02/26 01:29:53 INFO mapred.JobClient: Task Id : attempt_201402260118_0004_m_000000_0, Status : FAILED
  6. java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
  7. at org.apache.hadoop.mapreduce.lib.input.CombineFileRecordReader.initNextRecordReader(CombineFileRecordReader.java:164)
  8. at org.apache.hadoop.mapreduce.lib.input.CombineFileRecordReader.<init>(CombineFileRecordReader.java:126)
  9. at org.apache.mahout.text.MultipleTextFileInputFormat.createRecordReader(MultipleTextFileInputFormat.java:43)
  10. at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:488)
  11. at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731)
  12. at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
  13. at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
  14. at java.security.AccessController.doPrivileged(Native Method)
  15. at javax.security.auth.Subject.doAs(Subject.java:396)
  16. at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
  17. at org.apache.hadoop.mapred.Child.main(Child.java:249)
  18. Caused by: java.lang.reflect.InvocationTargetException
  19. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  20. at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
  21. at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
  22. at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
  23. at org.apache.hadoop.mapreduce.lib.input.CombineFileRecordReader.initNextRecordReader(CombineFileRecordReader.java:155)
  24. ... 10 more
  25. Caused by: java.lang.NoClassDefFoundError: org/apache/mahout/common/AbstractJob
  26. at java.lang.ClassLoader.defineClass1(Native Method)
  27. at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
  28. at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
  29. at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
  30. at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
  31. at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
  32. at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
  33. at java.security.AccessController.doPrivileged(Native Method)
  34. at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
  35. at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
  36. at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
  37. at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
  38. at org.apache.mahout.text.WholeFileRecordReader.<init>(WholeFileRecordReader.java:61)
  39. ... 15 more
  40. Caused by: java.lang.ClassNotFoundException: org.apache.mahout.common.AbstractJob
  41. at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
  42. at java.security.AccessController.doPrivileged(Native Method)
  43. at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
  44. at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
  45. at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
  46. at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
  47. ... 28 more

任何帮助都将不胜感激。

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题