添加外部jar时hadoop noclassdeffounderror

cedebl8k  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(522)

我正在尝试使用外部jar在hadoop上运行mapreduce作业。我已经使用 copyFromLocal 至 /user/hduser/lib/ .
在我的main方法中，我将jar添加到distributedcache。但是，当我运行mapreduce程序时，在mapper类中收到一个noclassdeffounderror。我已经尝试了许多解决方案张贴在所以从其他人谁有一个类似的错误，但我没有解决这个问题。感谢您的指导。
从主方法：

  1. Configuration conf = new Configuration();
  2. String jarToAdd1 = "/user/hduser/lib/joda-time-2.9.1-no-tzdb.jar";
  3. String jarToAdd2 = "/user/hduser/lib/joda-time-2.9.1.jar";
  4. addJarToDistributedCache(jarToAdd1, conf);
  5. addJarToDistributedCache(jarToAdd2, conf);
  6. .........

添加到分布式缓存：

  1. private static void addJarToDistributedCache(String jarToAdd, Configuration conf) throws IOException {
  3.         Path hdfsJar = new Path(jarToAdd);
  4.         DistributedCache.addFileToClassPath(hdfsJar,conf);
  5.     }

发生错误的Map器：

  1. public static class Map1 extends Mapper<LongWritable, Text, IntWritable, UserData> {
  3. Map<IntWritable, UserData> userLog = new HashMap<IntWritable, UserData>();
  5. public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
  7.     String line = value.toString();                             
  8.     StringTokenizer tokenizer = new StringTokenizer(line);      
  9.     DateTimeFormatter formatter = DateTimeFormat.forPattern("yyyyddmm HH:mm:ss");    //*********ERROR HAPPENS HERE**********

堆栈跟踪：

  1. 16/01/30 20:45:15 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  2. 16/01/30 20:45:16 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
  3. 16/01/30 20:45:16 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
  4. 16/01/30 20:45:16 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
  5. 16/01/30 20:45:16 INFO input.FileInputFormat: Total input paths to process : 1
  6. 16/01/30 20:45:16 INFO input.FileInputFormat: Total input paths to process : 1
  7. 16/01/30 20:45:16 INFO mapreduce.JobSubmitter: number of splits:3
  8. 16/01/30 20:45:17 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1787244132_0001
  9. 16/01/30 20:45:17 INFO mapred.LocalDistributedCacheManager: Creating symlink: /app/hadoop/tmp/mapred/local/1454204717451/joda-time-2.9.1-no-tzdb.jar <- /usr/local/hadoop/sbin/joda-time-2.9.1-no-tzdb.jar
  10. 16/01/30 20:45:17 INFO mapred.LocalDistributedCacheManager: Localized hdfs://localhost:54310/user/hduser/lib/joda-time-2.9.1-no-tzdb.jar as file:/app/hadoop/tmp/mapred/local/1454204717451/joda-time-2.9.1-no-tzdb.jar
  11. 16/01/30 20:45:17 INFO mapred.LocalDistributedCacheManager: Creating symlink: /app/hadoop/tmp/mapred/local/1454204717452/joda-time-2.9.1.jar <- /usr/local/hadoop/sbin/joda-time-2.9.1.jar
  12. 16/01/30 20:45:17 INFO mapred.LocalDistributedCacheManager: Localized hdfs://localhost:54310/user/hduser/lib/joda-time-2.9.1.jar as file:/app/hadoop/tmp/mapred/local/1454204717452/joda-time-2.9.1.jar
  13. 16/01/30 20:45:17 INFO mapred.LocalDistributedCacheManager: file:/app/hadoop/tmp/mapred/local/1454204717451/joda-time-2.9.1-no-tzdb.jar
  14. 16/01/30 20:45:17 INFO mapred.LocalDistributedCacheManager: file:/app/hadoop/tmp/mapred/local/1454204717452/joda-time-2.9.1.jar
  15. 16/01/30 20:45:17 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
  16. 16/01/30 20:45:17 INFO mapreduce.Job: Running job: job_local1787244132_0001
  17. 16/01/30 20:45:17 INFO mapred.LocalJobRunner: OutputCommitter set in config null
  18. 16/01/30 20:45:18 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
  19. 16/01/30 20:45:18 INFO mapred.LocalJobRunner: Waiting for map tasks
  20. 16/01/30 20:45:18 INFO mapred.LocalJobRunner: Starting task: attempt_local1787244132_0001_m_000000_0
  21. 16/01/30 20:45:18 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
  22. 16/01/30 20:45:18 INFO mapred.MapTask: Processing split: hdfs://localhost:54310/user/hduser/input/sentimentFeedback7.csv:0+143748596
  23. 16/01/30 20:45:18 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
  24. 16/01/30 20:45:18 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
  25. 16/01/30 20:45:18 INFO mapred.MapTask: soft limit at 83886080
  26. 16/01/30 20:45:18 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
  27. 16/01/30 20:45:18 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
  28. 16/01/30 20:45:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  29. 16/01/30 20:45:18 INFO mapred.MapTask: Starting flush of map output
  30. 16/01/30 20:45:18 INFO mapred.LocalJobRunner: Starting task: attempt_local1787244132_0001_m_000001_0
  31. 16/01/30 20:45:18 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
  32. 16/01/30 20:45:18 INFO mapred.MapTask: Processing split: hdfs://localhost:54310/user/hduser/input/allergyConsumption7.csv:0+134217728
  33. 16/01/30 20:45:18 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
  34. 16/01/30 20:45:18 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
  35. 16/01/30 20:45:18 INFO mapred.MapTask: soft limit at 83886080
  36. 16/01/30 20:45:18 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
  37. 16/01/30 20:45:18 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
  38. 16/01/30 20:45:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  39. 16/01/30 20:45:18 INFO mapred.MapTask: Starting flush of map output
  40. 16/01/30 20:45:18 INFO mapred.LocalJobRunner: Starting task: attempt_local1787244132_0001_m_000002_0
  41. 16/01/30 20:45:18 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
  42. 16/01/30 20:45:18 INFO mapred.MapTask: Processing split: hdfs://localhost:54310/user/hduser/input/allergyConsumption7.csv:134217728+105486421
  43. 16/01/30 20:45:18 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
  44. 16/01/30 20:45:18 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
  45. 16/01/30 20:45:18 INFO mapred.MapTask: soft limit at 83886080
  46. 16/01/30 20:45:18 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
  47. 16/01/30 20:45:18 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
  48. 16/01/30 20:45:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  49. 16/01/30 20:45:18 INFO mapred.MapTask: Starting flush of map output
  50. 16/01/30 20:45:18 INFO mapred.LocalJobRunner: map task executor complete.
  51. 16/01/30 20:45:18 WARN mapred.LocalJobRunner: job_local1787244132_0001
  52. java.lang.Exception: java.lang.NoClassDefFoundError: org/joda/time/format/DateTimeFormat
  53.     at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
  54.     at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
  55. Caused by: java.lang.NoClassDefFoundError: org/joda/time/format/DateTimeFormat
  56.     at org.peach.fooddiary.FoodDiaryMR$Map1.map(FoodDiaryMR.java:45)
  57.     at org.peach.fooddiary.FoodDiaryMR$Map1.map(FoodDiaryMR.java:1)
  58.     at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
  59.     at org.apache.hadoop.mapreduce.lib.input.DelegatingMapper.run(DelegatingMapper.java:55)
  60.     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
  61.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
  62.     at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
  63.     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  64.     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  65.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  66.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  67.     at java.lang.Thread.run(Thread.java:745)
  68. Caused by: java.lang.ClassNotFoundException: org.joda.time.format.DateTimeFormat
  69.     at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
  70.     at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  71.     at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  72.     ... 12 more
  73. 16/01/30 20:45:18 INFO mapreduce.Job: Job job_local1787244132_0001 running in uber mode : false
  74. 16/01/30 20:45:18 INFO mapreduce.Job:  map 0% reduce 0%
  75. 16/01/30 20:45:18 INFO mapreduce.Job: Job job_local1787244132_0001 failed with state FAILED due to: NA
  76. 16/01/30 20:45:19 INFO mapreduce.Job: Counters: 0
JavahadoopJardistributed-cacheNoClassDefFoundError

1条答案

按热度按时间
c7rzv4ha

c7rzv4ha1#

首先distributedcache.addfiletoclasspath现在已不推荐使用，但您可以使用：

  1. Configuration conf = getConf();
  2.     Job job = Job.getInstance(conf, "My Job Name");
  3.     job.addFileToClassPath(new Path("path"));

如果您仍然面临同样的问题，请手动执行，我的意思是通过将jar文件放入hadoop的类路径来将代码部署到hadoop本身：
1） 使用以下方法将需要执行的类导出到hadoop中：
在项目资源管理器中，右键单击项目名称并选择“导出…”
在“java”部分下，选择“jar文件”，然后选择“下一步”
选择要将导出的“jar”存储到的目录
2） 转到运行hadoop的服务器并运行以下命令以了解hadoop正在查找的目录，其中一个示例是“$hadoop\u home/share/hadoop/common/lib/*”

  1. $ hadoop classpath

3） 将导出的jar文件从步骤1复制到步骤2中的一个目录，然后再次开始测试。
提示最好直接在服务器中测试map reduce，并使用以下命令在服务器中运行它：

  1. $ hadoop jar JarFileName.jar PakageName.ClassName /path/to/input /path/to/output

有关详细说明的完整示例，请检查此项

展开查看全部
赞(0）分享  回复(0）举报  2021-05-29
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com