mapreduce程序中函数错误的不可Map参数

yrwegjxp  于 2021-05-30  发布在  Hadoop
关注(0)|答案(0)|浏览(225)

我正在eclipse(在ubuntu14.04lts上)中处理一个mapreducejava项目,为此我使用apacheavro序列化框架,因为我需要avro-tools-1.7.7.jar文件。我从apache网站下载了这个jar,并使用下载的jar编写了java代码。当我执行程序时,我得到java.lang.verifyerror。我从一些网站上看到,这个错误是由于jar中编译类文件的jdk版本与运行时jdk版本不匹配,所以我检查了下载的jar file.class版本和我的运行时jvm版本,结果不匹配,所以我将jdk从1.7降级到1.6,没有不匹配。jar中编译的类有50个主版本,我当前的项目类文件也是如此。但我还是犯了那个错误。

  1. srimanth@srimanth-Inspiron-N5110:~$ hadoop jar Desktop/AvroMapReduceExamples.jar practice.AvroSort file:///home/srimanth/avrofile.avro file:///home/srimanth/sorted/ test.avro
  2. 15/04/19 22:14:36 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  3. Exception in thread "main" java.lang.VerifyError: (class: org/apache/hadoop/mapred/JobTrackerInstrumentation, method: create signature: (Lorg/apache/hadoop/mapred/JobTracker;Lorg/apache/hadoop/mapred/JobConf;)Lorg/apache/hadoop/mapred/JobTrackerInstrumentation;) Incompatible argument to function
  4. at org.apache.hadoop.mapred.LocalJobRunner.<init>(LocalJobRunner.java:420)
  5. at org.apache.hadoop.mapred.JobClient.init(JobClient.java:470)
  6. at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:455)
  7. at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
  8. at practice.AvroSort.run(AvroSort.java:63)
  9. at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
  10. at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
  11. at practice.AvroSort.main(AvroSort.java:67)
  12. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  13. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  14. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  15. at java.lang.reflect.Method.invoke(Method.java:622)
  16. at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
  17. at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

这是我的java程序

  1. package practice;
  2. import java.io.File;
  3. import java.io.IOException;
  4. import org.apache.avro.Schema;
  5. import org.apache.avro.mapred.AvroCollector;
  6. import org.apache.avro.mapred.AvroJob;
  7. import org.apache.avro.mapred.AvroMapper;
  8. import org.apache.avro.mapred.AvroReducer;
  9. import org.apache.avro.mapred.Pair;
  10. import org.apache.hadoop.conf.Configured;
  11. import org.apache.hadoop.fs.Path;
  12. import org.apache.hadoop.mapred.FileInputFormat;
  13. import org.apache.hadoop.mapred.FileOutputFormat;
  14. import org.apache.hadoop.mapred.JobClient;
  15. import org.apache.hadoop.mapred.JobConf;
  16. import org.apache.hadoop.mapred.Reporter;
  17. import org.apache.hadoop.util.Tool;
  18. import org.apache.hadoop.util.ToolRunner;
  19. public class AvroSort extends Configured implements Tool {
  20. static class SortMapper<K> extends AvroMapper<K, Pair<K, K>> {
  21. public void map(K datum, AvroCollector<Pair<K, K>> collector,
  22. Reporter reporter) throws IOException {
  23. collector.collect(new Pair<K, K>(datum, null, datum, null));
  24. }
  25. }
  26. static class SortReducer<K> extends AvroReducer<K, K, K> {
  27. public void reduce(K key, Iterable<K> values,
  28. AvroCollector<K> collector,
  29. Reporter reporter) throws IOException {
  30. for (K value : values) {
  31. collector.collect(value);
  32. }
  33. }
  34. }
  35. @Override
  36. public int run(String[] args) throws Exception {
  37. if (args.length != 3) {
  38. System.err.printf(
  39. "Usage: %s [generic options] <input> <output> <schema-file>\n",
  40. getClass().getSimpleName());
  41. ToolRunner.printGenericCommandUsage(System.err);
  42. return -1;
  43. }
  44. String input = args[0];
  45. String output = args[1];
  46. String schemaFile = args[2];
  47. JobConf conf = new JobConf(getConf(), getClass());
  48. conf.setJobName("Avro sort");
  49. FileInputFormat.addInputPath(conf, new Path(input));
  50. FileOutputFormat.setOutputPath(conf, new Path(output));
  51. Schema schema = new Schema.Parser().parse(new File(schemaFile));
  52. AvroJob.setInputSchema(conf, schema);
  53. Schema intermediateSchema = Pair.getPairSchema(schema, schema);
  54. AvroJob.setMapOutputSchema(conf, intermediateSchema);
  55. AvroJob.setOutputSchema(conf, schema);
  56. AvroJob.setMapperClass(conf, SortMapper.class);
  57. AvroJob.setReducerClass(conf, SortReducer.class);
  58. JobClient.runJob(conf);
  59. return 0;
  60. }
  61. public static void main(String[] args) throws Exception {
  62. int exitCode = ToolRunner.run(new AvroSort(), args);
  63. System.exit(exitCode);
  64. }
  65. }

附加信息:jdk版本:1.6,hadoop版本:2.6.0,我没有使用maven。
请帮帮我,我一整天都被困在这里。我真的很感谢你的帮助。

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题