mapreduce:Map中的键中存在类型不匹配错误

dauxcl2d  于 2021-07-15  发布在  Hadoop
关注(0)|答案(0)|浏览(350)

我在ubuntuvm中使用hadoop-3.1.0
我得到这个错误，但不知道原因

  1. java.lang.Exception: java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.io.Text, received org.apache.hadoop.io.LongWritable

我的java代码

  1. package wc;
  3. import java.io.IOException;
  4. import java.util.StringTokenizer;
  5. import org.apache.hadoop.conf.Configuration;
  6. import org.apache.hadoop.fs.Path;
  7. import org.apache.hadoop.io.IntWritable;
  8. import org.apache.hadoop.io.Text;
  9. import org.apache.hadoop.mapreduce.Job;
  10. import org.apache.hadoop.mapreduce.Mapper;
  11. import org.apache.hadoop.mapreduce.Reducer;
  12. import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
  13. import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
  15. public class Wordcount { 
  17. //================================
  18. public static class TokenizerMapper extends Mapper 
  19. { 
  20. private final static IntWritable one = new IntWritable(1); 
  21. private Text word = new Text(); 
  23. public void map(Object key, Text value, Context context) throws
  24. IOException, InterruptedException { 
  26. StringTokenizer itr = new StringTokenizer(value.toString());
  27. while (itr.hasMoreTokens()) { 
  29. word.set(itr.nextToken());
  30. context.write(word, one); 
  32. }
  33. } 
  35. }
  37. //================================
  39. public static class IntSumReducer extends Reducer {
  41. private IntWritable result = new IntWritable(); 
  43. public void reduce(Text key, Iterable<IntWritable> values,Context context) 
  44.         throws IOException, InterruptedException { 
  46. int sum = 0;
  47. for (IntWritable val : values) { 
  49. sum += val.get();
  50. } 
  52. result.set(sum);
  53. context.write(key, result); 
  55. }
  56. } 
  57. //================================
  59. public static void main(String[] args) throws Exception
  60. { 
  62. Configuration conf = new Configuration(); 
  64. Job job = Job.getInstance(conf, "word count");
  65. job.setJarByClass(Wordcount.class);
  66. job.setMapperClass(TokenizerMapper.class);
  67. job.setCombinerClass(IntSumReducer.class);
  68. job.setReducerClass(IntSumReducer.class);
  69. job.setOutputKeyClass(Text.class);
  70. job.setOutputValueClass(IntWritable.class);
  71. FileInputFormat.addInputPath(job, new Path(args[0]));
  72. FileOutputFormat.setOutputPath(job, new Path(args[1]));
  73. System.out.println("Almost done");
  74. System.exit(job.waitForCompletion(true) ? 0 : 1); 
  76. }
  77. }

我以“matrix.txt”（关于矩阵乘法的简短文本）作为输入文件，以“output”作为输出文件夹来运行配置
这里是控制台输出

  1. 2021-02-09 06:51:27,505 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(60)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  2. Almost done
  3. 2021-02-09 06:51:27,832 INFO  [main] impl.MetricsConfig (MetricsConfig.java:loadFirst(118)) - loaded properties from hadoop-metrics2.properties
  4. 2021-02-09 06:51:27,884 INFO  [main] impl.MetricsSystemImpl (MetricsSystemImpl.java:startTimer(374)) - Scheduled Metric snapshot period at 0 second(s).
  5. 2021-02-09 06:51:27,884 INFO  [main] impl.MetricsSystemImpl (MetricsSystemImpl.java:start(191)) - JobTracker metrics system started
  6. 2021-02-09 06:51:27,954 WARN  [main] mapreduce.JobResourceUploader (JobResourceUploader.java:uploadResourcesInternal(149)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
  7. 2021-02-09 06:51:27,963 WARN  [main] mapreduce.JobResourceUploader (JobResourceUploader.java:uploadJobJar(482)) - No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
  8. 2021-02-09 06:51:27,990 INFO  [main] input.FileInputFormat (FileInputFormat.java:listStatus(292)) - Total input files to process : 1
  9. 2021-02-09 06:51:28,030 INFO  [main] mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(202)) - number of splits:1
  10. 2021-02-09 06:51:28,489 INFO  [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(298)) - Submitting tokens for job: job_local1153866518_0001
  11. 2021-02-09 06:51:28,492 INFO  [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(299)) - Executing with tokens: []
  12. 2021-02-09 06:51:28,601 INFO  [main] mapreduce.Job (Job.java:submit(1574)) - The url to track the job: http://localhost:8080/
  13. 2021-02-09 06:51:28,602 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1619)) - Running job: job_local1153866518_0001
  14. 2021-02-09 06:51:28,605 INFO  [Thread-21] mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(501)) - OutputCommitter set in config null
  15. 2021-02-09 06:51:28,609 INFO  [Thread-21] output.FileOutputCommitter (FileOutputCommitter.java:<init>(141)) - File Output Committer Algorithm version is 2
  16. 2021-02-09 06:51:28,610 INFO  [Thread-21] output.FileOutputCommitter (FileOutputCommitter.java:<init>(156)) - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
  17. 2021-02-09 06:51:28,610 INFO  [Thread-21] mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(519)) - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
  18. 2021-02-09 06:51:28,640 INFO  [Thread-21] mapred.LocalJobRunner (LocalJobRunner.java:runTasks(478)) - Waiting for map tasks
  19. 2021-02-09 06:51:28,640 INFO  [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(252)) - Starting task: attempt_local1153866518_0001_m_000000_0
  20. 2021-02-09 06:51:28,657 INFO  [LocalJobRunner Map Task Executor #0] output.FileOutputCommitter (FileOutputCommitter.java:<init>(141)) - File Output Committer Algorithm version is 2
  21. 2021-02-09 06:51:28,657 INFO  [LocalJobRunner Map Task Executor #0] output.FileOutputCommitter (FileOutputCommitter.java:<init>(156)) - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
  22. 2021-02-09 06:51:28,730 INFO  [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(625)) -  Using ResourceCalculatorProcessTree : [ ]
  23. 2021-02-09 06:51:28,733 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(768)) - Processing split: file:/home/hadoop/eclipse-workspace/WordCount/matrix.txt:0+509
  24. 2021-02-09 06:51:28,919 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1219)) - (EQUATOR) 0 kvi 26214396(104857584)
  25. 2021-02-09 06:51:28,919 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(1012)) - mapreduce.task.io.sort.mb: 100
  26. 2021-02-09 06:51:28,919 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(1013)) - soft limit at 83886080
  27. 2021-02-09 06:51:28,919 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(1014)) - bufstart = 0; bufvoid = 104857600
  28. 2021-02-09 06:51:28,920 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(1015)) - kvstart = 26214396; length = 6553600
  29. 2021-02-09 06:51:28,922 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:createSortingCollector(409)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  30. 2021-02-09 06:51:28,926 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1476)) - Starting flush of map output
  31. 2021-02-09 06:51:28,939 INFO  [Thread-21] mapred.LocalJobRunner (LocalJobRunner.java:runTasks(486)) - map task executor complete.
  32. 2021-02-09 06:51:28,940 WARN  [Thread-21] mapred.LocalJobRunner (LocalJobRunner.java:run(590)) - job_local1153866518_0001
  33. java.lang.Exception: java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.io.Text, received org.apache.hadoop.io.LongWritable
  34.     at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
  35.     at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
  36. Caused by: java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.io.Text, received org.apache.hadoop.io.LongWritable
  37.     at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1088)
  38.     at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:727)
  39.     at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
  40.     at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
  41.     at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:125)
  42.     at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
  43.     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
  44.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
  45.     at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
  46.     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  47.     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  48.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  49.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  50.     at java.lang.Thread.run(Thread.java:748)
  51. 2021-02-09 06:51:29,605 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1640)) - Job job_local1153866518_0001 running in uber mode : false
  52. 2021-02-09 06:51:29,606 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1647)) -  map 0% reduce 0%
  53. 2021-02-09 06:51:29,607 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1660)) - Job job_local1153866518_0001 failed with state FAILED due to: NA
  54. 2021-02-09 06:51:29,612 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1665)) - Counters: 0
hadoopmapreduceDictionary

暂无答案！

目前还没有任何答案，快来回答吧！

我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com