无法在mapreduce 2.x中运行wordcount示例

67up9zun  于 2021-06-02  发布在  Hadoop
关注(0)|答案(3)|浏览(326)

我正试图在java的MapReduce2.x中执行mapreduce字数示例。。。。我已经创建了jar,但是在执行它的时候显示了一个错误,比如wordmapper类在我的包中找不到,但是我已经声明在我的包中…..请帮助我解决这个问题。。。。。。
这是我的wordcount驱动程序代码:

  1. package com.mapreduce2.x;
  2. public class WordCount {
  3. public static void main(String args[]) throws IOException, ClassNotFoundException, InterruptedException
  4. {
  5. Configuration conf=new Configuration();
  6. org.apache.hadoop.mapreduce.Job job= new org.apache.hadoop.mapreduce.Job(conf, "Word_Count");
  7. job.setMapperClass(WordMapper.class);
  8. job.setReducerClass(WordReducer.class);
  9. job.setMapOutputKeyClass(Text.class);
  10. job.setMapOutputValueClass(IntWritable.class);
  11. job.setOutputKeyClass(Text.class);
  12. job.setOutputValueClass(IntWritable.class);
  13. org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(job, new Path(args[0]));
  14. org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(job, new Path(args[1]));
  15. job.waitForCompletion(true);
  16. }}

这是我的wordmapper类:

  1. public class WordMapper extends Mapper<LongWritable, Text, Text,IntWritable>{
  2. private final static IntWritable one=new IntWritable(1);
  3. private Text word=new Text();
  4. public void map(LongWritable key, Text value, org.apache.hadoop.mapreduce.Reducer.Context context) throws IOException, InterruptedException
  5. {
  6. String line=value.toString();
  7. StringTokenizer tokenizer=new StringTokenizer(line);
  8. while(tokenizer.hasMoreTokens())
  9. {
  10. word.set(tokenizer.nextToken());
  11. context.write(word, one);
  12. }
  13. }}

减词器代码-

  1. public class WordReducer extends Reducer<Text, IntWritable, Text, IntWritable>{
  2. public void reduce(Text key, Iterator<IntWritable> values,Context context) throws IOException, InterruptedException
  3. {
  4. int sum =0;
  5. while(values.hasNext())
  6. {
  7. sum= sum+values.next().get();
  8. }
  9. context.write(key, new IntWritable(sum));
  10. }}

它在执行时显示以下错误-

  1. 15/05/29 10:12:26 INFO mapreduce.Job: map 0% reduce 0%
  2. 15/05/29 10:12:33 INFO mapreduce.Job: Task Id : attempt_1432876892622_0005_m_000000_0, Status : FAILED
  3. Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.mapreduce2.x.WordMapper not found
  4. at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2076)
  5. at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:186)
  6. at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:742)
  7. at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
  8. at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
  9. at java.security.AccessController.doPrivileged(Native Method)
  10. at javax.security.auth.Subject.doAs(Subject.java:415)
  11. at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
  12. at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
  13. Caused by: java.lang.ClassNotFoundException: Class com.mapreduce2.x.WordMapper not found
  14. at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1982)
  15. at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2074)
  16. ... 8 more
o7jaxewo

o7jaxewo1#

在运行jar文件时包含类名,或者可以在创建jar文件时指定主类名。
如果运行时没有类名,那么在运行jar时指定类名。
使用命令hadoop jar word.jar com.mapreduce2.x.wordmapper/input/output
这里word.jar是jar文件名。
或者
在创建jar文件时,还可以包含主类名。步骤:文件-->导出-->jar-->位置-->然后单击下一步-->它要求选择主类-->选择类并单击确定
之后,可以使用命令运行jar文件
hadoop jar word.jar/输入/输出
希望这能解决你的问题。

8ljdwjyq

8ljdwjyq2#

尝试在job=newjob(conf,“wordcount”)下面添加注解行//job.setjarbyclass(wordcount.class);
它对我有用

xzlaal3s

xzlaal3s3#

你可以试试这个:(在linux/unix中)
删除java代码中的包名。
在包含java程序的目录中,创建一个名为classes的新目录。前任: Hadoop-Wordcount -> classes , WordCount.java 编译: javac -classpath $HADOOP_HOME/hadoop-common-2.7.1.jar:$HADOOP_HOME/hadoop-mapreduce-client-core-2.7.1.jar:$HADOOP_HOME/hadoop-annotations-2.7.1.jar:$HADOOP_HOME/commons-cli-1.2.jar -d ./classes WordCount.java 创建一个jar jar -cvf wordcount.jar -C ./classes/ . 5.运行 bin/hadoop jar $HADOOP_HOME/Hadoop-WordCount/wordcount.jar WordCount input output

相关问题