如何让wordcount在cloudera中使用新的java库?

7uzetpgm  于 2021-05-29  发布在  Hadoop
关注(0)|答案(0)|浏览(277)

我是hadoop新手,在这里尝试WordCountv1.0的例子:
https://www.cloudera.com/documentation/other/tutorial/cdh5/topics/ht_usage.html
但是,当我使用以下行编译wordcount.java时:

  1. javac -cp /usr/lib/hadoop/*:/usr/lib/hadoop-mapreduce/* WordCount.java -d build -Xlint

代码似乎使用了旧版本的.jar文件,并给出了以下警告(如图所示)。但是,当我检查我声明的类路径时,有一些.jar文件似乎是需要的.jar文件的更新版本。

所以我的问题是如何让我的wordcount.java使用更新的文件?我试图在wordcount.java代码中查看哪些行使用了那些必需的.jar文件,但看不到它们。提前谢谢你的帮助。
wordcount.java的代码

  1. import java.io.IOException;
  2. import java.util.StringTokenizer;
  3. import org.apache.hadoop.conf.Configuration;
  4. import org.apache.hadoop.fs.Path;
  5. import org.apache.hadoop.io.IntWritable;
  6. import org.apache.hadoop.io.Text;
  7. import org.apache.hadoop.mapreduce.Job;
  8. import org.apache.hadoop.mapreduce.Mapper;
  9. import org.apache.hadoop.mapreduce.Reducer;
  10. import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
  11. import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
  12. public class WordCount {
  13. public static class TokenizerMapper
  14. extends Mapper<Object, Text, Text, IntWritable>{
  15. private final static IntWritable one = new IntWritable(1);
  16. private Text word = new Text();
  17. public void map(Object key, Text value, Context context
  18. ) throws IOException, InterruptedException {
  19. StringTokenizer itr = new StringTokenizer(value.toString());
  20. while (itr.hasMoreTokens()) {
  21. word.set(itr.nextToken());
  22. context.write(word, one);
  23. }
  24. }
  25. }
  26. public static class IntSumReducer
  27. extends Reducer<Text,IntWritable,Text,IntWritable> {
  28. private IntWritable result = new IntWritable();
  29. public void reduce(Text key, Iterable<IntWritable> values,
  30. Context context
  31. ) throws IOException, InterruptedException {
  32. int sum = 0;
  33. for (IntWritable val : values) {
  34. sum += val.get();
  35. }
  36. result.set(sum);
  37. context.write(key, result);
  38. }
  39. }
  40. public static void main(String[] args) throws Exception {
  41. Configuration conf = new Configuration();
  42. Job job = Job.getInstance(conf, "word count");
  43. job.setJarByClass(WordCount.class);
  44. job.setMapperClass(TokenizerMapper.class);
  45. job.setCombinerClass(IntSumReducer.class);
  46. job.setReducerClass(IntSumReducer.class);
  47. job.setOutputKeyClass(Text.class);
  48. job.setOutputValueClass(IntWritable.class);
  49. FileInputFormat.addInputPath(job, new Path(args[0]));
  50. FileOutputFormat.setOutputPath(job, new Path(args[1]));
  51. System.exit(job.waitForCompletion(true) ? 0 : 1);
  52. }
  53. }

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题