2.6.0中运行我自己的wordcount.java版本

2sbarzqh  于 2021-05-30  发布在  Hadoop
关注(0)|答案(5)|浏览(375)

我正在尝试创建我自己的wordcount版本并执行它。为此,我尝试通过执行以下命令(如下所述)来创建wordcount.jarhttp://cs.smith.edu/dftwiki/index.php/hadoop_tutorial_1_--_running_wordcount 对于hadoop-2以外的早期版本。*):

  1. javac -classpath /usr/local/hadoop-2.6.0/share/hadoop/common/*:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/* -d wordcount_classes/ WordCount.java
  2. jar -cvf wordcount.jar -C wordcount_classes/ .

问题是我在试图编译wordcount类时在第一个命令中遇到错误。在第一个命令中,我尝试包含存在于common和mapreduce目录中的所有jar,因为我没有找到任何名为“hadoop-0.19.2-core.jar”的jar,正如我在互联网上找到的hadoop-0教程中所描述的那样*

  1. javac -classpath /home/hadoop/hadoop/hadoop-0.19.2-core.jar -d wordcount_classes WordCount.java

当我按如下方式执行给定的wordcount时,它工作得非常好,但我无法编译自己的版本:

  1. hadoop jar /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-*-examples-2.6.0.jar wordcount input output

我在互联网上搜索过,但我发现大多数教程都是针对hadoop-1.*的,它不同于2.6.0版本,2.6.0版本不包含任何java源文件,如我需要的wordcount.java,而且它的结构也完全不同。因此,我不得不单独下载wordcount.java文件https://github.com/apache/hadoop-common/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/wordcount.java 并复制当前目录中的wordcount.java,在该目录中执行创建jar的命令。
我已经从这个来源下载了hadoophttp://mirror.its.dal.ca/apache/hadoop/common/hadoop-2.6.0/
以下是我得到的错误的详细信息:

  1. /usr/local/hadoop-2.6.0/share/hadoop/common/hadoop-common-2.6.0.jar(org /apache/hadoop/fs/Path.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate': class file for org.apache.hadoop.classification.InterfaceAudience not found
  2. WordCount.java:54: error: cannot access Options
  3. String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
  4. ^
  5. class file for org.apache.commons.cli.Options not found
  6. WordCount.java:68: error: method waitForCompletion in class Job cannot be applied to given types;
  7. System.exit(job.waitForCompletion() ? 0 : 1);
  8. ^
  9. required: boolean
  10. found: no arguments
  11. reason: actual and formal argument lists differ in length
  12. Note: WordCount.java uses or overrides a deprecated API.
  13. Note: Recompile with -Xlint:deprecation for details.
  14. 2 errors
  15. 1 warning

下面是wordcount.java:

  1. //package org.apache.hadoop.examples;
  2. package org.myorg;
  3. import java.io.IOException;
  4. import java.util.StringTokenizer;
  5. import org.apache.hadoop.conf.Configuration;
  6. import org.apache.hadoop.fs.Path;
  7. import org.apache.hadoop.io.IntWritable;
  8. import org.apache.hadoop.io.Text;
  9. import org.apache.hadoop.mapreduce.Job;
  10. import org.apache.hadoop.mapreduce.Mapper;
  11. import org.apache.hadoop.mapreduce.Reducer;
  12. import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
  13. import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
  14. import org.apache.hadoop.util.GenericOptionsParser;
  15. public class WordCount {
  16. public static class TokenizerMapper
  17. extends Mapper<Object, Text, Text, IntWritable>{
  18. private final static IntWritable one = new IntWritable(1);
  19. private Text word = new Text();
  20. public void map(Object key, Text value, Context context
  21. ) throws IOException, InterruptedException {
  22. StringTokenizer itr = new StringTokenizer(value.toString());
  23. while (itr.hasMoreTokens()) {
  24. word.set(itr.nextToken());
  25. context.write(word, one);
  26. }
  27. }
  28. }
  29. public static class IntSumReducer
  30. extends Reducer<Text,IntWritable,Text,IntWritable> {
  31. private IntWritable result = new IntWritable();
  32. public void reduce(Text key, Iterable<IntWritable> values,
  33. Context context
  34. ) throws IOException, InterruptedException {
  35. int sum = 0;
  36. for (IntWritable val : values) {
  37. sum += val.get();
  38. }
  39. result.set(sum);
  40. context.write(key, result);
  41. }
  42. }
  43. public static void main(String[] args) throws Exception {
  44. Configuration conf = new Configuration();
  45. String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
  46. if (otherArgs.length != 2) {
  47. System.err.println("Usage: wordcount <in> <out>");
  48. System.exit(2);
  49. }
  50. Job job = new Job(conf, "word count");
  51. job.setJarByClass(WordCount.class);
  52. job.setMapperClass(TokenizerMapper.class);
  53. job.setCombinerClass(IntSumReducer.class);
  54. job.setReducerClass(IntSumReducer.class);
  55. job.setOutputKeyClass(Text.class);
  56. job.setOutputValueClass(IntWritable.class);
  57. FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
  58. FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
  59. System.exit(job.waitForCompletion() ? 0 : 1);
  60. }
  61. }

事先谢谢你的帮助

jjhzyzn0

jjhzyzn01#

我解决了在wordcount项目中添加jar的问题:share/hadoop/common/lib/commons-cli-1.2.jar

643ylb08

643ylb082#

试试这个:

  1. javac --classpath `hadoop classpath` .... (rest of command)

在ubuntu和debian上测试。
这工作对我来说很好。

9udxz4iz

9udxz4iz3#

在我看来,这更像是一个java问题,而不是hadoop问题。我建议使用与hadoopversion2.2.0配合使用的eclipse和hadoopdeveloper工具(除非您特别需要2.6.0)。
您也可以尝试%hadoop\u home%\share\hadoop\mapreduce\hadoop-mapreduce-client-core-version\u number.jar
另外,您在第54行得到的第一条错误消息也可以通过实现 org.apache.hadoop.util.Tool 我在apachedocs的options上找到了这个信息。
在第68行,你漏了一个论点 job.waitForCompletion() 方法调用。我建议通过考试 true 此方法的参数。

at0kjp5o

at0kjp5o4#

system.exit(job.waitforcompletion())处有小的修正?0 : 1); 它应该是这样的:system.exit(job.waitforcompletion(true)?0 : 1); 因为它将返回布尔值。。

dgtucam1

dgtucam15#

我通过执行以下操作解决了这个问题(现在使用hadoop-2.6.0更容易了)(有关更多详细信息,请参阅本教程)http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapreducetutorial.html#usage):
将以下内容添加到~/.profile文件中(以后不要忘记执行“source~/.profile”):
导出java\u home=/usr/java/default
导出路径=$java\u home/bin:$path
export hadoop_classpath=$java_home/lib/tools.jar#这对于简化java类的编译非常重要。
然后编译wordcount.java并创建一个jar:
$bin/hadoop com.sun.tools.javac.main wordcount.java
$jar cf wc.jar wordcount*.class类
然后使用如下命令执行作业:
$hadoop jar wc.jar wordcount输入/输出/

相关问题