2.6.0中运行我自己的wordcount.java版本

2sbarzqh  于 2021-05-30  发布在  Hadoop
关注(0)|答案(5)|浏览(350)

我正在尝试创建我自己的wordcount版本并执行它。为此,我尝试通过执行以下命令(如下所述)来创建wordcount.jarhttp://cs.smith.edu/dftwiki/index.php/hadoop_tutorial_1_--_running_wordcount 对于hadoop-2以外的早期版本。*):

javac -classpath /usr/local/hadoop-2.6.0/share/hadoop/common/*:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/* -d wordcount_classes/ WordCount.java
jar -cvf wordcount.jar -C wordcount_classes/ .

问题是我在试图编译wordcount类时在第一个命令中遇到错误。在第一个命令中,我尝试包含存在于common和mapreduce目录中的所有jar,因为我没有找到任何名为“hadoop-0.19.2-core.jar”的jar,正如我在互联网上找到的hadoop-0教程中所描述的那样*

javac -classpath /home/hadoop/hadoop/hadoop-0.19.2-core.jar -d wordcount_classes WordCount.java

当我按如下方式执行给定的wordcount时,它工作得非常好,但我无法编译自己的版本:

hadoop jar /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-*-examples-2.6.0.jar wordcount input output

我在互联网上搜索过,但我发现大多数教程都是针对hadoop-1.*的,它不同于2.6.0版本,2.6.0版本不包含任何java源文件,如我需要的wordcount.java,而且它的结构也完全不同。因此,我不得不单独下载wordcount.java文件https://github.com/apache/hadoop-common/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/wordcount.java 并复制当前目录中的wordcount.java,在该目录中执行创建jar的命令。
我已经从这个来源下载了hadoophttp://mirror.its.dal.ca/apache/hadoop/common/hadoop-2.6.0/
以下是我得到的错误的详细信息:

/usr/local/hadoop-2.6.0/share/hadoop/common/hadoop-common-2.6.0.jar(org  /apache/hadoop/fs/Path.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate': class file for org.apache.hadoop.classification.InterfaceAudience not found
  WordCount.java:54: error: cannot access Options
String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
                     ^
class file for org.apache.commons.cli.Options not found
WordCount.java:68: error: method waitForCompletion in class Job cannot be applied to given types;
System.exit(job.waitForCompletion() ? 0 : 1);
               ^
required: boolean
found: no arguments
reason: actual and formal argument lists differ in length
Note: WordCount.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
2 errors
1 warning

下面是wordcount.java:

//package org.apache.hadoop.examples;
package org.myorg;

import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;

public class WordCount {

public static class TokenizerMapper 
   extends Mapper<Object, Text, Text, IntWritable>{

private final static IntWritable one = new IntWritable(1);
private Text word = new Text();

public void map(Object key, Text value, Context context
                ) throws IOException, InterruptedException {
  StringTokenizer itr = new StringTokenizer(value.toString());
  while (itr.hasMoreTokens()) {
    word.set(itr.nextToken());
    context.write(word, one);
   }
 }
}

public static class IntSumReducer 
   extends Reducer<Text,IntWritable,Text,IntWritable> {
private IntWritable result = new IntWritable();

public void reduce(Text key, Iterable<IntWritable> values, 
                   Context context
                   ) throws IOException, InterruptedException {
  int sum = 0;
  for (IntWritable val : values) {
    sum += val.get();
  }
  result.set(sum);
  context.write(key, result);
  }
}

public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
if (otherArgs.length != 2) {
  System.err.println("Usage: wordcount <in> <out>");
  System.exit(2);
}
Job job = new Job(conf, "word count");
job.setJarByClass(WordCount.class);
job.setMapperClass(TokenizerMapper.class);
job.setCombinerClass(IntSumReducer.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
System.exit(job.waitForCompletion() ? 0 : 1);
 }
}

事先谢谢你的帮助

jjhzyzn0

jjhzyzn01#

我解决了在wordcount项目中添加jar的问题:share/hadoop/common/lib/commons-cli-1.2.jar

643ylb08

643ylb082#

试试这个:

javac --classpath `hadoop classpath` .... (rest of command)

在ubuntu和debian上测试。
这工作对我来说很好。

9udxz4iz

9udxz4iz3#

在我看来,这更像是一个java问题,而不是hadoop问题。我建议使用与hadoopversion2.2.0配合使用的eclipse和hadoopdeveloper工具(除非您特别需要2.6.0)。
您也可以尝试%hadoop\u home%\share\hadoop\mapreduce\hadoop-mapreduce-client-core-version\u number.jar
另外,您在第54行得到的第一条错误消息也可以通过实现 org.apache.hadoop.util.Tool 我在apachedocs的options上找到了这个信息。
在第68行,你漏了一个论点 job.waitForCompletion() 方法调用。我建议通过考试 true 此方法的参数。

at0kjp5o

at0kjp5o4#

system.exit(job.waitforcompletion())处有小的修正?0 : 1); 它应该是这样的:system.exit(job.waitforcompletion(true)?0 : 1); 因为它将返回布尔值。。

dgtucam1

dgtucam15#

我通过执行以下操作解决了这个问题(现在使用hadoop-2.6.0更容易了)(有关更多详细信息,请参阅本教程)http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapreducetutorial.html#usage):
将以下内容添加到~/.profile文件中(以后不要忘记执行“source~/.profile”):
导出java\u home=/usr/java/default
导出路径=$java\u home/bin:$path
export hadoop_classpath=$java_home/lib/tools.jar#这对于简化java类的编译非常重要。
然后编译wordcount.java并创建一个jar:
$bin/hadoop com.sun.tools.javac.main wordcount.java
$jar cf wc.jar wordcount*.class类
然后使用如下命令执行作业:
$hadoop jar wc.jar wordcount输入/输出/

相关问题