我正试图在一些wikipedia转储文件(以压缩的bz2形式)上运行一个带有javaMap器/reducer的hadoop流作业。我正在尝试使用wikihadoop,这是wikimedia最近发布的一个接口。
wikireader\u mapper.java
package courseproj.example;
// Mapper: emits (token, 1) for every article occurrence.
public class WikiReader_Mapper extends MapReduceBase implements Mapper<Text, Text, Text, IntWritable> {
// Reuse objects to save overhead of object creation.
private final static Text KEY = new Text();
private final static IntWritable VALUE = new IntWritable(1);
@Override
public void map(Text key, Text value, OutputCollector<Text, IntWritable> collector, Reporter reporter)
throws IOException {
KEY.set("article count");
collector.collect(KEY, VALUE);
}
}
维基阅读器.java
package courseproj.example;
//Reducer: sums up all the counts.
public class WikiReader_Reducer extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
private final static IntWritable SUM = new IntWritable();
public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> collector,
Reporter reporter) throws IOException {
int sum = 0;
while (values.hasNext()) {
sum += values.next().get();
}
SUM.set(sum);
collector.collect(key, SUM);
}
}
我执行的命令是
hadoop jar lib/hadoop-streaming-2.0.0-cdh4.2.0.jar \
-libjars lib2/wikihadoop-0.2.jar \
-D mapreduce.input.fileinputformat.split.minsize=300000000 \
-D mapreduce.task.timeout=6000000 \
-D org.wikimedia.wikihadoop.previousRevision=false \
-input enwiki-latest-pages-articles10.xml-p000925001p001325000.bz2 \
-output out \
-inputformat org.wikimedia.wikihadoop.StreamWikiDumpInputFormat \
-mapper WikiReader_Mapper \
-reducer WikiReader_Reducer
我收到的错误信息是
Error: java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:424)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:103)
Caused by: java.io.IOException: Cannot run program "WikiReader_Mapper": java.io.IOException: error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:209)
我更熟悉新的hadoopapi和旧的。因为我的mapper和reducer代码在两个不同的文件中,所以我在哪里定义作业的jobconf配置参数,同时遵循hadoop streaming的命令结构(显式设置mapper和reducer类)。有没有一种方法可以将mapper和reducer代码打包到一个类中(扩展configured和implements工具,这是在新的api中完成的),并将类名传递给hadoop流式处理命令行,而不是分别设置map和reduce类?
1条答案
按热度按时间zi8p0yeb1#
流媒体使用旧的api(
org.apache.hadoop.mapred
)-但是Map器和还原器类扩展了新的api类(org.apache.hadoop.mapreduce
).尝试更改Map器以实现
org.apache.hadoop.mapred.Mapper
,以及要实现的减速器org.apache.hadoop.mapred.Reducer
,例如: