编辑:问题解决了-我犯了一个相当愚蠢的错误。
我有一个mapreduce管道,由map、reduce、map和reduce组成。我对第一个reduce使用sequencefileoutputformat,对第二个map使用sequencefileinputformat。我看了一下它的用法,似乎我用对了。我在这里输入的类型是intwritable和IntPairaryWritable(一个自定义arraywritable子类,使用mahout中的intpairwritable)。问题是,当读取第二个Map中的IntPairaryWritable时,当我尝试获取单个IntPairWritable时,会出现classcastexception。我不确定这是因为我使用arraywritable类时出错,还是因为我使用的sequencefile{input,output}格式有问题。我在这里和其他地方看了很多例子,我觉得这两个都做对了,但还是有一个错误。有什么帮助吗?
具体内容:
这是我的第一节课:
public static class WalkIdReducer extends MapReduceBase implements
Reducer<IntWritable, IntPairWritable, IntWritable, IntPairArrayWritable> {
@Override
public void reduce(IntWritable walk_id, Iterator<IntPairWritable> values,
OutputCollector<IntWritable, IntPairArrayWritable> output,
Reporter reporter) throws IOException {
ArrayList<IntPairWritable> value_array = new ArrayList<IntPairWritable>();
while (values.hasNext()) {
value_array.add(values.next());
}
output.collect(walk_id, IntPairArrayWritable.fromArrayList(value_array));
}
}
第二个mapper类:
public static class NodePairMapper extends MapReduceBase implements
Mapper<IntWritable, IntPairArrayWritable, IntPairWritable, Text> {
@Override
public void map(IntWritable key, IntPairArrayWritable value,
OutputCollector<IntPairWritable, Text> output,
Reporter reporter) throws IOException {
// The following line gives a ClassCastException;
// See IntPairArrayWritable.toArrayList(), below
ArrayList<IntPairWritable> values = value.toArrayList();
// other unimportant stuff
}
}
第一个mapreduce的作业配置的相关部分:
conf.setReducerClass(WalkIdReducer.class);
conf.setOutputKeyClass(IntWritable.class);
conf.setOutputValueClass(IntPairArrayWritable.class);
conf.setOutputFormat(SequenceFileOutputFormat.class);
对于第二个mapreduce:
conf.setInputFormat(SequenceFileInputFormat.class);
conf.setMapperClass(NodePairMapper.class);
最后,我的arraywritable子类:
public static class IntPairArrayWritable extends ArrayWritable
{
// These two methods are what people say is all you need for
// creating an ArrayWritable subclass
public IntPairArrayWritable() {
super(IntPairArrayWritable.class);
}
public IntPairArrayWritable(IntPairWritable[] values) {
super(IntPairArrayWritable.class, values);
}
// Some convenience methods, so I can use ArrayLists in
// other parts of the code
public static IntPairArrayWritable fromArrayList(
ArrayList<IntPairWritable> array) {
IntPairArrayWritable writable = new IntPairArrayWritable();
IntPairWritable[] values = new IntPairWritable[array.size()];
for (int i=0; i<array.size(); i++) {
values[i] = array.get(i);
}
writable.set(values);
return writable;
}
public ArrayList<IntPairWritable> toArrayList() {
ArrayList<IntPairWritable> array = new ArrayList<IntPairWritable>();
for (Writable pair : this.get()) {
// This line is what kills it. I get a ClassCastException here.
IntPairWritable int_pair = (IntPairWritable) pair;
array.add(int_pair);
}
return array;
}
}
我得到的具体错误如下:
java.lang.ClassCastException: WalkAnalyzer$IntPairArrayWritable cannot be cast to org.apache.mahout.common.IntPairWritable
at WalkAnalyzer$IntPairArrayWritable.toArrayList(WalkAnalyzer.java:231)
at WalkAnalyzer$NodePairMapper.map(WalkAnalyzer.java:84)
at WalkAnalyzer$NodePairMapper.map(WalkAnalyzer.java:77)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
我很困惑,为什么从arraywritable得到的get()方法是 WalkAnalyzer$IntPairArrayWritable
-我希望get()返回 IntPairArrayWritable
,如api中所述。
编辑
我发现了问题。我是如何为intpairarraywriteable编写构造函数的。我打电话来了 super(IntPairArrayWritable.class);
当我应该打电话的时候 super(IntPairWritable.class);
. 代码实际上应该是这样的:
public static class IntPairArrayWritable extends ArrayWritable
{
// These two methods are what people say is all you need for
// creating an ArrayWritable subclass
public IntPairArrayWritable() {
super(IntPairWritable.class);
}
public IntPairArrayWritable(IntPairWritable[] values) {
super(IntPairWritable.class, values);
}
}
我想对arraywritable子类使用一个不那么明显混淆的名称是个好主意,这样就更容易发现错误。
1条答案
按热度按时间4szc88ey1#
检查导入语句中的intpairwritable。看起来您在Map程序中选择了错误的包名称,因此正在强制转换到一个不同的类,即使它的名称也是intpairwritable。