我有一个pig脚本,它使用宏两次,在同一个关系上,但参数不同;对于每次使用,我在不同的字段中过滤相同的关系。宏的形状如下:
DEFINE doubleGroupJoin (mainField, mainRelation) returns out {
valid = FILTER $mainRelation BY $mainField != '';
r1 = FOREACH (GROUP valid BY $mainField) GENERATE
field1_1, field1_2, ...;
r2 = FOREACH (GROUP valid BY ($mainField, otherfield1, ...) GENERATE
field2_1, field2_2, ...;
$out = FOREACH (JOIN R1 BY field1_1, R2 BY field1_2) GENERATE
final1, final2, ...;
}
在脚本中,我有以下内容:
-- Output1
finalR1 = doubleGroupJoin('field1', initialData);
STORE finalR1 INTO '$output/R1';
-- Output2
finalR2 = doubleGroupJoin('field2', initialData);
STORE finalR2 INTO '$output/R2';
如果我注解掉output1或output2块,则作业可以正常工作,但如果我尝试同时使用这两个块,则会出现以下错误:
java.lang.ClassCastException: org.apache.pig.data.BinSedesTuple cannot be cast to java.lang.String
at org.apache.pig.backend.hadoop.HDataType.getWritableComparableTypes(HDataType.java:106)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map.collect(PigGenericMapReduce.java:111)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:284)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:277)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
在这里使用pig 0.12.0。有什么关于为什么会这样的建议吗?
暂无答案!
目前还没有任何答案,快来回答吧!