在pig中将具有多个字段的元组拆分为具有单个字段的元组

hmtdttj4  于 2021-06-24  发布在  Pig
关注(0)|答案(0)|浏览(218)

我有不同长度的元组。我试图将它们转换为只有一个字段的元组(每个字段都是一个Map)。
原始数据:

  1. dump entryArray;
  2. ([symbol#HIG,security_type#EQUITY,foreign_entry_id#743094])
  3. ([symbol#PEW,security_type#EQUITY,foreign_entry_id#743084])
  4. ([symbol#AFFY,security_type#EQUITY,foreign_entry_id#5585],[symbol#RFG,security_type#ETF,foreign_entry_id#5586],[symbol#SCHW,security_type#EQUITY,foreign_entry_id#5587],[symbol#VWO,security_type#ETF,foreign_entry_id#5588])

我希望输出是(每个字段仍然是map):

  1. ([symbol#HIG,security_type#EQUITY,foreign_entry_id#743094])
  2. ([symbol#PEW,security_type#EQUITY,foreign_entry_id#743084])
  3. ([symbol#AFFY,security_type#EQUITY,foreign_entry_id#5585])
  4. ([symbol#RFG,security_type#ETF,foreign_entry_id#5586])
  5. ([symbol#SCHW,security_type#EQUITY,foreign_entry_id#5587])
  6. ([symbol#VWO,security_type#ETF,foreign_entry_id#5588])

我试过: entry = FOREACH entryArray GENERATE FLATTEN(TOBAG()); 输出的格式相同,但字段似乎不再是Map:

  1. entry = FOREACH entryArray GENERATE FLATTEN(TOBAG());
  2. dump entry;
  3. ([symbol#HIG,security_type#EQUITY,foreign_entry_id#743094])
  4. ([symbol#PEW,security_type#EQUITY,foreign_entry_id#743084])
  5. ([symbol#AFFY,security_type#EQUITY,foreign_entry_id#5585])
  6. ([symbol#RFG,security_type#ETF,foreign_entry_id#5586])
  7. ([symbol#SCHW,security_type#EQUITY,foreign_entry_id#5587])
  8. ([symbol#VWO,security_type#ETF,foreign_entry_id#5588])
  9. security_type = FOREACH entry GENERATE FLATTEN($0#'security_type');
  10. it throws:
  11. ERROR 1052: Cannot cast bytearray to map with schema :map
  12. org.apache.pig.impl.logicalLayer.validators.TypeCheckerException: ERROR 1059: <line 18, column 16> Problem while reconciling output schema of ForEach
  13. at org.apache.pig.newplan.logical.visitor.TypeCheckingRelVisitor.throwTypeCheckerException(TypeCheckingRelVisitor.java:141)
  14. at org.apache.pig.newplan.logical.visitor.TypeCheckingRelVisitor.visit(TypeCheckingRelVisitor.java:181)
  15. at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:75)
  16. ......

如有任何建议,我们将不胜感激。谢谢!

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题