pig:扁平化错误

kfgdxczn 于 2021-06-25 发布在 Pig

关注(0)|答案(1)|浏览(500)

我有亲戚 X 有结构的 X: {group: chararray,inboundCount: {(name: chararray,inb: long)},outboundCount: {(name: chararray,out: long)}} 具体如下：

(IAD,{},{(IAD,25)})
(LAX,{},{(LAX,2)})
(ORD,{(ORD,27)},{})
(PDX,{},{(PDX,3)}) 
(SFO,{(SFO,3)},{})

我想要一个如下结构的输出 final: {airport: chararray,inbound: long,outbound: long} 不带输出：

(IAD,,25)
(LAX,,2)
(ORD,27,)
(PDX,,3)
(SFO,3,)

我尝试了下面的代码，它给出了我想要的输出结构。但什么也没印出来。是因为空值包吗？。

final = foreach X generate group as airport,FLATTEN(inboundCount.inb) as inbound,FLATTEN(outboundCount.out) as outbound;

请帮帮我。
编辑我得到了这个关系 x 通过执行以下命令。

A= load '/user/hduser/airline.csv' using PigStorage(',') as (year:int,month:int,dayofmonth:int,dayofweek:int,dep:int,CRS:int,Arr:int,CRSArr:int,UniqueCarrier:chararray,FlightNum:int,TailNum:chararray,ActualElapsedTime:int,CRSElapsed:int,AirTime:int,ArrDelay:int,DepDelay:int,Origin:chararray,Dest:chararray,Distance:int,TaxiIn:int,TaxiOut:int,Cancelled:int,CancelCode:chararray,Diverted:int,CarrierDelay:int,WeatherDelay:int,NASDelay:int,SecurityDelay:int,LateAircraft:int);
B= foreach A generate year,month,UniqueCarrier,FlightNum,TailNum,Origin,Dest;
inbound = group B by Dest;
inboundCount = foreach inbound generate group,COUNT(B.FlightNum) as inb;
outbound = group B by Origin;
outboundCount = foreach outbound generate group,COUNT(B.FlightNum) as out;
X = COGROUP inboundCount BY name, outboundCount BY name;

样品输入记录：
2008,1,31,41757115524001758，ua，114，n845ua，243243217362362，lax，作战需求文件，1745,11,15,0，，0,0,0362,0,0

apache-pig Flatten

来源：https://stackoverflow.com/questions/32146582/pig-flatten-error

1条答案

按热度按时间

6tdlim6h1#

你就快到了。请试试这个。用求和而不是展平

A= load '/user/hduser/airline.csv' using PigStorage(',') as (year:int,month:int,dayofmonth:int,dayofweek:int,dep:int,CRS:int,Arr:int,CRSArr:int,UniqueCarrier:chararray,FlightNum:int,TailNum:chararray,ActualElapsedTime:int,CRSElapsed:int,AirTime:int,ArrDelay:int,DepDelay:int,Origin:chararray,Dest:chararray,Distance:int,TaxiIn:int,TaxiOut:int,Cancelled:int,CancelCode:chararray,Diverted:int,CarrierDelay:int,WeatherDelay:int,NASDelay:int,SecurityDelay:int,LateAircraft:int);

B= foreach A generate year,month,UniqueCarrier,FlightNum,TailNum,Origin,Dest;

inbound = group B by Dest;

inboundCount = foreach inbound generate group,COUNT(B.FlightNum) as inb;

outbound = group B by Origin;

outboundCount = foreach outbound generate group,COUNT(B.FlightNum) as out;

X = COGROUP inboundCount BY name, outboundCount BY name;

final_data = FOREACH X GENERATE group as airport, SUM(inboundCount.inb) as inb, SUM(outboundCount.out) as out;

dump final_data;

最终数据的转储将为您提供预期的结果。

(IAD,,25)
(LAX,,2)
(ORD,27,)
(PDX,,3)
(SFO,3,)

如果需要，您仍然可以将null计数替换为0

final_null_check = FOREACH final_data GENERATE airport,(inb is null ? 0 :inb) as inb_cnt, (out is null ? 0 : out) as out_cnt;

在空检查之后，如果您转储final\u null\u check关系，您将得到如下输出

(IAD,0,25)
 (LAX,0,2)
 (ORD,27,0)
 (PDX,0,3)
 (SFO,3,0)

赞(0）回复(0）举报 2021-06-26

我来回答

pig:扁平化错误

1条答案

相关问题

热门标签

最新问答