在pig中按连接关系计数

brccelvz  于 2021-06-24  发布在  Pig
关注(0)|答案(1)|浏览(297)

我正在尝试从hdfs加载两个文件到pig。当我加入司机关系和卡车关系后,我想数一数。如何计算相关行数?我试过这个,但它给我的计数与组没有一个计数:

truck_temp = FOREACH (GROUP truck_join BY  drivers_info::driverId) { GENERATE  group, COUNT(truck_join); };
drivers_load = LOAD '/Pig-Practice/drivers.csv' USING PigStorage(',') AS (driverId:int,name:chararray,ssn:biginteger,location:chararray,certified:chararray,wageplan:chararray);
drivers_info = FOREACH ( GROUP drivers_load BY (driverId,name)) GENERATE group.driverId,group.name;

event_load = LOAD '/Pig-Practice/truck_event_text_partition.csv' USING PigStorage(',') AS (driverId:int, truckId:int, eventTime:chararray,
eventType:chararray, longitude:double, latitude:double,
eventKey:chararray, correlationId:long, driverName:chararray,
routeId:long,routeName:chararray,eventDate:chararray);
truck_events1 = FILTER event_load BY $0 >1;

truck_events2 =  FOREACH (GROUP truck_events1 BY (driverId,driverName,routeId,routeName) ) GENERATE group.driverId,group.driverName,group.routeId,group.routeName;

truck_join =  JOIN drivers_info BY driverId, truck_events2 BY driverId;
jjjwad0x

jjjwad0x1#

为了获得一个连接后的总计数,您需要对所有连接进行分组。
count要求前面的group all语句用于全局计数,group by语句用于组计数。引用:count

truck_temp = FOREACH (GROUP truck_join ALL) 
             { 
                 GENERATE  COUNT(truck_join); 
             };

相关问题