从pig中的sum获取max

3lxsmp7m  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(301)
player = LOAD 'ass2_player' USING org.apache.hive.hcatalog.pig.HCatLoader();
player = FOREACH player GENERATE
    (chararray)$3 AS tmID,
    (int)$1 AS year,
    (int)$8 AS points;
group_data = GROUP player BY (year, tmID);
sum_data = FOREACH group_data GENERATE group, SUM(player.points) AS tot_points;
max_data = FOREACH sum_data GENERATE FLATTEN(group), MAX(sum_data.tot_points);
DUMP max_data;

我只想选择每年得分最高的球队的tmid。
如何获取整行或部分字段或具有最大值的行。比如,分组后,分组只包含“year”,tuple取“tmid,tot\u points”。我如何得到像:(年,tmid,总积分)为每年。

bvjveswy

bvjveswy1#

你就快到了。以下是 sum_data :

((year, tmID), tot_points)

从这里开始,你需要 group 年复一年 maxtot_points . 如果你愿意的话就容易多了 flatten 中的组 sum_data 仅步骤,例如:

sum_data = FOREACH group_data GENERATE flatten(group) as (year, tmID), SUM(player.points) AS tot_points;

sum_data_grouped = GROUP sum_data BY year;
max_data = FOREACH sum_data_grouped GENERATE group AS year, MAX(sum_data.tot_points) AS max_points, sum_data.tmpID AS tmID;

您的最终脚本应该如下所示:

player = LOAD 'ass2_player' USING org.apache.hive.hcatalog.pig.HCatLoader();
player = FOREACH player GENERATE (chararray)$3 AS tmID, (int)$1 AS year, (int)$8 AS points;
group_data = GROUP player BY (year, tmID);
sum_data = FOREACH group_data GENERATE flatten(group) AS (year, tmID), SUM(player.points) AS tot_points;
sum_data_grouped = GROUP sum_data BY year;
max_data = FOREACH sum_data_grouped GENERATE group AS year, MAX(sum_data.tot_points) AS max_points, sum_data.tmpID AS tmID;

ps:我在手机上写的,没有测试。告诉我你得到了什么。

相关问题