在pig中对包的不同值执行计数

woobm2wo  于 2021-06-24  发布在  Pig
关注(0)|答案(1)|浏览(364)

我有一个关于Pig的问题,当执行似乎是两个级别的分组。例如,假设我有一些示例输入数据,如:

email_id:chararray    from:chararray        to:bag{recipients:tuple(recipient:chararray)}
e1                    user1@example.com     {(friend1@example.com),(friend2@example.com),(friend3@myusers.com)}
e2                    user1@example.com     {(friend1@example.com),(friend4@example.com)}
e3                    user1@example.com     {(friend5@example.com)}
e4                    user2@example.com     {(friend2@example.com),(friend4@example.com)}

因此,每一行都是一封来自用户“from”到用户“to”的电子邮件。
我最终想要一份所有发件人和他们发送电子邮件给的所有人的列表,包括每个人发送的电子邮件数量,从高到低排序,例如:

user1@example.com     {(friend1@example.com, 2), (friend2@example.com, 1), (friend3@example.com, 1), (friend4@example.com, 1), (friend5@example.com, 1)}
user2@example.com     {(friend2@example.com, 1), (friend4@example.com, 1)}

最好的方法来解决Pig的想法将不胜感激!

xuo3flqw

xuo3flqw1#

以下是脚本的一个版本:

inpt = load '/pig_data/pig_fun/input/from_senders.txt' as (email_id:chararray, from:chararray, to:bag{recipients:tuple(recipient:chararray)});

pivot = foreach inpt generate from, FLATTEN(to);
pivot = foreach pivot generate from, to::recipient as recipient;
dump pivot;
/*
(user1@example.com,friend1@example.com)
(user1@example.com,friend2@example.com)
(user1@example.com,friend3@myusers.com)
(user1@example.com,friend1@example.com)
(user1@example.com,friend4@example.com)
(user1@example.com,friend5@example.com)
(user2@example.com,friend2@example.com)
(user2@example.com,friend4@example.com)

* /

grp = group pivot by (from, recipient);
with_count = foreach grp generate FLATTEN(group), COUNT(pivot) as count;
dump with_count;
/*
(user1@example.com,friend1@example.com,2)
(user1@example.com,friend2@example.com,1)
(user1@example.com,friend3@myusers.com,1)
(user1@example.com,friend4@example.com,1)
(user1@example.com,friend5@example.com,1)
(user2@example.com,friend2@example.com,1)
(user2@example.com,friend4@example.com,1)

* /

to_bag = group with_count by from;
result = foreach to_bag {
    order_by_count = order with_count by count desc;
    generate group as from, order_by_count.(recipient, count);
};
dump result;
/*
(user1@example.com,{(friend1@example.com,2),(friend2@example.com,1),(friend3@myusers.com,1),(friend4@example.com,1),(friend5@example.com,1)})
(user2@example.com,{(friend2@example.com,1),(friend4@example.com,1)})

* /

希望有帮助。

相关问题