我有以下输入称为movieusertagfltr:
(260,{(260,starwars),(260,George Lucas),(260,sci-fi),(260,cult classic),(260,Science Fiction),(260,classic),(260,supernatural powers),(260,nerdy),(260,Science Fiction),(260,critically acclaimed),(260,Science Fiction),(260,action),(260,script),(260,"imaginary world),(260,space),(260,Science Fiction),(260,"space epic),(260,Syfy),(260,series),(260,classic sci-fi),(260,space adventure),(260,jedi),(260,awesome soundtrack),(260,awesome),(260,coming of age)})
(858,{(858,Katso Sanna!)})
(924,{(924,slow),(924,boring)})
(1256,{(1256,Marx Brothers)})
它遵循以下模式: (movieId:int, tags:bag{(movieId:int, tag:cararray),...})
基本上,第一个数字代表一个电影id,随后的包包含与该电影相关联的所有关键字。我想对这些关键词进行分组,这样我就有了这样的输出:
(260,{(1,starwars),(1,George Lucas),(1,sci-fi),(1,cult classic),(4,Science Fiction),(1,classic),(1,supernatural powers),(1,nerdy),(1,critically acclaimed),(1,action),(1,script),(1,"imaginary world),(1,space),(1,"space epic),(1,Syfy),(1,series),(1,classic sci-fi),(1,space adventure),(1,jedi),(1,awesome soundtrack),(1,awesome),(1,coming of age)})
(858,{(1,Katso Sanna!)})
(924,{(1,slow),(1,boring)})
(1256,{(1,Marx Brothers)})
请注意,标签科幻小说已经出现了4次与id 260的电影。使用groupby和count,我使用以下脚本为每部电影计算不同的关键字:
sum = FOREACH group_data {
unique_tags = DISTINCT movieUserTagFltr.tags::tag;
GENERATE group, COUNT(unique_tags) as tag;
};
但它只返回一个全局计数,我需要一个局部计数。所以我想的逻辑是:
result = iterate over each tuple of group_data {
generate a tuple with $0, and a bag with {
foreach distinct tag that group_data has on it's $1 variable do {
generate a tuple like: (tag_name, count of how many times that tag appeared on $1)
}
}
}
1条答案
按热度按时间mmvthczy1#
您可以将原始输入展平,以便每个
movieID
以及tag
是他们自己的记录。然后分组movieID
以及tag
得到每个组合的计数。最后,分组movieID
这样你就有了一袋标签,每部电影都会有计数。假设你从
movieUserTagFltr
使用您描述的模式:您的最终方案是: