如何在Hive SQL中分别按多列分组?

t98cgbkg  于 2023-10-18  发布在  Hive
关注(0)|答案(1)|浏览(168)

假设我有一个名为t的表,有两列foobar
| Foo|酒吧|
| --|--|
| 1 | 11 |
| 1 | 11 |
| 2 | 11 |
| 2 | 11 |
| 2 | 11 |
| 3 | 11 |
| 3 | 12 |
| 3 | 12 |
现在我想分别计算foobar的不同值的出现次数,并将它们聚合为ARRAY<MAP<BIGINT, BIGINT>>
在本示例中:

  • foo == 1出现2次;
  • foo = 2出现3次;
  • foo = 3出现3次;
  • bar == 11出现6次;
  • bar == 12出现了2次。

因此,生成的表应该如下所示:
| 名称|CNT|
| --|--|
| “foo”|[{1:2},{2:3},{3:3}]|
| “酒吧”|[2019 - 02 - 16 00:01:00]|
我目前的做法有点像这样:

WITH t_foo AS (
  SELECT
    "foo" AS name,
   COLLECT_LIST(MAP(val, cnt)) AS cnt
  FROM (
    SELECT
      foo AS val,
      COUNT(*) AS cnt
    FROM
      t
    GROUP BY
      foo
  ) AS tt
),
t_bar AS (
  SELECT
    "bar" AS name,
   COLLECT_LIST(MAP(val, cnt)) AS cnt
  FROM (
    SELECT
      bar AS val,
      COUNT(*) AS cnt
    FROM
      t
    GROUP BY
      bar
  ) AS tt
)
SELECT * FROM t_foo
UNION ALL 
SELECT * FROM t_bar

这是可行的,但它似乎是重复的。事实上,我不仅有foobar,还有十几个其他列需要处理。有没有更聪明的方法来解决这个问题?

iq3niunx

iq3niunx1#

为了推广这段代码,您应该需要动态查询,但是这种方法可能很繁重,并且容易受到sql注入攻击。
然而,您仍然可以在不使用动态查询的情况下做一些事情,那就是:

  • 一次计算所有计数,使用窗口函数而不是聚合
  • 应用所有COLLECT_LIST操作的并集。
WITH cte AS (
    SELECT DISTINCT foo, COUNT(*) OVER(PARTITION BY foo) AS cnt_foo,
                    bar, COUNT(*) OVER(PARTITION BY bar) AS cnt_bar
    FROM t
)
SELECT "foo" AS name, COLLECT_LIST(MAP(foo, cnt_foo)) AS cnt FROM cte
UNION ALL 
SELECT "bar" AS name, COLLECT_LIST(MAP(bar, cnt_bar)) AS cnt FROM cte

应该比原来的表现更好。

相关问题