postgresql 连接以合并多个重复数据消除策略

xv8emn3q  于 2023-06-05  发布在  PostgreSQL
关注(0)|答案(1)|浏览(542)

我有一个数据集,我试图用两种方法同时清理和消除重复数据。
例如-如果我们忽略标点符号(包括空格)、词序和缩写,这些公司名称都是一样的:

company_id |    company_name    |        words        | words_wo_spaces |    words_sorted     | sorted_words_wo_spaces 
------------+--------------------+---------------------+-----------------+---------------------+------------------------
         17 | the usa co         | {THE,USA,COMPANY}   | THEUSACOMPANY   | {COMPANY,THE,USA}   | COMPANYTHEUSA     
         18 | the u s a company  | {THE,U,S,A,COMPANY} | THEUSACOMPANY   | {A,COMPANY,S,THE,U} | ACOMPANYSTHEU     
         19 | usa company, the   | {USA,COMPANY,THE}   | USACOMPANYTHE   | {COMPANY,THE,USA}   | COMPANYTHEUSA     
         20 | u s a company, the | {U,S,A,COMPANY,THE} | USACOMPANYTHE   | {A,COMPANY,S,THE,U} | ACOMPANYSTHEU

除了company_idcompany_name之外,我有一些帮助函数来创建所有的列,这些函数用于拆分和排序单词,并重新编码单词以删除常见的缩写(例如,用“company”替换“co”)。我不把那些辅助函数放在这个问题上。
如果我们看words_wo_spaces列,有两对匹配的公司:17 / 1819 / 20。如果我们按sorted_words_wo_spaces分组,则有两对不同的匹配公司:17 / 1918 / 20
我真正想要的是一个查询(或两个或三个),它可以使用两种方法对公司进行分组,以确定它们都是相同的。
查询1

SELECT 
  array_agg(company_id) ids
  , array_agg(company_name) names
FROM companies
GROUP BY words_wo_spaces

-- Returns

--    ids   | words_wo_spaces 
-- ---------+-----------------
--  {19,20} | USACOMPANYTHE
--  {17,18} | THEUSACOMPANY

问题二

SELECT 
  array_agg(company_id) ids
  , array_agg(company_name) names
FROM companies
GROUP BY sorted_words_wo_spaces

-- Returns

--    ids   | sorted_words_wo_spaces 
-- ---------+------------------------
--  {18,20} | ACOMPANYSTHEU
--  {17,19} | COMPANYTHEUSA

但我不知道如何将两者结合起来。如果17=18,18=20,17=19,那么它们都是一样的。有没有可能把这个逻辑放到sql查询中?
我确实尝试将这两个查询的结果连接起来,但这没有帮助(正如预期的那样,一旦我解决了它)。

WITH
  unsorted_groups AS <query 1>
  , sorted_groups AS <query 2>
SELECT 
  array_cat(sorted_groups.ids, unsorted_groups.ids) ids
FROM sorted_groups 
JOIN unsorted_groups 
  ON unsorted_groups.ids && sorted_groups.ids;

-- Returns

--      ids   
-- ---------------
--  {18,20,19,20}
--  {18,20,17,18}
--  {17,19,19,20}
--  {17,19,17,18}

有用的是一个返回这样一个表的查询,这样id数组就可以被排序&然后用于将所有行分组为一个(这个理想的结果在每行的ids数组中有每个id)。

--      ids   
-- ---------------
--  {18,17,19,20}
--  {18,20,17,19}
--  {17,19,18,20}
--  {17,19,20,18}
vybvopom

vybvopom1#

您可以使用overlap operator &&将两个查询连接起来,如下:

WITH Q1 AS
(
  SELECT 
    array_agg(company_id ORDER BY company_id) ids -- The order by is necessary for the aggregation in the joined query
  FROM companies
  GROUP BY words_wo_spaces
),
Q2 AS
(
  SELECT 
    array_agg(company_id ORDER BY company_id) ids -- The order by is necessary
  FROM companies
  GROUP BY sorted_words_wo_spaces
),
Joined_Q AS
(
  SELECT Q1.ids Q1_ids,
         UNNEST(Q2.ids) Q2_ids -- for each array of ids from q1, unnest the matched array of ids from q2
  FROM Q1 JOIN Q2
  ON Q1.ids && Q2.ids
)
SELECT DISTINCT ARRAY_AGG(Q2_ids ORDER BY Q2_ids) -- aggregate the unnested q2 ids as a single array.
FROM Joined_Q
GROUP BY Q1_ids

Demo

相关问题