使用hive/sparksql在url中进行字符串匹配

jtoj6r0c  于 2021-06-24  发布在  Hive
关注(0)|答案(1)|浏览(587)

我有两个表,一个包含url列表,另一个包含单词列表。我的要求是过滤掉包含单词的url。例如:

URL

https://www.techhive.com/article/3409153/65-inch-oled-4k-tv-from-lg-at-a-1300-dollar-discount.html
https://www.techradar.com/in/news/lg-c9-oled-65-inch-4ktv-price-drop
https://www.t3.com/news/cheap-oled-tv-deals-currys-august
https://indianexpress.com/article/technology/gadgets/lg-bets-big-on-oled-tvs-in-india-to-roll-out-rollable-tv-by-year-end-5823635/
https://www.sony.co.in/electronics/televisions/a1-series
https://www.amazon.in/Sony-138-8-inches-Bravia-KD-55A8F/dp/B07BWKVBYW
https://www.91mobiles.com/list-of-tvs/sony-oled-tv

Words

Sony
Samsung
Deal
Bravia

现在我想过滤任何网址有任何的话。通常我会做一个

Select url from url_table where url not like '%Sony%' or url not like '%Samsung%' or url not like '%Deal%' or not like '%Bravia%';

但这是一种繁琐且不可扩展的方法。实现这一目标的最佳方法是什么?如何对words表使用not like函数?

cigdeys3

cigdeys31#

使用正则表达式:

where url not rlike '(?i)Sony|Samsung|Deal|Bravia'
``` `(?i)` 表示大小写不敏感。
现在让我们用单词从表中构建相同的regexp。
您可以聚合表中的单词列表并将其传递给rlike。请参见此示例:

with

initial_data as (--replace with your table
select stack(7,
'https://www.techhive.com/article/3409153/65-inch-oled-4k-tv-from-lg-at-a-1300-dollar-discount.html',
'https://www.techradar.com/in/news/lg-c9-oled-65-inch-4ktv-price-drop',
'https://www.t3.com/news/cheap-oled-tv-deals-currys-august',
'https://indianexpress.com/article/technology/gadgets/lg-bets-big-on-oled-tvs-in-india-to-roll-out-rollable-tv-by-year-end-5823635/',
'https://www.sony.co.in/electronics/televisions/a1-series',
'https://www.amazon.in/Sony-138-8-inches-Bravia-KD-55A8F/dp/B07BWKVBYW',
'https://www.91mobiles.com/list-of-tvs/sony-oled-tv'
) as url ) ,

words as (-- replace with your words table
select stack (4, 'Sony','Samsung','Deal','Bravia') as word
),

sub as (--aggregate list of words for rlike
select concat('''','(?i)',concat_ws('|',collect_set(word)),'''') words_regex from words
)

select s.url
from initial_data s cross join sub --cross join with words_regex
where url not rlike sub.words_regex --rlike works fine

结果:

OK
url
https://www.techhive.com/article/3409153/65-inch-oled-4k-tv-from-lg-at-a-1300-dollar-discount.html
https://www.techradar.com/in/news/lg-c9-oled-65-inch-4ktv-price-drop
https://indianexpress.com/article/technology/gadgets/lg-bets-big-on-oled-tvs-in-india-to-roll-out-rollable-tv-by-year-end-5823635/
Time taken: 10.145 seconds, Fetched: 3 row(s)

您还可以单独计算子查询,并将其结果作为变量传递,而不是我的示例中的交叉联接。希望你有这个想法。

相关问题