python 从网站的项目列表中提取所有链接

13z8s7eq 于 2023-08-02 发布在 Python

关注(0)|答案(1)|浏览(129)

我试图使用Python从这个网站中提取所有壁画链接：https://streetartcities.com/cities/losangeles/artworks。但我的代码不能正常工作，并给我一个空输出。我不知道网站是否包含JavaScript或其他编程语言。有谁能简单地解释一下这个问题并帮助我调试代码吗？谢谢你，谢谢

from requests_html import HTMLSession
import warnings

warnings.filterwarnings("ignore", category=DeprecationWarning) ## ignore the Deprecation warning message

s = HTMLSession()

def get_mural_links():
    url = 'https://streetartcities.com/cities/losangeles/artworks'
    links = []
    r = s.get(url)

    grid = r.html.find('div.p-5 div.grid.mt-4.grid-cols-2.md\:grid-cols-3.gap-3')
    for item in grid:
        links.append(item.find('a', first=True).attrs['href'])
    return links

print(get_mural_links())

字符串

python

来源：https://stackoverflow.com/questions/76808290/extract-all-links-from-a-list-of-items-from-a-website

1条答案

按热度按时间

pkbketx91#

您尝试查找的元素稍后会使用JavaScript延迟添加到站点。这就是说，你需要Selenium。Selenium以比requests更人性化的方式浏览页面。

import time

from selenium import webdriver
from selenium.webdriver.common.by import By

def get_mural_links():
  url = 'https://streetartcities.com/cities/losangeles/artworks'
  grid_selector = 'div.p-5 div.grid.mt-4.grid-cols-2.md\:grid-cols-3.gap-3'

  # Instantiate a driver instance.
  # This will be the medium for us to scrape the links.
  driver = webdriver.Chrome()
  
  # Go to the given page, then wait for two seconds.
  driver.get(url)
  time.sleep(2)
  
  # Find the grid and iterate over its children.
  grid = driver.find_element(By.CSS_SELECTOR, grid_selector)
  links = []
  
  for child in grid.find_elements(By.CSS_SELECTOR, 'a'):
    links.append(child.get_attribute('href'))
  
  return links

字符串
试试看：

print(get_mural_links()[:10])

'''
[
  'https://streetartcities.com/cities/losangeles/markers/e406c20c-7774-4aad-841a-7c5e10ec707b',
  'https://streetartcities.com/cities/losangeles/markers/736dabb7-5d3b-4231-8c11-1c289b643125',
  'https://streetartcities.com/cities/losangeles/markers/906d66d7-60c6-4e0d-8afc-db5591f724ca',
  'https://streetartcities.com/cities/losangeles/markers/4ad5536b-7447-41d3-8f32-7c7835da9d75',
  'https://streetartcities.com/cities/losangeles/markers/94cfc0d9-cd03-44d3-97cf-90dadbaa3f1b',
  'https://streetartcities.com/cities/losangeles/markers/5597',
  'https://streetartcities.com/cities/losangeles/markers/11514',
  'https://streetartcities.com/cities/losangeles/markers/c13e8d5b-a066-4a35-8236-8a3446b82251',
  'https://streetartcities.com/cities/losangeles/markers/4923',
  'https://streetartcities.com/cities/losangeles/markers/11991'
]
'''

型

赞(0）回复(0）举报 2023-08-02

我来回答

python 从网站的项目列表中提取所有链接

1条答案

相关问题

热门标签

最新问答