selenium 网页抓取动态下拉列表

6l7fqoea  于 2023-01-26  发布在  其他
关注(0)|答案(1)|浏览(140)

我试着把所有的产品都整理成:url ='https://www. danfoss. com/zh/products/dps/阀门和执行器/阀门/pvg比例阀/# tab-概述'
我的问题是最后两个产品的html标签在inspect和view页面源代码中都不存在,除非我点击显示更多,我怎么才能把它们刮下来呢?
下面是我代码:

a=soup.find('div',attrs={"class":"tile-group-component product-group"})
b=a.find_all("li") 
for c in b:
    d=c.find('a',href=True) 
    print(d['href'])

但是它没有显示最后两个

rnmwe5a2

rnmwe5a21#

下面是获取数据的一种方法,作为 Dataframe (从该页面抓取API端点):

import requests
import pandas as pd

pd.set_option('display.max_columns', None, 'display.max_colwidth', None)

headers = {
    'content-type': 'application/json',
    'origin': 'https://www.danfoss.com',
    'accept': 'application/json, text/plain, */*',
    'referer': 'https://www.danfoss.com/en/products/dps/valves-and-actuators/valves/pvg-proportional-valves/',
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36'
}

s = requests.Session()
s.headers.update(headers)
big_df = pd.DataFrame()
for x in range(0, 10, 5):
    payload = '{"culture":"en","currentPageId":2133,"offset":'+ str(x) + ',"pageSize":7,"tags":"int_products=valves/pvg-proportional-valves,segments=dps","types":"product,productExternalContent","sortOption":""}'
    r = s.post('https://www.danfoss.com/umbraco/Api/TagBasedGroupComponentsApi/GetProductGroupComponentItems', data=payload)
    df = pd.json_normalize(r.json()['items'])
    big_df = pd.concat([big_df, df], axis=0, ignore_index=True)
print(big_df)

最终结果:

id  formattedStartDate  formattedEndDate    isExternalLink  title   image   imageAltText    description startDate   endDate location    eventTypes  hasErrors   hasWarnings errors  warnings    link.newWindow  link.url
0   https://danfoss-webex-umbraco-prodblue.azurewebsites.net/en/products/dps/valves-and-actuators/valves/pvg-proportional-valves/pvg-16-proportional-valves/    None    None    False   PVG 16 proportional valves  /media/14374/pvg-16-proportional-valve.jpg  PVG 16 proportional valve   <p>Low-flow PVG 16 covers simple to complex needs. Proven technology and components for customized valve stacks.</p>    None    None    None    None    False   False   []  []  False   /en/products/dps/valves-and-actuators/valves/pvg-proportional-valves/pvg-16-proportional-valves/
1   https://danfoss-webex-umbraco-prodblue.azurewebsites.net/en/products/dps/valves-and-actuators/valves/pvg-proportional-valves/pvg-32-proportional-valves/    None    None    False   PVG 32 proportional valves  /media/14375/pvg-32-proportional-valve-with-pve-series-7-electrohydraulic-actuators.jpg PVG 32 proportional valves  <p>PVG 32 is designed for maximum flexibility configurable as an advanced electrically controlled proportional valve. As well as a load sensing directional control valve.</p>  None    None    None    None    False   False   []  []  False   /en/products/dps/valves-and-actuators/valves/pvg-proportional-valves/pvg-32-proportional-valves/
2   https://danfoss-webex-umbraco-prodblue.azurewebsites.net/en/products/dps/valves-and-actuators/valves/pvg-proportional-valves/pvg-100-proportional-valves/   None    None    False   PVG 100 proportional valves /media/14379/pvg-100-load-independent-proportional-valve.jpg    PVG 100 proportional valves <p>The flow-sharing technology of the post-compensated, load-sensing PVG 100 is ideal in different systems. Especially in systems where flow demand regularly pushes supply to the limits.</p>  None    None    None    None    False   False   []  []  False   /en/products/dps/valves-and-actuators/valves/pvg-proportional-valves/pvg-100-proportional-valves/
3   https://danfoss-webex-umbraco-prodblue.azurewebsites.net/en/products/dps/valves-and-actuators/valves/pvg-proportional-valves/pvg-120-proportional-valves/   None    None    False   PVG 120 proportional valves /media/14376/pvg-120-proportional-valve-with-pve-series-7-electrohydraulic-actuator.jpg PVG 120 proportional valve  <p>Load sensing valve suitable for high flow and pressure. The PVG 120 is built to take control.</p>    None    None    None    None    False   False   []  []  False   /en/products/dps/valves-and-actuators/valves/pvg-proportional-valves/pvg-120-proportional-valves/
4   https://danfoss-webex-umbraco-prodblue.azurewebsites.net/en/products/dps/valves-and-actuators/valves/pvg-proportional-valves/pvg-128-256-proportional-valves/   None    None    False   PVG 128/256 proportional valves /media/14391/pvg-128-256-proportional-valve.jpg PVG 128/256 proportional valves <p>For the most demanding applications, only the highest flow and the highest pressure will do. The PVG 128 and PVG 256 are made for controlling the toughest applications.</p> None    None    None    None    False   False   []  []  False   /en/products/dps/valves-and-actuators/valves/pvg-proportional-valves/pvg-128-256-proportional-valves/
5   https://danfoss-webex-umbraco-prodblue.azurewebsites.net/en/products/dps/valves-and-actuators/valves/pvg-proportional-valves/pvg-combinations/  None    None    False   PVG combinations    /media/14378/pvg-proportional-valves-combinations.jpg   PVG combinations    <p>Danfoss offers solutions which makes it possible to combine up to four different valve families.</p> None    None    None    None    False   False   []  []  False   /en/products/dps/valves-and-actuators/valves/pvg-proportional-valves/pvg-combinations/
6   https://danfoss-webex-umbraco-prodblue.azurewebsites.net/en/products/dps/valves-and-actuators/valves/pvg-proportional-valves/pvg-ex-proportional-valves/    None    None    False   PVG-EX proportional valves  /media/14390/pvg-ex-32-proportional-valve.jpg   PVG-EX proportional valves  <p>Our PVG-EX program is an explosion proof PVG designed to be used in Ex hazardous areas like mining and oil and gas industries. The PVG-EX shares the same features as the standard PVG. This enables stack combinations of flexible slice sections, making it possible to build valve groups meeting the exact requirements, now for use in hazardous areas.</p> None    None    None    None    False   False   []  []  False   /en/products/dps/valves-and-actuators/valves/pvg-proportional-valves/pvg-ex-proportional-valves/
7   https://danfoss-webex-umbraco-prodblue.azurewebsites.net/en/products/dps/valves-and-actuators/valves/pvg-proportional-valves/pvg-combinations/  None    None    False   PVG combinations    /media/14378/pvg-proportional-valves-combinations.jpg   PVG combinations    <p>Danfoss offers solutions which makes it possible to combine up to four different valve families.</p> None    None    None    None    False   False   []  []  False   /en/products/dps/valves-and-actuators/valves/pvg-proportional-valves/pvg-combinations/
8   https://danfoss-webex-umbraco-prodblue.azurewebsites.net/en/products/dps/valves-and-actuators/valves/pvg-proportional-valves/pvg-ex-proportional-valves/    None    None    False   PVG-EX proportional valves  /media/14390/pvg-ex-32-proportional-valve.jpg   PVG-EX proportional valves  <p>Our PVG-EX program is an explosion proof PVG designed to be used in Ex hazardous areas like mining and oil and gas industries. The PVG-EX shares the same features as the standard PVG. This enables stack combinations of flexible slice sections, making it possible to build valve groups meeting the exact requirements, now for use in hazardous areas.</p> None    None    None    None    False   False   []  []  False   /en/products/dps/valves-and-actuators/valves/pvg-proportional-valves/pvg-ex-proportional-valves/
​

相关问题