scrapy 如何在抓取数据时循环遍历同级标记

eaf3rand  于 2022-11-23  发布在  其他
关注(0)|答案(2)|浏览(136)

我正在尝试使用python scrapy框架从this page中抓取编辑器数据。
我面临的问题是每个标签都是兄弟标签,编辑器角色在h3标签内,名称在div标签内。所有这些都在id为“editors-section”的div标签内。
response.css("#editors-section>div.row.align-items-center")
并收集编辑姓名和组织,
而是如何收集它们各自的角色,如何循环遍历所有的标签。

f8rj6qna

f8rj6qna1#

您可以使用相对xpath表达式和following-sibling指令沿着使用selectors root.tag属性来测试相邻的角色头,这样就可以准确地确定每个人的角色。
例如:

for header in response.xpath("//h2"):
    role = header.xpath("./text()").get()
    for sibling in header.xpath("./following-sibling::*"):
        if sibling.root.tag == "h2":
            break
        name = sibling.xpath(".//h3/*/text()").get()
        location = sibling.xpath(".//p[@class='mb-2']/text()").get()
        if name and location:
            yield{
                "role": role.strip(),
                "name": name.strip(),
                "location": location.strip()
            }

输出

[
  {
    "role": "Editors-in-Chief",
    "name": "Hua Wang",
    "location": "University of Electronic Science and Technology of China, China"
  },
  {
    "role": "Editors-in-Chief",
    "name": "Gabriele Morra",
    "location": "University of Louisiana at Lafayette, USA"
  },
  {
    "role": "Board Members",
    "name": "Luca Caricchi",
    "location": "University of Geneva, Switzerland"
  },
  {
    "role": "Board Members",
    "name": "Michael Fehler",
    "location": "Massachusetts Institute of Technology, USA"
  },
  {
    "role": "Board Members",
    "name": "Peter Gerstoft",
    "location": "Scripps Institution of Oceanography, USA"
  },
  {
    "role": "Board Members",
    "name": "Forrest M. Hoffman",
    "location": "Oak Ridge National Laboratory, United States of America"
  },
  {
    "role": "Board Members",
    "name": "Xiangyun Hu",
    "location": "China University of Geosciences, China"
  },
  {
    "role": "Board Members",
    "name": "Guangmin Hu",
    "location": "University of Electronic Science and Technology of China, China"
  },
  {
    "role": "Board Members",
    "name": "Qingkai Kong",
    "location": "UC Berkeley, USA"
  },
  {
    "role": "Board Members",
    "name": "Yuemin Li",
    "location": "University of Electronic Science and Technology of China, China"
  },
  {
    "role": "Board Members",
    "name": "Hongjun Lin",
    "location": "Zhejiang Normal University, China"
  },
  {
    "role": "Board Members",
    "name": "Aldo Lipani",
    "location": "University College London, United Kingdom"
  },
  {
    "role": "Board Members",
    "name": "Zhigang Peng",
    "location": "Georgia Institute of Technology, USA"
  },
  {
    "role": "Board Members",
    "name": "Piero Poli",
    "location": "Grenoble Alpes University, France"
  },
  {
    "role": "Board Members",
    "name": "Kunfeng Qiu",
    "location": "China University of Geoscience, China"
  },
  {
    "role": "Board Members",
    "name": "Calogero Schillaci",
    "location": "JRC European Commission, Italy"
  },
  {
    "role": "Board Members",
    "name": "Hosein Shahnas",
    "location": "University of Toronto, Canada"
  },
  {
    "role": "Board Members",
    "name": "Byung-Dal So",
    "location": "Kangwon National University, South Korea"
  },
  {
    "role": "Board Members",
    "name": "Rui Wang",
    "location": "China University of Geoscience, China"
  },
  {
    "role": "Board Members",
    "name": "Yong Wang",
    "location": "East Carolina University, USA"
  },
  {
    "role": "Board Members",
    "name": "Zhiguo Wang",
    "location": "Xi'an Jiaotong University, China"
  },
  {
    "role": "Board Members",
    "name": "Jun Xia",
    "location": "Wuhan University, China"
  },
  {
    "role": "Board Members",
    "name": "Lizhi Xiao",
    "location": "China University of Petroleum(Beijing), China"
  },
  {
    "role": "Board Members",
    "name": "Chicheng Xu",
    "location": "Aramco Services Company, USA"
  },
  {
    "role": "Board Members",
    "name": "Zhibing Yang",
    "location": "Wuhan University, China"
  },
  {
    "role": "Board Members",
    "name": "Nana Yoshimitsu",
    "location": "Kyoto University, Japan"
  },
  {
    "role": "Board Members",
    "name": "Hongyan Zhang",
    "location": "Wuhan University, China"
  }
]
pieyvz9o

pieyvz9o2#

相同的结果,但使用了另一种方法(和一个for循环)。我找到每个h3元素(name),并使用preceding XPath表达式得到role(上面的第一个h2元素):

def parse(self, response):
    for h3_node in response.xpath('//div[@class="container"]//h3'):
        role = h3_node.xpath('normalize-space(./preceding::h2[1])').get()
        name = h3_node.xpath('normalize-space(.)').get()
        location = h3_node.xpath("normalize-space(./following-sibling::p[1])").get()
        if name and location:
            yield{
                "role": role,
                "name": name,
                "location": location,
            }

相关问题