如何使用新的contex为每一个要求与scrapy-剧作家?

pwuypxnk  于 2022-11-09  发布在  其他
关注(0)|答案(1)|浏览(153)

下面是我的做法,但我不确定它是否为每个新请求创建和使用新上下文:

class TestSpider(scrapy.Spider):
    name = 'test'
    start_urls = [...]
    cnt = 0

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url=url,
                                 meta={'playwright': True,
                                       'playwright_context': f'{self.cnt}'})

    def parse(self, response):
        self.cnt += 1
        for res in response.xpath('//div[@id="contenu"]'):
            url = res.xpath('.//h2/a/@href').get()
            yield scrapy.Request(url=url,
                                 callback=self.get_content,
                                 meta={'playwright': True,
                                       'playwright_context': f'{self.cnt}'})

这段代码是按照我的要求运行还是出错了?

ej83mcc0

ej83mcc01#

在发送请求之前/之后,self.cnt += 1应位于for循环中,以便在每次发送请求后创建一个编号递增的新上下文

Class TestSpider(scrapy.Spider):
    name = 'test'
    start_urls = [...]
    cnt = 0

    def start_requests(self):
        for url in self.start_urls: 
            self.cnt += 1   # <------ increment the count here
            yield scrapy.Request(url=url,
                                 meta={'playwright': True,
                                       'playwright_context': f'{self.cnt}'})

    def parse(self, response):
        for res in response.xpath('//div[@id="contenu"]'):
            url = res.xpath('.//h2/a/@href').get()
            self.cnt += 1    # <------ increment the count here
            yield scrapy.Request(url=url,
                                 callback=self.get_content,
                                 meta={'playwright': True,
                                       'playwright_context': f'{self.cnt}'})

相关问题