我不知道我代码发生了什么。我编写了spider并加载了https://docs.scrapy.org/en/latest/topics/media-pipeline.html中描述的项目,但Scrapy不下载任何文件:
2022-07-19 01:35:09 [scrapy.core.engine] INFO: Spider opened
2022-07-19 01:35:09 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-07-19 01:35:09 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2022-07-19 01:35:09 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.tp-link.com/br/support/download/> from <GET https://www.tp-link.com/br/support/download>
2022-07-19 01:35:09 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.tp-link.com/br/support/download/> (referer: None)
2022-07-19 01:35:10 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.tp-link.com/br/support/download/archer-c7/> (referer: https://www.tp-link.com/br/support/download/)
2022-07-19 01:35:10 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.tp-link.com/br/support/download/archer-c7/v4/> (referer: https://www.tp-link.com/br/support/download/archer-c7/)
2022-07-19 01:35:10 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.tp-link.com/br/support/download/archer-c7/v4/>
{'file_urls': ['https://static.tp-link.com/2019/201912/20191206/Archer%20C7(US)_V4_190411.zip',
'https://static.tp-link.com/2018/201804/20180428/Archer%20C7(US)_V4_180425.zip',
'https://static.tp-link.com/2017/201712/20171221/Archer%20C7(US)_V4_171101.zip'],
'model': 'Archer C7 ',
'vendor': 'TP-Link',
'version': 'V4'}
2022-07-19 01:35:10 [scrapy.core.engine] INFO: Closing spider (finished)
2022-07-19 01:35:10 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 1082,
'downloader/request_count': 4,
'downloader/request_method_count/GET': 4,
'downloader/response_bytes': 77004,
'downloader/response_count': 4,
'downloader/response_status_count/200': 3,
'downloader/response_status_count/301': 1,
'elapsed_time_seconds': 1.45197,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2022, 7, 19, 1, 35, 10, 957079),
'httpcompression/response_bytes': 667235,
'httpcompression/response_count': 3,
'item_scraped_count': 1,
'log_count/DEBUG': 6,
'log_count/INFO': 10,
'memusage/max': 93999104,
'memusage/startup': 93999104,
'request_depth_max': 2,
'response_received_count': 3,
'scheduler/dequeued': 4,
'scheduler/dequeued/memory': 4,
'scheduler/enqueued': 4,
'scheduler/enqueued/memory': 4,
'start_time': datetime.datetime(2022, 7, 19, 1, 35, 9, 505109)}
2022-07-19 01:35:10 [scrapy.core.engine] INFO: Spider closed (finished)
我的项目分类我定义了一些自定义字段,但file_urls和files被创建为零碎的文档:
class FirmwareItem(scrapy.Item):
# define the fields for your item here like:
vendor = scrapy.Field()
model = scrapy.Field()
version = scrapy.Field()
file_urls = scrapy.Field()
files = scrapy.Field()
ParseMethod在这里,我在页面上获得zip文件链接,并将它们作为列表加载到“file_urls”:
def parseVersion(self,response):
firmware = FirmwareItem()
firmware['vendor'] = "TP-Link"
firmware['model'] = response.xpath('//em[@id="model-version-name"]/text()').get()
firmware['version'] = response.xpath('//em[@id="model-version-name"]/span/text()').get()
urls = response.xpath('//div[@data-id="Firmware"]//a[@class="download-resource-btn ga-click"]/@href').getall()
firmware['file_urls'] = [url.replace(' ',"%20") for url in urls]
yield firmware
设置启用项目管道并设置默认目录:
ITEM_PIPELINES = {
'router.pipelines.RouterPipeline': 300,
}
FILES_STORE = "downloaded"
管道保留为默认值:
from itemadapter import ItemAdapter
class RouterPipeline:
def process_item(self, item, spider):
return item
1条答案
按热度按时间xv8emn3q1#
就快完成了...请确保您的www.example.com中已设置了所有这些设置settings.py: