我刚刚从THIS问题（也在THIS repo中提到）中编写了Scrapy spider。看起来它在一年前工作，但现在book24.ru阻止了spider并返回520错误。
Shell响应520错误：
2023-01-20 22:10:23 [scrapy.utils.log] INFO: Scrapy 2.7.1 started (bot: bestsellers)
2023-01-20 22:10:23 [scrapy.utils.log] INFO: Versions: lxml 4.9.2.0, libxml2 2.9.14, cssselect 1.2.0, parsel 1.7.0, w3lib 2.1.1, Twisted 22.10.0, Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0], pyOpenSSL 23.0.0 (OpenSSL 3.0.7 1 Nov 2022), cryptography 39.0.0, Platform Linux-5.15.0-58-generic-x86_64-with-glibc2.35
2023-01-20 22:10:23 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'bestsellers',
'NEWSPIDER_MODULE': 'bestsellers.spiders',
'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7',
'SPIDER_MODULES': ['bestsellers.spiders'],
'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'}
2023-01-20 22:10:23 [asyncio] DEBUG: Using selector: EpollSelector
2023-01-20 22:10:23 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor
2023-01-20 22:10:23 [scrapy.utils.log] DEBUG: Using asyncio event loop: asyncio.unix_events._UnixSelectorEventLoop
2023-01-20 22:10:23 [scrapy.extensions.telnet] INFO: Telnet Password: cb384b87b1e215df
2023-01-20 22:10:23 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.memusage.MemoryUsage',
'scrapy.extensions.feedexport.FeedExporter',
'scrapy.extensions.logstats.LogStats']
2023-01-20 22:10:23 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2023-01-20 22:10:23 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2023-01-20 22:10:23 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2023-01-20 22:10:23 [scrapy.core.engine] INFO: Spider opened
2023-01-20 22:10:23 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2023-01-20 22:10:23 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2023-01-20 22:10:23 [filelock] DEBUG: Attempting to acquire lock 140382385887728 on /home/user/.cache/python-tldextract/3.10.6.final__first_scrapy-rekTiTXv__400173__tldextract-3.4.0/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2023-01-20 22:10:23 [filelock] DEBUG: Lock 140382385887728 acquired on /home/user/.cache/python-tldextract/3.10.6.final__first_scrapy-rekTiTXv__400173__tldextract-3.4.0/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2023-01-20 22:10:23 [filelock] DEBUG: Attempting to release lock 140382385887728 on /home/user/.cache/python-tldextract/3.10.6.final__first_scrapy-rekTiTXv__400173__tldextract-3.4.0/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2023-01-20 22:10:23 [filelock] DEBUG: Lock 140382385887728 released on /home/user/.cache/python-tldextract/3.10.6.final__first_scrapy-rekTiTXv__400173__tldextract-3.4.0/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2023-01-20 22:10:23 [scrapy.core.engine] DEBUG: Crawled (520) <GET https://book24.ru/product/transhumanism-inc-6015821/> (referer: None)
2023-01-20 22:10:23 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <520 https://book24.ru/product/transhumanism-inc-6015821/>: HTTP status code is not handled or not allowed
2023-01-20 22:10:23 [scrapy.core.engine] INFO: Closing spider (finished)
2023-01-20 22:10:23 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 243,
'downloader/request_count': 1,
'downloader/request_method_count/GET': 1,
'downloader/response_bytes': 7640,
'downloader/response_count': 1,
'downloader/response_status_count/520': 1,
'elapsed_time_seconds': 0.364996,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2023, 1, 20, 19, 10, 23, 800623),
'httperror/response_ignored_count': 1,
'httperror/response_ignored_status_count/520': 1,
'log_count/DEBUG': 8,
'log_count/INFO': 11,
'memusage/max': 65613824,
'memusage/startup': 65613824,
'response_received_count': 1,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2023, 1, 20, 19, 10, 23, 435627)}
2023-01-20 22:10:23 [scrapy.core.engine] INFO: Spider closed (finished)第一次尝试-尝试添加headers cause HERE，因为有些网站会阻止robot请求，您应该添加标题，使自己看起来像是使用Web浏览器的普通用户。但它没有帮助，并再次返回520错误。
import scrapy
class BookSpider(scrapy.Spider):
name = 'book24-2'
start_urls = ['https://book24.ru/knigi-bestsellery/']
def parse(self, response):
headers = {
'user-agent' : 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'
}
for link in response.css('div.product-card__image-holder a::attr(href)', headers=headers).get():
yield response.follow(link, callback=self.parse_book, headers=headers)
for i in range(1, 25):
next_page = f'https://book24.ru/knigi-bestsellery/page-{i}/'
yield response.follow(next_page, callback=self.parse, headers=headers)
def parse_book(self, response):
yield{
'name': response.css('h1.product-detail-page__title::text').get(),
'buy': response.css('p.product-detail-page__purchased-text::text').get().split()[1],
'type': response.css('div.product-characteristic__value a::attr(title)')[2].get()
}第二次尝试-尝试在www.example.com中将ROBOTSTXT_OBEY = True更改为x1m2 n1settings.py- 520错误仍然存在。
第三次尝试-尝试使用fake_useragent lib -无效。代码：
import scrapy
from fake_useragent import UserAgent
# print(scrapy.Request('https://book24.ru/knigi-bestsellery/'))
class BookSpider(scrapy.Spider):
name = 'book24-2'
start_urls = ['https://book24.ru/knigi-bestsellery/']
# self значит, что остаемся в пределах этого класса
# response это ответ, который видели в scrapy shell, к примеру, response.css('...')
def parse(self, response):
# headers = {
# 'user-agent' : 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'
# }
headers = {
'user-agent' : UserAgent().random
}
for link in response.css('div.product-card__image-holder a::attr(href)', headers=headers).get():
# yield - ключевое слово в scrapy, которое позволяет выполнять разные действия
# yield scrapy.Request('https://book24.ru/knigi-bestsellery/', headers=headers)
# print('++++++++++++'+response)
yield response.follow(link, callback=self.parse_book, headers=headers)
for i in range(1, 25):
next_page = f'https://book24.ru/knigi-bestsellery/page-{i}/'
yield response.follow(next_page, callback=self.parse, headers=headers)
def parse_book(self, response):
headers = {
'user-agent' : UserAgent().random
}
yield{
'name': response.css('h1.product-detail-page__title::text', headers=headers).get(),
'buy': response.css('p.product-detail-page__purchased-text::text', headers=headers).get().split()[1],
'type': response.css('div.product-characteristic__value a::attr(title)', headers=headers)[2].get()
}
1条答案
按热度按时间iih3973s1#
您的spider代码有多个问题需要解决。
1.你的css选择器有一个关键字参数头，这是防止你从爬行的书页。
1.据我所知，只有14页的书，你有你的for循环范围设置为25。
1.你的
'type'字段在你的项目是寻址一个特定的索引，然而，不是所有的图书特征是统一的格式，所以在超过一半的图书页面，你试图刮取的元素不存在.所有这些都是非常简单的修复方法。对于
type字段，您可以简单地迭代特征标签，直到到达您正在寻找的标签，然后提取信息并中断迭代。例如：
运行上面的代码，这是我输出的一部分。
如您所见，蜘蛛生成了390个项目。