该网站受cloudflare保护。

https://steamdb.info/graph/ is using Cloudflare CDN/Proxy!

https://steamdb.info/graph/ is using Cloudflare SSL!

它与cloudscraper一起工作，这相当于requests模块可以处理云耀斑保护。

import cloudscraper
scraper = cloudscraper.create_scraper(delay=10,   browser={'custom': 'ScraperBot/1.0',})
url = 'https://steamdb.info/graph/'
req = scraper.get(url)
print(req)

输出：

<Response [200]>

这是因为该站点不存在-https：steamdb.info/graphs/转到404
谢谢

我解决了这个问题。如果一个网站使用的是cloudfare，你可以使用未检测到的chrome驱动程序，并将其作为scrapy middleware使用。
将此添加到Middleware.py:

class SeleniumMiddleWare(object):

    def __init__(self):
        path = "G:/Downloads/chromedriver.exe"
        options = uc.ChromeOptions()
        options.headless=True
        chrome_prefs = {}
        options.experimental_options["prefs"] = chrome_prefs
        chrome_prefs["profile.default_content_settings"] = {"images": 2}
        chrome_prefs["profile.managed_default_content_settings"] = {"images": 2}
        #self.driver = uc.Chrome(options=options)
        self.driver=  uc.Chrome(options= options, use_subprocess=True, driver_executable_path = path)

    def process_request(self, request, spider):
        try:
            self.driver.get(request.url)
        except:
            pass
        content = self.driver.page_source
        self.driver.quit()

        return HtmlResponse(request.url, encoding='utf-8', body=content, request=request)

    def process_response(self, request, response, spider):
        return response

Settings.py:

DOWNLOADER_MIDDLEWARES = {
    'my_scraper.middlewares.SeleniumMiddleWare': 491 #change my_scraper to your scraper's name
}

my_scraper.py：我的电脑

class SeleniumSpider(scrapy.Spider):
    name = 'steamdb'
    allowed_domains = ['steamdb.info']
    start_urls = ['https://steamdb.info/graph/']

    def parse(self, response):
        yield {"title": response.css("h1::text").get()}