我正试图从亚马逊刮一些产品，以便获得一些竞争对手的信息，这是我正在采用的过程：

Make a query in the search bar ->
Visit every product page of the results gotten from the query -> 
Gather information from that product ->
Check if the product matches the quantity that we looked for (I.E. We might want to collect only products sold in a pack of n items ... like a kit of n toner cartridges)
    -> If it does, yield the item.
    -> If not, find a variation in that ad that represents a pack of such n items
         -> If such a variation exists, go visit that variation of the product, modify some information of the item (such as price and asin) and yield that item.

这里我有一个特殊的例子。我不会发布我拥有的全部函数，但我宁愿发布一些代表性的函数（为了保持它更短，更通用，以便将来可能对其他人有用）。
下面是我的代码结构：

def start_requests(self):
        for i, prod in enumerate(products):
            url = 'https://www.amazon.it/s?' + urlencode({'k': prod['query']})
            competitors = scrapy.Request(url=url, callback=self.parse_keyword_response, meta={'prod':prod})
            yield competitors

def parse_keyword_response(self, response):
        # Function that loops on the results of the query made, 
        # and collects all the products that actually match our search
        products = response.xpath('//*[@data-asin]')
        prod = response.meta['prod']

        competitors =[]

        for product in products:
            asin = product.xpath('@data-asin').extract_first()
            product_url = f"https://www.amazon.it/dp/{asin}"
            competitor = scrapy.Request(url=product_url, callback=self.parse_competitor_product_page, meta={'asin': asin, 'prod':prod})
            yield competitor
            competitors.append(competitor)

def parse_competitor_product_page(self, response):
        # Function that scrapes information from a product page and yields the competitor
        # only if it actually matches our search.

        ' Do some work and scrape required product attributes'

        competitor = ProductItem()
        competitor['product'] = prod_name
        competitor['asin'] = asin
        competitor['Title'] = title
        competitor['producer'] = producer
        competitor['MainImage'] = image
        competitor['Rating'] = rating
        competitor['NumberOfReviews'] = number_of_reviews
        competitor['price'] = price
        competitor['AvailableSizes'] = sizes
        competitor['AvailableColors'] = colors
        competitor['Varieties'] = varieties
        competitor['BulletPoints'] = bullet_points
        competitor['SellerRank'] = seller_rank

        if self.is_right_product(prod, competitor, response):
            yield competitor

def is_right_product(self, product, competitor, response):
       # Function that checks whether a resulting competitor actually matches the product that 
       # we looked for. It returns a boolean if it does. It also alters some attributes of that
       # competitor if a right variation is found on its page.

      ' I will omit some if else branches as those work well and I will only post the faulty 
           branch (which happens to be the one that should modify the competitor object because 
           a right variation is found on its page. '

      if product_is_right_quantity(competitor):
           return True
      else:
           variation = find_variation_of_right_quantity(product['quantity'], competitor)
           if vatiation is not None:
                competitor = self..update_product_to_right_variation(competitor, variation, response)
                print("variation check done")
                return True
           else:
                return False

def update_product_to_right_variation(self, product, variation_name, response):
        print("IN UPDATE PRODUCT TO RIGHT VARIATION")
        variation_asin = response.xpath(f'//div[@id="variation_color_name"]/ul/li[contains(@title, \'{variation_name}\')]/@data-defaultasin').get()
        product_url = f"https://www.amazon.it/dp/{variation_asin}"
        print(product_url)
        yield scrapy.Request(url=product_url, callback=self.update_competitor_from_product_page, errback=self.errback_http, meta={'prod':product, 'asin':variation_asin})

def update_competitor_from_product_page(self, response):
        print("INSIIDE UPDATE COMPETITOR FROM PRODUCT PAGE")
        prod = response.meta['prod']
        asin = response.meta['asin']

        price = response.xpath('//*[@id="priceblock_ourprice"]/text()').extract_first()

        prod['price'] = price
        prod['Title'] = title
        prod['asin'] = asin

        response.meta['prod'] = prod
        print(prod['price'])
        return prod

正如您所看到的，我放置了一些print语句用于调试目的。

update_competitor_from_product_page中的打印语句从未得到输出。

所有其他的都是这样。所以应该被用作update_product_to_right_variation中所做请求的回调函数的那个函数永远不会被调用。因此，竞争者对象保持不变。
我是异步编程的新手，也是Scrapy的新手。
首先，我想知道为什么我的回调函数从来没有被调用过。其次，我该怎么做我想做的事情？