如何解决Scrapy请求-嵌套请求中未调用回调函数
我正尝试从亚马逊刮擦一些产品，以获取有关我的竞争对手的一些信息。这是我正在采用的过程：
Make a query in the search bar ->
Visit every product page of the results gotten from the query ->
Gather information from that product ->
Check if the product matches the quantity that we looked for (I.E. We might want to collect only products sold in a pack of n items ... like a kit of n toner cartridges)
-> If it does,yield the item.
-> If not,find a variation in that ad that represents a pack of such n items
-> If such a variation exists,go visit that variation of the product,modify some information of the item (such as price and asin) and yield that item.
我在这里有一个特殊的情况。我不会发布我拥有的所有功能，而是发布一些代表性的功能（为了使它更简短，更通用，以便将来对其他人有用 ）。
这是我的代码的结构：
def start_requests(self):
for i,prod in enumerate(products):
url = 'https://www.amazon.it/s?' + urlencode({'k': prod['query']})
competitors = scrapy.Request(url=url,callback=self.parse_keyword_response,meta={'prod':prod})
yield competitors
def parse_keyword_response(self,response):
# Function that loops on the results of the query made,# and collects all the products that actually match our search
products = response.xpath('//*[@data-asin]')
prod = response.meta['prod']
competitors =[]
for product in products:
asin = product.xpath('@data-asin').extract_first()
product_url = f"https://www.amazon.it/dp/{asin}"
competitor = scrapy.Request(url=product_url,callback=self.parse_competitor_product_page,meta={'asin': asin,'prod':prod})
yield competitor
competitors.append(competitor)
def parse_competitor_product_page(self,response):
# Function that scrapes information from a product page and yields the competitor
# only if it actually matches our search.
' Do some work and scrape required product attributes'
competitor = ProductItem()
competitor['product'] = prod_name
competitor['asin'] = asin
competitor['Title'] = title
competitor['producer'] = producer
competitor['MainImage'] = image
competitor['Rating'] = rating
competitor['NumberOfReviews'] = number_of_reviews
competitor['price'] = price
competitor['AvailableSizes'] = sizes
competitor['AvailableColors'] = colors
competitor['Varieties'] = varieties
competitor['BulletPoints'] = bullet_points
competitor['SellerRank'] = seller_rank
if self.is_right_product(prod,competitor,response):
yield competitor
def is_right_product(self,product,response):
# Function that checks whether a resulting competitor actually matches the product that
# we looked for. It returns a boolean if it does. It also alters some attributes of that
# competitor if a right variation is found on its page.
' I will omit some if else branches as those work well and I will only post the faulty
branch (which happens to be the one that should modify the competitor object because
a right variation is found on its page. '
if product_is_right_quantity(competitor):
return True
else:
variation = find_variation_of_right_quantity(product['quantity'],competitor)
if vatiation is not None:
competitor = self..update_product_to_right_variation(competitor,variation,response)
print("variation check done")
return True
else:
return False
def update_product_to_right_variation(self,variation_name,response):
print("IN UPDATE PRODUCT TO RIGHT VARIATION")
variation_asin = response.xpath(f'//div[@id="variation_color_name"]/ul/li[contains(@title,\'{variation_name}\')]/@data-defaultasin').get()
product_url = f"https://www.amazon.it/dp/{variation_asin}"
print(product_url)
yield scrapy.Request(url=product_url,callback=self.update_competitor_from_product_page,errback=self.errback_http,meta={'prod':product,'asin':variation_asin})
def update_competitor_from_product_page(self,response):
print("INSIIDE UPDATE COMPETITOR FROM PRODUCT PAGE")
prod = response.meta['prod']
asin = response.meta['asin']
price = response.xpath('//*[@id="priceblock_ourprice"]/text()').extract_first()
prod['price'] = price
prod['Title'] = title
prod['asin'] = asin
response.meta['prod'] = prod
print(prod['price'])
return prod
如您所见，我放置了一些打印语句以用于调试。
update_competitor_from_product_page中的打印语句永远不会输出。
所有其他人都这样做。因此，永远不会调用应用作update_product_to_right_variation中发出的请求的回调函数的函数。结果，竞争者对象保持不变。
我是异步编程的新手，也是Scrapy的新手。
首先，我想知道为什么从未调用过我的回调函数。其次，我该怎么做？
解决方法
我无法测试它，但是问题可能是您尝试在函数yield Request中执行的函数parse_competitor_product_page()中is_right_product()-在parse_competitor_product_page()中执行-函数yield中的return / parse_competitor_product_page()无法直接将其发送到Scrapy Engine，但会将其发送到先前的函数is_right_product()，该函数应yield / {{1 }}到先前的功能return-在parse_competitor_product_page()中，您应该parse_competitor_product_page()，然后将其发送给yield引擎，该引擎将执行它。
在您的代码中，您从Scrapy yield Request到parse_competitor_product_page()，但是在is_right_product()中，您发送了is_right_product() / return True，因此它不会发送return False到Request，并且无法将其发送到Scrapy引擎
我认为您需要这样的东西
parse_competitor_product_page()版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。