为什么刮y蜘蛛在刮scrap亚马逊时不返回任何输出？

如何解决为什么刮y蜘蛛在刮scrap亚马逊时不返回任何输出？

我在Amazon畅销书页面上测试了那抓不动的蜘蛛（请参阅下面的URL），但是它返回的价格怪异，或者就像您在最后的输出中所见完全没有输出（我仅共享了1页）。 CSS选择器可能有问题，但是我不确定。我希望蜘蛛将输出保存在JSON文件中，以便可以将其快速转换为pandas数据框以进行某些分析。这是我在终端中编写的用于运行Spider的代码：scrapy crawl amazon_booksUK -o somefilename.json

我知道这是一门值得研究的东西，但是如果您有时间的话，那真的会对我有所帮助！：）

url = https://www.amazon.co.uk/s?i=stripbooks&bbn=266239&rh=n%3A266239%2Cp_72%3A184315031%2Cp_36%3A389028011&dc&page=1&fst=as%3Aoff&qid=1598942460&rnid=389022011&ref=sr_pg_1

1。蜘蛛码：

import scrapy
from ..items import AmazonscrapeItem

class AmazonSpiderSpider(scrapy.Spider):
    page_number = 2
    name = 'amazon_booksUK'
    start_urls = [
        'https://www.amazon.co.uk/s?i=stripbooks&bbn=266239&rh=n%3A266239%2Cp_72%3A184315031%2Cp_36%3A389028011&dc&page=1&fst=as%3Aoff&qid=1598942460&rnid=389022011&ref=sr_pg_1'
    ]

    def parse(self,response):
        items = AmazonscrapeItem()

        # if multiple classes --> .css("::text").extract()
        product_name = response.css('.a-color-base.a-text-normal::text').extract()
        product_author = response.css('.a-color-secondary .a-size-base.a-link-normal').css('::text').extract()
        product_nbr_reviews = response.css('.a-size-small .a-link-normal .a-size-base').css('::text').extract()
        product_type = response.css('.a-spacing-top-small .a-link-normal.a-text-bold').css('::text').extract()
        product_price = response.css('.a-spacing-top-small .a-price-whole').css('::text').extract()
        product_more_choice = response.css('.a-spacing-top-mini .a-color-secondary .a-link-normal').css('::text').extract()
        # this only selects the element that has the image --> need stuff inside src (source attr)
        product_imagelink = response.css('.s-image::attr(src)').extract() # want attr of src

        items['product_name'] = product_name
        items['product_author'] = product_author
        items['product_nbr_reviews'] = product_nbr_reviews
        items['product_type'] = product_type
        items['product_price'] = product_price
        items['product_more_choice'] = product_more_choice
        items['product_imagelink'] = product_imagelink

        yield items

        next_page = 'https://www.amazon.co.uk/s?i=stripbooks&bbn=266239&rh=n%3A266239%2Cp_72%3A184315031%2Cp_36%3A389028011&dc&page='+ str(AmazonSpiderSpider.page_number)+'&fst=as%3Aoff&qid=1598942460&rnid=389022011&ref=sr_pg_'+ str(AmazonSpiderSpider.page_number)
        if AmazonSpiderSpider.page_number <3:
            AmazonSpiderSpider.page_number += 1
            yield response.follow(next_page,callback=self.parse)

2。 items.py：

import scrapy

    class AmazonscrapeItem(scrapy.Item):
        # define the fields for your item here like:
        product_name = scrapy.Field()
        product_author = scrapy.Field()
        product_nbr_reviews = scrapy.Field()
        product_type = scrapy.Field()
        product_price = scrapy.Field()
        product_more_choice = scrapy.Field()
        product_imagelink = scrapy.Field()

3。 settings.py： 我使用了Google bot用户代理程序，以避免在测试刮板时被禁止。

USER_AGENT = 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'

4。 pipelines.py：

class AmazonscrapePipeline:
    def process_item(self,item,spider):
        return item

输出

The default interactive shell is now zsh.
To update your account to use zsh,please run `chsh -s /bin/zsh`.
For more details,please visit https://support.apple.com/kb/HT208050.
(venv) (base) Eriks-MBP:scrapyTutorial erikhren$ cd amazonscrape
(venv) (base) Eriks-MBP:amazonscrape erikhren$ scrapy crawl amazon_booksUK -o mm.json
2020-09-01 15:01:43 [scrapy.utils.log] INFO: Scrapy 2.3.0 started (bot: amazonscrape)
2020-09-01 15:01:43 [scrapy.utils.log] INFO: Versions: lxml 4.5.2.0,libxml2 2.9.10,cssselect 1.1.0,parsel 1.6.0,w3lib 1.22.0,Twisted 20.3.0,Python 3.7.3 (default,Mar  6 2020,22:34:30) - [Clang 11.0.3 (clang-1103.0.32.29)],pyOpenSSL 19.1.0 (OpenSSL 1.1.1g  21 Apr 2020),cryptography 3.1,Platform Darwin-19.6.0-x86_64-i386-64bit
2020-09-01 15:01:43 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2020-09-01 15:01:43 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'amazonscrape','NEWSPIDER_MODULE': 'amazonscrape.spiders','ROBOTSTXT_OBEY': True,'SPIDER_MODULES': ['amazonscrape.spiders'],'USER_AGENT': 'Mozilla/5.0 (compatible; Googlebot/2.1; '
               '+http://www.google.com/bot.html)'}
2020-09-01 15:01:43 [scrapy.extensions.telnet] INFO: Telnet Password: e57aa4639df179b5
2020-09-01 15:01:43 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats','scrapy.extensions.telnet.TelnetConsole','scrapy.extensions.memusage.MemoryUsage','scrapy.extensions.feedexport.FeedExporter','scrapy.extensions.logstats.LogStats']
2020-09-01 15:01:44 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware','scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware','scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware','scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware','scrapy.downloadermiddlewares.useragent.UserAgentMiddleware','scrapy.downloadermiddlewares.retry.RetryMiddleware','scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware','scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware','scrapy.downloadermiddlewares.redirect.RedirectMiddleware','scrapy.downloadermiddlewares.cookies.CookiesMiddleware','scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware','scrapy.downloadermiddlewares.stats.DownloaderStats']
2020-09-01 15:01:44 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware','scrapy.spidermiddlewares.offsite.OffsiteMiddleware','scrapy.spidermiddlewares.referer.RefererMiddleware','scrapy.spidermiddlewares.urllength.UrlLengthMiddleware','scrapy.spidermiddlewares.depth.DepthMiddleware']
2020-09-01 15:01:44 [scrapy.middleware] INFO: Enabled item pipelines:
['amazonscrape.pipelines.AmazonscrapePipeline']
2020-09-01 15:01:44 [scrapy.core.engine] INFO: Spider opened
2020-09-01 15:01:44 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min),scraped 0 items (at 0 items/min)
2020-09-01 15:01:44 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-09-01 15:01:44 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.co.uk/robots.txt> (referer: None)
2020-09-01 15:01:44 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.amazon.co.uk/b?ie=UTF8&node=266239> from <GET https://www.amazon.co.uk/s?i=stripbooks&bbn=266239&rh=n%3A266239%2Cp_72%3A184315031%2Cp_36%3A389028011&dc&page=1&fst=as%3Aoff&qid=1598942460&rnid=389022011&ref=sr_pg_1>
2020-09-01 15:01:44 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.amazon.co.uk/books-used-books-textbooks/b?ie=UTF8&node=266239> from <GET https://www.amazon.co.uk/b?ie=UTF8&node=266239>
2020-09-01 15:01:45 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.co.uk/books-used-books-textbooks/b?ie=UTF8&node=266239> (referer: None)
2020-09-01 15:01:45 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.co.uk/books-used-books-textbooks/b?ie=UTF8&node=266239>
{'product_author': [],'product_imagelink': [],'product_more_choice': [],'product_name': [],'product_nbr_reviews': [],'product_price': ['11','.','12','9','6','8','30','13','11','10','7','24','39','20','34','48','52','47','14','17','71','44','89','40','31','66','68','289','26','100','60','107','222','21','61','22','18','.'],'product_type': []}

解决方法

您只需要像这样使用用户代理：

USER_AGENT = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/74.0.3729.131 Safari/537.36'

类似真实浏览器的东西

为什么刮y蜘蛛在刮scrap亚马逊时不返回任何输出？

如何解决为什么刮y蜘蛛在刮scrap亚马逊时不返回任何输出？

输出

解决方法

相关推荐