如何解决使用子流程时，如何使用scrapy-redis处理非法URL？
我想向html请求推送到redis的URL，它运行良好。
如果我使用cmdline运行，则可以轻松处理ValueError并继续进行操作。
但是，当我使用python（在子进程下）运行时，一旦遇到来自非法“ URL”的ValueError（例如javascript：void（0），电子邮件链接），它将立即完成，而其余URL则不运行。
当使用python.subprocess运行scrapy-redis时，你们如何处理非法URL？
# request html (or other info)
# TODO: dupefilter implemented by scratch
# push urls to redis
r = redis.Redis(host='localhost',port=6379,db=0)
r.delete('MySpider:start_urls')
r.delete("MySpider:items")
for url in url_list:
print(start_urls)
print('-----------------')
print(url_list)
r.lpush('MySpider:start_urls',url)
# scrap info from urls
urls_from_scrapy = []
html_strs_from_scrapy = []
worker = subprocess.Popen("scrapy crawl MySpider".split()) # error when LOG arg added
worker.wait(timeout=None)
这是我看到的错误：
2020-08-11 09:26:21 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method RedisMixin.spider_idle of <MySpider 'MySpider' at 0x201fde4ef48>>
Traceback (most recent call last):
File "c:\programdata\anaconda3\lib\site-packages\scrapy\utils\signal.py",line 32,in send_catch_log
response = robustApply(receiver,signal=signal,sender=sender,*arguments,**named)
File "c:\programdata\anaconda3\lib\site-packages\pydispatch\robustapply.py",line 55,in robustApply
return receiver(*arguments,**named)
File "D:\PycharmProjects\genetic_algo_crawl\demo\scrapy_redis\spiders.py",line 128,in spider_idle
self.schedule_next_requests()
File "D:\PycharmProjects\genetic_algo_crawl\demo\scrapy_redis\spiders.py",line 122,in schedule_next_requests
for req in self.next_requests():
File "D:\PycharmProjects\genetic_algo_crawl\demo\scrapy_redis\spiders.py",line 91,in next_requests
req = self.make_request_from_data(data)
File "D:\PycharmProjects\genetic_algo_crawl\demo\scrapy_redis\spiders.py",line 117,in make_request_from_data
return self.make_requests_from_url(url)
File "c:\programdata\anaconda3\lib\site-packages\scrapy\spiders\__init__.py",line 87,in make_requests_from_url
return Request(url,dont_filter=True)
File "c:\programdata\anaconda3\lib\site-packages\scrapy\http\request\__init__.py",line 25,in __init__
self._set_url(url)
File "c:\programdata\anaconda3\lib\site-packages\scrapy\http\request\__init__.py",line 69,in _set_url
raise ValueError('Missing scheme in request url: %s' % self._url)
ValueError: Missing scheme in request url: mailto:sjxx@xidian.edu.cn
2020-08-11 09:26:21 [scrapy.core.engine] INFO: Closing spider (finished)
版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。