如何解决为什么Beautifulsoup中的select_one函数返回None值
我正在使用python 3和Beautifulsoup
我的以下代码返回无值列表
import bs4 as bs
import urllib.request
import pandas as pd
from requests_html import HTMLSession
review_dict = {'review':[],'author':[]}
page = 1
while page != 10:
session = HTMLSession()
url = 'https://www.goodreads.com/book/show/2932708?from_search=true&from_srp=true&qid=OOQwYQkG9A&rank=1' + str(++page)
grURL = session.get(url)
soup = bs.BeautifulSoup(grURL.content,'html.parser')
prod_containers = soup.find('div',id = 'lazy_loadable_view')
firstelement = prod_containers.find_all('div',attrs={'class': 'left bodycol'})
for rows in firstelement:
review = rows.select_one('p > div.reviewText stacked > span.readable > span')
author = rows.select_one('p > div.reviewHeader uitext stacked > span > a[title]')
review_dict['review'].append(review)
review_dict['author'].append(author)
if page == 10:
break
page += 1
sword_reviews = pd.DataFrame(review_dict)
sword_reviews
当我使用.text
功能时,Jupiter Notebook给我这个错误:
AttributeError: 'NoneType' object has no attribute 'text'
如何调整我的代码以正确刮取评论和评论者的姓名?
解决方法
要获取所需的数据,您需要更改select_one
中的搜索字符串。将多个类与“。”一起加入
尝试以下代码:
import bs4 as bs
import urllib.request
import pandas as pd
from requests_html import HTMLSession
review_dict = {'review':[],'author':[]}
page = 1
while page != 10:
session = HTMLSession()
url = 'https://www.goodreads.com/book/show/2932708?from_search=true&from_srp=true&qid=OOQwYQkG9A&rank=1' + str(++page)
grURL = session.get(url)
soup = bs.BeautifulSoup(grURL.content,'html.parser')
prod_containers = soup.find('div',id = 'lazy_loadable_view')
firstelement = prod_containers.find_all('div',attrs={'class': 'left bodycol'})
for rows in firstelement:
review = rows.select_one('div.reviewText.stacked > span.readable > span')
author = rows.select_one('div.reviewHeader.uitext.stacked > span > a[title]')
review_dict['review'].append(review)
review_dict['author'].append(author)
if page == 10:
break
page += 1
sword_reviews = pd.DataFrame(review_dict)
print(sword_reviews)
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。