如何解决底部滚动到无限加载的页面,并在向上滚动时单击页面上存在的链接
我正在使用selenium
和python
设计程序,点击一系列帖子(n = 235),从最早的帖子(#235)开始直到最新(#1)。到目前为止,我有一个版本,可以部分完成应做的工作。我之所以这样说,部分是因为我要合并到该程序中的某些任务尚不存在(并且我在编写代码时遇到了麻烦)。
所以,到目前为止,我有一个程序可以一直滚动到网页末尾,找到最旧的帖子并单击它。
这是代码
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
import time
#for i in range(1000):
lenOfPage = driver.execute_script("window.scrollTo(0,document.body.scrollHeight);var lenOfPage=document.body.scrollHeight;return lenOfPage;")
match=False
while(match==False):
lastCount = lenOfPage
time.sleep(3)
lenOfPage = driver.execute_script("window.scrollTo(0,document.body.scrollHeight);var lenOfPage=document.body.scrollHeight;return lenOfPage;")
if lastCount==lenOfPage:
match=True
time.sleep(5)
totalLink=235#len(driver.find_elements_by_xpath("//div//div[1]//div[2]//ul[1]//li[3]//a[1]//i[1]"))
for i in range (1,totalLink-1):
linkxPath = "//div["+str(235-i+1)+"]//div[1]//div[2]//ul[1]//li[3]//a[1]//i[1]"
WebDriverWait(driver,20).until(EC.presence_of_element_located((By.XPATH,linkxPath))).click()
driver.execute_script("arguments[0].scrollIntoView();",driver.find_element_by_xpath(linkxPath))
#driver.find_element_by_xpath(linkxPath).click()
driver.find_element_by_xpath("//span[@class='share-title']").click()
#driver.refresh()
问题:
- 我无法合并向上滚动功能。现在,我正在做的是刷新页面并一直滚动到页面末尾,找到最旧的帖子并单击。我正在处理的网页具有此功能,一旦我单击较旧的帖子,该帖子就会变成新帖子(例如,如果我单击帖子#235,则刷新页面后,它会变成帖子1)。
- 我还无法找到一种使点击功能自动化的方法,现在我的代码中有3行用于定位项目和对其进行点击(效率不高,因为我需要更新235个帖子,一次执行3次)。通过查看xpath项,我注意到只有一个数字会更改,但是我不确定是否可以修改。
元素xpath:
//div[235]//div[1]//div[2]//ul[1]//li[3]//a[1]//i[1]
//div[234]//div[1]//div[2]//ul[1]//li[3]//a[1]//i[1]
.
.
.
//div[1]//div[1]//div[2]//ul[1]//li[3]//a[1]//i[1]
错误
Traceback (most recent call last):
File "/home/pi/Documents/Posh_Auto.py",line 39,in <module>
WebDriverWait(driver,20).until(EC.element_to_be_clickable((By.XPATH,linkxPath))).click()
File "/usr/local/lib/python3.7/dist-packages/selenium/webdriver/remote/webelement.py",line 80,in click
self._execute(Command.CLICK_ELEMENT)
File "/usr/local/lib/python3.7/dist-packages/selenium/webdriver/remote/webelement.py",line 633,in _execute
return self._parent.execute(command,params)
File "/usr/local/lib/python3.7/dist-packages/selenium/webdriver/remote/webdriver.py",line 321,in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.7/dist-packages/selenium/webdriver/remote/errorhandler.py",line 242,in check_response
raise exception_class(message,screen,stacktrace)
selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element <i class="icon share-gray"></i> is not clickable at point (860,8). Other element would receive the click: <div aria-expanded="false" aria-haspopup="true" class="dropdown-toggle" data-toggle="dropdown">...</div>
(Session info: chrome=78.0.3904.108)
请让我知道您的想法和评论。
谢谢
解决方法
driver.get("https://poshmark.com/closet/alyssascott688?sort_by=added_desc&just_in_closet=true")
# Waiting for Page to load successfully
WebDriverWait(driver,30).until(EC.presence_of_element_located((By.XPATH,"//img[@title='poshmark-logo']")))
# Since page is loading new elements after each scroll to bottom
# Get scroll height
last_height = driver.execute_script("return document.body.scrollHeight")
while True:
# Scroll down to bottom
driver.execute_script("window.scrollTo(0,document.body.scrollHeight);")
# Wait to load page
time.sleep(2)
# Calculate new scroll height and compare with last scroll height
new_height = driver.execute_script("return document.body.scrollHeight")
if new_height == last_height:
break
last_height = new_height
shareLinks = driver.find_elements_by_xpath("//i[@class='icon share-gray']")
#Total such share link
print(len(shareLinks))
for i in range(1,len(shareLinks)+1):
xpathLink = "(//i[@class='icon share-gray'])["+str(226-i+1)+"]"
#Creating Fresh elements so won't get stale element exception
clickLink = driver.find_element_by_xpath(xpathLink)
#Scroll to element and click
driver.execute_script("arguments[0].scrollIntoView();",clickLink)
driver.execute_script("arguments[0].click();",clickLink)
#After clicking on Share Link,I am copying the link. You can perform any other action if you want
WebDriverWait(driver,30).until(EC.element_to_be_clickable((By.XPATH,"//i[@class='icon copy-white']"))).click()
# To insure which link its clicking,I am printing link number
print("copied : " + str(226-i+1))
# Waiting for links to be clickable before go to next
WebDriverWait(driver,xpathLink)))
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。