如何解决Python:使用Selenium进行仪表板Web抓取-无法滚动内部仪表板栏
我想从以下在线发布的仪表板的第三页表中提取数据:
我设法编写了一个Python脚本,该脚本使用“ driver.find_element_by_xpath”遍历可见的表元素,但是我无法使用selenium方法自动滚动表。我只能提取表格的20个可见元素而无需滚动。
我的代码在下面:
import pandas as pd
from selenium import webdriver
chromeOptions = webdriver.ChromeOptions()
chromeOptions.add_experimental_option('useAutomationExtension',False)
# path to the chromeDriver
chromedriver = 'C:/Users/chromedriver.exe'
# I dont have full admin rights on my machine and that was the only way to run driver
driver = webdriver.Chrome(chromedriver,chrome_options=chromeOptions,desired_capabilities=chromeOptions.to_capabilities())
#link to the dashboard
url2 = 'https://app.powerbi.com/view?r=eyJrIjoiM2UzZjZjNGItYzc0OS00NjVkLTgyYzYtYjk4NmRhYjQ3OTQ0IiwidCI6IjA3ZTZlZTM1LTY4MTQtNDc5MC04NjY5LTgwNzY3Njk0YzI4ZCIsImMiOjl9'
driver.get(url2)
#at this point I was going manually to the third tab of the opened browser
### Main loop to extract elements
# declare empty tables
df_all= pd.DataFrame()
# 12 as we have 12 columns in dashboard
df = pd.DataFrame(columns=['1','2','3','4','5','6','7','8','9','10','11','12'])
#iterate through columns
for j in range(1,13):
# put extracted data here
ticker_list = []
# set variable for column numeration
col = j
# as Xpath for columns 11 and 12 has a different argument {k} it has to be adjusted
if j <= 10:
k = 1
# set x to get value of j
x = j
else:
k = 2
# set x to get last number from j ( 1 from 11,2 from 12)
x = int(str(j)[1:])
# Iterate over rows. I am able to extract 20 rows at once so no point of declaring bigger range
for i in range(1,21):
row = i
try:
print(i)
# get value from dashboard
tickerpath_2 = f"""//*[@id="pvExplorationHost"]/div/div/exploration/div/explore-canvas-modern/div/div[2]/div/div[2]/div[2]/visual-container-repeat/visual-container-modern[2]/transform/div/div[3]/div/visual-modern/div/div/div[2]/div[1]/div[4]/div/div[{k}]/div[{x}]/div[{row}]"""
ticker = driver.find_element_by_xpath(tickerpath_2)
print(ticker.text)
# append element to the list
ticker_list.append(ticker.text)
except:
continue
# append whole list as a pandas column
df[str(col)] = ticker_list
df_all = df_all.append(df)
现在在变量df_all中,您应该具有表的前20行。没关系但是,我无法自动滚动右侧的仪表板栏。我尝试了类似帖子中的各种选项,例如:
driver.execute_script("window.scrollTo(0,document.body.scrollHeight)")
或
scr1 = driver.find_element_by_xpath('/')
driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight",scr1)
或
read_more_buttons = driver.find_elements_by_class_name("scroll-bar-part-bar")
driver.execute_script("arguments[0].scrollIntoView(true);",read_more_buttons[1])
driver.execute_script("arguments[0].click()",read_more_buttons[1])
似乎这些方法在Dashboard的内部滚动条上不起作用。也许有人遇到过类似的问题? 否则,我将必须手动滚动每20行之后停止脚本:(
感谢您提前获得的所有帮助
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。