BeautifulSoup找不到所有tr标签

如何解决BeautifulSoup找不到所有tr标签

我在这里完全不知所措。我正在尝试从以下网址https://www.baseball-reference.com/leagues/MLB/2019-appearances-fielding.shtml中提取表格。

在此页面上，有两个表。第一个是团队外观，第二个是玩家外观。

以下代码片段使我可以毫无问题地获取页面的html。

url = 'https://www.baseball-reference.com/leagues/MLB/2019-appearances-fielding.shtml'
html_content = requests.get(url).text
soup = BeautifulSoup(html_content,'lxml')
print(soup.prettify())

一切都检查完了。当我尝试解析以获取标头信息时，我没有遇到任何麻烦。为此，我正在使用：

header = soup.find_all('tr',attrs ={"class": "thead"})
header

这里发生的是，这只是第一次出现，这是在“团队出场”表中进行的，而不是两次。

[<tr class="thead">
 <th aria-label="Tm" class="sort_default_asc show_partial_when_sorting center" data-stat="team_ID">Tm</th>
 <th aria-label="G_team" class="center" data-stat="G_team"></th>
 <th aria-label="Games Played" class="center" data-stat="G_all" data-tip="All games played">G</th>
 <th aria-label="Games Started" class="center" data-stat="GS" data-tip="Games Started">GS</th>
 <th aria-label="Games Played" class="center" data-stat="G_batting" data-tip="Games appeared in the batting order,&lt;br&gt;but may not have batted.">Batting</th>
 <th aria-label="Games Played" class="center" data-stat="G_defense" data-tip="Games in lineup at a defensive position.">Defense</th>
 <th aria-label="Games Played" class="center" data-stat="G_p_app" data-tip="Games in lineup or announced as a pitcher">P</th>
 <th aria-label="Games Played" class="center" data-stat="G_c" data-tip="Games in lineup as a catcher">C</th>
 <th aria-label="Games Played" class="center" data-stat="G_1b" data-tip="Games in lineup as a first baseman">1B</th>
 <th aria-label="Games Played" class="center" data-stat="G_2b" data-tip="Games in lineup as a second baseman">2B</th>
 <th aria-label="Games Played" class="center" data-stat="G_3b" data-tip="Games in lineup as a third baseman">3B</th>
 <th aria-label="Games Played" class="center" data-stat="G_ss" data-tip="Games in lineup as a shortstop">SS</th>
 <th aria-label="Games Played" class="center" data-stat="G_lf_app" data-tip="Games in lineup as a left fielder">LF</th>
 <th aria-label="Games Played" class="center" data-stat="G_cf_app" data-tip="Games in lineup as a center fielder">CF</th>
 <th aria-label="Games Played" class="center" data-stat="G_rf_app" data-tip="Games in lineup as a right fielder">RF</th>
 <th aria-label="Games Played" class="center" data-stat="G_of_app" data-tip="Games in lineup as an outfielder">OF</th>
 <th aria-label="Games Played" class="center" data-stat="G_dh" data-tip="Games in lineup as a designated hitter">DH</th>
 <th aria-label="Games Played" class="center" data-stat="G_ph" data-tip="Games in lineup as a pinch hitter&lt;br&gt;May have played another position as well.">PH</th>
 <th aria-label="Games Played" class="center" data-stat="G_pr" data-tip="Games in lineup as a pinch runner&lt;br&gt;May have played another position as well.">PR</th>
 </tr>]

当我检查页面的html时，Player Appearances表也具有此类，因此我不确定为什么两者都没有填充。此外，如果我将代码更改为类似

table = soup.find_all('tr',attrs ={"class": "full_table"})
table

适合实际数据的类，我什么也没得到。

我已经工作了一段时间了，很茫然。我只是在这里忽略一些简单的东西吗？我们将不胜感激。

谢谢！

解决方法

您可以使用comments = soup.find_all(string=lambda text: isinstance(text,Comment))提取注释，然后解析其中的表。

import requests
from bs4 import BeautifulSoup,Comment
import pandas as pd

headers = {'User-Agent': 
           'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/47.0.2526.106 Safari/537.36'}

# Get list of tables with pandas    
url = "https://www.baseball-reference.com/leagues/MLB/2019-appearances-fielding.shtml"
tables = pd.read_html(url)


# then pull out comments and get the tables within those
response = requests.get(url,headers=headers)
soup = BeautifulSoup(response.content,'html.parser')
comments = soup.find_all(string=lambda text: isinstance(text,Comment))

for each in comments:
    if '<table' in each:
        tables.append(pd.read_html(each)[0])