使用 Python ElementTree 抓取具有命名空间的多个 URL 的 rss/xml

如何解决使用 Python ElementTree 抓取具有命名空间的多个 URL 的 rss/xml

我正在制作一个抓取工具，它可以循环浏览带有 RSS 提要的 url 列表，然后检索其博客文章的标题、博客文章的日期以及博客文章的链接。这后来被提交给 postgres。我已经制作了另一个脚本，它在没有命名空间的情况下检索 xml，并将这个脚本基于该代码。我认为我无法理解的最大问题是我有很多网站，其中各种网站都有备用名称空间，所以我不完全明白我将如何以优雅的方式检索这些不同的提要。>

以下是我遇到的一些 XML 示例。

    <rss xmlns:dc="http://purl.org/dc/elements/1.1/" version="2.0">
    <rss xmlns:atom="http://www.w3.org/2005/Atom" version="2.0">
    <rss xmlns:content="http://purl.org/rss/1.0/modules/content/">
    <rss xmlns:sy="http://purl.org/rss/1.0/modules/syndication/">
    <rss xmlns:georss="http://www.georss.org/georss">
    <rss mlns:slash="http://purl.org/rss/1.0/modules/slash/">
    <rss xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#" version="2.0">
    <rss xmlns:wfw="http://wellformedweb.org/CommentAPI/>

事实上，其中一个事件在单个 rss 标签中包含所有 xml 命名空间。

    <rss xmlns:content="http://purl.org/rss/1.0/modules/content/" 
    xmlns:wfw="http://wellformedweb.org/CommentAPI/" 
    xmlns:dc="http://purl.org/dc/elements/1.1/" 
    xmlns:atom="http://www.w3.org/2005/Atom" 
    xmlns:sy="http://purl.org/rss/1.0/modules/syndication/" 
    xmlns:slash="http://purl.org/rss/1.0/modules/slash/" 
    xmlns:georss="http://www.georss.org/georss" 
    xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#" version="2.0> 

这是我的刮板的代码。问题从第二次尝试开始

import asyncio
import httpx
import xml.etree.ElementTree as ET
import psycopg2
import pdb

#open initial connection
conn = psycopg2.connect("")

#open initial cursor
cur = conn.cursor()

URLS =  [
        "https://www.sitename1.com/feed.xml","https://www.sitename2.com/atom.xml","https://www.sitename3.com/feed.xml","https://www.sitename4.com/index.xml","https://www.sitename5.com/links/rss.xml","https://www.sitename6.com/blog/rss.xml","http://www.sitename7.com/rss.xml" ] 
        
        # there's at least 10 more sites after this,but you get the picture.

async def main():

    async with httpx.AsyncClient() as client:
        for url in URLS:
            response = await client.get(url)
            
            try:
                root = ET.fromstring(response.text)
                print("ROOT:",root)
                    
            except:
                continue
            try:
                links = [x for x in root if x.tag in ("entry","items")]
                #links = [x for x in root if x.tag in ("entry","items")]
                print("LINKS:",links)
            except:
                print("URL {} is rejected".format(url))
                continue

            for link in links:
                title = [x.text for x in link if x.tag == "title"]
                link_url = [x.attrib["href"] for x in link if x.tag == "link"]
                if title and link_url:
                    print("Found {} with HREF {}".format(title,link_url))
                    #cur.execute("INSERT INTO posts (host_title,post_url) VALUES (%s,%s)",#(title[0],link_url[0]))
                    #conn.commit()
                    print("committed")
                    print(f"{title} and {link_url} submitted to database.")
                    
    cur.execute("SELECT * FROM posts;")
    rows = cur.fetchall()
    for r in rows:
        print(f"{r[0]} and {r[1]}")
    cur.close()
    conn.close()  

if __name__ == '__main__':
    asyncio.run(main())

这是目前的输出，我通过抛出异常停止了它，但现在它输出的只是一个空列表。未检索任何数据。

    ROOT: <Element 'rss' at 0x7f4b249a7220>
    LINKS: []
    ROOT: <Element 'rss' at 0x7f4b2528ab30>
    LINKS: []
    ROOT: <Element 'rss' at 0x7f4b249707c0>
    LINKS: []
    ROOT: <Element 'rss' at 0x7f4b24978130>
    LINKS: []
    ROOT: <Element 'rss' at 0x7f4b249990e0>
    LINKS: []
    ROOT: <Element 'rss' at 0x7f4b249a7d60>
    LINKS: []
    ROOT: <Element 'rss' at 0x7f4b249bb860>
    LINKS: []
    ROOT: <Element 'rss' at 0x7f4b249c3720>
    LINKS: []
    ROOT: <Element 'rss' at 0x7f4b2497d270>
    LINKS: []

我相信问题出在这行代码上。

 try:
      links = [x for x in root if x.tag[1] in ("entry","items")]
      #links = [x for x in root if x.tag in ("channel","items")]
      print("LINKS:",links)

人们会怎么做呢？我浏览了 python 文档，阅读了 stackoverflow 线程，并挖掘了一堆博客线程，我尝试了各种提供的解决方案，但最终无济于事。任何帮助将不胜感激。