httplib没有获得所有重定向代码

如何解决httplib没有获得所有重定向代码

| 我正在尝试获取似乎多次重定向的页面的最终URL。在浏览器中尝试以下示例URL，并将其与我的代码段底部的最终URL进行比较： 链接多次重定向 这是我正在运行的测试代码，请注意，获得代码200的最终URL与浏览器中的URL不相同。我有什么选择？

Python 2.7.1+ (r271:86832,Apr 11 2011,18:13:53)
[GCC 4.5.2] on linux2
Type \"help\",\"copyright\",\"credits\" or \"license\" for more information.
>>> import httplib
>>> from urlparse import urlparse
>>> url = \'http://www.usmc.mil/units/hqmc/\'
>>> host = urlparse(url)[1]
>>> req = \'\'.join(urlparse(url)[2:5])
>>> conn = httplib.HTTPConnection(host)
>>> conn.request(\'HEAD\',req)
>>> resp = conn.getresponse()
>>> print resp.status
    301
>>> print resp.msg.dict[\'location\']
    http://www.marines.mil/units/hqmc/

>>> url = \'http://www.marines.mil/units/hqmc/\'
>>> host = urlparse(url)[1]
>>> req = \'\'.join(urlparse(url)[2:5])
>>> conn = httplib.HTTPConnection(host)
>>> conn.request(\'HEAD\',req)
>>> resp = conn.getresponse()
>>> print resp.status
    302
>>> print resp.msg.dict[\'location\']
    http://www.marines.mil/units/hqmc/default.aspx

>>> url = \'http://www.marines.mil/units/hqmc/default.aspx\'
>>> host = urlparse(url)[1]
>>> req = \'\'.join(urlparse(url)[2:5])
>>> conn = httplib.HTTPConnection(host)
>>> conn.request(\'HEAD\',req)
>>> resp = conn.getresponse()
>>> print resp.status
    200
>>> print resp.msg.dict[\'location\']
    Traceback (most recent call last):
    File \"<stdin>\",line 1,in <module>
    KeyError: \'location\'
>>> print url
    http://www.marines.mil/units/hqmc/default.aspx //THIS URL DOES NOT RETURN A 200 IN ANY BROWSER I HAVE TRIED 

    

解决方法

        您可以尝试将User-Agent标头设置为浏览器的User-Agent。 ps： urllib2自动重定向 编辑：

In [2]: import urllib2
In [3]: resp = urllib2.urlopen(\'http://www.usmc.mil/units/hqmc/\')
In [4]: resp.geturl()
Out[4]: \'http://www.marines.mil/units/hqmc/default.aspx

    ,        您可以使用HttpLib2来获取URL的实际位置：

import httplib2

def getContentLocation(link):
    h = httplib2.Http(\".cache_httplib\")
    h.follow_all_redirects = True
    resp = h.request(link,\"GET\")[0]
    contentLocation = resp[\'content-location\']
    return contentLocation

if __name__ == \'__main__\':
    link = \'http://podcast.at/podcast_url344476.html\'
    print getContentLocation(link)

执行看起来像这样：

$ python2.7 getContentLocation.py
http://keyinvest.podcaster.de/8uhr30.rss

请注意，此示例还使用了缓存（urllib和httplib均不支持该缓存）。因此，这将重复运行明显更快。这对于爬网/抓取可能很有趣。如果不想缓存，请将

h = httplib2.Http(\".cache_httplib\")

替换为

h = httplib2.Http()

。