Scrapy - urlparse.urljoin的行为方式与str.join相同吗？

Question

我试图在Scrapy蜘蛛中使用urlparse.urljoin来编译一个url列表。目前，我的蜘蛛没有返回，但没有发现任何错误。所以我试图检查我是否正确地编译了这些URL。

我的尝试是使用str.join在闲置测试，如下图所示：

>>> href = ['lphs.asp?id=598&city=london', 
'lphs.asp?id=480&city=london', 
'lphs.asp?id=1808&city=london', 
'lphs.asp?id=1662&city=london', 
'lphs.asp?id=502&city=london',] 
>>> for x in href: 
    base = "http:/www.url-base.com/destination/" 
    final_url = str.join(base, x) 
    print(final_url) 

的返回什么一个行：

lhttp:/www.url-base.com/destination/phttp:/www.url-base.com/destination/hhttp:/www.url-base.com/destination/shttp:/www.url-base.com/destination/.http:/www.url-base.com/destination/ahttp:/www.url-base.com/destination/shttp:/www.url-base.com/destination/phttp:/www.url-base.com/destination/?http:/www.url-base.com/destination/ihttp:/www.url-base.com/destination/dhttp:/www.url-base.com/destination/=http:/www.url-base.com/destination/5http:/www.url-base.com/destination/9http:/www.url-base.com/destination/8http:/www.url-base.com/destination/&http:/www.url-base.com/destination/chttp:/www.url-base.com/destination/ihttp:/www.url-base.com/destination/thttp:/www.url-base.com/destination/yhttp:/www.url-base.com/destination/=http:/www.url-base.com/destination/lhttp:/www.url-base.com/destination/ohttp:/www.url-base.com/destination/nhttp:/www.url-base.com/destination/dhttp:/www.url-base.com/destination/ohttp:/www.url-base.com/destination/n

我认为，从我的例子是很明显，str.join不会以相同的方式表现 - 如果确实如此，那么这就是为什么我的蜘蛛没有遵循这些链接！ - 但是，对此有确认是很好的。

如果这不是正确的测试方法，我该如何测试这个过程？

更新使用以下urlparse.urljoin 尝试： 从进口的urllib.parse里urlparse

>>> from urllib.parse import urlparse 
    >>> for x in href: 
     base = "http:/www.url-base.com/destination/" 
     final_url = urlparse.urljoin(base, x) 
     print(final_url) 

这是投掷AttributeError: 'function' object has no attribute 'urljoin'

更新 - 相关

def parse_links(self, response): 
    room_links = response.xpath('//form/table/tr/td/table//a[div]/@href').extract() # insert xpath which contains the href for the rooms 
    for link in room_links: 
     base_url = "http://www.example.com/followthrough" 
     final_url = urlparse.urljoin(base_url, link) 
     print(final_url) 
     # This is not joing the final_url right 
     yield Request(final_url, callback=parse_links) 

蜘蛛功能

更新

我只是再次测试空闲：

>>> from urllib.parse import urljoin 
>>> from urllib import parse 
>>> room_links = ['lphs.asp?id=562&city=london', 
'lphs.asp?id=1706&city=london', 
'lphs.asp?id=1826&city=london', 
'lphs.asp?id=541&city=london', 
'lphs.asp?id=1672&city=london', 
'lphs.asp?id=509&city=london', 
'lphs.asp?id=428&city=london', 
'lphs.asp?id=614&city=london', 
'lphs.asp?id=336&city=london', 
'lphs.asp?id=412&city=london', 
'lphs.asp?id=611&city=london',] 
>>> for link in room_links: 
    base_url = "http:/www.url-base.com/destination/" 
    final_url = urlparse.urljoin(base_url, link) 
    print(final_url) 

其中抛出此：

Traceback (most recent call last): 
    File "<pyshell#34>", line 3, in <module> 
    final_url = urlparse.urljoin(base_url, link) 
AttributeError: 'function' object has no attribute 'urljoin' 

Answer 1

你看，因为这给出的输出：

for x in href: 
    base = "http:/www.url-base.com/destination/" 
    final_url = str.join(base, href) # <-- 'x' instead of 'href' probably intended here 
    print(final_url) 

urljoin来自urllib库的行为不同，请参阅文档。这不是简单的字符串连接。

编辑： 基于您的评论，我想你正在使用Python 3.该import语句，您导入urlparse功能。这就是你得到这个错误的原因。无论是进口和直接使用功能：

from urllib.parse import urljoin 
... 
final_url = urljoin(base, x) 

或导入parse模块，并使用这样的功能：

from urllib import parse 
... 
final_url = parse.urljoin(base, x)