Scrapy通过兄弟节点迭代 - xpath生成空列表

Question

我遍历元素的兄弟节点。但是，我无法使用xpath函数来提取其中的数据。我似乎总是获得一个空的列表。

# Determine if the cast members are listed in a table or unordered list. 
cast_siblings = response.xpath('//h2/span[starts-with(@id,"Cast")]/../following-sibling::*') 
for sibling in cast_siblings: 
    desc_str = "".join(sibling.xpath('./descendant::text()').extract()) 
    if "Trailer" in desc_str or "Film Festival" in desc_str or "Comment" in desc_str: 
     break 

    # Extract the name to get the type of the node. 
    node_type = sibling.xpath('name()').extract_first() 
    if node_type == 'ul': 
     cast_list = sibling.xpath('./ul[1]/li') 
     self.parse_list_into_cast_table(cast_list, _id, conn) 
    elif node_type == 'table': 
     cast_tables = sibling.xpath('./table') 
     self.parse_table_into_cast_table(cast_tables, _id, conn) 
    else: 
     print "Unknown cast element type! " + _id 

在上面的代码，我一直在经历的兄弟节点，直到我打它包含“拖车”，“电影节”或“评论”的地方文本的元素。否则，对于每个兄弟节点，我首先得到它的名字来找出它是什么样的节点 - 我对表和无序列表感兴趣。

但是，失败的部分是当我调用sibling.xpath（'./ table'）和/或sibling.xpath（'ul [1]/li'）时。

生成的对象始终是一个空列表。

我已经使用scrapy shell并检查了从cast_siblings获得的选择器列表。每个选择器似乎都是有效的，当我提取它们时，我可以看到内部元素。我只是无法使用xpath到达他们。

下面是我的控制台显示空表结果的摘录：

>>> a = response.xpath('//h2/span[starts-with(@id,"Cas")]/../following-sibling::*') 
>>> a 
[<Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h3> <span class="mw-headline" id="Sudo_'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h3> <span class="mw-headline" id="Maruy'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h3> <span class="mw-headline" id="Azumi'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h3> <span class="mw-headline" id="Prese'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<p><b>Additional Cast Members:</b>\n</p>'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<ul><li> <a href="/Yukari_Ito" title="Yu'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h2> <span class="mw-headline" id="Trail'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<p>\n<script type="text/javascript" src="'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<div id="mediaplayer"></div>'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<script type="text/javascript">\n jwplay'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h2> <span class="mw-headline" id="Comme'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<br>'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<div id="comment-outer"><hr><div id="com'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<div align="center"></div>'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table cellpadding="2" style="border:0px'>] 
>>> a[0] 
<Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h3> <span class="mw-headline" id="Sudo_'> 
>>> a[1] 
<Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'> 
>>> a[1].xpath('./table') 
[] 

这是很奇怪的，因为A [1]在上面的例子清楚地表明内的表。

Answer 1

好像你正在寻找已经是一个表节点的表儿童：

node_type = sibling.xpath('name()').extract_first() 
if node_type == 'ul': 
    ... 
elif node_type == 'table': 
    cast_tables = sibling.xpath('./table') <--- 

这里你已经知道，当前节点表，你正在试图寻找它的下面另一个表。所以在这种情况下，它应该仅仅是：

cast_tables = sibling 

另外值得一提的是，当你测试scrapy选择的结果包括根节点的标签，即'//h1'将返回<h1>...</h1>