Scrapy SGML链接提取与自定义标记和属性

Question

我想知道为什么我的链接提取没有得到任何链接。我正在扩展CrawlSpider并在链接提取器中设置tags和attrs属性。我试图从网页上的选择框中取消链接（http://www.realclearpolitics.com/epolls/latest_polls/president/），所以我正在寻找带有值属性的选项标签。

结帐代码在这里：https://github.com/jtfairbank/RCP-Poll-Scraper/blob/master/polldata/spiders/pres.py

而这里的根目录：https://github.com/jtfairbank/RCP-Poll-Scraper

任何想法？谢谢！

Answer 1

你的链接提取没有得到任何的联系，因为你与tags=['option']寻找<option>标签和有上this page没有选择或选项标签。

尝试restrict_xpaths='//td[@class="lp-race"]'：

>>> SgmlLinkExtractor(
...  allow=(r"epolls/2012/president/[a-z]{2}/[a-z]+_romney_vs_obama-[0-9]{4}\.html"), 
...  allow_domains=('realclearpolitics.com',), 
...  restrict_xpaths='//td[@class="lp-race"]', 
...).extract_links(response) 

[Link(url='http://www.realclearpolitics.com/epolls/2012/president/fl/florida_romney_vs_ob...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/oh/ohio_romney_vs_obama...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/va/virginia_romney_vs_o...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/ia/iowa_romney_vs_obama...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/wi/wisconsin_romney_vs_...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/or/oregon_romney_vs_oba...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/mo/missouri_romney_vs_o...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/ca/california_romney_vs...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/md/maryland_romney_vs_o...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/mi/michigan_romney_vs_o...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/co/colorado_romney_vs_o...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/mn/minnesota_romney_vs_...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/wa/washington_romney_vs...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/ct/connecticut_romney_v...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/nv/nevada_romney_vs_oba...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/mt/montana_romney_vs_ob...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/pa/pennsylvania_romney_...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/in/indiana_romney_vs_ob...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/ma/massachusetts_romney...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/ga/georgia_romney_vs_ob...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/id/idaho_romney_vs_obam...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/az/arizona_romney_vs_ob...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/me/maine_romney_vs_obam...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/la/louisiana_romney_vs_...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/hi/hawaii_romney_vs_oba...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/tx/texas_romney_vs_obam...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/ar/arkansas_romney_vs_o...', 
Link(url='http://www.realclearpolitics.com/epolls/2012/president/ne/nebraska_romney_vs_o...',