Scrapy不是爬行网页

Question

我不知道什么地方错了这种蜘蛛，但它不会抓取任何网页：

from scrapy import log 
from scrapy.contrib.spiders import CrawlSpider, Rule 
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor 
from scrapy.selector import HtmlXPathSelector 

from paper_crawler.items import PaperCrawlerItem 

class PlosGeneticsSpider(CrawlSpider): 
    name = 'plosgenetics' 
    allowed_domains = ['plosgenetics.org'] 
    start_urls = ['http://www.plosgenetics.org/article/browse/volume'] 

    rules = [ 
     Rule(SgmlLinkExtractor(allow=(),restrict_xpaths=('//ul[@id="journal_slides"]')), callback='parse_item', follow=True) 
     ] 


    def parse_item(self, response): 
     self.log(response.url) 
     print response.url 

     hxs = HtmlXPathSelector(response) 
     titles = hxs.select('//div[@class="item cf"]') 
     items = [] 
     for title in titles: 
      item = PaperCrawlerItem() 
      item['title'] = "".join(title.xpath('.//div[@class="header"]//h3//a[contains(@href,"article")]/text()').extract()).strip() 
      item['URL'] = title.xpath('.//div[@class="header"]//h3//a[contains(@href,"article")]/@href').extract() 
      item['authors'] = "".join(title.xpath('.//div[@class="header"]//div[@class="authors"]/text()').extract()).replace('\n', "") 
      items.append(item) 

     return(items) 

语法是正确的，但它一直对如何说INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)

任何想法我搞砸了？

Answer 1

我可以发现的最明显的错误是您在CrawlSpider.rules的定义中应该是一个元组，您将在this question的第一个答案中找到修复。

但在短期，CrawlSpider.rules应该是：

rules = (Rule(...),Rule(...),) 

公告支架和逗号。