Scrapy Craigslist的脚本

Question

我想创建一个Scrapy脚本来凑所有结果电脑演出在Craigslist的任意子域： 例如这里：http://losangeles.craigslist.org/search/cpg/ 此查询返回的许多文章的列表，我试图刮这些结果（不仅是第一页上的结果）的标题和href无法使用CrawlSpider和linkExtractor，但脚本不会返回任何结果。 我会在这里贴上我的剧本，感谢

import scrapy 
    from scrapy.spiders import Rule,CrawlSpider 
    from scrapy.linkextractors import LinkExtractor 

    class CraigspiderSpider(CrawlSpider): 
     name = "CraigSpider" 
     allowed_domains = ["http://losangeles.craigslist.org"] 
     start_urls = (
        'http://losangeles.craigslist.org/search/cpg/', 
     ) 

     rules = (Rule(LinkExtractor(allow=(), restrict_xpaths=('//a[@class="button next"]',)), callback="parse_page", follow= True),) 

     def parse_page(self, response): 
      items = response.selector.xpath("//p[@class='row']") 
     for i in items: 
      link = i.xpath("./span[@class='txt']/span[@class='pl']/a/@href").extract() 
      title = i.xpath("./span[@class='txt']/span[@class='pl']/a/span[@id='titletextonly']/text()").extract() 
      print link,title 

Answer 1

根据您粘贴代码，parse_page：

1. 不返回/产生任何东西，
2. 只包含一条线：“项目= response.selector ...“

以上＃2的原因是for循环未正确缩进。

尝试缩进for循环：

class CraigspiderSpider(CrawlSpider): 
    name = "CraigSpider" 
    allowed_domains = ["http://losangeles.craigslist.org"] 
    start_urls = ('http://losangeles.craigslist.org/search/cpg/',) 

    rules = (Rule(
     LinkExtractor(allow=(), restrict_xpaths=('//a[@class="button next"]',)), 
     callback="parse_page", follow= True)) 

    def parse_page(self, response): 
     items = response.selector.xpath("//p[@class='row']") 

     for i in items: 
      link = i.xpath("./span[@class='txt']/span[@class='pl']/a/@href").extract() 
      title = i.xpath("./span[@class='txt']/span[@class='pl']/a/span[@id='titletextonly']/text()").extract() 
      print link, title 
      yield dict(link=link, title=title)