如何抓取分类的网站

Question

我正在尝试使用Scrapy编写抓取工具来抓取分类类型（目标）网站并从目标网站上的链接获取信息。关于Scrapy的教程只帮助我从目标URL获取链接，但不能获得我寻求的第二层数据收集。任何线索？

因此，例如，目标站点将是：

start_url = "http://newyork.craigslist.org/search/cta" 

和所有目标站点上我想去给每个上市，并获得价格和卖家，地点，电话或电子邮件

链接

Answer 1

import scrapy 
from scrapy.contrib.spiders import CrawlSpider, Rule 
from scrapy.selector import Selector 
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor 
from urlparse import urljoin 


class CompItem(scrapy.Item): 
    name = scrapy.Field() 
    price = scrapy.Field() 
    location = scrapy.Field() 




class criticspider(CrawlSpider): 
    name = "craig" 
    allowed_domains = ["newyork.craigslist.org"] 
    start_urls = ["http://newyork.craigslist.org/search/cta"] 


    def parse(self, response): 
     sites = response.xpath('//div[@class="content"]') 
     items = [] 

     for site in sites: 
      item = CompItem() 
      item['name'] = site.xpath('.//p[@class="row"]/span[@class="txt"]/span[@class="pl"]/a/text()').extract(). 
      item['price'] = site.xpath('.//p[@class="row"]/span[@class="txt"]/span[@class="l2"]/span[@class="price"]/text()').extract() 
      item['location'] = site.xpath('.//p[@class="row"]/span[@class="txt"]/span[@class="l2"]/span[@class="pnr"]/small/text()').extract() 
      items.append(item) 
      return items