Scrapy在几页后停止爬行

Question

我只是拿起Scrapy和网站爬虫的基本知识，所以我会非常感谢您的意见。我从Scrapy构建了一个简单而简单的抓取工具，在教程的指导下进行。

它工作正常，但它不会抓取所有页面，因为它应该。

我的蜘蛛的代码是：

from scrapy.spider  import BaseSpider 
from scrapy.selector  import HtmlXPathSelector 
from scrapy.http.request import Request 
from fraist.items  import FraistItem 
import re 

class fraistspider(BaseSpider): 
    name = "fraistspider" 
    allowed_domain = ["99designs.com"] 
    start_urls = ["http://99designs.com/designer-blog/"] 

    def parse(self, response): 
     hxs = HtmlXPathSelector(response) 
     links = hxs.select("//div[@class='pagination']/a/@href").extract() 

     #We stored already crawled links in this list 
     crawledLinks = [] 

     #Pattern to check proper link 
     linkPattern  = re.compile("^(?:ftp|http|https):\/\/(?:[\w\.\-\+]+:{0,1}[\w\.\-\+]*@)?(?:[a-z0-9\-\.]+)(?::[0-9]+)?(?:\/|\/(?:[\w#!:\.\?\+=&%@!\-\/\(\)]+)|\?(?:[\w#!:\.\?\+=&%@!\-\/\(\)]+))?$") 

     for link in links: 
      # If it is a proper link and is not checked yet, yield it to the Spider 
      if linkPattern.match(link) and not link in crawledLinks: 
       crawledLinks.append(link) 
       yield Request(link, self.parse) 

     posts = hxs.select("//article[@class='content-summary']") 
     items = [] 
     for post in posts: 
      item = FraistItem() 
      item["title"] = post.select("div[@class='summary']/h3[@class='entry-title']/a/text()").extract() 
      item["link"] = post.select("div[@class='summary']/h3[@class='entry-title']/a/@href").extract() 
      item["content"] = post.select("div[@class='summary']/p/text()").extract() 
      items.append(item) 
     for item in items: 
      yield item 

，输出是：

  'title': [u'Design a poster in the style of Saul Bass']} 
2015-05-20 16:22:41+0100 [fraistspider] DEBUG: Scraped from <200 http://nnbdesig 
ner.wpengine.com/designer-blog/> 
     {'content': [u'Helping a company come up with a branding strategy can be 
exciting\xa0and intimidating, all at once. It gives a designer the opportunity 
to make a great visual impact with a brand, but requires skills in logo, print a 
nd digital design. If you\u2019ve been hesitating to join a 99designs Brand Iden 
tity Pack contest, here are a... '], 
     'link': [u'http://99designs.com/designer-blog/2015/05/07/tips-brand-ide 
ntity-pack-design-success/'], 
     'title': [u'99designs\u2019 tips for a successful Brand Identity Pack d 
esign']} 
2015-05-20 16:22:41+0100 [fraistspider] DEBUG: Redirecting (301) to <GET http:// 
nnbdesigner.wpengine.com/> from <GET http://99designs.com/designer-blog/page/10/ 
> 
2015-05-20 16:22:41+0100 [fraistspider] DEBUG: Redirecting (301) to <GET http:// 
nnbdesigner.wpengine.com/> from <GET http://99designs.com/designer-blog/page/11/ 
> 
2015-05-20 16:22:41+0100 [fraistspider] INFO: Closing spider (finished) 
2015-05-20 16:22:41+0100 [fraistspider] INFO: Stored csv feed (100 items) in: da 
ta.csv 
2015-05-20 16:22:41+0100 [fraistspider] INFO: Dumping Scrapy stats: 
     {'downloader/request_bytes': 4425, 
     'downloader/request_count': 16, 
     'downloader/request_method_count/GET': 16, 
     'downloader/response_bytes': 126915, 
     'downloader/response_count': 16, 
     'downloader/response_status_count/200': 11, 
     'downloader/response_status_count/301': 5, 
     'dupefilter/filtered': 41, 
     'finish_reason': 'finished', 
     'finish_time': datetime.datetime(2015, 5, 20, 15, 22, 41, 738000), 
     'item_scraped_count': 100, 
     'log_count/DEBUG': 119, 
     'log_count/INFO': 8, 
     'request_depth_max': 5, 
     'response_received_count': 11, 
     'scheduler/dequeued': 16, 
     'scheduler/dequeued/memory': 16, 
     'scheduler/enqueued': 16, 
     'scheduler/enqueued/memory': 16, 
     'start_time': datetime.datetime(2015, 5, 20, 15, 22, 40, 718000)} 
2015-05-20 16:22:41+0100 [fraistspider] INFO: Spider closed (finished) 

正如你可以看到'item_scraped_count'是100，虽然它应该是更因为有122页总，每页10篇文章。

从输出我可以看到有301重定向问题，但我不明白为什么这会导致问题。我已经尝试了另一种方法来重写我的蜘蛛代码，但是它在同一部分的几次输入之后再次破坏。

任何帮助将不胜感激。谢谢！

Answer 1

好像你正在击中http://doc.scrapy.org/en/latest/topics/settings.html#concurrent-items中定义的默认100个项目。

对于这种情况，我将使用CrawlSpider来抓取多个页面，因此您必须定义与99designs.com中的页面相匹配的rule，并轻松修改您的解析函数以处理该项目。

ç& P中的example code from the Scrapy docs：

import scrapy 
from scrapy.contrib.spiders import CrawlSpider, Rule 
from scrapy.contrib.linkextractors import LinkExtractor 

class MySpider(CrawlSpider): 
    name = 'example.com' 
    allowed_domains = ['example.com'] 
    start_urls = ['http://www.example.com'] 

    rules = (
     # Extract links matching 'category.php' (but not matching 'subsection.php') 
     # and follow links from them (since no callback means follow=True by default). 
     Rule(LinkExtractor(allow=('category\.php',), deny=('subsection\.php',))), 

     # Extract links matching 'item.php' and parse them with the spider's method parse_item 
     Rule(LinkExtractor(allow=('item\.php',)), callback='parse_item'), 
    ) 

    def parse_item(self, response): 
     self.log('Hi, this is an item page! %s' % response.url) 
     item = scrapy.Item() 
     item['id'] = response.xpath('//td[@id="item_id"]/text()').re(r'ID: (\d+)') 
     item['name'] = response.xpath('//td[@id="item_name"]/text()').extract() 
     item['description'] = response.xpath('//td[@id="item_description"]/text()').extract() 
     return item 

编辑：我刚刚发现this blog post包含一个有用的例子。