获取scrapy蜘蛛抓取整个网站

Question

我使用scrapy抓取我拥有的旧网站，我将下面的代码用作我的蜘蛛。我不介意为每个网页输出文件，或者包含所有内容的数据库。但我确实需要能够有蜘蛛爬行用了我不得不把在我目前不得不做的每一个网址

import scrapy 

class DmozSpider(scrapy.Spider): 
    name = "dmoz" 
    allowed_domains = ["www.example.com"] 
    start_urls = [ 
     "http://www.example.com/contactus" 
    ] 

    def parse(self, response): 
     filename = response.url.split("/")[-2] + '.html' 
     with open(filename, 'wb') as f: 
      f.write(response.body) 

Answer 1

要抓取整个网站的整个事情，你应该使用CrawlSpider代替在scrapy.Spider

Here's an example

你的目的尝试使用这样的：

import scrapy 
from scrapy.spiders import CrawlSpider, Rule 
from scrapy.linkextractors import LinkExtractor 

class MySpider(CrawlSpider): 
    name = 'example.com' 
    allowed_domains = ['example.com'] 
    start_urls = ['http://www.example.com'] 

    rules = (
     Rule(LinkExtractor(), callback='parse_item', follow=True), 
    ) 

    def parse_item(self, response): 
     filename = response.url.split("/")[-2] + '.html' 
     with open(filename, 'wb') as f: 
      f.write(response.body) 

此外，请看看这个article