scrapy InitSpider：在__init__中设置规则？

Question

我正在构建一个可选登录的递归webspider。我想通过json配置文件使大多数设置动态化。

在我的__init__函数中，我正在阅读此文件并尝试填充所有变量，但是，这不适用于Rules。

class CrawlpySpider(InitSpider): 

... 

#---------------------------------------------------------------------- 
def __init__(self, *args, **kwargs): 
    """Constructor: overwrite parent __init__ function""" 

    # Call parent init 
    super(CrawlpySpider, self).__init__(*args, **kwargs) 

    # Get command line arg provided configuration param 
    config_file = kwargs.get('config') 

    # Validate configuration file parameter 
    if not config_file: 
     logging.error('Missing argument "-a config"') 
     logging.error('Usage: scrapy crawl crawlpy -a config=/path/to/config.json') 
     self.abort = True 

    # Check if it is actually a file 
    elif not os.path.isfile(config_file): 
     logging.error('Specified config file does not exist') 
     logging.error('Not found in: "' + config_file + '"') 
     self.abort = True 

    # All good, read config 
    else: 
     # Load json config 
     fpointer = open(config_file) 
     data = fpointer.read() 
     fpointer.close() 

     # convert JSON to dict 
     config = json.loads(data) 

     # config['rules'] is simply a string array which looks like this: 
     # config['rules'] = [ 
     # 'password', 
     # 'reset', 
     # 'delete', 
     # 'disable', 
     # 'drop', 
     # 'logout', 
     # ] 

     CrawlpySpider.rules = (
      Rule(
       LinkExtractor(
        allow_domains=(self.allowed_domains), 
        unique=True, 
        deny=tuple(config['rules']) 
       ), 
       callback='parse', 
       follow=False 
      ), 
     ) 

Scrapy仍然爬存在于config['rules']，因此也打0​​页面的页面。所以指定的页面不会被拒绝。我在这里错过了什么？

更新：

我已经通过设置CrawlpySpider.rules = ...以及self.rules = ...内__init__尝试。两种变体都不起作用。

• 蜘蛛：InitSpider
• 规则：LinkExtractor
• 爬行前：做事先登录爬行

我甚至试图否认，在我的parse功能

# Dive deeper? 
    # The nesting depth is now handled via a custom middle-ware (middlewares.py) 
    #if curr_depth < self.max_depth or self.max_depth == 0: 
    links = LinkExtractor().extract_links(response) 
    for link in links: 
     for ignore in self.ignores: 
      if (ignore not in link.url) and (ignore.lower() not in link.url.lower()) and link.url.find(ignore) == -1: 
       yield Request(link.url, meta={'depth': curr_depth+1, 'referer': response.url}) 

Answer 1

你是在要设置实例属性的位置设置类属性：

# this: 
CrawlpySpider.rules = (
# should be this: 
self.rules = (
<...>