来自命令行参数的蜘蛛

Question

以下蜘蛛代码输出文件的格式为k.1375093834.0.txt。我要的是形式kickstarter.com.1375093834.0.txt

任何建议代码变化将是非常有益的

class shnurl(CrawlSpider): 
    name = "shnurl" 
    #start_urls = [ 
    # "http://www.blogger.com" 
    # ] 
    rules = [ 
     Rule(SgmlLinkExtractor(),follow=True, callback="parse") 
    ] 

    def __init__(self, *args, **kwargs): 

     #Initialize the parent class. 
     super(shnurl, self).__init__(*args, **kwargs) 

     #Get the start URL from the command line. 
     self.start_urls = [kwargs.get('start_url')] 

     #Create a results file based on the start_url + current time. 
     self.fname = '{0}.{1}.{2}'.format(self.start_url[12], time.time(),'txt') 
     self.fileout = open(self.fname, 'w+') 

     #Create a logfile based on the start_url + current time. 
     #Log file stores the errors, debug & info prints. 
     logfname = '{0}.{1}.{2}'.format(self.start_url[12], time.time(),'log') 
     #log.start(logfile='./runtime.log', loglevel=log.INFO) 
     log.start(logfile=logfname, loglevel=log.INFO) 
     self.log('Output will be written to: {0}'.format(self.fname), log.INFO) 
     #End of constructor 

使用的文件名： -

scrapy crawl shnurl -a start_url="https://www.kickstarter.com" 

Answer 1

假设我已经认识这个问题，你想在start_url上做一个切片，但是你错误地定义了它。根据以下方括号中的12后面加一个冒号，这将解决问题：

self.fname = '{0}.{1}.{2}'.format(self.start_url[12:], time.time(),'txt') 
    logfname = '{0}.{1}.{2}'.format(self.start_url[12:], time.time(),'log')