刮：刮URL

Question

的第二电平在下面的代码解析函数执行大约32倍（福尔环32 HREF的实测值）在同一开头每个子链路该去刮去数据（32个单个网址parse_next功能） 。但parse_next功能只执行一次（单程）/不叫（输出CSV文件empty.can任何人能帮助我在哪里，我没有错

import scrapy 
import logging 

logger = logging.getLogger('mycustomlogger') 

from ScrapyTestProject.items import ScrapytestprojectItem 
class QuotesSpider(scrapy.Spider): 
    name = "nestedurl" 
    allowed_domains = ['www.grohe.in'] 
    start_urls = [ 
     'https://www.grohe.com/in/7780/bathroom/bathroom-faucets/essence/', 
def parse(self, response): 
    logger.info("Parse function called on %s", response.url) 
    for divs in response.css('div.viewport div.workspace div.float-box'): 
     item = {'producturl': divs.css('a::attr(href)').extract_first(), 
       'imageurl': divs.css('a img::attr(src)').extract_first(), 
       'description' : divs.css('a div.text::text').extract() + divs.css('a span.nowrap::text').extract()} 
     next_page = response.urljoin(item['producturl']) 
     #logger.info("This is an information %s", next_page) 
     yield scrapy.Request(next_page, callback=self.parse_next, meta={'item': item}) 
     #yield item 

def parse_next(self, response): 
    item = response.meta['item'] 
    logger.info("Parse function called on2 %s", response.url) 
    item['headline'] = response.css('div#content a.headline::text').extract() 
    return item 
    #response.css('div#product-variants a::attr(href)').extract() 

Answer 1

确定，所以一些事情出错：

• 压痕
• start_urls列表不与[
• allowed_domains封闭使用的域扩展。在同时要刮.COM

工作下面的代码：

import scrapy 
import logging 

class QuotesSpider(scrapy.Spider): 
    name = "nestedurl" 
    allowed_domains = ['www.grohe.com'] 
    start_urls = [ 
     'https://www.grohe.com/in/7780/bathroom/bathroom-faucets/essence/' 
    ] 
    def parse(self, response): 
     # logger.info("Parse function called on %s", response.url) 
     for divs in response.css('div.viewport div.workspace div.float-box'): 
      item = {'producturl': divs.css('a::attr(href)').extract_first(), 
        'imageurl': divs.css('a img::attr(src)').extract_first(), 
        'description' : divs.css('a div.text::text').extract() + divs.css('a span.nowrap::text').extract()} 
      next_page = response.urljoin(item['producturl']) 
      #logger.info("This is an information %s", next_page) 
      yield scrapy.Request(next_page, callback=self.parse_next, meta={'item': item}) 
      #yield item 

    def parse_next(self, response): 
     item = response.meta['item'] 
     # logger.info("Parse function called on2 %s", response.url) 
     item['headline'] = response.css('div#content a.headline::text').extract() 
     return item 
     #response.css('div#product-variants a::attr(href)').extract() 

注意：删除一些日志记录/项目管道，因为这些没有在我的机器上定义。