502在使用Scrapy飞溅领英时发生错误

我尝试使用Scrapy和Splash为Netflix刮取LinkedIn公司页面。当我使用scrapy shell时，它工作得很好，但当我运行脚本时会产生502错误。502在使用Scrapy飞溅领英时发生错误

错误：

2017-01-06 16:06:45 [scrapy.core.engine] INFO: Spider opened 
2017-01-06 16:06:45 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 
2017-01-06 16:06:52 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.linkedin.com/company/netflix via http://localhost:8050/render.html> (failed 1 times): 502 Bad Gateway 
2017-01-06 16:06:55 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.linkedin.com/company/netflix via http://localhost:8050/render.html> (failed 2 times): 502 Bad Gateway 
2017-01-06 16:07:05 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET https://www.linkedin.com/company/netflix via http://localhost:8050/render.html> (failed 3 times): 502 Bad Gateway 
2017-01-06 16:07:05 [scrapy.core.engine] DEBUG: Crawled (502) <GET https://www.linkedin.com/company/netflix via http://localhost:8050/render.html> (referer: None) 
2017-01-06 16:07:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <502 https://www.linkedin.com/company/netflix>: HTTP status code is not handled or not allowed 
2017-01-06 16:07:05 [scrapy.core.engine] INFO: Closing spider (finished)

在飞溅终端：

2017-01-06 10:36:52.186410 [render] [139764812670456] loadFinished: RenderErrorInfo(type='HTTP', code=999, text='Request denied', url='https://www.linkedin.com/company/netflix') 
2017-01-06 10:36:52.205523 [events] {"fds": 18, "qsize": 0, "args": {"url": "https://www.linkedin.com/company/netflix", "headers": {"User-Agent": "Scrapy/1.3.0 (+http://scrapy.org)", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language": "en"}, "uid": 139764812670456, "wait": 0.5}, "rendertime": 6.674675464630127, "timestamp": 1483699012, "user-agent": "Scrapy/1.3.0 (+http://scrapy.org)", "maxrss": 87956, "error": {"info": {"url": "https://www.linkedin.com/company/netflix", "code": 999, "type": "HTTP", "text": "Request denied"}, "error": 502, "description": "Error rendering page", "type": "RenderError"}, "active": 0, "load": [0.51, 0.67, 0.8], "status_code": 502, "client_ip": "172.17.0.1", "method": "POST", "_id": 139764812670456, "path": "/render.html"} 
2017-01-06 10:36:52.206259 [-] "172.17.0.1" - - [06/Jan/2017:10:36:51 +0000] "POST /render.html HTTP/1.1" 502 192 "-" "Scrapy/1.3.0 (+http://scrapy.org)"

代码蜘蛛：

import scrapy 
from scrapy_splash import SplashRequest 
from linkedin.items import LinkedinItem 


class LinkedinScrapy(scrapy.Spider): 
    name = 'linkedin_spider' # spider name 
    allowed_domains = ['linkedin.com'] 
    start_urls = ['https://www.linkedin.com/company/netflix'] 

    def start_requests(self): 
     for url in self.start_urls: 
      yield SplashRequest(url, self.parse, 
          endpoint='render.html', args={'wait': 0.5) 

    def parse(self, response): 
     item = LinkedinItem() 
     item['name'] = response.xpath('//*[@id="stream-promo-top-bar"]/div[2]/div[1]/div[1]/div/h1/span/text()').extract_first() 
     item['followers'] = response.xpath('//*[@id = "biz-follow-mod"]/div/div/div/p/text()').extract_first().split()[0] 
     item['description'] = response.xpath('//*[@id="stream-about-section"]/div[2]/div[1]/div/p/text()').extract_first() 
       yield item

来源

2017-01-06 rd616

这可能是因为LinkedIn否认，因为用户代理字符串访问您的请求被使用：

"User-Agent": "Scrapy/1.3.0 (+http://scrapy.org)"

你应该在你的蜘蛛更改用户代理别的东西，看到mozillas documentation for that

来源

2017-01-06 10:50:55 Granitosaurus

那么，如果LinkedIn的T＆C'S说，它允许刮他们...哦，等一下事实并非如此。另外，如果他们阻止他进行刮擦，你会认为他们不允许这样做，而有人真的花时间阻止他的刮擦尝试。 – zerohero

@zerohero数据是公开的，因此条款和条件是irellavent。在离开这些评论之前，请教你自己网络欺诈的合法性。 – Granitosaurus

我认为你是需要教育的人。这些数据可能是公开的，但与您的条款相比，使用自动抓取实用程序是违背他们的条款的，因为它会导致无用的服务器资源在无法从中产生收入时使用。我一直在围绕着这个问题，并围绕它多次提出合法性问题。 – zerohero

502在使用Scrapy飞溅领英时发生错误

回答

相关问题