4
我有,我不能用机械化访问的网址,我不知道为什么:红宝石机械化404 =>网:: HTTPNotFound
# Use ruby 2.1.6
require 'mechanize'
require 'axlsx' # 2.0.1
require 'roo' # 1.13.2
mechanize = Mechanize.new
mechanize.request_headers = { "Accept-Encoding" => "" }
mechanize.ignore_bad_chunking = true
mechanize.follow_meta_refresh = true
xlsx = Roo::Excelx.new("./base_list.xlsx")
xlsx.each_with_pagename do |page, sheet|
sheet.each do |row|
page = mechanize.get(row[0])
end
end
当我重复我的名单上,我得到的网址,如: https://angel.co/_helencousins,我可以用我的浏览器,但不与机械化访问它,我有这样的错误:
/.rvm/gems/ruby-2.1.6/gems/mechanize-2.7.4/lib/mechanize/http/agent.rb:316:in `fetch': 404 => Net::HTTPNotFound for https://angel.co/_helencousins -- unhandled response (Mechanize::ResponseCodeError)
from /Users/xxx/.rvm/gems/ruby-2.1.6/gems/mechanize-2.7.4/lib/mechanize.rb:464:in `get'
from scraper.rb:15:in `block (2 levels) in <main>'
from /Users/xxx/.rvm/gems/ruby-2.1.6/gems/roo-1.13.2/lib/roo/base.rb:428:in `block in each'
from /Users/xxx/.rvm/gems/ruby-2.1.6/gems/roo-1.13.2/lib/roo/base.rb:427:in `upto'
from /Users/xxx/.rvm/gems/ruby-2.1.6/gems/roo-1.13.2/lib/roo/base.rb:427:in `each'
from scraper.rb:14:in `block in <main>'
from /Users/xxx/.rvm/gems/ruby-2.1.6/gems/roo-1.13.2/lib/roo/base.rb:398:in `block in each_with_pagename'
from /Users/xxx/.rvm/gems/ruby-2.1.6/gems/roo-1.13.2/lib/roo/base.rb:397:in `each'
from /Users/xxx/.rvm/gems/ruby-2.1.6/gems/roo-1.13.2/lib/roo/base.rb:397:in `each_with_pagename'
from scraper.rb:13:in `<main>'
可能的问题:http://stackoverflow.com/questions/8567973/why-does-accessing-a-ssl-site-with-mechanize-on-windows-fail但在mac工作 – 7stud
它们很好地嗅探请求的用户代理信息并拒绝任何不是标准浏览器的东西。首先,你应该看看他们是否有一个API,如果是的话,使用它。如果他们不寻找他们的服务条款,并看看他们是否允许机械化刮擦。如果是这样,请尝试更改您的用户代理字符串或联系他们的网站管理员并寻求帮助。 –