Python龙卷风httplib重试

Question

我使用tornado异常地使用HTTP代理来获取许多网页。所以，我的许多feth'es完成与错误（我的代理是不可靠的）。我想用另一个代理立即重试它们。下面是例子：

from tornado import ioloop 
from tornado import httpclient 

def handle_request(response): 
    if response.error: 
     print "Error:", response.error 
     // HERE i want to put my retry with another proxy 
    else: 
     print response.body 
    ioloop.IOLoop.instance().stop() 

http_client = httpclient.AsyncHTTPClient() 
http_client.fetch("http://www.google.com/", handle_request) 
ioloop.IOLoop.instance().start() 

但如何我可以我添加新的handle_request电流环取事件？另外，我如何将变量传递到handle_request（列出所有我的代理）。

Answer 1

你问两个问题 -

我会考虑使用谐音http://docs.python.org/library/functools.html#partial-objects

from functools import partial 

PROXIES = [A, B, C, D] # As appropriate 
... 
def handle_request(proxies, response): 
    if ...BAD RESPONSE...: 
     return http_client.fetch(response.request.url, partial(handle_request, proxies[1:])) 
    # Now handle the case that you have a good result or you're out of proxies 

http_client.fetch("http://www.google.com/", partial(handle_request, PROXIES[:])) 

当然的另一种选择是让一个对象。

class ProxyRequest(object): 
    PROXIES = [A, B, C] 

    def __init__(self, url): 
      self.url = url 
      self.proxies = self.PROXIES[:] 
      self.fetch() 

    def fetch(self): 
      p, self.proxies = self.proxies[0], self.proxies[1:] 

      http_client.fetch(self.url, self.handle, proxy=p) 

    def handle(self, response): 
      if response.error: 
       if self.proxies: 
        return self.fetch() 
       else: 
        ...error case... 

      ...stop the ioloop if you want...