随机超时使用heroku + gunicorn + aiohttp的错误

我开发了一个服务，充当网关，将请求重定向到不同的微服务。为此，我使用aiohttp来处理重定向请求，gunicorn（w/aiohttp.worker.GunicornWebWorker）服务，Heroku作为主机。随机超时使用heroku + gunicorn + aiohttp的错误

在本地工作一切都很完美，100％的请求返回响应，客户端总是收到所需的信息，但是当我部署到Heroku并重定向一些请求时（每分钟5k），我发现有3到7个HTTP状态请求503超时错误。没什么好担心的，因为好的解决请求比例非常高（99.9994），但我想知道发生了什么。该异常引发只是超时之前是这样的：

[2017-02-10 17:03:48 +0000] [683] [INFO] Worker exiting (pid: 683) 
ERROR:asyncio:Task was destroyed but it is pending! 
[2017-02-10 17:03:48 +0000] [683] [INFO] Stopping server: 683, connections: 1 
Exception ignored in: <generator object GunicornWebWorker._run at 0x7f18b1d2f518> 
Traceback (most recent call last): 
    yield from self.close() 
    yield from self.wsgi.shutdown() 
File "/app/.heroku/python/lib/python3.5/site-packages/aiohttp/web.py", line 199, in shutdown 
    yield from self.on_shutdown.send(self) 
File "/app/.heroku/python/lib/python3.5/site-packages/aiohttp/signals.py", line 48, in send 
    yield from self._send(*args, **kwargs) 
File "/app/.heroku/python/lib/python3.5/site-packages/aiohttp/signals.py", line 16, in _send 
    yield from res 
File "/app/app/app.py", line 14, in close_redis 
    app.redis_pool.close() 
File "/app/.heroku/python/lib/python3.5/site-packages/aioredis/pool.py", line 135, in close 
    self._close_state.set() 
File "/app/.heroku/python/lib/python3.5/asyncio/locks.py", line 242, in set 
    fut.set_result(True) 
File "/app/.heroku/python/lib/python3.5/asyncio/futures.py", line 332, in set_result 
    self._schedule_callbacks() 
File "/app/.heroku/python/lib/python3.5/asyncio/futures.py", line 242, in _schedule_callbacks 
    self._loop.call_soon(callback, self) 
File "/app/.heroku/python/lib/python3.5/asyncio/base_events.py", line 497, in call_soon 
    handle = self._call_soon(callback, args) 
File "/app/.heroku/python/lib/python3.5/asyncio/base_events.py", line 506, in _call_soon 
    self._check_closed() 
File "/app/.heroku/python/lib/python3.5/asyncio/base_events.py", line 334, in _check_closed 
    raise RuntimeError('Event loop is closed') 
RuntimeError: Event loop is closed 
ERROR:asyncio:Task was destroyed but it is pending! 
task: <Task pending coro=<ServerHttpProtocol.start() running at /app/.heroku/python/lib/python3.5/site-packages/aiohttp/server.py:261>> 
[2017-02-10 17:03:48 +0000] [4] [CRITICAL] WORKER TIMEOUT (pid:683)

随后的Heroku /路由器显示这样的错误：

at=error code=H12 desc="Request timeout" method=GET path="https://stackoverflow.com/users/21324/posts/" host=superapp.herokuapp.com request_id=209bd839-baac-4e72-a04e-657d85348f45 fwd="84.78.56.97" dyno=web.2 connect=0ms service=30000ms status=503 bytes=0

我运行的应用程序有：

gunicorn --pythonpath app app.app:aio_app --worker-class aiohttp.worker.GunicornWebWorker --workers 3

主要代码是：

def init(asyncio_loop): 
    app = web.Application(loop=asyncio_loop, middlewares=[middlewares.auth_middleware, 
                  middlewares.logging_middleware]) 

    # INIT DBs 
    app.redis_pool = asyncio_loop.run_until_complete(
     create_pool((settings.REDIS['host'], settings.REDIS['port']), 
        password=settings.REDIS['password'])) 

    # Clean connections on stop 
    app.on_shutdown.append(close_redis) 

    # Add rollbar 
    rollbar.init(settings.ROLLBAR_TOKEN, 'production') # access_token, environment 

    # Bind routes 
    for r in routes: 
     app.router.add_route(r[0], r[1], r[2]) 

    return app 


# Create app 
loop = asyncio.get_event_loop() 
aio_app = init(loop)

以及重定向例如：

async with aiohttp.ClientSession() as s: 
    try: 
     async with s.request(method=method, 
          url=new_url, 
          headers=new_headers, 
          data=body, 
          allow_redirects=False, 
          timeout=25) as response: 
      # Clean response 
      resp_headers = MSRepository.filter_response_headers(response.headers) 
      resp_body = (await response.read()) 

      return ResponseDataEntity(resp_headers, response.status, resp_body) 
    except asyncio.TimeoutError: 
     raise MSConnectionError("Request timeout") 
    except Exception as e: 
     rollbar.report_message(str(e), extra_data={ 
      "url": new_url, 
      "data": body, 
      "method": method, 
      "headers": new_headers 
     }) 
     raise MSConnectionError(str(e))

正如你可以看到有25秒的超时发出请求和异常与超时30秒时提高。

任何人都有任何线索发生了什么？

（注：当我写改向我的意思并不是说HTTP 302我的意思是处理该请求，编辑标题，检查权威性，使异步请求中选取适当的MS，处理响应和返回该响应）

来源

2017-02-10 Jairo Vadillo

最后问题出现在一个处理程序中。我不知道发生了什么，因为超时对于所有端点是完全随机的，但是在6小时完成超过10k个请求后，我确信问题是这样。下面是代码之前和修复程序后：

async def bad_handler(request): 
    # Read body in ALL cases to not to block requests 
    if '/event-log/action/' == request.path: 
     if request.query_string != '': 
      action_type = request.query_string 
     else: 
      try: 
       request_body = await request.json() 
       action_type = request_body.get('type', '') 
      except: 
       action_type = '' 

     print("Action_type={}".format(action_type)) 

    # Response to client 
    return aiohttp.web.json_response(data={}, status=200) 

async def good_handler(request): 
    # Read body in ALL cases to not to block requests 
    try: 
     request_body = await request.json() 
    except: 
     request_body = None 

    if '/event-log/action/' == request.path: 
     if request.query_string != '': 
      action_type = request.query_string 
     else: 
      if request_body is not None: 
       action_type = request_body.get('type', '') 
      else: 
       action_type = '' 

     print("Action_type={}".format(action_type)) 

    # Response to client 
    return aiohttp.web.json_response(data={}, status=200)

正如你看到的，唯一的区别是，在一个情况下，我们一直在等待身体和在其他情况下没有。

我将离开这个问题打开只是跳一个人回答我为什么现在工作。 :)

来源

2017-02-23 16:47:37

随机超时使用heroku + gunicorn + aiohttp的错误

回答

相关问题