服务结构无状态标识服务器3服务在加载时失败

我们正在将基于标识服务器3（https://github.com/IdentityServer/IdentityServer3）的无状态服务迁移到服务结构。服务结构无状态标识服务器3服务在加载时失败

该项目在本地开发框和正常开发负载下的生产群集上运行良好，但当以约20-30个请求每秒投入生产时，它会迅速停止响应请求，并且ARR中配置的运行状况检查变得不健康。

该服务由执行SSL卸载的IIS ARR（应用程序请求路由）群集控制。

身份服务器日志输出以下两个错误，一个看起来与入站请求相关，另一个与出站请求有关，用于身份服务器持久性的azure存储。

Microsoft.WindowsAzure.Storage.StorageException: The client could not finish the operation within specified timeout. ---> System.TimeoutException: The client could not finish the operation within specified timeout. 
    --- End of inner exception stack trace --- 
    at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 50 
    at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass1`1.<CreateCallback>b__0(IAsyncResult ar) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\AsyncExtensions.cs:line 66 
--- End of stack trace from previous location where exception was thrown --- 
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) 
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) 
    at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() 
    at Collabco.Myday.Identity.IdSvr.BaseStore`1.<ExecuteQueryAsync>d__21.MoveNext() in C:\Dev\myday-identity\IdentityServer\IdSvr\BaseStore.cs:line 258 
--- End of stack trace from previous location where exception was thrown --- 
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) 
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) 
    at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() 
    at Collabco.Myday.Identity.IdSvr.ScopeStore.<GetScopesAsync>d__3.MoveNext() in C:\Dev\myday-identity\IdentityServer\IdSvr\ScopeStore.cs:line 43 
--- End of stack trace from previous location where exception was thrown --- 
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) 
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) 
    at IdentityServer3.Core.Endpoints.DiscoveryEndpointController.<GetConfiguration>d__11.MoveNext() in c:\local\identity\server3\IdentityServer3\source\Core\Endpoints\Connect\DiscoveryEndpointController.cs:line 73 
--- End of stack trace from previous location where exception was thrown --- 
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) 
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) 
    at System.Threading.Tasks.System.Web.Http910180.TaskHelpersExtensions.<CastToObject>d__3`1.MoveNext() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0 
--- End of stack trace from previous location where exception was thrown --- 
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) 
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) 
    at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__0.MoveNext() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0 
--- End of stack trace from previous location where exception was thrown --- 
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) 
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) 
    at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__2.MoveNext() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0 
--- End of stack trace from previous location where exception was thrown --- 
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) 
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) 
    at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__1.MoveNext() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0 
Request Information 
RequestID: 
RequestDate: 
StatusMessage:

，我们看到另一个例外是：

System.ObjectDisposedException: Cannot access a disposed object. 
Object name: 'System.Net.HttpListenerRequest'. 
    at System.Net.HttpListenerRequest.CheckDisposed() 
    at System.Net.HttpListenerRequest.get_LocalEndPoint() 
    at System.Net.HttpListenerRequest.get_IsLocal() 
    at Microsoft.Owin.Host.HttpListener.RequestProcessing.OwinHttpListenerContext.GetServerIsLocal() 
    at Microsoft.Owin.Host.HttpListener.RequestProcessing.CallEnvironment.get_ServerIsLocal() 
    at Microsoft.Owin.Host.HttpListener.RequestProcessing.CallEnvironment.PropertiesTryGetValue(String key, Object& value) 
    at Microsoft.Owin.Host.HttpListener.RequestProcessing.CallEnvironment.TryGetValue(String key, Object& value) 
    at Microsoft.Owin.OwinContext.Get[T](String key) in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0 
    at System.Web.Http.Owin.OwinHttpRequestContext.get_IsLocal() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0 
    at System.Web.Http.Owin.OwinHttpRequestContext.get_IncludeErrorDetail() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0 
    at System.Net.Http.System.Web.Http910180.HttpRequestMessageExtensions.CreateErrorResponse(HttpRequestMessage request, HttpStatusCode statusCode, Func`2 errorCreator) in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0 
    at System.Web.Http.ExceptionHandling.DefaultExceptionHandler.Handle(ExceptionHandlerContext context) in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0 
    at System.Web.Http.ExceptionHandling.DefaultExceptionHandler.HandleAsync(ExceptionHandlerContext context, CancellationToken cancellationToken) in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0 
    at System.Web.Http.ExceptionHandling.LastChanceExceptionHandler.HandleAsync(ExceptionHandlerContext context, CancellationToken cancellationToken) in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0 
    at System.Web.Http.ExceptionHandling.ExceptionHandlerExtensions.<HandleAsyncCore>d__0.MoveNext() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0 
--- End of stack trace from previous location where exception was thrown --- 
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) 
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) 
    at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__1.MoveNext() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0 
--- End of stack trace from previous location where exception was thrown --- 
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) 
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) 
    at System.Web.Http.Owin.PassiveAuthenticationMessageHandler.<SendAsync>d__0.MoveNext() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0 
--- End of stack trace from previous location where exception was thrown --- 
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) 
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) 
    at System.Web.Http.HttpServer.<SendAsync>d__0.MoveNext() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0

后约5-10分钟，没有任何回收除去负荷后通过服务起死回生。服务结构在不稳定时期也检测不到故障。

任何任何想法？

来源

2016-10-14 David Bown

'Microsoft.WindowsAzure.Storage.StorageException：客户端无法在指定的超时时间内完成操作。 ---> System.TimeoutException：客户端无法在指定的超时时间内完成操作。看起来您的Azure存储在处理请求方面速度很慢。顺便说一下，如果调用代码引发TimeoutException，则服务结构代理的默认重试策略是重试该调用10-11次。您是否处理Azure存储的异常？怎么样？ – cassandrad

@cassandrad我的问题是为什么这首先发生，因为我有完全相同的代码，禁止几个owin自主服务结构细节，在azure web应用程序上运行，没有存储超时记录。我会研究异常处理，也许这个特定的一个没有被处理。顺便说一下，我正在使用Azure存储客户端中的内置重试支持。我想知道是否有什么特定的服务结构我需要配置有关http请求或连接服务点？ –

我会看两件事情：从哪个节点调用即将到来的，我的意思是如果您有一个无状态的API网关，那么与对存储到从该网关执行的DB的服务的调用相比，该网关发生了多少次实际调用。在存储数据的服务中是否有例外，如果存在，那么哪一个例外。此外，我还会测量单个存储到天蓝色操作所需的平均时间，实际上是一次Insert方法的调用。没有什么具体的需要配置。 – cassandrad

解决这个是：

确保超时异常，正在捕捉和处理，并记录下来，在某些情况下，他们没有。
调整了重试策略以使重试更接近（延迟100ms）并且限制在不超过10秒。还实现了重试的日志记录。实际上我到目前为止没有重试。
增加了连接限制在桌子上的服务端点1000

最显著的因素似乎是3号虽然有些改进是通过实施1，我做到了第一这是一种明显的真指出。

为此，我在存储库类（身份服务器术语中的基本存储类）上使用了一个静态构造函数，该代码依赖于StorageAccount实例。除最后一行外，其他所有行都已存在

 var tableServicePoint = System.Net.ServicePointManager.FindServicePoint(storageAccount.TableEndpoint); 
     tableServicePoint.UseNagleAlgorithm = false; 
     tableServicePoint.Expect100Continue = false; 
     tableServicePoint.ConnectionLimit = 1000;

下面的文章帮助了这一点：https://azure.microsoft.com/en-gb/documentation/articles/storage-performance-checklist/

综上所述，我的结论是默认的连接限制（2个或10个不同的文件冲突）是导致请求表存储排队，并最终超时并最终导致服务崩溃。

使用azure网络应用程序时，要确认设置连接限制是不必要的，因此服务结构/ owin自主主机项目在使用影响天蓝色存储的连接方面有所不同。

来源

2016-10-19 08:33:19

服务结构无状态标识服务器3服务在加载时失败

回答

相关问题