2016-05-03 76 views
0

我最近将ES从版本1.1.0升级到1.6。所有CONFIGS保持不变,但通话的近5%开始呕吐EsRejectedExecutionException当我运行ES 1.1.0弹性搜索从1.1.0升级到1.6引发异常EsRejectedExecutionException

这从来没有发生过完整的堆栈跟踪低于:

org.elasticsearch.action.search.ReduceSearchPhaseException: Failed to execute phase [merge], [reduce] 
at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction$1.onFailure(TransportSearchQueryAndFetchAction.java:93) 
at org.elasticsearch.common.util.concurrent.AbstractRunnable.onRejection(AbstractRunnable.java:65) 
at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:85) 
at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.moveToSecondPhase(TransportSearchQueryAndFetchAction.java:78) 
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.innerMoveToSecondPhase(TransportSearchTypeAction.java:403) 
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:202) 
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onResult(TransportSearchTypeAction.java:178) 
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onResult(TransportSearchTypeAction.java:175) 
at org.elasticsearch.search.action.SearchServiceTransportAction$12.handleResponse(SearchServiceTransportAction.java:346) 
at org.elasticsearch.search.action.SearchServiceTransportAction$12.handleResponse(SearchServiceTransportAction.java:337) 
at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:163) 
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:132) 
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) 
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) 
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) 
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296) 
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) 
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) 
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) 
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) 
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) 
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) 
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268) 
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255) 
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) 
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) 
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) 
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) 
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) 
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) 
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
at java.lang.Thread.run(Thread.java:745) 
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 500) on org.elasticsearch.acti[email protected]4cd69494 
at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62) 
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) 
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372) 
at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:79) 

所有配置保持不变,而升级,所以我想知道可能是什么原因造成的问题。线程池大小似乎已满,所以我试图通过增加线程池和Queue_size来进行测试。但我无法理解是什么从1.1到1.6导致了这个问题。

我阅读发行说明并没有发现任何与此相关的内容。我正在寻找任何配置参数的默认值更改,但我没有找到任何看起来相关的东西。

我发现this问题,这是非常相似,我的问题,但是这是在1.1.2版本解决了,所以这不应该导致错误。

如果有人能指导我如何寻找问题,这将是一个很大的帮助。

+0

你有多少个节点,每个节点有多少个CPU? – Val

+0

@Val我有2个群集,每个3个节点。每个节点有32个CPU。使用相同的配置,我从来没有收到1.1例外,但有1.6 – Atri

+0

错误我读了http://stackoverflow.com/questions/27793530/esrejectedexecutionexception-in-elasticsearch-for-parallel-search,这将缓解这个问题,但没有解决它。我认为它与2群集配置有关。 – Atri

回答

0

我升级到1.7.5,一切正常。我想我的映射和查询的设计方式在ES 1.6.0中存在一些问题。