# Unresponsive ES2.3 cluster with SearchGuard 2.3 installed

**URL:** <https://forum.search-guard.com/t/unresponsive-es2-3-cluster-with-searchguard-2-3-installed/976>\
**Category:** Search Guard\
**Created:** [June 26, 2018, 7:09pm UTC](https://forum.search-guard.com/t/unresponsive-es2-3-cluster-with-searchguard-2-3-installed/976 "2018-06-26T19:09:56Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sai\_Hariharan](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@Sai\_Hariharan](https://forum.search-guard.com/u/Sai_Hariharan)\
**Post date:** [June 26, 2018, 7:09pm UTC](https://forum.search-guard.com/t/unresponsive-es2-3-cluster-with-searchguard-2-3-installed/976/1 "2018-06-26T19:09:56Z")

</div>

Hi there

We had an unresponsive Elasticsearch cluster (5 node host-[06-10] , with 32GB of heap space assigned to each host). Upon checking cluster against each host only one of it (host-08) would respond after a while - Cluster state as Yelllow and having only 4 hosts. The rest would either timeout or return SSL error.

Also, found that most hosts have reached a heap space (\_cat/nodes) of almost 99%

Upon checking the logs we found the exceptions listed in the log messages section below.

I checked the validity of the certificates in JKS and they are good till Aug 20th 2018.

Eventually I had restart the cluster to resolve this and it took almost 5-6 hours with ~330GB of index size. The cluster is now reen and healthy. However, I still see the below error messages in the log

[2018-06-26 15:49:17,539] [ERROR][com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] [host-10] SSL Problem null cert chain  
javax.net.ssl.SSLHandshakeException: null cert chain

Any thoughts on what could’ve gone wrong, please let me know your views.

- Search Guard and Elasticsearch version

Elasticsearch version - 2.3.3

Search Guard version -

“component”: “search-guard-2”,  
“version”: “2.3.3.1”,

“component”: “search-guard-ssl”,  
“version”: “2.3.3.13”,

- JVM version and operating system version

java version “1.7.0\_51”

OS : Oracle Linux 6.9

- Search Guard configuration files

searchguard.ssl.transport.enabled: true  
searchguard.ssl.transport.keystore\_type: JKS  
searchguard.ssl.transport.keystore\_filepath: jks1/server.jks  
searchguard.ssl.transport.keystore\_password: \<\>  
searchguard.ssl.transport.truststore\_type: JKS  
searchguard.ssl.transport.truststore\_filepath: jks1/server.jks  
searchguard.ssl.transport.truststore\_password: \<\>  
searchguard.ssl.http.enabled: true  
searchguard.ssl.http.keystore\_type: JKS  
searchguard.ssl.http.keystore\_filepath: jks1/server.jks  
searchguard.ssl.http.keystore\_password: \<\>  
searchguard.ssl.http.truststore\_type: JKS  
searchguard.ssl.http.truststore\_filepath: jks1/server.jks  
searchguard.ssl.http.truststore\_password: \<\>  
searchguard.ssl.http.clientauth\_mode: NONE

searchguard.enabled: true  
security.manager.enabled: false

searchguard.authcz.admin\_dn:  
- “C=US,ST=California,O=A\* Inc.,OU=management:group.752639,[CN=rody.com](http://CN=rody.com)”  
- “C=US,ST=California,O=A\* Inc.,OU=management:group.752639,[CN=pwk-rody.com](http://CN=pwk-rody.com)”  
- “C=US,ST=California,O=A\* Inc.,OU=management:group.752639,[CN=pwk-f5p-lnn01.com](http://CN=pwk-f5p-lnn01.com)”

searchguard.authcz.impersonation\_dn:  
“C=US,ST=California,O=A\* Inc.,OU=management:group.752639,[CN=rody.com](http://CN=rody.com)”:  
- '_’  
searchguard.authcz.impersonation\_dn:  
"C=US,ST=California,O=A_ Inc.,OU=management:group.752639,[CN=pwk.rody.com](http://CN=pwk.rody.com)":  
- '_’  
searchguard.authcz.impersonation\_dn:  
"C=US,ST=California,O=A_ Inc.,OU=management:group.752639,CN=pwk-f5p-lnn01…com":

- Elasticsearch log messages on debug level

[2018-06-19 18:56:30,600][WARN][netty.handler.ssl.SslHandler] Unexpected leftover data after SSLEngine.unwrap(): status=OK handshakeStatus=NEED\_WRAP consumed=0 produced=0 remaining=7 data=15030300020100

//Several instances of GC cycles  
[2018-06-19 18:56:30,593][WARN][monitor.jvm] [host-07] [gc][old][1374850][5979] duration [2.1m], collections [1]/[2.1m], total [2.1m]/[4.6d], memory [31.7gb]-\>[31.7gb]/[31.7gb], all\_pools {[young] [1.8gb]-\>[1.8gb]/[1.8gb]}{[survivor] [230mb]-\>[230.1mb]/[232.9mb]}{[old] [29.7gb]-\>[29.7gb]/[29.7gb]}

[2018-06-19 18:56:30,589][DEBUG][action.search] [host-07] [14368933] Failed to execute query phase  
RemoteTransportException[[host-10][17.160.69.238:9300][indices:data/read/search[phase/query/id]]]; nested: SearchContextMissingException[No search context found for id [14368933]];  
Caused by: SearchContextMissingException[No search context found for id [14368933]]  
at org.elasticsearch.search.SearchService.findContext(SearchService.java:613 )  
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:425 )  
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:376 )  
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:373 )  
at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33 )  
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLTransportService.messageReceivedDecorate(SearchGuardSSLTransportService.java:161 )  
at com.floragunn.searchguard.transport.SearchGuardTransportService.messageReceivedDecorate(SearchGuardTransportService.java:293 )  
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLTransportService$Interceptor.messageReceived(SearchGuardSSLTransportService.java:128 )  
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75 )  
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:300 )  
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37 )  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145 )  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615 )  
at java.lang.Thread.run(Thread.java:744)

[2018-06-19 19:59:16,722][WARN][discovery.zen.ping.unicast] [host-07] failed to send ping to [{#zen\_unicast\_3#}{17.160.69.236}{host-08/17.99.69.266:9300}]  
ReceiveTimeoutTransportException[[host-08m/17.99.69.266:9300][internal:discovery/zen/unicast] request\_id [80874756] timed out after [124435ms]]  
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:679 )  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145 )  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615 )  
at java.lang.Thread.run(Thread.java:744)

[2018-06-19 20:17:07,562][WARN][com.floragunn.searchguard.transport.SearchGuardTransportService] [host-07] Received response for a request that has timed out, sent [1195273ms] ago, timed out [1070838ms] ago, action [internal:discovery/zen/unicast], node [{#zen\_unicast\_3#}{17.160.69.236}{[ma2-rodyp-lcb08.corp.apple.com/17.160.69.236:9300](http://ma2-rodyp-lcb08.corp.apple.com/17.160.69.236:9300)}], id [80874756]

[2018-06-20 03:07:58,459][WARN][com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] [host-07] exception caught on transport layer [[id: 0x0f3baf15, /17.96.16.84:48738 =\> /17.160.69.235:9300]], closing connection  
java.lang.OutOfMemoryError: Java heap space  
at sun.security.ssl.EngineInputRecord.compareMacTags(EngineInputRecord.java:330 )  
at sun.security.ssl.EngineInputRecord.checkMacTags(EngineInputRecord.java:313 )  
at sun.security.ssl.EngineInputRecord.decrypt(EngineInputRecord.java:244)

[2018-06-20 12:08:05,960][ERROR][com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] [host-08] SSL Problem Received close\_notify during handshake  
javax.net.ssl.SSLException: Received close\_notify during handshake  
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208 )  
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1619 )  
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1587 )  
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1732 )  
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1060 )  
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:884 )  
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:758 )  
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624 )  
at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1218 )  
at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:852 )  
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425 )  
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303 )  
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70 )  
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564 )  
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559 )  
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268 )  
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255 )  
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88 )  
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108 )  
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337 )  
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89 )  
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178 )  
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108 )  
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42 )  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145 )  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615 )  
at java.lang.Thread.run(Thread.java:744)

[2018-06-20 15:34:35,383][ERROR][com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] [host-08] SSL Problem null cert chain  
javax.net.ssl.SSLHandshakeException: null cert chain  
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1290 )  
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:513 )  
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:790 )  
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:758 )  
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624 )  
at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1218 )  
at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:852 )  
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425 )  
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303 )  
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70 )  
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564 )  
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559 )  
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268 )  
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255 )  
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88 )  
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108 )  
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337 )  
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89 )  
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178 )  
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108 )  
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42 )  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145 )  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615 )  
at java.lang.Thread.run(Thread.java:744)  
Caused by: javax.net.ssl.SSLHandshakeException: null cert chain  
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192 )  
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1619 )  
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:278 )  
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:266 )  
at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1631 )  
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:176 )  
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868 )  
at sun.security.ssl.Handshaker$1.run(Handshaker.java:808 )  
at sun.security.ssl.Handshaker$1.run(Handshaker.java:806 )  
at java.security.AccessController.doPrivileged(Native Method )  
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1227 )  
at org.jboss.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1392 )  
at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1255 )  
… 18 more

[2018-06-21 21:29:00,075][WARN][gateway] [host-08] [searchguard][0]: failed to list shard for shard\_store on node [kxb4QDvVR8eNNwOoPT-uVQ]  
FailedNodeException[total failure in fetching]; nested: ElasticsearchException[unauthenticated request internal:cluster/nodes/indices/shard/store for user User [name=\_sg\_internal, roles=]];  
at org.elasticsearch.gateway.AsyncShardFetch$1.onFailure(AsyncShardFetch.java:277 )  
at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:95 )  
at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:135 )  
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170 )  
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:144 )  
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:85 )  
at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.list(TransportNodesListShardStoreMetaData.java:88 )  
at org.elasticsearch.gateway.AsyncShardFetch.asyncFetch(AsyncShardFetch.java:267 )  
at org.elasticsearch.gateway.AsyncShardFetch.fetchData(AsyncShardFetch.java:117 )  
at org.elasticsearch.gateway.GatewayAllocator$InternalReplicaShardAllocator.fetchData(GatewayAllocator.java:183 )  
at org.elasticsearch.gateway.ReplicaShardAllocator.allocateUnassigned(ReplicaShardAllocator.java:137 )  
at org.elasticsearch.gateway.GatewayAllocator.allocateUnassigned(GatewayAllocator.java:123 )  
at org.elasticsearch.cluster.routing.allocation.allocator.ShardsAllocators.allocateUnassigned(ShardsAllocators.java:70 )  
at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:258 )  
at org.elasticsearch.cluster.routing.allocation.AllocationService.applyStartedShards(AllocationService.java:86 )  
at org.elasticsearch.cluster.action.shard.ShardStateAction$ShardStartedClusterStateHandler.execute(ShardStateAction.java:218 )  
at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:468 )  
at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772 )  
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231 )  
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194 )  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145 )  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615 )  
at java.lang.Thread.run(Thread.java:744)  
Caused by: ElasticsearchException[unauthenticated request internal:cluster/nodes/indices/shard/store for user User [name=\_sg\_internal, roles=]]  
… 21 more

- Other installed Elasticsearch or Kibana plugins, if any

delete-by-query - 2.3.3

head

---

<div class="post-metadata">

**Author:** ![Sai\_Hariharan](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@Sai\_Hariharan](https://forum.search-guard.com/u/Sai_Hariharan)\
**Post date:** [June 29, 2018, 6:57pm UTC](https://forum.search-guard.com/t/unresponsive-es2-3-cluster-with-searchguard-2-3-installed/976/2 "2018-06-29T18:57:50Z")

</div>

Hi

Has anyone faced a similar issue as illustrated below? I know its a pretty old version that we have and we have also been pushing for an upgrade without much success. Until then we need to keep this up and running.

Let me know if you need any further information on this.

Regards,

Hari

> **···**
>
> On Tuesday, June 26, 2018 at 12:09:56 PM UTC-7, ark.202 wrote:
> 
> > Hi there
> 
> > 
> 
> > 
> 
> > We had an unresponsive Elasticsearch cluster (5 node host-[06-10] , with 32GB of heap space assigned to each host). Upon checking cluster against each host only one of it (host-08) would respond after a while - Cluster state as Yelllow and having only 4 hosts. The rest would either timeout or return SSL error.
> 
> > 
> 
> > Also, found that most hosts have reached a heap space (\_cat/nodes) of almost 99%
> 
> > 
> 
> > Upon checking the logs we found the exceptions listed in the log messages section below.
> 
> > 
> 
> > I checked the validity of the certificates in JKS and they are good till Aug 20th 2018.
> 
> > 
> 
> > Eventually I had restart the cluster to resolve this and it took almost 5-6 hours with ~330GB of index size. The cluster is now reen and healthy. However, I still see the below error messages in the log
> 
> > 
> 
> > [2018-06-26 15:49:17,539] [ERROR][com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] [host-10] SSL Problem null cert chain  
> > javax.net.ssl.SSLHandshakeException: null cert chain
> 
> > 
> 
> > Any thoughts on what could’ve gone wrong, please let me know your views.
> 
> > 
> 
> > 
> 
> > 
> 
> > - Search Guard and Elasticsearch version
> 
> > Elasticsearch version - 2.3.3
> 
> > Search Guard version -
> 
> > “component”: “search-guard-2”,  
> > “version”: “2.3.3.1”,
> 
> > “component”: “search-guard-ssl”,  
> > “version”: “2.3.3.13”,
> 
> > 
> 
> > 
> 
> > - JVM version and operating system version
> 
> > java version “1.7.0\_51”
> 
> > OS : Oracle Linux 6.9
> 
> > 
> 
> > 
> 
> > 
> 
> > - Search Guard configuration files
> 
> > 
> 
> > 
> 
> > searchguard.ssl.transport.enabled: true  
> > searchguard.ssl.transport.keystore\_type: JKS  
> > searchguard.ssl.transport.keystore\_filepath: jks1/server.jks  
> > searchguard.ssl.transport.keystore\_password: \<\>  
> > searchguard.ssl.transport.truststore\_type: JKS  
> > searchguard.ssl.transport.truststore\_filepath: jks1/server.jks  
> > searchguard.ssl.transport.truststore\_password: \<\>  
> > searchguard.ssl.http.enabled: true  
> > searchguard.ssl.http.keystore\_type: JKS  
> > searchguard.ssl.http.keystore\_filepath: jks1/server.jks  
> > searchguard.ssl.http.keystore\_password: \<\>  
> > searchguard.ssl.http.truststore\_type: JKS  
> > searchguard.ssl.http.truststore\_filepath: jks1/server.jks  
> > searchguard.ssl.http.truststore\_password: \<\>  
> > searchguard.ssl.http.clientauth\_mode: NONE
> > 
> > searchguard.enabled: true  
> > security.manager.enabled: false
> 
> > 
> 
> > 
> 
> > 
> 
> > searchguard.authcz.admin\_dn:  
> > - “C=US,ST=California,O=A\* Inc.,OU=management:group.752639,CN=[rody.com](http://rody.com)”  
> > - “C=US,ST=California,O=A\* Inc.,OU=management:group.752639,CN=[pwk-rody.com](http://pwk-rody.com)”  
> > - “C=US,ST=California,O=A\* Inc.,OU=management:group.752639,CN=[pwk-f5p-lnn01.com](http://pwk-f5p-lnn01.com)”
> > 
> > searchguard.authcz.impersonation\_dn:  
> > “C=US,ST=California,O=A\* Inc.,OU=management:group.752639,CN=[rody.com](http://rody.com)”:  
> > - '_’  
> > searchguard.authcz.impersonation\_dn:  
> > "C=US,ST=California,O=A_ Inc.,OU=management:group.752639,CN=[pwk.rody.com](http://pwk.rody.com)":  
> > - '_’  
> > searchguard.authcz.impersonation\_dn:  
> > "C=US,ST=California,O=A_ Inc.,OU=management:group.752639,CN=pwk-f5p-lnn01…com":
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > - Elasticsearch log messages on debug level
> 
> > 
> 
> > 
> 
> > 
> 
> > [2018-06-19 18:56:30,600][WARN][netty.handler.ssl.SslHandler] Unexpected leftover data after SSLEngine.unwrap(): status=OK handshakeStatus=NEED\_WRAP consumed=0 produced=0 remaining=7 data=15030300020100
> 
> > 
> 
> > 
> 
> > //Several instances of GC cycles  
> > [2018-06-19 18:56:30,593][WARN][monitor.jvm] [host-07] [gc][old][1374850][5979] duration [2.1m], collections [1]/[2.1m], total [2.1m]/[4.6d], memory [31.7gb]-\>[31.7gb]/[31.7gb], all\_pools {[young] [1.8gb]-\>[1.8gb]/[1.8gb]}{[survivor] [230mb]-\>[230.1mb]/[232.9mb]}{[old] [29.7gb]-\>[29.7gb]/[29.7gb]}
> 
> > 
> 
> > 
> 
> > [2018-06-19 18:56:30,589][DEBUG][action.search] [host-07] [14368933] Failed to execute query phase  
> > RemoteTransportException[[host-10][17.160.69.238:9300][indices:data/read/search[phase/query/id]]]; nested: SearchContextMissingException[No search context found for id [14368933]];  
> > Caused by: SearchContextMissingException[No search context found for id [14368933]]  
> > at org.elasticsearch.search.SearchService.findContext(SearchService.java:613 )  
> > at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:425 )  
> > at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:376 )  
> > at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:373 )  
> > at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33 )  
> > at com.floragunn.searchguard.ssl.transport.SearchGuardSSLTransportService.messageReceivedDecorate(SearchGuardSSLTransportService.java:161 )  
> > at com.floragunn.searchguard.transport.SearchGuardTransportService.messageReceivedDecorate(SearchGuardTransportService.java:293 )  
> > at com.floragunn.searchguard.ssl.transport.SearchGuardSSLTransportService$Interceptor.messageReceived(SearchGuardSSLTransportService.java:128 )  
> > at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75 )  
> > at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:300 )  
> > at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37 )  
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145 )  
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615 )  
> > at java.lang.Thread.run(Thread.java:744)
> 
> > 
> 
> > 
> 
> > 
> 
> > [2018-06-19 19:59:16,722][WARN][discovery.zen.ping.unicast] [host-07] failed to send ping to [{#zen\_unicast\_3#}{17.160.69.236}{host-08/17.99.69.266:9300}]  
> > ReceiveTimeoutTransportException[[host-08m/17.99.69.266:9300][internal:discovery/zen/unicast] request\_id [80874756] timed out after [124435ms]]  
> > at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:679 )  
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145 )  
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615 )  
> > at java.lang.Thread.run(Thread.java:744)
> 
> > 
> 
> > [2018-06-19 20:17:07,562][WARN][com.floragunn.searchguard.transport.SearchGuardTransportService] [host-07] Received response for a request that has timed out, sent [1195273ms] ago, timed out [1070838ms] ago, action [internal:discovery/zen/unicast], node [{#zen\_unicast\_3#}{17.160.69.236}{[ma2-rodyp-lcb08.corp.apple.com/17.160.69.236:9300](http://ma2-rodyp-lcb08.corp.apple.com/17.160.69.236:9300)}], id [80874756]
> 
> > 
> 
> > [2018-06-20 03:07:58,459][WARN][com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] [host-07] exception caught on transport layer [[id: 0x0f3baf15, /17.96.16.84:48738 =\> /17.160.69.235:9300]], closing connection  
> > java.lang.OutOfMemoryError: Java heap space  
> > at sun.security.ssl.EngineInputRecord.compareMacTags(EngineInputRecord.java:330 )  
> > at sun.security.ssl.EngineInputRecord.checkMacTags(EngineInputRecord.java:313 )  
> > at sun.security.ssl.EngineInputRecord.decrypt(EngineInputRecord.java:244)
> 
> > 
> 
> > 
> 
> > [2018-06-20 12:08:05,960][ERROR][com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] [host-08] SSL Problem Received close\_notify during handshake  
> > javax.net.ssl.SSLException: Received close\_notify during handshake  
> > at sun.security.ssl.Alerts.getSSLException(Alerts.java:208 )  
> > at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1619 )  
> > at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1587 )  
> > at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1732 )  
> > at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1060 )  
> > at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:884 )  
> > at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:758 )  
> > at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624 )  
> > at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1218 )  
> > at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:852 )  
> > at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425 )  
> > at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303 )  
> > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70 )  
> > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564 )  
> > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559 )  
> > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268 )  
> > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255 )  
> > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88 )  
> > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108 )  
> > at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337 )  
> > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89 )  
> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178 )  
> > at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108 )  
> > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42 )  
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145 )  
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615 )  
> > at java.lang.Thread.run(Thread.java:744)
> 
> > 
> 
> > [2018-06-20 15:34:35,383][ERROR][com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] [host-08] SSL Problem null cert chain  
> > javax.net.ssl.SSLHandshakeException: null cert chain  
> > at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1290 )  
> > at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:513 )  
> > at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:790 )  
> > at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:758 )  
> > at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624 )  
> > at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1218 )  
> > at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:852 )  
> > at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425 )  
> > at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303 )  
> > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70 )  
> > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564 )  
> > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559 )  
> > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268 )  
> > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255 )  
> > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88 )  
> > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108 )  
> > at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337 )  
> > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89 )  
> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178 )  
> > at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108 )  
> > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42 )  
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145 )  
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615 )  
> > at java.lang.Thread.run(Thread.java:744)  
> > Caused by: javax.net.ssl.SSLHandshakeException: null cert chain  
> > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192 )  
> > at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1619 )  
> > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:278 )  
> > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:266 )  
> > at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1631 )  
> > at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:176 )  
> > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868 )  
> > at sun.security.ssl.Handshaker$1.run(Handshaker.java:808 )  
> > at sun.security.ssl.Handshaker$1.run(Handshaker.java:806 )  
> > at java.security.AccessController.doPrivileged(Native Method )  
> > at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1227 )  
> > at org.jboss.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1392 )  
> > at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1255 )  
> > … 18 more
> 
> > 
> 
> > 
> 
> > 
> 
> > [2018-06-21 21:29:00,075][WARN][gateway] [host-08] [searchguard][0]: failed to list shard for shard\_store on node [kxb4QDvVR8eNNwOoPT-uVQ]  
> > FailedNodeException[total failure in fetching]; nested: ElasticsearchException[unauthenticated request internal:cluster/nodes/indices/shard/store for user User [name=\_sg\_internal, roles=]];  
> > at org.elasticsearch.gateway.AsyncShardFetch$1.onFailure(AsyncShardFetch.java:277 )  
> > at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:95 )  
> > at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:135 )  
> > at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170 )  
> > at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:144 )  
> > at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:85 )  
> > at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.list(TransportNodesListShardStoreMetaData.java:88 )  
> > at org.elasticsearch.gateway.AsyncShardFetch.asyncFetch(AsyncShardFetch.java:267 )  
> > at org.elasticsearch.gateway.AsyncShardFetch.fetchData(AsyncShardFetch.java:117 )  
> > at org.elasticsearch.gateway.GatewayAllocator$InternalReplicaShardAllocator.fetchData(GatewayAllocator.java:183 )  
> > at org.elasticsearch.gateway.ReplicaShardAllocator.allocateUnassigned(ReplicaShardAllocator.java:137 )  
> > at org.elasticsearch.gateway.GatewayAllocator.allocateUnassigned(GatewayAllocator.java:123 )  
> > at org.elasticsearch.cluster.routing.allocation.allocator.ShardsAllocators.allocateUnassigned(ShardsAllocators.java:70 )  
> > at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:258 )  
> > at org.elasticsearch.cluster.routing.allocation.AllocationService.applyStartedShards(AllocationService.java:86 )  
> > at org.elasticsearch.cluster.action.shard.ShardStateAction$ShardStartedClusterStateHandler.execute(ShardStateAction.java:218 )  
> > at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:468 )  
> > at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772 )  
> > at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231 )  
> > at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194 )  
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145 )  
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615 )  
> > at java.lang.Thread.run(Thread.java:744)  
> > Caused by: ElasticsearchException[unauthenticated request internal:cluster/nodes/indices/shard/store for user User [name=\_sg\_internal, roles=]]  
> > … 21 more
> 
> > 
> 
> > 
> 
> > 
> 
> > - Other installed Elasticsearch or Kibana plugins, if any
> 
> > 
> 
> > delete-by-query - 2.3.3
> 
> > head

---

<div class="post-metadata">

**Author:** ![jkressin](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/jkressin/32/9_2.png) [@jkressin](https://forum.search-guard.com/u/jkressin)\
**Post date:** [July 1, 2018, 10:56am UTC](https://forum.search-guard.com/t/unresponsive-es2-3-cluster-with-searchguard-2-3-installed/976/3 "2018-07-01T10:56:51Z")

</div>

Are you really running Search Guard 2.3.3.1? The last version for ES 2.3.3 was Search Guard 2.3.3.12.

But in any case we do no support this version anymore, neither does Elasticsearch. So I’m very sorry but it seems you need to upgrade …

> **···**
>
> On Tuesday, June 26, 2018 at 9:09:56 PM UTC+2, ark.202 wrote:
> 
> > Hi there
> 
> > 
> 
> > 
> 
> > We had an unresponsive Elasticsearch cluster (5 node host-[06-10] , with 32GB of heap space assigned to each host). Upon checking cluster against each host only one of it (host-08) would respond after a while - Cluster state as Yelllow and having only 4 hosts. The rest would either timeout or return SSL error.
> 
> > 
> 
> > Also, found that most hosts have reached a heap space (\_cat/nodes) of almost 99%
> 
> > 
> 
> > Upon checking the logs we found the exceptions listed in the log messages section below.
> 
> > 
> 
> > I checked the validity of the certificates in JKS and they are good till Aug 20th 2018.
> 
> > 
> 
> > Eventually I had restart the cluster to resolve this and it took almost 5-6 hours with ~330GB of index size. The cluster is now reen and healthy. However, I still see the below error messages in the log
> 
> > 
> 
> > [2018-06-26 15:49:17,539] [ERROR][com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] [host-10] SSL Problem null cert chain  
> > javax.net.ssl.SSLHandshakeException: null cert chain
> 
> > 
> 
> > Any thoughts on what could’ve gone wrong, please let me know your views.
> 
> > 
> 
> > 
> 
> > 
> 
> > - Search Guard and Elasticsearch version
> 
> > Elasticsearch version - 2.3.3
> 
> > Search Guard version -
> 
> > “component”: “search-guard-2”,  
> > “version”: “2.3.3.1”,
> 
> > “component”: “search-guard-ssl”,  
> > “version”: “2.3.3.13”,
> 
> > 
> 
> > 
> 
> > - JVM version and operating system version
> 
> > java version “1.7.0\_51”
> 
> > OS : Oracle Linux 6.9
> 
> > 
> 
> > 
> 
> > 
> 
> > - Search Guard configuration files
> 
> > 
> 
> > 
> 
> > searchguard.ssl.transport.enabled: true  
> > searchguard.ssl.transport.keystore\_type: JKS  
> > searchguard.ssl.transport.keystore\_filepath: jks1/server.jks  
> > searchguard.ssl.transport.keystore\_password: \<\>  
> > searchguard.ssl.transport.truststore\_type: JKS  
> > searchguard.ssl.transport.truststore\_filepath: jks1/server.jks  
> > searchguard.ssl.transport.truststore\_password: \<\>  
> > searchguard.ssl.http.enabled: true  
> > searchguard.ssl.http.keystore\_type: JKS  
> > searchguard.ssl.http.keystore\_filepath: jks1/server.jks  
> > searchguard.ssl.http.keystore\_password: \<\>  
> > searchguard.ssl.http.truststore\_type: JKS  
> > searchguard.ssl.http.truststore\_filepath: jks1/server.jks  
> > searchguard.ssl.http.truststore\_password: \<\>  
> > searchguard.ssl.http.clientauth\_mode: NONE
> > 
> > searchguard.enabled: true  
> > security.manager.enabled: false
> 
> > 
> 
> > 
> 
> > 
> 
> > searchguard.authcz.admin\_dn:  
> > - “C=US,ST=California,O=A\* Inc.,OU=management:group.752639,CN=[rody.com](http://rody.com)”  
> > - “C=US,ST=California,O=A\* Inc.,OU=management:group.752639,CN=[pwk-rody.com](http://pwk-rody.com)”  
> > - “C=US,ST=California,O=A\* Inc.,OU=management:group.752639,CN=[pwk-f5p-lnn01.com](http://pwk-f5p-lnn01.com)”
> > 
> > searchguard.authcz.impersonation\_dn:  
> > “C=US,ST=California,O=A\* Inc.,OU=management:group.752639,CN=[rody.com](http://rody.com)”:  
> > - '_’  
> > searchguard.authcz.impersonation\_dn:  
> > "C=US,ST=California,O=A_ Inc.,OU=management:group.752639,CN=[pwk.rody.com](http://pwk.rody.com)":  
> > - '_’  
> > searchguard.authcz.impersonation\_dn:  
> > "C=US,ST=California,O=A_ Inc.,OU=management:group.752639,CN=pwk-f5p-lnn01…com":
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > - Elasticsearch log messages on debug level
> 
> > 
> 
> > 
> 
> > 
> 
> > [2018-06-19 18:56:30,600][WARN][netty.handler.ssl.SslHandler] Unexpected leftover data after SSLEngine.unwrap(): status=OK handshakeStatus=NEED\_WRAP consumed=0 produced=0 remaining=7 data=15030300020100
> 
> > 
> 
> > 
> 
> > //Several instances of GC cycles  
> > [2018-06-19 18:56:30,593][WARN][monitor.jvm] [host-07] [gc][old][1374850][5979] duration [2.1m], collections [1]/[2.1m], total [2.1m]/[4.6d], memory [31.7gb]-\>[31.7gb]/[31.7gb], all\_pools {[young] [1.8gb]-\>[1.8gb]/[1.8gb]}{[survivor] [230mb]-\>[230.1mb]/[232.9mb]}{[old] [29.7gb]-\>[29.7gb]/[29.7gb]}
> 
> > 
> 
> > 
> 
> > [2018-06-19 18:56:30,589][DEBUG][action.search] [host-07] [14368933] Failed to execute query phase  
> > RemoteTransportException[[host-10][17.160.69.238:9300][indices:data/read/search[phase/query/id]]]; nested: SearchContextMissingException[No search context found for id [14368933]];  
> > Caused by: SearchContextMissingException[No search context found for id [14368933]]  
> > at org.elasticsearch.search.SearchService.findContext(SearchService.java:613 )  
> > at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:425 )  
> > at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:376 )  
> > at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:373 )  
> > at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33 )  
> > at com.floragunn.searchguard.ssl.transport.SearchGuardSSLTransportService.messageReceivedDecorate(SearchGuardSSLTransportService.java:161 )  
> > at com.floragunn.searchguard.transport.SearchGuardTransportService.messageReceivedDecorate(SearchGuardTransportService.java:293 )  
> > at com.floragunn.searchguard.ssl.transport.SearchGuardSSLTransportService$Interceptor.messageReceived(SearchGuardSSLTransportService.java:128 )  
> > at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75 )  
> > at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:300 )  
> > at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37 )  
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145 )  
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615 )  
> > at java.lang.Thread.run(Thread.java:744)
> 
> > 
> 
> > 
> 
> > 
> 
> > [2018-06-19 19:59:16,722][WARN][discovery.zen.ping.unicast] [host-07] failed to send ping to [{#zen\_unicast\_3#}{17.160.69.236}{host-08/17.99.69.266:9300}]  
> > ReceiveTimeoutTransportException[[host-08m/17.99.69.266:9300][internal:discovery/zen/unicast] request\_id [80874756] timed out after [124435ms]]  
> > at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:679 )  
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145 )  
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615 )  
> > at java.lang.Thread.run(Thread.java:744)
> 
> > 
> 
> > [2018-06-19 20:17:07,562][WARN][com.floragunn.searchguard.transport.SearchGuardTransportService] [host-07] Received response for a request that has timed out, sent [1195273ms] ago, timed out [1070838ms] ago, action [internal:discovery/zen/unicast], node [{#zen\_unicast\_3#}{17.160.69.236}{[ma2-rodyp-lcb08.corp.apple.com/17.160.69.236:9300](http://ma2-rodyp-lcb08.corp.apple.com/17.160.69.236:9300)}], id [80874756]
> 
> > 
> 
> > [2018-06-20 03:07:58,459][WARN][com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] [host-07] exception caught on transport layer [[id: 0x0f3baf15, /17.96.16.84:48738 =\> /17.160.69.235:9300]], closing connection  
> > java.lang.OutOfMemoryError: Java heap space  
> > at sun.security.ssl.EngineInputRecord.compareMacTags(EngineInputRecord.java:330 )  
> > at sun.security.ssl.EngineInputRecord.checkMacTags(EngineInputRecord.java:313 )  
> > at sun.security.ssl.EngineInputRecord.decrypt(EngineInputRecord.java:244)
> 
> > 
> 
> > 
> 
> > [2018-06-20 12:08:05,960][ERROR][com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] [host-08] SSL Problem Received close\_notify during handshake  
> > javax.net.ssl.SSLException: Received close\_notify during handshake  
> > at sun.security.ssl.Alerts.getSSLException(Alerts.java:208 )  
> > at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1619 )  
> > at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1587 )  
> > at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1732 )  
> > at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1060 )  
> > at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:884 )  
> > at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:758 )  
> > at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624 )  
> > at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1218 )  
> > at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:852 )  
> > at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425 )  
> > at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303 )  
> > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70 )  
> > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564 )  
> > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559 )  
> > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268 )  
> > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255 )  
> > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88 )  
> > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108 )  
> > at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337 )  
> > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89 )  
> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178 )  
> > at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108 )  
> > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42 )  
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145 )  
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615 )  
> > at java.lang.Thread.run(Thread.java:744)
> 
> > 
> 
> > [2018-06-20 15:34:35,383][ERROR][com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] [host-08] SSL Problem null cert chain  
> > javax.net.ssl.SSLHandshakeException: null cert chain  
> > at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1290 )  
> > at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:513 )  
> > at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:790 )  
> > at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:758 )  
> > at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624 )  
> > at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1218 )  
> > at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:852 )  
> > at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425 )  
> > at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303 )  
> > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70 )  
> > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564 )  
> > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559 )  
> > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268 )  
> > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255 )  
> > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88 )  
> > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108 )  
> > at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337 )  
> > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89 )  
> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178 )  
> > at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108 )  
> > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42 )  
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145 )  
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615 )  
> > at java.lang.Thread.run(Thread.java:744)  
> > Caused by: javax.net.ssl.SSLHandshakeException: null cert chain  
> > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192 )  
> > at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1619 )  
> > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:278 )  
> > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:266 )  
> > at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1631 )  
> > at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:176 )  
> > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868 )  
> > at sun.security.ssl.Handshaker$1.run(Handshaker.java:808 )  
> > at sun.security.ssl.Handshaker$1.run(Handshaker.java:806 )  
> > at java.security.AccessController.doPrivileged(Native Method )  
> > at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1227 )  
> > at org.jboss.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1392 )  
> > at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1255 )  
> > … 18 more
> 
> > 
> 
> > 
> 
> > 
> 
> > [2018-06-21 21:29:00,075][WARN][gateway] [host-08] [searchguard][0]: failed to list shard for shard\_store on node [kxb4QDvVR8eNNwOoPT-uVQ]  
> > FailedNodeException[total failure in fetching]; nested: ElasticsearchException[unauthenticated request internal:cluster/nodes/indices/shard/store for user User [name=\_sg\_internal, roles=]];  
> > at org.elasticsearch.gateway.AsyncShardFetch$1.onFailure(AsyncShardFetch.java:277 )  
> > at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:95 )  
> > at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:135 )  
> > at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170 )  
> > at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:144 )  
> > at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:85 )  
> > at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.list(TransportNodesListShardStoreMetaData.java:88 )  
> > at org.elasticsearch.gateway.AsyncShardFetch.asyncFetch(AsyncShardFetch.java:267 )  
> > at org.elasticsearch.gateway.AsyncShardFetch.fetchData(AsyncShardFetch.java:117 )  
> > at org.elasticsearch.gateway.GatewayAllocator$InternalReplicaShardAllocator.fetchData(GatewayAllocator.java:183 )  
> > at org.elasticsearch.gateway.ReplicaShardAllocator.allocateUnassigned(ReplicaShardAllocator.java:137 )  
> > at org.elasticsearch.gateway.GatewayAllocator.allocateUnassigned(GatewayAllocator.java:123 )  
> > at org.elasticsearch.cluster.routing.allocation.allocator.ShardsAllocators.allocateUnassigned(ShardsAllocators.java:70 )  
> > at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:258 )  
> > at org.elasticsearch.cluster.routing.allocation.AllocationService.applyStartedShards(AllocationService.java:86 )  
> > at org.elasticsearch.cluster.action.shard.ShardStateAction$ShardStartedClusterStateHandler.execute(ShardStateAction.java:218 )  
> > at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:468 )  
> > at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772 )  
> > at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231 )  
> > at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194 )  
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145 )  
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615 )  
> > at java.lang.Thread.run(Thread.java:744)  
> > Caused by: ElasticsearchException[unauthenticated request internal:cluster/nodes/indices/shard/store for user User [name=\_sg\_internal, roles=]]  
> > … 21 more
> 
> > 
> 
> > 
> 
> > 
> 
> > - Other installed Elasticsearch or Kibana plugins, if any
> 
> > 
> 
> > delete-by-query - 2.3.3
> 
> > head