ES 6.8.9 + SG 25.6: Connection reset by peer at each request

This is a double post of https://git.floragunn.com/search-guard/search-guard/-/issues/3 as we had no answer on the GitLab issue, and it seems the good place to get support is here :slight_smile:

Elasticsearch version: tested with 6.8.6 and 6.8.9

Search Guard version: 25.5 and 25.6

Describe the issue:

We have been deploying Elastcsearch along with Search Guard for a while and since the last release, we get a weird behavior which is flooding our logs. Each requests to the cluster outputs a “Connection reset by peer” error. Any request leads to this error but the one we execute for the following example is GET / and the result is 200 OK with the correct content.

2020-05-19 08:58:20.715755243 +0200 CEST [elasticsearch-1] [2020-05-19T06:58:20,712][ERROR][c.f.s.s.h.n.SearchGuardSSLNettyHttpServerTransport] [7c2252c4-8c22-4eee-804c-85602ad09d1c.azerty-1581.elasticsearch.dbs.172.17.0.1.xip.st-sc.fr] Exception during establishing a SSL connection: java.io.IOException: Connection reset by peer
2020-05-19 08:58:20.715763272 +0200 CEST [elasticsearch-1] java.io.IOException: Connection reset by peer
2020-05-19 08:58:20.715764838 +0200 CEST [elasticsearch-1] 	at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:?]
2020-05-19 08:58:20.715765965 +0200 CEST [elasticsearch-1] 	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:?]
2020-05-19 08:58:20.715766851 +0200 CEST [elasticsearch-1] 	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276) ~[?:?]
2020-05-19 08:58:20.715767616 +0200 CEST [elasticsearch-1] 	at sun.nio.ch.IOUtil.read(IOUtil.java:245) ~[?:?]
2020-05-19 08:58:20.715816539 +0200 CEST [elasticsearch-1] 	at sun.nio.ch.IOUtil.read(IOUtil.java:223) ~[?:?]
2020-05-19 08:58:20.715818174 +0200 CEST [elasticsearch-1] 	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:358) ~[?:?]
2020-05-19 08:58:20.715819183 +0200 CEST [elasticsearch-1] 	at io.netty.buffer.PooledHeapByteBuf.setBytes(PooledHeapByteBuf.java:261) ~[netty-buffer-4.1.32.Final.jar:4.1.32.Final]
2020-05-19 08:58:20.715820180 +0200 CEST [elasticsearch-1] 	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132) ~[netty-buffer-4.1.32.Final.jar:4.1.32.Final]
2020-05-19 08:58:20.715842799 +0200 CEST [elasticsearch-1] 	at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:347) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
2020-05-19 08:58:20.715844245 +0200 CEST [elasticsearch-1] 	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
2020-05-19 08:58:20.715845217 +0200 CEST [elasticsearch-1] 	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
2020-05-19 08:58:20.715846179 +0200 CEST [elasticsearch-1] 	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
2020-05-19 08:58:20.715868128 +0200 CEST [elasticsearch-1] 	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
2020-05-19 08:58:20.715869388 +0200 CEST [elasticsearch-1] 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
2020-05-19 08:58:20.715884696 +0200 CEST [elasticsearch-1] 	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
2020-05-19 08:58:20.715885932 +0200 CEST [elasticsearch-1] 	at java.lang.Thread.run(Thread.java:834) [?:?]

This is kind of annoying as it really floods the logs on every request.

The previous version was Elasticsearch 6.8.2 along with Search Guard 25.4, and it was working as expected (i.e. no “Connection reset by peer” log at each request)

Is it a known issue on Search Guard side? Do you have any idea on where does it come from?

Thanks a lot for your help :slight_smile:

For your information, I downloaded Search Guard 25.4, updated the file plugin-descriptor.properties so that the line elasticsearch.version equals 6.8.9, and re-created the archive.

I can now start an Elasticsearch and query it without having my logs flooded :slight_smile:

But I would prefer if it would be possible to use the latest version (25.6) of Search Guard