ES 6.8.9 + SG 25.6: Connection reset by peer at each request

EtienneM · May 19, 2020, 7:16am

This is a double post of ES 6.8.x + Searchguard 25.5 Connection reset by peer at each request (#3) · Issues · search-guard / legacy / Search Guard · GitLab as we had no answer on the GitLab issue, and it seems the good place to get support is here

Elasticsearch version: tested with 6.8.6 and 6.8.9

Search Guard version: 25.5 and 25.6

Describe the issue:

We have been deploying Elastcsearch along with Search Guard for a while and since the last release, we get a weird behavior which is flooding our logs. Each requests to the cluster outputs a “Connection reset by peer” error. Any request leads to this error but the one we execute for the following example is GET / and the result is 200 OK with the correct content.

2020-05-19 08:58:20.715755243 +0200 CEST [elasticsearch-1] [2020-05-19T06:58:20,712][ERROR][c.f.s.s.h.n.SearchGuardSSLNettyHttpServerTransport] [7c2252c4-8c22-4eee-804c-85602ad09d1c.azerty-1581.elasticsearch.dbs.172.17.0.1.xip.st-sc.fr] Exception during establishing a SSL connection: java.io.IOException: Connection reset by peer
2020-05-19 08:58:20.715763272 +0200 CEST [elasticsearch-1] java.io.IOException: Connection reset by peer
2020-05-19 08:58:20.715764838 +0200 CEST [elasticsearch-1] 	at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:?]
2020-05-19 08:58:20.715765965 +0200 CEST [elasticsearch-1] 	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:?]
2020-05-19 08:58:20.715766851 +0200 CEST [elasticsearch-1] 	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276) ~[?:?]
2020-05-19 08:58:20.715767616 +0200 CEST [elasticsearch-1] 	at sun.nio.ch.IOUtil.read(IOUtil.java:245) ~[?:?]
2020-05-19 08:58:20.715816539 +0200 CEST [elasticsearch-1] 	at sun.nio.ch.IOUtil.read(IOUtil.java:223) ~[?:?]
2020-05-19 08:58:20.715818174 +0200 CEST [elasticsearch-1] 	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:358) ~[?:?]
2020-05-19 08:58:20.715819183 +0200 CEST [elasticsearch-1] 	at io.netty.buffer.PooledHeapByteBuf.setBytes(PooledHeapByteBuf.java:261) ~[netty-buffer-4.1.32.Final.jar:4.1.32.Final]
2020-05-19 08:58:20.715820180 +0200 CEST [elasticsearch-1] 	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132) ~[netty-buffer-4.1.32.Final.jar:4.1.32.Final]
2020-05-19 08:58:20.715842799 +0200 CEST [elasticsearch-1] 	at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:347) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
2020-05-19 08:58:20.715844245 +0200 CEST [elasticsearch-1] 	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
2020-05-19 08:58:20.715845217 +0200 CEST [elasticsearch-1] 	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
2020-05-19 08:58:20.715846179 +0200 CEST [elasticsearch-1] 	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
2020-05-19 08:58:20.715868128 +0200 CEST [elasticsearch-1] 	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
2020-05-19 08:58:20.715869388 +0200 CEST [elasticsearch-1] 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
2020-05-19 08:58:20.715884696 +0200 CEST [elasticsearch-1] 	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
2020-05-19 08:58:20.715885932 +0200 CEST [elasticsearch-1] 	at java.lang.Thread.run(Thread.java:834) [?:?]

This is kind of annoying as it really floods the logs on every request.

The previous version was Elasticsearch 6.8.2 along with Search Guard 25.4, and it was working as expected (i.e. no “Connection reset by peer” log at each request)

Is it a known issue on Search Guard side? Do you have any idea on where does it come from?

Thanks a lot for your help

EtienneM · May 19, 2020, 7:24am

For your information, I downloaded Search Guard 25.4, updated the file plugin-descriptor.properties so that the line elasticsearch.version equals 6.8.9, and re-created the archive.

I can now start an Elasticsearch and query it without having my logs flooded

But I would prefer if it would be possible to use the latest version (25.6) of Search Guard

cgrard · July 23, 2020, 10:46am

We’re having the exact same issue, can’t find where it comes from as every service accessing to Elastic is working properly without reporting any error whatsoever. Our setup is slightly different as it is Elastic 6.8.10 with SearchGuard 25.6 but the error is the same:

[ERROR][c.f.s.s.h.n.SearchGuardSSLNettyHttpServerTransport] [elasticsearch] Exception during establishing a SSL connection: java.io.IOException: Connection reset by peer
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:?]
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:?]
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:?]
	at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[?:?]
	at io.netty.buffer.PooledHeapByteBuf.setBytes(PooledHeapByteBuf.java:261) ~[netty-buffer-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132) ~[netty-buffer-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:347) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]

srgbnd · July 23, 2020, 12:55pm

@cgrard please post the following configs to help reproduce the issue:

elasticsearch.yml
sg_config.yml

cgrard · July 23, 2020, 1:17pm

Here is my sg_config.yml

searchguard:
  dynamic:
    authc:
      basic_internal_auth:
        enabled: true
        order: 1
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: internal

And here is my elasticsearch.yml is a bit complicated because it is dockerized with various variables, but here is a readable version of it:

cluster.name: "elasticsearch"

node:
  name: NODE-1
  master: true
  data: true
  ingest: true

discovery.zen:
  minimum_master_nodes: 1
  ping.unicast.hosts: "127.0.0.1, [::1]"
network.host: "0.0.0.0"

http:
  compression: true
  cors:
    enabled: true
    allow-origin: *

path.repo: ["/opt/elasticsearch/backup"]

searchguard:
  enterprise_modules_enabled: false
  restapi:
    roles_enabled:
      - "admin"
  ssl.transport:
    enabled: true
    enable_openssl_if_available: true
    keystore_type: JKS
    keystore_filepath: searchguard/ssl/NODE-1-keystore.jks
    keystore_password: "changeme"
    truststore_type: JKS
    truststore_filepath: searchguard/ssl/truststore.jks
    truststore_password: "changeme"
    enforce_hostname_verification: false
  ssl.http:
    enabled: true
    clientauth_mode: OPTIONAL
    enable_openssl_if_available: true
    keystore_type: JKS
    keystore_filepath: searchguard/ssl/NODE-1-keystore.jks
    keystore_password: "changeme"
    truststore_type: JKS
    truststore_filepath: searchguard/ssl/truststore.jks
    truststore_password: "changeme"
  authcz.admin_dn:
    - "CN=elastic ,OU=devops, C=COM"

searchguard.enable_snapshot_restore_privilege: true

jkressin · July 23, 2020, 3:08pm

Indeed, this looks like an issue that has been fixed for Search Guard 7, but it seems the fix has not made it into the 6.x line of Search Guard.

As a workaround, until a fix is released, you can lower the log level of the respective class since the error is not critical. You can do this dynamically or statically.

In log4j.properties, add:

logger.sgnetty.name = com.floragunn.searchguard.ssl.http.netty.SearchGuardSSLNettyHttpServerTransport
logger.sgnetty.level = fatal

Or dynamically in a running cluster by using the settings API:

curl -u admin:admin --insecure -X PUT "https://localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d '{
  "transient": {
    "logger.com.floragunn.searchguard.ssl.http.netty.SearchGuardSSLNettyHttpServerTransport": "fatal"
  }
}'

Sorry for the inconvenience!

cgrard · July 23, 2020, 3:51pm

Thanks, it seems that the dynamic API call solved the issue, I will wait for the fix to be included in the 6.x line of Search Guard and update my Docker image accordingly at that moment.

Thanks again!

Topic		Replies	Views
Exception during establishing a SSL connection: java.net.SocketException: Connection reset Search Guard	8	7274	August 24, 2020
Logstash communication failed with Elasticsearch search guard Search Guard	2	530	February 7, 2018
TLS setup failing - ES 6.1.2 Search Guard	0	942	March 8, 2018
Search Guard Refresh Interval Error Search Guard	5	434	November 15, 2018
SSL Connection Problem between Logstash and ElasticSearch with SearchGuard Search Guard	5	756	March 16, 2018

ES 6.8.9 + SG 25.6: Connection reset by peer at each request

Related topics