Node cannot join cluster due to "close_notify during handshake"

Search Guard
1

Hi,
I am using Elasticsearch cluster version 6.5.4 and the corresponding searchguard version
I have a cluster with 1 master eligible node and all other are data nodes.
In that while trying to add a node I am getting the following exception in master node:

[2019-05-02T12:39:02,161][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [MASTER] SSL Problem Received close_notify during handshake
javax.net.ssl.SSLException: Received close_notify during handshake
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634) ~[?:?]
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1776) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_51]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:294) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1297) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-common-4.1.30.Final.jar:4.1.30.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_51]

The following exception occurs in data node:

[2019-05-02T14:44:04,236][WARN ][c.f.s.s.t.SearchGuardSSLNettyTransport] [test-dc1] send message failed [channel: NettyTcpChannel{localAddress=/172.24.159.127:56086, remoteAddress=/172.21.205.160:9300}]
javax.net.ssl.SSLException: SSLEngine closed already
at io.netty.handler.ssl.SslHandler.wrap(…)(Unknown Source) ~[?:?]
[2019-05-02T14:44:07,231][WARN ][o.e.d.z.ZenDiscovery ] [test-dc1] not enough master nodes discovered during pinging (found [], but needed [1]), pinging again

The node is not joining the cluster

Please help me…
Thanks in Advance

3

Please attach the full logs for the data and master node (zipped and with sensitive informations stripped). Please also provide these informations

  • Operating system details
  • JVM version and vendor
  • OpenSSL version (if applicable)
  • How did you generate the SSL certificates
  • elasticsearch.yml of all nodes
4

Master node details:
Windows 10

Data node details:
Windows server 2008 R2

Java version(Oracle Java) : 1.8.0_20

I generated the certificates with my own code using bouncycastle library

Master Node ES.yml

cluster.name:CLUSTER
node.name: MASTER
node.master: true
node.data: true
path.data: “./…/esdata”
path.logs: “./…/logs/eslogs”
http.port: 9200
network.host: 172.21.205.160
cluster.routing.allocation.disk.watermark.low: 2gb
cluster.routing.allocation.disk.watermark.high: 1.5gb
cluster.routing.allocation.disk.watermark.flood_stage: 1gb
discovery.zen.minimum_master_nodes: 1
transport.tcp.port: 9300
searchguard.enterprise_modules_enabled: false
searchguard.ssl.transport.pemcert_filepath: node.pem
searchguard.ssl.transport.pemkey_filepath: node.key
searchguard.ssl.transport.pemtrustedcas_filepath: root_ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: client.pem
searchguard.ssl.http.pemkey_filepath: client.key
searchguard.ssl.http.pemtrustedcas_filepath: root_ca.pem
searchguard.ssl.http.clientauth_mode: REQUIRE
searchguard.nodes_dn:

  • CN=*.testing, OU=none, O=none, L=none, ST=US, C=US
    searchguard.authcz.admin_dn:
  • CN=admin.testing, OU=none, O=none, L=none, ST=US, C=US

Data node ES.yml

http.enabled: false
cluster.name:CLUSTER
cluster.routing.allocation.disk.watermark.flood_stage: 1gb
cluster.routing.allocation.disk.watermark.low: 2gb
cluster.routing.allocation.disk.watermark.high: 1.5gb
http.port: 9200
discovery.zen.ping.unicast.hosts: [“172.21.205.160:9300”]
node.name: tester-dc1
node.master: false
node.data: true
node.ingest: false
path.data: “./…/esdata”
path.logs: “./…/eslogs”
network.host: test-dc1
transport.tcp.port: 9300
node.attr.location: C:\testing
node.attr.hostname: test-dc1
discovery.zen.minimum_master_nodes: 1
searchguard.enterprise_modules_enabled: false
searchguard.ssl.transport.pemcert_filepath: node.pem
searchguard.ssl.transport.pemkey_filepath: node.key
searchguard.ssl.transport.pemtrustedcas_filepath: root_ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: false
searchguard.nodes_dn:

  • CN=*.testing, OU=none, O=none, L=none, ST=US, C=US
    searchguard.authcz.admin_dn:
  • CN=admin.testing, OU=none, O=none, L=none, ST=US, C=US
5

Please upgrade Java on all machine to at least 1.8.0u111 (or even better to Java 1.8.0u212).

6

Even after upgrading the same thing is happening

7

Please provide the full logs of both nodes

8

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.