SG :None of the configured nodes are available [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version

  • Installed and used enterprise modules, if any

  • JVM version and operating system version

  • Search Guard configuration files

  • Elasticsearch log messages on debug level

  • Other installed Elasticsearch or Kibana plugins, if any

Hello,

I configured searchguard in a single node elastic search environment (elasticsearch 6.2.2 and searchguard 6.2.2-22.0) and I have 2 problems :

[o.e.m.j.JvmGcMonitorService] [hub-iot-monitoring-elasticsearch] [gc][10] overhead, spent [403ms] collecting in the last [1.1s]

[INFO ][c.f.s.h.SearchGuardHttpServerTransport] publish_address {10.128.12.210:9200}, bound_addresses {[::]:9200}

[INFO ][o.e.n.Node ] started

[c.f.s.s.t.SearchGuardSSLNettyTransport] SSL Problem Received close_notify during handshake

javax.net.ssl.SSLException: Received close_notify during handshake

at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]

at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[?:?]

at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634) ~[?:?]

at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1776) ~[?:?]

at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083) ~[?:?]

at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907) ~[?:?]

at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]

at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_151]

at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{X34NHNr_TIm8FUSje0kA7w}{localhost}{127.0.0.1:9300}]]

at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347)

at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245)

at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:60)

at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:371)

at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:405)

at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:394)

at com.floragunn.searchguard.tools.SearchGuardAdmin.main0(SearchGuardAdmin.java:444)

at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:123)

My elasticsearch.yml :

cluster.name: ${CLUSTER_NAME}
node.name: ${NODE_NAME}
node.master: true
node.data: true
node.ingest: true
network.host: 0.0.0.0
http.enabled: true
http.port: 9200
http.compression: true
http.cors.enabled: true
http.cors.allow-origin: *
bootstrap.memory_lock: false
discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.unicast.hosts: 127.0.0.1, [::1]
searchguard.enterprise_modules_enabled: false

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.enable_openssl_if_available: true
searchguard.ssl.transport.keystore_type: JKS
searchguard.ssl.transport.keystore_filepath: searchguard/ssl/${NODE_NAME}-keystore.jks
searchguard.ssl.transport.keystore_password: ${KS_PWD}
searchguard.ssl.transport.truststore_type: JKS
searchguard.ssl.transport.truststore_filepath: searchguard/ssl/truststore.jks
searchguard.ssl.transport.truststore_password: ${TS_PWD}
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.clientauth_mode: OPTIONAL
searchguard.ssl.http.enable_openssl_if_available: true
searchguard.ssl.http.keystore_type: JKS
searchguard.ssl.http.keystore_filepath: searchguard/ssl/${NODE_NAME}-keystore.jks
searchguard.ssl.http.keystore_password: ${KS_PWD}
searchguard.ssl.http.truststore_type: JKS
searchguard.ssl.http.truststore_filepath: searchguard/ssl/truststore.jks
searchguard.ssl.http.truststore_password: ${TS_PWD}
searchguard.authcz.admin_dn:
  - "CN=elastic ,OU=SSL, C=FR"

and i run sgadmin like this :

/elasticsearch/plugins/search-guard-6/tools/sgadmin.sh \
-cd /elasticsearch/config/searchguard \
-ks /elasticsearch/config/searchguard/ssl/elastic-keystore.jks \
-ts /elasticsearch/config/searchguard/ssl/truststore.jks \
-kspass $KS_PWD \
-tspass $TS_PWD \
-nhnv \
-icl

Usually, this means that the TLS handshake is taking too long and netty is closing the connection.

Are you running Elasticsearch on a very machine with not enough memory? This message here:

[o.e.m.j.JvmGcMonitorService] [hub-iot-monitoring-elasticsearch] [gc][10] overhead, spent [403ms] collecting in the last [1.1s]

``

would also indicate this. BTW, 22.0 is quite old, the current SG version is for 6.2.2 is 23.0.

ยทยทยท

On Wednesday, October 17, 2018 at 7:59:05 PM UTC+2, sara qasmi wrote:

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version
  • Installed and used enterprise modules, if any
  • JVM version and operating system version
  • Search Guard configuration files
  • Elasticsearch log messages on debug level
  • Other installed Elasticsearch or Kibana plugins, if any

Hello,

I configured searchguard in a single node elastic search environment (elasticsearch 6.2.2 and searchguard 6.2.2-22.0) and I have 2 problems :

[o.e.m.j.JvmGcMonitorService] [hub-iot-monitoring-elasticsearch] [gc][10] overhead, spent [403ms] collecting in the last [1.1s]

[INFO ][c.f.s.h.SearchGuardHttpServerTransport] publish_address {10.128.12.210:9200}, bound_addresses {[::]:9200}

[INFO ][o.e.n.Node ] started

[c.f.s.s.t.SearchGuardSSLNettyTransport] SSL Problem Received close_notify during handshake

javax.net.ssl.SSLException: Received close_notify during handshake

at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]

at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[?:?]

at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634) ~[?:?]

at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1776) ~[?:?]

at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083) ~[?:?]

at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907) ~[?:?]

at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]

at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_151]

at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{X34NHNr_TIm8FUSje0kA7w}{localhost}{127.0.0.1:9300}]]

at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347)

at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245)

at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:60)

at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:371)

at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:405)

at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:394)

at com.floragunn.searchguard.tools.SearchGuardAdmin.main0(SearchGuardAdmin.java:444)

at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:123)

My elasticsearch.yml :

[cluster.name](http://cluster.name): ${CLUSTER_NAME}
[node.name](http://node.name): ${NODE_NAME}
node.master: true
node.data: true
node.ingest: true
network.host: 0.0.0.0
http.enabled: true
http.port: 9200
http.compression: true
http.cors.enabled: true
http.cors.allow-origin: *
bootstrap.memory_lock: false
discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.unicast.hosts: 127.0.0.1, [::1]
searchguard.enterprise_modules_enabled: false

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.enable_openssl_if_available: true
searchguard.ssl.transport.keystore_type: JKS
searchguard.ssl.transport.keystore_filepath: searchguard/ssl/${NODE_NAME}-keystore.jks
searchguard.ssl.transport.keystore_password: ${KS_PWD}
searchguard.ssl.transport.truststore_type: JKS
searchguard.ssl.transport.truststore_filepath: searchguard/ssl/truststore.jks
searchguard.ssl.transport.truststore_password: ${TS_PWD}
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.clientauth_mode: OPTIONAL
searchguard.ssl.http.enable_openssl_if_available: true
searchguard.ssl.http.keystore_type: JKS
searchguard.ssl.http.keystore_filepath: searchguard/ssl/${NODE_NAME}-keystore.jks
searchguard.ssl.http.keystore_password: ${KS_PWD}
searchguard.ssl.http.truststore_type: JKS
searchguard.ssl.http.truststore_filepath: searchguard/ssl/truststore.jks
searchguard.ssl.http.truststore_password: ${TS_PWD}
searchguard.authcz.admin_dn:
  - "CN=elastic ,OU=SSL, C=FR"

and i run sgadmin like this :

/elasticsearch/plugins/search-guard-6/tools/sgadmin.sh \
-cd /elasticsearch/config/searchguard \
-ks /elasticsearch/config/searchguard/ssl/elastic-keystore.jks \
-ts /elasticsearch/config/searchguard/ssl/truststore.jks \
-kspass $KS_PWD \
-tspass $TS_PWD \
-nhnv \
-icl