[2019-03-26T04:55:50,759][INFO ][c.f.s.SearchGuardPlugin ] [10.49.112.126-ingest-node-0]4 Search Guard modules loaded so far: [Module [type=MULTITENANCY, implementing class=com.floragunn.searchguard.configuration.PrivilegesInterceptorImpl], Module [type=REST_MANAGEMENT_API, implementing class=com.floragunn.searchguard.dlic.rest.api.SearchGuardRestApiActions], Module [type=DLSFLS, implementing class=com.floragunn.searchguard.configuration.SearchGuardFlsDlsIndexSearcherWrapper], Module [type=AUDITLOG, implementing class=com.floragunn.searchguard.auditlog.impl.AuditLogImpl]]
[2019-03-26T04:55:50,840][INFO ][c.f.s.c.ComplianceConfig ] [10.49.112.126-ingest-node-0]Compliance features are enabled
[2019-03-26T04:55:50,841][INFO ][c.f.s.c.IndexBaseConfigurationRepository] [10.49.112.126-ingest-node-0]Search Guard License Info: SearchGuardLicense [uid=00000000-0000-0000-0000-000000000000, type=TRIAL, features=[COMPLIANCE], issueDate=2019-03-14, expiryDate=2019-05-13, issuedTo=The world, issuer=floragunn GmbH, startDate=2019-03-14, majorVersion=6, clusterName=*, allowedNodeCount=2147483647, msgs=, expiresInDays=48, isExpired=false, valid=true, action=, prodUsage=Yes, one cluster with all commercial features and unlimited nodes per cluster., clusterService=org.elasticsearch.cluster.service.ClusterService@28a1449c, getMsgs()=, getExpiresInDays()=48, isExpired()=false, isValid()=true, getAction()=, getProdUsage()=Yes, one cluster with all commercial features and unlimited nodes per cluster.]
[2019-03-26T04:55:50,841][INFO ][c.f.s.c.IndexBaseConfigurationRepository] [10.49.112.126-ingest-node-0]Search Guard License Type: TRIAL, valid
[2019-03-26T04:55:50,841][INFO ][c.f.s.c.IndexBaseConfigurationRepository] [10.49.112.126-ingest-node-0]Node ‘10.49.112.126-ingest-node-0’ initialized
[2019-03-26T04:55:54,664][ERROR][c.f.s.s.h.n.SearchGuardSSLNettyHttpServerTransport] [10.49.112.126-ingest-node-0]SSL Problem Received fatal alert: bad_certificate
javax.net.ssl.SSLException: Received fatal alert: bad_certificate
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1647) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1615) ~[?:?]
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1781) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1070) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:896) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_191]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1301) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
``
From the log above, it throws the error after it tried to initializing the node.
In my ES cluster, I have 2 hot data nodes (2 instances/node), 2 warm data nodes (4 instances/node), 3 master nodes and 2 ingest nodes.
It errors on 2 ingest nodes only. What I did wrong?
I also used sgdiag.sh validates the certificate for the nodes, and it passes.
When asking questions, please provide the following information:
- Search Guard and Elasticsearch version
24.1 and 6.6.0
- Installed and used enterprise modules, if any
No
- JVM version and operating system version
1.8
-
Search Guard configuration files
-
Elasticsearch log messages on debug level
-
Other installed Elasticsearch or Kibana plugins, if any