java.security.cert.CertificateExpiredException: NotAfter

Hi:
My es version is 5.5.1,search guard version is v5.Es and search guard have been running normally about two years,but suddenly one day when I execute the following command in my running es cluster to update es access rights:“/opt/elasticsearch-5.5.1/plugins/search-guard-5/tools/sgadmin.sh -cd /opt/elasticsearch-5.5.1/plugins/search-guard-5/sgconfig -cn faceid-es -ks /opt/elasticsearch-5.5.1/config/kirk.jks -ts /opt/elasticsearch-5.5.1/config/truststore.jks -nhnv”,I get the error log,the log as fllow:
[10.104.42.108] run: tools/sgadmin_demo.sh
[10.104.42.108] out: Search Guard Admin v5
[10.104.42.108] out: Will connect to localhost:9300 … done
[10.104.42.108] out:
[10.104.42.108] out: ### LICENSE NOTICE Search Guard ###
[10.104.42.108] out:
[10.104.42.108] out: If you use one or more of the following features in production
[10.104.42.108] out: make sure you have a valid Search Guard license
[10.104.42.108] out: (See https://floragunn.com/searchguard-validate-license)
[10.104.42.108] out:
[10.104.42.108] out: * Kibana Multitenancy
[10.104.42.108] out: * LDAP authentication/authorization
[10.104.42.108] out: * Active Directory authentication/authorization
[10.104.42.108] out: * REST Management API
[10.104.42.108] out: * JSON Web Token (JWT) authentication/authorization
[10.104.42.108] out: * Kerberos authentication/authorization
[10.104.42.108] out: * Document- and Fieldlevel Security (DLS/FLS)
[10.104.42.108] out: * Auditlogging
[10.104.42.108] out:
[10.104.42.108] out: In case of any doubt mail to sales@floragunn.com
[10.104.42.108] out: ###################################
[10.104.42.108] out: Contacting elasticsearch cluster ‘facepp-es’ and wait for YELLOW clusterstate …
[10.104.42.108] out: Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{roaP-g5qRri23cL9i81YvQ}{localhost}{10.104.42.108:9300}]. This is not an error, will keep on trying …
[10.104.42.108] out: * Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
[10.104.42.108] out: * Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
[10.104.42.108] out: * If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
[10.104.42.108] out: * Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
[10.104.42.108] out: 12:34:32.302 [elasticsearch[client][transport_client_boss][T#1]] ERROR com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport - SSL Problem General SSLEngine problem
[10.104.42.108] out: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
[10.104.42.108] out: at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1478) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:1.8.0_144]
[10.104.42.108] out: at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_144]
[10.104.42.108] out: at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:254) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
[10.104.42.108] out: at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1156) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
[10.104.42.108] out: at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
[10.104.42.108] out: at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
[10.104.42.108] out: at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
[10.104.42.108] out: at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
[10.104.42.108] out: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
[10.104.42.108] out: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
[10.104.42.108] out: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
[10.104.42.108] out: at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
[10.104.42.108] out: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
[10.104.42.108] out: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
[10.104.42.108] out: at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
[10.104.42.108] out: at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
[10.104.42.108] out: at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
[10.104.42.108] out: at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
[10.104.42.108] out: at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
[10.104.42.108] out: at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
[10.104.42.108] out: at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.11.Final.jar:4.1.11.Final]
[10.104.42.108] out: at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
[10.104.42.108] out: Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
[10.104.42.108] out: at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.Handshaker$1.run(Handshaker.java:966) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.Handshaker$1.run(Handshaker.java:963) ~[?:1.8.0_144]
[10.104.42.108] out: at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416) ~[?:1.8.0_144]
[10.104.42.108] out: at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1295) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
[10.104.42.108] out: at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1208) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
[10.104.42.108] out: … 18 more
[10.104.42.108] out: Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: timestamp check failed
[10.104.42.108] out: at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:362) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.validator.Validator.validate(Validator.java:260) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1501) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.Handshaker$1.run(Handshaker.java:966) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.Handshaker$1.run(Handshaker.java:963) ~[?:1.8.0_144]
[10.104.42.108] out: at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416) ~[?:1.8.0_144]
[10.104.42.108] out: at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1295) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
[10.104.42.108] out: at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1208) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
[10.104.42.108] out: … 18 more
[10.104.42.108] out: Caused by: java.security.cert.CertPathValidatorException: timestamp check failed
[10.104.42.108] out: at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:223) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:140) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:79) ~[?:1.8.0_144]
[10.104.42.108] out: at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:357) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.validator.Validator.validate(Validator.java:260) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1501) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.Handshaker$1.run(Handshaker.java:966) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.Handshaker$1.run(Handshaker.java:963) ~[?:1.8.0_144]
[10.104.42.108] out: at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416) ~[?:1.8.0_144]
[10.104.42.108] out: at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1295) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
[10.104.42.108] out: at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1208) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
[10.104.42.108] out: … 18 more
[10.104.42.108] out: Caused by: java.security.cert.CertificateExpiredException: NotAfter: Wed Sep 02 02:38:10 CST 2020
[10.104.42.108] out: at sun.security.x509.CertificateValidity.valid(CertificateValidity.java:274) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:629) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.provider.certpath.BasicChecker.verifyTimestamp(BasicChecker.java:190) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:144) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:223) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:140) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:79) ~[?:1.8.0_144]
[10.104.42.108] out: at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:357) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.validator.Validator.validate(Validator.java:260) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1501) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.Handshaker$1.run(Handshaker.java:966) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.Handshaker$1.run(Handshaker.java:963) ~[?:1.8.0_144]
[10.104.42.108] out: at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
[10.104.42.108] out: at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416) ~[?:1.8.0_144]

It seems certificate expired,how can I solve this problem?
Thanks!

Hi.
Did you do any action to the Search Guard, Elasticsearch or network connection recently that could cause problems? For example, installed a new version, changed configuration, etc.

[10.104.42.108] out: Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{roaP-g5qRri23cL9i81YvQ}{localhost}{10.104.42.108:9300}]. This is not an error, will keep on trying …

Are you sure you can reach the Elasticsearch server transport port (10.104.42.108:9300) from the machine you run the sgadmin from? You can check the connection with netcat, for example

$ nc -vz 127.0.0.1 9200
localhost [127.0.0.1] 9200 (wap-wsp) open
$ nc -vz 127.0.0.1 9300
localhost [127.0.0.1] 9300 (vrace) open

Show the cluster health and state

curl -k -u admin:admin -X GET https://localhost:9200/_cluster/health?pretty
curl -k -u admin:admin -X GET https://localhost:9200/_cluster/state?pretty

Thank you very much for your reply @srgbnd.
I am sure i can reach the Elasticsearch server transport por (9200 and 9300).

When i execute comand : “/opt/elasticsearch-5.5.1/plugins/search-guard-5/tools/sgadmin.sh -cd /opt/elasticsearch-5.5.1/plugins/search-guard-5/sgconfig -cn megauth-es -ks /opt/elasticsearch-5.5.1/config/kirk.jks -ts /opt/elasticsearch-5.5.1/config/truststore.jks -nhnv” to update config of access to elasticsearch, the above error will appear and the es.log as following:
[2021-01-09T22:25:13,473][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [megauth-es-master1] SSL Problem Received fatal alert: certificate_unknown
javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634) ~[?:?]
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_144]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:254) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1156) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.11.Final.jar:4.1.11.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
[2021-01-09T22:25:18,297][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [megauth-es-master1] SSL Problem Received fatal alert: certificate_unknown
javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634) ~[?:?]
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083) ~[?:?]

I have generated TLS certificate by using Online TLS certificate generator, and update the files of CN=node-keystore.jks,CN=sgadmin-keystore.jks,truststore.jks,the problem is solved:
es@iZ2ze31n0q38u2u2vehkg7Z:/opt/elasticsearch-5.5.1/plugins/search-guard-5/tools$ /opt/elasticsearch-5.5.1/plugins/search-guard-5/tools/sgadmin.sh -cd /opt/elasticsearch-5.5.1/plugins/search-guard-5/sgconfig -cn megauth-es -ks /opt/elasticsearch-5.5.1/config/CN=sgadmin-keystore.jks -kspass xxxxxxxxx -ts /opt/elasticsearch-5.5.1/config/truststore-new.jks -tspass xxxxxxxx -nhnv
Search Guard Admin v5
Will connect to localhost:9300 … done

LICENSE NOTICE Search Guard

If you use one or more of the following features in production
make sure you have a valid Search Guard license
(See https://floragunn.com/searchguard-validate-license)

  • Kibana Multitenancy
  • LDAP authentication/authorization
  • Active Directory authentication/authorization
  • REST Management API
  • JSON Web Token (JWT) authentication/authorization
  • Kerberos authentication/authorization
  • Document- and Fieldlevel Security (DLS/FLS)
  • Auditlogging

In case of any doubt mail to sales@floragunn.com
###################################
Contacting elasticsearch cluster ‘megauth-es’ and wait for YELLOW clusterstate …
Clustername: megauth-es
Clusterstate: GREEN
Number of nodes: 3
Number of data nodes: 3
searchguard index already exists, so we do not need to create one.
Populate config from /opt/elasticsearch-5.5.1/plugins/search-guard-5/sgconfig/
Will update ‘config’ with /opt/elasticsearch-5.5.1/plugins/search-guard-5/sgconfig/sg_config.yml
SUCC: Configuration for ‘config’ created or updated
Will update ‘roles’ with /opt/elasticsearch-5.5.1/plugins/search-guard-5/sgconfig/sg_roles.yml
SUCC: Configuration for ‘roles’ created or updated
Will update ‘rolesmapping’ with /opt/elasticsearch-5.5.1/plugins/search-guard-5/sgconfig/sg_roles_mapping.yml
SUCC: Configuration for ‘rolesmapping’ created or updated
Will update ‘internalusers’ with /opt/elasticsearch-5.5.1/plugins/search-guard-5/sgconfig/sg_internal_users.yml
SUCC: Configuration for ‘internalusers’ created or updated
Will update ‘actiongroups’ with /opt/elasticsearch-5.5.1/plugins/search-guard-5/sgconfig/sg_action_groups.yml
SUCC: Configuration for ‘actiongroups’ created or updated
Done with success

I want to know if the certificate has a life cycle,If so, what is the life cycle?
The certificate generated by online TLS certificate generator cannot be used in a production environment?
What’s the difference betwee online TLS certificate generator and TLS tool?

Sorry, I was inattentive when read your question the first time. I see the first error when you use -cn faceid-es is about an expired certificate.

Caused by: java.security.cert.CertificateExpiredException: NotAfter: Wed Sep 02 02:38:10 CST 2020

And the second error when you use -cn megauth-es is about an untrusted certificate.

javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

I want to know if the certificate has a life cycle,If so, what is the life cycle?

You need to create or obtain new TLS certificates instead of the expired ones. All the certificates applied to the Elasticsearch nodes must be signed by the same root CA.

The certificate generated by online TLS certificate generator cannot be used in a production environment?
What’s the difference betwee online TLS certificate generator and TLS tool?

I advise against using the online TLS certificate generator for the production certificates. If you use the generator, it automatically creates the certificates and sends a link that you can use to download the certificates. It means that anyone who read your mail and got the link can download the certificates. The generator is for testing purposes.

I advise using the Search Guard TLS tool. You can tweak the certificate options in any way you want, and you generate the certificates locally in a secure environment.

Thank you very much for your reply @srgbnd.
Let me explain that the faceid-es cluster is my prod environment,megauth-es cluster is my test environment, and their certification have expired.

1.At the same time, i want to confirm which files need to be updated? In my application i only use es and kibana service. If only update three files: truststore.jks,CN=node-keystore.jks and CN=sgadmin-keystore.jks?
2.If the certificate expires and I restart the cluster, can the cluster still be started?
3.If i just need to create a node certificate(CN=sgadmin-keystore.jks) and all machines use the same node certificate?
4.To update the certificate, you need to shut down all machines first, and then start the master nodes and data nodes in sequence ?(In my prod environment, the cluster contains 5 master nodes and 20 data nodes)

Thank you very much and look forward to your reply!

1.At the same time, i want to confirm which files need to be updated? In my application i only use es and kibana service. If only update three files: truststore.jks,CN=node-keystore.jks and CN=sgadmin-keystore.jks?

If you want to use the Search Guard TLS tool, you need to be aware that it generates the PEM format certificates, not JKS. And the configuration options for PEM differs from the configuration options for JKS. A typical configuration looks like the following
elasticsearch.yml

searchguard.ssl.transport.pemcert_filepath: node1.pem
searchguard.ssl.transport.pemkey_filepath: node1.key
searchguard.ssl.transport.pemkey_password: QxoQyHTsXipE
searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: node1_http.pem
searchguard.ssl.http.pemkey_filepath: node1_http.key
searchguard.ssl.http.pemkey_password: AqrumRQqSIqo
searchguard.ssl.http.pemtrustedcas_filepath: root-ca.pem
searchguard.nodes_dn:
- CN=node1.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com
- CN=node2.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com
- CN=node3.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com
searchguard.authcz.admin_dn:
- CN=kirk.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

Here you can read about the options Configuring TLS | Security for Elasticsearch | Search Guard

If you want to stick with JKS, you can look for a howto on the Web, for example, To Generate a Certificate by Using keytool (Oracle GlassFish Server 3.0.1 Administration Guide), https://stackoverflow.com/a/11315084/2393924

2.If the certificate expires and I restart the cluster, can the cluster still be started?

No, it can’t. As a side note, Search Guard v7.x-35.0.0 and newer supports the hot-reload for certificates to change the certificates and keys without the cluster reload.

3.If i just need to create a node certificate(CN=sgadmin-keystore.jks) and all machines use the same node certificate?

It is not secure. Every node needs to have its own certificate. All the node certificates must be signed by the same root CA or intermediate CA.

4.To update the certificate, you need to shut down all machines first, and then start the master nodes and data nodes in sequence ?(In my prod environment, the cluster contains 5 master nodes and 20 data nodes)

Yes, start the master nodes first. You may want to temporarily disable the shards allocation before the restart. Look at the Elasticsearch guide for this Full cluster restart upgrade | Elasticsearch Reference [5.5] | Elastic But instead of the upgrade, change the certificates.