java.security.cert.CertificateExpiredException: NotAfter

fly · January 7, 2021, 3:35am

Hi:
My es version [10.104.42.108] [10.104.42.108] [10.104.42.108] [10.104.42.108] out:
[10.104.42.108] [10.104.42.108] out:
[10.104.42.108] [10.104.42.108] [10.104.42.108] [10.104.42.108] out:
[10.104.42.108] [10.104.42.108] [10.104.42.108] [10.104.42.108] [10.104.42.108] [10.104.42.108] [10.104.42.108] [10.104.42.108] out: * Auditlogging
[10.104.42.108] out:
[10.104.42.108] [10.104.42.108] [10.104.42.108] [10.104.42.108] [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] [10.104.42.108] [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: … 18 more
[10.104.42.108] [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: … 18 more
[10.104.42.108] [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: … 18 more
[10.104.42.108] [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: [10.104.42.108] out: is 5.5.1,search guard version is v5.Es and search guard have been running normally about two years，but suddenly one day when I execute the following command in my running es cluster to update es access rights：“/opt/elasticsearch-5.5.1/plugins/search-guard-5/tools/sgadmin.sh -cd /opt/elasticsearch-5.5.1/plugins/search-guard-5/sgconfig -cn faceid-es -ks /opt/elasticsearch-5.5.1/config/kirk.jks -ts /opt/elasticsearch-5.5.1/config/truststore.jks -nhnv”，I get the error log，the log as fllow:
run: tools/sgadmin_demo.sh
out: Search Guard Admin v5
out: Will connect to localhost:9300 … done
out: ### LICENSE NOTICE Search Guard ###
out: If you use one or more of the following features in production
out: make sure you have a valid Search Guard license
out: (See Licensing | Search Guard Community, Enterprise and Compliance Edition)
out: * Kibana Multitenancy
out: * LDAP authentication/authorization
out: * Active Directory authentication/authorization
out: * REST Management API
out: * JSON Web Token (JWT) authentication/authorization
out: * Kerberos authentication/authorization
out: * Document- and Fieldlevel Security (DLS/FLS)
out: In case of any doubt mail to sales@floragunn.com
out: ###################################
out: Contacting elasticsearch cluster ‘facepp-es’ and wait for YELLOW clusterstate …
out: Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{roaP-g5qRri23cL9i81YvQ}{localhost}{10.104.42.108:9300}]. This is not an error, will keep on trying …
* Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
* Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
* If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
* Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
out: 12:34:32.302 [elasticsearch[client][transport_client_boss][T#1]] ERROR com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport - SSL Problem General SSLEngine problem
out: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1478) ~[?:1.8.0_144]
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[?:1.8.0_144]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[?:1.8.0_144]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:1.8.0_144]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_144]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:254) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1156) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.11.Final.jar:4.1.11.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
out: Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:1.8.0_144]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) ~[?:1.8.0_144]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304) ~[?:1.8.0_144]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) ~[?:1.8.0_144]
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514) ~[?:1.8.0_144]
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[?:1.8.0_144]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026) ~[?:1.8.0_144]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:966) ~[?:1.8.0_144]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:963) ~[?:1.8.0_144]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416) ~[?:1.8.0_144]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1295) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1208) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
out: Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: timestamp check failed
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:362) ~[?:1.8.0_144]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270) ~[?:1.8.0_144]
at sun.security.validator.Validator.validate(Validator.java:260) ~[?:1.8.0_144]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:1.8.0_144]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[?:1.8.0_144]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:1.8.0_144]
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1501) ~[?:1.8.0_144]
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[?:1.8.0_144]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026) ~[?:1.8.0_144]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:966) ~[?:1.8.0_144]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:963) ~[?:1.8.0_144]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416) ~[?:1.8.0_144]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1295) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1208) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
out: Caused by: java.security.cert.CertPathValidatorException: timestamp check failed
at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135) ~[?:1.8.0_144]
at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:223) ~[?:1.8.0_144]
at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:140) ~[?:1.8.0_144]
at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:79) ~[?:1.8.0_144]
at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292) ~[?:1.8.0_144]
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:357) ~[?:1.8.0_144]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270) ~[?:1.8.0_144]
at sun.security.validator.Validator.validate(Validator.java:260) ~[?:1.8.0_144]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:1.8.0_144]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[?:1.8.0_144]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:1.8.0_144]
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1501) ~[?:1.8.0_144]
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[?:1.8.0_144]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026) ~[?:1.8.0_144]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:966) ~[?:1.8.0_144]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:963) ~[?:1.8.0_144]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416) ~[?:1.8.0_144]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1295) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1208) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
out: Caused by: java.security.cert.CertificateExpiredException: NotAfter: Wed Sep 02 02:38:10 CST 2020
at sun.security.x509.CertificateValidity.valid(CertificateValidity.java:274) ~[?:1.8.0_144]
at sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:629) ~[?:1.8.0_144]
at sun.security.provider.certpath.BasicChecker.verifyTimestamp(BasicChecker.java:190) ~[?:1.8.0_144]
at sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:144) ~[?:1.8.0_144]
at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125) ~[?:1.8.0_144]
at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:223) ~[?:1.8.0_144]
at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:140) ~[?:1.8.0_144]
at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:79) ~[?:1.8.0_144]
at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292) ~[?:1.8.0_144]
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:357) ~[?:1.8.0_144]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270) ~[?:1.8.0_144]
at sun.security.validator.Validator.validate(Validator.java:260) ~[?:1.8.0_144]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:1.8.0_144]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[?:1.8.0_144]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:1.8.0_144]
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1501) ~[?:1.8.0_144]
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[?:1.8.0_144]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026) ~[?:1.8.0_144]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:966) ~[?:1.8.0_144]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:963) ~[?:1.8.0_144]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416) ~[?:1.8.0_144]

It seems certificate expired，how can I solve this problem?
Thanks!

srgbnd · January 7, 2021, 4:46pm

Hi.
Did you do any action to the Search Guard, Elasticsearch or network connection recently that could cause problems? For example, installed a new version, changed configuration, etc.

[10.104.42.108] out: Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{roaP-g5qRri23cL9i81YvQ}{localhost}{10.104.42.108:9300}]. This is not an error, will keep on trying …

Are you sure you can reach the Elasticsearch server transport port (10.104.42.108:9300) from the machine you run the sgadmin from? You can check the connection with netcat, for example

$ nc -vz 127.0.0.1 9200
localhost [127.0.0.1] 9200 (wap-wsp) open
$ nc -vz 127.0.0.1 9300
localhost [127.0.0.1] 9300 (vrace) open

Show the cluster health and state

curl -k -u admin:admin -X GET https://localhost:9200/_cluster/health?pretty
curl -k -u admin:admin -X GET https://localhost:9200/_cluster/state?pretty

fly · January 9, 2021, 2:47pm

Thank you very much for your reply @srgbnd.
I am sure i can reach the Elasticsearch server transport por (9200 and 9300).

When i execute comand : “/opt/elasticsearch-5.5.1/plugins/search-guard-5/tools/sgadmin.sh -cd /opt/elasticsearch-5.5.1/plugins/search-guard-5/sgconfig -cn megauth-es -ks /opt/elasticsearch-5.5.1/config/kirk.jks -ts /opt/elasticsearch-5.5.1/config/truststore.jks -nhnv” to update config of access to elasticsearch, the above error will appear and the es.log as following:
[2021-01-09T22:25:13,473][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [megauth-es-master1] SSL Problem Received fatal alert: certificate_unknown
javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634) ~[?:?]
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_144]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:254) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1156) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.11.Final.jar:4.1.11.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
[2021-01-09T22:25:18,297][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [megauth-es-master1] SSL Problem Received fatal alert: certificate_unknown
javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634) ~[?:?]
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083) ~[?:?]

I have generated TLS certificate by using Online TLS certificate generator, and update the files of CN=node-keystore.jks,CN=sgadmin-keystore.jks,truststore.jks，the problem is solved:
es@iZ2ze31n0q38u2u2vehkg7Z:/opt/elasticsearch-5.5.1/plugins/search-guard-5/tools$ /opt/elasticsearch-5.5.1/plugins/search-guard-5/tools/sgadmin.sh -cd /opt/elasticsearch-5.5.1/plugins/search-guard-5/sgconfig -cn megauth-es -ks /opt/elasticsearch-5.5.1/config/CN=sgadmin-keystore.jks -kspass xxxxxxxxx -ts /opt/elasticsearch-5.5.1/config/truststore-new.jks -tspass xxxxxxxx -nhnv
Search Guard Admin v5
Will connect to localhost:9300 … done

LICENSE NOTICE Search Guard

If you use one or more of the following features in production
make sure you have a valid Search Guard license
(See Licensing | Search Guard Community, Enterprise and Compliance Edition)

Kibana Multitenancy
LDAP authentication/authorization
Active Directory authentication/authorization
REST Management API
JSON Web Token (JWT) authentication/authorization
Kerberos authentication/authorization
Document- and Fieldlevel Security (DLS/FLS)
Auditlogging

In case of any doubt mail to sales@floragunn.com
###################################
Contacting elasticsearch cluster ‘megauth-es’ and wait for YELLOW clusterstate …
Clustername: megauth-es
Clusterstate: GREEN
Number of nodes: 3
Number of data nodes: 3
searchguard index already exists, so we do not need to create one.
Populate config from /opt/elasticsearch-5.5.1/plugins/search-guard-5/sgconfig/
Will update ‘config’ with /opt/elasticsearch-5.5.1/plugins/search-guard-5/sgconfig/sg_config.yml
SUCC: Configuration for ‘config’ created or updated
Will update ‘roles’ with /opt/elasticsearch-5.5.1/plugins/search-guard-5/sgconfig/sg_roles.yml
SUCC: Configuration for ‘roles’ created or updated
Will update ‘rolesmapping’ with /opt/elasticsearch-5.5.1/plugins/search-guard-5/sgconfig/sg_roles_mapping.yml
SUCC: Configuration for ‘rolesmapping’ created or updated
Will update ‘internalusers’ with /opt/elasticsearch-5.5.1/plugins/search-guard-5/sgconfig/sg_internal_users.yml
SUCC: Configuration for ‘internalusers’ created or updated
Will update ‘actiongroups’ with /opt/elasticsearch-5.5.1/plugins/search-guard-5/sgconfig/sg_action_groups.yml
SUCC: Configuration for ‘actiongroups’ created or updated
Done with success

I want to know if the certificate has a life cycle，If so, what is the life cycle？
The certificate generated by online TLS certificate generator cannot be used in a production environment?
What’s the difference betwee online TLS certificate generator and TLS tool?

srgbnd · January 11, 2021, 1:51pm

Sorry, I was inattentive when read your question the first time. I see the first error when you use -cn faceid-es is about an expired certificate.

Caused by: java.security.cert.CertificateExpiredException: NotAfter: Wed Sep 02 02:38:10 CST 2020

And the second error when you use -cn megauth-es is about an untrusted certificate.

javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

I want to know if the certificate has a life cycle，If so, what is the life cycle？

You need to create or obtain new TLS certificates instead of the expired ones. All the certificates applied to the Elasticsearch nodes must be signed by the same root CA.

The certificate generated by online TLS certificate generator cannot be used in a production environment?
What’s the difference betwee online TLS certificate generator and TLS tool?

I advise against using the online TLS certificate generator for the production certificates. If you use the generator, it automatically creates the certificates and sends a link that you can use to download the certificates. It means that anyone who read your mail and got the link can download the certificates. The generator is for testing purposes.

I advise using the Search Guard TLS tool. You can tweak the certificate options in any way you want, and you generate the certificates locally in a secure environment.

fly · January 12, 2021, 4:11am

Thank you very much for your reply @srgbnd.
Let me explain that the faceid-es cluster is my prod environment，megauth-es cluster is my test environment, and their certification have expired.

1.At the same time, i want to confirm which files need to be updated? In my application i only use es and kibana service. If only update three files: truststore.jks,CN=node-keystore.jks and CN=sgadmin-keystore.jks?
2.If the certificate expires and I restart the cluster, can the cluster still be started?
3.If i just need to create a node certificate(CN=sgadmin-keystore.jks) and all machines use the same node certificate?
4.To update the certificate, you need to shut down all machines first, and then start the master nodes and data nodes in sequence ?(In my prod environment, the cluster contains 5 master nodes and 20 data nodes)

Thank you very much and look forward to your reply!

srgbnd · January 12, 2021, 9:53am

1.At the same time, i want to confirm which files need to be updated? In my application i only use es and kibana service. If only update three files: truststore.jks,CN=node-keystore.jks and CN=sgadmin-keystore.jks?

If you want to use the Search Guard TLS tool, you need to be aware that it generates the PEM format certificates, not JKS. And the configuration options for PEM differs from the configuration options for JKS. A typical configuration looks like the following
elasticsearch.yml

searchguard.ssl.transport.pemcert_filepath: node1.pem
searchguard.ssl.transport.pemkey_filepath: node1.key
searchguard.ssl.transport.pemkey_password: QxoQyHTsXipE
searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: node1_http.pem
searchguard.ssl.http.pemkey_filepath: node1_http.key
searchguard.ssl.http.pemkey_password: AqrumRQqSIqo
searchguard.ssl.http.pemtrustedcas_filepath: root-ca.pem
searchguard.nodes_dn:
- CN=node1.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com
- CN=node2.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com
- CN=node3.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com
searchguard.authcz.admin_dn:
- CN=kirk.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

Here you can read about the options Configuring TLS | Security for Elasticsearch | Search Guard

If you want to stick with JKS, you can look for a howto on the Web, for example, To Generate a Certificate by Using keytool (Oracle GlassFish Server 3.0.1 Administration Guide), security - generate key and certificate using keytool - Stack Overflow

2.If the certificate expires and I restart the cluster, can the cluster still be started?

No, it can’t. As a side note, Search Guard v7.x-35.0.0 and newer supports the hot-reload for certificates to change the certificates and keys without the cluster reload.

3.If i just need to create a node certificate(CN=sgadmin-keystore.jks) and all machines use the same node certificate?

It is not secure. Every node needs to have its own certificate. All the node certificates must be signed by the same root CA or intermediate CA.

4.To update the certificate, you need to shut down all machines first, and then start the master nodes and data nodes in sequence ?(In my prod environment, the cluster contains 5 master nodes and 20 data nodes)

Yes, start the master nodes first. You may want to temporarily disable the shards allocation before the restart. Look at the Elasticsearch guide for this Full cluster restart upgrade | Elasticsearch Reference [5.5] | Elastic But instead of the upgrade, change the certificates.

fly · February 2, 2021, 6:45am

Thank you very much for your reply @srgbnd

system · February 23, 2021, 6:46am

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
How to access new certificates for Search Guard 5.4.0 Search Guard	1	424	May 25, 2018
replace expired certificate failed Search Guard	0	517	February 14, 2019
Searchguard not failing for expired certificates Search Guard	14	1543	August 14, 2020
Certificate issue when deploying last release ELK + Search Guard Search Guard	4	2151	January 17, 2019
SSL Problem Received fatal alert: certificate_unknown and Caused by: java.security.cert.CertificateExpiredException: NotAfter Search Guard	3	2754	November 7, 2019

java.security.cert.CertificateExpiredException: NotAfter

LICENSE NOTICE Search Guard

Related topics