Demo certificates rollover

Search Guard
1

Hi,

the demo certificates that are shipped with Search Guard are about to expire on 4th of May. Anyone still running a PoC with these demo certificates should replace them soon.

We have prepared a new set of certificates, and will also take this as an opportunity to replace the old root CA for security reasons. The new certificates can be downloaded from here:

https://downloads.search-guard.com/tls-demo-certificates

The zip contains the following files:

  • root-ca.pem: The new root CA
  • esnode.pem: Node certificate
  • esnode-key.pem: Private key of the node certificate, no password
  • kirk.pem: Admin certificate
  • kirk-key.pem: Private key of the admin certificate, no password
  • spock.pem: Client certificate
  • spock-key.pem: Private key of the client certificate, no password

The certificates are a drop-in replacement for the old ones. This means you can simply unzip them in the config directory of Elasticsearch (overwriting the old ones) and restart the node(s). No further configuration changes are required.

In addition, we will ship a point release 22.1 soon containing the new certs.

Sorry for any inconvenience!

Thanks,

Jochen

Ā·Ā·Ā·

Search Guard (Ā®) is an Elasticsearch plugin that offers encryption, authentication and authorisation.

Coded with love in Berlin, Denmark, Sweden and the US.

Search Guard is a trademark of floragunn GmbH, registered in the U.S. and in other countries.

Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries.

2

Hi - I just tried replacing the files as instructed and ElasticSearch will no longer start:

[2018-04-25T04:40:50,191][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [node-1] SSL Problem General SSLEngine problem
javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529) ~[?:?]
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_162]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) ~[?:?]
at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1979) ~[?:?]
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:237) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_162]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]
ā€¦ 19 more
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:362) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:279) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:130) ~[?:?]
at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1966) ~[?:?]
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:237) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_162]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]
ā€¦ 19 more
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:153) ~[?:?]
at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:79) ~[?:?]
at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292) ~[?:1.8.0_162]
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:357) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:279) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:130) ~[?:?]
at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1966) ~[?:?]
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:237) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_162]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]
ā€¦ 19 more

``

Do you know whatā€™s gone wrong here?

Ā·Ā·Ā·

On Tuesday, 24 April 2018 07:51:45 UTC+3, Jochen Kressin wrote:

Hi,

the demo certificates that are shipped with Search Guard are about to expire on 4th of May. Anyone still running a PoC with these demo certificates should replace them soon.

We have prepared a new set of certificates, and will also take this as an opportunity to replace the old root CA for security reasons. The new certificates can be downloaded from here:

https://downloads.search-guard.com/tls-demo-certificates

The zip contains the following files:

  • root-ca.pem: The new root CA
  • esnode.pem: Node certificate
  • esnode-key.pem: Private key of the node certificate, no password
  • kirk.pem: Admin certificate
  • kirk-key.pem: Private key of the admin certificate, no password
  • spock.pem: Client certificate
  • spock-key.pem: Private key of the client certificate, no password

The certificates are a drop-in replacement for the old ones. This means you can simply unzip them in the config directory of Elasticsearch (overwriting the old ones) and restart the node(s). No further configuration changes are required.

In addition, we will ship a point release 22.1 soon containing the new certs.

Sorry for any inconvenience!

Thanks,

Jochen

Search Guard (Ā®) is an Elasticsearch plugin that offers encryption, authentication and authorisation.

Coded with love in Berlin, Denmark, Sweden and the US.

Search Guard is a trademark of floragunn GmbH, registered in the U.S. and in other countries.

Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries.

3

Have you replaced all the files on all node - including the root-ca.pem? We also exchanged the Root CA, so it needs to be updated as well.

And, do you by chance have different certificates for transport and HTTP?

Oh, and what exact version do you use - should not make a difference, but always useful to know.

Ā·Ā·Ā·

On Tuesday, April 24, 2018 at 9:42:50 PM UTC-7, Ross Coundon wrote:

Hi - I just tried replacing the files as instructed and ElasticSearch will no longer start:

[2018-04-25T04:40:50,191][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [node-1] SSL Problem General SSLEngine problem
javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529) ~[?:?]
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_162]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) ~[?:?]
at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1979) ~[?:?]
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:237) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_162]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]
ā€¦ 19 more
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:362) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:279) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:130) ~[?:?]
at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1966) ~[?:?]
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:237) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_162]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]
ā€¦ 19 more
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:153) ~[?:?]
at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:79) ~[?:?]
at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292) ~[?:1.8.0_162]
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:357) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:279) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:130) ~[?:?]
at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1966) ~[?:?]
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:237) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_162]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]
ā€¦ 19 more

``

Do you know whatā€™s gone wrong here?

On Tuesday, 24 April 2018 07:51:45 UTC+3, Jochen Kressin wrote:

Hi,

the demo certificates that are shipped with Search Guard are about to expire on 4th of May. Anyone still running a PoC with these demo certificates should replace them soon.

We have prepared a new set of certificates, and will also take this as an opportunity to replace the old root CA for security reasons. The new certificates can be downloaded from here:

https://downloads.search-guard.com/tls-demo-certificates

The zip contains the following files:

  • root-ca.pem: The new root CA
  • esnode.pem: Node certificate
  • esnode-key.pem: Private key of the node certificate, no password
  • kirk.pem: Admin certificate
  • kirk-key.pem: Private key of the admin certificate, no password
  • spock.pem: Client certificate
  • spock-key.pem: Private key of the client certificate, no password

The certificates are a drop-in replacement for the old ones. This means you can simply unzip them in the config directory of Elasticsearch (overwriting the old ones) and restart the node(s). No further configuration changes are required.

In addition, we will ship a point release 22.1 soon containing the new certs.

Sorry for any inconvenience!

Thanks,

Jochen

Search Guard (Ā®) is an Elasticsearch plugin that offers encryption, authentication and authorisation.

Coded with love in Berlin, Denmark, Sweden and the US.

Search Guard is a trademark of floragunn GmbH, registered in the U.S. and in other countries.

Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries.

4

Hi - yes, thereā€™s only a single node at present while we experiment. I unpacked the zip file over the top of the files and accepted all replacements and the root-ca.pqm was replaced too.
Iā€™m using the following configuration:

searchguard.ssl.transport.pemcert_filepath: esnode.pem

searchguard.ssl.transport.pemkey_filepath: esnode-key.pem

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: esnode.pem

searchguard.ssl.http.pemkey_filepath: esnode-key.pem

searchguard.ssl.http.pemtrustedcas_filepath: root-ca.pem

searchguard.allow_unsafe_democertificates: true

searchguard.allow_default_init_sgindex: true

searchguard.authcz.admin_dn:

  • CN=kirk,OU=client,O=client,L=test, C=de

Version of ES is:

ā€œversionā€: {
ā€œnumberā€: ā€œ6.2.3ā€,
ā€œbuild_hashā€: ā€œc59ff00ā€,
ā€œbuild_dateā€: ā€œ2018-03-13T10:06:29.741383Zā€,
ā€œbuild_snapshotā€: false,
ā€œlucene_versionā€: ā€œ7.2.1ā€,
ā€œminimum_wire_compatibility_versionā€: ā€œ5.6.0ā€,
ā€œminimum_index_compatibility_versionā€: ā€œ5.0.0ā€
},

``

and search-guard is:

version=6.2.3-22.0

``

Ā·Ā·Ā·

On Wednesday, 25 April 2018 08:06:34 UTC+3, Jochen Kressin wrote:

Have you replaced all the files on all node - including the root-ca.pem? We also exchanged the Root CA, so it needs to be updated as well.

And, do you by chance have different certificates for transport and HTTP?

Oh, and what exact version do you use - should not make a difference, but always useful to know.

On Tuesday, April 24, 2018 at 9:42:50 PM UTC-7, Ross Coundon wrote:

Hi - I just tried replacing the files as instructed and ElasticSearch will no longer start:

[2018-04-25T04:40:50,191][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [node-1] SSL Problem General SSLEngine problem
javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529) ~[?:?]
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_162]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) ~[?:?]
at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1979) ~[?:?]
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:237) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_162]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]
ā€¦ 19 more
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:362) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:279) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:130) ~[?:?]
at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1966) ~[?:?]
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:237) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_162]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]
ā€¦ 19 more
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:153) ~[?:?]
at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:79) ~[?:?]
at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292) ~[?:1.8.0_162]
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:357) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:279) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:130) ~[?:?]
at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1966) ~[?:?]
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:237) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_162]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]
ā€¦ 19 more

``

Do you know whatā€™s gone wrong here?

On Tuesday, 24 April 2018 07:51:45 UTC+3, Jochen Kressin wrote:

Hi,

the demo certificates that are shipped with Search Guard are about to expire on 4th of May. Anyone still running a PoC with these demo certificates should replace them soon.

We have prepared a new set of certificates, and will also take this as an opportunity to replace the old root CA for security reasons. The new certificates can be downloaded from here:

https://downloads.search-guard.com/tls-demo-certificates

The zip contains the following files:

  • root-ca.pem: The new root CA
  • esnode.pem: Node certificate
  • esnode-key.pem: Private key of the node certificate, no password
  • kirk.pem: Admin certificate
  • kirk-key.pem: Private key of the admin certificate, no password
  • spock.pem: Client certificate
  • spock-key.pem: Private key of the client certificate, no password

The certificates are a drop-in replacement for the old ones. This means you can simply unzip them in the config directory of Elasticsearch (overwriting the old ones) and restart the node(s). No further configuration changes are required.

In addition, we will ship a point release 22.1 soon containing the new certs.

Sorry for any inconvenience!

Thanks,

Jochen

Search Guard (Ā®) is an Elasticsearch plugin that offers encryption, authentication and authorisation.

Coded with love in Berlin, Denmark, Sweden and the US.

Search Guard is a trademark of floragunn GmbH, registered in the U.S. and in other countries.

Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries.

5

Strange, an upgrade from a vanilla 6.2.3-22.0 to the new certificates seems to work without problems here. This one:

Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException:Path does not chain with any of the trust anchors

``

That usually means the node certificate (and intermediate certificates if present) cannot be validated against the configured root CA. The new demo certificates do not contain any intermediate cert, so the chain is very simple:

esnode.pem ā†’ root-ca.pem

``

If you have OpenSSL installed it would be very helpful to see the output of:

openssl x509 -in ./esnode.pem -text -noout

``

and

penssl x509 -in ./root-ca.pem -text -noout

``

This will print out the details about the certificates, and the trust chain.

Also, which JDK do you use?

Thanks!

Ā·Ā·Ā·

On Tuesday, April 24, 2018 at 11:11:10 PM UTC-7, Ross Coundon wrote:

Hi - yes, thereā€™s only a single node at present while we experiment. I unpacked the zip file over the top of the files and accepted all replacements and the root-ca.pqm was replaced too.
Iā€™m using the following configuration:

searchguard.ssl.transport.pemcert_filepath: esnode.pem

searchguard.ssl.transport.pemkey_filepath: esnode-key.pem

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: esnode.pem

searchguard.ssl.http.pemkey_filepath: esnode-key.pem

searchguard.ssl.http.pemtrustedcas_filepath: root-ca.pem

searchguard.allow_unsafe_democertificates: true

searchguard.allow_default_init_sgindex: true

searchguard.authcz.admin_dn:

  • CN=kirk,OU=client,O=client,L=test, C=de

Version of ES is:

ā€œversionā€: {
ā€œnumberā€: ā€œ6.2.3ā€,
ā€œbuild_hashā€: ā€œc59ff00ā€,
ā€œbuild_dateā€: ā€œ2018-03-13T10:06:29.741383Zā€,
ā€œbuild_snapshotā€: false,
ā€œlucene_versionā€: ā€œ7.2.1ā€,
ā€œminimum_wire_compatibility_versionā€: ā€œ5.6.0ā€,
ā€œminimum_index_compatibility_versionā€: ā€œ5.0.0ā€
},

``

and search-guard is:

version=6.2.3-22.0

``

On Wednesday, 25 April 2018 08:06:34 UTC+3, Jochen Kressin wrote:

Have you replaced all the files on all node - including the root-ca.pem? We also exchanged the Root CA, so it needs to be updated as well.

And, do you by chance have different certificates for transport and HTTP?

Oh, and what exact version do you use - should not make a difference, but always useful to know.

On Tuesday, April 24, 2018 at 9:42:50 PM UTC-7, Ross Coundon wrote:

Hi - I just tried replacing the files as instructed and ElasticSearch will no longer start:

[2018-04-25T04:40:50,191][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [node-1] SSL Problem General SSLEngine problem
javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529) ~[?:?]
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_162]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) ~[?:?]
at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1979) ~[?:?]
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:237) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_162]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]
ā€¦ 19 more
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:362) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:279) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:130) ~[?:?]
at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1966) ~[?:?]
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:237) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_162]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]
ā€¦ 19 more
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:153) ~[?:?]
at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:79) ~[?:?]
at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292) ~[?:1.8.0_162]
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:357) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:279) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:130) ~[?:?]
at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1966) ~[?:?]
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:237) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_162]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]
ā€¦ 19 more

``

Do you know whatā€™s gone wrong here?

On Tuesday, 24 April 2018 07:51:45 UTC+3, Jochen Kressin wrote:

Hi,

the demo certificates that are shipped with Search Guard are about to expire on 4th of May. Anyone still running a PoC with these demo certificates should replace them soon.

We have prepared a new set of certificates, and will also take this as an opportunity to replace the old root CA for security reasons. The new certificates can be downloaded from here:

https://downloads.search-guard.com/tls-demo-certificates

The zip contains the following files:

  • root-ca.pem: The new root CA
  • esnode.pem: Node certificate
  • esnode-key.pem: Private key of the node certificate, no password
  • kirk.pem: Admin certificate
  • kirk-key.pem: Private key of the admin certificate, no password
  • spock.pem: Client certificate
  • spock-key.pem: Private key of the client certificate, no password

The certificates are a drop-in replacement for the old ones. This means you can simply unzip them in the config directory of Elasticsearch (overwriting the old ones) and restart the node(s). No further configuration changes are required.

In addition, we will ship a point release 22.1 soon containing the new certs.

Sorry for any inconvenience!

Thanks,

Jochen

Search Guard (Ā®) is an Elasticsearch plugin that offers encryption, authentication and authorisation.

Coded with love in Berlin, Denmark, Sweden and the US.

Search Guard is a trademark of floragunn GmbH, registered in the U.S. and in other countries.

Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries.

6

Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1524368626614 (0x162eb7353b6)
Signature Algorithm: sha256WithRSAEncryption
Issuer: DC=com, DC=example, O=Example Com Inc., OU=Example Com Inc. Root CA, CN=Example Com Inc. Root CA
Validity
Not Before: Apr 22 03:43:47 2018 GMT
Not After : Apr 19 03:43:47 2028 GMT
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:96:be:7f:8e:fa:bc:b0:7e:03:02:b9:dd:b8:98:
07:25:30:37:ee:34:0f:c8:cc:22:8b:c6:5e:6e:b0:
81:3f:3e:f5:26:ec:f3:df:5d:0d:78:2d:f4:21:35:
05:ea:3a:e6:83:f5:f8:95:33:e1:ce:d4:1c:ca:c2:
63:77:8f:88:3b:78:72:27:47:57:31:10:da:0d:18:
a1:5a:d0:5a:fd:11:79:d4:bf:cb:1f:c3:2a:1b:3c:
3f:0d:4e:ef:5e:68:7e:d3:f9:de:9f:f6:8a:30:f9:
0e:27:c5:bf:57:8a:7e:48:45:1f:e9:70:9f:2f:ef:
31:23:71:7a:59:69:97:a3:71:25:38:89:56:74:3d:
1d:83:8b:81:fd:ad:f7:bd:48:4c:91:e7:02:eb:b1:
50:5e:3c:1d:cb:8d:a2:f5:b8:ae:1b:64:5d:e7:fc:
91:a0:0d:ed:c1:37:2d:4f:80:f5:3e:3b:e1:42:cd:
08:a9:04:14:f2:25:64:02:8d:de:22:4d:15:d5:6c:
c6:b4:d4:f8:25:01:1f:39:3b:dc:3a:35:70:29:04:
bc:96:74:64:58:e9:d1:9d:f2:f3:02:d8:fe:0a:96:
19:f1:95:c8:0f:65:d8:25:2a:78:86:4d:7f:9e:4f:
34:fb:46:cc:ea:ef:bc:e3:62:ba:2e:3c:bc:12:87:
d4:bb
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Authority Key Identifier:
keyid:92:35:0C:E0:0F:1E:2B:45:F6:4D:39:F3:7B:5F:A2:E6:12:97:40:73
DirName:/DC=com/DC=example/O=Example Com Inc./OU=Example Com Inc. Root CA/CN=Example Com Inc. Root CA
serial:01

        X509v3 Subject Key Identifier:
            AC:AF:EF:C6:66:16:35:4A:33:D8:3B:A4:C0:A8:9D:81:FB:15:50:47
        X509v3 Basic Constraints: critical
            CA:FALSE
        X509v3 Key Usage: critical
            Digital Signature, Non Repudiation, Key Encipherment
        X509v3 Extended Key Usage: critical
            TLS Web Server Authentication, TLS Web Client Authentication
        X509v3 Subject Alternative Name:
            Registered ID:1.2.3.4.5.5, DNS:node-0.example.com, DNS:localhost, IP Address:127.0.0.1
Signature Algorithm: sha256WithRSAEncryption
     83:8a:bb:25:ec:15:f1:af:d6:12:3f:2e:4a:5d:ff:bd:d0:36:
     a8:ea:25:dc:50:72:55:f9:ec:63:8c:58:d6:ce:33:91:f4:e7:
     ff:40:38:86:79:25:0b:16:50:b3:4a:37:be:da:1e:32:d1:af:
     8a:30:ab:68:c6:6e:97:3f:67:a9:00:77:e7:a2:6a:d9:1c:a6:
     76:ed:6c:6a:e1:2a:93:ad:a0:46:72:f0:ab:ac:97:09:1e:8b:
     1a:73:2c:33:48:49:26:e4:78:ba:57:cf:8c:49:23:51:13:30:
     df:d5:1b:c6:59:3f:56:e3:ce:51:f3:88:71:c3:bb:42:4d:67:
     a4:e7:37:32:ab:5f:30:86:30:2a:21:15:f7:a6:f2:f9:ca:36:
     72:94:9d:e3:10:32:f5:dd:de:bc:d4:68:08:2f:b5:fe:c7:73:
     62:d3:06:57:f8:7d:9c:d9:17:51:24:c1:d4:97:85:a3:00:d6:
     59:1f:1e:2a:8e:07:1b:60:78:32:f1:08:71:12:67:67:ea:81:
     5a:ac:59:7f:ad:de:a1:d0:7e:2b:dc:3d:6e:ad:c6:d0:f2:ac:
     53:d1:74:93:86:86:23:06:cd:3f:ed:7b:ff:64:90:0b:50:46:
     0a:53:6f:7b:24:61:d2:0e:39:43:95:d1:61:90:eb:49:09:94:
     58:40:cc:8b

and

Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1 (0x1)
Signature Algorithm: sha256WithRSAEncryption
Issuer: DC=com, DC=example, O=Example Com Inc., OU=Example Com Inc. Root CA, CN=Example Com Inc. Root CA
Validity
Not Before: Apr 22 03:43:46 2018 GMT
Not After : Apr 19 03:43:46 2028 GMT
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:af:ee:f8:60:11:3f:98:a7:9e:1a:57:2b:47:34:
ab:bb:35:4a:ed:55:4c:46:88:82:66:6b:f1:55:88:
e9:2f:1a:99:fe:5c:53:79:2d:57:b5:93:f5:2b:95:
4e:c0:26:da:2d:80:e4:ff:82:b7:0e:e2:66:47:e7:
1d:69:6c:0b:71:e1:3d:47:1d:ea:6b:f3:19:9e:26:
a3:19:da:98:ce:eb:f9:af:68:b5:1a:77:a3:06:28:
19:2b:57:ca:55:53:42:eb:00:8d:ba:bd:76:8f:02:
31:5e:21:70:14:de:a4:27:7e:d3:0d:2f:e2:1e:94:
95:75:3c:c6:38:63:d7:17:94:23:3e:03:29:b4:60:
7f:7e:aa:d2:bb:f8:54:85:f8:e9:7e:f6:ac:c2:52:
11:32:8e:4b:1b:b0:2e:4a:2f:d5:93:95:6d:f4:a5:
3d:ac:a0:5c:8c:6a:b0:75:65:8f:58:8c:91:84:5b:
42:66:93:89:be:97:58:72:9f:32:26:c3:6a:a0:de:
8c:e8:6e:92:40:a3:ce:9a:6d:19:93:8f:15:0f:34:
d3:65:2d:4d:33:6f:d5:38:9a:2b:19:23:31:02:4d:
c3:3e:a3:7d:9e:77:c2:cd:df:87:52:34:45:64:fa:
59:f3:38:a1:e1:51:16:7c:85:46:67:38:b9:84:d4:
80:09
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Authority Key Identifier:
keyid:92:35:0C:E0:0F:1E:2B:45:F6:4D:39:F3:7B:5F:A2:E6:12:97:40:73

        X509v3 Subject Key Identifier:
            92:35:0C:E0:0F:1E:2B:45:F6:4D:39:F3:7B:5F:A2:E6:12:97:40:73
        X509v3 Key Usage: critical
            Digital Signature, Certificate Sign, CRL Sign
Signature Algorithm: sha256WithRSAEncryption
     68:40:7b:f0:b1:1d:f8:84:63:b6:9b:ca:95:47:d9:d0:e4:a9:
     68:e4:76:0f:c9:de:b2:48:a3:5c:4f:7e:8e:67:80:10:7d:a0:
     86:b2:4f:92:79:c4:e2:df:94:05:44:72:f7:83:6a:9f:7c:40:
     f8:b4:a4:74:44:13:46:41:28:22:2d:ab:e6:1c:60:a1:dd:8a:
     43:ba:92:aa:db:18:61:11:e4:bd:a0:19:90:cf:16:a7:17:05:
     85:a1:de:13:9e:7b:06:d9:c0:9e:8f:24:7d:59:7a:11:cc:78:
     ac:c3:42:89:59:eb:8d:97:08:d4:74:96:34:c6:79:f5:ea:ca:
     e6:d7:32:ff:33:f7:f4:3e:f0:b2:87:d1:d4:d6:61:75:8a:f9:
     ce:4a:a6:c3:0f:66:7b:25:21:b2:72:48:0a:69:dd:4e:9a:c8:
     3f:ae:be:57:62:d0:9e:c8:97:97:50:f7:26:a5:e5:fa:7a:b5:
     89:24:d4:d4:87:ac:96:0b:f8:58:1b:f7:45:0b:8c:6b:26:17:
     d7:c3:3e:99:d3:2b:54:ca:02:4e:df:66:c9:1b:83:69:da:21:
     80:c2:fb:e0:23:d1:1f:c7:31:2f:fc:a3:fe:14:6a:c9:3a:f4:
     09:02:ae:3f:05:4d:fa:64:06:bc:d1:6f:fc:4c:19:ea:65:39:
     0d:9e:a6:55

JDK is
openjdk version ā€œ1.8.0_162ā€
OpenJDK Runtime Environment (build 1.8.0_162-8u162-b12-0ubuntu0.16.04.2-b12)
OpenJDK 64-Bit Server VM (build 25.162-b12, mixed mode)

Ā·Ā·Ā· 
    Subject: DC=de, L=test, O=node, OU=node, CN=node-0.example.com
    Subject: DC=com, DC=example, O=Example Com Inc., OU=Example Com Inc. Root CA, CN=Example Com Inc. Root CA

On Wednesday, 25 April 2018 09:36:02 UTC+3, Jochen Kressin wrote:

Strange, an upgrade from a vanilla 6.2.3-22.0 to the new certificates seems to work without problems here. This one:

Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException:Path does not chain with any of the trust anchors

``

That usually means the node certificate (and intermediate certificates if present) cannot be validated against the configured root CA. The new demo certificates do not contain any intermediate cert, so the chain is very simple:

esnode.pem ā†’ root-ca.pem

``

If you have OpenSSL installed it would be very helpful to see the output of:

openssl x509 -in ./esnode.pem -text -noout

``

and

penssl x509 -in ./root-ca.pem -text -noout

``

This will print out the details about the certificates, and the trust chain.

Also, which JDK do you use?

Thanks!

On Tuesday, April 24, 2018 at 11:11:10 PM UTC-7, Ross Coundon wrote:

Hi - yes, thereā€™s only a single node at present while we experiment. I unpacked the zip file over the top of the files and accepted all replacements and the root-ca.pqm was replaced too.
Iā€™m using the following configuration:

searchguard.ssl.transport.pemcert_filepath: esnode.pem

searchguard.ssl.transport.pemkey_filepath: esnode-key.pem

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: esnode.pem

searchguard.ssl.http.pemkey_filepath: esnode-key.pem

searchguard.ssl.http.pemtrustedcas_filepath: root-ca.pem

searchguard.allow_unsafe_democertificates: true

searchguard.allow_default_init_sgindex: true

searchguard.authcz.admin_dn:

  • CN=kirk,OU=client,O=client,L=test, C=de

Version of ES is:

ā€œversionā€: {
ā€œnumberā€: ā€œ6.2.3ā€,
ā€œbuild_hashā€: ā€œc59ff00ā€,
ā€œbuild_dateā€: ā€œ2018-03-13T10:06:29.741383Zā€,
ā€œbuild_snapshotā€: false,
ā€œlucene_versionā€: ā€œ7.2.1ā€,
ā€œminimum_wire_compatibility_versionā€: ā€œ5.6.0ā€,
ā€œminimum_index_compatibility_versionā€: ā€œ5.0.0ā€
},

``

and search-guard is:

version=6.2.3-22.0

``

On Wednesday, 25 April 2018 08:06:34 UTC+3, Jochen Kressin wrote:

Have you replaced all the files on all node - including the root-ca.pem? We also exchanged the Root CA, so it needs to be updated as well.

And, do you by chance have different certificates for transport and HTTP?

Oh, and what exact version do you use - should not make a difference, but always useful to know.

On Tuesday, April 24, 2018 at 9:42:50 PM UTC-7, Ross Coundon wrote:

Hi - I just tried replacing the files as instructed and ElasticSearch will no longer start:

[2018-04-25T04:40:50,191][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [node-1] SSL Problem General SSLEngine problem
javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529) ~[?:?]
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_162]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) ~[?:?]
at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1979) ~[?:?]
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:237) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_162]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]
ā€¦ 19 more
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:362) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:279) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:130) ~[?:?]
at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1966) ~[?:?]
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:237) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_162]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]
ā€¦ 19 more
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:153) ~[?:?]
at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:79) ~[?:?]
at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292) ~[?:1.8.0_162]
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:357) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:279) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:130) ~[?:?]
at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1966) ~[?:?]
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:237) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_162]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]
ā€¦ 19 more

``

Do you know whatā€™s gone wrong here?

On Tuesday, 24 April 2018 07:51:45 UTC+3, Jochen Kressin wrote:

Hi,

the demo certificates that are shipped with Search Guard are about to expire on 4th of May. Anyone still running a PoC with these demo certificates should replace them soon.

We have prepared a new set of certificates, and will also take this as an opportunity to replace the old root CA for security reasons. The new certificates can be downloaded from here:

https://downloads.search-guard.com/tls-demo-certificates

The zip contains the following files:

  • root-ca.pem: The new root CA
  • esnode.pem: Node certificate
  • esnode-key.pem: Private key of the node certificate, no password
  • kirk.pem: Admin certificate
  • kirk-key.pem: Private key of the admin certificate, no password
  • spock.pem: Client certificate
  • spock-key.pem: Private key of the client certificate, no password

The certificates are a drop-in replacement for the old ones. This means you can simply unzip them in the config directory of Elasticsearch (overwriting the old ones) and restart the node(s). No further configuration changes are required.

In addition, we will ship a point release 22.1 soon containing the new certs.

Sorry for any inconvenience!

Thanks,

Jochen

Search Guard (Ā®) is an Elasticsearch plugin that offers encryption, authentication and authorisation.

Coded with love in Berlin, Denmark, Sweden and the US.

Search Guard is a trademark of floragunn GmbH, registered in the U.S. and in other countries.

Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries.

7

Hi Ross, Has your problem been solved? I have meet the same problem with you.

åœØ 2018幓4꜈25ę—„ę˜ŸęœŸäø‰ UTC+8äø‹åˆ3:31:40ļ¼ŒRoss Coundon写道ļ¼š

Ā·Ā·Ā·

Strange, an upgrade from a vanilla 6.2.3-22.0 to the new certificates seems to work without problems here. This one:

Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException:Path does not chain with any of the trust anchors

``

That usually means the node certificate (and intermediate certificates if present) cannot be validated against the configured root CA. The new demo certificates do not contain any intermediate cert, so the chain is very simple:

esnode.pem ā†’ root-ca.pem

``

If you have OpenSSL installed it would be very helpful to see the output of:

openssl x509 -in ./esnode.pem -text -noout

``

and

penssl x509 -in ./root-ca.pem -text -noout

``

This will print out the details about the certificates, and the trust chain.

Also, which JDK do you use?

Thanks!

On Tuesday, April 24, 2018 at 11:11:10 PM UTC-7, Ross Coundon wrote:

Hi - yes, thereā€™s only a single node at present while we experiment. I unpacked the zip file over the top of the files and accepted all replacements and the root-ca.pqm was replaced too.
Iā€™m using the following configuration:

searchguard.ssl.transport.pemcert_filepath: esnode.pem

searchguard.ssl.transport.pemkey_filepath: esnode-key.pem

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: esnode.pem

searchguard.ssl.http.pemkey_filepath: esnode-key.pem

searchguard.ssl.http.pemtrustedcas_filepath: root-ca.pem

searchguard.allow_unsafe_democertificates: true

searchguard.allow_default_init_sgindex: true

searchguard.authcz.admin_dn:

  • CN=kirk,OU=client,O=client,L=test, C=de

Version of ES is:

ā€œversionā€: {
ā€œnumberā€: ā€œ6.2.3ā€,
ā€œbuild_hashā€: ā€œc59ff00ā€,
ā€œbuild_dateā€: ā€œ2018-03-13T10:06:29.741383Zā€,
ā€œbuild_snapshotā€: false,
ā€œlucene_versionā€: ā€œ7.2.1ā€,
ā€œminimum_wire_compatibility_versionā€: ā€œ5.6.0ā€,
ā€œminimum_index_compatibility_versionā€: ā€œ5.0.0ā€
},

``

and search-guard is:

version=6.2.3-22.0

``

On Wednesday, 25 April 2018 08:06:34 UTC+3, Jochen Kressin wrote:

Have you replaced all the files on all node - including the root-ca.pem? We also exchanged the Root CA, so it needs to be updated as well.

And, do you by chance have different certificates for transport and HTTP?

Oh, and what exact version do you use - should not make a difference, but always useful to know.

On Tuesday, April 24, 2018 at 9:42:50 PM UTC-7, Ross Coundon wrote:

Hi - I just tried replacing the files as instructed and ElasticSearch will no longer start:

[2018-04-25T04:40:50,191][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [node-1] SSL Problem General SSLEngine problem
javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529) ~[?:?]
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_162]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) ~[?:?]
at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1979) ~[?:?]
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:237) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_162]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]
ā€¦ 19 more
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:362) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:279) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:130) ~[?:?]
at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1966) ~[?:?]
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:237) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_162]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]
ā€¦ 19 more
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:153) ~[?:?]
at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:79) ~[?:?]
at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292) ~[?:1.8.0_162]
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:357) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:279) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:130) ~[?:?]
at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1966) ~[?:?]
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:237) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_162]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]
ā€¦ 19 more

``

Do you know whatā€™s gone wrong here?

On Tuesday, 24 April 2018 07:51:45 UTC+3, Jochen Kressin wrote:

Hi,

the demo certificates that are shipped with Search Guard are about to expire on 4th of May. Anyone still running a PoC with these demo certificates should replace them soon.

We have prepared a new set of certificates, and will also take this as an opportunity to replace the old root CA for security reasons. The new certificates can be downloaded from here:

https://downloads.search-guard.com/tls-demo-certificates

The zip contains the following files:

  • root-ca.pem: The new root CA
  • esnode.pem: Node certificate
  • esnode-key.pem: Private key of the node certificate, no password
  • kirk.pem: Admin certificate
  • kirk-key.pem: Private key of the admin certificate, no password
  • spock.pem: Client certificate
  • spock-key.pem: Private key of the client certificate, no password

The certificates are a drop-in replacement for the old ones. This means you can simply unzip them in the config directory of Elasticsearch (overwriting the old ones) and restart the node(s). No further configuration changes are required.

In addition, we will ship a point release 22.1 soon containing the new certs.

Sorry for any inconvenience!

Thanks,

Jochen

Search Guard (Ā®) is an Elasticsearch plugin that offers encryption, authentication and authorisation.

Coded with love in Berlin, Denmark, Sweden and the US.

Search Guard is a trademark of floragunn GmbH, registered in the U.S. and in other countries.

Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries.

Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1524368626614 (0x162eb7353b6)
Signature Algorithm: sha256WithRSAEncryption
Issuer: DC=com, DC=example, O=Example Com Inc., OU=Example Com Inc. Root CA, CN=Example Com Inc. Root CA
Validity
Not Before: Apr 22 03:43:47 2018 GMT
Not After : Apr 19 03:43:47 2028 GMT
Subject: DC=de, L=test, O=node, OU=node, CN=node-0.example.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:96:be:7f:8e:fa:bc:b0:7e:03:02:b9:dd:b8:98:
07:25:30:37:ee:34:0f:c8:cc:22:8b:c6:5e:6e:b0:
81:3f:3e:f5:26:ec:f3:df:5d:0d:78:2d:f4:21:35:
05:ea:3a:e6:83:f5:f8:95:33:e1:ce:d4:1c:ca:c2:
63:77:8f:88:3b:78:72:27:47:57:31:10:da:0d:18:
a1:5a:d0:5a:fd:11:79:d4:bf:cb:1f:c3:2a:1b:3c:
3f:0d:4e:ef:5e:68:7e:d3:f9:de:9f:f6:8a:30:f9:
0e:27:c5:bf:57:8a:7e:48:45:1f:e9:70:9f:2f:ef:
31:23:71:7a:59:69:97:a3:71:25:38:89:56:74:3d:
1d:83:8b:81:fd:ad:f7:bd:48:4c:91:e7:02:eb:b1:
50:5e:3c:1d:cb:8d:a2:f5:b8:ae:1b:64:5d:e7:fc:
91:a0:0d:ed:c1:37:2d:4f:80:f5:3e:3b:e1:42:cd:
08:a9:04:14:f2:25:64:02:8d:de:22:4d:15:d5:6c:
c6:b4:d4:f8:25:01:1f:39:3b:dc:3a:35:70:29:04:
bc:96:74:64:58:e9:d1:9d:f2:f3:02:d8:fe:0a:96:
19:f1:95:c8:0f:65:d8:25:2a:78:86:4d:7f:9e:4f:
34:fb:46:cc:ea:ef:bc:e3:62:ba:2e:3c:bc:12:87:
d4:bb
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Authority Key Identifier:
keyid:92:35:0C:E0:0F:1E:2B:45:F6:4D:39:F3:7B:5F:A2:E6:12:97:40:73
DirName:/DC=com/DC=example/O=Example Com Inc./OU=Example Com Inc. Root CA/CN=Example Com Inc. Root CA
serial:01

        X509v3 Subject Key Identifier:
            AC:AF:EF:C6:66:16:35:4A:33:D8:3B:A4:C0:A8:9D:81:FB:15:50:47
        X509v3 Basic Constraints: critical
            CA:FALSE
        X509v3 Key Usage: critical
            Digital Signature, Non Repudiation, Key Encipherment
        X509v3 Extended Key Usage: critical
            TLS Web Server Authentication, TLS Web Client Authentication
        X509v3 Subject Alternative Name:
            Registered ID:1.2.3.4.5.5, DNS:[node-0.example.com](http://node-0.example.com), DNS:localhost, IP Address:127.0.0.1
Signature Algorithm: sha256WithRSAEncryption
     83:8a:bb:25:ec:15:f1:af:d6:12:3f:2e:4a:5d:ff:bd:d0:36:
     a8:ea:25:dc:50:72:55:f9:ec:63:8c:58:d6:ce:33:91:f4:e7:
     ff:40:38:86:79:25:0b:16:50:b3:4a:37:be:da:1e:32:d1:af:
     8a:30:ab:68:c6:6e:97:3f:67:a9:00:77:e7:a2:6a:d9:1c:a6:
     76:ed:6c:6a:e1:2a:93:ad:a0:46:72:f0:ab:ac:97:09:1e:8b:
     1a:73:2c:33:48:49:26:e4:78:ba:57:cf:8c:49:23:51:13:30:
     df:d5:1b:c6:59:3f:56:e3:ce:51:f3:88:71:c3:bb:42:4d:67:
     a4:e7:37:32:ab:5f:30:86:30:2a:21:15:f7:a6:f2:f9:ca:36:
     72:94:9d:e3:10:32:f5:dd:de:bc:d4:68:08:2f:b5:fe:c7:73:
     62:d3:06:57:f8:7d:9c:d9:17:51:24:c1:d4:97:85:a3:00:d6:
     59:1f:1e:2a:8e:07:1b:60:78:32:f1:08:71:12:67:67:ea:81:
     5a:ac:59:7f:ad:de:a1:d0:7e:2b:dc:3d:6e:ad:c6:d0:f2:ac:
     53:d1:74:93:86:86:23:06:cd:3f:ed:7b:ff:64:90:0b:50:46:
     0a:53:6f:7b:24:61:d2:0e:39:43:95:d1:61:90:eb:49:09:94:
     58:40:cc:8b

and

Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1 (0x1)
Signature Algorithm: sha256WithRSAEncryption
Issuer: DC=com, DC=example, O=Example Com Inc., OU=Example Com Inc. Root CA, CN=Example Com Inc. Root CA
Validity
Not Before: Apr 22 03:43:46 2018 GMT
Not After : Apr 19 03:43:46 2028 GMT
Subject: DC=com, DC=example, O=Example Com Inc., OU=Example Com Inc. Root CA, CN=Example Com Inc. Root CA
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:af:ee:f8:60:11:3f:98:a7:9e:1a:57:2b:47:34:
ab:bb:35:4a:ed:55:4c:46:88:82:66:6b:f1:55:88:
e9:2f:1a:99:fe:5c:53:79:2d:57:b5:93:f5:2b:95:
4e:c0:26:da:2d:80:e4:ff:82:b7:0e:e2:66:47:e7:
1d:69:6c:0b:71:e1:3d:47:1d:ea:6b:f3:19:9e:26:
a3:19:da:98:ce:eb:f9:af:68:b5:1a:77:a3:06:28:
19:2b:57:ca:55:53:42:eb:00:8d:ba:bd:76:8f:02:
31:5e:21:70:14:de:a4:27:7e:d3:0d:2f:e2:1e:94:
95:75:3c:c6:38:63:d7:17:94:23:3e:03:29:b4:60:
7f:7e:aa:d2:bb:f8:54:85:f8:e9:7e:f6:ac:c2:52:
11:32:8e:4b:1b:b0:2e:4a:2f:d5:93:95:6d:f4:a5:
3d:ac:a0:5c:8c:6a:b0:75:65:8f:58:8c:91:84:5b:
42:66:93:89:be:97:58:72:9f:32:26:c3:6a:a0:de:
8c:e8:6e:92:40:a3:ce:9a:6d:19:93:8f:15:0f:34:
d3:65:2d:4d:33:6f:d5:38:9a:2b:19:23:31:02:4d:
c3:3e:a3:7d:9e:77:c2:cd:df:87:52:34:45:64:fa:
59:f3:38:a1:e1:51:16:7c:85:46:67:38:b9:84:d4:
80:09
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Authority Key Identifier:
keyid:92:35:0C:E0:0F:1E:2B:45:F6:4D:39:F3:7B:5F:A2:E6:12:97:40:73

        X509v3 Subject Key Identifier:
            92:35:0C:E0:0F:1E:2B:45:F6:4D:39:F3:7B:5F:A2:E6:12:97:40:73
        X509v3 Key Usage: critical
            Digital Signature, Certificate Sign, CRL Sign
Signature Algorithm: sha256WithRSAEncryption
     68:40:7b:f0:b1:1d:f8:84:63:b6:9b:ca:95:47:d9:d0:e4:a9:
     68:e4:76:0f:c9:de:b2:48:a3:5c:4f:7e:8e:67:80:10:7d:a0:
     86:b2:4f:92:79:c4:e2:df:94:05:44:72:f7:83:6a:9f:7c:40:
     f8:b4:a4:74:44:13:46:41:28:22:2d:ab:e6:1c:60:a1:dd:8a:
     43:ba:92:aa:db:18:61:11:e4:bd:a0:19:90:cf:16:a7:17:05:
     85:a1:de:13:9e:7b:06:d9:c0:9e:8f:24:7d:59:7a:11:cc:78:
     ac:c3:42:89:59:eb:8d:97:08:d4:74:96:34:c6:79:f5:ea:ca:
     e6:d7:32:ff:33:f7:f4:3e:f0:b2:87:d1:d4:d6:61:75:8a:f9:
     ce:4a:a6:c3:0f:66:7b:25:21:b2:72:48:0a:69:dd:4e:9a:c8:
     3f:ae:be:57:62:d0:9e:c8:97:97:50:f7:26:a5:e5:fa:7a:b5:
     89:24:d4:d4:87:ac:96:0b:f8:58:1b:f7:45:0b:8c:6b:26:17:
     d7:c3:3e:99:d3:2b:54:ca:02:4e:df:66:c9:1b:83:69:da:21:
     80:c2:fb:e0:23:d1:1f:c7:31:2f:fc:a3:fe:14:6a:c9:3a:f4:
     09:02:ae:3f:05:4d:fa:64:06:bc:d1:6f:fc:4c:19:ea:65:39:
     0d:9e:a6:55

JDK is
openjdk version ā€œ1.8.0_162ā€
OpenJDK Runtime Environment (build 1.8.0_162-8u162-b12-0ubuntu0.16.04.2-b12)
OpenJDK 64-Bit Server VM (build 25.162-b12, mixed mode)

On Wednesday, 25 April 2018 09:36:02 UTC+3, Jochen Kressin wrote:

8

guys, thats really strange - we cannot reproduce this
pls make sure you replace all the certs. best would be first to delete every cert/key to have no leftovers and then copy over the new ones.

@yvanh1994 pls share your steps what you did to rollover the certs

Ā·Ā·Ā·

On Monday, 7 May 2018 10:20:52 UTC+2, yvaā€¦@gā€¦com wrote:

Hi Ross, Has your problem been solved? I have meet the same problem with you.

åœØ 2018幓4꜈25ę—„ę˜ŸęœŸäø‰ UTC+8äø‹åˆ3:31:40ļ¼ŒRoss Coundon写道ļ¼š

Strange, an upgrade from a vanilla 6.2.3-22.0 to the new certificates seems to work without problems here. This one:

Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException:Path does not chain with any of the trust anchors

``

That usually means the node certificate (and intermediate certificates if present) cannot be validated against the configured root CA. The new demo certificates do not contain any intermediate cert, so the chain is very simple:

esnode.pem ā†’ root-ca.pem

``

If you have OpenSSL installed it would be very helpful to see the output of:

openssl x509 -in ./esnode.pem -text -noout

``

and

penssl x509 -in ./root-ca.pem -text -noout

``

This will print out the details about the certificates, and the trust chain.

Also, which JDK do you use?

Thanks!

On Tuesday, April 24, 2018 at 11:11:10 PM UTC-7, Ross Coundon wrote:

Hi - yes, thereā€™s only a single node at present while we experiment. I unpacked the zip file over the top of the files and accepted all replacements and the root-ca.pqm was replaced too.
Iā€™m using the following configuration:

searchguard.ssl.transport.pemcert_filepath: esnode.pem

searchguard.ssl.transport.pemkey_filepath: esnode-key.pem

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: esnode.pem

searchguard.ssl.http.pemkey_filepath: esnode-key.pem

searchguard.ssl.http.pemtrustedcas_filepath: root-ca.pem

searchguard.allow_unsafe_democertificates: true

searchguard.allow_default_init_sgindex: true

searchguard.authcz.admin_dn:

  • CN=kirk,OU=client,O=client,L=test, C=de

Version of ES is:

ā€œversionā€: {
ā€œnumberā€: ā€œ6.2.3ā€,
ā€œbuild_hashā€: ā€œc59ff00ā€,
ā€œbuild_dateā€: ā€œ2018-03-13T10:06:29.741383Zā€,
ā€œbuild_snapshotā€: false,
ā€œlucene_versionā€: ā€œ7.2.1ā€,
ā€œminimum_wire_compatibility_versionā€: ā€œ5.6.0ā€,
ā€œminimum_index_compatibility_versionā€: ā€œ5.0.0ā€
},

``

and search-guard is:

version=6.2.3-22.0

``

On Wednesday, 25 April 2018 08:06:34 UTC+3, Jochen Kressin wrote:

Have you replaced all the files on all node - including the root-ca.pem? We also exchanged the Root CA, so it needs to be updated as well.

And, do you by chance have different certificates for transport and HTTP?

Oh, and what exact version do you use - should not make a difference, but always useful to know.

On Tuesday, April 24, 2018 at 9:42:50 PM UTC-7, Ross Coundon wrote:

Hi - I just tried replacing the files as instructed and ElasticSearch will no longer start:

[2018-04-25T04:40:50,191][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [node-1] SSL Problem General SSLEngine problem
javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529) ~[?:?]
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_162]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) ~[?:?]
at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1979) ~[?:?]
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:237) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_162]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]
ā€¦ 19 more
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:362) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:279) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:130) ~[?:?]
at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1966) ~[?:?]
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:237) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_162]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]
ā€¦ 19 more
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:153) ~[?:?]
at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:79) ~[?:?]
at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292) ~[?:1.8.0_162]
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:357) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:279) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:130) ~[?:?]
at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1966) ~[?:?]
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:237) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_162]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]
ā€¦ 19 more

``

Do you know whatā€™s gone wrong here?

On Tuesday, 24 April 2018 07:51:45 UTC+3, Jochen Kressin wrote:

Hi,

the demo certificates that are shipped with Search Guard are about to expire on 4th of May. Anyone still running a PoC with these demo certificates should replace them soon.

We have prepared a new set of certificates, and will also take this as an opportunity to replace the old root CA for security reasons. The new certificates can be downloaded from here:

https://downloads.search-guard.com/tls-demo-certificates

The zip contains the following files:

  • root-ca.pem: The new root CA
  • esnode.pem: Node certificate
  • esnode-key.pem: Private key of the node certificate, no password
  • kirk.pem: Admin certificate
  • kirk-key.pem: Private key of the admin certificate, no password
  • spock.pem: Client certificate
  • spock-key.pem: Private key of the client certificate, no password

The certificates are a drop-in replacement for the old ones. This means you can simply unzip them in the config directory of Elasticsearch (overwriting the old ones) and restart the node(s). No further configuration changes are required.

In addition, we will ship a point release 22.1 soon containing the new certs.

Sorry for any inconvenience!

Thanks,

Jochen

Search Guard (Ā®) is an Elasticsearch plugin that offers encryption, authentication and authorisation.

Coded with love in Berlin, Denmark, Sweden and the US.

Search Guard is a trademark of floragunn GmbH, registered in the U.S. and in other countries.

Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries.

Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1524368626614 (0x162eb7353b6)
Signature Algorithm: sha256WithRSAEncryption
Issuer: DC=com, DC=example, O=Example Com Inc., OU=Example Com Inc. Root CA, CN=Example Com Inc. Root CA
Validity
Not Before: Apr 22 03:43:47 2018 GMT
Not After : Apr 19 03:43:47 2028 GMT
Subject: DC=de, L=test, O=node, OU=node, CN=node-0.example.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:96:be:7f:8e:fa:bc:b0:7e:03:02:b9:dd:b8:98:
07:25:30:37:ee:34:0f:c8:cc:22:8b:c6:5e:6e:b0:
81:3f:3e:f5:26:ec:f3:df:5d:0d:78:2d:f4:21:35:
05:ea:3a:e6:83:f5:f8:95:33:e1:ce:d4:1c:ca:c2:
63:77:8f:88:3b:78:72:27:47:57:31:10:da:0d:18:
a1:5a:d0:5a:fd:11:79:d4:bf:cb:1f:c3:2a:1b:3c:
3f:0d:4e:ef:5e:68:7e:d3:f9:de:9f:f6:8a:30:f9:
0e:27:c5:bf:57:8a:7e:48:45:1f:e9:70:9f:2f:ef:
31:23:71:7a:59:69:97:a3:71:25:38:89:56:74:3d:
1d:83:8b:81:fd:ad:f7:bd:48:4c:91:e7:02:eb:b1:
50:5e:3c:1d:cb:8d:a2:f5:b8:ae:1b:64:5d:e7:fc:
91:a0:0d:ed:c1:37:2d:4f:80:f5:3e:3b:e1:42:cd:
08:a9:04:14:f2:25:64:02:8d:de:22:4d:15:d5:6c:
c6:b4:d4:f8:25:01:1f:39:3b:dc:3a:35:70:29:04:
bc:96:74:64:58:e9:d1:9d:f2:f3:02:d8:fe:0a:96:
19:f1:95:c8:0f:65:d8:25:2a:78:86:4d:7f:9e:4f:
34:fb:46:cc:ea:ef:bc:e3:62:ba:2e:3c:bc:12:87:
d4:bb
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Authority Key Identifier:
keyid:92:35:0C:E0:0F:1E:2B:45:F6:4D:39:F3:7B:5F:A2:E6:12:97:40:73
DirName:/DC=com/DC=example/O=Example Com Inc./OU=Example Com Inc. Root CA/CN=Example Com Inc. Root CA
serial:01

        X509v3 Subject Key Identifier:
            AC:AF:EF:C6:66:16:35:4A:33:D8:3B:A4:C0:A8:9D:81:FB:15:50:47
        X509v3 Basic Constraints: critical
            CA:FALSE
        X509v3 Key Usage: critical
            Digital Signature, Non Repudiation, Key Encipherment
        X509v3 Extended Key Usage: critical
            TLS Web Server Authentication, TLS Web Client Authentication
        X509v3 Subject Alternative Name:
            Registered ID:1.2.3.4.5.5, DNS:[node-0.example.com](http://node-0.example.com), DNS:localhost, IP Address:127.0.0.1
Signature Algorithm: sha256WithRSAEncryption
     83:8a:bb:25:ec:15:f1:af:d6:12:3f:2e:4a:5d:ff:bd:d0:36:
     a8:ea:25:dc:50:72:55:f9:ec:63:8c:58:d6:ce:33:91:f4:e7:
     ff:40:38:86:79:25:0b:16:50:b3:4a:37:be:da:1e:32:d1:af:
     8a:30:ab:68:c6:6e:97:3f:67:a9:00:77:e7:a2:6a:d9:1c:a6:
     76:ed:6c:6a:e1:2a:93:ad:a0:46:72:f0:ab:ac:97:09:1e:8b:
     1a:73:2c:33:48:49:26:e4:78:ba:57:cf:8c:49:23:51:13:30:
     df:d5:1b:c6:59:3f:56:e3:ce:51:f3:88:71:c3:bb:42:4d:67:
     a4:e7:37:32:ab:5f:30:86:30:2a:21:15:f7:a6:f2:f9:ca:36:
     72:94:9d:e3:10:32:f5:dd:de:bc:d4:68:08:2f:b5:fe:c7:73:
     62:d3:06:57:f8:7d:9c:d9:17:51:24:c1:d4:97:85:a3:00:d6:
     59:1f:1e:2a:8e:07:1b:60:78:32:f1:08:71:12:67:67:ea:81:
     5a:ac:59:7f:ad:de:a1:d0:7e:2b:dc:3d:6e:ad:c6:d0:f2:ac:
     53:d1:74:93:86:86:23:06:cd:3f:ed:7b:ff:64:90:0b:50:46:
     0a:53:6f:7b:24:61:d2:0e:39:43:95:d1:61:90:eb:49:09:94:
     58:40:cc:8b

and

Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1 (0x1)
Signature Algorithm: sha256WithRSAEncryption
Issuer: DC=com, DC=example, O=Example Com Inc., OU=Example Com Inc. Root CA, CN=Example Com Inc. Root CA
Validity
Not Before: Apr 22 03:43:46 2018 GMT
Not After : Apr 19 03:43:46 2028 GMT
Subject: DC=com, DC=example, O=Example Com Inc., OU=Example Com Inc. Root CA, CN=Example Com Inc. Root CA
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:af:ee:f8:60:11:3f:98:a7:9e:1a:57:2b:47:34:
ab:bb:35:4a:ed:55:4c:46:88:82:66:6b:f1:55:88:
e9:2f:1a:99:fe:5c:53:79:2d:57:b5:93:f5:2b:95:
4e:c0:26:da:2d:80:e4:ff:82:b7:0e:e2:66:47:e7:
1d:69:6c:0b:71:e1:3d:47:1d:ea:6b:f3:19:9e:26:
a3:19:da:98:ce:eb:f9:af:68:b5:1a:77:a3:06:28:
19:2b:57:ca:55:53:42:eb:00:8d:ba:bd:76:8f:02:
31:5e:21:70:14:de:a4:27:7e:d3:0d:2f:e2:1e:94:
95:75:3c:c6:38:63:d7:17:94:23:3e:03:29:b4:60:
7f:7e:aa:d2:bb:f8:54:85:f8:e9:7e:f6:ac:c2:52:
11:32:8e:4b:1b:b0:2e:4a:2f:d5:93:95:6d:f4:a5:
3d:ac:a0:5c:8c:6a:b0:75:65:8f:58:8c:91:84:5b:
42:66:93:89:be:97:58:72:9f:32:26:c3:6a:a0:de:
8c:e8:6e:92:40:a3:ce:9a:6d:19:93:8f:15:0f:34:
d3:65:2d:4d:33:6f:d5:38:9a:2b:19:23:31:02:4d:
c3:3e:a3:7d:9e:77:c2:cd:df:87:52:34:45:64:fa:
59:f3:38:a1:e1:51:16:7c:85:46:67:38:b9:84:d4:
80:09
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Authority Key Identifier:
keyid:92:35:0C:E0:0F:1E:2B:45:F6:4D:39:F3:7B:5F:A2:E6:12:97:40:73

        X509v3 Subject Key Identifier:
            92:35:0C:E0:0F:1E:2B:45:F6:4D:39:F3:7B:5F:A2:E6:12:97:40:73
        X509v3 Key Usage: critical
            Digital Signature, Certificate Sign, CRL Sign
Signature Algorithm: sha256WithRSAEncryption
     68:40:7b:f0:b1:1d:f8:84:63:b6:9b:ca:95:47:d9:d0:e4:a9:
     68:e4:76:0f:c9:de:b2:48:a3:5c:4f:7e:8e:67:80:10:7d:a0:
     86:b2:4f:92:79:c4:e2:df:94:05:44:72:f7:83:6a:9f:7c:40:
     f8:b4:a4:74:44:13:46:41:28:22:2d:ab:e6:1c:60:a1:dd:8a:
     43:ba:92:aa:db:18:61:11:e4:bd:a0:19:90:cf:16:a7:17:05:
     85:a1:de:13:9e:7b:06:d9:c0:9e:8f:24:7d:59:7a:11:cc:78:
     ac:c3:42:89:59:eb:8d:97:08:d4:74:96:34:c6:79:f5:ea:ca:
     e6:d7:32:ff:33:f7:f4:3e:f0:b2:87:d1:d4:d6:61:75:8a:f9:
     ce:4a:a6:c3:0f:66:7b:25:21:b2:72:48:0a:69:dd:4e:9a:c8:
     3f:ae:be:57:62:d0:9e:c8:97:97:50:f7:26:a5:e5:fa:7a:b5:
     89:24:d4:d4:87:ac:96:0b:f8:58:1b:f7:45:0b:8c:6b:26:17:
     d7:c3:3e:99:d3:2b:54:ca:02:4e:df:66:c9:1b:83:69:da:21:
     80:c2:fb:e0:23:d1:1f:c7:31:2f:fc:a3:fe:14:6a:c9:3a:f4:
     09:02:ae:3f:05:4d:fa:64:06:bc:d1:6f:fc:4c:19:ea:65:39:
     0d:9e:a6:55

JDK is
openjdk version ā€œ1.8.0_162ā€
OpenJDK Runtime Environment (build 1.8.0_162-8u162-b12-0ubuntu0.16.04.2-b12)
OpenJDK 64-Bit Server VM (build 25.162-b12, mixed mode)

On Wednesday, 25 April 2018 09:36:02 UTC+3, Jochen Kressin wrote:

9

Unfortunately not, I had to give up and uninstall search guard altogether as the certs expired and pretty much everything then stopped working

10

Sounds like you used the demo certificates for production?

Ā·Ā·Ā·

On Monday, 7 May 2018 12:27:31 UTC+2, Ross Coundon wrote:

Unfortunately not, I had to give up and uninstall search guard altogether as the certs expired and pretty much everything then stopped working

11

No, I just ran out of time to evaluate.

Ā·Ā·Ā·

On Monday, 7 May 2018 15:42:53 UTC+3, Search Guard wrote:

Sounds like you used the demo certificates for production?

On Monday, 7 May 2018 12:27:31 UTC+2, Ross Coundon wrote:

Unfortunately not, I had to give up and uninstall search guard altogether as the certs expired and pretty much everything then stopped working

12

if there is no sensitive data in there can you just zip the stuff and send it to us? happy to investigate this.

Ā·Ā·Ā·

On Monday, 7 May 2018 20:15:35 UTC+2, Ross Coundon wrote:

No, I just ran out of time to evaluate.

On Monday, 7 May 2018 15:42:53 UTC+3, Search Guard wrote:

Sounds like you used the demo certificates for production?

On Monday, 7 May 2018 12:27:31 UTC+2, Ross Coundon wrote:

Unfortunately not, I had to give up and uninstall search guard altogether as the certs expired and pretty much everything then stopped working

13

Hi there,

We have the same error here.

After we got the error ā€œjava.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchorsā€, we ran the sg_admin.sh script.
It returns another error indicating"ElasticsearchException[Empty file path for searchguard.ssl.transport.pemkey_filepath]".

It appears that the search guard canā€™t pick up the file path correctly.

I hope this helps your further investigation.

Kind regards,
Rex

Ā·Ā·Ā·

On Tuesday, 8 May 2018 04:20:19 UTC+10, Search Guard wrote:

if there is no sensitive data in there can you just zip the stuff and send it to us? happy to investigate this.

On Monday, 7 May 2018 20:15:35 UTC+2, Ross Coundon wrote:

No, I just ran out of time to evaluate.

On Monday, 7 May 2018 15:42:53 UTC+3, Search Guard wrote:

Sounds like you used the demo certificates for production?

On Monday, 7 May 2018 12:27:31 UTC+2, Ross Coundon wrote:

Unfortunately not, I had to give up and uninstall search guard altogether as the certs expired and pretty much everything then stopped working

14

my elasticsearch version is 6.2.2 and search guard version is com.floragunn:search-guard-6:6.2.2-22.0. I have remove all pem files under /etc/elasticsearch and download new certificates from https://downloads.search-guard.com/tls-demo-certificates. But elasticsearch service is fail to start and get following errors:

[2018-05-08T02:18:26,590][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [sjtsetelk01] SSL Problem General SSLEngine problem

javax.net.ssl.SSLHandshakeException: General SSLEngine problem

at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1409) ~[?:?]

at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[?:?]

at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[?:?]

at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]

at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_31]

at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_31]

Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem

at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]

at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) ~[?:?]

at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304) ~[?:?]

at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) ~[?:?]

at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1478) ~[?:?]

at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:212) ~[?:?]

at sun.security.ssl.Handshaker.processLoop(Handshaker.java:957) ~[?:?]

at sun.security.ssl.Handshaker$1.run(Handshaker.java:897) ~[?:?]

at sun.security.ssl.Handshaker$1.run(Handshaker.java:894) ~[?:?]

at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_31]

at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1347) ~[?:?]

at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]

at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]

ā€¦ 19 more

Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:352) ~[?:?]

at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:260) ~[?:?]

at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]

at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]

at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[?:?]

at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:?]

at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1465) ~[?:?]

at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:212) ~[?:?]

at sun.security.ssl.Handshaker.processLoop(Handshaker.java:957) ~[?:?]

at sun.security.ssl.Handshaker$1.run(Handshaker.java:897) ~[?:?]

at sun.security.ssl.Handshaker$1.run(Handshaker.java:894) ~[?:?]

at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_31]

at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1347) ~[?:?]

at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]

at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]

ā€¦ 19 more

Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:153) ~[?:?]

at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:79) ~[?:?]

at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292) ~[?:1.8.0_31]

at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:347) ~[?:?]

at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:260) ~[?:?]

at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]

at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]

at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[?:?]

at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:?]

at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1465) ~[?:?]

at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:212) ~[?:?]

at sun.security.ssl.Handshaker.processLoop(Handshaker.java:957) ~[?:?]

at sun.security.ssl.Handshaker$1.run(Handshaker.java:897) ~[?:?]

at sun.security.ssl.Handshaker$1.run(Handshaker.java:894) ~[?:?]

at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_31]

at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1347) ~[?:?]

at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]

at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]

ā€¦ 19 more

[2018-05-08T02:18:26,585][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [sjtsetelk01] SSL Problem General SSLEngine problem

javax.net.ssl.SSLHandshakeException: General SSLEngine problem

at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1409) ~[?:?]

at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[?:?]

at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[?:?]

at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]

Ā·Ā·Ā·

2018-05-08 2:20 GMT+08:00 Search Guard info@search-guard.com:

if there is no sensitive data in there can you just zip the stuff and send it to us? happy to investigate this.

On Monday, 7 May 2018 20:15:35 UTC+2, Ross Coundon wrote:

No, I just ran out of time to evaluate.

On Monday, 7 May 2018 15:42:53 UTC+3, Search Guard wrote:

Sounds like you used the demo certificates for production?

On Monday, 7 May 2018 12:27:31 UTC+2, Ross Coundon wrote:

Unfortunately not, I had to give up and uninstall search guard altogether as the certs expired and pretty much everything then stopped working

ā€“

You received this message because you are subscribed to the Google Groups ā€œSearch Guard Community Forumā€ group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/1114f33e-cec1-46ca-8bdc-e4688853b366%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

15

i cannot reproduce this

to proof that it works i created a demo for that, pls look here https://gist.github.com/floragunncom/3f2734aaa692fd165fc1cce3d931c10b
this installs ES and SG from the scratch, install old expired demo certificates, overwrite them with the new ones and it works

Ā·Ā·Ā·

Am 08.05.2018 um 11:27 schrieb Yvan He <yvanh1994@gmail.com>:

my elasticsearch version is 6.2.2 and search guard version is com.floragunn:search-guard-6:6.2.2-22.0. I have remove all pem files under /etc/elasticsearch and download new certificates from https://downloads.search-guard.com/tls-demo-certificates\. But elasticsearch service is fail to start and get following errors:

[2018-05-08T02:18:26,590][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [sjtsetelk01] SSL Problem General SSLEngine problem
javax.net.ssl.SSLHandshakeException: General SSLEngine problem
        at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1409) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_31]
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_31]
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) ~[?:?]
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304) ~[?:?]
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) ~[?:?]
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1478) ~[?:?]
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:212) ~[?:?]
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:957) ~[?:?]
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:897) ~[?:?]
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:894) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_31]
        at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1347) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]
        ... 19 more
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:352) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:260) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:?]
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1465) ~[?:?]
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:212) ~[?:?]
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:957) ~[?:?]
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:897) ~[?:?]
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:894) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_31]
        at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1347) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]
        ... 19 more
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:153) ~[?:?]
        at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:79) ~[?:?]
        at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292) ~[?:1.8.0_31]
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:347) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:260) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:?]
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1465) ~[?:?]
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:212) ~[?:?]
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:957) ~[?:?]
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:897) ~[?:?]
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:894) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_31]
        at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1347) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]
        ... 19 more
[2018-05-08T02:18:26,585][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [sjtsetelk01] SSL Problem General SSLEngine problem
javax.net.ssl.SSLHandshakeException: General SSLEngine problem
        at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1409) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]

2018-05-08 2:20 GMT+08:00 Search Guard <info@search-guard.com>:
if there is no sensitive data in there can you just zip the stuff and send it to us? happy to investigate this.

On Monday, 7 May 2018 20:15:35 UTC+2, Ross Coundon wrote:
No, I just ran out of time to evaluate.

On Monday, 7 May 2018 15:42:53 UTC+3, Search Guard wrote:
Sounds like you used the demo certificates for production?

On Monday, 7 May 2018 12:27:31 UTC+2, Ross Coundon wrote:
Unfortunately not, I had to give up and uninstall search guard altogether as the certs expired and pretty much everything then stopped working

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/1114f33e-cec1-46ca-8bdc-e4688853b366%40googlegroups.com\.

For more options, visit https://groups.google.com/d/optout\.

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/CAAiY4T%3DQGnN_xtm%3DL%3D3U9T66foUG9pDnhjzKSxX5-rFYbMwVOg%40mail.gmail.com\.
For more options, visit https://groups.google.com/d/optout\.

16

hi guys, i meet the same problem on searchguard 5.3. after cert expired, es cluster stop work even when I replace with a new cert of self-signed (by example-pki-scripts)
why ? pls give some help

åœØ 2018幓5꜈9ę—„ę˜ŸęœŸäø‰ UTC+8äøŠåˆ12:18:53ļ¼ŒSearch Guard写道ļ¼š

Ā·Ā·Ā·

i cannot reproduce this

to proof that it works i created a demo for that, pls look here https://gist.github.com/floragunncom/3f2734aaa692fd165fc1cce3d931c10b

this installs ES and SG from the scratch, install old expired demo certificates, overwrite them with the new ones and it works

Am 08.05.2018 um 11:27 schrieb Yvan He yvan...@gmail.com:

my elasticsearch version is 6.2.2 and search guard version is com.floragunn:search-guard-6:6.2.2-22.0. I have remove all pem files under /etc/elasticsearch and download new certificates from https://downloads.search-guard.com/tls-demo-certificates. But elasticsearch service is fail to start and get following errors:

[2018-05-08T02:18:26,590][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [sjtsetelk01] SSL Problem General SSLEngine problem

javax.net.ssl.SSLHandshakeException: General SSLEngine problem

    at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1409) ~[?:?]

    at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[?:?]

    at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[?:?]

    at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]

    at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_31]

    at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]

    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]

    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]

    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]

    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_31]

Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem

    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]

    at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) ~[?:?]

    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304) ~[?:?]

    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) ~[?:?]

    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1478) ~[?:?]

    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:212) ~[?:?]

    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:957) ~[?:?]

    at sun.security.ssl.Handshaker$1.run(Handshaker.java:897) ~[?:?]

    at sun.security.ssl.Handshaker$1.run(Handshaker.java:894) ~[?:?]

    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_31]

    at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1347) ~[?:?]

    at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]

    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]

    ... 19 more

Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

    at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:352) ~[?:?]

    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:260) ~[?:?]

    at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]

    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]

    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[?:?]

    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:?]

    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1465) ~[?:?]

    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:212) ~[?:?]

    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:957) ~[?:?]

    at sun.security.ssl.Handshaker$1.run(Handshaker.java:897) ~[?:?]

    at sun.security.ssl.Handshaker$1.run(Handshaker.java:894) ~[?:?]

    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_31]

    at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1347) ~[?:?]

    at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]

    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]

    ... 19 more

Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

    at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:153) ~[?:?]

    at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:79) ~[?:?]

    at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292) ~[?:1.8.0_31]

    at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:347) ~[?:?]

    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:260) ~[?:?]

    at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]

    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]

    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[?:?]

    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:?]

    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1465) ~[?:?]

    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:212) ~[?:?]

    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:957) ~[?:?]

    at sun.security.ssl.Handshaker$1.run(Handshaker.java:897) ~[?:?]

    at sun.security.ssl.Handshaker$1.run(Handshaker.java:894) ~[?:?]

    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_31]

    at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1347) ~[?:?]

    at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[?:?]

    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[?:?]

    ... 19 more

[2018-05-08T02:18:26,585][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [sjtsetelk01] SSL Problem General SSLEngine problem

javax.net.ssl.SSLHandshakeException: General SSLEngine problem

    at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1409) ~[?:?]

    at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[?:?]

    at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[?:?]

    at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]

2018-05-08 2:20 GMT+08:00 Search Guard in...@search-guard.com:

if there is no sensitive data in there can you just zip the stuff and send it to us? happy to investigate this.

On Monday, 7 May 2018 20:15:35 UTC+2, Ross Coundon wrote:

No, I just ran out of time to evaluate.

On Monday, 7 May 2018 15:42:53 UTC+3, Search Guard wrote:

Sounds like you used the demo certificates for production?

On Monday, 7 May 2018 12:27:31 UTC+2, Ross Coundon wrote:

Unfortunately not, I had to give up and uninstall search guard altogether as the certs expired and pretty much everything then stopped working

ā€“
You received this message because you are subscribed to the Google Groups ā€œSearch Guard Community Forumā€ group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/1114f33e-cec1-46ca-8bdc-e4688853b366%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

ā€“
You received this message because you are subscribed to the Google Groups ā€œSearch Guard Community Forumā€ group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/CAAiY4T%3DQGnN_xtm%3DL%3D3U9T66foUG9pDnhjzKSxX5-rFYbMwVOg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

17

Can you please share a link on how to generate production ready certificates that can be used here. I am new to this and not sure how generate the certificates, and what king of certificates are these.

18

By far the easiest way is to use the offline TLS generator:

You will find a configuration example in the download, and the tool also generates configuration snippets for your elasticsearch.yml.

If you want to read more about the types of certificates, please see here:

1 Like