Sgadmin failing with certificate_unknown error with self-signed certificates

Elasticsearch version: 7.0.1
Deploying elasticsearch in k8s env as pods - 1 master node, 1 data node, 1 client node.

Describe the issue:
I have generated two self-signed certificates for node and client certificates respectively.
Node cert -

Owner: CN=elasticsearch.shiv, C=ELK
Issuer: CN=elasticsearch.shiv, C=ELK

SubjectAlternativeName [
DNSName: elasticsearch.shiv
DNSName: localhost
IPAddress: 127.0.0.1
OIDName: 1.2.3.4.5.5
]

Client certificate:

Owner: CN=admin, C=ELK
Issuer: CN=admin, C=ELK

  • Added the node certificate to keystore (searchguard.ssl.transport.keystore_filepath)
  • and the client certificate to another (client_keystore_filepath).
  • Added both the self-signed certs to another keystore and configured it as searchguard.ssl.transport.truststore_filepath.
    Truststore looks like -

Alias name: admin
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=admin, C=ELK
Issuer: CN=admin, C=ELK

Alias name: elasticsearch
Entry type: trustedCertEntry
Owner: CN=elasticsearch.shiv, C=ELK
Issuer: CN=elasticsearch.shiv, C=ELK

With this setup, master and data nodes are able to form the cluster. But sgadmin fails with this error -

/usr/share/elasticsearch/plugins/search-guard-7/tools/sgadmin.sh -cd $SG_CONFIG_DIRECTORY -ts $TRUSTSTORE_FILEPATH -ks $CLIENT_KEYSTORE_FILEPATH -cn $CLUSTER_NAME -kspass $KS_PWD -tspass $TS_PWD -h $HOSTNAME -nhnv

sgadmin started
Search Guard Admin v7
Will connect to es-elk-elasticsearch-client-64474bfddb-g4cfm:9300 … done
18:17:57.706 [elasticsearch[client][transport_worker][T#1]] ERROR com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport - SSL Problem Received fatal alert: certificate_unknown
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:307) ~[?:?]
at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:291) ~[?:?]
at sun.security.ssl.TransportContext.dispatch(TransportContext.java:180) ~[?:?]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:164) ~[?:?]
at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:668) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:623) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:441) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:420) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:634) ~[?:?]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1301) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]
ERR: Cannot connect to Elasticsearch. Please refer to elasticsearch logfile for more information
Trace:
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{okMhf-ztRoGluRtVyEByCA}{es-elk-elasticsearch-client-64474bfddb-g4cfm}{192.168.1.95:9300}]]
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:352)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:248)
at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:57)
at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:386)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:393)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:382)
at com.floragunn.searchguard.tools.SearchGuardAdmin.execute(SearchGuardAdmin.java:510)
at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:142)
sgadmin ended

Could you help me understand what is the issue here? Should the truststore be configured differently? Can it not work with self signed certs?

Provide configuration:
elasticsearch/config/elasticsearch.yml

cluster:
name: {CLUSTER_NAME} initial_master_nodes: {CLUSTER_INITIAL_MASTER_NODES}

node:
master: {NODE_MASTER} data: {NODE_DATA}
name: {NODE_NAME} ingest: {NODE_INGEST}
max_local_storage_nodes: ${MAX_LOCAL_STORAGE_NODES}

network.host: ${NETWORK_HOST}

path:
data: /data/data
logs: /data/log
repo: ${PATH_REPO}

#bootstrap.memory_lock: true

http:
compression: true
cors:
enabled: {HTTP_CORS_ENABLE} allow-origin: {HTTP_CORS_ALLOW_ORIGIN}

discovery:
seed_hosts: ${DISCOVERY_SERVICE}

searchguard:
ssl.transport:
enabled: {HTTP_SSL} enable_openssl_if_available: false keystore_type: {KEYSTORE_TYPE}
keystore_filepath: {KEYSTORE_FILEPATH} keystore_password: {KS_PWD}
truststore_type: {TRUSTSTORE_TYPE} truststore_filepath: {TRUSTSTORE_FILEPATH}
truststore_password: {TS_PWD} enforce_hostname_verification: false ssl.http: enabled: true clientauth_mode: OPTIONAL enable_openssl_if_available: true keystore_type: {KEYSTORE_TYPE}
keystore_filepath: {KEYSTORE_FILEPATH} keystore_password: {KS_PWD}
truststore_type: {TRUSTSTORE_TYPE} truststore_filepath: {TRUSTSTORE_FILEPATH}
truststore_password: {TS_PWD} authcz.admin_dn: - "{AUTH_ADMIN_DN}"
enterprise_modules_enabled: false

Elasticsearch Logs on running sgadmin:

{“type”:“log”,“host”:“es-elk-elasticsearch-client-64474bfddb-g4cfm”,“level”:“ERROR”,“systemid”:“e0d808ded4c343bc984b648ac8bfb732”,“system”:“ELK”,“time”: “2020-06-06T19:08:31.528Z”,“logger”:“c.f.s.s.t.SearchGuardSSLNettyTransport”,“timezone”:“UTC”,“marker”:"[es-elk-elasticsearch-client-64474bfddb-g4cfm] ",“log”:“SSL Problem PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target”}
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:320) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:263) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:258) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkClientCerts(CertificateMessage.java:1276) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1179) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1156) ~[?:?]
at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1048) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:995) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1464) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1369) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:276) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:135) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkClientCerts(CertificateMessage.java:1254) ~[?:?]
… 29 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[?:?]
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[?:?]
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297) ~[?:?]
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:276) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:135) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkClientCerts(CertificateMessage.java:1254) ~[?:?]
… 29 more

Can it not work with self signed certs?

It works with self-signed certificates.

elasticsearch.yml

 searchguard.authcz.admin_dn:
   - "{AUTH_ADMIN_DN}"

What is the value of AUTH_ADMIN_DN variable? Does it exist? Does the value match the admin certificate DN?

sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

This error means that the certificate you use in your sgadmin call could not be validated against the root (and intermediate certs if you have any) certificate. Look at this answer Certifiicates for Search Guard Causing an Issue.

Validate the chain of certificates.

Yes, the value of AUTH_ADMIN_DN matched the admin cert DN - “CN=admin,C=ELK”.

Thanks. There was something weird with my truststore. One alias had a chain length of 1, while the other alias did not show something like that.

Alias name: admin
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=admin, C=ELK
Issuer: CN=admin, C=ELK

Alias name: elasticsearch
Entry type: trustedCertEntry
Owner: CN=elasticsearch.shiv, C=ELK
Issuer: CN=elasticsearch.shiv, C=ELK

I regenerated the truststore by importing the certs like this -
keytool -import -alias elasticsearch -file es-cert.pem -keystore truststore.jks
keytool -import -alias admin -file admin-cert.pem -keystore truststore.jks

With this new truststore, sgadmin ran successfully.

1 Like