Unable to setup certificates

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version - 6.5.1

  • Installed and used enterprise modules, if any

  • JVM version and operating system version ubuntu 16.04 - java 8

  • Search Guard configuration files

elasticsearch.yml:

snippet generated by tlstool

searchguard.ssl.transport.pemcert_filepath: [[server domain name]].pem
searchguard.ssl.transport.pemkey_filepath: [[server domain name]].key
searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: false
searchguard.ssl.http.enabled: false
searchguard.nodes_dn:

  • CN=[[server domain name]],OU=Ops,O=[[company name]],DC=[[company domain name]],DC=com
    searchguard.authcz.admin_dn:
  • CN=[[server domain name]],OU=Ops,O=[[company name]],DC=[[company domain name]],DC=com
···

searchguard.allow_unsafe_democertificates: true

searchguard.allow_default_init_sgindex: true
searchguard.audit.type: internal_elasticsearch
searchguard.enable_snapshot_restore_privilege: true
searchguard.check_snapshot_restore_write_privileges: true
searchguard.restapi.roles_enabled: [“sg_all_access”]
cluster.routing.allocation.disk.threshold_enabled: false

cluster.name: searchguard_demo

network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1
node.max_local_storage_nodes: 3
xpack.security.enabled: false

``

tlsconfig.yml: (but we’ve tried many things, including all defaults)

[[data]] for anonymisation, tell me if it hides crucial data, two same tags have the exact same value.

ca:
root:
dn: CN=root.[[server domain name]],OU=CA,O=[[company name]],DC=[[company domain name]],DC=com
keysize: 2048
pkPassword: none
validityDays: 3650
file: root-ca.pem

defaults:
validityDays: 3650
pkPassword: none
httpsEnabled: false
verifyHostnames: false
resolveHostnames: false

nodes:

  • dn: CN=[[server domain name]],OU=Ops,[[company name]],DC=[[company domain name]],DC=com
    #dns: [[server domain name]]

clients:

  • name: kirk
    dn: CN=[[server domain name]],OU=Ops,O=[[company name]],DC=[[company domain name]],DC=com
    admin: true

``

  • Elasticsearch log messages on debug level

Trying to run sgadmin:

./sgadmin.sh -cd config/ -cacert out/root-ca.pem -cert out/kirk.pem -key out/kirk.key -icl

WARNING: JAVA_HOME not set, will use /usr/bin/java
Search Guard Admin v6
Will connect to localhost:9300 … done
ES Config path is not set
OpenSSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSL
No config directory, key- and truststore files are resolved absolutely
TLS Transport Client Provider : JDK
TLS Transport Server Provider : JDK
TLS HTTP Provider : null
Enabled TLS protocols for transport layer : [TLSv1.1, TLSv1.2]
Enabled TLS protocols for HTTP layer :
Clustername: elasticsearch
no modules loaded
loaded plugin [com.floragunn.searchguard.SearchGuardPlugin]
loaded plugin [org.elasticsearch.transport.Netty4Plugin]
send message failed [channel: NettyTcpChannel{localAddress=/127.0.0.1:43932, remoteAddress=localhost/127.0.0.1:9300}]
javax.net.ssl.SSLException: SSLEngine closed already
at io.netty.handler.ssl.SslHandler.wrap(…)(Unknown Source) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
Unable to check whether cluster is sane: None of the configured nodes are available: [{#transport#-1}{U63iKke9TV2IwEgtgZ8eKg}{localhost}{127.0.0.1:9300}]
SSL Problem General SSLEngine problem
javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1521) ~[?:1.8.0_191]
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:528) ~[?:1.8.0_191]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:802) ~[?:1.8.0_191]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766) ~[?:1.8.0_191]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_191]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:294) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1297) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-common-4.1.30.Final.jar:4.1.30.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:1.8.0_191]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1709) ~[?:1.8.0_191]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:318) ~[?:1.8.0_191]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) ~[?:1.8.0_191]
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) ~[?:1.8.0_191]
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) ~[?:1.8.0_191]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:1.8.0_191]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) ~[?:1.8.0_191]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) ~[?:1.8.0_191]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) ~[?:1.8.0_191]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1457) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1365) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
… 19 more
Caused by: java.security.cert.CertificateException: No name matching localhost found
at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:231) ~[?:1.8.0_191]
at sun.security.util.HostnameChecker.match(HostnameChecker.java:96) ~[?:1.8.0_191]
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455) ~[?:1.8.0_191]
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:436) ~[?:1.8.0_191]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:252) ~[?:1.8.0_191]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:1.8.0_191]
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1626) ~[?:1.8.0_191]
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) ~[?:1.8.0_191]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:1.8.0_191]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) ~[?:1.8.0_191]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) ~[?:1.8.0_191]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) ~[?:1.8.0_191]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1457) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1365) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
… 19 more
ERR: Cannot connect to Elasticsearch. Please refer to elasticsearch logfile for more information
Trace:

``

elasticsearch.log:

[2019-01-03T10:16:25,894][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [u8Zx_wh] SSL Problem Received fatal alert: certificate_unknown
javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1647) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1615) ~[?:?]
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1781) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1070) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:896) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_191]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:294) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1297) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-common-4.1.30.Final.jar:4.1.30.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
[2019-01-03T10:16:25,902][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [u8Zx_wh] SSL Problem Received fatal alert: certificate_unknown
javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1647) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1615) ~[?:?]
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1781) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1070) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:896) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_191]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:294) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1297) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:405) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:372) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:355) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1050) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1429) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:947) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-common-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) [netty-common-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-common-4.1.30.Final.jar:4.1.30.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]

``

  • Other installed Elasticsearch or Kibana plugins, if any

Hello,

After having a successful local instance of SG running, and having tweaked ES/LS/KB to our desire, we’re trying to deploy it on our servers following a production setup.

This implies not using the demo configuration.

We have been struggling really hard to setup the certificates correctly, and have unfortunately been stuck for two days on that part. We have only basic knowledge of certificates.

What we’d like to achieve: set SG on a 1 cluster setup, where everything is hosted on 1 server. We don’t need the fancy stuff but we can’t even get the basics running. For the moment encryption is not our concern, since the elastic stack is all on the same server.
We understand that SG requires a minimum of TLS setup and that’s what we’re looking to achieve.

So we just want self-signed certificates with: the root CA, one certificate for the only node, and one certificate for administration (use of SGadmin) that wil sit on the server as well. Not interested in DNS lookup etc.

We’ve tried tweaking the tlsconfig.yml over and over again, but the furthest we’ve been able to go is having ES and SG start ok, but then getting an error whenever we want to use SGadmin.

Taking the demo certificates from our local install did work, so the ES config

It seems strange to me that we’re having an error regarding ‘localhost’ in SGAdmin’s output.

Also we’ve been trying to run it with the very default provided by the documentation of the TLS tool (leaving example company name and DNS, since it shouldn’t matter with lookup disabled), which doesn’t work.

I guess there is something we’re missing or not understanding right, but can you see what?

Regards.

Your hostnames seems not matching.

Caused by: java.security.cert.CertificateException: No name matching localhost found at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:231) ~[?:1.8.0_191] at sun.security.util.HostnameChecker.match(HostnameChecker.java:96) ~[?:1.8.0_191] at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455) ~[?:1.8.0_191]

You can disable hostname verification applying the “-nhnv” flag to the sgadmin command

./sgadmin.sh -cd config/ -cacert out/root-ca.pem -cert out/kirk.pem -key out/kirk.key -icl -nhnv

See also https://docs.search-guard.com/latest/sgadmin#certificate-validation-settings

···

On Thursday, 3 January 2019 11:04:25 UTC+1, … wrote:

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version - 6.5.1
  • Installed and used enterprise modules, if any
  • JVM version and operating system version ubuntu 16.04 - java 8
  • Search Guard configuration files

elasticsearch.yml:

snippet generated by tlstool

searchguard.ssl.transport.pemcert_filepath: [[server domain name]].pem
searchguard.ssl.transport.pemkey_filepath: [[server domain name]].key
searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: false
searchguard.ssl.http.enabled: false
searchguard.nodes_dn:

  • CN=[[server domain name]],OU=Ops,O=[[company name]],DC=[[company domain name]],DC=com
    searchguard.authcz.admin_dn:
  • CN=[[server domain name]],OU=Ops,O=[[company name]],DC=[[company domain name]],DC=com

searchguard.allow_unsafe_democertificates: true

searchguard.allow_default_init_sgindex: true
searchguard.audit.type: internal_elasticsearch
searchguard.enable_snapshot_restore_privilege: true
searchguard.check_snapshot_restore_write_privileges: true
searchguard.restapi.roles_enabled: [“sg_all_access”]
cluster.routing.allocation.disk.threshold_enabled: false

cluster.name: searchguard_demo

network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1
node.max_local_storage_nodes: 3
xpack.security.enabled: false

``

tlsconfig.yml: (but we’ve tried many things, including all defaults)

[[data]] for anonymisation, tell me if it hides crucial data, two same tags have the exact same value.

ca:
root:
dn: CN=root.[[server domain name]],OU=CA,O=[[company name]],DC=[[company domain name]],DC=com
keysize: 2048
pkPassword: none
validityDays: 3650
file: root-ca.pem

defaults:
validityDays: 3650
pkPassword: none
httpsEnabled: false
verifyHostnames: false
resolveHostnames: false

nodes:

  • dn: CN=[[server domain name]],OU=Ops,[[company name]],DC=[[company domain name]],DC=com
    #dns: [[server domain name]]

clients:

  • name: kirk
    dn: CN=[[server domain name]],OU=Ops,O=[[company name]],DC=[[company domain name]],DC=com
    admin: true

``

  • Elasticsearch log messages on debug level

Trying to run sgadmin:

./sgadmin.sh -cd config/ -cacert out/root-ca.pem -cert out/kirk.pem -key out/kirk.key -icl

WARNING: JAVA_HOME not set, will use /usr/bin/java
Search Guard Admin v6
Will connect to localhost:9300 … done
ES Config path is not set
OpenSSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSL
No config directory, key- and truststore files are resolved absolutely
TLS Transport Client Provider : JDK
TLS Transport Server Provider : JDK
TLS HTTP Provider : null
Enabled TLS protocols for transport layer : [TLSv1.1, TLSv1.2]
Enabled TLS protocols for HTTP layer :
Clustername: elasticsearch
no modules loaded
loaded plugin [com.floragunn.searchguard.SearchGuardPlugin]
loaded plugin [org.elasticsearch.transport.Netty4Plugin]
send message failed [channel: NettyTcpChannel{localAddress=/127.0.0.1:43932, remoteAddress=localhost/127.0.0.1:9300}]
javax.net.ssl.SSLException: SSLEngine closed already
at io.netty.handler.ssl.SslHandler.wrap(…)(Unknown Source) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
Unable to check whether cluster is sane: None of the configured nodes are available: [{#transport#-1}{U63iKke9TV2IwEgtgZ8eKg}{localhost}{127.0.0.1:9300}]
SSL Problem General SSLEngine problem
javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1521) ~[?:1.8.0_191]
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:528) ~[?:1.8.0_191]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:802) ~[?:1.8.0_191]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766) ~[?:1.8.0_191]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_191]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:294) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1297) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-common-4.1.30.Final.jar:4.1.30.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:1.8.0_191]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1709) ~[?:1.8.0_191]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:318) ~[?:1.8.0_191]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) ~[?:1.8.0_191]
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) ~[?:1.8.0_191]
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) ~[?:1.8.0_191]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:1.8.0_191]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) ~[?:1.8.0_191]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) ~[?:1.8.0_191]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) ~[?:1.8.0_191]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1457) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1365) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
… 19 more
Caused by: java.security.cert.CertificateException: No name matching localhost found
at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:231) ~[?:1.8.0_191]
at sun.security.util.HostnameChecker.match(HostnameChecker.java:96) ~[?:1.8.0_191]
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455) ~[?:1.8.0_191]
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:436) ~[?:1.8.0_191]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:252) ~[?:1.8.0_191]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:1.8.0_191]
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1626) ~[?:1.8.0_191]
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) ~[?:1.8.0_191]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:1.8.0_191]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) ~[?:1.8.0_191]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) ~[?:1.8.0_191]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) ~[?:1.8.0_191]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1457) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1365) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
… 19 more
ERR: Cannot connect to Elasticsearch. Please refer to elasticsearch logfile for more information
Trace:

``

elasticsearch.log:

[2019-01-03T10:16:25,894][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [u8Zx_wh] SSL Problem Received fatal alert: certificate_unknown
javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1647) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1615) ~[?:?]
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1781) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1070) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:896) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_191]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:294) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1297) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-common-4.1.30.Final.jar:4.1.30.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
[2019-01-03T10:16:25,902][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [u8Zx_wh] SSL Problem Received fatal alert: certificate_unknown
javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1647) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1615) ~[?:?]
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1781) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1070) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:896) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_191]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:294) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1297) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:405) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:372) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:355) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1050) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1429) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:947) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-common-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) [netty-common-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-common-4.1.30.Final.jar:4.1.30.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]

``

  • Other installed Elasticsearch or Kibana plugins, if any

Hello,

After having a successful local instance of SG running, and having tweaked ES/LS/KB to our desire, we’re trying to deploy it on our servers following a production setup.

This implies not using the demo configuration.

We have been struggling really hard to setup the certificates correctly, and have unfortunately been stuck for two days on that part. We have only basic knowledge of certificates.

What we’d like to achieve: set SG on a 1 cluster setup, where everything is hosted on 1 server. We don’t need the fancy stuff but we can’t even get the basics running. For the moment encryption is not our concern, since the elastic stack is all on the same server.
We understand that SG requires a minimum of TLS setup and that’s what we’re looking to achieve.

So we just want self-signed certificates with: the root CA, one certificate for the only node, and one certificate for administration (use of SGadmin) that wil sit on the server as well. Not interested in DNS lookup etc.

We’ve tried tweaking the tlsconfig.yml over and over again, but the furthest we’ve been able to go is having ES and SG start ok, but then getting an error whenever we want to use SGadmin.

Taking the demo certificates from our local install did work, so the ES config

It seems strange to me that we’re having an error regarding ‘localhost’ in SGAdmin’s output.

Also we’ve been trying to run it with the very default provided by the documentation of the TLS tool (leaving example company name and DNS, since it shouldn’t matter with lookup disabled), which doesn’t work.

I guess there is something we’re missing or not understanding right, but can you see what?

Regards.

Solved my issue, then had to solve an issue regarding the admin certificate and node one being too similar, thus SG thinking I was connecting with a node certificate.

Thank you very much!