UnicastZenPing failing due to a "bad header". Any hints as to how I can locate this header and fix?

Search Guard
1

Has anyone encountered this problem before?

[2017-07-12T09:28:26,518][WARN ][o.e.d.z.ZenDiscovery     ] [client_hostname.domain] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again

[2017-07-12T09:28:29,184][WARN ][o.e.d.z.UnicastZenPing   ] [client_hostname.domain] [3] failed send ping to {#zen_unicast_hostname.domain:9301_0#}{79tDJstFSJ-enaTA4DiWLw}{hostname.domain}{XXX.XXX.XXX.XXX:9301}

java.lang.IllegalStateException: handshake failed with {#zen_unicast_hostname.domain:9301_0#}{79tDJstFSJ-enaTA4DiWLw}{hostname.domain}{XXX.XXX.XXX.XXX:9301}

at org.elasticsearch.transport.TransportService.handshake(TransportService.java:386) ~[elasticsearch-5.4.2.jar:5.4.2]

at org.elasticsearch.transport.TransportService.handshake(TransportService.java:353) ~[elasticsearch-5.4.2.jar:5.4.2]

at org.elasticsearch.discovery.zen.UnicastZenPing$PingingRound.getOrConnect(UnicastZenPing.java:401) ~[elasticsearch-5.4.2.jar:5.4.2]

at org.elasticsearch.discovery.zen.UnicastZenPing$3.doRun(UnicastZenPing.java:508) [elasticsearch-5.4.2.jar:5.4.2]

at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.4.2.jar:5.4.2]

at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.4.2.jar:5.4.2]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_102]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_102]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102]

Caused by: org.elasticsearch.transport.RemoteTransportException: [master_hostname.domain][XXX.XXX.XXX.XXX:9301][internal:transport/handshake]

Caused by: org.elasticsearch.ElasticsearchException: bad header found

at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:158) ~[?:?]

at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:139) ~[?:?]

at com.floragunn.searchguard.SearchGuardPlugin$2$1.messageReceived(SearchGuardPlugin.java:338) ~[?:?]



<details class='elided'>
<summary title='Show trimmed content'>&#183;&#183;&#183;</summary>

=================
elasticsearch.yml
=================
searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.keystore_filepath: /PATH/serverKeyStore.jks
searchguard.ssl.transport.truststore_filepath: /PATH/serverTrustStore.jks
searchguard.ssl.http.enabled: true
searchguard.ssl.http.keystore_filepath: /PATH/serverKeyStore.jks
searchguard.ssl.http.truststore_filepath: /PATH/serverTrustStore.jks
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.keystore_password: XXX
searchguard.ssl.transport.truststore_password: XXX
searchguard.ssl.http.keystore_password: XXX
searchguard.ssl.http.truststore_password: XXX
searchguard.ssl.transport.keystore_alias: srvalias
searchguard.ssl.transport.truststore_alias: srvalias
searchguard.ssl.http.keystore_alias: srvalias
searchguard.ssl.http.truststore_alias: srvalias
searchguard.ssl.http.enabled_protocols:
- "TLSv1"
- "TLSv1.1"
- "TLSv1.2"


=======================
Env details
=======================
search-guard-5
elasticsearch-5.4.2
OpenJDK
Red Hat Enterprise Linux Server release 7.3 (Maipo)
1-node
2

Update:
The above error was from my client log. After looking at the master log, I see this error:

[2017-07-12T10:37:57,673][ERROR][c.f.s.t.SearchGuardRequestHandler] Error validating headers

[2017-07-12T10:37:58,359][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [master_hostname.domain] SSL Problem null cert chain

javax.net.ssl.SSLHandshakeException: null cert chain

at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1431) ~[?:?]

at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[?:?]

at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[?:?]

at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]

at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_102]

at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:254) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]

at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1156) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]

at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]

at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]

at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]

at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.11.Final.jar:4.1.11.Final]

at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.11.Final.jar:4.1.11.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]

at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.11.Final.jar:4.1.11.Final]

at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.11.Final.jar:4.1.11.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.11.Final.jar:4.1.11.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579) [netty-transport-4.1.11.Final.jar:4.1.11.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496) [netty-transport-4.1.11.Final.jar:4.1.11.Final]

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.11.Final.jar:4.1.11.Final]

at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.11.Final.jar:4.1.11.Final]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102]

Caused by: javax.net.ssl.SSLHandshakeException: null cert chain

at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]

at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[?:?]

at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304) ~[?:?]

at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:292) ~[?:?]

at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1865) ~[?:?]

at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:230) ~[?:?]

at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979) ~[?:?]

at sun.security.ssl.Handshaker$1.run(Handshaker.java:919) ~[?:?]

at sun.security.ssl.Handshaker$1.run(Handshaker.java:916) ~[?:?]

at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_102]

at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1369) ~[?:?]

at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1295) ~[?:?]

at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1208) ~[?:?]

… 18 more

3

Update again…

I was looking at the source code, seeing where the error message about the bad headers was called, and I see this:

try {
HeaderHelper.checkSGHeader(request);
} catch (Exception e) {
auditLog.logBadHeaders(request);
log.error("Error validating headers "+e, e);
transportChannel.sendResponse(ExceptionsHelper.convertToElastic(e));
return;
}

**I then searched for that “checkSGHeader” method, and I saw that it thows an exception if there is a “sg” at the start of the header. I also noticed that it does an audit of a “logBadHeaders”, which, I see on this page, https://github.com/floragunncom/search-guard-docs/blob/master/auditlogging.md, —> **

  • BAD_HEADERS—an attempt was made to spoof a request to Elasticsearch with Search Guard internal headers.

Ok. Now I believe I can see what is happening – the next step is to try to figure out why this is happening, and think of a way to fix it. Anybody know?

4

"javax.net.ssl.SSLHandshakeException: null cert chain" points to a SSL/cert misconfiguration and is likely the root cause for your problem.

I strongly recommend to use Security and Alerting for Elasticsearch and Kibana | Search Guard to get it up and running and then switch to your own certificates.
Especially if you are not familiar with SSL/TLS you spend a lot of time getting this to work before you can explore the functionality and features.

Alternatively use GitHub - floragunncom/search-guard: Search Guard Plugin - Security for Elasticsearch

···

Am 12.07.2017 um 17:05 schrieb Steve Haertel <stevehaertel@gmail.com>:

Update again...

I was looking at the source code, seeing where the error message about the bad headers was called, and I see this:

try {
HeaderHelper.checkSGHeader(request);
} catch (Exception e) {
auditLog.logBadHeaders(request);
log.error("Error validating headers "+e, e);
transportChannel.sendResponse(ExceptionsHelper.convertToElastic(e));
return;
}

I then searched for that "checkSGHeader" method, and I saw that it thows an exception if there is a "_sg_" at the start of the header. I also noticed that it does an audit of a "logBadHeaders", which, I see on this page, https://github.com/floragunncom/search-guard-docs/blob/master/auditlogging.md, --->
  • BAD_HEADERS—an attempt was made to spoof a request to Elasticsearch with Search Guard internal headers.

Ok. Now I believe I can see what is happening -- the next step is to try to figure out why this is happening, and think of a way to fix it. Anybody know?

--
You received this message because you are subscribed to the Google Groups "Search Guard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/2c4ad824-c74f-4b18-924b-d4cf577c9953%40googlegroups.com\.
For more options, visit https://groups.google.com/d/optout\.

5

Hmm yeah. Actually this may sound weird, but exploring the function/features isn’t my goal. Getting my certificates to do the same thing as the generated ones (which I did get!) is my primary objective here. I have to figure out just what the heck are the differences between my keystore certs and the generated ones…

···

On Wednesday, July 12, 2017 at 11:27:36 AM UTC-4, Search Guard wrote:

“javax.net.ssl.SSLHandshakeException: null cert chain” points to a SSL/cert misconfiguration and is likely the root cause for your problem.

I strongly recommend to use https://floragunn.com/tls-certificate-generator/ to get it up and running and then switch to your own certificates.

Especially if you are not familiar with SSL/TLS you spend a lot of time getting this to work before you can explore the functionality and features.

Alternatively use https://github.com/floragunncom/search-guard/wiki/Search-Guard-Bundle

Am 12.07.2017 um 17:05 schrieb Steve Haertel steveh...@gmail.com:

Update again…

I was looking at the source code, seeing where the error message about the bad headers was called, and I see this:

try {

HeaderHelper.checkSGHeader(request);

} catch (Exception e) {

auditLog.logBadHeaders(request);

log.error("Error validating headers "+e, e);

transportChannel.sendResponse(ExceptionsHelper.convertToElastic(e));

return;

}

I then searched for that “checkSGHeader” method, and I saw that it thows an exception if there is a “sg” at the start of the header. I also noticed that it does an audit of a “logBadHeaders”, which, I see on this page, https://github.com/floragunncom/search-guard-docs/blob/master/auditlogging.md, —>
• BAD_HEADERS—an attempt was made to spoof a request to Elasticsearch with Search Guard internal headers.

Ok. Now I believe I can see what is happening – the next step is to try to figure out why this is happening, and think of a way to fix it. Anybody know?

–
You received this message because you are subscribed to the Google Groups “Search Guard” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/2c4ad824-c74f-4b18-924b-d4cf577c9953%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

6

We can narrow down what the problem is at least…

**I went ahead and used the generated certificates in place of my own, but I ended up with the same error. So at least we know the certificates themselves are not the problem. Probably some config problem? **

Anyone, help? :frowning:

···

======================================

logger.com.floragunn.searchguard.ssl: DEBUG

searchguard.ssl.http.clientauth_mode: REQUIRE

searchguard.ssl.transport.enabled: true

searchguard.ssl.transport.keystore_filepath: stores/serverKeyStore.jks

searchguard.ssl.transport.truststore_filepath: stores/serverTrustStore.jks

searchguard.ssl.http.enabled: true

searchguard.ssl.http.keystore_filepath: stores/serverKeyStore.jks

searchguard.ssl.http.truststore_filepath: stores/serverTrustStore.jks

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.keystore_password: kpwd

searchguard.ssl.transport.truststore_password: tpwd

searchguard.ssl.http.keystore_password: kpwd

searchguard.ssl.http.truststore_password: tpwd

searchguard.ssl.transport.keystore_alias: cn=hostname.domain

searchguard.ssl.transport.truststore_alias: root-ca-chain

searchguard.ssl.http.keystore_alias: cn=hostname.domain

searchguard.ssl.http.truststore_alias: root-ca-chain

searchguard.ssl.transport.enabled_protocols:

  • “TLSv1.2”

searchguard.ssl.http.enabled_protocols:

  • “TLSv1.2”

==============================================

[2017-07-12T15:32:22,922][INFO ][o.e.n.Node ] [master_hostname.domain] initializing …

[2017-07-12T15:32:23,437][INFO ][o.e.e.NodeEnvironment ] [master_hostname.domain] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [6.3gb], net total_space [16.9gb], spins? [unknown], types [rootfs]

[2017-07-12T15:32:23,437][INFO ][o.e.e.NodeEnvironment ] [master_hostname.domain] heap size [1.9gb], compressed ordinary object pointers [true]

[2017-07-12T15:32:23,520][INFO ][o.e.n.Node ] [master_hostname.domain] node name [master_hostname.domain], node ID [jH94B8oWQTiKrNfu-3eQuA]

[2017-07-12T15:32:23,520][INFO ][o.e.n.Node ] [master_hostname.domain] version[5.4.2], pid[6068], build[929b078/2017-06-15T02:29:28.122Z], OS[Linux/3.10.0-514.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_102/25.102-b14]

[2017-07-12T15:32:23,521][INFO ][o.e.n.Node ] [master_hostname.domain] JVM arguments [-Xms2g, -Xmx2g, -Djavax.net.ssl.trustStore=/PATH/serverTrustStore.jks, -Djavax.net.ssl.trustAnchors=/PATH/serverTrustStore.jks, -Djavax.net.ssl.keyStore=/PATH/serverKeyStore.jks, -Djavax.net.debug=ssl:handshake, -Dlog4j2.disable.jmx=true, -Djava.security.policy=/PATH/plugin-security.policy, -Des.path.home=/PATH]

[2017-07-12T15:32:34,739][INFO ][c.f.s.SearchGuardPlugin ] Clustername: elk-CwS2.2.1-0705

[2017-07-12T15:32:34,989][INFO ][c.f.s.SearchGuardPlugin ] Node [master_hostname.domain] is a transportClient: false/tribeNode: false/tribeNodeClient: false

[2017-07-12T15:32:35,101][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSL

[2017-07-12T15:32:35,102][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.version: 1.8.0_102

[2017-07-12T15:32:35,102][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vendor: Oracle Corporation

[2017-07-12T15:32:35,102][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.specification.version: 1.8

[2017-07-12T15:32:35,102][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.specification.vendor: Oracle Corporation

[2017-07-12T15:32:35,102][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.specification.name: Java Virtual Machine Specification

[2017-07-12T15:32:35,102][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.name: OpenJDK 64-Bit Server VM

[2017-07-12T15:32:35,102][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.vendor: Oracle Corporation

[2017-07-12T15:32:35,102][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.specification.version: 1.8

[2017-07-12T15:32:35,102][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.specification.vendor: Oracle Corporation

[2017-07-12T15:32:35,103][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.specification.name: Java Platform API Specification

[2017-07-12T15:32:35,103][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] os.name: Linux

[2017-07-12T15:32:35,103][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] os.arch: amd64

[2017-07-12T15:32:35,103][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] os.version: 3.10.0-514.el7.x86_64

[2017-07-12T15:32:36,835][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] JVM supports the following 82 ciphers for https [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV, TLS_DH_anon_WITH_AES_256_GCM_SHA384, TLS_DH_anon_WITH_AES_128_GCM_SHA256, TLS_DH_anon_WITH_AES_256_CBC_SHA256, TLS_ECDH_anon_WITH_AES_256_CBC_SHA, TLS_DH_anon_WITH_AES_256_CBC_SHA, TLS_DH_anon_WITH_AES_128_CBC_SHA256, TLS_ECDH_anon_WITH_AES_128_CBC_SHA, TLS_DH_anon_WITH_AES_128_CBC_SHA, TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA, SSL_DH_anon_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_DH_anon_WITH_DES_CBC_SHA, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA, SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA, TLS_RSA_WITH_NULL_SHA256, TLS_ECDHE_ECDSA_WITH_NULL_SHA, TLS_ECDHE_RSA_WITH_NULL_SHA, SSL_RSA_WITH_NULL_SHA, TLS_ECDH_ECDSA_WITH_NULL_SHA, TLS_ECDH_RSA_WITH_NULL_SHA, TLS_ECDH_anon_WITH_NULL_SHA, SSL_RSA_WITH_NULL_MD5, TLS_KRB5_WITH_3DES_EDE_CBC_SHA, TLS_KRB5_WITH_3DES_EDE_CBC_MD5, TLS_KRB5_WITH_DES_CBC_SHA, TLS_KRB5_WITH_DES_CBC_MD5, TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA, TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5]

[2017-07-12T15:32:36,875][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] JVM supports the following 82 ciphers for transport [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV, TLS_DH_anon_WITH_AES_256_GCM_SHA384, TLS_DH_anon_WITH_AES_128_GCM_SHA256, TLS_DH_anon_WITH_AES_256_CBC_SHA256, TLS_ECDH_anon_WITH_AES_256_CBC_SHA, TLS_DH_anon_WITH_AES_256_CBC_SHA, TLS_DH_anon_WITH_AES_128_CBC_SHA256, TLS_ECDH_anon_WITH_AES_128_CBC_SHA, TLS_DH_anon_WITH_AES_128_CBC_SHA, TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA, SSL_DH_anon_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_DH_anon_WITH_DES_CBC_SHA, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA, SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA, TLS_RSA_WITH_NULL_SHA256, TLS_ECDHE_ECDSA_WITH_NULL_SHA, TLS_ECDHE_RSA_WITH_NULL_SHA, SSL_RSA_WITH_NULL_SHA, TLS_ECDH_ECDSA_WITH_NULL_SHA, TLS_ECDH_RSA_WITH_NULL_SHA, TLS_ECDH_anon_WITH_NULL_SHA, SSL_RSA_WITH_NULL_MD5, TLS_KRB5_WITH_3DES_EDE_CBC_SHA, TLS_KRB5_WITH_3DES_EDE_CBC_MD5, TLS_KRB5_WITH_DES_CBC_SHA, TLS_KRB5_WITH_DES_CBC_MD5, TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA, TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5]

[2017-07-12T15:32:36,877][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Config directory is /PATH/integration/elk/conf/elasticsearch/, from there the key- and truststore files are resolved relatively

[2017-07-12T15:32:36,885][DEBUG][c.f.s.s.u.SSLCertificateHelper] Keystore has 1 entries/aliases

[2017-07-12T15:32:36,885][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias cn=hostname.domain: is a certificate entry?false/is a key entry?true

[2017-07-12T15:32:36,886][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias cn=hostname.domain: chain len 3

[2017-07-12T15:32:36,886][DEBUG][c.f.s.s.u.SSLCertificateHelper] cert CN=hostname.domain of type -1 → false

[2017-07-12T15:32:36,888][DEBUG][c.f.s.s.u.SSLCertificateHelper] cert CN=ORGANIZATION Canada Signing CA, OU=ORGANIZATION Canada Signing CA, O=ORGANIZATION Canada of type 0 → false

[2017-07-12T15:32:36,889][DEBUG][c.f.s.s.u.SSLCertificateHelper] cert CN=ORGANIZATION Canada Root CA, OU=ORGANIZATION Canada Root CA, O=ORGANIZATION Canada of type 2147483647 → true

[2017-07-12T15:32:36,889][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias cn=hostname.domain: single cert CN=hostname.domain of type -1 → false

[2017-07-12T15:32:36,889][WARN ][c.f.s.s.u.SSLCertificateHelper] Certificate chain for alias cn=hostname.domain contains a root certificate

[2017-07-12T15:32:36,890][DEBUG][c.f.s.s.u.SSLCertificateHelper] Keystore has 1 entries/aliases

[2017-07-12T15:32:36,890][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias cn=hostname.domain: is a certificate entry?false/is a key entry?true

[2017-07-12T15:32:36,891][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias cn=hostname.domain: chain len 3

[2017-07-12T15:32:36,983][DEBUG][c.f.s.s.u.SSLCertificateHelper] cert CN=hostname.domain of type -1 → false

[2017-07-12T15:32:36,983][DEBUG][c.f.s.s.u.SSLCertificateHelper] cert CN=ORGANIZATION Canada Signing CA, OU=ORGANIZATION Canada Signing CA, O=ORGANIZATION Canada of type 0 → false

[2017-07-12T15:32:36,983][DEBUG][c.f.s.s.u.SSLCertificateHelper] cert CN=ORGANIZATION Canada Root CA, OU=ORGANIZATION Canada Root CA, O=ORGANIZATION Canada of type 2147483647 → true

[2017-07-12T15:32:36,983][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias cn=hostname.domain: single cert CN=hostname.domain of type -1 → false

[2017-07-12T15:32:36,996][DEBUG][c.f.s.s.u.SSLCertificateHelper] Keystore has 1 entries/aliases

[2017-07-12T15:32:36,996][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias root-ca-chain: is a certificate entry?true/is a key entry?false

[2017-07-12T15:32:36,996][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias root-ca-chain: single cert CN=ORGANIZATION Canada Root CA, OU=ORGANIZATION Canada Root CA, O=ORGANIZATION Canada of type 2147483647 → true

[2017-07-12T15:32:37,170][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] HTTPS client auth mode REQUIRE

[2017-07-12T15:32:37,234][DEBUG][c.f.s.s.u.SSLCertificateHelper] Keystore has 1 entries/aliases

[2017-07-12T15:32:37,234][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias cn=hostname.domain: is a certificate entry?false/is a key entry?true

[2017-07-12T15:32:37,234][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias cn=hostname.domain: chain len 3

[2017-07-12T15:32:37,234][DEBUG][c.f.s.s.u.SSLCertificateHelper] cert CN=hostname.domain of type -1 → false

[2017-07-12T15:32:37,234][DEBUG][c.f.s.s.u.SSLCertificateHelper] cert CN=ORGANIZATION Canada Signing CA, OU=ORGANIZATION Canada Signing CA, O=ORGANIZATION Canada of type 0 → false

[2017-07-12T15:32:37,235][DEBUG][c.f.s.s.u.SSLCertificateHelper] cert CN=ORGANIZATION Canada Root CA, OU=ORGANIZATION Canada Root CA, O=ORGANIZATION Canada of type 2147483647 → true

[2017-07-12T15:32:37,235][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias cn=hostname.domain: single cert CN=hostname.domain of type -1 → false

[2017-07-12T15:32:37,235][WARN ][c.f.s.s.u.SSLCertificateHelper] Certificate chain for alias cn=hostname.domain contains a root certificate

[2017-07-12T15:32:37,235][DEBUG][c.f.s.s.u.SSLCertificateHelper] Keystore has 1 entries/aliases

[2017-07-12T15:32:37,235][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias cn=hostname.domain: is a certificate entry?false/is a key entry?true

[2017-07-12T15:32:37,235][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias cn=hostname.domain: chain len 3

[2017-07-12T15:32:37,235][DEBUG][c.f.s.s.u.SSLCertificateHelper] cert CN=hostname.domain of type -1 → false

[2017-07-12T15:32:37,235][DEBUG][c.f.s.s.u.SSLCertificateHelper] cert CN=ORGANIZATION Canada Signing CA, OU=ORGANIZATION Canada Signing CA, O=ORGANIZATION Canada of type 0 → false

[2017-07-12T15:32:37,236][DEBUG][c.f.s.s.u.SSLCertificateHelper] cert CN=ORGANIZATION Canada Root CA, OU=ORGANIZATION Canada Root CA, O=ORGANIZATION Canada of type 2147483647 → true

[2017-07-12T15:32:37,236][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias cn=hostname.domain: single cert CN=hostname.domain of type -1 → false

[2017-07-12T15:32:37,237][DEBUG][c.f.s.s.u.SSLCertificateHelper] Keystore has 1 entries/aliases

[2017-07-12T15:32:37,237][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias root-ca-chain: is a certificate entry?true/is a key entry?false

[2017-07-12T15:32:37,237][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias root-ca-chain: single cert CN=ORGANIZATION Canada Root CA, OU=ORGANIZATION Canada Root CA, O=ORGANIZATION Canada of type 2147483647 → true

[2017-07-12T15:32:37,245][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslTransportClientProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]

[2017-07-12T15:32:37,245][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslTransportServerProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]

[2017-07-12T15:32:37,246][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslHTTPProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]

[2017-07-12T15:32:37,246][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslTransport protocols [TLSv1.2]

[2017-07-12T15:32:37,246][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslHTTP protocols [TLSv1.2]

[2017-07-12T15:32:37,249][INFO ][o.e.p.PluginsService ] [master_hostname.domain] loaded module [aggs-matrix-stats]

[2017-07-12T15:32:37,249][INFO ][o.e.p.PluginsService ] [master_hostname.domain] loaded module [ingest-common]

[2017-07-12T15:32:37,249][INFO ][o.e.p.PluginsService ] [master_hostname.domain] loaded module [lang-expression]

[2017-07-12T15:32:37,249][INFO ][o.e.p.PluginsService ] [master_hostname.domain] loaded module [lang-groovy]

[2017-07-12T15:32:37,249][INFO ][o.e.p.PluginsService ] [master_hostname.domain] loaded module [lang-mustache]

[2017-07-12T15:32:37,249][INFO ][o.e.p.PluginsService ] [master_hostname.domain] loaded module [lang-painless]

[2017-07-12T15:32:37,250][INFO ][o.e.p.PluginsService ] [master_hostname.domain] loaded module [percolator]

[2017-07-12T15:32:37,250][INFO ][o.e.p.PluginsService ] [master_hostname.domain] loaded module [reindex]

[2017-07-12T15:32:37,250][INFO ][o.e.p.PluginsService ] [master_hostname.domain] loaded module [transport-netty3]

[2017-07-12T15:32:37,250][INFO ][o.e.p.PluginsService ] [master_hostname.domain] loaded module [transport-netty4]

[2017-07-12T15:32:37,250][INFO ][o.e.p.PluginsService ] [master_hostname.domain] loaded plugin [search-guard-5]

[2017-07-12T15:32:47,316][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin com.floragunn.searchguard.SearchGuardPlugin

[2017-07-12T15:32:47,651][INFO ][c.f.s.a.BackendRegistry ] Register EgoAuthenticationBackend()

[2017-07-12T15:32:47,930][INFO ][o.e.d.DiscoveryModule ] [master_hostname.domain] using discovery type [zen]

[2017-07-12T15:32:52,995][INFO ][o.e.n.Node ] [master_hostname.domain] initialized

[2017-07-12T15:32:52,995][INFO ][o.e.n.Node ] [master_hostname.domain] starting …

[2017-07-12T15:32:53,414][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport] [master_hostname.domain] using profile[default], worker_count[4], port[9301], bind_host[null], publish_host[null], compress[false], connect_timeout[30s], connections_per_node[0/3/6/1/1], receive_predictor[64kb->64kb]

[2017-07-12T15:32:53,420][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport] [master_hostname.domain] binding server bootstrap to: [XXX.XXX.XXX.XXX]

[2017-07-12T15:32:54,196][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport] [master_hostname.domain] Bound profile [default] to address {XXX.XXX.XXX.XXX:9301}

[2017-07-12T15:32:54,197][INFO ][o.e.t.TransportService ] [master_hostname.domain] publish_address {XXX.XXX.XXX.XXX:9301}, bound_addresses {XXX.XXX.XXX.XXX:9301}

[2017-07-12T15:32:54,204][INFO ][o.e.b.BootstrapChecks ] [master_hostname.domain] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks

[2017-07-12T15:32:57,992][INFO ][o.e.c.s.ClusterService ] [master_hostname.domain] new_master {master_hostname.domain}{jH94B8oWQTiKrNfu-3eQuA}{aJMQWztPRlmVg5LnNlAvkg}{hostname.domain}{XXX.XXX.XXX.XXX:9301}, reason: zen-disco-elected-as-master ([0] nodes joined)

[2017-07-12T15:32:58,592][INFO ][c.f.s.h.SearchGuardHttpServerTransport] [master_hostname.domain] publish_address {127.0.0.1:9201}, bound_addresses {XXX.XXX.XXX.XXX:9201}

[2017-07-12T15:32:58,646][INFO ][o.e.n.Node ] [master_hostname.domain] started

[2017-07-12T15:33:01,290][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport] [master_hostname.domain] connected to node [{data_hostname.domain}{2y1AAE8SSB-ZiIeHJNRvTg}{EgseokcITtKt6fEt52FrtA}{hostname.domain}{XXX.XXX.XXX.XXX:9302}]

[2017-07-12T15:33:02,284][INFO ][o.e.g.GatewayService ] [master_hostname.domain] recovered [15] indices into cluster_state

[2017-07-12T15:33:03,195][INFO ][o.e.c.s.ClusterService ] [master_hostname.domain] added {{data_hostname.domain}{2y1AAE8SSB-ZiIeHJNRvTg}{EgseokcITtKt6fEt52FrtA}{hostname.domain}{XXX.XXX.XXX.XXX:9302},}, reason: zen-disco-node-join[{data_hostname.domain}{2y1AAE8SSB-ZiIeHJNRvTg}{EgseokcITtKt6fEt52FrtA}{hostname.domain}{XXX.XXX.XXX.XXX:9302}]

[2017-07-12T15:33:03,212][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport] [master_hostname.domain] connected to node [{client_hostname.domain}{svr4h2-EQVWWKJdek19CQg}{PjFrAfHxSeOXMtlK07F1dQ}{hostname.domain}{XXX.XXX.XXX.XXX:9300}]

[2017-07-12T15:33:03,502][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [master_hostname.domain] SSL Problem null cert chain

javax.net.ssl.SSLHandshakeException: null cert chain

at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1431) ~[?:?]

at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[?:?]

at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[?:?]

at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]

at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_102]

7

(I should have specified in my previous post – I used the generated trust store, and node keystore and renamed the files to the same name/location that I had before when I was trying my own key/trust stores.)

8

try skipping the ks/ts aliases:

searchguard.ssl.http.clientauth_mode: REQUIRE
searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.keystore_filepath: stores/serverKeyStore.jks
searchguard.ssl.transport.truststore_filepath: stores/serverTrustStore.jks
searchguard.ssl.http.enabled: true
searchguard.ssl.http.keystore_filepath: stores/serverKeyStore.jks
searchguard.ssl.http.truststore_filepath: stores/serverTrustStore.jks
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.keystore_password: kpwd
searchguard.ssl.transport.truststore_password: tpwd
searchguard.ssl.http.keystore_password: kpwd
searchguard.ssl.http.truststore_password: tpwd
###searchguard.ssl.transport.keystore_alias: cn=hostname.domain #use defaults
###searchguard.ssl.transport.truststore_alias: root-ca-chain #use defaults
###searchguard.ssl.http.keystore_alias: cn=hostname.domain #use defaults
###searchguard.ssl.http.truststore_alias: root-ca-chain #use defaults
searchguard.ssl.transport.enabled_protocols:
  - "TLSv1.2"
searchguard.ssl.http.enabled_protocols:
  - "TLSv1.2"

···

Am 12.07.2017 um 21:42 schrieb Steve Haertel <stevehaertel@gmail.com>:

(I should have specified in my previous post -- I used the generated trust store, and node keystore and renamed the files to the same name/location that I had before when I was trying my own key/trust stores.)

--
You received this message because you are subscribed to the Google Groups "Search Guard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/d367905b-a5d5-4429-bfe0-1e0c007ab156%40googlegroups.com\.
For more options, visit https://groups.google.com/d/optout\.

9

SOLVED.

I had an idea last night and tested it out today.

My problem wasn’t the certificates from the elasticsearch.yml. Those were all OK. They were OK as soon as I got the es master to show “started”

The reason why these null cert errors were happening after that point is because I have a whole application cluster set up and some OTHER services try to query the master after it starts up. It was these other services that weren’t modified to handle ssl things yet that seemed to be causing the errors to show up in the master log.

If anyone from elasticsearch/searchguard development is reading this (or heck, literally any software developer), it just goes to show you why log messages are super important. So many times through this whole process I was getting error messages like “alias not found” or “cert not found” but nowhere did it actually tell me where it was looking when the error happened. (keystore? truststore? who’s making the call? )

10

thx for pointing this out.

We will add this to our feature backlog and try to improve the error messages in future versions

···

Am 13.07.2017 um 15:54 schrieb Steve Haertel <stevehaertel@gmail.com>:

SOLVED.

I had an idea last night and tested it out today.

My problem wasn't the certificates from the elasticsearch.yml. Those were all OK. They were OK as soon as I got the es master to show "started"

The reason why these null cert errors were happening after that point is because I have a whole application cluster set up and some OTHER services try to query the master after it starts up. It was these other services that weren't modified to handle ssl things yet that seemed to be causing the errors to show up in the master log.

If anyone from elasticsearch/searchguard development is reading this (or heck, literally any software developer), it just goes to show you why log messages are super important. So many times through this whole process I was getting error messages like "alias not found" or "cert not found" but nowhere did it actually tell me where it was looking when the error happened. (keystore? truststore? who's making the call? )

--
You received this message because you are subscribed to the Google Groups "Search Guard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/3989f4e4-b3a3-4217-82dd-56ff67c25e91%40googlegroups.com\.
For more options, visit https://groups.google.com/d/optout\.