centos search-guard-2-2.2.0.0-alpha2 for es2.2.0 connect 9300 causes exception caught on transport

es version: 2.2.0

sg2 version: 2-2.2.0.0-alpha2

sg-ssl version: search-guard-ssl-2.2.0.6

openssl version: 1.0.2h

elasticeasrch.yml:

network.host: 0.0.0.0

searchguard.authcz.admin_dn:

  • “CN=kirk,OU=client,O=client,l=tEst, C=De”
    searchguard.allow_all_from_loopback: true
    searchguard.ssl.transport.enforce_hostname_verification: false
    searchguard.ssl.transport.resolve_hostname: false
    searchguard.ssl.transport.enable_openssl_if_available: false
    searchguard.ssl.transport.truststore_filepath: truststore.jks
    searchguard.ssl.transport.keystore_filepath: node-0-keystore.jks
    searchguard.ssl.http.enable_openssl_if_available: false
    searchguard.ssl.http.truststore_filepath: truststore.jks
    searchguard.ssl.http.keystore_filepath: node-0-keystore.jks

``

I use static linked jar as mentioned in openssl.md, that is netty-tcnative-openssl-static-1.1.33.Fork16-fg01-linux-x86_64.jar

When es is started as normal, I got the follow exception:

[com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] [Katrina Luisa van Horne] exception caught on transport layer [[id: 0xbef734e4, /0:0:0:0:0:0:0:1:10804 => /0:0:0:0:0:0:0:1:9300]], closing connection
java.lang.RuntimeException: java.security.KeyException
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1429)
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1218)
at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:852)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.security.ProviderException: java.security.KeyException
at sun.security.ec.ECKeyPairGenerator.generateKeyPair(ECKeyPairGenerator.java:147)
at java.security.KeyPairGenerator$Delegate.generateKeyPair(KeyPairGenerator.java:703)
at sun.security.ssl.ECDHCrypt.(ECDHCrypt.java:64)
at sun.security.ssl.ServerHandshaker.setupEphemeralECDHKeys(ServerHandshaker.java:1432)
at sun.security.ssl.ServerHandshaker.trySetCipherSuite(ServerHandshaker.java:1219)
at sun.security.ssl.ServerHandshaker.chooseCipherSuite(ServerHandshaker.java:1023)
at sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:738)
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:221)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:919)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:916)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1369)
at org.jboss.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1392)
at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1255)
… 18 more
Caused by: java.security.KeyException
at sun.security.ec.ECKeyPairGenerator.generateECKeyPair(Native Method)
at sun.security.ec.ECKeyPairGenerator.generateKeyPair(ECKeyPairGenerator.java:128)
… 32 more

``

and after enable_openssl_if_available is changed to true, the follow exception shows:

[com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] [Matt Murdock] exception caught on transport layer [[id: 0x83b7f149, /127.0.0.1:52052 => /127.0.0.1:9300]], closing connection
java.lang.AbstractMethodError
at org.apache.tomcat.jni.SSL.readFromSSL(Native Method)
at io.netty.handler.ssl.OpenSslEngine.readPlaintextData(OpenSslEngine.java:364)
at io.netty.handler.ssl.OpenSslEngine.unwrap(OpenSslEngine.java:697)
at io.netty.handler.ssl.OpenSslEngine.unwrap(OpenSslEngine.java:803)
at io.netty.handler.ssl.OpenSslEngine.unwrap(OpenSslEngine.java:846)
at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1218)
at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:852)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

``

And both “/_searchguard/sslinfo?pretty” api show “Search Guard not initialized (SG11)”

···

As I have configed them successfully in Windows7 before(JDK ssl provider), after I copy the whole WINDOWS ES folder to CentOS and replace the static linked tcnative, then the “/_searchguard/sslinfo?pretty” api successfully responses as follow:

{
principal: null,
peer_certificates: “0”,
ssl_protocol: null,
ssl_cipher: null,
ssl_openssl_available: true,
ssl_openssl_version: 268443791,
ssl_openssl_version_string: “OpenSSL 1.0.2h 3 May 2016”,
ssl_openssl_non_available_cause: “”,
ssl_provider_http: null,
ssl_provider_transport_server: “JDK”,
ssl_provider_transport_client: “JDK”}

``

no matter I use openssl provider or jdk provider, and the Basic Auth works fine. But I still get “exception caught on transport layer” exception when starting es, and can’t update SG config with sgadmin – when sgadmin runs, the same “exception caught on transport layer” exception alerts.

see inline comments

es version: 2.2.0
sg2 version: 2-2.2.0.0-alpha2
sg-ssl version: search-guard-ssl-2.2.0.6
openssl version: 1.0.2h

elasticeasrch.yml:

network.host: 0.0.0.0
searchguard.authcz.admin_dn:
  - "CN=kirk,OU=client,O=client,l=tEst, C=De"
searchguard.allow_all_from_loopback: true
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: false
searchguard.ssl.transport.enable_openssl_if_available: false
searchguard.ssl.transport.truststore_filepath: truststore.jks
searchguard.ssl.transport.keystore_filepath: node-0-keystore.jks
searchguard.ssl.http.enable_openssl_if_available: false
searchguard.ssl.http.truststore_filepath: truststore.jks
searchguard.ssl.http.keystore_filepath: node-0-keystore.jks

I use static linked jar as mentioned in openssl.md, that is netty-tcnative-openssl-static-1.1.33.Fork16-fg01-linux-x86_64.jar

When es is started as normal, I got the follow exception:
[com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] [Katrina Luisa van Horne] exception caught on transport layer [[id: 0xbef734e4, /0:0:0:0:0:0:0:1:10804 => /0:0:0:0:0:0:0:1:9300]], closing connection
java.lang.RuntimeException: java.security.KeyException
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1429)
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1218)
at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:852)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.security.ProviderException: java.security.KeyException
at sun.security.ec.ECKeyPairGenerator.generateKeyPair(ECKeyPairGenerator.java:147)
at java.security.KeyPairGenerator$Delegate.generateKeyPair(KeyPairGenerator.java:703)
at sun.security.ssl.ECDHCrypt.<init>(ECDHCrypt.java:64)
at sun.security.ssl.ServerHandshaker.setupEphemeralECDHKeys(ServerHandshaker.java:1432)
at sun.security.ssl.ServerHandshaker.trySetCipherSuite(ServerHandshaker.java:1219)
at sun.security.ssl.ServerHandshaker.chooseCipherSuite(ServerHandshaker.java:1023)
at sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:738)
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:221)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:919)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:916)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1369)
at org.jboss.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1392)
at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1255)
... 18 more
Caused by: java.security.KeyException
at sun.security.ec.ECKeyPairGenerator.generateECKeyPair(Native Method)
at sun.security.ec.ECKeyPairGenerator.generateKeyPair(ECKeyPairGenerator.java:128)
... 32 more

Do you use Java 7 or OpenJDK? Pls try Orcale Java 8 - if this exception remains pls give us more datils about your JVM Version and operating system.

and after enable_openssl_if_available is changed to true, the follow exception shows:
[com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] [Matt Murdock] exception caught on transport layer [[id: 0x83b7f149, /127.0.0.1:52052 => /127.0.0.1:9300]], closing connection
java.lang.AbstractMethodError
at org.apache.tomcat.jni.SSL.readFromSSL(Native Method)
at io.netty.handler.ssl.OpenSslEngine.readPlaintextData(OpenSslEngine.java:364)
at io.netty.handler.ssl.OpenSslEngine.unwrap(OpenSslEngine.java:697)
at io.netty.handler.ssl.OpenSslEngine.unwrap(OpenSslEngine.java:803)
at io.netty.handler.ssl.OpenSslEngine.unwrap(OpenSslEngine.java:846)
at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1218)
at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:852)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Would it possible for you to upgrade to ES 2.3.3 and SG SSL 2.3.3.13 and SG 2.3.3.1 ?
The static 1.1.33.Fork16 library is not compatible with the SG version you use.

BTW: There is no "searchguard.allow_all_from_loopback" property anymore so you can safley remove this from your configuration.

···

Am 27.06.2016 um 12:47 schrieb Anders Huang <hmh.the.one@gmail.com>:

And both "/_searchguard/sslinfo?pretty" api show "Search Guard not initialized (SG11)"

--------------------------------
As I have configed them successfully in Windows7 before(JDK ssl provider), after I copy the whole WINDOWS ES folder to CentOS and replace the static linked tcnative, then the "/_searchguard/sslinfo?pretty" api successfully responses as follow:
{
principal: null,
peer_certificates: "0",
ssl_protocol: null,
ssl_cipher: null,
ssl_openssl_available: true,
ssl_openssl_version: 268443791,
ssl_openssl_version_string: "OpenSSL 1.0.2h 3 May 2016",
ssl_openssl_non_available_cause: "",
ssl_provider_http: null,
ssl_provider_transport_server: "JDK",
ssl_provider_transport_client: "JDK"}

no matter I use openssl provider or jdk provider, and the Basic Auth works fine. But I still get "exception caught on transport layer" exception when starting es, and can't update SG config with sgadmin -- when sgadmin runs, the same "exception caught on transport layer" exception alerts.

--
You received this message because you are subscribed to the Google Groups "Search Guard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/58ec4240-2c6b-4752-b45a-93c7b782437a%40googlegroups.com\.
For more options, visit https://groups.google.com/d/optout\.

JDK version:

openjdk version “1.8.0_91”

OpenJDK Runtime Environment (build 1.8.0_91-b14)

OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)

OS info:

Linux 2.6.32-504.30.3.el6.x86_64 #1 SMP Wed Jul 15 10:13:09 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

OK, I’ll try the new version.

Thank you for your reply.

ES 2.3.3 and SG SSL 2.3.3.13 and SG 2.3.3.1 does work!