Cannot initialize SearchGuard

valentin · January 25, 2018, 12:23pm

Hi guys,

I have a 7 node ES cluster and I installed SG on it.

For some reason I **cannot initialize the SG plugin. Any idea why? **

I’m using generated (online by SG) certificates and those seems to be OK.

When I try to initialize SG I do it using the following command:

sgadmin.sh -cacert root-ca.pem -cert sgadmin.crt.pem -key sgadmin.key.pem -keypass blabla -nhnv -cd …/sgconfig/ -h 1.2.3.4 -p 9300 -cn MyCluster

``

My SG configuration block present in ES is the following:

####### SEARCH GUARD #######

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.pemcert_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.crtfull.pem
searchguard.ssl.transport.pemkey_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.key.pem
searchguard.ssl.transport.pemkey_password: e4aae4f746361c10e3aa
searchguard.ssl.transport.pemtrustedcas_filepath: certificates/search-guard-certificates/chain-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.crtfull.pem
searchguard.ssl.http.pemkey_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.key.pem
searchguard.ssl.http.pemkey_password: e4aae4f746361c10e3aa
searchguard.ssl.http.pemtrustedcas_filepath: certificates/search-guard-certificates/chain-ca.pem
searchguard.nodes_dn:

‘*’
searchguard.authcz.admin_dn:
CN=sgadmin

######## End Search Guard Configuration ########

``

Search Guard and Elasticsearch version

SG → 5-5.5.2-16, ES → 5.5.2

``

Installed and used enterprise modules, if any

→ none

``

JVM version and operating system version

→ openjdk version “1.8.0_144”

``

Search Guard configuration files

→ more exactly?

``

Elasticsearch log messages on debug level

→

ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
Trace:
ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1544)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1501)
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1385)
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1267)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at java.lang.Thread.run(Thread.java:748)

``

Other installed Elasticsearch or Kibana plugins, if any

→ none

``

jkressin · January 25, 2018, 12:50pm

This usually happens why you try to use a node certificate as an admin certificate. We improved the error messages in SG6 when this happens, in SG5 it’s not so obvious

So can you please check if the certificate you use in the sgadmin call:

sgadmin.crt.pem

Is also a node certificate, means has the OID set?

···

On Thursday, January 25, 2018 at 1:23:53 PM UTC+1, valentin@servergeek.at wrote:

Hi guys,

I have a 7 node ES cluster and I installed SG on it.

For some reason I **cannot initialize the SG plugin. Any idea why? **

I’m using generated (online by SG) certificates and those seems to be OK.

When I try to initialize SG I do it using the following command:

sgadmin.sh -cacert root-ca.pem -cert sgadmin.crt.pem -key sgadmin.key.pem -keypass blabla -nhnv -cd …/sgconfig/ -h 1.2.3.4 -p 9300 -cn MyCluster

``

My SG configuration block present in ES is the following:

####### SEARCH GUARD #######

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.pemcert_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.crtfull.pem
searchguard.ssl.transport.pemkey_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.key.pem
searchguard.ssl.transport.pemkey_password: e4aae4f746361c10e3aa
searchguard.ssl.transport.pemtrustedcas_filepath: certificates/search-guard-certificates/chain-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.crtfull.pem
searchguard.ssl.http.pemkey_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.key.pem
searchguard.ssl.http.pemkey_password: e4aae4f746361c10e3aa
searchguard.ssl.http.pemtrustedcas_filepath: certificates/search-guard-certificates/chain-ca.pem
searchguard.nodes_dn:

‘*’
searchguard.authcz.admin_dn:

CN=sgadmin

######## End Search Guard Configuration ########

``

Search Guard and Elasticsearch version

SG → 5-5.5.2-16, ES → 5.5.2

``

Installed and used enterprise modules, if any

→ none

``

JVM version and operating system version

→ openjdk version “1.8.0_144”

``

Search Guard configuration files

→ more exactly?

``

Elasticsearch log messages on debug level

→

ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
Trace:
ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1544)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1501)
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1385)
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1267)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at java.lang.Thread.run(Thread.java:748)

``

Other installed Elasticsearch or Kibana plugins, if any

→ none

``

valentin · January 25, 2018, 1:07pm

Hi Jochen,

The certificate has the following properties:

Certificate:

Data:

Version: 3 (0x2)

Serial Number: 1 (0x1)

Signature Algorithm: sha256WithRSAEncryption

Issuer: O=SaS, OU=SaS Signing CA, CN=SaS Signing CA

Validity

Not Before: Jan 23 07:47:48 2018 GMT

Not After : Jan 23 07:47:48 2020 GMT

Subject Public Key Info:

Public Key Algorithm: rsaEncryption

Public-Key: (2048 bit)

…

X509v3 extensions:

X509v3 Key Usage: critical

Digital Signature, Key Encipherment

X509v3 Basic Constraints:

CA:FALSE

X509v3 Extended Key Usage:

TLS Web Server Authentication, TLS Web Client Authentication

X509v3 Subject Key Identifier:

…

X509v3 Authority Key Identifier:

keyid:…

…

···

On Thu, Jan 25, 2018 at 1:50 PM, Jochen Kressin jkressin@floragunn.com wrote:

This usually happens why you try to use a node certificate as an admin certificate. We improved the error messages in SG6 when this happens, in SG5 it’s not so obvious

So can you please check if the certificate you use in the sgadmin call:

sgadmin.crt.pem

Is also a node certificate, means has the OID set?

On Thursday, January 25, 2018 at 1:23:53 PM UTC+1, valentin@servergeek.at wrote:

Hi guys,

I have a 7 node ES cluster and I installed SG on it.

For some reason I **cannot initialize the SG plugin. Any idea why? **

I’m using generated (online by SG) certificates and those seems to be OK.

When I try to initialize SG I do it using the following command:

sgadmin.sh -cacert root-ca.pem -cert sgadmin.crt.pem -key sgadmin.key.pem -keypass blabla -nhnv -cd …/sgconfig/ -h 1.2.3.4 -p 9300 -cn MyCluster

``

My SG configuration block present in ES is the following:

####### SEARCH GUARD #######

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.pemcert_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.crtfull.pem
searchguard.ssl.transport.pemkey_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.key.pem
searchguard.ssl.transport.pemkey_password: e4aae4f746361c10e3aa
searchguard.ssl.transport.pemtrustedcas_filepath: certificates/search-guard-certificates/chain-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.crtfull.pem
searchguard.ssl.http.pemkey_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.key.pem
searchguard.ssl.http.pemkey_password: e4aae4f746361c10e3aa
searchguard.ssl.http.pemtrustedcas_filepath: certificates/search-guard-certificates/chain-ca.pem
searchguard.nodes_dn:

‘*’
searchguard.authcz.admin_dn:

CN=sgadmin

######## End Search Guard Configuration ########

``

Search Guard and Elasticsearch version

SG → 5-5.5.2-16, ES → 5.5.2

``

Installed and used enterprise modules, if any

→ none

``

JVM version and operating system version

→ openjdk version “1.8.0_144”

``

Search Guard configuration files

→ more exactly?

``

Elasticsearch log messages on debug level

→

ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
Trace:
ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1544)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1501)
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1385)
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1267)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at java.lang.Thread.run(Thread.java:748)

``

Other installed Elasticsearch or Kibana plugins, if any

→ none

``

–

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/ruyB5QVFAds/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/3a8548ea-14d4-42f4-b0be-d6a2e2389c62%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

jkressin · January 25, 2018, 1:41pm

Can you please also post the SAN section of the key? The OID field in question is in that section. Thx!

···

On Thursday, January 25, 2018 at 2:07:16 PM UTC+1, Valentin Fischer wrote:

Hi Jochen,

The certificate has the following properties:

Certificate:

Data:

Version: 3 (0x2)

Serial Number: 1 (0x1)

Signature Algorithm: sha256WithRSAEncryption

Issuer: O=SaS, OU=SaS Signing CA, CN=SaS Signing CA

Validity

Not Before: Jan 23 07:47:48 2018 GMT

Not After : Jan 23 07:47:48 2020 GMT

Subject: CN=sgadmin

Subject Public Key Info:

Public Key Algorithm: rsaEncryption

Public-Key: (2048 bit)

…

X509v3 extensions:

X509v3 Key Usage: critical

Digital Signature, Key Encipherment

X509v3 Basic Constraints:

CA:FALSE

X509v3 Extended Key Usage:

TLS Web Server Authentication, TLS Web Client Authentication

X509v3 Subject Key Identifier:

…

X509v3 Authority Key Identifier:

keyid:…

…

On Thu, Jan 25, 2018 at 1:50 PM, Jochen Kressin jkressin@floragunn.com wrote:

This usually happens why you try to use a node certificate as an admin certificate. We improved the error messages in SG6 when this happens, in SG5 it’s not so obvious

So can you please check if the certificate you use in the sgadmin call:

sgadmin.crt.pem

Is also a node certificate, means has the OID set?

On Thursday, January 25, 2018 at 1:23:53 PM UTC+1, valentin@servergeek.at wrote:

Hi guys,

I have a 7 node ES cluster and I installed SG on it.

For some reason I **cannot initialize the SG plugin. Any idea why? **

I’m using generated (online by SG) certificates and those seems to be OK.

When I try to initialize SG I do it using the following command:

sgadmin.sh -cacert root-ca.pem -cert sgadmin.crt.pem -key sgadmin.key.pem -keypass blabla -nhnv -cd …/sgconfig/ -h 1.2.3.4 -p 9300 -cn MyCluster

``

My SG configuration block present in ES is the following:

####### SEARCH GUARD #######

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.pemcert_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.crtfull.pem
searchguard.ssl.transport.pemkey_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.key.pem
searchguard.ssl.transport.pemkey_password: e4aae4f746361c10e3aa
searchguard.ssl.transport.pemtrustedcas_filepath: certificates/search-guard-certificates/chain-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.crtfull.pem
searchguard.ssl.http.pemkey_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.key.pem
searchguard.ssl.http.pemkey_password: e4aae4f746361c10e3aa
searchguard.ssl.http.pemtrustedcas_filepath: certificates/search-guard-certificates/chain-ca.pem
searchguard.nodes_dn:

‘*’
searchguard.authcz.admin_dn:

CN=sgadmin

######## End Search Guard Configuration ########

``

Search Guard and Elasticsearch version

SG → 5-5.5.2-16, ES → 5.5.2

``

Installed and used enterprise modules, if any

→ none

``

JVM version and operating system version

→ openjdk version “1.8.0_144”

``

Search Guard configuration files

→ more exactly?

``

Elasticsearch log messages on debug level

→

ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
Trace:
ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1544)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1501)
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1385)
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1267)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at java.lang.Thread.run(Thread.java:748)

``

Other installed Elasticsearch or Kibana plugins, if any

→ none

``

–

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/ruyB5QVFAds/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/3a8548ea-14d4-42f4-b0be-d6a2e2389c62%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

valentin · January 25, 2018, 2:05pm

Hmmm… no idea how to get that. How do I get/see it ?

···

On Thu, Jan 25, 2018 at 2:41 PM, Jochen Kressin jkressin@floragunn.com wrote:

Can you please also post the SAN section of the key? The OID field in question is in that section. Thx!

On Thursday, January 25, 2018 at 2:07:16 PM UTC+1, Valentin Fischer wrote:

Hi Jochen,

The certificate has the following properties:

Certificate:

Data:

Version: 3 (0x2)

Serial Number: 1 (0x1)

Signature Algorithm: sha256WithRSAEncryption

Issuer: O=SaS, OU=SaS Signing CA, CN=SaS Signing CA

Validity

Not Before: Jan 23 07:47:48 2018 GMT

Not After : Jan 23 07:47:48 2020 GMT

Subject: CN=sgadmin

Subject Public Key Info:

Public Key Algorithm: rsaEncryption

Public-Key: (2048 bit)

…

X509v3 extensions:

X509v3 Key Usage: critical

Digital Signature, Key Encipherment

X509v3 Basic Constraints:

CA:FALSE

X509v3 Extended Key Usage:

TLS Web Server Authentication, TLS Web Client Authentication

X509v3 Subject Key Identifier:

…

X509v3 Authority Key Identifier:

keyid:…

…

On Thu, Jan 25, 2018 at 1:50 PM, Jochen Kressin jkressin@floragunn.com wrote:

This usually happens why you try to use a node certificate as an admin certificate. We improved the error messages in SG6 when this happens, in SG5 it’s not so obvious

So can you please check if the certificate you use in the sgadmin call:

sgadmin.crt.pem

Is also a node certificate, means has the OID set?

On Thursday, January 25, 2018 at 1:23:53 PM UTC+1, valentin@servergeek.at wrote:

Hi guys,

I have a 7 node ES cluster and I installed SG on it.

For some reason I **cannot initialize the SG plugin. Any idea why? **

I’m using generated (online by SG) certificates and those seems to be OK.

When I try to initialize SG I do it using the following command:

sgadmin.sh -cacert root-ca.pem -cert sgadmin.crt.pem -key sgadmin.key.pem -keypass blabla -nhnv -cd …/sgconfig/ -h 1.2.3.4 -p 9300 -cn MyCluster

``

My SG configuration block present in ES is the following:

####### SEARCH GUARD #######

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.pemcert_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.crtfull.pem
searchguard.ssl.transport.pemkey_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.key.pem
searchguard.ssl.transport.pemkey_password: e4aae4f746361c10e3aa
searchguard.ssl.transport.pemtrustedcas_filepath: certificates/search-guard-certificates/chain-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.crtfull.pem
searchguard.ssl.http.pemkey_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.key.pem
searchguard.ssl.http.pemkey_password: e4aae4f746361c10e3aa
searchguard.ssl.http.pemtrustedcas_filepath: certificates/search-guard-certificates/chain-ca.pem
searchguard.nodes_dn:

‘*’
searchguard.authcz.admin_dn:

CN=sgadmin

######## End Search Guard Configuration ########

``

Search Guard and Elasticsearch version

SG → 5-5.5.2-16, ES → 5.5.2

``

Installed and used enterprise modules, if any

→ none

``

JVM version and operating system version

→ openjdk version “1.8.0_144”

``

Search Guard configuration files

→ more exactly?

``

Elasticsearch log messages on debug level

→

ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
Trace:
ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1544)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1501)
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1385)
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1267)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at java.lang.Thread.run(Thread.java:748)

``

Other installed Elasticsearch or Kibana plugins, if any

→ none

``

–

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/ruyB5QVFAds/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/3a8548ea-14d4-42f4-b0be-d6a2e2389c62%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

–

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/ruyB5QVFAds/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/90c70109-9415-44d9-8bb3-c95234166c07%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

jkressin · January 25, 2018, 2:32pm

It’s in the X509v3 extensions part, I think you just cutted it off. Should be something like:

   X509v3 extensions:

        X509v3 Key Usage: critical

            Digital Signature, Key Encipherment

        X509v3 Basic Constraints:

            CA:FALSE

        X509v3 Extended Key Usage:

            TLS Web Server Authentication, TLS Web Client Authentication

        X509v3 Subject Key Identifier:

            7D:A1:DE:12:4D:AE:D6:79:9D:CF:A8:57:7E:30:08:8B:BA:8E:59:D8

        X509v3 Authority Key Identifier:

            keyid:35:03:23:13:30:30:21:1F:8F:BD:F3:DF:5E:C1:B0:A9:20:88:2C:B0

        **X509v3 Subject Alternative Name:**

** DNS:node-0.example.com, DNS:localhost, IP Address:127.0.0.1, Registered ID:1.2.3.4.5.5**

``

If you see the “Registered ID” part here, it means this is a node certificate.

···

On Thursday, January 25, 2018 at 3:05:33 PM UTC+1, Valentin Fischer wrote:

Hmmm… no idea how to get that. How do I get/see it ?

On Thu, Jan 25, 2018 at 2:41 PM, Jochen Kressin jkressin@floragunn.com wrote:

Can you please also post the SAN section of the key? The OID field in question is in that section. Thx!

On Thursday, January 25, 2018 at 2:07:16 PM UTC+1, Valentin Fischer wrote:

Hi Jochen,

The certificate has the following properties:

Certificate:

Data:

Version: 3 (0x2)

Serial Number: 1 (0x1)

Signature Algorithm: sha256WithRSAEncryption

Issuer: O=SaS, OU=SaS Signing CA, CN=SaS Signing CA

Validity

Not Before: Jan 23 07:47:48 2018 GMT

Not After : Jan 23 07:47:48 2020 GMT

Subject: CN=sgadmin

Subject Public Key Info:

Public Key Algorithm: rsaEncryption

Public-Key: (2048 bit)

…

X509v3 extensions:

X509v3 Key Usage: critical

Digital Signature, Key Encipherment

X509v3 Basic Constraints:

CA:FALSE

X509v3 Extended Key Usage:

TLS Web Server Authentication, TLS Web Client Authentication

X509v3 Subject Key Identifier:

…

X509v3 Authority Key Identifier:

keyid:…

…

On Thu, Jan 25, 2018 at 1:50 PM, Jochen Kressin jkressin@floragunn.com wrote:

This usually happens why you try to use a node certificate as an admin certificate. We improved the error messages in SG6 when this happens, in SG5 it’s not so obvious

So can you please check if the certificate you use in the sgadmin call:

sgadmin.crt.pem

Is also a node certificate, means has the OID set?

On Thursday, January 25, 2018 at 1:23:53 PM UTC+1, valentin@servergeek.at wrote:

Hi guys,

I have a 7 node ES cluster and I installed SG on it.

For some reason I **cannot initialize the SG plugin. Any idea why? **

I’m using generated (online by SG) certificates and those seems to be OK.

When I try to initialize SG I do it using the following command:

sgadmin.sh -cacert root-ca.pem -cert sgadmin.crt.pem -key sgadmin.key.pem -keypass blabla -nhnv -cd …/sgconfig/ -h 1.2.3.4 -p 9300 -cn MyCluster

``

My SG configuration block present in ES is the following:

####### SEARCH GUARD #######

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.pemcert_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.crtfull.pem
searchguard.ssl.transport.pemkey_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.key.pem
searchguard.ssl.transport.pemkey_password: e4aae4f746361c10e3aa
searchguard.ssl.transport.pemtrustedcas_filepath: certificates/search-guard-certificates/chain-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.crtfull.pem
searchguard.ssl.http.pemkey_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.key.pem
searchguard.ssl.http.pemkey_password: e4aae4f746361c10e3aa
searchguard.ssl.http.pemtrustedcas_filepath: certificates/search-guard-certificates/chain-ca.pem
searchguard.nodes_dn:

‘*’
searchguard.authcz.admin_dn:

CN=sgadmin

######## End Search Guard Configuration ########

``

Search Guard and Elasticsearch version

SG → 5-5.5.2-16, ES → 5.5.2

``

Installed and used enterprise modules, if any

→ none

``

JVM version and operating system version

→ openjdk version “1.8.0_144”

``

Search Guard configuration files

→ more exactly?

``

Elasticsearch log messages on debug level

→

ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
Trace:
ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1544)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1501)
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1385)
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1267)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at java.lang.Thread.run(Thread.java:748)

``

Other installed Elasticsearch or Kibana plugins, if any

→ none

``

–

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/ruyB5QVFAds/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/3a8548ea-14d4-42f4-b0be-d6a2e2389c62%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

–

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/ruyB5QVFAds/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/90c70109-9415-44d9-8bb3-c95234166c07%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

valentin · January 26, 2018, 5:21am

Hi,

There is no SAN part in it.

X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Basic Constraints:
CA:FALSE
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Subject Key Identifier:
72:E6:02:00:BF:2A:F0:E4:BB:18:EB:E7:5E:DC:ED:9F:A8:FD:BC:67
X509v3 Authority Key Identifier:
keyid:E9:4A:18:64:74:6D:C6:EF:46:FA:C1:BB:53:62:98:B2:C8:6C:75:4A

``

···

On Thursday, January 25, 2018 at 3:32:45 PM UTC+1, Jochen Kressin wrote:

It’s in the X509v3 extensions part, I think you just cutted it off. Should be something like:

   X509v3 extensions:

        X509v3 Key Usage: critical

            Digital Signature, Key Encipherment

        X509v3 Basic Constraints:

            CA:FALSE

        X509v3 Extended Key Usage:

            TLS Web Server Authentication, TLS Web Client Authentication

        X509v3 Subject Key Identifier:

            7D:A1:DE:12:4D:AE:D6:79:9D:CF:A8:57:7E:30:08:8B:BA:8E:59:D8

        X509v3 Authority Key Identifier:

            keyid:35:03:23:13:30:30:21:1F:8F:BD:F3:DF:5E:C1:B0:A9:20:88:2C:B0

        **X509v3 Subject Alternative Name:**

** DNS:node-0.example.com, DNS:localhost, IP Address:127.0.0.1, Registered ID:1.2.3.4.5.5**

``

If you see the “Registered ID” part here, it means this is a node certificate.

On Thursday, January 25, 2018 at 3:05:33 PM UTC+1, Valentin Fischer wrote:

Hmmm… no idea how to get that. How do I get/see it ?

On Thu, Jan 25, 2018 at 2:41 PM, Jochen Kressin jkre...@floragunn.com wrote:

Can you please also post the SAN section of the key? The OID field in question is in that section. Thx!

On Thursday, January 25, 2018 at 2:07:16 PM UTC+1, Valentin Fischer wrote:

Hi Jochen,

The certificate has the following properties:

Certificate:

Data:

Version: 3 (0x2)

Serial Number: 1 (0x1)

Signature Algorithm: sha256WithRSAEncryption

Issuer: O=SaS, OU=SaS Signing CA, CN=SaS Signing CA

Validity

Not Before: Jan 23 07:47:48 2018 GMT

Not After : Jan 23 07:47:48 2020 GMT

Subject: CN=sgadmin

Subject Public Key Info:

Public Key Algorithm: rsaEncryption

Public-Key: (2048 bit)

…

X509v3 extensions:

X509v3 Key Usage: critical

Digital Signature, Key Encipherment

X509v3 Basic Constraints:

CA:FALSE

X509v3 Extended Key Usage:

TLS Web Server Authentication, TLS Web Client Authentication

X509v3 Subject Key Identifier:

…

X509v3 Authority Key Identifier:

keyid:…

…

On Thu, Jan 25, 2018 at 1:50 PM, Jochen Kressin jkre...@floragunn.com wrote:

This usually happens why you try to use a node certificate as an admin certificate. We improved the error messages in SG6 when this happens, in SG5 it’s not so obvious

So can you please check if the certificate you use in the sgadmin call:

sgadmin.crt.pem

Is also a node certificate, means has the OID set?

On Thursday, January 25, 2018 at 1:23:53 PM UTC+1, vale…@servergeek.at wrote:

Hi guys,

I have a 7 node ES cluster and I installed SG on it.

For some reason I **cannot initialize the SG plugin. Any idea why? **

I’m using generated (online by SG) certificates and those seems to be OK.

When I try to initialize SG I do it using the following command:

sgadmin.sh -cacert root-ca.pem -cert sgadmin.crt.pem -key sgadmin.key.pem -keypass blabla -nhnv -cd …/sgconfig/ -h 1.2.3.4 -p 9300 -cn MyCluster

``

My SG configuration block present in ES is the following:

####### SEARCH GUARD #######

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.pemcert_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.crtfull.pem
searchguard.ssl.transport.pemkey_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.key.pem
searchguard.ssl.transport.pemkey_password: e4aae4f746361c10e3aa
searchguard.ssl.transport.pemtrustedcas_filepath: certificates/search-guard-certificates/chain-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.crtfull.pem
searchguard.ssl.http.pemkey_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.key.pem
searchguard.ssl.http.pemkey_password: e4aae4f746361c10e3aa
searchguard.ssl.http.pemtrustedcas_filepath: certificates/search-guard-certificates/chain-ca.pem
searchguard.nodes_dn:

‘*’
searchguard.authcz.admin_dn:

CN=sgadmin

######## End Search Guard Configuration ########

``

Search Guard and Elasticsearch version

SG → 5-5.5.2-16, ES → 5.5.2

``

Installed and used enterprise modules, if any

→ none

``

JVM version and operating system version

→ openjdk version “1.8.0_144”

``

Search Guard configuration files

→ more exactly?

``

Elasticsearch log messages on debug level

→

ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
Trace:
ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1544)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1501)
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1385)
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1267)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at java.lang.Thread.run(Thread.java:748)

``

Other installed Elasticsearch or Kibana plugins, if any

→ none

``

–

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/ruyB5QVFAds/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/3a8548ea-14d4-42f4-b0be-d6a2e2389c62%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

–

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/ruyB5QVFAds/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/90c70109-9415-44d9-8bb3-c95234166c07%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

valentin · January 29, 2018, 5:50am

Any updates on this ?

I want to have a running cluster and for some reason I cannot initialize it.

···

On Friday, January 26, 2018 at 6:21:26 AM UTC+1, vale…@servergeek.at wrote:

Hi,

There is no SAN part in it.

X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Basic Constraints:
CA:FALSE
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Subject Key Identifier:
72:E6:02:00:BF:2A:F0:E4:BB:18:EB:E7:5E:DC:ED:9F:A8:FD:BC:67
X509v3 Authority Key Identifier:
keyid:E9:4A:18:64:74:6D:C6:EF:46:FA:C1:BB:53:62:98:B2:C8:6C:75:4A

``

On Thursday, January 25, 2018 at 3:32:45 PM UTC+1, Jochen Kressin wrote:

It’s in the X509v3 extensions part, I think you just cutted it off. Should be something like:

   X509v3 extensions:

        X509v3 Key Usage: critical

            Digital Signature, Key Encipherment

        X509v3 Basic Constraints:

            CA:FALSE

        X509v3 Extended Key Usage:

            TLS Web Server Authentication, TLS Web Client Authentication

        X509v3 Subject Key Identifier:

            7D:A1:DE:12:4D:AE:D6:79:9D:CF:A8:57:7E:30:08:8B:BA:8E:59:D8

        X509v3 Authority Key Identifier:

            keyid:35:03:23:13:30:30:21:1F:8F:BD:F3:DF:5E:C1:B0:A9:20:88:2C:B0

        **X509v3 Subject Alternative Name:**

** DNS:node-0.example.com, DNS:localhost, IP Address:127.0.0.1, Registered ID:1.2.3.4.5.5**

``

If you see the “Registered ID” part here, it means this is a node certificate.

On Thursday, January 25, 2018 at 3:05:33 PM UTC+1, Valentin Fischer wrote:

Hmmm… no idea how to get that. How do I get/see it ?

On Thu, Jan 25, 2018 at 2:41 PM, Jochen Kressin jkre...@floragunn.com wrote:

Can you please also post the SAN section of the key? The OID field in question is in that section. Thx!

On Thursday, January 25, 2018 at 2:07:16 PM UTC+1, Valentin Fischer wrote:

Hi Jochen,

The certificate has the following properties:

Certificate:

Data:

Version: 3 (0x2)

Serial Number: 1 (0x1)

Signature Algorithm: sha256WithRSAEncryption

Issuer: O=SaS, OU=SaS Signing CA, CN=SaS Signing CA

Validity

Not Before: Jan 23 07:47:48 2018 GMT

Not After : Jan 23 07:47:48 2020 GMT

Subject: CN=sgadmin

Subject Public Key Info:

Public Key Algorithm: rsaEncryption

Public-Key: (2048 bit)

…

X509v3 extensions:

X509v3 Key Usage: critical

Digital Signature, Key Encipherment

X509v3 Basic Constraints:

CA:FALSE

X509v3 Extended Key Usage:

TLS Web Server Authentication, TLS Web Client Authentication

X509v3 Subject Key Identifier:

…

X509v3 Authority Key Identifier:

keyid:…

…

On Thu, Jan 25, 2018 at 1:50 PM, Jochen Kressin jkre...@floragunn.com wrote:

This usually happens why you try to use a node certificate as an admin certificate. We improved the error messages in SG6 when this happens, in SG5 it’s not so obvious

So can you please check if the certificate you use in the sgadmin call:

sgadmin.crt.pem

Is also a node certificate, means has the OID set?

On Thursday, January 25, 2018 at 1:23:53 PM UTC+1, vale…@servergeek.at wrote:

Hi guys,

I have a 7 node ES cluster and I installed SG on it.

For some reason I **cannot initialize the SG plugin. Any idea why? **

I’m using generated (online by SG) certificates and those seems to be OK.

When I try to initialize SG I do it using the following command:

sgadmin.sh -cacert root-ca.pem -cert sgadmin.crt.pem -key sgadmin.key.pem -keypass blabla -nhnv -cd …/sgconfig/ -h 1.2.3.4 -p 9300 -cn MyCluster

``

My SG configuration block present in ES is the following:

####### SEARCH GUARD #######

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.pemcert_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.crtfull.pem
searchguard.ssl.transport.pemkey_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.key.pem
searchguard.ssl.transport.pemkey_password: e4aae4f746361c10e3aa
searchguard.ssl.transport.pemtrustedcas_filepath: certificates/search-guard-certificates/chain-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.crtfull.pem
searchguard.ssl.http.pemkey_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.key.pem
searchguard.ssl.http.pemkey_password: e4aae4f746361c10e3aa
searchguard.ssl.http.pemtrustedcas_filepath: certificates/search-guard-certificates/chain-ca.pem
searchguard.nodes_dn:

‘*’
searchguard.authcz.admin_dn:

CN=sgadmin

######## End Search Guard Configuration ########

``

Search Guard and Elasticsearch version

SG → 5-5.5.2-16, ES → 5.5.2

``

Installed and used enterprise modules, if any

→ none

``

JVM version and operating system version

→ openjdk version “1.8.0_144”

``

Search Guard configuration files

→ more exactly?

``

Elasticsearch log messages on debug level

→

ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
Trace:
ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1544)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1501)
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1385)
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1267)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at java.lang.Thread.run(Thread.java:748)

``

Other installed Elasticsearch or Kibana plugins, if any

→ none

``

–

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/ruyB5QVFAds/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/3a8548ea-14d4-42f4-b0be-d6a2e2389c62%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

–

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/ruyB5QVFAds/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/90c70109-9415-44d9-8bb3-c95234166c07%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

jkressin · January 29, 2018, 11:49am

Ah, sorry, it’s obvious but somehow I totally overlooked it:

It’s this part of the config that causes problems:

searchguard.nodes_dn:

‘*’
If you just use a wildcard here it means that Search Guard will treat every certificate, including the admin one, as node certificate. And that’s why sgadmin chokes with the error message you posted. You need to make sure that the pattern you are using here does not match the DN of the admin certificate. Depending on the DNs of your node certificates that might be something like:

CN=*.it.internal

···

On Monday, January 29, 2018 at 6:50:14 AM UTC+1, valentin@servergeek.at wrote:

Any updates on this ?

I want to have a running cluster and for some reason I cannot initialize it.

On Friday, January 26, 2018 at 6:21:26 AM UTC+1, vale…@servergeek.at wrote:

Hi,

There is no SAN part in it.

X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Basic Constraints:
CA:FALSE
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Subject Key Identifier:
72:E6:02:00:BF:2A:F0:E4:BB:18:EB:E7:5E:DC:ED:9F:A8:FD:BC:67
X509v3 Authority Key Identifier:
keyid:E9:4A:18:64:74:6D:C6:EF:46:FA:C1:BB:53:62:98:B2:C8:6C:75:4A

``

On Thursday, January 25, 2018 at 3:32:45 PM UTC+1, Jochen Kressin wrote:

It’s in the X509v3 extensions part, I think you just cutted it off. Should be something like:

   X509v3 extensions:

        X509v3 Key Usage: critical

            Digital Signature, Key Encipherment

        X509v3 Basic Constraints:

            CA:FALSE

        X509v3 Extended Key Usage:

            TLS Web Server Authentication, TLS Web Client Authentication

        X509v3 Subject Key Identifier:

            7D:A1:DE:12:4D:AE:D6:79:9D:CF:A8:57:7E:30:08:8B:BA:8E:59:D8

        X509v3 Authority Key Identifier:

            keyid:35:03:23:13:30:30:21:1F:8F:BD:F3:DF:5E:C1:B0:A9:20:88:2C:B0

        **X509v3 Subject Alternative Name:**

** DNS:node-0.example.com, DNS:localhost, IP Address:127.0.0.1, Registered ID:1.2.3.4.5.5**

``

If you see the “Registered ID” part here, it means this is a node certificate.

On Thursday, January 25, 2018 at 3:05:33 PM UTC+1, Valentin Fischer wrote:

Hmmm… no idea how to get that. How do I get/see it ?

On Thu, Jan 25, 2018 at 2:41 PM, Jochen Kressin jkre...@floragunn.com wrote:

Can you please also post the SAN section of the key? The OID field in question is in that section. Thx!

On Thursday, January 25, 2018 at 2:07:16 PM UTC+1, Valentin Fischer wrote:

Hi Jochen,

The certificate has the following properties:

Certificate:

Data:

Version: 3 (0x2)

Serial Number: 1 (0x1)

Signature Algorithm: sha256WithRSAEncryption

Issuer: O=SaS, OU=SaS Signing CA, CN=SaS Signing CA

Validity

Not Before: Jan 23 07:47:48 2018 GMT

Not After : Jan 23 07:47:48 2020 GMT

Subject: CN=sgadmin

Subject Public Key Info:

Public Key Algorithm: rsaEncryption

Public-Key: (2048 bit)

…

X509v3 extensions:

X509v3 Key Usage: critical

Digital Signature, Key Encipherment

X509v3 Basic Constraints:

CA:FALSE

X509v3 Extended Key Usage:

TLS Web Server Authentication, TLS Web Client Authentication

X509v3 Subject Key Identifier:

…

X509v3 Authority Key Identifier:

keyid:…

…

On Thu, Jan 25, 2018 at 1:50 PM, Jochen Kressin jkre...@floragunn.com wrote:

This usually happens why you try to use a node certificate as an admin certificate. We improved the error messages in SG6 when this happens, in SG5 it’s not so obvious

So can you please check if the certificate you use in the sgadmin call:

sgadmin.crt.pem

Is also a node certificate, means has the OID set?

On Thursday, January 25, 2018 at 1:23:53 PM UTC+1, vale…@servergeek.at wrote:

Hi guys,

I have a 7 node ES cluster and I installed SG on it.

For some reason I **cannot initialize the SG plugin. Any idea why? **

I’m using generated (online by SG) certificates and those seems to be OK.

When I try to initialize SG I do it using the following command:

sgadmin.sh -cacert root-ca.pem -cert sgadmin.crt.pem -key sgadmin.key.pem -keypass blabla -nhnv -cd …/sgconfig/ -h 1.2.3.4 -p 9300 -cn MyCluster

``

My SG configuration block present in ES is the following:

####### SEARCH GUARD #######

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.pemcert_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.crtfull.pem
searchguard.ssl.transport.pemkey_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.key.pem
searchguard.ssl.transport.pemkey_password: e4aae4f746361c10e3aa
searchguard.ssl.transport.pemtrustedcas_filepath: certificates/search-guard-certificates/chain-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.crtfull.pem
searchguard.ssl.http.pemkey_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.key.pem
searchguard.ssl.http.pemkey_password: e4aae4f746361c10e3aa
searchguard.ssl.http.pemtrustedcas_filepath: certificates/search-guard-certificates/chain-ca.pem
searchguard.nodes_dn:

‘*’
searchguard.authcz.admin_dn:

CN=sgadmin

######## End Search Guard Configuration ########

``

Search Guard and Elasticsearch version

SG → 5-5.5.2-16, ES → 5.5.2

``

Installed and used enterprise modules, if any

→ none

``

JVM version and operating system version

→ openjdk version “1.8.0_144”

``

Search Guard configuration files

→ more exactly?

``

Elasticsearch log messages on debug level

→

ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
Trace:
ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1544)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1501)
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1385)
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1267)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at java.lang.Thread.run(Thread.java:748)

``

Other installed Elasticsearch or Kibana plugins, if any

→ none

``

–

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/ruyB5QVFAds/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/3a8548ea-14d4-42f4-b0be-d6a2e2389c62%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

–

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/ruyB5QVFAds/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/90c70109-9415-44d9-8bb3-c95234166c07%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

valentin · January 30, 2018, 9:09am

Hi Jochen,

Thank you for the info! I have applied the change and also switched to “official” certificates and I have one BIG issue…

[2018-01-30T10:05:13,481][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [ES_MASTER1] SSL Problem error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate
javax.net.ssl.SSLHandshakeException: error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:955) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:914) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:978) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1021) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:205) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1156) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.11.Final.jar:4.1.11.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]

``

Any ideas why is it throwing this error? The certificate and key are OK

(openssl x509 -noout -modulus -in node.pem | openssl md5 ; openssl rsa -noout -modulus -in node.key | openssl md5) | uniq
(stdin)= a5ab322f1cd213600e8ac367a471e06f

``

This is starting to drive me crazy…

Another question would be if the Registered ID:1.2.3.4.5.5 is really needed in the SAN.

Thank you!!!

···

On Monday, January 29, 2018 at 12:49:58 PM UTC+1, Jochen Kressin wrote:

Ah, sorry, it’s obvious but somehow I totally overlooked it:

It’s this part of the config that causes problems:

searchguard.nodes_dn:

‘*’
If you just use a wildcard here it means that Search Guard will treat every certificate, including the admin one, as node certificate. And that’s why sgadmin chokes with the error message you posted. You need to make sure that the pattern you are using here does not match the DN of the admin certificate. Depending on the DNs of your node certificates that might be something like:

CN=*.it.internal

On Monday, January 29, 2018 at 6:50:14 AM UTC+1, vale…@servergeek.at wrote:

Any updates on this ?

I want to have a running cluster and for some reason I cannot initialize it.

On Friday, January 26, 2018 at 6:21:26 AM UTC+1, vale…@servergeek.at wrote:

Hi,

There is no SAN part in it.

X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Basic Constraints:
CA:FALSE
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Subject Key Identifier:
72:E6:02:00:BF:2A:F0:E4:BB:18:EB:E7:5E:DC:ED:9F:A8:FD:BC:67
X509v3 Authority Key Identifier:
keyid:E9:4A:18:64:74:6D:C6:EF:46:FA:C1:BB:53:62:98:B2:C8:6C:75:4A

``

On Thursday, January 25, 2018 at 3:32:45 PM UTC+1, Jochen Kressin wrote:

It’s in the X509v3 extensions part, I think you just cutted it off. Should be something like:

   X509v3 extensions:

        X509v3 Key Usage: critical

            Digital Signature, Key Encipherment

        X509v3 Basic Constraints:

            CA:FALSE

        X509v3 Extended Key Usage:

            TLS Web Server Authentication, TLS Web Client Authentication

        X509v3 Subject Key Identifier:

            7D:A1:DE:12:4D:AE:D6:79:9D:CF:A8:57:7E:30:08:8B:BA:8E:59:D8

        X509v3 Authority Key Identifier:

            keyid:35:03:23:13:30:30:21:1F:8F:BD:F3:DF:5E:C1:B0:A9:20:88:2C:B0

        **X509v3 Subject Alternative Name:**

** DNS:node-0.example.com, DNS:localhost, IP Address:127.0.0.1, Registered ID:1.2.3.4.5.5**

``

If you see the “Registered ID” part here, it means this is a node certificate.

On Thursday, January 25, 2018 at 3:05:33 PM UTC+1, Valentin Fischer wrote:

Hmmm… no idea how to get that. How do I get/see it ?

On Thu, Jan 25, 2018 at 2:41 PM, Jochen Kressin jkre...@floragunn.com wrote:

Can you please also post the SAN section of the key? The OID field in question is in that section. Thx!

On Thursday, January 25, 2018 at 2:07:16 PM UTC+1, Valentin Fischer wrote:

Hi Jochen,

The certificate has the following properties:

Certificate:

Data:

Version: 3 (0x2)

Serial Number: 1 (0x1)

Signature Algorithm: sha256WithRSAEncryption

Issuer: O=SaS, OU=SaS Signing CA, CN=SaS Signing CA

Validity

Not Before: Jan 23 07:47:48 2018 GMT

Not After : Jan 23 07:47:48 2020 GMT

Subject: CN=sgadmin

Subject Public Key Info:

Public Key Algorithm: rsaEncryption

Public-Key: (2048 bit)

…

X509v3 extensions:

X509v3 Key Usage: critical

Digital Signature, Key Encipherment

X509v3 Basic Constraints:

CA:FALSE

X509v3 Extended Key Usage:

TLS Web Server Authentication, TLS Web Client Authentication

X509v3 Subject Key Identifier:

…

X509v3 Authority Key Identifier:

keyid:…

…

On Thu, Jan 25, 2018 at 1:50 PM, Jochen Kressin jkre...@floragunn.com wrote:

This usually happens why you try to use a node certificate as an admin certificate. We improved the error messages in SG6 when this happens, in SG5 it’s not so obvious

So can you please check if the certificate you use in the sgadmin call:

sgadmin.crt.pem

Is also a node certificate, means has the OID set?

On Thursday, January 25, 2018 at 1:23:53 PM UTC+1, vale…@servergeek.at wrote:

Hi guys,

I have a 7 node ES cluster and I installed SG on it.

For some reason I **cannot initialize the SG plugin. Any idea why? **

I’m using generated (online by SG) certificates and those seems to be OK.

When I try to initialize SG I do it using the following command:

sgadmin.sh -cacert root-ca.pem -cert sgadmin.crt.pem -key sgadmin.key.pem -keypass blabla -nhnv -cd …/sgconfig/ -h 1.2.3.4 -p 9300 -cn MyCluster

``

My SG configuration block present in ES is the following:

####### SEARCH GUARD #######

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.pemcert_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.crtfull.pem
searchguard.ssl.transport.pemkey_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.key.pem
searchguard.ssl.transport.pemkey_password: e4aae4f746361c10e3aa
searchguard.ssl.transport.pemtrustedcas_filepath: certificates/search-guard-certificates/chain-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.crtfull.pem
searchguard.ssl.http.pemkey_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.key.pem
searchguard.ssl.http.pemkey_password: e4aae4f746361c10e3aa
searchguard.ssl.http.pemtrustedcas_filepath: certificates/search-guard-certificates/chain-ca.pem
searchguard.nodes_dn:

‘*’
searchguard.authcz.admin_dn:

CN=sgadmin

######## End Search Guard Configuration ########

``

Search Guard and Elasticsearch version

SG → 5-5.5.2-16, ES → 5.5.2

``

Installed and used enterprise modules, if any

→ none

``

JVM version and operating system version

→ openjdk version “1.8.0_144”

``

Search Guard configuration files

→ more exactly?

``

Elasticsearch log messages on debug level

→

ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
Trace:
ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1544)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1501)
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1385)
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1267)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at java.lang.Thread.run(Thread.java:748)

``

Other installed Elasticsearch or Kibana plugins, if any

→ none

``

–

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/ruyB5QVFAds/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/3a8548ea-14d4-42f4-b0be-d6a2e2389c62%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

–

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/ruyB5QVFAds/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/90c70109-9415-44d9-8bb3-c95234166c07%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

valentin · January 30, 2018, 9:11am

My current ES settings are the following:

####### SEARCH GUARD #######

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.pemcert_filepath: certificates/node.pem
searchguard.ssl.transport.pemkey_filepath: certificates/node.key
searchguard.ssl.transport.pemkey_password: blabla
searchguard.ssl.transport.pemtrustedcas_filepath: certificates/chain.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: certificates/node.pem
searchguard.ssl.http.pemkey_filepath: certificates/node.key
searchguard.ssl.http.pemkey_password: blabla
searchguard.ssl.http.pemtrustedcas_filepath: certificates/chain.pem
searchguard.nodes_dn:

CN=*.it.internal
searchguard.authcz.admin_dn:
CN=admin

``

···

On Tuesday, January 30, 2018 at 10:09:09 AM UTC+1, vale…@servergeek.at wrote:

Hi Jochen,

Thank you for the info! I have applied the change and also switched to “official” certificates and I have one BIG issue…

[2018-01-30T10:05:13,481][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [ES_MASTER1] SSL Problem error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate
javax.net.ssl.SSLHandshakeException: error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:955) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:914) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:978) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1021) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:205) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1156) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.11.Final.jar:4.1.11.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]

``

Any ideas why is it throwing this error? The certificate and key are OK

(openssl x509 -noout -modulus -in node.pem | openssl md5 ; openssl rsa -noout -modulus -in node.key | openssl md5) | uniq
(stdin)= a5ab322f1cd213600e8ac367a471e06f

``

This is starting to drive me crazy…

Another question would be if the Registered ID:1.2.3.4.5.5 is really needed in the SAN.

Thank you!!!

On Monday, January 29, 2018 at 12:49:58 PM UTC+1, Jochen Kressin wrote:

Ah, sorry, it’s obvious but somehow I totally overlooked it:

It’s this part of the config that causes problems:

searchguard.nodes_dn:

‘*’
If you just use a wildcard here it means that Search Guard will treat every certificate, including the admin one, as node certificate. And that’s why sgadmin chokes with the error message you posted. You need to make sure that the pattern you are using here does not match the DN of the admin certificate. Depending on the DNs of your node certificates that might be something like:

CN=*.it.internal

On Monday, January 29, 2018 at 6:50:14 AM UTC+1, vale…@servergeek.at wrote:

Any updates on this ?

I want to have a running cluster and for some reason I cannot initialize it.

On Friday, January 26, 2018 at 6:21:26 AM UTC+1, vale…@servergeek.at wrote:

Hi,

There is no SAN part in it.

X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Basic Constraints:
CA:FALSE
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Subject Key Identifier:
72:E6:02:00:BF:2A:F0:E4:BB:18:EB:E7:5E:DC:ED:9F:A8:FD:BC:67
X509v3 Authority Key Identifier:
keyid:E9:4A:18:64:74:6D:C6:EF:46:FA:C1:BB:53:62:98:B2:C8:6C:75:4A

``

On Thursday, January 25, 2018 at 3:32:45 PM UTC+1, Jochen Kressin wrote:

It’s in the X509v3 extensions part, I think you just cutted it off. Should be something like:

   X509v3 extensions:

        X509v3 Key Usage: critical

            Digital Signature, Key Encipherment

        X509v3 Basic Constraints:

            CA:FALSE

        X509v3 Extended Key Usage:

            TLS Web Server Authentication, TLS Web Client Authentication

        X509v3 Subject Key Identifier:

            7D:A1:DE:12:4D:AE:D6:79:9D:CF:A8:57:7E:30:08:8B:BA:8E:59:D8

        X509v3 Authority Key Identifier:

            keyid:35:03:23:13:30:30:21:1F:8F:BD:F3:DF:5E:C1:B0:A9:20:88:2C:B0

        **X509v3 Subject Alternative Name:**

** DNS:node-0.example.com, DNS:localhost, IP Address:127.0.0.1, Registered ID:1.2.3.4.5.5**

``

If you see the “Registered ID” part here, it means this is a node certificate.

On Thursday, January 25, 2018 at 3:05:33 PM UTC+1, Valentin Fischer wrote:

Hmmm… no idea how to get that. How do I get/see it ?

On Thu, Jan 25, 2018 at 2:41 PM, Jochen Kressin jkre...@floragunn.com wrote:

Can you please also post the SAN section of the key? The OID field in question is in that section. Thx!

On Thursday, January 25, 2018 at 2:07:16 PM UTC+1, Valentin Fischer wrote:

Hi Jochen,

The certificate has the following properties:

Certificate:

Data:

Version: 3 (0x2)

Serial Number: 1 (0x1)

Signature Algorithm: sha256WithRSAEncryption

Issuer: O=SaS, OU=SaS Signing CA, CN=SaS Signing CA

Validity

Not Before: Jan 23 07:47:48 2018 GMT

Not After : Jan 23 07:47:48 2020 GMT

Subject: CN=sgadmin

Subject Public Key Info:

Public Key Algorithm: rsaEncryption

Public-Key: (2048 bit)

…

X509v3 extensions:

X509v3 Key Usage: critical

Digital Signature, Key Encipherment

X509v3 Basic Constraints:

CA:FALSE

X509v3 Extended Key Usage:

TLS Web Server Authentication, TLS Web Client Authentication

X509v3 Subject Key Identifier:

…

X509v3 Authority Key Identifier:

keyid:…

…

On Thu, Jan 25, 2018 at 1:50 PM, Jochen Kressin jkre...@floragunn.com wrote:

This usually happens why you try to use a node certificate as an admin certificate. We improved the error messages in SG6 when this happens, in SG5 it’s not so obvious

So can you please check if the certificate you use in the sgadmin call:

sgadmin.crt.pem

Is also a node certificate, means has the OID set?

On Thursday, January 25, 2018 at 1:23:53 PM UTC+1, vale…@servergeek.at wrote:

Hi guys,

I have a 7 node ES cluster and I installed SG on it.

For some reason I **cannot initialize the SG plugin. Any idea why? **

I’m using generated (online by SG) certificates and those seems to be OK.

When I try to initialize SG I do it using the following command:

sgadmin.sh -cacert root-ca.pem -cert sgadmin.crt.pem -key sgadmin.key.pem -keypass blabla -nhnv -cd …/sgconfig/ -h 1.2.3.4 -p 9300 -cn MyCluster

``

My SG configuration block present in ES is the following:

####### SEARCH GUARD #######

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.pemcert_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.crtfull.pem
searchguard.ssl.transport.pemkey_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.key.pem
searchguard.ssl.transport.pemkey_password: e4aae4f746361c10e3aa
searchguard.ssl.transport.pemtrustedcas_filepath: certificates/search-guard-certificates/chain-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.crtfull.pem
searchguard.ssl.http.pemkey_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.key.pem
searchguard.ssl.http.pemkey_password: e4aae4f746361c10e3aa
searchguard.ssl.http.pemtrustedcas_filepath: certificates/search-guard-certificates/chain-ca.pem
searchguard.nodes_dn:

‘*’
searchguard.authcz.admin_dn:

CN=sgadmin

######## End Search Guard Configuration ########

``

Search Guard and Elasticsearch version

SG → 5-5.5.2-16, ES → 5.5.2

``

Installed and used enterprise modules, if any

→ none

``

JVM version and operating system version

→ openjdk version “1.8.0_144”

``

Search Guard configuration files

→ more exactly?

``

Elasticsearch log messages on debug level

→

ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
Trace:
ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1544)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1501)
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1385)
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1267)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at java.lang.Thread.run(Thread.java:748)

``

Other installed Elasticsearch or Kibana plugins, if any

→ none

``

–

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/ruyB5QVFAds/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/3a8548ea-14d4-42f4-b0be-d6a2e2389c62%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

–

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/ruyB5QVFAds/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/90c70109-9415-44d9-8bb3-c95234166c07%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

jkressin · January 30, 2018, 1:18pm

I know, TLS can be tricky from time to time …

First regarding the OID. Background of all of this is that Search Guard needs to reliably identify traffic between the nodes in your cluster (inter-node traffic). Inter-node traffic has elevated privileges, so we need to make sure only trusted nodes talk to each other. For this we use TLS. This also shields from the attack vector where an attacker would start a node, let if join your cluster, and then sniffs traffic.

When a node joins the cluster or sends a request to another node, we check if sends a node certificate. We identify a certificate as node certificate by:

Checking if it has the OID in the SAN part OR
Checking the DN against the configures list of DNs in searchguard.nodes_dn

The first approach is more flexible, since it does not require you to make changes in elasticsearch.yml should something with the DNs change. However, not all PKIs are able to add an OID, that’s why we offer the second approach, listing the DNs of the node certificates in elasticsearch.yml. So if you use this second approach, you don’t need to add the OID.

Regarding the exception: According to the stack trace this exception happens on the REST layer (port 9200). It complains that it cannot find a client certificate in the request. What did you do in order for this error to show up? Maybe you pointed sgadmin to the REST port (9200) instead of the transport port (9300)?

···

On Tuesday, January 30, 2018 at 10:11:41 AM UTC+1, valentin@servergeek.at wrote:

My current ES settings are the following:

####### SEARCH GUARD #######

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.pemcert_filepath: certificates/node.pem
searchguard.ssl.transport.pemkey_filepath: certificates/node.key
searchguard.ssl.transport.pemkey_password: blabla
searchguard.ssl.transport.pemtrustedcas_filepath: certificates/chain.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: certificates/node.pem
searchguard.ssl.http.pemkey_filepath: certificates/node.key
searchguard.ssl.http.pemkey_password: blabla
searchguard.ssl.http.pemtrustedcas_filepath: certificates/chain.pem
searchguard.nodes_dn:

CN=*.it.internal
searchguard.authcz.admin_dn:

CN=admin

``

On Tuesday, January 30, 2018 at 10:09:09 AM UTC+1, vale…@servergeek.at wrote:

Hi Jochen,

Thank you for the info! I have applied the change and also switched to “official” certificates and I have one BIG issue…

[2018-01-30T10:05:13,481][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [ES_MASTER1] SSL Problem error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate
javax.net.ssl.SSLHandshakeException: error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:955) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:914) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:978) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1021) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:205) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1156) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.11.Final.jar:4.1.11.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]

``

Any ideas why is it throwing this error? The certificate and key are OK

(openssl x509 -noout -modulus -in node.pem | openssl md5 ; openssl rsa -noout -modulus -in node.key | openssl md5) | uniq
(stdin)= a5ab322f1cd213600e8ac367a471e06f

``

This is starting to drive me crazy…

Another question would be if the Registered ID:1.2.3.4.5.5 is really needed in the SAN.

Thank you!!!

On Monday, January 29, 2018 at 12:49:58 PM UTC+1, Jochen Kressin wrote:

Ah, sorry, it’s obvious but somehow I totally overlooked it:

It’s this part of the config that causes problems:

searchguard.nodes_dn:

‘*’
If you just use a wildcard here it means that Search Guard will treat every certificate, including the admin one, as node certificate. And that’s why sgadmin chokes with the error message you posted. You need to make sure that the pattern you are using here does not match the DN of the admin certificate. Depending on the DNs of your node certificates that might be something like:

CN=*.it.internal

On Monday, January 29, 2018 at 6:50:14 AM UTC+1, vale…@servergeek.at wrote:

Any updates on this ?

I want to have a running cluster and for some reason I cannot initialize it.

On Friday, January 26, 2018 at 6:21:26 AM UTC+1, vale…@servergeek.at wrote:

Hi,

There is no SAN part in it.

X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Basic Constraints:
CA:FALSE
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Subject Key Identifier:
72:E6:02:00:BF:2A:F0:E4:BB:18:EB:E7:5E:DC:ED:9F:A8:FD:BC:67
X509v3 Authority Key Identifier:
keyid:E9:4A:18:64:74:6D:C6:EF:46:FA:C1:BB:53:62:98:B2:C8:6C:75:4A

``

On Thursday, January 25, 2018 at 3:32:45 PM UTC+1, Jochen Kressin wrote:

It’s in the X509v3 extensions part, I think you just cutted it off. Should be something like:

   X509v3 extensions:

        X509v3 Key Usage: critical

            Digital Signature, Key Encipherment

        X509v3 Basic Constraints:

            CA:FALSE

        X509v3 Extended Key Usage:

            TLS Web Server Authentication, TLS Web Client Authentication

        X509v3 Subject Key Identifier:

            7D:A1:DE:12:4D:AE:D6:79:9D:CF:A8:57:7E:30:08:8B:BA:8E:59:D8

        X509v3 Authority Key Identifier:

            keyid:35:03:23:13:30:30:21:1F:8F:BD:F3:DF:5E:C1:B0:A9:20:88:2C:B0

        **X509v3 Subject Alternative Name:**

** DNS:node-0.example.com, DNS:localhost, IP Address:127.0.0.1, Registered ID:1.2.3.4.5.5**

``

If you see the “Registered ID” part here, it means this is a node certificate.

On Thursday, January 25, 2018 at 3:05:33 PM UTC+1, Valentin Fischer wrote:

Hmmm… no idea how to get that. How do I get/see it ?

On Thu, Jan 25, 2018 at 2:41 PM, Jochen Kressin jkre...@floragunn.com wrote:

Can you please also post the SAN section of the key? The OID field in question is in that section. Thx!

On Thursday, January 25, 2018 at 2:07:16 PM UTC+1, Valentin Fischer wrote:

Hi Jochen,

The certificate has the following properties:

Certificate:

Data:

Version: 3 (0x2)

Serial Number: 1 (0x1)

Signature Algorithm: sha256WithRSAEncryption

Issuer: O=SaS, OU=SaS Signing CA, CN=SaS Signing CA

Validity

Not Before: Jan 23 07:47:48 2018 GMT

Not After : Jan 23 07:47:48 2020 GMT

Subject: CN=sgadmin

Subject Public Key Info:

Public Key Algorithm: rsaEncryption

Public-Key: (2048 bit)

…

X509v3 extensions:

X509v3 Key Usage: critical

Digital Signature, Key Encipherment

X509v3 Basic Constraints:

CA:FALSE

X509v3 Extended Key Usage:

TLS Web Server Authentication, TLS Web Client Authentication

X509v3 Subject Key Identifier:

…

X509v3 Authority Key Identifier:

keyid:…

…

On Thu, Jan 25, 2018 at 1:50 PM, Jochen Kressin jkre...@floragunn.com wrote:

This usually happens why you try to use a node certificate as an admin certificate. We improved the error messages in SG6 when this happens, in SG5 it’s not so obvious

So can you please check if the certificate you use in the sgadmin call:

sgadmin.crt.pem

Is also a node certificate, means has the OID set?

On Thursday, January 25, 2018 at 1:23:53 PM UTC+1, vale…@servergeek.at wrote:

Hi guys,

I have a 7 node ES cluster and I installed SG on it.

For some reason I **cannot initialize the SG plugin. Any idea why? **

I’m using generated (online by SG) certificates and those seems to be OK.

When I try to initialize SG I do it using the following command:

sgadmin.sh -cacert root-ca.pem -cert sgadmin.crt.pem -key sgadmin.key.pem -keypass blabla -nhnv -cd …/sgconfig/ -h 1.2.3.4 -p 9300 -cn MyCluster

``

My SG configuration block present in ES is the following:

####### SEARCH GUARD #######

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.pemcert_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.crtfull.pem
searchguard.ssl.transport.pemkey_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.key.pem
searchguard.ssl.transport.pemkey_password: e4aae4f746361c10e3aa
searchguard.ssl.transport.pemtrustedcas_filepath: certificates/search-guard-certificates/chain-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.crtfull.pem
searchguard.ssl.http.pemkey_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.key.pem
searchguard.ssl.http.pemkey_password: e4aae4f746361c10e3aa
searchguard.ssl.http.pemtrustedcas_filepath: certificates/search-guard-certificates/chain-ca.pem
searchguard.nodes_dn:

‘*’
searchguard.authcz.admin_dn:

CN=sgadmin

######## End Search Guard Configuration ########

``

Search Guard and Elasticsearch version

SG → 5-5.5.2-16, ES → 5.5.2

``

Installed and used enterprise modules, if any

→ none

``

JVM version and operating system version

→ openjdk version “1.8.0_144”

``

Search Guard configuration files

→ more exactly?

``

Elasticsearch log messages on debug level

→

ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
Trace:
ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1544)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1501)
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1385)
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1267)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at java.lang.Thread.run(Thread.java:748)

``

Other installed Elasticsearch or Kibana plugins, if any

→ none

``

–

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/ruyB5QVFAds/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/3a8548ea-14d4-42f4-b0be-d6a2e2389c62%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

–

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/ruyB5QVFAds/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/90c70109-9415-44d9-8bb3-c95234166c07%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

valentin · January 30, 2018, 1:25pm

Hi,

Thanks for the reply!

The certificate has the OID: X509v3 Key Usage: critical
Digital Signature, Non Repudiation, Key Encipherment, Data Encipherment
X509v3 Extended Key Usage:
TLS Web Client Authentication, TLS Web Server Authentication
X509v3 Subject Key Identifier:
B5:D9:1F:4C:01:2E:E1:84:FC:84:B6:F7:72:9F:1B:F0:19:09:D2:BB
X509v3 Subject Alternative Name:
DNS:avl2923t.it.internal, Registered ID:1.2.3.4.5.5
X509v3 Authority Key Identifier:
keyid:4E:6F:0C:C1:18:62:1C:2B:A4:E2:7B:C9:A3:D0:5F:1E:57:4A:F8:41

The exception is happening when I start ES:

[2018-01-30T14:21:50,500][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [ES_MASTER1] SSL Problem error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate
javax.net.ssl.SSLHandshakeException: error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:955) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:914) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:978) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1021) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:205) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1156) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.11.Final.jar:4.1.11.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]

The exception is raised every 2 seconds or so.

Any ideas ?

···

On Tue, Jan 30, 2018 at 2:18 PM, Jochen Kressin jkressin@floragunn.com wrote:

I know, TLS can be tricky from time to time …

First regarding the OID. Background of all of this is that Search Guard needs to reliably identify traffic between the nodes in your cluster (inter-node traffic). Inter-node traffic has elevated privileges, so we need to make sure only trusted nodes talk to each other. For this we use TLS. This also shields from the attack vector where an attacker would start a node, let if join your cluster, and then sniffs traffic.

When a node joins the cluster or sends a request to another node, we check if sends a node certificate. We identify a certificate as node certificate by:

Checking if it has the OID in the SAN part OR

Checking the DN against the configures list of DNs in searchguard.nodes_dn

The first approach is more flexible, since it does not require you to make changes in elasticsearch.yml should something with the DNs change. However, not all PKIs are able to add an OID, that’s why we offer the second approach, listing the DNs of the node certificates in elasticsearch.yml. So if you use this second approach, you don’t need to add the OID.

Regarding the exception: According to the stack trace this exception happens on the REST layer (port 9200). It complains that it cannot find a client certificate in the request. What did you do in order for this error to show up? Maybe you pointed sgadmin to the REST port (9200) instead of the transport port (9300)?

On Tuesday, January 30, 2018 at 10:11:41 AM UTC+1, valentin@servergeek.at wrote:

My current ES settings are the following:

####### SEARCH GUARD #######

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.pemcert_filepath: certificates/node.pem
searchguard.ssl.transport.pemkey_filepath: certificates/node.key
searchguard.ssl.transport.pemkey_password: blabla
searchguard.ssl.transport.pemtrustedcas_filepath: certificates/chain.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: certificates/node.pem
searchguard.ssl.http.pemkey_filepath: certificates/node.key
searchguard.ssl.http.pemkey_password: blabla
searchguard.ssl.http.pemtrustedcas_filepath: certificates/chain.pem
searchguard.nodes_dn:

CN=*.it.internal
searchguard.authcz.admin_dn:

CN=admin

``

On Tuesday, January 30, 2018 at 10:09:09 AM UTC+1, vale…@servergeek.at wrote:

Hi Jochen,

Thank you for the info! I have applied the change and also switched to “official” certificates and I have one BIG issue…

[2018-01-30T10:05:13,481][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [ES_MASTER1] SSL Problem error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate
javax.net.ssl.SSLHandshakeException: error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:955) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:914) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:978) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1021) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:205) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1156) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.11.Final.jar:4.1.11.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]

``

Any ideas why is it throwing this error? The certificate and key are OK

(openssl x509 -noout -modulus -in node.pem | openssl md5 ; openssl rsa -noout -modulus -in node.key | openssl md5) | uniq
(stdin)= a5ab322f1cd213600e8ac367a471e06f

``

This is starting to drive me crazy…

Another question would be if the Registered ID:1.2.3.4.5.5 is really needed in the SAN.

Thank you!!!

On Monday, January 29, 2018 at 12:49:58 PM UTC+1, Jochen Kressin wrote:

Ah, sorry, it’s obvious but somehow I totally overlooked it:

It’s this part of the config that causes problems:

searchguard.nodes_dn:

‘*’
If you just use a wildcard here it means that Search Guard will treat every certificate, including the admin one, as node certificate. And that’s why sgadmin chokes with the error message you posted. You need to make sure that the pattern you are using here does not match the DN of the admin certificate. Depending on the DNs of your node certificates that might be something like:

CN=*.it.internal

On Monday, January 29, 2018 at 6:50:14 AM UTC+1, vale…@servergeek.at wrote:

Any updates on this ?

I want to have a running cluster and for some reason I cannot initialize it.

On Friday, January 26, 2018 at 6:21:26 AM UTC+1, vale…@servergeek.at wrote:

Hi,

There is no SAN part in it.

X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Basic Constraints:
CA:FALSE
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Subject Key Identifier:
72:E6:02:00:BF:2A:F0:E4:BB:18:EB:E7:5E:DC:ED:9F:A8:FD:BC:67
X509v3 Authority Key Identifier:
keyid:E9:4A:18:64:74:6D:C6:EF:46:FA:C1:BB:53:62:98:B2:C8:6C:75:4A

``

On Thursday, January 25, 2018 at 3:32:45 PM UTC+1, Jochen Kressin wrote:

It’s in the X509v3 extensions part, I think you just cutted it off. Should be something like:

   X509v3 extensions:

        X509v3 Key Usage: critical

            Digital Signature, Key Encipherment

        X509v3 Basic Constraints:

            CA:FALSE

        X509v3 Extended Key Usage:

            TLS Web Server Authentication, TLS Web Client Authentication

        X509v3 Subject Key Identifier:

            7D:A1:DE:12:4D:AE:D6:79:9D:CF:A8:57:7E:30:08:8B:BA:8E:59:D8

        X509v3 Authority Key Identifier:

            keyid:35:03:23:13:30:30:21:1F:8F:BD:F3:DF:5E:C1:B0:A9:20:88:2C:B0

        **X509v3 Subject Alternative Name:**

** DNS:node-0.example.com, DNS:localhost, IP Address:127.0.0.1, Registered ID:1.2.3.4.5.5**

``

If you see the “Registered ID” part here, it means this is a node certificate.

On Thursday, January 25, 2018 at 3:05:33 PM UTC+1, Valentin Fischer wrote:

Hmmm… no idea how to get that. How do I get/see it ?

On Thu, Jan 25, 2018 at 2:41 PM, Jochen Kressin jkre...@floragunn.com wrote:

Can you please also post the SAN section of the key? The OID field in question is in that section. Thx!

On Thursday, January 25, 2018 at 2:07:16 PM UTC+1, Valentin Fischer wrote:

Hi Jochen,

The certificate has the following properties:

Certificate:

Data:

Version: 3 (0x2)

Serial Number: 1 (0x1)

Signature Algorithm: sha256WithRSAEncryption

Issuer: O=SaS, OU=SaS Signing CA, CN=SaS Signing CA

Validity

Not Before: Jan 23 07:47:48 2018 GMT

Not After : Jan 23 07:47:48 2020 GMT

Subject: CN=sgadmin

Subject Public Key Info:

Public Key Algorithm: rsaEncryption

Public-Key: (2048 bit)

…

X509v3 extensions:

X509v3 Key Usage: critical

Digital Signature, Key Encipherment

X509v3 Basic Constraints:

CA:FALSE

X509v3 Extended Key Usage:

TLS Web Server Authentication, TLS Web Client Authentication

X509v3 Subject Key Identifier:

…

X509v3 Authority Key Identifier:

keyid:…

…

On Thu, Jan 25, 2018 at 1:50 PM, Jochen Kressin jkre...@floragunn.com wrote:

This usually happens why you try to use a node certificate as an admin certificate. We improved the error messages in SG6 when this happens, in SG5 it’s not so obvious

So can you please check if the certificate you use in the sgadmin call:

sgadmin.crt.pem

Is also a node certificate, means has the OID set?

On Thursday, January 25, 2018 at 1:23:53 PM UTC+1, vale…@servergeek.at wrote:

Hi guys,

I have a 7 node ES cluster and I installed SG on it.

For some reason I **cannot initialize the SG plugin. Any idea why? **

I’m using generated (online by SG) certificates and those seems to be OK.

When I try to initialize SG I do it using the following command:

sgadmin.sh -cacert root-ca.pem -cert sgadmin.crt.pem -key sgadmin.key.pem -keypass blabla -nhnv -cd …/sgconfig/ -h 1.2.3.4 -p 9300 -cn MyCluster

``

My SG configuration block present in ES is the following:

####### SEARCH GUARD #######

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.pemcert_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.crtfull.pem
searchguard.ssl.transport.pemkey_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.key.pem
searchguard.ssl.transport.pemkey_password: e4aae4f746361c10e3aa
searchguard.ssl.transport.pemtrustedcas_filepath: certificates/search-guard-certificates/chain-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.crtfull.pem
searchguard.ssl.http.pemkey_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.key.pem
searchguard.ssl.http.pemkey_password: e4aae4f746361c10e3aa
searchguard.ssl.http.pemtrustedcas_filepath: certificates/search-guard-certificates/chain-ca.pem
searchguard.nodes_dn:

‘*’
searchguard.authcz.admin_dn:

CN=sgadmin

######## End Search Guard Configuration ########

``

Search Guard and Elasticsearch version

SG → 5-5.5.2-16, ES → 5.5.2

``

Installed and used enterprise modules, if any

→ none

``

JVM version and operating system version

→ openjdk version “1.8.0_144”

``

Search Guard configuration files

→ more exactly?

``

Elasticsearch log messages on debug level

→

ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
Trace:
ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1544)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1501)
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1385)
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1267)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at java.lang.Thread.run(Thread.java:748)

``

Other installed Elasticsearch or Kibana plugins, if any

→ none

``

–

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/ruyB5QVFAds/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/3a8548ea-14d4-42f4-b0be-d6a2e2389c62%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

–

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/ruyB5QVFAds/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/90c70109-9415-44d9-8bb3-c95234166c07%40googlegroups.com.

–

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/ruyB5QVFAds/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/6961a039-a75b-4f7a-b5c8-51cbed8c6ffa%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

valentin · January 30, 2018, 1:29pm

You’re suggestion about something else talking with the node opened my eyes!
I did a tcpdump and seen that an older node was trying to talk with this node…

Stopped it and this node is starting “clean”.

···

Now I’ll add another one to see if they can talk to each other.

Thanks for the replies!

Topic		Replies	Views
Can't init searchguard after new certificates Search Guard	4	1572	April 24, 2019
sgadmin won't start Search Guard	3	487	September 27, 2017
Search Guard not initialized (SG11). Search Guard	1	1748	January 18, 2017
We cant initialize SeaarchGuard Search Guard	6	373	May 19, 2016
Search Guard not initialized (SG11) searchguard 6.7.1 Search Guard	12	2181	June 6, 2019

Cannot initialize SearchGuard

Related Topics