sgadmin failure with PEM certificates

Hi there,

I’m trying to setup searchguard but sgadmin fails with the following output

vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ sudo ./sgadmin.sh -cert ~/admin001.crt -cacert /etc/elasticsearch/rest.ca -key ~/admin001.key --disable-host-name-verification --diagnose -icl
WARNING: JAVA_HOME not set, will use /usr/bin/java

Search Guard Admin v5

Will connect to localhost:9300 … done

LICENSE NOTICE Search Guard

If you use one or more of the following features in production
make sure you have a valid Search Guard license
(See https://floragunn.com/searchguard-validate-license)

  • Kibana Multitenancy
  • LDAP authentication/authorization
  • Active Directory authentication/authorization
  • REST Management API
  • JSON Web Token (JWT) authentication/authorization
  • Kerberos authentication/authorization
  • Document- and Fieldlevel Security (DLS/FLS)
  • Auditlogging

In case of any doubt mail to sales@floragunn.com

···

###################################
Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-5/tools/sgadmin_diag_trace_2017-Nov-22_17-10-58.txt
Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying …

Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)

  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
  • Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
    Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying …
    Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)
  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
  • Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
    Clustername: elasticsearch
    Clusterstate: GREEN
    Number of nodes: 1
    Number of data nodes: 1
    ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
    Trace:
    ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
    at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
    at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
    at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
    at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
    at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
    at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
    at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
    at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1553)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
    at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
    at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1510)
    at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1393)
    at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1273)
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1084)
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
    at java.lang.Thread.run(Thread.java:748)

``

Meanwhile, in the elasticsearch logs, we have (traceback redacted for readability);

Nov 22 17:10:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.ConfigurationLoader] Failure no such index retrieving configuration for [roles] (index=searchguard)
Nov 22 17:11:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.IndexBaseConfigurationRepository] Unable to load configuration because of java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for roles
Nov 22 17:11:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.f.SearchGuardFilter] Search Guard not initialized (SG11) for indices:monitor/stats
Nov 22 17:12:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.ConfigurationLoader] Failure no such index retrieving configuration for [roles] (index=searchguard)
Nov 22 17:13:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.IndexBaseConfigurationRepository] Unable to load configuration because of java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for roles
Nov 22 17:13:04 elasticsearch elasticsearch[1180]: java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for roles
Nov 22 17:13:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.f.SearchGuardFilter] Search Guard not initialized (SG11) for indices:admin/exists

``

Here we show that the admin certificate is indeed valid against the specified CA

vagrant@elasticsearch:~ openssl x509 -noout -text -in ~/admin001.crt | grep Subject: Subject: CN=admin001 vagrant@elasticsearch:~ sudo openssl verify -CAfile /etc/elasticsearch/rest.ca ~/admin001.crt
/home/vagrant/admin001.crt: OK

``

Here is the elasticsearch.yml configuration

vagrant@elasticsearch:~ sudo grep -e '^’ -e ‘^#’ --invert-match /etc/elasticsearch/elasticsearch.yml
cluster.name: “elasticsearch”
node.name: “elasticsearchminion”
node.master: true
node.data: true
network.bind_host: 0.0.0.0
network.publish_host: 0.0.0.0
network.host: 0.0.0.0
http.port: 9201
discovery.zen.ping.unicast.hosts: [
]
cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.low: 15gb
cluster.routing.allocation.disk.watermark.high: 5gb
searchguard.ssl.transport.pemkey_filepath: transport.key
searchguard.ssl.transport.pemcert_filepath: transport.cert
searchguard.ssl.transport.pemtrustedcas_filepath: transport.ca
searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.keystore_type: x509
searchguard.ssl.http.pemkey_filepath: rest.key
searchguard.ssl.http.pemcert_filepath: rest.cert
searchguard.ssl.http.pemtrustedcas_filepath: rest.ca
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.authcz.admin_dn:

  • ‘*’
    searchguard.nodes_dn:
  • ‘*’
    searchguard.audit.type: com.payplug.auditlog.impl.StdoutAuditLog
    searchguard.audit.enable_request_details: true
    searchguard.audit.ignore_users:
  • kibanaserver

``

What am I missing here ? For what it’s worth, all cryptographic material have been generated using openssl/easyrsa (I’m planning on documenting it).
Please let me know if I can provide any extra informations.

Arthur

searchguard.authcz.admin_dn can not be a wildcard

···

Am 23.11.2017 um 10:39 schrieb calvinh34@gmail.com:

Hi there,

I'm trying to setup searchguard but sgadmin fails with the following output

vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ sudo ./sgadmin.sh -cert ~/admin001.crt -cacert /etc/elasticsearch/rest.ca -key ~/admin001.key --disable-host-name-verification --diagnose -icl
WARNING: JAVA_HOME not set, will use /usr/bin/java

Search Guard Admin v5

Will connect to localhost:9300 ... done

### LICENSE NOTICE Search Guard ###

If you use one or more of the following features in production
make sure you have a valid Search Guard license
(See https://floragunn.com/searchguard-validate-license)

* Kibana Multitenancy
* LDAP authentication/authorization
* Active Directory authentication/authorization
* REST Management API
* JSON Web Token (JWT) authentication/authorization
* Kerberos authentication/authorization
* Document- and Fieldlevel Security (DLS/FLS)
* Auditlogging

In case of any doubt mail to <sales@floragunn.com>
###################################
Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-5/tools/sgadmin_diag_trace_2017-Nov-22_17-10-58.txt
Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying ...

Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)
* Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
* Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
* If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
* Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying ...
Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)
* Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
* Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
* If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
* Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
Clustername: elasticsearch
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
Trace:
ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1553)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1510)
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1393)
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1273)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1084)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at java.lang.Thread.run(Thread.java:748)

Meanwhile, in the elasticsearch logs, we have (traceback redacted for readability);

Nov 22 17:10:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.ConfigurationLoader] Failure no such index retrieving configuration for [roles] (index=searchguard)
Nov 22 17:11:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.IndexBaseConfigurationRepository] Unable to load configuration because of java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for [roles](index=searchguard)
Nov 22 17:11:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.f.SearchGuardFilter] Search Guard not initialized (SG11) for indices:monitor/stats
Nov 22 17:12:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.ConfigurationLoader] Failure no such index retrieving configuration for [roles] (index=searchguard)
Nov 22 17:13:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.IndexBaseConfigurationRepository] Unable to load configuration because of java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for [roles](index=searchguard)
Nov 22 17:13:04 elasticsearch elasticsearch[1180]: java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for [roles](index=searchguard)
Nov 22 17:13:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.f.SearchGuardFilter] Search Guard not initialized (SG11) for indices:admin/exists

Here we show that the admin certificate is indeed valid against the specified CA

vagrant@elasticsearch:~ openssl x509 \-noout \-text \-in \~/admin001\.crt | grep Subject: Subject: CN=admin001 vagrant@elasticsearch:\~ sudo openssl verify -CAfile /etc/elasticsearch/rest.ca ~/admin001.crt
/home/vagrant/admin001.crt: OK

Here is the elasticsearch.yml configuration

vagrant@elasticsearch:~ sudo grep \-e &#39;^' -e '^#' --invert-match /etc/elasticsearch/elasticsearch.yml
cluster.name: "elasticsearch"
node.name: "elasticsearchminion"
node.master: true
node.data: true
network.bind_host: 0.0.0.0
network.publish_host: 0.0.0.0
network.host: 0.0.0.0
http.port: 9201
discovery.zen.ping.unicast.hosts: [
]
cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.low: 15gb
cluster.routing.allocation.disk.watermark.high: 5gb
searchguard.ssl.transport.pemkey_filepath: transport.key
searchguard.ssl.transport.pemcert_filepath: transport.cert
searchguard.ssl.transport.pemtrustedcas_filepath: transport.ca
searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.keystore_type: x509
searchguard.ssl.http.pemkey_filepath: rest.key
searchguard.ssl.http.pemcert_filepath: rest.cert
searchguard.ssl.http.pemtrustedcas_filepath: rest.ca
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.authcz.admin_dn:
- '*'
searchguard.nodes_dn:
- '*'
searchguard.audit.type: com.payplug.auditlog.impl.StdoutAuditLog
searchguard.audit.enable_request_details: true
searchguard.audit.ignore_users:
- kibanaserver

What am I missing here ? For what it's worth, all cryptographic material have been generated using openssl/easyrsa (I'm planning on documenting it).
Please let me know if I can provide any extra informations.

Arthur

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/166328f7-872a-4aec-804a-27187b477b97%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Thanks for the quick answer !

Not much success tho;

vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ sudo grep -e ‘^$’ -e ‘^#’ --invert-match /etc/elasticsearch/elasticsearch.yml
cluster.name: “elasticsearch”
node.name: “elasticsearchminion”
node.master: true
node.data: true
network.bind_host: 0.0.0.0
network.publish_host: 0.0.0.0
network.host: 0.0.0.0
http.port: 9201
discovery.zen.ping.unicast.hosts: [
]
cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.low: 15gb
cluster.routing.allocation.disk.watermark.high: 5gb
searchguard.ssl.transport.pemkey_filepath: transport.key
searchguard.ssl.transport.pemcert_filepath: transport.cert
searchguard.ssl.transport.pemtrustedcas_filepath: transport.ca
searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.keystore_type: x509
searchguard.ssl.http.pemkey_filepath: rest.key
searchguard.ssl.http.pemcert_filepath: rest.cert
searchguard.ssl.http.pemtrustedcas_filepath: rest.ca
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.authcz.admin_dn:

  • CN=admin001
    searchguard.nodes_dn:
  • ‘*’
    searchguard.audit.type: com.payplug.auditlog.impl.StdoutAuditLog
    searchguard.audit.enable_request_details: true
    searchguard.audit.ignore_users:
  • kibanaserver
    vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ sudo service elasticsearch restart
    vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ sudo ./sgadmin.sh -cert ~/admin001.crt -cacert /etc/elasticsearch/rest.ca -key ~/admin001.key --disable-host-name-verification --diagnose -icl
    WARNING: JAVA_HOME not set, will use /usr/bin/java
    Search Guard Admin v5
    Will connect to localhost:9300 … done

LICENSE NOTICE Search Guard

If you use one or more of the following features in production
make sure you have a valid Search Guard license
(See https://floragunn.com/searchguard-validate-license)

  • Kibana Multitenancy
  • LDAP authentication/authorization
  • Active Directory authentication/authorization
  • REST Management API
  • JSON Web Token (JWT) authentication/authorization
  • Kerberos authentication/authorization
  • Document- and Fieldlevel Security (DLS/FLS)
  • Auditlogging

In case of any doubt mail to sales@floragunn.com

···

###################################
Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-5/tools/sgadmin_diag_trace_2017-Nov-23_10-52-14.txt
Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …
Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{dhWeO-uaT4KaitGsjFknKA}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying …
Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{dhWeO-uaT4KaitGsjFknKA}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)

  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
  • Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
    Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{dhWeO-uaT4KaitGsjFknKA}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying …
    Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{dhWeO-uaT4KaitGsjFknKA}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)
  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
  • Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
    Clustername: elasticsearch
    Clusterstate: GREEN
    Number of nodes: 1
    Number of data nodes: 1
    ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
    Trace:
    ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
    at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
    at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
    at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
    at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
    at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
    at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
    at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
    at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1553)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
    at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
    at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1510)
    at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1393)
    at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1273)
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1084)
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
    at java.lang.Thread.run(Thread.java:748)

``

The log on the Elasticsearch side are the same, final entry is still

[ERROR][c.f.s.f.SearchGuardFilter] Search Guard not initialized (SG11) for indices:admin/exists

``

On Thursday, November 23, 2017 at 11:09:10 AM UTC+1, Search Guard wrote:

searchguard.authcz.admin_dn can not be a wildcard

Am 23.11.2017 um 10:39 schrieb calv...@gmail.com:

Hi there,

I’m trying to setup searchguard but sgadmin fails with the following output

vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ sudo ./sgadmin.sh -cert ~/admin001.crt -cacert /etc/elasticsearch/rest.ca -key ~/admin001.key --disable-host-name-verification --diagnose -icl
WARNING: JAVA_HOME not set, will use /usr/bin/java

Search Guard Admin v5

Will connect to localhost:9300 … done

LICENSE NOTICE Search Guard

If you use one or more of the following features in production
make sure you have a valid Search Guard license
(See https://floragunn.com/searchguard-validate-license)

  • Kibana Multitenancy
  • LDAP authentication/authorization
  • Active Directory authentication/authorization
  • REST Management API
  • JSON Web Token (JWT) authentication/authorization
  • Kerberos authentication/authorization
  • Document- and Fieldlevel Security (DLS/FLS)
  • Auditlogging

In case of any doubt mail to sa...@floragunn.com
###################################
Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-5/tools/sgadmin_diag_trace_2017-Nov-22_17-10-58.txt
Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying …

Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)

  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
  • Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
    Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying …
    Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)
  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
  • Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
    Clustername: elasticsearch
    Clusterstate: GREEN
    Number of nodes: 1
    Number of data nodes: 1
    ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md

Trace:
ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1553)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1510)
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1393)
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1273)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1084)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at java.lang.Thread.run(Thread.java:748)

Meanwhile, in the elasticsearch logs, we have (traceback redacted for readability);

Nov 22 17:10:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.ConfigurationLoader] Failure no such index retrieving configuration for [roles] (index=searchguard)
Nov 22 17:11:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.IndexBaseConfigurationRepository] Unable to load configuration because of java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for roles
Nov 22 17:11:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.f.SearchGuardFilter] Search Guard not initialized (SG11) for indices:monitor/stats
Nov 22 17:12:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.ConfigurationLoader] Failure no such index retrieving configuration for [roles] (index=searchguard)
Nov 22 17:13:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.IndexBaseConfigurationRepository] Unable to load configuration because of java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for roles
Nov 22 17:13:04 elasticsearch elasticsearch[1180]: java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for roles
Nov 22 17:13:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.f.SearchGuardFilter] Search Guard not initialized (SG11) for indices:admin/exists

Here we show that the admin certificate is indeed valid against the specified CA

vagrant@elasticsearch:~ openssl x509 -noout -text -in ~/admin001.crt | grep Subject: Subject: CN=admin001 vagrant@elasticsearch:~ sudo openssl verify -CAfile /etc/elasticsearch/rest.ca ~/admin001.crt
/home/vagrant/admin001.crt: OK

Here is the elasticsearch.yml configuration

vagrant@elasticsearch:~ sudo grep -e '^’ -e ‘^#’ --invert-match /etc/elasticsearch/elasticsearch.yml
cluster.name: “elasticsearch”
node.name: “elasticsearchminion”
node.master: true
node.data: true
network.bind_host: 0.0.0.0
network.publish_host: 0.0.0.0
network.host: 0.0.0.0
http.port: 9201
discovery.zen.ping.unicast.hosts: [
]
cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.low: 15gb
cluster.routing.allocation.disk.watermark.high: 5gb
searchguard.ssl.transport.pemkey_filepath: transport.key
searchguard.ssl.transport.pemcert_filepath: transport.cert
searchguard.ssl.transport.pemtrustedcas_filepath: transport.ca

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.keystore_type: x509
searchguard.ssl.http.pemkey_filepath: rest.key
searchguard.ssl.http.pemcert_filepath: rest.cert
searchguard.ssl.http.pemtrustedcas_filepath: rest.ca

searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.authcz.admin_dn:

  • ‘*’
    searchguard.nodes_dn:
  • ‘*’
    searchguard.audit.type: com.payplug.auditlog.impl.StdoutAuditLog
    searchguard.audit.enable_request_details: true
    searchguard.audit.ignore_users:
  • kibanaserver

What am I missing here ? For what it’s worth, all cryptographic material have been generated using openssl/easyrsa (I’m planning on documenting it).
Please let me know if I can provide any extra informations.

Arthur


You received this message because you are subscribed to the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/166328f7-872a-4aec-804a-27187b477b97%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

I am also using pem certs and facing the same issue. I have put in an exact DN match.

Should I be changing any settings in sg_config dir?

Thanks

···

On Thursday, November 23, 2017 at 3:09:09 PM UTC+5:30, calv...@gmail.com wrote:

Hi there,

I’m trying to setup searchguard but sgadmin fails with the following output

vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ sudo ./sgadmin.sh -cert ~/admin001.crt -cacert /etc/elasticsearch/rest.ca -key ~/admin001.key --disable-host-name-verification --diagnose -icl
WARNING: JAVA_HOME not set, will use /usr/bin/java

Search Guard Admin v5

Will connect to localhost:9300 … done

LICENSE NOTICE Search Guard

If you use one or more of the following features in production
make sure you have a valid Search Guard license
(See https://floragunn.com/searchguard-validate-license)

  • Kibana Multitenancy
  • LDAP authentication/authorization
  • Active Directory authentication/authorization
  • REST Management API
  • JSON Web Token (JWT) authentication/authorization
  • Kerberos authentication/authorization
  • Document- and Fieldlevel Security (DLS/FLS)
  • Auditlogging

In case of any doubt mail to sales@floragunn.com
###################################
Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-5/tools/sgadmin_diag_trace_2017-Nov-22_17-10-58.txt
Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying …

Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)

  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
  • Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
    Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying …
    Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)
  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
  • Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
    Clustername: elasticsearch
    Clusterstate: GREEN
    Number of nodes: 1
    Number of data nodes: 1
    ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
    Trace:
    ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
    at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
    at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
    at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
    at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
    at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
    at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
    at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
    at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1553)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
    at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
    at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1510)
    at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1393)
    at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1273)
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1084)
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
    at java.lang.Thread.run(Thread.java:748)

``

Meanwhile, in the elasticsearch logs, we have (traceback redacted for readability);

Nov 22 17:10:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.ConfigurationLoader] Failure no such index retrieving configuration for [roles] (index=searchguard)
Nov 22 17:11:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.IndexBaseConfigurationRepository] Unable to load configuration because of java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for roles
Nov 22 17:11:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.f.SearchGuardFilter] Search Guard not initialized (SG11) for indices:monitor/stats
Nov 22 17:12:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.ConfigurationLoader] Failure no such index retrieving configuration for [roles] (index=searchguard)
Nov 22 17:13:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.IndexBaseConfigurationRepository] Unable to load configuration because of java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for roles
Nov 22 17:13:04 elasticsearch elasticsearch[1180]: java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for roles
Nov 22 17:13:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.f.SearchGuardFilter] Search Guard not initialized (SG11) for indices:admin/exists

``

Here we show that the admin certificate is indeed valid against the specified CA

vagrant@elasticsearch:~ openssl x509 -noout -text -in ~/admin001.crt | grep Subject: Subject: CN=admin001 vagrant@elasticsearch:~ sudo openssl verify -CAfile /etc/elasticsearch/rest.ca ~/admin001.crt
/home/vagrant/admin001.crt: OK

``

Here is the elasticsearch.yml configuration

vagrant@elasticsearch:~ sudo grep -e '^’ -e ‘^#’ --invert-match /etc/elasticsearch/elasticsearch.yml
cluster.name: “elasticsearch”
node.name: “elasticsearchminion”
node.master: true
node.data: true
network.bind_host: 0.0.0.0
network.publish_host: 0.0.0.0
network.host: 0.0.0.0
http.port: 9201
discovery.zen.ping.unicast.hosts: [
]
cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.low: 15gb
cluster.routing.allocation.disk.watermark.high: 5gb
searchguard.ssl.transport.pemkey_filepath: transport.key
searchguard.ssl.transport.pemcert_filepath: transport.cert
searchguard.ssl.transport.pemtrustedcas_filepath: transport.ca
searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.keystore_type: x509
searchguard.ssl.http.pemkey_filepath: rest.key
searchguard.ssl.http.pemcert_filepath: rest.cert
searchguard.ssl.http.pemtrustedcas_filepath: rest.ca
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.authcz.admin_dn:

  • ‘*’
    searchguard.nodes_dn:
  • ‘*’
    searchguard.audit.type: com.payplug.auditlog.impl.StdoutAuditLog
    searchguard.audit.enable_request_details: true
    searchguard.audit.ignore_users:
  • kibanaserver

``

What am I missing here ? For what it’s worth, all cryptographic material have been generated using openssl/easyrsa (I’m planning on documenting it).
Please let me know if I can provide any extra informations.

Arthur

Can you post (or mail) the output of

openssl x509 -in ~/admin001.crt -text -noout

···

Am 23.11.2017 um 12:09 schrieb calvinh34@gmail.com:

Thanks for the quick answer !

Not much success tho;

vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ sudo grep -e '^&#39; \-e &#39;^\#&#39; \-\-invert\-match /etc/elasticsearch/elasticsearch\.yml cluster\.name: &quot;elasticsearch&quot; node\.name: &quot;elasticsearchminion&quot; node\.master: true node\.data: true network\.bind\_host: 0\.0\.0\.0 network\.publish\_host: 0\.0\.0\.0 network\.host: 0\.0\.0\.0 http\.port: 9201 discovery\.zen\.ping\.unicast\.hosts: \[ \] cluster\.routing\.allocation\.disk\.threshold\_enabled: true cluster\.routing\.allocation\.disk\.watermark\.low: 15gb cluster\.routing\.allocation\.disk\.watermark\.high: 5gb searchguard\.ssl\.transport\.pemkey\_filepath: transport\.key searchguard\.ssl\.transport\.pemcert\_filepath: transport\.cert searchguard\.ssl\.transport\.pemtrustedcas\_filepath: transport\.ca searchguard\.ssl\.transport\.enabled: true searchguard\.ssl\.transport\.keystore\_type: x509 searchguard\.ssl\.http\.pemkey\_filepath: rest\.key searchguard\.ssl\.http\.pemcert\_filepath: rest\.cert searchguard\.ssl\.http\.pemtrustedcas\_filepath: rest\.ca searchguard\.ssl\.transport\.enforce\_hostname\_verification: false searchguard\.ssl\.http\.enabled: true searchguard\.authcz\.admin\_dn: &nbsp;&nbsp;\- CN=admin001 searchguard\.nodes\_dn: &nbsp;&nbsp;\- &#39;\*&#39; searchguard\.audit\.type: com\.payplug\.auditlog\.impl\.StdoutAuditLog searchguard\.audit\.enable\_request\_details: true searchguard\.audit\.ignore\_users: &nbsp;&nbsp;\- kibanaserver vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search\-guard\-5/tools sudo service elasticsearch restart
vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ sudo ./sgadmin.sh -cert ~/admin001.crt -cacert /etc/elasticsearch/rest.ca -key ~/admin001.key --disable-host-name-verification --diagnose -icl
WARNING: JAVA_HOME not set, will use /usr/bin/java
Search Guard Admin v5
Will connect to localhost:9300 ... done

### LICENSE NOTICE Search Guard ###

If you use one or more of the following features in production
make sure you have a valid Search Guard license
(See https://floragunn.com/searchguard-validate-license)

* Kibana Multitenancy
* LDAP authentication/authorization
* Active Directory authentication/authorization
* REST Management API
* JSON Web Token (JWT) authentication/authorization
* Kerberos authentication/authorization
* Document- and Fieldlevel Security (DLS/FLS)
* Auditlogging

In case of any doubt mail to <sales@floragunn.com>
###################################
Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-5/tools/sgadmin_diag_trace_2017-Nov-23_10-52-14.txt
Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{dhWeO-uaT4KaitGsjFknKA}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying ...
  Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{dhWeO-uaT4KaitGsjFknKA}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)
   * Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
   * Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
   * If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{dhWeO-uaT4KaitGsjFknKA}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying ...
  Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{dhWeO-uaT4KaitGsjFknKA}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)
   * Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
   * Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
   * If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
Clustername: elasticsearch
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
Trace:
ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
        at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
        at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
        at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
        at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
        at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
        at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
        at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
        at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1553)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
        at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
        at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1510)
        at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1393)
        at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1273)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1084)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
        at java.lang.Thread.run(Thread.java:748)

The log on the Elasticsearch side are the same, final entry is still

[ERROR][c.f.s.f.SearchGuardFilter] Search Guard not initialized (SG11) for indices:admin/exists

On Thursday, November 23, 2017 at 11:09:10 AM UTC+1, Search Guard wrote:
searchguard.authcz.admin_dn can not be a wildcard

> Am 23.11.2017 um 10:39 schrieb calv...@gmail.com:
>
> Hi there,
>
> I'm trying to setup searchguard but sgadmin fails with the following output
>
> vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ sudo ./sgadmin.sh -cert ~/admin001.crt -cacert /etc/elasticsearch/rest.ca -key ~/admin001.key --disable-host-name-verification --diagnose -icl
> WARNING: JAVA_HOME not set, will use /usr/bin/java
>
> Search Guard Admin v5
>
> Will connect to localhost:9300 ... done
>
>
> ### LICENSE NOTICE Search Guard ###
>
> If you use one or more of the following features in production
> make sure you have a valid Search Guard license
> (See https://floragunn.com/searchguard-validate-license)
>
> * Kibana Multitenancy
> * LDAP authentication/authorization
> * Active Directory authentication/authorization
> * REST Management API
> * JSON Web Token (JWT) authentication/authorization
> * Kerberos authentication/authorization
> * Document- and Fieldlevel Security (DLS/FLS)
> * Auditlogging
>
> In case of any doubt mail to <sa...@floragunn.com>
> ###################################
> Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-5/tools/sgadmin_diag_trace_2017-Nov-22_17-10-58.txt
> Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
>
> Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying ...
>
> Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)
> * Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
> * Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
> * If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
> * Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
> Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying ...
> Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)
> * Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
> * Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
> * If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
> * Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
> Clustername: elasticsearch
> Clusterstate: GREEN
> Number of nodes: 1
> Number of data nodes: 1
> ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
> Trace:
> ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
> at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
> at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
> at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
> at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
> at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
> at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
> at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
> at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
> at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
> at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
> at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1553)
> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
> at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
> at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1510)
> at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1393)
> at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
> at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
> at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
> at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1273)
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1084)
> at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
> at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
> at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
> at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
> at java.lang.Thread.run(Thread.java:748)
>
>
>
>
> Meanwhile, in the elasticsearch logs, we have (traceback redacted for readability);
>
> Nov 22 17:10:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.ConfigurationLoader] Failure no such index retrieving configuration for [roles] (index=searchguard)
> Nov 22 17:11:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.IndexBaseConfigurationRepository] Unable to load configuration because of java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for [roles](index=searchguard)
> Nov 22 17:11:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.f.SearchGuardFilter] Search Guard not initialized (SG11) for indices:monitor/stats
> Nov 22 17:12:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.ConfigurationLoader] Failure no such index retrieving configuration for [roles] (index=searchguard)
> Nov 22 17:13:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.IndexBaseConfigurationRepository] Unable to load configuration because of java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for [roles](index=searchguard)
> Nov 22 17:13:04 elasticsearch elasticsearch[1180]: java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for [roles](index=searchguard)
> Nov 22 17:13:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.f.SearchGuardFilter] Search Guard not initialized (SG11) for indices:admin/exists
>
> Here we show that the admin certificate is indeed valid against the specified CA
>
> vagrant@elasticsearch:~ openssl x509 \-noout \-text \-in \~/admin001\.crt | grep Subject: &gt; Subject: CN=admin001 &gt; vagrant@elasticsearch:\~ sudo openssl verify -CAfile /etc/elasticsearch/rest.ca ~/admin001.crt
> /home/vagrant/admin001.crt: OK
>
>
>
>
> Here is the elasticsearch.yml configuration
>
>
> vagrant@elasticsearch:~ sudo grep \-e &#39;^' -e '^#' --invert-match /etc/elasticsearch/elasticsearch.yml
> cluster.name: "elasticsearch"
> node.name: "elasticsearchminion"
> node.master: true
> node.data: true
> network.bind_host: 0.0.0.0
> network.publish_host: 0.0.0.0
> network.host: 0.0.0.0
> http.port: 9201
> discovery.zen.ping.unicast.hosts: [
> ]
> cluster.routing.allocation.disk.threshold_enabled: true
> cluster.routing.allocation.disk.watermark.low: 15gb
> cluster.routing.allocation.disk.watermark.high: 5gb
> searchguard.ssl.transport.pemkey_filepath: transport.key
> searchguard.ssl.transport.pemcert_filepath: transport.cert
> searchguard.ssl.transport.pemtrustedcas_filepath: transport.ca
> searchguard.ssl.transport.enabled: true
> searchguard.ssl.transport.keystore_type: x509
> searchguard.ssl.http.pemkey_filepath: rest.key
> searchguard.ssl.http.pemcert_filepath: rest.cert
> searchguard.ssl.http.pemtrustedcas_filepath: rest.ca
> searchguard.ssl.transport.enforce_hostname_verification: false
> searchguard.ssl.http.enabled: true
> searchguard.authcz.admin_dn:
> - '*'
> searchguard.nodes_dn:
> - '*'
> searchguard.audit.type: com.payplug.auditlog.impl.StdoutAuditLog
> searchguard.audit.enable_request_details: true
> searchguard.audit.ignore_users:
> - kibanaserver
>
>
>
> What am I missing here ? For what it's worth, all cryptographic material have been generated using openssl/easyrsa (I'm planning on documenting it).
> Please let me know if I can provide any extra informations.
>
> Arthur
>
> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/166328f7-872a-4aec-804a-27187b477b97%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/1e926131-b185-495d-a894-27eeb6c66603%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Here you are. Maybe it’s missing an ExtendedKeyUsage attribute ?

vagrant@elasticsearch:…$ openssl x509 -in ~/admin001.crt -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 6 (0x6)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=mycompany test ca
Validity
Not Before: Nov 22 15:15:31 2017 GMT
Not After : Nov 20 15:15:31 2027 GMT
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:f2:7c:8a:fd:5b:d2:1e:1e:01:52:32:9f:ae:57:
fd:c1:8c:94:52:dd:e7:3a:2f:8c:3f:71:44:ab:81:
79:37:64:08:d5:76:a8:36:be:29:60:27:13:fd:23:
92:db:bb:f9:de:cc:3e:88:c5:7d:69:e3:48:ca:0b:
3d:8e:d1:81:73:7a:14:05:95:a0:95:8b:70:ef:d5:
65:81:01:57:39:45:fa:c2:28:81:52:f2:4f:de:fd:
38:1a:f1:11:e6:9c:36:6a:51:3a:b8:5a:b1:51:c1:
04:3d:fe:b1:55:24:32:a6:3f:f3:83:7b:e4:77:1c:
45:03:49:9f:ac:e2:dc:5f:f5:8a:34:ac:3b:c2:73:
a3:70:5a:63:e5:32:4a:b4:99:4a:53:1c:9d:10:dd:
6c:ba:72:88:86:29:c7:da:7c:5a:60:ed:d8:74:cd:
0f:47:d8:b3:6f:be:75:25:fa:5d:23:43:fd:2c:c3:
b7:74:57:17:e1:04:76:6f:b9:82:08:c5:af:2b:ce:
f5:14:d2:4c:02:f6:47:f3:0b:2a:c9:80:4a:fd:23:
be:be:00:3c:4d:af:ff:b5:65:24:fb:49:d5:20:24:
d4:4a:26:cc:c2:71:30:94:31:68:78:7b:8b:df:d0:
e8:f8:eb:34:d6:ba:1c:e6:95:9a:54:f3:0c:29:2b:
6f:2f
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
X509v3 Subject Key Identifier:
0B:8F:E0:5C:5C:02:36:C7:37:8B:17:90:0E:D8:04:D9:C8:25:29:11
X509v3 Authority Key Identifier:
keyid:74:32:94:50:67:DF:4C:95:03:18:D0:51:08:A6:50:14:E0:8A:42:C8
DirName:/CN=mycompany test ca
serial:A7:CD:62:39:B3:FF:48:76

        X509v3 Extended Key Usage:
            TLS Web Client Authentication
        X509v3 Key Usage:
            Digital Signature
Signature Algorithm: sha256WithRSAEncryption
     a1:97:f8:e7:19:9a:18:40:af:a1:91:7d:35:14:34:2a:1a:14:
     a4:02:ed:65:27:26:00:be:02:37:dc:4e:b2:27:16:4d:06:a7:
     4a:00:44:ec:02:43:9b:8c:ae:81:6b:84:34:64:1d:1b:85:ff:
     6d:ab:0e:cd:a1:43:92:15:fb:7e:6b:0e:9b:cf:aa:b1:0a:c1:
     65:14:59:29:4f:94:93:b5:91:16:f1:22:5a:12:2a:ab:a4:59:
     33:f1:47:03:3f:03:b6:3a:ad:df:2a:90:ef:71:db:ef:5f:d7:
     e2:3a:4f:6d:1c:8f:76:e1:7c:5f:a0:bb:19:b1:83:c7:1f:b3:
     f0:40:f8:c6:66:38:74:be:07:e5:5d:8d:f9:25:ca:f0:d8:cd:
     fc:ad:35:1b:67:40:1b:91:54:57:53:16:e7:a3:e0:67:9c:4c:
     7f:ad:0c:11:27:9f:c6:f3:da:88:db:38:17:04:6b:29:ff:f4:
     a4:34:ea:55:27:8e:e2:49:b4:f1:75:63:78:60:3e:1b:cc:0a:
     f7:87:d1:6f:2e:66:a4:8b:a8:87:eb:b8:16:9b:1f:75:46:d8:
     d3:fd:9c:55:30:4a:11:9c:b7:a6:f6:85:62:f4:45:0c:4e:34:
     00:38:ef:16

``

···
    Subject: CN=admin001
     de:c2:5f:3f:26:36:e6:9e:19:0b:67:4f:71:3a:38:84:7a:de:

On Friday, November 24, 2017 at 1:37:28 PM UTC+1, Search Guard wrote:

Can you post (or mail) the output of

openssl x509 -in ~/admin001.crt -text -noout

Am 23.11.2017 um 12:09 schrieb calv...@gmail.com:

Thanks for the quick answer !

Not much success tho;

vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ sudo grep -e ‘^$’ -e ‘^#’ --invert-match /etc/elasticsearch/elasticsearch.yml

cluster.name: “elasticsearch”

node.name: “elasticsearchminion”

node.master: true

node.data: true

network.bind_host: 0.0.0.0

network.publish_host: 0.0.0.0

network.host: 0.0.0.0

http.port: 9201

discovery.zen.ping.unicast.hosts: [

]

cluster.routing.allocation.disk.threshold_enabled: true

cluster.routing.allocation.disk.watermark.low: 15gb

cluster.routing.allocation.disk.watermark.high: 5gb

searchguard.ssl.transport.pemkey_filepath: transport.key

searchguard.ssl.transport.pemcert_filepath: transport.cert

searchguard.ssl.transport.pemtrustedcas_filepath: transport.ca

searchguard.ssl.transport.enabled: true

searchguard.ssl.transport.keystore_type: x509

searchguard.ssl.http.pemkey_filepath: rest.key

searchguard.ssl.http.pemcert_filepath: rest.cert

searchguard.ssl.http.pemtrustedcas_filepath: rest.ca

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.http.enabled: true

searchguard.authcz.admin_dn:

  • CN=admin001

searchguard.nodes_dn:

  • ‘*’

searchguard.audit.type: com.payplug.auditlog.impl.StdoutAuditLog

searchguard.audit.enable_request_details: true

searchguard.audit.ignore_users:

  • kibanaserver

vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ sudo service elasticsearch restart
vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ sudo ./sgadmin.sh -cert ~/admin001.crt -cacert /etc/elasticsearch/rest.ca -key ~/admin001.key --disable-host-name-verification --diagnose -icl

WARNING: JAVA_HOME not set, will use /usr/bin/java

Search Guard Admin v5
Will connect to localhost:9300 … done

LICENSE NOTICE Search Guard

If you use one or more of the following features in production

make sure you have a valid Search Guard license

(See https://floragunn.com/searchguard-validate-license)

  • Kibana Multitenancy
  • LDAP authentication/authorization
  • Active Directory authentication/authorization
  • REST Management API
  • JSON Web Token (JWT) authentication/authorization
  • Kerberos authentication/authorization
  • Document- and Fieldlevel Security (DLS/FLS)
  • Auditlogging

In case of any doubt mail to sa...@floragunn.com

###################################

Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-5/tools/sgadmin_diag_trace_2017-Nov-23_10-52-14.txt

Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{dhWeO-uaT4KaitGsjFknKA}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying …

Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{dhWeO-uaT4KaitGsjFknKA}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)

  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
  • Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{dhWeO-uaT4KaitGsjFknKA}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying …

Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{dhWeO-uaT4KaitGsjFknKA}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)

  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
  • Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.

Clustername: elasticsearch

Clusterstate: GREEN

Number of nodes: 1

Number of data nodes: 1

ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md

Trace:

ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]

    at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)                                                                                                                                                                                  
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
    at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
    at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
    at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
    at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
    at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
    at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
    at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1553)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
    at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
    at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1510)
    at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1393)
    at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1273)
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1084)
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
    at java.lang.Thread.run(Thread.java:748)

The log on the Elasticsearch side are the same, final entry is still

[ERROR][c.f.s.f.SearchGuardFilter] Search Guard not initialized (SG11) for indices:admin/exists

On Thursday, November 23, 2017 at 11:09:10 AM UTC+1, Search Guard wrote:

searchguard.authcz.admin_dn can not be a wildcard

Am 23.11.2017 um 10:39 schrieb calv...@gmail.com:

Hi there,

I’m trying to setup searchguard but sgadmin fails with the following output

vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ sudo ./sgadmin.sh -cert ~/admin001.crt -cacert /etc/elasticsearch/rest.ca -key ~/admin001.key --disable-host-name-verification --diagnose -icl
WARNING: JAVA_HOME not set, will use /usr/bin/java

Search Guard Admin v5

Will connect to localhost:9300 … done

LICENSE NOTICE Search Guard

If you use one or more of the following features in production
make sure you have a valid Search Guard license
(See https://floragunn.com/searchguard-validate-license)

  • Kibana Multitenancy
  • LDAP authentication/authorization
  • Active Directory authentication/authorization
  • REST Management API
  • JSON Web Token (JWT) authentication/authorization
  • Kerberos authentication/authorization
  • Document- and Fieldlevel Security (DLS/FLS)
  • Auditlogging

In case of any doubt mail to sa...@floragunn.com
###################################
Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-5/tools/sgadmin_diag_trace_2017-Nov-22_17-10-58.txt
Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying …

Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)

  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
  • Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
    Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying …
    Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.

seems like "Key Encipherment" and "TLS Web Server Authentication" as X509v3 Key Usage is missing

should look like

X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Basic Constraints:
                CA:FALSE
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication

···

Am 24.11.2017 um 14:24 schrieb calvinh34@gmail.com:

Here you are. Maybe it's missing an ExtendedKeyUsage attribute ?

vagrant@elasticsearch:…$ openssl x509 -in ~/admin001.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 6 (0x6)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=mycompany test ca
        Validity
            Not Before: Nov 22 15:15:31 2017 GMT
            Not After : Nov 20 15:15:31 2027 GMT
        Subject: CN=admin001
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:f2:7c:8a:fd:5b:d2:1e:1e:01:52:32:9f:ae:57:
                    fd:c1:8c:94:52:dd:e7:3a:2f:8c:3f:71:44:ab:81:
                    79:37:64:08:d5:76:a8:36:be:29:60:27:13:fd:23:
                    92:db:bb:f9:de:cc:3e:88:c5:7d:69:e3:48:ca:0b:
                    3d:8e:d1:81:73:7a:14:05:95:a0:95:8b:70:ef:d5:
                    65:81:01:57:39:45:fa:c2:28:81:52:f2:4f:de:fd:
                    38:1a:f1:11:e6:9c:36:6a:51:3a:b8:5a:b1:51:c1:
                    04:3d:fe:b1:55:24:32:a6:3f:f3:83:7b:e4:77:1c:
                    45:03:49:9f:ac:e2:dc:5f:f5:8a:34:ac:3b:c2:73:
                    a3:70:5a:63:e5:32:4a:b4:99:4a:53:1c:9d:10:dd:
                    6c:ba:72:88:86:29:c7:da:7c:5a:60:ed:d8:74:cd:
                    0f:47:d8:b3:6f:be:75:25:fa:5d:23:43:fd:2c:c3:
                    b7:74:57:17:e1:04:76:6f:b9:82:08:c5:af:2b:ce:
                    f5:14:d2:4c:02:f6:47:f3:0b:2a:c9:80:4a:fd:23:
                    be:be:00:3c:4d:af:ff:b5:65:24:fb:49:d5:20:24:
                    d4:4a:26:cc:c2:71:30:94:31:68:78:7b:8b:df:d0:
                    e8:f8:eb:34:d6:ba:1c:e6:95:9a:54:f3:0c:29:2b:
                    6f:2f
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            X509v3 Subject Key Identifier:
                0B:8F:E0:5C:5C:02:36:C7:37:8B:17:90:0E:D8:04:D9:C8:25:29:11
            X509v3 Authority Key Identifier:
                keyid:74:32:94:50:67:DF:4C:95:03:18:D0:51:08:A6:50:14:E0:8A:42:C8
                DirName:/CN=mycompany test ca
                serial:A7:CD:62:39:B3:FF:48:76

            X509v3 Extended Key Usage:
                TLS Web Client Authentication
            X509v3 Key Usage:
                Digital Signature
    Signature Algorithm: sha256WithRSAEncryption
         a1:97:f8:e7:19:9a:18:40:af:a1:91:7d:35:14:34:2a:1a:14:
         a4:02:ed:65:27:26:00:be:02:37:dc:4e:b2:27:16:4d:06:a7:
         de:c2:5f:3f:26:36:e6:9e:19:0b:67:4f:71:3a:38:84:7a:de:
         4a:00:44:ec:02:43:9b:8c:ae:81:6b:84:34:64:1d:1b:85:ff:
         6d:ab:0e:cd:a1:43:92:15:fb:7e:6b:0e:9b:cf:aa:b1:0a:c1:
         65:14:59:29:4f:94:93:b5:91:16:f1:22:5a:12:2a:ab:a4:59:
         33:f1:47:03:3f:03:b6:3a:ad:df:2a:90:ef:71:db:ef:5f:d7:
         e2:3a:4f:6d:1c:8f:76:e1:7c:5f:a0:bb:19:b1:83:c7:1f:b3:
         f0:40:f8:c6:66:38:74:be:07:e5:5d:8d:f9:25:ca:f0:d8:cd:
         fc:ad:35:1b:67:40:1b:91:54:57:53:16:e7:a3:e0:67:9c:4c:
         7f:ad:0c:11:27:9f:c6:f3:da:88:db:38:17:04:6b:29:ff:f4:
         a4:34:ea:55:27:8e:e2:49:b4:f1:75:63:78:60:3e:1b:cc:0a:
         f7:87:d1:6f:2e:66:a4:8b:a8:87:eb:b8:16:9b:1f:75:46:d8:
         d3:fd:9c:55:30:4a:11:9c:b7:a6:f6:85:62:f4:45:0c:4e:34:
         00:38:ef:16

On Friday, November 24, 2017 at 1:37:28 PM UTC+1, Search Guard wrote:
Can you post (or mail) the output of

openssl x509 -in ~/admin001.crt -text -noout

> Am 23.11.2017 um 12:09 schrieb calv...@gmail.com:
>
> Thanks for the quick answer !
>
> Not much success tho;
>
> vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ sudo grep -e '^&#39; \-e &#39;^\#&#39; \-\-invert\-match /etc/elasticsearch/elasticsearch\.yml &gt; cluster\.name: &quot;elasticsearch&quot; &gt; node\.name: &quot;elasticsearchminion&quot; &gt; node\.master: true &gt; node\.data: true &gt; network\.bind\_host: 0\.0\.0\.0 &gt; network\.publish\_host: 0\.0\.0\.0 &gt; network\.host: 0\.0\.0\.0 &gt; http\.port: 9201 &gt; discovery\.zen\.ping\.unicast\.hosts: \[ &gt; \] &gt; cluster\.routing\.allocation\.disk\.threshold\_enabled: true &gt; cluster\.routing\.allocation\.disk\.watermark\.low: 15gb &gt; cluster\.routing\.allocation\.disk\.watermark\.high: 5gb &gt; searchguard\.ssl\.transport\.pemkey\_filepath: transport\.key &gt; searchguard\.ssl\.transport\.pemcert\_filepath: transport\.cert &gt; searchguard\.ssl\.transport\.pemtrustedcas\_filepath: transport\.ca &gt; searchguard\.ssl\.transport\.enabled: true &gt; searchguard\.ssl\.transport\.keystore\_type: x509 &gt; searchguard\.ssl\.http\.pemkey\_filepath: rest\.key &gt; searchguard\.ssl\.http\.pemcert\_filepath: rest\.cert &gt; searchguard\.ssl\.http\.pemtrustedcas\_filepath: rest\.ca &gt; searchguard\.ssl\.transport\.enforce\_hostname\_verification: false &gt; searchguard\.ssl\.http\.enabled: true &gt; searchguard\.authcz\.admin\_dn: &gt; \- CN=admin001 &gt; searchguard\.nodes\_dn: &gt; \- &#39;\*&#39; &gt; searchguard\.audit\.type: com\.payplug\.auditlog\.impl\.StdoutAuditLog &gt; searchguard\.audit\.enable\_request\_details: true &gt; searchguard\.audit\.ignore\_users: &gt; \- kibanaserver &gt; vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search\-guard\-5/tools sudo service elasticsearch restart
> vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ sudo ./sgadmin.sh -cert ~/admin001.crt -cacert /etc/elasticsearch/rest.ca -key ~/admin001.key --disable-host-name-verification --diagnose -icl
> WARNING: JAVA_HOME not set, will use /usr/bin/java
> Search Guard Admin v5
> Will connect to localhost:9300 ... done
>
> ### LICENSE NOTICE Search Guard ###
>
> If you use one or more of the following features in production
> make sure you have a valid Search Guard license
> (See https://floragunn.com/searchguard-validate-license)
>
> * Kibana Multitenancy
> * LDAP authentication/authorization
> * Active Directory authentication/authorization
> * REST Management API
> * JSON Web Token (JWT) authentication/authorization
> * Kerberos authentication/authorization
> * Document- and Fieldlevel Security (DLS/FLS)
> * Auditlogging
>
> In case of any doubt mail to <sa...@floragunn.com>
> ###################################
> Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-5/tools/sgadmin_diag_trace_2017-Nov-23_10-52-14.txt
> Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
> Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{dhWeO-uaT4KaitGsjFknKA}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying ...
> Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{dhWeO-uaT4KaitGsjFknKA}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)
> * Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
> * Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
> * If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
> * Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
> Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{dhWeO-uaT4KaitGsjFknKA}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying ...
> Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{dhWeO-uaT4KaitGsjFknKA}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)
> * Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
> * Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
> * If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
> * Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
> Clustername: elasticsearch
> Clusterstate: GREEN
> Number of nodes: 1
> Number of data nodes: 1
> ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
> Trace:
> ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
> at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
> at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
> at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
> at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
> at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
> at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
> at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
> at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
> at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
> at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
> at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1553)
> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
> at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
> at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1510)
> at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1393)
> at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
> at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
> at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
> at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1273)
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1084)
> at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
> at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
> at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
> at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
> at java.lang.Thread.run(Thread.java:748)
>
>
> The log on the Elasticsearch side are the same, final entry is still
>
> [ERROR][c.f.s.f.SearchGuardFilter] Search Guard not initialized (SG11) for indices:admin/exists
>
>
>
> On Thursday, November 23, 2017 at 11:09:10 AM UTC+1, Search Guard wrote:
> searchguard.authcz.admin_dn can not be a wildcard
>
> > Am 23.11.2017 um 10:39 schrieb calv...@gmail.com:
> >
> > Hi there,
> >
> > I'm trying to setup searchguard but sgadmin fails with the following output
> >
> > vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ sudo ./sgadmin.sh -cert ~/admin001.crt -cacert /etc/elasticsearch/rest.ca -key ~/admin001.key --disable-host-name-verification --diagnose -icl
> > WARNING: JAVA_HOME not set, will use /usr/bin/java
> >
> > Search Guard Admin v5
> >
> > Will connect to localhost:9300 ... done
> >
> >
> > ### LICENSE NOTICE Search Guard ###
> >
> > If you use one or more of the following features in production
> > make sure you have a valid Search Guard license
> > (See https://floragunn.com/searchguard-validate-license)
> >
> > * Kibana Multitenancy
> > * LDAP authentication/authorization
> > * Active Directory authentication/authorization
> > * REST Management API
> > * JSON Web Token (JWT) authentication/authorization
> > * Kerberos authentication/authorization
> > * Document- and Fieldlevel Security (DLS/FLS)
> > * Auditlogging
> >
> > In case of any doubt mail to <sa...@floragunn.com>
> > ###################################
> > Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-5/tools/sgadmin_diag_trace_2017-Nov-22_17-10-58.txt
> > Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
> >
> > Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying ...
> >
> > Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)
> > * Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
> > * Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
> > * If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
> > * Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
> > Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying ...
> > Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)
> > * Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
> > * Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
> > * If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
> > * Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
> > Clustername: elasticsearch
> > Clusterstate: GREEN
> > Number of nodes: 1
> > Number of data nodes: 1
> > ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
> > Trace:
> > ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
> > at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
> > at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
> > at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
> > at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
> > at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
> > at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
> > at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
> > at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
> > at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
> > at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
> > at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1553)
> > at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
> > at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
> > at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1510)
> > at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1393)
> > at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
> > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> > at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
> > at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
> > at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
> > at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
> > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> > at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> > at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)
> > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> > at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1273)
> > at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1084)
> > at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
> > at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
> > at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
> > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> > at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
> > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> > at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
> > at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
> > at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
> > at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
> > at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
> > at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
> > at java.lang.Thread.run(Thread.java:748)
> >
> >
> >
> >
> > Meanwhile, in the elasticsearch logs, we have (traceback redacted for readability);
> >
> > Nov 22 17:10:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.ConfigurationLoader] Failure no such index retrieving configuration for [roles] (index=searchguard)
> > Nov 22 17:11:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.IndexBaseConfigurationRepository] Unable to load configuration because of java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for [roles](index=searchguard)
> > Nov 22 17:11:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.f.SearchGuardFilter] Search Guard not initialized (SG11) for indices:monitor/stats
> > Nov 22 17:12:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.ConfigurationLoader] Failure no such index retrieving configuration for [roles] (index=searchguard)
> > Nov 22 17:13:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.IndexBaseConfigurationRepository] Unable to load configuration because of java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for [roles](index=searchguard)
> > Nov 22 17:13:04 elasticsearch elasticsearch[1180]: java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for [roles](index=searchguard)
> > Nov 22 17:13:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.f.SearchGuardFilter] Search Guard not initialized (SG11) for indices:admin/exists
> >
> > Here we show that the admin certificate is indeed valid against the specified CA
> >
> > vagrant@elasticsearch:~ openssl x509 \-noout \-text \-in \~/admin001\.crt | grep Subject: &gt; &gt; Subject: CN=admin001 &gt; &gt; vagrant@elasticsearch:\~ sudo openssl verify -CAfile /etc/elasticsearch/rest.ca ~/admin001.crt
> > /home/vagrant/admin001.crt: OK
> >
> >
> >
> >
> > Here is the elasticsearch.yml configuration
> >
> >
> > vagrant@elasticsearch:~ sudo grep \-e &#39;^' -e '^#' --invert-match /etc/elasticsearch/elasticsearch.yml
> > cluster.name: "elasticsearch"
> > node.name: "elasticsearchminion"
> > node.master: true
> > node.data: true
> > network.bind_host: 0.0.0.0
> > network.publish_host: 0.0.0.0
> > network.host: 0.0.0.0
> > http.port: 9201
> > discovery.zen.ping.unicast.hosts: [
> > ]
> > cluster.routing.allocation.disk.threshold_enabled: true
> > cluster.routing.allocation.disk.watermark.low: 15gb
> > cluster.routing.allocation.disk.watermark.high: 5gb
> > searchguard.ssl.transport.pemkey_filepath: transport.key
> > searchguard.ssl.transport.pemcert_filepath: transport.cert
> > searchguard.ssl.transport.pemtrustedcas_filepath: transport.ca
> > searchguard.ssl.transport.enabled: true
> > searchguard.ssl.transport.keystore_type: x509
> > searchguard.ssl.http.pemkey_filepath: rest.key
> > searchguard.ssl.http.pemcert_filepath: rest.cert
> > searchguard.ssl.http.pemtrustedcas_filepath: rest.ca
> > searchguard.ssl.transport.enforce_hostname_verification: false
> > searchguard.ssl.http.enabled: true
> > searchguard.authcz.admin_dn:
> > - '*'
> > searchguard.nodes_dn:
> > - '*'
> > searchguard.audit.type: com.payplug.auditlog.impl.StdoutAuditLog
> > searchguard.audit.enable_request_details: true
> > searchguard.audit.ignore_users:
> > - kibanaserver
> >
> >
> >
> > What am I missing here ? For what it's worth, all cryptographic material have been generated using openssl/easyrsa (I'm planning on documenting it).
> > Please let me know if I can provide any extra informations.
> >
> > Arthur
> >
> > --
> > You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> > To post to this group, send email to search...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/166328f7-872a-4aec-804a-27187b477b97%40googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/1e926131-b185-495d-a894-27eeb6c66603%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/1012f7f6-3a55-4333-aa47-7d56cfbda0f4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Still no luck, we can see the certificate now have the KeyEncipherment and WebServerAuth attributes. Also I updated elasticsearch.yml to match the admin002 CN;

vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ openssl x509 -in ~/admin002.crt -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 7 (0x7)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=company test ca
Validity
Not Before: Nov 24 14:13:35 2017 GMT
Not After : Nov 22 14:13:35 2027 GMT
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:be:95:33:6f:a8:ac:b1:4f:17:68:26:92:ec:45:
74:9c:eb:17:6e:b3:eb:aa:47:51:62:be:6a:6e:cd:
63:cf:6b:38:5b:56:e2:45:09:f9:77:bd:00:00:1e:
10:99:8b:9e:01:89:1b:20:80:ae:b8:a3:ca:33:6c:
43:97:b9:1c:39:a6:4f:fb:4e:4e:8b:91:68:4a:0e:
52:42:fe:d9:1c:9a:5b:ba:6a:8f:ad:23:af:a0:f5:
ed:57:e2:3e:a2:97:ec:dc:9e:91:00:ef:04:b2:bd:
ec:b5:28:89:7f:3c:7f:e1:4d:5a:b3:f3:d8:ec:8a:
db:54:32:67:67:b1:57:45:30:48:9a:10:96:ed:31:
37:9c:73:62:d8:b2:8e:26:99:dc:d2:53:29:62:ee:
3f:68:e3:ff:e0:8d:e6:d1:77:d6:99:64:2e:81:9d:
ba:a3:c1:66:82:57:b2:75:bc:83:22:4e:94:45:2d:
e9:c4:c4:c6:a8:38:7f:21:28:5c:c5:a2:77:40:70:
2b:47:ed:1f:3b:74:60:4d:52:08:92:46:7b:c6:4d:
44:2d:c9:f5:ee:a0:95:c0:bb:2c:ae:41:e1:6c:3e:
74:bd:49:34:a9:00:9d:d0:b9:7b:d4:05:01:cc:a6:
9c:1d:0f:95:80:4e:87:97:f3:7d:9e:7d:4a:fc:2b:
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
X509v3 Subject Key Identifier:
F9:6F:FE:01:F4:32:14:36:9A:83:1E:47:09:72:FD:59:95:6C:AA:64
X509v3 Authority Key Identifier:
keyid:74:32:94:50:67:DF:4C:95:03:18:D0:51:08:A6:50:14:E0:8A:42:C8
DirName:/CN=company test ca
serial:A7:CD:62:39:B3:FF:48:76
X509v3 Extended Key Usage:
TLS Web Client Authentication, TLS Web Server Authentication
X509v3 Key Usage:
Digital Signature, Key Encipherment
Signature Algorithm: sha256WithRSAEncryption
95:d5:dc:75:57:83:a6:e0:89:93:5d:b7:e7:6a:51:75:83:5a:
be:e5:ce:16:48:47:45:1e:6e:c5:e0:86:ce:d5:58:3e:20:f9:
8d:b4:b2:5e:d6:8b:a0:49:94:9e:77:c1:20:64:d7:da:a9:4c:
f8:98:2e:44:ed:1f:b9:88:56:88:a9:eb:f3:13:34:04:cf:0c:
2e:89:c5:be:25:15:e2:dd:bd:4c:66:d5:d6:df:9d:cc:5c:73:
f4:63:0e:2f:dd:7d:24:da:0e:2b:5d:6d:9a:30:9b:e0:11:d9:
34:17:d1:14:44:4e:9e:b0:7f:46:87:2b:c7:25:20:a1:3e:fb:
f2:de:38:3d:42:cc:eb:35:48:30:b3:60:6a:ff:23:fd:f0:cb:
59:a7:e1:f8:89:fd:a5:52:44:95:d2:ab:a5:fd:75:df:9e:4c:
a8:a8:8b:c2:0a:12:1c:17:aa:f4:84:91:54:4d:37:92:eb:4b:
11:9c:0a:a6:e1:56:ed:03:aa:16:4e:66:1c:ae:10:4b:9a:4d:
a0:ab:a7:21:61:5a:c8:cc:b3:a9:6b:53:35:7e:70:d9:97:a5:
3d:ac:b9:cd:66:aa:92:2d:8c:70:73:b5:fe:9a:5b:ba:33:4c:
65:27:3f:34:ec:2e:80:ce:f2:25:f8:e2:d7:3b:09:6f:d7:95:
47:48:77:fe
vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ sudo ./sgadmin.sh -cert ~/admin002.crt -cacert /etc/elasticsearch/rest.ca -key ~/admin002.key --disable-host-name-verification --diagnose -icl
WARNING: JAVA_HOME not set, will use /usr/bin/java
Search Guard Admin v5
Will connect to localhost:9300 … done

LICENSE NOTICE Search Guard

If you use one or more of the following features in production
make sure you have a valid Search Guard license
(See https://floragunn.com/searchguard-validate-license)

  • Kibana Multitenancy
  • LDAP authentication/authorization
  • Active Directory authentication/authorization
  • REST Management API
  • JSON Web Token (JWT) authentication/authorization
  • Kerberos authentication/authorization
  • Document- and Fieldlevel Security (DLS/FLS)
  • Auditlogging

In case of any doubt mail to sales@floragunn.com

···
    Subject: CN=admin002                                                                                                                                                    
                cc:f7

###################################
Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-5/tools/sgadmin_diag_trace_2017-Nov-24_14-28-20.txt
Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …
Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{skJ0nA_FQNyFDAGkW0PS_g}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying …
Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{skJ0nA_FQNyFDAGkW0PS_g}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)

  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
  • Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
    Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{skJ0nA_FQNyFDAGkW0PS_g}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying …
    Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{skJ0nA_FQNyFDAGkW0PS_g}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)
  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
  • Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
    Clustername: elasticsearch
    Clusterstate: GREEN
    Number of nodes: 1
    Number of data nodes: 1
    ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
    Trace:
    ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
    at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
    at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
    at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
    at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
    at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
    at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
    at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
    at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1553)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
    at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
    at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1510)
    at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1393)
    at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1273)
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1084)
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
    at java.lang.Thread.run(Thread.java:748)

``

Any way to increase the debug output ?

Hi,

what can I do to move on this issue ?

Arthur

···

On Thursday, November 23, 2017 at 10:39:09 AM UTC+1, calv...@gmail.com wrote:

Hi there,

I’m trying to setup searchguard but sgadmin fails with the following output

vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ sudo ./sgadmin.sh -cert ~/admin001.crt -cacert /etc/elasticsearch/rest.ca -key ~/admin001.key --disable-host-name-verification --diagnose -icl
WARNING: JAVA_HOME not set, will use /usr/bin/java

Search Guard Admin v5

Will connect to localhost:9300 … done

LICENSE NOTICE Search Guard

If you use one or more of the following features in production
make sure you have a valid Search Guard license
(See https://floragunn.com/searchguard-validate-license)

  • Kibana Multitenancy
  • LDAP authentication/authorization
  • Active Directory authentication/authorization
  • REST Management API
  • JSON Web Token (JWT) authentication/authorization
  • Kerberos authentication/authorization
  • Document- and Fieldlevel Security (DLS/FLS)
  • Auditlogging

In case of any doubt mail to sales@floragunn.com
###################################
Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-5/tools/sgadmin_diag_trace_2017-Nov-22_17-10-58.txt
Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying …

Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)

  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
  • Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
    Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying …
    Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)
  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
  • Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
    Clustername: elasticsearch
    Clusterstate: GREEN
    Number of nodes: 1
    Number of data nodes: 1
    ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
    Trace:
    ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
    at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
    at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
    at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
    at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
    at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
    at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
    at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
    at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1553)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
    at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
    at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1510)
    at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1393)
    at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1273)
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1084)
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
    at java.lang.Thread.run(Thread.java:748)

``

Meanwhile, in the elasticsearch logs, we have (traceback redacted for readability);

Nov 22 17:10:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.ConfigurationLoader] Failure no such index retrieving configuration for [roles] (index=searchguard)
Nov 22 17:11:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.IndexBaseConfigurationRepository] Unable to load configuration because of java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for roles
Nov 22 17:11:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.f.SearchGuardFilter] Search Guard not initialized (SG11) for indices:monitor/stats
Nov 22 17:12:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.ConfigurationLoader] Failure no such index retrieving configuration for [roles] (index=searchguard)
Nov 22 17:13:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.IndexBaseConfigurationRepository] Unable to load configuration because of java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for roles
Nov 22 17:13:04 elasticsearch elasticsearch[1180]: java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for roles
Nov 22 17:13:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.f.SearchGuardFilter] Search Guard not initialized (SG11) for indices:admin/exists

``

Here we show that the admin certificate is indeed valid against the specified CA

vagrant@elasticsearch:~ openssl x509 -noout -text -in ~/admin001.crt | grep Subject: Subject: CN=admin001 vagrant@elasticsearch:~ sudo openssl verify -CAfile /etc/elasticsearch/rest.ca ~/admin001.crt
/home/vagrant/admin001.crt: OK

``

Here is the elasticsearch.yml configuration

vagrant@elasticsearch:~ sudo grep -e '^’ -e ‘^#’ --invert-match /etc/elasticsearch/elasticsearch.yml
cluster.name: “elasticsearch”
node.name: “elasticsearchminion”
node.master: true
node.data: true
network.bind_host: 0.0.0.0
network.publish_host: 0.0.0.0
network.host: 0.0.0.0
http.port: 9201
discovery.zen.ping.unicast.hosts: [
]
cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.low: 15gb
cluster.routing.allocation.disk.watermark.high: 5gb
searchguard.ssl.transport.pemkey_filepath: transport.key
searchguard.ssl.transport.pemcert_filepath: transport.cert
searchguard.ssl.transport.pemtrustedcas_filepath: transport.ca
searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.keystore_type: x509
searchguard.ssl.http.pemkey_filepath: rest.key
searchguard.ssl.http.pemcert_filepath: rest.cert
searchguard.ssl.http.pemtrustedcas_filepath: rest.ca
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.authcz.admin_dn:

  • ‘*’
    searchguard.nodes_dn:
  • ‘*’
    searchguard.audit.type: com.payplug.auditlog.impl.StdoutAuditLog
    searchguard.audit.enable_request_details: true
    searchguard.audit.ignore_users:
  • kibanaserver

``

What am I missing here ? For what it’s worth, all cryptographic material have been generated using openssl/easyrsa (I’m planning on documenting it).
Please let me know if I can provide any extra informations.

Arthur

can you pls mail or attach the certificates?

···

On Tuesday, 5 December 2017 18:46:02 UTC+1, cal…@g…m wrote:

Hi,

what can I do to move on this issue ?

Arthur

On Thursday, November 23, 2017 at 10:39:09 AM UTC+1, calv...@gmail.com wrote:

Hi there,

I’m trying to setup searchguard but sgadmin fails with the following output

vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ sudo ./sgadmin.sh -cert ~/admin001.crt -cacert /etc/elasticsearch/rest.ca -key ~/admin001.key --disable-host-name-verification --diagnose -icl
WARNING: JAVA_HOME not set, will use /usr/bin/java

Search Guard Admin v5

Will connect to localhost:9300 … done

LICENSE NOTICE Search Guard

If you use one or more of the following features in production
make sure you have a valid Search Guard license
(See https://floragunn.com/searchguard-validate-license)

  • Kibana Multitenancy
  • LDAP authentication/authorization
  • Active Directory authentication/authorization
  • REST Management API
  • JSON Web Token (JWT) authentication/authorization
  • Kerberos authentication/authorization
  • Document- and Fieldlevel Security (DLS/FLS)
  • Auditlogging

In case of any doubt mail to sales@floragunn.com
###################################
Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-5/tools/sgadmin_diag_trace_2017-Nov-22_17-10-58.txt
Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying …

Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)

  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
  • Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
    Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying …
    Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)
  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
  • Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
    Clustername: elasticsearch
    Clusterstate: GREEN
    Number of nodes: 1
    Number of data nodes: 1
    ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
    Trace:
    ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
    at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
    at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
    at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
    at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
    at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
    at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
    at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
    at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1553)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
    at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
    at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1510)
    at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1393)
    at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1273)
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1084)
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
    at java.lang.Thread.run(Thread.java:748)

``

Meanwhile, in the elasticsearch logs, we have (traceback redacted for readability);

Nov 22 17:10:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.ConfigurationLoader] Failure no such index retrieving configuration for [roles] (index=searchguard)
Nov 22 17:11:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.IndexBaseConfigurationRepository] Unable to load configuration because of java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for roles
Nov 22 17:11:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.f.SearchGuardFilter] Search Guard not initialized (SG11) for indices:monitor/stats
Nov 22 17:12:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.ConfigurationLoader] Failure no such index retrieving configuration for [roles] (index=searchguard)
Nov 22 17:13:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.IndexBaseConfigurationRepository] Unable to load configuration because of java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for roles
Nov 22 17:13:04 elasticsearch elasticsearch[1180]: java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for roles
Nov 22 17:13:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.f.SearchGuardFilter] Search Guard not initialized (SG11) for indices:admin/exists

``

Here we show that the admin certificate is indeed valid against the specified CA

vagrant@elasticsearch:~ openssl x509 -noout -text -in ~/admin001.crt | grep Subject: Subject: CN=admin001 vagrant@elasticsearch:~ sudo openssl verify -CAfile /etc/elasticsearch/rest.ca ~/admin001.crt
/home/vagrant/admin001.crt: OK

``

Here is the elasticsearch.yml configuration

vagrant@elasticsearch:~ sudo grep -e '^’ -e ‘^#’ --invert-match /etc/elasticsearch/elasticsearch.yml
cluster.name: “elasticsearch”
node.name: “elasticsearchminion”
node.master: true
node.data: true
network.bind_host: 0.0.0.0
network.publish_host: 0.0.0.0
network.host: 0.0.0.0
http.port: 9201
discovery.zen.ping.unicast.hosts: [
]
cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.low: 15gb
cluster.routing.allocation.disk.watermark.high: 5gb
searchguard.ssl.transport.pemkey_filepath: transport.key
searchguard.ssl.transport.pemcert_filepath: transport.cert
searchguard.ssl.transport.pemtrustedcas_filepath: transport.ca
searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.keystore_type: x509
searchguard.ssl.http.pemkey_filepath: rest.key
searchguard.ssl.http.pemcert_filepath: rest.cert
searchguard.ssl.http.pemtrustedcas_filepath: rest.ca
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.authcz.admin_dn:

  • ‘*’
    searchguard.nodes_dn:
  • ‘*’
    searchguard.audit.type: com.payplug.auditlog.impl.StdoutAuditLog
    searchguard.audit.enable_request_details: true
    searchguard.audit.ignore_users:
  • kibanaserver

``

What am I missing here ? For what it’s worth, all cryptographic material have been generated using openssl/easyrsa (I’m planning on documenting it).
Please let me know if I can provide any extra informations.

Arthur

I am also facing the same issue, may I know what is the solution for this? I can’t share the PEM files for Policy reasons.

Error as per diagnose file:
ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:monitor/stats. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:122)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)

Regards
Anil