Cannot retrieve cluster state due to: No user found for cluster:monitor/health.

Search Guard
1

Been trying to upgrade to ES 6.1.1 and SG6

Was working fine on SG5 and ES5, upgraded to 6, cluster is fine and shows as green (at one point at least), before i deleted the searchguard index to try and start fresh my old users would login in kibana but no sgadmin…

  • Search Guard and Elasticsearch version

ES 6.1.1

SG6 6.1.1-20.1

  • Installed and used enterprise modules, if any

none

  • JVM version and operating system version

Ubuntu 16.04

openjdk version “1.8.0_151”

OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)

OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

  • Search Guard configuration files

tried with blank (the initial files) and my old ones, error is the same

  • Other installed Elasticsearch or Kibana plugins, if any

none

sudo ./sgadmin.sh -cd …/sgconfig/original/ -ks sgadmin.jks -kspass password -ksalias client -ts truststore.jks -tspass password -icl -nhnv -h 10.10.x.x

WARNING: JAVA_HOME not set, will use /usr/bin/java

Search Guard Admin v6

Will connect to 10.10.x.x:9300 … done

Connected as ---------------------------------------------------------

Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …

Cannot retrieve cluster state due to: No user found for cluster:monitor/health. This is not an error, will keep on trying …

Root cause: ElasticsearchSecurityException[No user found for cluster:monitor/health] (org.elasticsearch.ElasticsearchSecurityException/org.elasticsearch.ElasticsearchSecurityException)

  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)

  • Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml

  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)

  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.

2

can you pls post the output of

sudo ./sgadmin.sh -cd ../sgconfig/original/ -ks sgadmin.jks -kspass password -ksalias client -ts truststore.jks -tspass password -icl -nhnv -h 10.10.x.x -w

···

Am 17.01.2018 um 18:51 schrieb Fabio Rodrigues <fabiopbx@gmail.com>:

Been trying to upgrade to ES 6.1.1 and SG6

Was working fine on SG5 and ES5, upgraded to 6, cluster is fine and shows as green (at one point at least), before i deleted the searchguard index to try and start fresh my old users would login in kibana but no sgadmin...

* Search Guard and Elasticsearch version
ES 6.1.1
SG6 6.1.1-20.1

* Installed and used enterprise modules, if any
none

* JVM version and operating system version

Ubuntu 16.04

openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

* Search Guard configuration files

tried with blank (the initial files) and my old ones, error is the same

* Other installed Elasticsearch or Kibana plugins, if any
none

sudo ./sgadmin.sh -cd ../sgconfig/original/ -ks sgadmin.jks -kspass password -ksalias client -ts truststore.jks -tspass password -icl -nhnv -h 10.10.x.x

WARNING: JAVA_HOME not set, will use /usr/bin/java
Search Guard Admin v6
Will connect to 10.10.x.x:9300 ... done
Connected as ---------------------------------------------------------
Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
Cannot retrieve cluster state due to: No user found for cluster:monitor/health. This is not an error, will keep on trying ...

  Root cause: ElasticsearchSecurityException[No user found for cluster:monitor/health] (org.elasticsearch.ElasticsearchSecurityException/org.elasticsearch.ElasticsearchSecurityException)

   * Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
   * If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow sgadmin to operate on a red cluster.

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/8a764afc-7b60-44c1-840c-462568cec8bc%40googlegroups.com\.
For more options, visit https://groups.google.com/d/optout\.

3

These are the same certificates I created for SG5.

Search Guard Admin v6

Will connect to 10.10.x.x:9300 … done

Connected as EMAILADDRESS=ca@company.com,CN=sgadmin,OU=company,O=company Ltd,L=London,ST=England,C=UK

{

“whoami” : {

“dn” : “EMAILADDRESS=ca@company.com,CN=sgadmin,OU=company,O=company Ltd,L=London,ST=England,C=UK”,

“is_admin” : true,

“is_authenticated” : true,

“is_node_certificate_request” : true

}

}

4

SG6 is more strict about certificates than SG5.

Semms that you admin certificate is also a node certificate. Thats no longer permitted.
Did you encode the OID into the admin cert or do you use the nodes_dn property in elasticsearch.yml?

···

Am 18.01.2018 um 10:20 schrieb Fabio Rodrigues <fabiopbx@gmail.com>:

These are the same certificates I created for SG5.

Search Guard Admin v6
Will connect to 10.10.x.x:9300 ... done
Connected as EMAILADDRESS=ca@company.com,CN=sgadmin,OU=company,O=company Ltd,L=London,ST=England,C=UK
{
  "whoami" : {
    "dn" : "EMAILADDRESS=ca@company.com,CN=sgadmin,OU=company,O=company Ltd,L=London,ST=England,C=UK",
    "is_admin" : true,
    "is_authenticated" : true,
    "is_node_certificate_request" : true
  }
}

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/74026a2a-0b52-4e4e-8058-4bd789df6018%40googlegroups.com\.
For more options, visit https://groups.google.com/d/optout\.

5

I do remember encoding the OID into the admin cert and in elasticsearch i only have the

searchguard.authcz.admin_dn:

6

hen you need to recreate this certificate without an OID

···

Am 18.01.2018 um 11:37 schrieb Fabio Rodrigues <fabiopbx@gmail.com>:

I do remember encoding the OID into the admin cert and in elasticsearch i only have the

searchguard.authcz.admin_dn:
  - EMAILADDRESS=ca@company.com,CN=sgadmin,OU=company,O=company Ltd,L=London,ST=England,C=UK

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/ea90e9f7-355c-4f9b-9efa-dfcb0a15163a%40googlegroups.com\.
For more options, visit https://groups.google.com/d/optout\.

7

I see, ill give that a try, its nice that one can use the PEMs straight up tho!

Ill report back when i try.

8

was just looking at my cmds file, and the ID i added was this

subjectAltName = RID:1.2.3.4.5.5

as per openssl.conf

[ es_server_cert ]

Extensions for server certificates (man x509v3_config).

basicConstraints = CA:FALSE

subjectKeyIdentifier = hash

subjectAltName = RID:1.2.3.4.5.5

authorityKeyIdentifier = keyid,issuer:always

keyUsage = critical, digitalSignature, keyEncipherment

extendedKeyUsage = serverAuth,clientAuth

so indeed it seems i used the server (node) config for the sgadmin, tho i did have this note on my file:

sgadmin # after hours of trying, it turns out sgadmin also NEEDS “TLS Server Auth” -.- go figure

which was me trying to use, but never worked…

[ es_usr_cert ]

Extensions for client certificates (man x509v3_config).

basicConstraints = CA:FALSE

subjectKeyIdentifier = hash

authorityKeyIdentifier = keyid,issuer

keyUsage = critical, digitalSignature, keyEncipherment

extendedKeyUsage = clientAuth

what say you? use the client config or the server one without the RID, its been a while, cant remember what the RID was for :frowning:

9

see https://github.com/floragunncom/search-guard-ssl/tree/master/example-pki-scripts and Generating certificates | Security for Elasticsearch | Search Guard
You can also generate certificates here: Security and Alerting for Elasticsearch and Kibana | Search Guard

The sgadmin certificate must not have the RID and extendedKeyUsage = serverAuth,clientAuth, the dn of this certificate must be registered in elasticsearch.yml
The node certificates need to have the RID and extendedKeyUsage = serverAuth,clientAuth
Normal client certificate must not have the RID and extendedKeyUsage = clientAuth

Hope this helps

···

Am 18.01.2018 um 13:43 schrieb Fabio Rodrigues <fabiopbx@gmail.com>:

was just looking at my cmds file, and the ID i added was this

subjectAltName = RID:1.2.3.4.5.5

as per openssl.conf

[ es_server_cert ]
# Extensions for server certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
subjectKeyIdentifier = hash
subjectAltName = RID:1.2.3.4.5.5
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth,clientAuth

so indeed it seems i used the server (node) config for the sgadmin, tho i did have this note on my file:

# sgadmin # after hours of trying, it turns out sgadmin also NEEDS "TLS Server Auth" -.- go figure

which was me trying to use, but never worked....

[ es_usr_cert ]
# Extensions for client certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth

what say you? use the client config or the server one without the RID, its been a while, cant remember what the RID was for :frowning:

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/089631a4-816f-4c24-b05e-7a5a589c1d4a%40googlegroups.com\.
For more options, visit https://groups.google.com/d/optout\.

10

I will add this to the documentation backlog, seems we are not clear enough here.

···

On Thursday, January 18, 2018 at 2:09:31 PM UTC+1, Search Guard wrote:

see https://github.com/floragunncom/search-guard-ssl/tree/master/example-pki-scripts and http://docs.search-guard.com/latest/generating-tls-certificates

You can also generate certificates here: https://floragunn.com/tls-certificate-generator/

The sgadmin certificate must not have the RID and extendedKeyUsage = serverAuth,clientAuth, the dn of this certificate must be registered in elasticsearch.yml

The node certificates need to have the RID and extendedKeyUsage = serverAuth,clientAuth

Normal client certificate must not have the RID and extendedKeyUsage = clientAuth

Hope this helps

Am 18.01.2018 um 13:43 schrieb Fabio Rodrigues fabiopbx@gmail.com:

was just looking at my cmds file, and the ID i added was this

subjectAltName = RID:1.2.3.4.5.5

as per openssl.conf

[ es_server_cert ]

Extensions for server certificates (man x509v3_config).

basicConstraints = CA:FALSE

subjectKeyIdentifier = hash

subjectAltName = RID:1.2.3.4.5.5

authorityKeyIdentifier = keyid,issuer:always

keyUsage = critical, digitalSignature, keyEncipherment

extendedKeyUsage = serverAuth,clientAuth

so indeed it seems i used the server (node) config for the sgadmin, tho i did have this note on my file:

sgadmin # after hours of trying, it turns out sgadmin also NEEDS “TLS Server Auth” -.- go figure

which was me trying to use, but never worked…

[ es_usr_cert ]

Extensions for client certificates (man x509v3_config).

basicConstraints = CA:FALSE

subjectKeyIdentifier = hash

authorityKeyIdentifier = keyid,issuer

keyUsage = critical, digitalSignature, keyEncipherment

extendedKeyUsage = clientAuth

what say you? use the client config or the server one without the RID, its been a while, cant remember what the RID was for :frowning:

–
You received this message because you are subscribed to the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/089631a4-816f-4c24-b05e-7a5a589c1d4a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

11

i did have a look for changes that might break this, but didnt find any on this :open_mouth:

also recreated as above but now i get this…

i have tried encrypted PEM and encrypted PEM, with password and no password, the keys are RSA and have verified with openssl.

running sgadmin as per Examples | Security for Elasticsearch | Search Guard

sudo ./sgadmin.sh -cd …/sgconfig/original/ -cacert sgadmin-full.cert.pem -cert sgadmin.cert.pem -key sgadmin.NOkey.pem -icl -nhnv -h 10.10.x.x

WARNING: JAVA_HOME not set, will use /usr/bin/java
Search Guard Admin v6
Will connect to 10.10.x.x:9300 … done
13:23:52.062 [main] ERROR com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore - Your keystore or PEM does not contain a key. If you sepcified a key password try removing it. If you not sepcified a key password maybe you one because the key is password protected. Maybe you just confused keys and certificates.
ERR: An unexpected IllegalStateException occured: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
Trace:
java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:452)
at org.elasticsearch.plugins.PluginsService.(PluginsService.java:105)
at org.elasticsearch.client.transport.TransportClient.newPluginService(TransportClient.java:103)
at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:128)
at org.elasticsearch.client.transport.TransportClient.(TransportClient.java:251)
at com.floragunn.searchguard.tools.SearchGuardAdmin$TransportClientImpl.(SearchGuardAdmin.java:823)
at com.floragunn.searchguard.tools.SearchGuardAdmin.main0(SearchGuardAdmin.java:403)
at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:120)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:443)
… 7 more
Caused by: ElasticsearchSecurityException[Error while initializing transport SSL layer from PEM: java.lang.IllegalArgumentException: File does not contain valid private key: sgadmin.NOkey.pem]; nested: IllegalArgumentException[File does not contain valid private key: sgadmin.NOkey.pem]; nested: InvalidKeySpecException[Neither RSA, DSA nor EC worked]; nested: InvalidKeySpecException[java.security.InvalidKeyException: IOException : algid parse error, not a sequence]; nested: InvalidKeyException[IOException : algid parse error, not a sequence];
at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:292)
at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.(DefaultSearchGuardKeyStore.java:145)
at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.(SearchGuardSSLPlugin.java:192)
at com.floragunn.searchguard.SearchGuardPlugin.(SearchGuardPlugin.java:182)
… 12 more
Caused by: java.lang.IllegalArgumentException: File does not contain valid private key: sgadmin.NOkey.pem
at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:267)
at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:90)
at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.buildSSLServerContext(DefaultSearchGuardKeyStore.java:613)
at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:287)
… 15 more
Caused by: java.security.spec.InvalidKeySpecException: Neither RSA, DSA nor EC worked
at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1024)
at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:993)
at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:265)
… 18 more
Caused by: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: IOException : algid parse error, not a sequence
at sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:169)
at java.security.KeyFactory.generatePrivate(KeyFactory.java:372)
at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1022)
… 20 more
Caused by: java.security.InvalidKeyException: IOException : algid parse error, not a sequence
at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:352)
at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:357)
at sun.security.ec.ECPrivateKeyImpl.(ECPrivateKeyImpl.java:73)
at sun.security.ec.ECKeyFactory.implGeneratePrivate(ECKeyFactory.java:237)
at sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:165)
… 22 more

``

12

looks like it worked with a .jks of the same new cert :slight_smile:

thank you for all the help guys :D/