Elasticsearch failed to start after installing searchguard along with valid certificates

Search Guard
1

Hi All,

I have installed Search Guard version 5.6.4-18 plugin to my Elasticsearch cluster (Version 5.6.4). Search Guard was working fine when I generated the self signed certificates using the TLS Certificate Generator, but now we got the valid certificates from third party entrust and they provided three files which has,

  • chain-crt.pem
  • server-crt.pem
  • server-key.pem
    JAVA version “1.8.0_151”

OS “Ubuntu”

VERSION=“16.04.3 LTS (Xenial Xerus)”

I have configured the pem files in elasticsearch.yml as like below,

Configuring TLS on each node

searchguard.ssl.transport.pemcert_filepath: /home/elasticsearch-uat/elasticsearch-5.6.4/config/server-crt.pem

searchguard.ssl.transport.pemkey_filepath: /home/elasticsearch-uat/elasticsearch-5.6.4/config/server-key.pem

#searchguard.ssl.transport.pemkey_password: changeit

searchguard.ssl.transport.pemtrustedcas_filepath: /home/elasticsearch-uat/elasticsearch-5.6.4/config/chain-crt.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: /home/elasticsearch-uat/elasticsearch-5.6.4/config/server-crt.pem

searchguard.ssl.http.pemkey_filepath: /home/elasticsearch-uat/elasticsearch-5.6.4/config/server-key.pem

#searchguard.ssl.http.pemkey_password: changeit

searchguard.ssl.http.pemtrustedcas_filepath: /home/elasticsearch-uat/elasticsearch-5.6.4/config/chain-crt.pem

searchguard.authcz.admin_dn:

The valid certificates CN has wildcard based type and it wil be like this *.servicenow.com

After the above configuration changes made in elasticsearch.yml, I tried starting the elasticsearch but however I am getting the below exception in ES logs,

2018-02-08T13:54:43,754][ERROR][c.f.s.s.DefaultSearchGuardKeyStore] Your keystore or PEM does not contain a key. If you sepcified a key password try removing it. If you not sepcified a key password maybe you one because the key is password protected. Maybe you just confused keys and certificates.

[2018-02-08T13:54:43,828][ERROR][o.e.b.Bootstrap ] Exception

org.elasticsearch.ElasticsearchException: Failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]

at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:438) ~[elasticsearch-5.6.4.jar:5.6.4]

Please find the attached complete ES logs & Elasticsearch.yml and kindly share your thoughts. Please correct me if I am doing anything wrong in the setup and it would be very helpful.

Regards,
Ganeshbabu R

Elasticsearch1.yml (3.81 KB)

es_logs.txt (22.9 KB)

2

I’m not familiar with Entrust, but usually your CA does not provide you with the private key of your certificate. The whole point is that you keep your private key secret at all times. The usual steps to get a certificate from a CA is:

  • Generate a private key

  • Generate a CSR with that key

  • (both steps can also be combined in one command)

  • Send the CSR to the CA, keep your key secret

  • You will get back your certificate, the root CA and all intermediates (if any)

You need to use the private key that has been generated when you created the CSR.

···

On Thursday, February 8, 2018 at 3:49:05 PM UTC+1, Ganesh Babu wrote:

Hi All,

I have installed Search Guard version 5.6.4-18 plugin to my Elasticsearch cluster (Version 5.6.4). Search Guard was working fine when I generated the self signed certificates using the TLS Certificate Generator, but now we got the valid certificates from third party entrust and they provided three files which has,

  • chain-crt.pem
  • server-crt.pem
  • server-key.pem
    JAVA version “1.8.0_151”

OS “Ubuntu”

VERSION=“16.04.3 LTS (Xenial Xerus)”

I have configured the pem files in elasticsearch.yml as like below,

Configuring TLS on each node

searchguard.ssl.transport.pemcert_filepath: /home/elasticsearch-uat/elasticsearch-5.6.4/config/server-crt.pem

searchguard.ssl.transport.pemkey_filepath: /home/elasticsearch-uat/elasticsearch-5.6.4/config/server-key.pem

#searchguard.ssl.transport.pemkey_password: changeit

searchguard.ssl.transport.pemtrustedcas_filepath: /home/elasticsearch-uat/elasticsearch-5.6.4/config/chain-crt.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: /home/elasticsearch-uat/elasticsearch-5.6.4/config/server-crt.pem

searchguard.ssl.http.pemkey_filepath: /home/elasticsearch-uat/elasticsearch-5.6.4/config/server-key.pem

#searchguard.ssl.http.pemkey_password: changeit

searchguard.ssl.http.pemtrustedcas_filepath: /home/elasticsearch-uat/elasticsearch-5.6.4/config/chain-crt.pem

searchguard.authcz.admin_dn:

The valid certificates CN has wildcard based type and it wil be like this *.servicenow.com

After the above configuration changes made in elasticsearch.yml, I tried starting the elasticsearch but however I am getting the below exception in ES logs,

2018-02-08T13:54:43,754][ERROR][c.f.s.s.DefaultSearchGuardKeyStore] Your keystore or PEM does not contain a key. If you sepcified a key password try removing it. If you not sepcified a key password maybe you one because the key is password protected. Maybe you just confused keys and certificates.

[2018-02-08T13:54:43,828][ERROR][o.e.b.Bootstrap ] Exception

org.elasticsearch.ElasticsearchException: Failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]

at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:438) ~[elasticsearch-5.6.4.jar:5.6.4]

Please find the attached complete ES logs & Elasticsearch.yml and kindly share your thoughts. Please correct me if I am doing anything wrong in the setup and it would be very helpful.

Regards,
Ganeshbabu R

3

Hi Jochen,

Thanks for your reply and sorry for the confusion about the certificates.

What you said above is correct and Yes I got the valid certificate from the Certificate Authority. Later I converted to .pem format using the openssl command. As I mentioned above, I have configured the pem files in elasticsearch.yml and tried to start elasticsearch but failed to do so.

Kindly check it once of my elasticsearch logs and yml file and correct me if I am doing anything wrong.
As I forgot to mention above my elasticsearch node hostname is elaticsearch-data1 and my certificate CN is *.servicenow.com
Should the certificates has to be generated based on the hostname of ES nodes?

Please correct me if my understanding is wrong.

Regards,

Ganeshbabu R

4

Do make SSL hostname verification working your hostnames and the contents of the certificate need to be aligned!

···

Am 12.02.2018 um 08:37 schrieb Ganesh Babu <babu.ganesh0708@gmail.com>:

Hi Jochen,

Thanks for your reply and sorry for the confusion about the certificates.

What you said above is correct and Yes I got the valid certificate from the Certificate Authority. Later I converted to .pem format using the openssl command. As I mentioned above, I have configured the pem files in elasticsearch.yml and tried to start elasticsearch but failed to do so.

Kindly check it once of my elasticsearch logs and yml file and correct me if I am doing anything wrong.

As I forgot to mention above my elasticsearch node hostname is elaticsearch-data1 and my certificate CN is *.servicenow.com

Should the certificates has to be generated based on the hostname of ES nodes?

Please correct me if my understanding is wrong.

Regards,
Ganeshbabu R

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/6c8347ea-c5f0-4f35-a659-9d246c6ab113%40googlegroups.com\.
For more options, visit https://groups.google.com/d/optout\.

5

I had similar issue. One thing you may want to double check is what type of key you have. SG supports PKCS12 and PKCS8. Say if you have PKCS1 format, you may run into these issues. Make sure your server-key is of one those. You can convert them using openssl command.
openssl pkcs8 -topk8 -inform PEM -outform PEM -nocrypt -in server-key.pem -out server-key8.pem

Jalaja

···

On Thursday, February 8, 2018 at 6:49:05 AM UTC-8, Ganesh Babu wrote:

Hi All,

I have installed Search Guard version 5.6.4-18 plugin to my Elasticsearch cluster (Version 5.6.4). Search Guard was working fine when I generated the self signed certificates using the TLS Certificate Generator, but now we got the valid certificates from third party entrust and they provided three files which has,

  • chain-crt.pem
  • server-crt.pem
  • server-key.pem
    JAVA version “1.8.0_151”

OS “Ubuntu”

VERSION=“16.04.3 LTS (Xenial Xerus)”

I have configured the pem files in elasticsearch.yml as like below,

Configuring TLS on each node

searchguard.ssl.transport.pemcert_filepath: /home/elasticsearch-uat/elasticsearch-5.6.4/config/server-crt.pem

searchguard.ssl.transport.pemkey_filepath: /home/elasticsearch-uat/elasticsearch-5.6.4/config/server-key.pem

#searchguard.ssl.transport.pemkey_password: changeit

searchguard.ssl.transport.pemtrustedcas_filepath: /home/elasticsearch-uat/elasticsearch-5.6.4/config/chain-crt.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: /home/elasticsearch-uat/elasticsearch-5.6.4/config/server-crt.pem

searchguard.ssl.http.pemkey_filepath: /home/elasticsearch-uat/elasticsearch-5.6.4/config/server-key.pem

#searchguard.ssl.http.pemkey_password: changeit

searchguard.ssl.http.pemtrustedcas_filepath: /home/elasticsearch-uat/elasticsearch-5.6.4/config/chain-crt.pem

searchguard.authcz.admin_dn:

The valid certificates CN has wildcard based type and it wil be like this *.servicenow.com

After the above configuration changes made in elasticsearch.yml, I tried starting the elasticsearch but however I am getting the below exception in ES logs,

2018-02-08T13:54:43,754][ERROR][c.f.s.s.DefaultSearchGuardKeyStore] Your keystore or PEM does not contain a key. If you sepcified a key password try removing it. If you not sepcified a key password maybe you one because the key is password protected. Maybe you just confused keys and certificates.

[2018-02-08T13:54:43,828][ERROR][o.e.b.Bootstrap ] Exception

org.elasticsearch.ElasticsearchException: Failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]

at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:438) ~[elasticsearch-5.6.4.jar:5.6.4]

Please find the attached complete ES logs & Elasticsearch.yml and kindly share your thoughts. Please correct me if I am doing anything wrong in the setup and it would be very helpful.

Regards,
Ganeshbabu R