Unable to execute sgadmin.sh

Search Guard
1

Hi All,

I deployed Search Guard on non prod linux ES instance.
I had to generate certificates using offline tls-generator tool by Search Guard because it involved adding IP address to the certificate.

After this I copied the certificates to /config folder.

ES%20config

I created a new internal user and changed the basic config files fo SG.

Now I was trying to execute sgadmin to add new users to searchguard index. But unfortunately it is giving me the following error:-

Search Guard Admin v6
Will connect to 10.123.222.89:9300 ... done
ERR: An unexpected IllegalStateException occured: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
Trace:
java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:563)
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:104)
        at org.elasticsearch.client.transport.TransportClient.newPluginService(TransportClient.java:105)
        at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:130)
        at org.elasticsearch.client.transport.TransportClient.<init>(TransportClient.java:262)
        at com.floragunn.searchguard.tools.SearchGuardAdmin$TransportClientImpl.<init>(SearchGuardAdmin.java:886)
        at com.floragunn.searchguard.tools.SearchGuardAdmin.main0(SearchGuardAdmin.java:441)
        at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:123)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:554)
        ... 7 more
Caused by: ElasticsearchException[Unable to read ../../../config/kirk-key.pem (../../../config/kirk-key.pem). Please make sure this files exists and is readable regarding to permissions. Property: searchguard.ssl.transport.pemkey_filepath]
        at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.checkPath(DefaultSearchGuardKeyStore.java:709)
        at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.resolve(DefaultSearchGuardKeyStore.java:193)
        at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:282)
        at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.<init>(DefaultSearchGuardKeyStore.java:145)
        at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.<init>(SearchGuardSSLPlugin.java:193)
        at com.floragunn.searchguard.SearchGuardPlugin.<init>(SearchGuardPlugin.java:197)
        ... 12 more

kirk was generated as admin certificate.
I believe its because its key is not in kirk-key.pem format.
I have a file as kirk.key. But if I feed in kirk.key instead, it seems to demand it in .pem format.

Can anyone tell what should I do differently to make it work?

My elasticsearch.yml is:

searchguard.ssl.transport.pemcert_filepath: esnode.pem
searchguard.ssl.transport.pemkey_filepath: esnode.key
searchguard.ssl.transport.pemkey_password: aBcDeFgHi12
searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: esnode_http.pem
searchguard.ssl.http.pemkey_filepath: esnode_http.key
searchguard.ssl.http.pemkey_password: jKlMnOpQrS34
searchguard.ssl.http.pemtrustedcas_filepath: root-ca.pem
searchguard.nodes_dn:
- CN=kirk,OU=client,O=client,L=test,C=de
searchguard.authcz.admin_dn:
- CN=kirk,OU=client,O=client,L=test,C=de
2

As far as I can see there are two issues here.

First, you use the same kirk certificate as admin and also as node certificate:

searchguard.nodes_dn: 
  - CN=kirk,OU=client,O=client,L=test,C=de 
searchguard.authcz.admin_dn: 
  - CN=kirk,OU=client,O=client,L=test,C=de

Due to security considerations this is not allowed. You need an admin certificate (kirk in your case) that is separate from a node certificate. In your case the node certificate is es-node.pem.

The second issue is that when you use sgadmin the path to the kirk certificate seems to be wrong:

So make sure that the path you use in the actual sgadmin call:

../../../config/kirk-key.pem

is correct.

The filename, by the way, does not really specify/tell you what format the certificate or key has. The kirk.key file already contains the correct format. In other words, you can name the file whatever you like.

4

Hi Jochen,

There is no file as “kirk-key.pem” produced by tlsgenerator.
Instead I gave in “kirk.key”(produced by tlsgenerator) which I think is not accepted by the sgadmin.sh script.

Do you think its because it demands .pem format?

If yes, how can I convert kirk.key into .pem format?.

5

You do not need to convert anything. The private key the generator produces is already in the correct PEM format. From what I can see in your posts, the private key is:

kirk.key

And the certificate is:

kirk.pem

From the error message I can see that when executing sgadmin you refer to the (non-existing) file kirk-key.pem:

Unable to read ../../../config/kirk-key.pem

Replacing the (non-existing) kirk-key.pem with the (existing) kirk.key file in the sgadmin call should do the trick.

Can you please post the sgadmin call you are issuing?

6

HI jochen,

Thanks for your response.

At first I executed this statement,

./sgadmin.sh -cd …/sgconfig/ -icl -nhnv -cacert …/…/…/config/root-ca.pem -cert …/…/…/config/kirk.pem -key …/…/…/config/kirk.key -h 10.123.211.87

Which gave me the following error, which I assume is because of no keypass.

Search Guard Admin v6
Will connect to 10.123.211.87:9300 ... done
15:34:32.924 [main] ERROR com.floragunn.searchguard.ssl.DefaultSearchGuardKeySto                                                                                        re - Your keystore or PEM does not contain a key. If you specified a key passwor                                                                                        d, try removing it. If you did not specify a key password, perhaps you need to i                                                                                        f the key is in fact password-protected. Maybe you just confused keys and certif                                                                                        icates.
ERR: An unexpected IllegalStateException occured: failed to load plugin class [c                                                                                        om.floragunn.searchguard.SearchGuardPlugin]
Trace:
java.lang.IllegalStateException: failed to load plugin class [com.floragunn.sear                                                                                        chguard.SearchGuardPlugin]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.ja                                                                                        va:563)
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:1                                                                                        04)
        at org.elasticsearch.client.transport.TransportClient.newPluginService(T                                                                                        ransportClient.java:105)
        at org.elasticsearch.client.transport.TransportClient.buildTemplate(Tran                                                                                        sportClient.java:130)
        at org.elasticsearch.client.transport.TransportClient.<init>(TransportCl                                                                                        ient.java:262)
        at com.floragunn.searchguard.tools.SearchGuardAdmin$TransportClientImpl.                                                                                        <init>(SearchGuardAdmin.java:886)
        at com.floragunn.searchguard.tools.SearchGuardAdmin.main0(SearchGuardAdm                                                                                        in.java:441)
        at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmi                                                                                        n.java:123)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct                                                                                        orAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC                                                                                        onstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.ja                                                                                        va:554)
        ... 7 more
Caused by: ElasticsearchSecurityException[Error while initializing transport SSL                                                                                         layer from PEM: java.lang.IllegalArgumentException: File does not contain valid                                                                                         private key: ../../../config/kirk.key]; nested: IllegalArgumentException[File d                                                                                        oes not contain valid private key: ../../../config/kirk.key]; nested: InvalidKey                                                                                        SpecException[Neither RSA, DSA nor EC worked]; nested: InvalidKeySpecException[j                                                                                        ava.security.InvalidKeyException: IOException : DER input, Integer tag error]; n                                                                                        ested: InvalidKeyException[IOException : DER input, Integer tag error];
        at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfi                                                                                        g(DefaultSearchGuardKeyStore.java:292)
        at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.<init>(Defau                                                                                        ltSearchGuardKeyStore.java:145)
        at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.<init>(SearchGuard                                                                                        SSLPlugin.java:193)
        at com.floragunn.searchguard.SearchGuardPlugin.<init>(SearchGuardPlugin.                                                                                        java:197)
        ... 12 more
Caused by: java.lang.IllegalArgumentException: File does not contain valid priva                                                                                        te key: ../../../config/kirk.key
        at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.j                                                                                        ava:267)
        at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.ja                                                                                        va:90)
        at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.buildSSLServ                                                                                        erContext(DefaultSearchGuardKeyStore.java:613)
        at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfi                                                                                        g(DefaultSearchGuardKeyStore.java:287)
        ... 15 more
Caused by: java.security.spec.InvalidKeySpecException: Neither RSA, DSA nor EC w                                                                                        orked
        at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContex                                                                                        t.java:1045)
        at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1014)
        at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.j                                                                                        ava:265)
        ... 18 more
Caused by: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyE                                                                                        xception: IOException : DER input, Integer tag error
        at sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:                                                                                        169)
        at java.security.KeyFactory.generatePrivate(KeyFactory.java:372)
        at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContex                                                                                        t.java:1043)
        ... 20 more
Caused by: java.security.InvalidKeyException: IOException : DER input, Integer t                                                                                        ag error
        at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:352)
        at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:357)
        at sun.security.ec.ECPrivateKeyImpl.<init>(ECPrivateKeyImpl.java:73)
        at sun.security.ec.ECKeyFactory.implGeneratePrivate(ECKeyFactory.java:23                                                                                        7)
        at sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:                                                                                        165)
        ... 22 more

Then I provide another option -keypass with the password provided by tlsgenerator in “client-certificates.readme” file.

./sgadmin.sh -cd …/sgconfig/ -icl -nhnv -cacert …/…/…/config/root-ca.pem -cert …/…/…/config/kirk.pem -key …/…/…/config/kirk.key -keypass TUpeLXjD3u5J -h 10.123.211.87

But still even after adding keypass I got the following error.

Search Guard Admin v6
Will connect to 10.123.211.87:9300 ... done
Unable to check whether cluster is sane: None of the configured nodes are available: [{#transport#-1}{Hyl106pgT0aZ8_spuz0jxA}{10.123.211.87}{10.123.211.87:9300}]
ERR: Cannot connect to Elasticsearch. Please refer to elasticsearch logfile for more information
Trace:
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{Hyl106pgT0aZ8_spuz0jxA}{10.123.211.87}{10.123.211.87:9300}]]
        at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347)
        at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245)
        at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:60)
        at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:371)
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:405)
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:394)
        at com.floragunn.searchguard.tools.SearchGuardAdmin.main0(SearchGuardAdmin.java:450)

Background story:-

After installing the plugin.
I used install_demo_.sh for setup…
Only to find out that to upload data the ip address should be present in the certificate subjectAltName.

I generated certificates by adding ip address under node conf area in tlsconfig.yml in tlsgenerator.

And just copied certificates to /config and changed elasticsearch.yml with provided snippet

I think there can be 2 possible scenarios.

  1. the dn for client and node certificates are same.
  2. Since I added generated certificates after using install_demo_conf script. I need to add cluster name details somewhere for search.

Which reason do you think is more likely or both?

If both, for first I need to generate certificates again with separate dn and again copyt new certs and change elasticsearch.yml, right?

What about the 2nd one?

7

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.