File does not contain valid private key: /etc/elasticsearch/node1.key

Li_Cui · August 27, 2018, 4:09am

Hello there.

Yesterday I setup the Serch-guard on ELK cluster after going through so many issues with the keys and certificates.

It was due to that the only pkcs8 keys were supported… I’m on the last Search Guard version, Version: 6.x-23.

Anyway, it worked yesterday I was able to send logs from the client via filebeat and the data showing on Kibana.

I shut down the cluster and shut down the AWS instances (Elassticsearch and kibana on one node, logstash on another node).

Today, I started the AWS instances and started the cluster, checked that all ELK were up and running, but when I used the same credential(the default admin)

But I wasn’t able to log on saying “The Search Guard license information could not be loaded. Please contact your system administrator.”

Here is my sg_config.yml:

searchguard:

dynamic:

authc:

  clientcert_auth_domain:

    http_enabled: true

    order: 1

    http_authenticator:

      type: clientcert

      config:

        username_attribute: cn #optional, if omitted DN becomes username

      challenge: false

    authentication_backend:

      type: noop

···

========================

Elasticsearch started fine and I can access on :https://:9200 and got:

{

“name” : “node1”,

“cluster_name” : “hls-test-elk”,

“cluster_uuid” : “A_9Ls_hiSzKM34WZ7Rjxnw”,

“version” : {

"number" : "6.3.2",

"build_flavor" : "default",

"build_type" : "rpm",

"build_hash" : "053779d",

"build_date" : "2018-07-20T05:20:23.451332Z",

"build_snapshot" : false,

"lucene_version" : "7.3.1",

"minimum_wire_compatibility_version" : "5.6.0",

"minimum_index_compatibility_version" : "5.0.0"

},

“tagline” : “You Know, for Search”

}

============================

./tools/sgadmin.sh -cd sgconfig/ -cacert /etc/elasticsearch/root-ca.pem -cert /etc/elasticsearch/node1.pem -key /etc/elasticsearch/node1.key -nhnv -h <elastic.IP> -cn hls-test-elk

WARNING: JAVA_HOME not set, will use /bin/java

Search Guard Admin v6

Will connect to <elastic.IP>:9300 … done

01:26:01.556 [main] ERROR com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore - Your keystore or PEM does not contain a key. If you specified a key password, try removing it. If you did not specify a key password, perhaps you need to if the key is in fact password-protected. Maybe you just confused keys and certificates.

ERR: An unexpected IllegalStateException occured: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]

Trace:

java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]

    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:701)

    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:114)

    at org.elasticsearch.client.transport.TransportClient.newPluginService(TransportClient.java:107)

    at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:132)

    at org.elasticsearch.client.transport.TransportClient.<init>(TransportClient.java:269)

    at com.floragunn.searchguard.tools.SearchGuardAdmin$TransportClientImpl.<init>(SearchGuardAdmin.java:886)

    at com.floragunn.searchguard.tools.SearchGuardAdmin.main0(SearchGuardAdmin.java:441)

    at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:123)

Caused by: java.lang.reflect.InvocationTargetException

    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:692)

    ... 7 more

Caused by: ElasticsearchSecurityException[Error while initializing transport SSL layer from PEM: java.lang.IllegalArgumentException: File does not contain valid private key: /etc/elasticsearch/node1.key]; nested: IllegalArgumentException[File does not contain valid private key: /etc/elasticsearch/node1.key]; nested: InvalidKeySpecException[Neither RSA, DSA nor EC worked]; nested: InvalidKeySpecException[java.security.InvalidKeyException: IOException : DER input, Integer tag error]; nested: InvalidKeyException[IOException : DER input, Integer tag error];

    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:292)

    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.<init>(DefaultSearchGuardKeyStore.java:145)

    at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.<init>(SearchGuardSSLPlugin.java:193)

    at com.floragunn.searchguard.SearchGuardPlugin.<init>(SearchGuardPlugin.java:197)

    ... 12 more

Caused by: java.lang.IllegalArgumentException: File does not contain valid private key: /etc/elasticsearch/node1.key

    at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:267)

    at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:90)

    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.buildSSLServerContext(DefaultSearchGuardKeyStore.java:613)

    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:287)

    ... 15 more

Caused by: java.security.spec.InvalidKeySpecException: Neither RSA, DSA nor EC worked

    at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1045)

    at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1014)

    at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:265)

    ... 18 more

Caused by: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: IOException : DER input, Integer tag error

    at sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:169)

    at java.security.KeyFactory.generatePrivate(KeyFactory.java:372)

    at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1043)

    ... 20 more

Caused by: java.security.InvalidKeyException: IOException : DER input, Integer tag error

    at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:352)

    at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:357)

    at sun.security.ec.ECPrivateKeyImpl.<init>(ECPrivateKeyImpl.java:73)

    at sun.security.ec.ECKeyFactory.implGeneratePrivate(ECKeyFactory.java:237)

    at sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:165)

    ... 22 more

==============================

I’ve changed the node1.key using “openssl pkcs8 topk8 -in node1.key -out node1-pkcs8.key” and didn’t work. the sg-internal-users.yml has never been changed.

Please help and let us know what’s next…

Thank you very much in advance

LI

jkressin · August 27, 2018, 6:38pm

Maybe you ran into this Java issue here:

https://github.com/floragunncom/search-guard/issues/524

tl;dr: OpenSSL changed the default PKCS#5, v1.5 format to PKCS#5, v2 in OpenSSL 1.1.0. The JDK has a long outstanding issue with PKCS#5, v2. In short, PKCS#5, v2 does not work.

Can you try to force OpenSSL to generate PKCS#5, v1.5 by using:

openssl pkcs8 -v1 "PBE-SHA1-3DES" -topk8 ...

···

On Monday, August 27, 2018 at 12:09:53 AM UTC-4, Li Cui wrote:

Hello there.

Yesterday I setup the Serch-guard on ELK cluster after going through so many issues with the keys and certificates.

It was due to that the only pkcs8 keys were supported… I’m on the last Search Guard version, Version: 6.x-23.

Anyway, it worked yesterday I was able to send logs from the client via filebeat and the data showing on Kibana.

I shut down the cluster and shut down the AWS instances (Elassticsearch and kibana on one node, logstash on another node).

Today, I started the AWS instances and started the cluster, checked that all ELK were up and running, but when I used the same credential(the default admin)

But I wasn’t able to log on saying “The Search Guard license information could not be loaded. Please contact your system administrator.”

Here is my sg_config.yml:

searchguard:

dynamic:

authc:

  clientcert_auth_domain:

    http_enabled: true

    order: 1

    http_authenticator:

      type: clientcert

      config:

        username_attribute: cn #optional, if omitted DN becomes username

      challenge: false

    authentication_backend:

      type: noop

========================

Elasticsearch started fine and I can access on :https://:9200 and got:

{

“name” : “node1”,

“cluster_name” : “hls-test-elk”,

“cluster_uuid” : “A_9Ls_hiSzKM34WZ7Rjxnw”,

“version” : {

"number" : "6.3.2",

"build_flavor" : "default",

"build_type" : "rpm",

"build_hash" : "053779d",

"build_date" : "2018-07-20T05:20:23.451332Z",

"build_snapshot" : false,

"lucene_version" : "7.3.1",

"minimum_wire_compatibility_version" : "5.6.0",

"minimum_index_compatibility_version" : "5.0.0"

},

“tagline” : “You Know, for Search”

}

============================

./tools/sgadmin.sh -cd sgconfig/ -cacert /etc/elasticsearch/root-ca.pem -cert /etc/elasticsearch/node1.pem -key /etc/elasticsearch/node1.key -nhnv -h <elastic.IP> -cn hls-test-elk

WARNING: JAVA_HOME not set, will use /bin/java

Search Guard Admin v6

Will connect to <elastic.IP>:9300 … done

01:26:01.556 [main] ERROR com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore - Your keystore or PEM does not contain a key. If you specified a key password, try removing it. If you did not specify a key password, perhaps you need to if the key is in fact password-protected. Maybe you just confused keys and certificates.

ERR: An unexpected IllegalStateException occured: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]

Trace:

java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]

    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:701)

    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:114)

    at org.elasticsearch.client.transport.TransportClient.newPluginService(TransportClient.java:107)

    at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:132)

    at org.elasticsearch.client.transport.TransportClient.<init>(TransportClient.java:269)

    at com.floragunn.searchguard.tools.SearchGuardAdmin$TransportClientImpl.<init>(SearchGuardAdmin.java:886)

    at com.floragunn.searchguard.tools.SearchGuardAdmin.main0(SearchGuardAdmin.java:441)

    at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:123)

Caused by: java.lang.reflect.InvocationTargetException

    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:692)

    ... 7 more

Caused by: ElasticsearchSecurityException[Error while initializing transport SSL layer from PEM: java.lang.IllegalArgumentException: File does not contain valid private key: /etc/elasticsearch/node1.key]; nested: IllegalArgumentException[File does not contain valid private key: /etc/elasticsearch/node1.key]; nested: InvalidKeySpecException[Neither RSA, DSA nor EC worked]; nested: InvalidKeySpecException[java.security.InvalidKeyException: IOException : DER input, Integer tag error]; nested: InvalidKeyException[IOException : DER input, Integer tag error];

    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:292)

    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.<init>(DefaultSearchGuardKeyStore.java:145)

    at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.<init>(SearchGuardSSLPlugin.java:193)

    at com.floragunn.searchguard.SearchGuardPlugin.<init>(SearchGuardPlugin.java:197)

    ... 12 more

Caused by: java.lang.IllegalArgumentException: File does not contain valid private key: /etc/elasticsearch/node1.key

    at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:267)

    at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:90)

    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.buildSSLServerContext(DefaultSearchGuardKeyStore.java:613)

    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:287)

    ... 15 more

Caused by: java.security.spec.InvalidKeySpecException: Neither RSA, DSA nor EC worked

    at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1045)

    at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1014)

    at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:265)

    ... 18 more

Caused by: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: IOException : DER input, Integer tag error

    at sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:169)

    at java.security.KeyFactory.generatePrivate(KeyFactory.java:372)

    at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1043)

    ... 20 more

Caused by: java.security.InvalidKeyException: IOException : DER input, Integer tag error

    at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:352)

    at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:357)

    at sun.security.ec.ECPrivateKeyImpl.<init>(ECPrivateKeyImpl.java:73)

    at sun.security.ec.ECKeyFactory.implGeneratePrivate(ECKeyFactory.java:237)

    at sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:165)

    ... 22 more

==============================

I’ve changed the node1.key using “openssl pkcs8 topk8 -in node1.key -out node1-pkcs8.key” and didn’t work. the sg-internal-users.yml has never been changed.

Please help and let us know what’s next…

Thank you very much in advance

LI

Li_Cui · August 28, 2018, 3:56am

Thank you…

I did: openssl pkcs8 -topk8 -v1 PBE-SHA1-3DES -in node1.key -out node1.key

Still got the same result:

[root@elk.com search-guard-6]# ./tools/sgadmin.sh -cd sgconfig/ -cacert /etc/elasticsearch/root-ca.pem -cert /etc/elasticsearch/node1.pem -key /etc/elasticsearch/node1.key -nhnv -h -cn elk-tsg

WARNING: JAVA_HOME not set, will use /bin/java

Search Guard Admin v6

Will connect to :9300 … done

03:47:55.215 [main] ERROR com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore - Your keystore or PEM does not contain a key. If you specified a key password, try removing it. If you did not specify a key password, perhaps you need to if the key is in fact password-protected. Maybe you just confused keys and certificates.

ERR: An unexpected IllegalStateException occured: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]

Trace:

java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]

at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:701)

at org.elasticsearch.plugins.PluginsService.(PluginsService.java:114)

at org.elasticsearch.client.transport.TransportClient.newPluginService(TransportClient.java:107)

at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:132)

at org.elasticsearch.client.transport.TransportClient.(TransportClient.java:269)

at com.floragunn.searchguard.tools.SearchGuardAdmin$TransportClientImpl.(SearchGuardAdmin.java:886)

at com.floragunn.searchguard.tools.SearchGuardAdmin.main0(SearchGuardAdmin.java:441)

at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:123)

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:692)

… 7 more

Caused by: ElasticsearchSecurityException[Error while initializing transport SSL layer from PEM: java.lang.IllegalArgumentException: File does not contain valid private key: /etc/elasticsearch/node1.key]; nested: IllegalArgumentException[File does not contain valid private key: /etc/elasticsearch/node1.key]; nested: InvalidKeySpecException[Neither RSA, DSA nor EC worked]; nested: InvalidKeySpecException[java.security.InvalidKeyException: IOException : DER input, Integer tag error]; nested: InvalidKeyException[IOException : DER input, Integer tag error];

at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:292)

at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.(DefaultSearchGuardKeyStore.java:145)

at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.(SearchGuardSSLPlugin.java:193)

at com.floragunn.searchguard.SearchGuardPlugin.(SearchGuardPlugin.java:197)

… 12 more

Caused by: java.lang.IllegalArgumentException: File does not contain valid private key: /etc/elasticsearch/node1.key

at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:267)

at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:90)

at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.buildSSLServerContext(DefaultSearchGuardKeyStore.java:613)

at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:287)

… 15 more

I can start the elasticsearch no problem, but can not initialize the cluster…

This is very urgent,… need to demo this…

Please help

Li

···

On Mon, Aug 27, 2018 at 1:38 PM Jochen Kressin jkressin@floragunn.com wrote:

Maybe you ran into this Java issue here:

https://github.com/floragunncom/search-guard/issues/524

tl;dr: OpenSSL changed the default PKCS#5, v1.5 format to PKCS#5, v2 in OpenSSL 1.1.0. The JDK has a long outstanding issue with PKCS#5, v2. In short, PKCS#5, v2 does not work.

Can you try to force OpenSSL to generate PKCS#5, v1.5 by using:

openssl pkcs8 -v1 "PBE-SHA1-3DES" -topk8 ...

On Monday, August 27, 2018 at 12:09:53 AM UTC-4, Li Cui wrote:

Hello there.

Yesterday I setup the Serch-guard on ELK cluster after going through so many issues with the keys and certificates.

It was due to that the only pkcs8 keys were supported… I’m on the last Search Guard version, Version: 6.x-23.

Anyway, it worked yesterday I was able to send logs from the client via filebeat and the data showing on Kibana.

I shut down the cluster and shut down the AWS instances (Elassticsearch and kibana on one node, logstash on another node).

Today, I started the AWS instances and started the cluster, checked that all ELK were up and running, but when I used the same credential(the default admin)

But I wasn’t able to log on saying “The Search Guard license information could not be loaded. Please contact your system administrator.”

Here is my sg_config.yml:

searchguard:

dynamic:

authc:

  clientcert_auth_domain:

    http_enabled: true

    order: 1

    http_authenticator:

      type: clientcert

      config:

        username_attribute: cn #optional, if omitted DN becomes username

      challenge: false

    authentication_backend:

      type: noop

========================

Elasticsearch started fine and I can access on :https://:9200 and got:

{

“name” : “node1”,

“cluster_name” : “hls-test-elk”,

“cluster_uuid” : “A_9Ls_hiSzKM34WZ7Rjxnw”,

“version” : {

"number" : "6.3.2",

"build_flavor" : "default",

"build_type" : "rpm",

"build_hash" : "053779d",

"build_date" : "2018-07-20T05:20:23.451332Z",

"build_snapshot" : false,

"lucene_version" : "7.3.1",

"minimum_wire_compatibility_version" : "5.6.0",

"minimum_index_compatibility_version" : "5.0.0"

},

“tagline” : “You Know, for Search”

}

============================

./tools/sgadmin.sh -cd sgconfig/ -cacert /etc/elasticsearch/root-ca.pem -cert /etc/elasticsearch/node1.pem -key /etc/elasticsearch/node1.key -nhnv -h <elastic.IP> -cn hls-test-elk

WARNING: JAVA_HOME not set, will use /bin/java

Search Guard Admin v6

Will connect to <elastic.IP>:9300 … done

01:26:01.556 [main] ERROR com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore - Your keystore or PEM does not contain a key. If you specified a key password, try removing it. If you did not specify a key password, perhaps you need to if the key is in fact password-protected. Maybe you just confused keys and certificates.

ERR: An unexpected IllegalStateException occured: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]

Trace:

java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]

    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:701)

    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:114)

    at org.elasticsearch.client.transport.TransportClient.newPluginService(TransportClient.java:107)

    at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:132)

    at org.elasticsearch.client.transport.TransportClient.<init>(TransportClient.java:269)

    at com.floragunn.searchguard.tools.SearchGuardAdmin$TransportClientImpl.<init>(SearchGuardAdmin.java:886)

    at com.floragunn.searchguard.tools.SearchGuardAdmin.main0(SearchGuardAdmin.java:441)

    at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:123)

Caused by: java.lang.reflect.InvocationTargetException

    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:692)

    ... 7 more

Caused by: ElasticsearchSecurityException[Error while initializing transport SSL layer from PEM: java.lang.IllegalArgumentException: File does not contain valid private key: /etc/elasticsearch/node1.key]; nested: IllegalArgumentException[File does not contain valid private key: /etc/elasticsearch/node1.key]; nested: InvalidKeySpecException[Neither RSA, DSA nor EC worked]; nested: InvalidKeySpecException[java.security.InvalidKeyException: IOException : DER input, Integer tag error]; nested: InvalidKeyException[IOException : DER input, Integer tag error];

    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:292)

    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.<init>(DefaultSearchGuardKeyStore.java:145)

    at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.<init>(SearchGuardSSLPlugin.java:193)

    at com.floragunn.searchguard.SearchGuardPlugin.<init>(SearchGuardPlugin.java:197)

    ... 12 more

Caused by: java.lang.IllegalArgumentException: File does not contain valid private key: /etc/elasticsearch/node1.key

    at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:267)

    at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:90)

    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.buildSSLServerContext(DefaultSearchGuardKeyStore.java:613)

    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:287)

    ... 15 more

Caused by: java.security.spec.InvalidKeySpecException: Neither RSA, DSA nor EC worked

    at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1045)

    at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1014)

    at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:265)

    ... 18 more

Caused by: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: IOException : DER input, Integer tag error

    at sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:169)

    at java.security.KeyFactory.generatePrivate(KeyFactory.java:372)

    at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1043)

    ... 20 more

Caused by: java.security.InvalidKeyException: IOException : DER input, Integer tag error

    at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:352)

    at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:357)

    at [sun.security.ec](http://sun.security.ec).ECPrivateKeyImpl.<init>(ECPrivateKeyImpl.java:73)

    at sun.security.ec.ECKeyFactory.implGeneratePrivate(ECKeyFactory.java:237)

    at sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:165)

    ... 22 more

==============================

I’ve changed the node1.key using “openssl pkcs8 topk8 -in node1.key -out node1-pkcs8.key” and didn’t work. the sg-internal-users.yml has never been changed.

Please help and let us know what’s next…

Thank you very much in advance

LI

–

You received this message because you are subscribed to the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/60d4864c-7d03-4697-acb5-3cb12b7a08e3%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

jkressin · August 28, 2018, 6:50pm

So this seems strange to me. You write that it worked at some point, but then after restart you got the exception complaining about the key? Something is odd here, because if the key stayed the same, you should not see this exception. I guess I want to say that the key does not suddenly become invalid. So something else must be going on here.

First thing to do would be to check the contents of the key:

openssl rsa -in config/node1.key -check

``

Second would be the question how you exactly generated the key? If you have a PKCS#12 for example and want to convert it to PKCS#5, v1.5 you would use something like:

openssl pkcs8 -v1 “PBE-SHA1-3DES” -in “node1.key.pkcs12” -topk8 -out “node1.key” -passin “pass:…” -passout “pass:…”

``

If your key does not have any password of course omit the passin and passout parameters.

Third thing to check is whether your key has a password set. Depending on how you generated the key, you might have header in your node1.key file like:

-----BEGIN ENCRYPTED PRIVATE KEY-----

``

or

-----BEGIN PRIVATE KEY-----

``

···

On Monday, August 27, 2018 at 11:56:46 PM UTC-4, Li Cui wrote:

Thank you…

I did: openssl pkcs8 -topk8 -v1 PBE-SHA1-3DES -in node1.key -out node1.key

Still got the same result:

[root@elk.com search-guard-6]# ./tools/sgadmin.sh -cd sgconfig/ -cacert /etc/elasticsearch/root-ca.pem -cert /etc/elasticsearch/node1.pem -key /etc/elasticsearch/node1.key -nhnv -h -cn elk-tsg

WARNING: JAVA_HOME not set, will use /bin/java

Search Guard Admin v6

Will connect to :9300 … done

03:47:55.215 [main] ERROR com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore - Your keystore or PEM does not contain a key. If you specified a key password, try removing it. If you did not specify a key password, perhaps you need to if the key is in fact password-protected. Maybe you just confused keys and certificates.

ERR: An unexpected IllegalStateException occured: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]

Trace:

java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]

at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:701)

at org.elasticsearch.plugins.PluginsService.(PluginsService.java:114)

at org.elasticsearch.client.transport.TransportClient.newPluginService(TransportClient.java:107)

at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:132)

at org.elasticsearch.client.transport.TransportClient.(TransportClient.java:269)

at com.floragunn.searchguard.tools.SearchGuardAdmin$TransportClientImpl.(SearchGuardAdmin.java:886)

at com.floragunn.searchguard.tools.SearchGuardAdmin.main0(SearchGuardAdmin.java:441)

at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:123)

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:692)

… 7 more

Caused by: ElasticsearchSecurityException[Error while initializing transport SSL layer from PEM: java.lang.IllegalArgumentException: File does not contain valid private key: /etc/elasticsearch/node1.key]; nested: IllegalArgumentException[File does not contain valid private key: /etc/elasticsearch/node1.key]; nested: InvalidKeySpecException[Neither RSA, DSA nor EC worked]; nested: InvalidKeySpecException[java.security.InvalidKeyException: IOException : DER input, Integer tag error]; nested: InvalidKeyException[IOException : DER input, Integer tag error];

at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:292)

at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.(DefaultSearchGuardKeyStore.java:145)

at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.(SearchGuardSSLPlugin.java:193)

at com.floragunn.searchguard.SearchGuardPlugin.(SearchGuardPlugin.java:197)

… 12 more

Caused by: java.lang.IllegalArgumentException: File does not contain valid private key: /etc/elasticsearch/node1.key

at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:267)

at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:90)

at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.buildSSLServerContext(DefaultSearchGuardKeyStore.java:613)

at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:287)

… 15 more

I can start the elasticsearch no problem, but can not initialize the cluster…

This is very urgent,… need to demo this…

Please help

Li

On Mon, Aug 27, 2018 at 1:38 PM Jochen Kressin jkressin@floragunn.com wrote:

Maybe you ran into this Java issue here:

https://github.com/floragunncom/search-guard/issues/524

tl;dr: OpenSSL changed the default PKCS#5, v1.5 format to PKCS#5, v2 in OpenSSL 1.1.0. The JDK has a long outstanding issue with PKCS#5, v2. In short, PKCS#5, v2 does not work.

Can you try to force OpenSSL to generate PKCS#5, v1.5 by using:

openssl pkcs8 -v1 "PBE-SHA1-3DES" -topk8 ...

On Monday, August 27, 2018 at 12:09:53 AM UTC-4, Li Cui wrote:

Hello there.

Yesterday I setup the Serch-guard on ELK cluster after going through so many issues with the keys and certificates.

It was due to that the only pkcs8 keys were supported… I’m on the last Search Guard version, Version: 6.x-23.

Anyway, it worked yesterday I was able to send logs from the client via filebeat and the data showing on Kibana.

I shut down the cluster and shut down the AWS instances (Elassticsearch and kibana on one node, logstash on another node).

Today, I started the AWS instances and started the cluster, checked that all ELK were up and running, but when I used the same credential(the default admin)

But I wasn’t able to log on saying “The Search Guard license information could not be loaded. Please contact your system administrator.”

Here is my sg_config.yml:

searchguard:

dynamic:

authc:

  clientcert_auth_domain:

    http_enabled: true

    order: 1

    http_authenticator:

      type: clientcert

      config:

        username_attribute: cn #optional, if omitted DN becomes username

      challenge: false

    authentication_backend:

      type: noop

========================

Elasticsearch started fine and I can access on :https://:9200 and got:

{

“name” : “node1”,

“cluster_name” : “hls-test-elk”,

“cluster_uuid” : “A_9Ls_hiSzKM34WZ7Rjxnw”,

“version” : {

"number" : "6.3.2",

"build_flavor" : "default",

"build_type" : "rpm",

"build_hash" : "053779d",

"build_date" : "2018-07-20T05:20:23.451332Z",

"build_snapshot" : false,

"lucene_version" : "7.3.1",

"minimum_wire_compatibility_version" : "5.6.0",

"minimum_index_compatibility_version" : "5.0.0"

},

“tagline” : “You Know, for Search”

}

============================

./tools/sgadmin.sh -cd sgconfig/ -cacert /etc/elasticsearch/root-ca.pem -cert /etc/elasticsearch/node1.pem -key /etc/elasticsearch/node1.key -nhnv -h <elastic.IP> -cn hls-test-elk

WARNING: JAVA_HOME not set, will use /bin/java

Search Guard Admin v6

Will connect to <elastic.IP>:9300 … done

01:26:01.556 [main] ERROR com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore - Your keystore or PEM does not contain a key. If you specified a key password, try removing it. If you did not specify a key password, perhaps you need to if the key is in fact password-protected. Maybe you just confused keys and certificates.

ERR: An unexpected IllegalStateException occured: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]

Trace:

java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]

    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:701)

    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:114)

    at org.elasticsearch.client.transport.TransportClient.newPluginService(TransportClient.java:107)

    at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:132)

    at org.elasticsearch.client.transport.TransportClient.<init>(TransportClient.java:269)

    at com.floragunn.searchguard.tools.SearchGuardAdmin$TransportClientImpl.<init>(SearchGuardAdmin.java:886)

    at com.floragunn.searchguard.tools.SearchGuardAdmin.main0(SearchGuardAdmin.java:441)

    at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:123)

Caused by: java.lang.reflect.InvocationTargetException

    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:692)

    ... 7 more

Caused by: ElasticsearchSecurityException[Error while initializing transport SSL layer from PEM: java.lang.IllegalArgumentException: File does not contain valid private key: /etc/elasticsearch/node1.key]; nested: IllegalArgumentException[File does not contain valid private key: /etc/elasticsearch/node1.key]; nested: InvalidKeySpecException[Neither RSA, DSA nor EC worked]; nested: InvalidKeySpecException[java.security.InvalidKeyException: IOException : DER input, Integer tag error]; nested: InvalidKeyException[IOException : DER input, Integer tag error];

    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:292)

    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.<init>(DefaultSearchGuardKeyStore.java:145)

    at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.<init>(SearchGuardSSLPlugin.java:193)

    at com.floragunn.searchguard.SearchGuardPlugin.<init>(SearchGuardPlugin.java:197)

    ... 12 more

Caused by: java.lang.IllegalArgumentException: File does not contain valid private key: /etc/elasticsearch/node1.key

    at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:267)

    at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:90)

    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.buildSSLServerContext(DefaultSearchGuardKeyStore.java:613)

    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:287)

    ... 15 more

Caused by: java.security.spec.InvalidKeySpecException: Neither RSA, DSA nor EC worked

    at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1045)

    at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1014)

    at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:265)

    ... 18 more

Caused by: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: IOException : DER input, Integer tag error

    at sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:169)

    at java.security.KeyFactory.generatePrivate(KeyFactory.java:372)

    at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1043)

    ... 20 more

Caused by: java.security.InvalidKeyException: IOException : DER input, Integer tag error

    at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:352)

    at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:357)

    at [sun.security.ec](http://sun.security.ec).ECPrivateKeyImpl.<init>(ECPrivateKeyImpl.java:73)

    at sun.security.ec.ECKeyFactory.implGeneratePrivate(ECKeyFactory.java:237)

    at sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:165)

    ... 22 more

==============================

I’ve changed the node1.key using “openssl pkcs8 topk8 -in node1.key -out node1-pkcs8.key” and didn’t work. the sg-internal-users.yml has never been changed.

Please help and let us know what’s next…

Thank you very much in advance

LI

–

You received this message because you are subscribed to the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/60d4864c-7d03-4697-acb5-3cb12b7a08e3%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Li_Cui · August 28, 2018, 8:27pm

Kressin,

Thank you very much for your help. By reading your suggestions, I did the following and got the same result:

[ec2-user@mycompany.com tools]$ ./sgtlstool.sh -c /home/ec2-user/config/tlsconfig.yml -ca -crt

WARNING: JAVA_HOME not set, will use /usr/bin/java

Root certificate has been sucessfully created.

The passwords of the private key files have been auto generated. You can find the passwords in root-ca.readme.

Created 2 node certificates.

Passwords for the private keys of the node certificates have been auto-generated. The passwords are stored in the config snippet files.

Created 1 client certificates.

Passwords for the private keys of the client certificates have been auto-generated. The passwords are stored in the file “client-certificates.readme”

2.[ec2-user@mycompany.com out]$ openssl rsa -in node1.key -check

Enter pass phrase for node1.key:

RSA key ok

writing RSA key

-----BEGIN RSA PRIVATE KEY-----

MIIEogIBAAKCAQEAmnD5vaOnsJdxfYGutvrkQ74vht/jvQWIxfm10fiZd0FCaQN+

LzyfizUW2ayLjs/F2H/GbZSvjtfoIatwdD/AEJgqvLzVJj4YAYeVAcqTk9iYmMQM

rOFaJQ3jbSc7XkB4ewGguuJO+oP3Sduh5PcRdCiTEZTMWxnCVOknv5k1vZq7e8Sq

ad+KKTzLRB1LfIKlu80lEbOQmjhIGoUKUhO6H60NLjAuE5vGgxRm0Xuhe4SG4DKh

H73cyylriOfyIyb92sg3CC1X71xqgjvVIPhuDOlJPEeBwTXyqnhhoftZotfQaN4D

SYMrJA397+l3wwWPxZ+OJ8YLSyid26dhmdzS/wIDAQABAoH/WY7fvP41ZOcc+jz8

Hz+QRk/SBhr/YCmBWBeeMvZeDhqyWa9q8+vYHKQ+xxzVTqFTq5JUvzhUd2yQlLYm

8lMxf7Ut+YdHjgMxUpgAbUPRbsEN+O4C4dOe2l606Zfve0wlpTIBezgUMgocnOeC

wtpw5fBw+mDQoqEIdMGmAyjLamviqX1W2y1/CeVTN5RvAkR25MhnYNlzkR0//4cV

M2zV3AXytfS8fHU59kBlDZmc5DFC3FBZTO79l2NVI25E9SBjgNAsa7QJSsYAHL3j

8S+EInuP+xt55kyQzDCe/1DxRAc3tcxMnxiercUyFRMEIeaz2cp+PfQW+dYRlS0u

Y1uJAoGBANarS6c4fhq2gtKjISZlxNjRgz4gp+wFGeBkHzzv8vSJsClJwnA1Z6V5

3lFJc0e5XtwWhwkCoxu3jCXmOt0YmCNyUWz/F/N++cZZZ+ECk+kZ/ahZddqiUI5W

bgrsgp2xnMDm/tKro0peEyDR1ONHX/DoUEwZGtO4qzkIjVzMTxYXAoGBALgtJUWi

2Att1hATly7QMnTZKJa2EbJS1AdRBI60aJh5OiTyWAMSz8Ozm65qjBuQsrzs8Xiv

/f4X+gYXVkimuWiImuWkTE1XII/D4Gce6Naw2jO/OIadTn7mmhVLoLmXhyysGA9r

X0KnhaTTrXk2vBqLgD7ksLdeuHgD2z+hDyNZAoGBALGs15IRRPaRJxh2IoLoapPW

XR035CSKWE+MLvq900YpDCAwR3meCipBAwwekgbEW8on7x47C4tuUNYxh3jl2U5F

WTWwsT89UjikWUE4Y/nkJDmJyezquWgGC1CJTtZ8YjSbPEBcBWwRDx3z7pMqyTLY

5VUG1lh0mWs8M9yh0beDAoGAP/4XrDlnnZj9Prq/1dkrHuRxMOR0A6S53Qzua3MN

wKSmaj32DKNVoybd2d1VbyJ4qCIfyJJFyyqjichEB5w+Ya7NOys6A8lZG/qwFsRr

nGT4e/BeAI/ZvUf8k5eicbnPktr8xSbGg49XZSzTYoF9HDAKUE//gscbiz+zx0j7

zIkCgYEAkhMaSadi/7m0nb20l9VuTOOs2RbLE4xgWOWU/wLEB6zPPLgJqS7paWr7

bDJuSA5bYKyf/hbN2wsZwc79RyJJ3a8b7F6H8J9ZtvkZUbG3s/hrGNDSu9mOJxWJ

vbePZ9s95qBmpJL4gFC1os+lPazr6Y1DqU/vhSVdEzIKd97VCIg=

-----END RSA PRIVATE KEY-----

3.[ec2-user@mycompany.com tools]$ ./sgtlsdiag.sh -ca /home/ec2-user/tools/out/root-ca.pem -crt /home/ec2-user/tools/out/node1.pem

WARNING: JAVA_HOME not set, will use /usr/bin/java

tlsconfig.yml (4.83 KB)

elasticsearch.yml (3.97 KB)

···

========================================================================

/home/ec2-user/tools/out/node1.pem

Certificate 1

SHA1 FPR: AA4CEF301E3A418066984FF54D043E3DB9D6EE3E

MD5 FPR: EEAE1AC4B2FF4D22FA73CC492A9AF10D

Subject DN [RFC2253]: CN=node1.hlx.dxc.com,OU=Ops,O=Example Com, Inc.,DC=hls.dxc,DC=com

Serial Number: 1535485513377

Issuer DN [RFC2253]: CN=root.ca.hls.dxc.com,OU=CA,O=Example Com, Inc.,DC=hls.dxc,DC=com

Not Before: Tue Aug 28 19:45:14 UTC 2018

Not After: Fri Aug 25 19:45:14 UTC 2028

Key Usage: digitalSignature nonRepudiation keyEncipherment

Signature Algorithm: SHA256WITHRSA

Version: 3

Extended Key Usage: id_kp_serverAuth id_kp_clientAuth

Basic Constraints: -1

SAN:

dNSName: node1.hls.dxc.com

iPAddress:

Trust anchor:

DC=com,DC=hls.dxc,O=Example Com, Inc.,OU=CA,CN=root.ca.hls.dxc.com

Replaced the new keys and certificates in elasticsearch.yml with the new ones and also node1_elasticsearch_config_snippet.yml
Restarted elasticsearch node.
Went to https:// :9200

Got:Search Guard not initialized (SG11). See http://docs.search-guard.com/v6/sgadmin

7.[root@mycompany.com tools]# /sgadmin.sh -cd sgconfig/ -cacert /etc/elasticsearch/root-ca.pem -cert /etc/elasticsearch/node1.pem -key /etc/elasticsearch/node1.key -nhnv -h -cn elk-tsg

bash: /sgadmin.sh: No such file or directory

[root@mycompany.com tools]# ./sgadmin.sh -cd sgconfig/ -cacert /etc/elasticsearch/root-ca.pem -cert /etc/elasticsearch/node1.pem -key /etc/elasticsearch/node1.key -nhnv -h -cn elk-tsg

WARNING: JAVA_HOME not set, will use /bin/java

Search Guard Admin v6

Will connect to :9300 … done

20:00:36.045 [main] ERROR com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore - Your keystore or PEM does not contain a key. If you specified a key password, try removing it. If you did not specify a key password, perhaps you need to if the key is in fact password-protected. Maybe you just confused keys and certificates.