"failed to load plugin class" When using Sgadmin

Search Guard
1

Elasticsearch and Search Guard version : 6.3.2

Entreprise Module : Disabled

JVM Version : 1.8.0_111

Operating System : Debian 9

Hello,

I want to change the ID and the password for the Admin user on Search Guard, so i used the hash.sh tool to generate my hashed password.

After that, i opened my sg_internal_user.yml located in the Search Guard directory of my master node.

After modified the file with my new hashed password, i had to apply my modification to the whole cluster, so

i runned sgadmin.sh. Then i get this error :

WARNING: Seems you want connect to the Elasticsearch HTTP port.

sgadmin connects on the transport port which is normally 9300.

Will connect to 10.15.20.160:9200 … done

10:27:12.088 [main] ERROR com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore - Your keystore or PEM does not contain a certificate. Maybe you confused keys and certificates.

ERR: An unexpected IllegalStateException occured: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]

Trace:

java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]

at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:701)

at org.elasticsearch.plugins.PluginsService.(PluginsService.java:114)

at org.elasticsearch.client.transport.TransportClient.newPluginService(TransportClient.java:107)

at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:132)

at org.elasticsearch.client.transport.TransportClient.(TransportClient.java:269)

at com.floragunn.searchguard.tools.SearchGuardAdmin$TransportClientImpl.(SearchGuardAdmin.java:886)

at com.floragunn.searchguard.tools.SearchGuardAdmin.main0(SearchGuardAdmin.java:441)

at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:123)

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:692)

… 7 more

Caused by: ElasticsearchSecurityException[Error while initializing transport SSL layer from PEM: java.lang.IllegalArgumentException: File does not contain valid certificates: /etc/elasticsearch/gandalf.cer]; nested: IllegalArgumentException[File does not contain valid certificates: /etc/elasticsearch/gandalf.cer]; nested: CertificateException[found no certificates in input stream];

at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:292)

at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.(DefaultSearchGuardKeyStore.java:145)

at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.(SearchGuardSSLPlugin.java:193)

at com.floragunn.searchguard.SearchGuardPlugin.(SearchGuardPlugin.java:197)

… 12 more

Caused by: java.lang.IllegalArgumentException: File does not contain valid certificates: /etc/elasticsearch/gandalf.cer

at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:262)

at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:90)

at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.buildSSLServerContext(DefaultSearchGuardKeyStore.java:613)

at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:287)

… 15 more

Caused by: java.security.cert.CertificateException: found no certificates in input stream

at io.netty.handler.ssl.PemReader.readCertificates(PemReader.java:98)

at io.netty.handler.ssl.PemReader.readCertificates(PemReader.java:64)

at io.netty.handler.ssl.SslContext.toX509Certificates(SslContext.java:1070)

at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:260)

… 18 more

``

My certificates are validate by the the root CA

Can you explain me what is the problem please ?

Thank You !

2

File does not contain valid certificates: /etc/elasticsearch/gandalf.cer]

Seems gandalf.cer does not contain a valid X509 certificate in pem format

···

Am 16.01.2019 um 10:30 schrieb Voortexx <ekham93@gmail.com>:

Elasticsearch and Search Guard version : 6.3.2

Entreprise Module : Disabled

JVM Version : 1.8.0_111

Operating System : Debian 9

Hello,

I want to change the ID and the password for the Admin user on Search Guard, so i used the hash.sh tool to generate my hashed password.

After that, i opened my sg_internal_user.yml located in the Search Guard directory of my master node.

After modified the file with my new hashed password, i had to apply my modification to the whole cluster, so

i runned sgadmin.sh. Then i get this error :

WARNING: Seems you want connect to the Elasticsearch HTTP port.
         sgadmin connects on the transport port which is normally 9300.
Will connect to 10.15.20.160:9200 ... done
10:27:12.088 [main] ERROR com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore - Your keystore or PEM does not contain a certificate. Maybe you confused keys and certificates.
ERR: An unexpected IllegalStateException occured: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
Trace:
java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:701)
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:114)
        at org.elasticsearch.client.transport.TransportClient.newPluginService(TransportClient.java:107)
        at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:132)
        at org.elasticsearch.client.transport.TransportClient.<init>(TransportClient.java:269)
        at com.floragunn.searchguard.tools.SearchGuardAdmin$TransportClientImpl.<init>(SearchGuardAdmin.java:886)
        at com.floragunn.searchguard.tools.SearchGuardAdmin.main0(SearchGuardAdmin.java:441)
        at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:123)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:692)
        ... 7 more
Caused by: ElasticsearchSecurityException[Error while initializing transport SSL layer from PEM: java.lang.IllegalArgumentException: File does not contain valid certificates: /etc/elasticsearch/gandalf.cer]; nested: IllegalArgumentException[File does not contain valid certificates: /etc/elasticsearch/gandalf.cer]; nested: CertificateException[found no certificates in input stream];
        at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:292)
        at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.<init>(DefaultSearchGuardKeyStore.java:145)
        at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.<init>(SearchGuardSSLPlugin.java:193)
        at com.floragunn.searchguard.SearchGuardPlugin.<init>(SearchGuardPlugin.java:197)
        ... 12 more
Caused by: java.lang.IllegalArgumentException: File does not contain valid certificates: /etc/elasticsearch/gandalf.cer
        at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:262)
        at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:90)
        at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.buildSSLServerContext(DefaultSearchGuardKeyStore.java:613)
        at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:287)
        ... 15 more
Caused by: java.security.cert.CertificateException: found no certificates in input stream
        at io.netty.handler.ssl.PemReader.readCertificates(PemReader.java:98)
        at io.netty.handler.ssl.PemReader.readCertificates(PemReader.java:64)
        at io.netty.handler.ssl.SslContext.toX509Certificates(SslContext.java:1070)
        at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:260)
        ... 18 more

My certificates are validate by the the root CA

Can you explain me what is the problem please ?

Thank You !

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/f01c8557-12d8-4a51-ac89-50621dc83f1d%40googlegroups.com\.
For more options, visit https://groups.google.com/d/optout\.

3

Thanks for help, I create new certificate and it works now.

However, I get a new error message. It says that it can’t check if the cluster is sane but when i run --diagnose i can see that sgadmin has

found the slaves nodes

Error Message :

Search Guard Admin v6

Will connect to 10.15.20.160:9300 … done

Unable to check whether cluster is sane: None of the configured nodes are available: [{#transport#-1}{hqNdlsB2TQSX1Jw3ukt_Cg}{10.15.20.160}{10.15.20.160:9300}]

ERR: Cannot connect to Elasticsearch. Please refer to elasticsearch logfile for more information

Trace:

NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{hqNdlsB2TQSX1Jw3ukt_Cg}{10.15.20.160}{10.15.20.160:9300}]]

    at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347)

    at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245)

    at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:60)

    at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:378)

    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:405)

    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:394)

    at com.floragunn.searchguard.tools.SearchGuardAdmin.main0(SearchGuardAdmin.java:450)

    at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:123)

``

4

Can you please attach your elasticsearch.yml (as file) and share the exact sgadmin command you are issueing?

Please also look into the logfiles if there is anything conspicuous

···

On Tuesday, 22 January 2019 15:51:45 UTC+1, Voortexx wrote:

Thanks for help, I create new certificate and it works now.

However, I get a new error message. It says that it can’t check if the cluster is sane but when i run --diagnose i can see that sgadmin has

found the slaves nodes

Error Message :

Search Guard Admin v6

Will connect to 10.15.20.160:9300 … done

Unable to check whether cluster is sane: None of the configured nodes are available: [{#transport#-1}{hqNdlsB2TQSX1Jw3ukt_Cg}{10.15.20.160}{10.15.20.160:9300}]

ERR: Cannot connect to Elasticsearch. Please refer to elasticsearch logfile for more information

Trace:

NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{hqNdlsB2TQSX1Jw3ukt_Cg}{10.15.20.160}{10.15.20.160:9300}]]

    at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347)

    at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245)

    at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:60)

    at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:378)

    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:405)

    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:394)

    at com.floragunn.searchguard.tools.SearchGuardAdmin.main0(SearchGuardAdmin.java:450)

    at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:123)

``

5

Here is my elasticsearch.yml :

======================== Elasticsearch Configuration =========================

sgadmin_diag_trace_2019-Jan-22_14-29-09.txt (268 KB)

···

NOTE: Elasticsearch comes with reasonable defaults for most settings.

Before you set out to tweleak and tune the configuration, make sure you understand what are you trying to accomplish and the consequences.

The primary way of configuring a node is via this file. This template lists the most important settings you may want to configure for a production cluster.

Please consult the documentation for further information on configuration options: Elasticsearch Guide | Elastic

---------------------------------- Cluster -----------------------------------

Use a descriptive name for your cluster:

cluster.name: ClustBase

------------------------------------ Node ------------------------------------

Use a descriptive name for the node:

node.master: true

node.name: Master

Add custom attributes to the node:

#node.attr.rack: r1

----------------------------------- Paths ------------------------------------

Path to directory where to store the data (separate multiple locations by comma):

path.data: /var/lib/elasticsearch

path.repo: /mnt/repo_snapshots

#Path to log files:

path.logs: /var/log/elasticsearch

----------------------------------- Memory -----------------------------------

Lock the memory on startup:

#bootstrap.memory_lock: true

Make sure that the heap size is set to about half the memory available on the system and that the owner of the process is allowed to use this limit.

Elasticsearch performs poorly when the system is swapping the memory.

---------------------------------- Network -----------------------------------

Set the bind address to a specific IP (IPv4 or IPv6):

network.host: 10.15.20.160

Set a custom port for HTTP:

http.port: 9200

For more information, consult the network module documentation.

--------------------------------- Discovery ----------------------------------

Pass an initial list of hosts to perform discovery when new node is started: The default list of hosts is [“127.0.0.1”, “[::1]”]

discovery.zen.ping.unicast.hosts: [“10.15.20.160”, “10.15.20.167”, “10.15.20.168”]

Prevent the “split brain” by configuring the majority of nodes (total number of master-eligible

#discovery.zen.minimum_master_nodes:

For more information, consult the zen discovery module documentation.

---------------------------------- Gateway -----------------------------------

Block initial recovery after a full cluster restart until N nodes are started:

#gateway.recover_after_nodes: 3

For more information, consult the gateway module documentation.

---------------------------------- Various -----------------------------------

Require explicit names when deleting indices:

#action.destructive_requires_name: true shared repo path_repo: [“mount/backups”, “/mount/longterm_backups”]

#path.repo: [“mnt/repo_snapshots”]

#---------------------------------Search-Guard---------------------------------

searchguard.ssl.transport.pemcert_filepath: 87.cer

searchguard.ssl.transport.pemkey_filepath: sdevfrekham.reyesholdings.com.key

searchguard.ssl.transport.pemkey_password: wd6qpyOQ32hw

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.cer

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: 88.cer

searchguard.ssl.http.pemkey_filepath: sdevfrekham.reyesholdings.com_http.key

searchguard.ssl.http.pemkey_password: EofEHuwj2zI1

searchguard.ssl.http.pemtrustedcas_filepath: root-ca.cer

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

#-----------------------------------X-Pack-------------------------------------

xpack.security.enabled: false

xpack.monitoring.enabled : false

xpack.graph.enabled : false

xpack.watcher.enabled : false

searchguard.enterprise_modules_enabled: false

``

Here is the command to push files on the whole cluster :

./sgadmin.sh -cd /usr/share/elasticsearch/plugins/search-guard-6/sgconfig/ -cacert /tmp/root-ca.cer -cert /etc/elasticsearch/superuser.reyesholdings.com.cer -key /etc/elasticsearch/superuser.reyesholdings.com.key -keypass EPyrMnFqFzsY -nhnv -icn -h 10.15.20.160

When i run --diagnose, i get the file “sgadmin_diag_trace_2019-Jan-22_14-29-09.txt”

6

There is not “-icn” sgadmin option. I guess you mean “-icl” ?

And in the diag trace i can see that your cluster is “red” because you have 18 unassigned shards (see How to Resolve Unassigned Shards in Elasticsearch | Datadog and Safely disable/enable replicas to fix unassigned shards - Elasticsearch - Discuss the Elastic Stack)

Add the “-arc” (allow red cluster) option to sgadmin to allow operations on a red cluster.

···

On Friday, 25 January 2019 10:08:18 UTC+1, Voortexx wrote:

Here is my elasticsearch.yml :

======================== Elasticsearch Configuration =========================

NOTE: Elasticsearch comes with reasonable defaults for most settings.

Before you set out to tweleak and tune the configuration, make sure you understand what are you trying to accomplish and the consequences.

The primary way of configuring a node is via this file. This template lists the most important settings you may want to configure for a production cluster.

Please consult the documentation for further information on configuration options: https://www.elastic.co/guide/en/elasticsearch/reference/index.html

---------------------------------- Cluster -----------------------------------

Use a descriptive name for your cluster:

cluster.name: ClustBase

------------------------------------ Node ------------------------------------

Use a descriptive name for the node:

node.master: true

node.name: Master

Add custom attributes to the node:

#node.attr.rack: r1

----------------------------------- Paths ------------------------------------

Path to directory where to store the data (separate multiple locations by comma):

path.data: /var/lib/elasticsearch

path.repo: /mnt/repo_snapshots

#Path to log files:

path.logs: /var/log/elasticsearch

----------------------------------- Memory -----------------------------------

Lock the memory on startup:

#bootstrap.memory_lock: true

Make sure that the heap size is set to about half the memory available on the system and that the owner of the process is allowed to use this limit.

Elasticsearch performs poorly when the system is swapping the memory.

---------------------------------- Network -----------------------------------

Set the bind address to a specific IP (IPv4 or IPv6):

network.host: 10.15.20.160

Set a custom port for HTTP:

http.port: 9200

For more information, consult the network module documentation.

--------------------------------- Discovery ----------------------------------

Pass an initial list of hosts to perform discovery when new node is started: The default list of hosts is [“127.0.0.1”, “[::1]”]

discovery.zen.ping.unicast.hosts: [“10.15.20.160”, “10.15.20.167”, “10.15.20.168”]

Prevent the “split brain” by configuring the majority of nodes (total number of master-eligible

#discovery.zen.minimum_master_nodes:

For more information, consult the zen discovery module documentation.

---------------------------------- Gateway -----------------------------------

Block initial recovery after a full cluster restart until N nodes are started:

#gateway.recover_after_nodes: 3

For more information, consult the gateway module documentation.

---------------------------------- Various -----------------------------------

Require explicit names when deleting indices:

#action.destructive_requires_name: true shared repo path_repo: [“mount/backups”, “/mount/longterm_backups”]

#path.repo: [“mnt/repo_snapshots”]

#---------------------------------Search-Guard---------------------------------

searchguard.ssl.transport.pemcert_filepath: 87.cer

searchguard.ssl.transport.pemkey_filepath: sdevfrekham.reyesholdings.com.key

searchguard.ssl.transport.pemkey_password: wd6qpyOQ32hw

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.cer

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: 88.cer

searchguard.ssl.http.pemkey_filepath: sdevfrekham.reyesholdings.com_http.key

searchguard.ssl.http.pemkey_password: EofEHuwj2zI1

searchguard.ssl.http.pemtrustedcas_filepath: root-ca.cer

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

#-----------------------------------X-Pack-------------------------------------

xpack.security.enabled: false

xpack.monitoring.enabled : false

xpack.graph.enabled : false

xpack.watcher.enabled : false

searchguard.enterprise_modules_enabled: false

``

Here is the command to push files on the whole cluster :

./sgadmin.sh -cd /usr/share/elasticsearch/plugins/search-guard-6/sgconfig/ -cacert /tmp/root-ca.cer -cert /etc/elasticsearch/superuser.reyesholdings.com.cer -key /etc/elasticsearch/superuser.reyesholdings.com.key -keypass EPyrMnFqFzsY -nhnv -icn -h 10.15.20.160

When i run --diagnose, i get the file “sgadmin_diag_trace_2019-Jan-22_14-29-09.txt”