Elasticsearch is not starting for Search Guard error

shuvo · July 5, 2020, 8:51am

I’m trying to configure Search Guard admin. here is my elasticsearch.yml configurations:

searchguard.ssl.transport.pemcert_filepath: /home/rsa-key-20200528/search-guard-ssl-master/example-pki-scripts/node-0.crt.pem
searchguard.ssl.transport.pemkey_filepath: /home/rsa-key-20200528/search-guard-ssl-master/example-pki-scripts/node-0.key.pem
searchguard.ssl.transport.pemkey_password: changeit
searchguard.ssl.transport.pemtrustedcas_filepath: /home/rsa-key-20200528/search-guard-ssl-master/example-pki-scripts/ca/root-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.authcz.admin_dn:
  - CN=kirk,OU=client,O=client,L=Test,C=DE

After this configurations, I’m unable to start my elasticsearch. Here is the full error from my “/var/log/elasticsearch/my-application.log” file.

[2020-07-05T04:51:39,896][INFO ][o.e.n.Node               ] [node-1] version[7.8.0], pid[2328], build[default/deb/757314695644ea9a1dc2fecd26d1a43856725e65/2020-06-14T19:35:50.234439Z], OS[Linux/4.9.0-12-amd64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/14.0.1/14.0.1+7]
[2020-07-05T04:51:39,907][INFO ][o.e.n.Node               ] [node-1] JVM home [/usr/share/elasticsearch/jdk]
[2020-07-05T04:51:39,908][INFO ][o.e.n.Node               ] [node-1] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-3499631201687762622, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2020-07-05T04:51:44,548][INFO ][c.f.s.s.SignalsPainlessExtension] [node-1] Loaded script whitelist: org.elasticsearch.painless.spi.Whitelist@3a5e2525
[2020-07-05T04:51:44,571][INFO ][c.f.s.SearchGuardPlugin  ] [node-1] ES Config path is /etc/elasticsearch
[2020-07-05T04:51:44,877][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] [node-1] JVM supports TLSv1.3
[2020-07-05T04:51:44,884][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] [node-1] Config directory is /etc/elasticsearch/, from there the key- and truststore files are resolved relatively
[2020-07-05T04:51:44,889][ERROR][o.e.b.Bootstrap          ] [node-1] Exception
java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:616) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:558) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:473) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:164) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.node.Node.<init>(Node.java:317) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) [elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) [elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) [elasticsearch-cli-7.8.0.jar:7.8.0]
	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.8.0.jar:7.8.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) [elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.8.0.jar:7.8.0]
Caused by: java.lang.reflect.InvocationTargetException
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) ~[elasticsearch-7.8.0.jar:7.8.0]
	... 15 more
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/home/rsa-key-20200528/search-guard-ssl-master/example-pki-scripts/node-0.crt.pem" "read")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]
	at java.security.AccessController.checkPermission(AccessController.java:1036) ~[?:?]
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:408) ~[?:?]
	at java.lang.SecurityManager.checkRead(SecurityManager.java:747) ~[?:?]
	at sun.nio.fs.UnixPath.checkRead(UnixPath.java:818) ~[?:?]
	at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:49) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149) ~[?:?]
	at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) ~[?:?]
	at java.nio.file.Files.readAttributes(Files.java:1843) ~[?:?]
	at java.nio.file.Files.isDirectory(Files.java:2314) ~[?:?]
	at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.checkPath(DefaultSearchGuardKeyStore.java:827) ~[?:?]
	at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.resolve(DefaultSearchGuardKeyStore.java:223) ~[?:?]
	at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:343) ~[?:?]
	at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.<init>(DefaultSearchGuardKeyStore.java:164) ~[?:?]
	at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.<init>(SearchGuardSSLPlugin.java:199) ~[?:?]
	at com.floragunn.searchguard.SearchGuardPlugin.<init>(SearchGuardPlugin.java:227) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) ~[elasticsearch-7.8.0.jar:7.8.0]
	... 15 more
[2020-07-05T04:51:44,908][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.8.0.jar:7.8.0]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.8.0.jar:7.8.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.8.0.jar:7.8.0]
Caused by: java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:616) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:558) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:473) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:164) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.node.Node.<init>(Node.java:317) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.8.0.jar:7.8.0]
	... 6 more
Caused by: java.lang.reflect.InvocationTargetException
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:558) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:473) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:164) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.node.Node.<init>(Node.java:317) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.8.0.jar:7.8.0]
	... 6 more
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/home/rsa-key-20200528/search-guard-ssl-master/example-pki-scripts/node-0.crt.pem" "read")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]
	at java.security.AccessController.checkPermission(AccessController.java:1036) ~[?:?]
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:408) ~[?:?]
	at java.lang.SecurityManager.checkRead(SecurityManager.java:747) ~[?:?]
	at sun.nio.fs.UnixPath.checkRead(UnixPath.java:818) ~[?:?]
	at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:49) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149) ~[?:?]
	at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) ~[?:?]
	at java.nio.file.Files.readAttributes(Files.java:1843) ~[?:?]
	at java.nio.file.Files.isDirectory(Files.java:2314) ~[?:?]
	at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.checkPath(DefaultSearchGuardKeyStore.java:827) ~[?:?]
	at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.resolve(DefaultSearchGuardKeyStore.java:223) ~[?:?]
	at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:343) ~[?:?]
	at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.<init>(DefaultSearchGuardKeyStore.java:164) ~[?:?]
	at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.<init>(SearchGuardSSLPlugin.java:199) ~[?:?]
	at com.floragunn.searchguard.SearchGuardPlugin.<init>(SearchGuardPlugin.java:227) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:558) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:473) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:164) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.node.Node.<init>(Node.java:317) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.8.0.jar:7.8.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.8.0.jar:7.8.0]
	... 6 more

I’m following this and this documentation.

what is wrong here? why elasticsearch is not starting?

srgbnd · July 7, 2020, 4:27pm

Does Elasticsearch user have rights to read the node-0.crt.pem file? I bet he doesn’t.

Another thing, if you don’t use OID in the TLS certificates, don’t forget to configure the searchguard.nodes_dn: Configuring TLS | Security for Elasticsearch | Search Guard

An example of the config for a cluster node:

elasticsearch.yml

node.name: sgssl-0.example.com
network.host: 0.0.0.0
http.port: 9200
cluster.name: es-docker-cluster
discovery.seed_hosts: ["sgssl-1.example.com", "sgssl-2.example.com"]
cluster.initial_master_nodes: ["sgssl-0.example.com", "sgssl-1.example.com", "sgssl-2.example.com"]
bootstrap.memory_lock: true
cluster.routing.allocation.disk.threshold_enabled: false
node.max_local_storage_nodes: 3

xpack.security.enabled: false

searchguard.allow_default_init_sgindex: true
searchguard.audit.type: internal_elasticsearch
searchguard.enable_snapshot_restore_privilege: true
searchguard.check_snapshot_restore_write_privileges: true
searchguard.restapi.roles_enabled: ["SGS_ALL_ACCESS"]
# This is a configuration snippet for the node node1
# This snippet needs to be inserted into the file config/elasticsearch.yml of the respective node.
# If the config file already contains SearchGuard configuration, this needs to be replaced.
# Furthermore, you need to copy the files referenced below into the same directory.
# Please refer to http://docs.search-guard.com/latest/configuring-tls for further configuration of your installation.

searchguard.ssl.transport.pemcert_filepath: node1.pem
searchguard.ssl.transport.pemkey_filepath: node1.key
searchguard.ssl.transport.pemkey_password: 6bbhtBZakqUT
searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: node1_http.pem
searchguard.ssl.http.pemkey_filepath: node1_http.key
searchguard.ssl.http.pemkey_password: dGaWWFn8pt33
searchguard.ssl.http.pemtrustedcas_filepath: root-ca.pem
searchguard.nodes_dn:
- CN=node1.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com
- CN=node2.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com
- CN=node3.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com
searchguard.authcz.admin_dn:
- CN=kirk.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

system · July 28, 2020, 4:28pm

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Elasticsearch failed to start after installing searchguard along with valid certificates Search Guard	4	588	April 3, 2018
Search Guard certificate error Search Guard	1	1191	February 1, 2018
Searchguard setting were not applied in Kibana Error Search Guard	4	1184	March 21, 2019
Unable to setup certificates Search Guard	2	1268	January 3, 2019
Please help configuring Elasticsearch with Search Guard Search Guard	6	1271	October 17, 2016

Elasticsearch is not starting for Search Guard error

Related Topics