Hello, I have got an issue with starting data node provisioned with search guard, the error shown in the log as below, and I wouldn’t get such error if no data exist in the node but I did get the error when my data node has a used disk space of 150GB.
[2020-02-29T06:56:53,916][WARN ][c.f.s.SearchGuardPlugin ] [data_es3] File /data/data_node/nodes/0/_state/node-305.st has insecure file permissions (should be 0600)
[2020-02-29T06:57:23,087][ERROR][o.e.b.Bootstrap ] [data_es3] Exception
java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.plugins.PluginsService.(PluginsService.java:163) ~[elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.node.Node.(Node.java:306) ~[elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.node.Node.(Node.java:251) ~[elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:211) ~[elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) [elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.0.1.jar:7.0.1]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.0.1.jar:7.0.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.0.1.jar:7.0.1]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_221]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.0.1.jar:7.0.1]
… 15 more
Caused by: java.lang.OutOfMemoryError: Required array size too large
When posting in this category, please add:
-
Your Search Guard configuration files
-
Your elasticsearch.yml configuration file
cluster.name: sg_es
node.name: data_es3
path.data: /data/data_node
path.logs: /data/data_node/logs
network.host: 0.0.0.0
transport.tcp.port: 9301
discovery.seed_hosts: [“10.20.16.165:9201”,“10.20.16.165:9301”,“10.20.16.165:9401”,“10.20.16.165:9501”,“10
.20.28.98:9201”,“10.20.28.98:9301”,“10.20.28.98:9401”,“10.20.28.98:9501”,“10.20.31.244:9201”,“10.20.31.244
:9301”,“10.20.31.244:9401”,“10.20.31.244:9501”]
cluster.initial_master_nodes: [“10.20.16.165:9201”,“10.20.28.98:9201”,“10.20.31.244:9201”]
node.master: true
node.data: true
node.ingest: false
node.ml: false
xpack.security.enabled: false
searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.pemcert_filepath: /data/data_node/out/data_es3.pem
searchguard.ssl.transport.pemkey_filepath: /data/data_node/out/data_es3.key
searchguard.ssl.transport.pemtrustedcas_filepath: /data/data_node/out/root-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: /data/data_node/out/data_es3_http.pem
searchguard.ssl.http.pemkey_filepath: /data/data_node/out/data_es3_http.key
searchguard.ssl.http.pemtrustedcas_filepath: /data/data_node/out/root-ca.pem
searchguard.nodes_dn:
- CN=master_es1.example.com,OU=Ops,O=Example Com, Inc.,DC=example,DC=com
- CN=data_es1.example.com,OU=Ops,O=Example Com, Inc.,DC=example,DC=com
- CN=master_es2.example.com,OU=Ops,O=Example Com, Inc.,DC=example,DC=com
- CN=data_es2.example.com,OU=Ops,O=Example Com, Inc.,DC=example,DC=com
- CN=master_es3.example.com,OU=Ops,O=Example Com, Inc.,DC=example,DC=com
- CN=data_es3.example.com,OU=Ops,O=Example Com, Inc.,DC=example,DC=com
searchguard.authcz.admin_dn: -
CN=kirk.example.com,OU=Ops,O=Example Com, Inc.,DC=example,DC=com
searchguard.cert.oid: 1.2.3.4.5.5
[JVM.option]
JVM configuration
################################################################
IMPORTANT: JVM heap size
################################################################
You should always set the min and max JVM heap
size to the same value. For example, to set
the heap to 4 GB, set:
-Xms4g
-Xmx4g
See Heap size settings | Elasticsearch Guide [8.4] | Elastic
for more information
################################################################
Xms represents the initial size of total heap space
Xmx represents the maximum size of total heap space
-Xms14g
-Xmx14g
################################################################
Expert settings
################################################################
All settings below this section are considered
expert settings. Don’t tamper with them unless
you understand what you are doing
################################################################
GC configuration
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
G1GC Configuration
NOTE: G1GC is only supported on JDK version 10 or later.
To use G1GC uncomment the lines below.
10-:-XX:-UseConcMarkSweepGC
10-:-XX:-UseCMSInitiatingOccupancyOnly
10-:-XX:+UseG1GC
10-:-XX:InitiatingHeapOccupancyPercent=75
DNS cache policy
cache ttl in seconds for positive DNS lookups noting that this overrides the
JDK security property networkaddress.cache.ttl; set to -1 to cache forever
-Des.networkaddress.cache.ttl=60
cache ttl in seconds for negative DNS lookups noting that this overrides the
JDK security property networkaddress.cache.negative ttl; set to -1 to cache
forever
-Des.networkaddress.cache.negative.ttl=10
optimizations
pre-touch memory pages used by the JVM during initialization
-XX:+AlwaysPreTouch
basic
explicitly set the stack size
-Xss1m
set to headless, just in case
-Djava.awt.headless=true
ensure UTF-8 encoding by default (e.g. filenames)
-Dfile.encoding=UTF-8
use our provided JNA always versus the system one
-Djna.nosys=true
turn off a JDK optimization that throws away stack traces for common
exceptions because stack traces are important for debugging
-XX:-OmitStackTraceInFastThrow
flags to configure Netty
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
log4j 2
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
#-Djava.io.tmpdir=${ES_TMPDIR}
-Djava.io.tmpdir=/data/data_node/javaiotmp
heap dumps
generate a heap dump when an allocation from the Java heap fails
heap dumps are created in the working directory of the JVM
-XX:+HeapDumpOnOutOfMemoryError
specify an alternative path for heap dumps; ensure the directory exists and
has sufficient space
-XX:HeapDumpPath=/data/data_node/logs
specify an alternative path for JVM fatal error logs
-XX:ErrorFile=/data/data_node/logs/hs_err_pid%p.log
JDK 8 GC logging
8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:/data/data_node/logs/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m
8:-XX:ReservedCodeCacheSize=128m
8:-XX:+PrintFlagsFinal
8:-XX:+UseCompressedOops
JDK 9+ GC logging
9-:-Xlog:gc*,gc+age=trace,safepoint:file=/data/data_node/logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m
due to internationalization enhancements in JDK 9 Elasticsearch need to set the provider to COMPAT otherwise
time/date parsing will break in an incompatible way for some date patterns and locals
9-:-Djava.locale.providers=COMPAT
If you are using Kibana, please also add:
- Your kibana.yml configuration file
(You can just drag a file from your local file manager to the compose window)