OOM when starting data node after completing the provisioning of search guard

Hello, I have got an issue with starting data node provisioned with search guard, the error shown in the log as below, and I wouldn’t get such error if no data exist in the node but I did get the error when my data node has a used disk space of 150GB.
[2020-02-29T06:56:53,916][WARN ][c.f.s.SearchGuardPlugin ] [data_es3] File /data/data_node/nodes/0/_state/node-305.st has insecure file permissions (should be 0600)
[2020-02-29T06:57:23,087][ERROR][o.e.b.Bootstrap ] [data_es3] Exception
java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.plugins.PluginsService.(PluginsService.java:163) ~[elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.node.Node.(Node.java:306) ~[elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.node.Node.(Node.java:251) ~[elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:211) ~[elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) [elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.0.1.jar:7.0.1]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.0.1.jar:7.0.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.0.1.jar:7.0.1]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_221]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.0.1.jar:7.0.1]
… 15 more
Caused by: java.lang.OutOfMemoryError: Required array size too large
When posting in this category, please add:

  • Elasticsearch logfiles on debug level

  • Your Search Guard configuration files

  • Your elasticsearch.yml configuration file
    cluster.name: sg_es
    node.name: data_es3
    path.data: /data/data_node
    path.logs: /data/data_node/logs
    network.host: 0.0.0.0
    transport.tcp.port: 9301
    discovery.seed_hosts: [“10.20.16.165:9201”,“10.20.16.165:9301”,“10.20.16.165:9401”,“10.20.16.165:9501”,“10
    .20.28.98:9201”,“10.20.28.98:9301”,“10.20.28.98:9401”,“10.20.28.98:9501”,“10.20.31.244:9201”,“10.20.31.244
    :9301”,“10.20.31.244:9401”,“10.20.31.244:9501”]
    cluster.initial_master_nodes: [“10.20.16.165:9201”,“10.20.28.98:9201”,“10.20.31.244:9201”]
    node.master: true
    node.data: true
    node.ingest: false
    node.ml: false
    xpack.security.enabled: false
    searchguard.ssl.transport.enabled: true
    searchguard.ssl.transport.pemcert_filepath: /data/data_node/out/data_es3.pem
    searchguard.ssl.transport.pemkey_filepath: /data/data_node/out/data_es3.key
    searchguard.ssl.transport.pemtrustedcas_filepath: /data/data_node/out/root-ca.pem
    searchguard.ssl.transport.enforce_hostname_verification: false
    searchguard.ssl.transport.resolve_hostname: false
    searchguard.ssl.http.enabled: true
    searchguard.ssl.http.pemcert_filepath: /data/data_node/out/data_es3_http.pem
    searchguard.ssl.http.pemkey_filepath: /data/data_node/out/data_es3_http.key
    searchguard.ssl.http.pemtrustedcas_filepath: /data/data_node/out/root-ca.pem
    searchguard.nodes_dn:

  • CN=master_es1.example.com,OU=Ops,O=Example Com, Inc.,DC=example,DC=com
  • CN=data_es1.example.com,OU=Ops,O=Example Com, Inc.,DC=example,DC=com
  • CN=master_es2.example.com,OU=Ops,O=Example Com, Inc.,DC=example,DC=com
  • CN=data_es2.example.com,OU=Ops,O=Example Com, Inc.,DC=example,DC=com
  • CN=master_es3.example.com,OU=Ops,O=Example Com, Inc.,DC=example,DC=com
  • CN=data_es3.example.com,OU=Ops,O=Example Com, Inc.,DC=example,DC=com
    searchguard.authcz.admin_dn:
  • CN=kirk.example.com,OU=Ops,O=Example Com, Inc.,DC=example,DC=com
    searchguard.cert.oid: 1.2.3.4.5.5

[JVM.option]

JVM configuration

################################################################

IMPORTANT: JVM heap size

################################################################

You should always set the min and max JVM heap

size to the same value. For example, to set

the heap to 4 GB, set:

-Xms4g

-Xmx4g

See Heap size settings | Elasticsearch Guide [8.4] | Elastic

for more information

################################################################

Xms represents the initial size of total heap space

Xmx represents the maximum size of total heap space

-Xms14g
-Xmx14g

################################################################

Expert settings

################################################################

All settings below this section are considered

expert settings. Don’t tamper with them unless

you understand what you are doing

################################################################

GC configuration

-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly

G1GC Configuration

NOTE: G1GC is only supported on JDK version 10 or later.

To use G1GC uncomment the lines below.

10-:-XX:-UseConcMarkSweepGC

10-:-XX:-UseCMSInitiatingOccupancyOnly

10-:-XX:+UseG1GC

10-:-XX:InitiatingHeapOccupancyPercent=75

DNS cache policy

cache ttl in seconds for positive DNS lookups noting that this overrides the

JDK security property networkaddress.cache.ttl; set to -1 to cache forever

-Des.networkaddress.cache.ttl=60

cache ttl in seconds for negative DNS lookups noting that this overrides the

JDK security property networkaddress.cache.negative ttl; set to -1 to cache

forever

-Des.networkaddress.cache.negative.ttl=10

optimizations

pre-touch memory pages used by the JVM during initialization

-XX:+AlwaysPreTouch

basic

explicitly set the stack size

-Xss1m

set to headless, just in case

-Djava.awt.headless=true

ensure UTF-8 encoding by default (e.g. filenames)

-Dfile.encoding=UTF-8

use our provided JNA always versus the system one

-Djna.nosys=true

turn off a JDK optimization that throws away stack traces for common

exceptions because stack traces are important for debugging

-XX:-OmitStackTraceInFastThrow

flags to configure Netty

-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0

log4j 2

-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true

#-Djava.io.tmpdir=${ES_TMPDIR}
-Djava.io.tmpdir=/data/data_node/javaiotmp

heap dumps

generate a heap dump when an allocation from the Java heap fails

heap dumps are created in the working directory of the JVM

-XX:+HeapDumpOnOutOfMemoryError

specify an alternative path for heap dumps; ensure the directory exists and

has sufficient space

-XX:HeapDumpPath=/data/data_node/logs

specify an alternative path for JVM fatal error logs

-XX:ErrorFile=/data/data_node/logs/hs_err_pid%p.log

JDK 8 GC logging

8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:/data/data_node/logs/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m

8:-XX:ReservedCodeCacheSize=128m
8:-XX:+PrintFlagsFinal
8:-XX:+UseCompressedOops

JDK 9+ GC logging

9-:-Xlog:gc*,gc+age=trace,safepoint:file=/data/data_node/logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m

due to internationalization enhancements in JDK 9 Elasticsearch need to set the provider to COMPAT otherwise

time/date parsing will break in an incompatible way for some date patterns and locals

9-:-Djava.locale.providers=COMPAT
If you are using Kibana, please also add:

  • Your kibana.yml configuration file

(You can just drag a file from your local file manager to the compose window)

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.