java.lang.OutOfMemoryError: Required array size too large

[silentfilm]

Elasticsearch version:
7.17.9
Server OS version:
3.10.0-1160.88.1.el7.x86_64 #1 SMP Tue Mar 7 15:41:52 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Kibana version (if relevant):
7.17.9

Describe the issue:
I have several lab nodes that are running Elasticsearch and SearchGuard successfully. However one is getting an Out of Memory error as soon as Elasticsearch starts up. What array does not have enough memory?

2023-09-12 14:18:04.415Z INFO  [main] org.elasticsearch.node.Node - version[7.17.9], pid[1363], build[default/docker/ef48222227ee6b9e70e502f0f0daa52435ee634d/2023-01-31T05:34:43.305517834Z], OS[Linux/3.10.0-1160.62.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/19.0.2/19.0.2+7-44]
2023-09-12 14:18:04.417Z INFO  [main] org.elasticsearch.node.Node - JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
2023-09-12 14:18:04.417Z INFO  [main] org.elasticsearch.node.Node - JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Djava.security.manager=allow, -Xms8G, -Xmx8G, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-783719792716376994, -Dlog4j2.formatMsgNoLookups=true, -Des.cgroups.hierarchy.override=/, -XX:MaxDirectMemorySize=4294967296, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]
2023-09-12 14:18:06.226Z INFO  [main] com.floragunn.searchguard.SearchGuardPlugin - ES Config path is /config
2023-09-12 14:18:06.246Z WARN  [main] stderr - SLF4J: No SLF4J providers were found.
2023-09-12 14:18:06.247Z WARN  [main] stderr - SLF4J: Defaulting to no-operation (NOP) logger implementation
2023-09-12 14:18:06.247Z WARN  [main] stderr - SLF4J: See https://www.slf4j.org/codes.html#noProviders for further details.
2023-09-12 14:18:06.248Z WARN  [main] stderr - SLF4J: Class path contains SLF4J bindings targeting slf4j-api versions 1.7.x or earlier.
2023-09-12 14:18:06.248Z WARN  [main] stderr - SLF4J: Ignoring binding found at [jar:file:/usr/share/elasticsearch/plugins/search-guard-flx/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2023-09-12 14:18:06.248Z WARN  [main] stderr - SLF4J: See https://www.slf4j.org/codes.html#ignoredBindings for an explanation.
2023-09-12 14:18:06.313Z INFO  [main] com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore - JVM supports TLSv1.3
2023-09-12 14:18:06.315Z INFO  [main] com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore - Config directory is /config/, from there the key- and truststore files are resolved relatively
2023-09-12 14:18:06.490Z INFO  [main] com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore - TLS Transport Client Provider : JDK
2023-09-12 14:18:06.490Z INFO  [main] com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore - TLS Transport Server Provider : JDK
2023-09-12 14:18:06.490Z INFO  [main] com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore - TLS HTTP Provider             : JDK
2023-09-12 14:18:06.491Z INFO  [main] com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore - Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
2023-09-12 14:18:06.491Z INFO  [main] com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore - Enabled TLS protocols for HTTP layer      : [TLSv1.3, TLSv1.2]
2023-09-12 14:18:06.611Z INFO  [main] com.floragunn.searchguard.SearchGuardPlugin - Clustername: nba_elasticsearch_cluster
2023-09-12 14:18:07.034Z ERROR [main] org.elasticsearch.bootstrap.Bootstrap - Exception
java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:811) ~[elasticsearch-7.17.9.jar:7.17.9]
        at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:750) ~[elasticsearch-7.17.9.jar:7.17.9]
        at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:533) ~[elasticsearch-7.17.9.jar:7.17.9]
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:170) ~[elasticsearch-7.17.9.jar:7.17.9]
        at org.elasticsearch.node.Node.<init>(Node.java:411) ~[elasticsearch-7.17.9.jar:7.17.9]
        at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.17.9.jar:7.17.9]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.17.9.jar:7.17.9]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.17.9.jar:7.17.9]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) [elasticsearch-7.17.9.jar:7.17.9]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169) [elasticsearch-7.17.9.jar:7.17.9]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:160) [elasticsearch-7.17.9.jar:7.17.9]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) [elasticsearch-7.17.9.jar:7.17.9]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) [elasticsearch-cli-7.17.9.jar:7.17.9]
        at org.elasticsearch.cli.Command.main(Command.java:77) [elasticsearch-cli-7.17.9.jar:7.17.9]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:125) [elasticsearch-7.17.9.jar:7.17.9]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) [elasticsearch-7.17.9.jar:7.17.9]
Caused by: java.lang.reflect.InvocationTargetException
        at jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:79) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:484) ~[?:?]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:802) ~[elasticsearch-7.17.9.jar:7.17.9]
        ... 15 more
Caused by: java.lang.OutOfMemoryError: Required array size too large
        at java.nio.file.Files.readAllBytes(Files.java:3293) ~[?:?]
        at com.floragunn.searchguard.SearchGuardPlugin.sha256(SearchGuardPlugin.java:416) ~[?:?]
        at com.floragunn.searchguard.SearchGuardPlugin.access$400(SearchGuardPlugin.java:186) ~[?:?]
        at com.floragunn.searchguard.SearchGuardPlugin$4.lambda$run$0(SearchGuardPlugin.java:366) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
        at java.util.stream.DistinctOps$1$2.accept(DistinctOps.java:174) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
        at java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]
        at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1921) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
        at com.floragunn.searchguard.SearchGuardPlugin$4.run(SearchGuardPlugin.java:366) ~[?:?]
        at com.floragunn.searchguard.SearchGuardPlugin$4.run(SearchGuardPlugin.java:360) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
        at com.floragunn.searchguard.SearchGuardPlugin.<init>(SearchGuardPlugin.java:360) ~[?:?]
        at jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:67) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:484) ~[?:?]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:802) ~[elasticsearch-7.17.9.jar:7.17.9]
        ... 15 more

Provide configuration:
The memory configuration is displayed at the top of the log messages.

[Eugene7]

Hi @silentfilm

To reproduce the error, I need the SearchGuard and ElasticSearch configurations. Please send all your sgconfig files and elasticsearch.yml.

[silentfilm]

elasticsearch.yml (2.8 KB)
sg_roles_mapping.yml (1.2 KB)
sg_roles.yml (2.8 KB)
sg_internal_users.yml (2.3 KB)
sg_blocks.yml (51 Bytes)
sg_authz.yml (262 Bytes)
sg_authc.yml (889 Bytes)
sg_action_groups.yml (57 Bytes)
sg_tenants.yml (52 Bytes)

I changed the authorization DN for security reasons. Our lab guy found that if he set

searchguard.allow_unsafe_democertificates: true

Then the array size issue went away. We do not have any large files in the /config directory.

root@elasticsearch-aln-nbaperf-d18:/config# ls -l
total 48
drwxr-xr-x. 3 elasticsearch elasticsearch 4096 Sep 13 12:31 certs
-rw-------. 1 elasticsearch elasticsearch 3427 Sep 13 12:26 curator-actions.yml
-rw-------. 1 elasticsearch elasticsearch  606 Sep 13 12:26 curator-config.yml
-rw-------. 1 elasticsearch elasticsearch 2719 Sep 13 12:26 curator-snapshot-restore.yml
-rw-------. 1 elasticsearch elasticsearch 1218 Sep 13 12:26 curator-snapshot.yml
-rw-rw----. 1 elasticsearch elasticsearch  199 Sep 13 12:26 elasticsearch.keystore
-rw-------. 1 elasticsearch elasticsearch 2610 Sep 13 12:26 elasticsearch.standalone.yml
-rw-------. 1 elasticsearch elasticsearch 2887 Sep 13 12:27 elasticsearch.yml
-rw-------. 1 elasticsearch elasticsearch  301 Sep 13 12:26 env.config.sh
-rw-------. 1 elasticsearch elasticsearch  127 Sep 13 12:26 global-component-template.json
drwx------. 2 elasticsearch elasticsearch    6 Sep 13 12:25 index
drwx------. 2 elasticsearch elasticsearch    6 Sep 13 12:25 ingest-geoip
-rw-------. 1 elasticsearch elasticsearch 1586 Sep 13 12:26 jvm.options
-rw-------. 1 elasticsearch elasticsearch 2322 Sep 13 12:26 log4j2.properties
drwx------. 2 elasticsearch elasticsearch    6 Sep 13 12:25 pipelines
drwx------. 2 elasticsearch elasticsearch    6 Sep 13 12:25 scripts
drwx------. 2 elasticsearch elasticsearch  194 Sep 13 12:26 sg

[Eugene7]

How have you deployed the cluster? Could you share elasticsearch.yml of one of the working nodes?
Had you made any changes to the cluster before the error appeared?

[silentfilm]

This is a single-node Elasticsearch installation. It is running in a Docker container. I’m helping a coworker troubleshoot his system. He says that he encountered the error so he reinstalled Elasticsearch but the problem persisted.

---
# Elasticsearch Configuration
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html

#
# Cluster and Discovery
#
cluster.name: xxx_elasticsearch_cluster
discovery.seed_hosts: [ "aln-nbaperf-d18"  ]
discovery.type: single-node
path.repo: /usr/share/elasticsearch/aln-nbaperf-d18

#
# Node
#
node.roles: [ master, ingest, data ]
node.max_local_storage_nodes: 1
node.name: esnode-aln-nbaperf-d18

#
# Paths
#
cluster.routing.allocation.disk.threshold_enabled: false
path.data: /data
path.logs: /logs

#
# Internals
#
bootstrap.memory_lock: false
script.painless.regex.enabled: true
indices.query.bool.max_clause_count: 2048
signals.enabled: false
action.auto_create_index: true

#
# Network
#
network.host: 0.0.0.0
network.publish_host: aln-nbaperf-d18.labs.server.com

http.publish_host: aln-nbaperf-d18.labs.server.com
http.port: 9200
http.max_content_length: 100mb
http.compression: false

transport.publish_host: aln-nbaperf-d18.labs.server.com
transport.port: 9300
transport.compress: true

#
# Security General
#
searchguard.enterprise_modules_enabled: false
searchguard.allow_unsafe_democertificates: true
searchguard.config_index_name: searchguard
searchguard.roles_mapping_resolution: MAPPING_ONLY
searchguard.nodes_dn:
  - "CN=esnode-*,OU=nBA,O=Company,L=Location,ST=XX,C=US"
searchguard.authcz.admin_dn:
  - "CN=esadmin,OU=nBA,O=Company,L=Location,ST=XX,C=US"
xpack.security.enabled: false
xpack.security.transport.ssl.enabled: false
searchguard.restapi.roles_enabled:
  - "SGS_ALL_ACCESS"
  - "SGS_LOGSTASH"
  - "SGS_KIBANA_SERVER"
  - "SGS_KIBANA_USER"
  - "SGS_READALL"
  - "SGS_OWN_INDEX"
  - "SGS_MANAGE_SNAPSHOTS"
  - "sg_index_maintenance"
  - "sg_alerting"
  - "sg_metrics"
  - "sg_data_integrity"

#
# Transport Layer Security
#
searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.pemcert_filepath: certs/esnode.pem
searchguard.ssl.transport.pemkey_filepath: certs/esnode.key
searchguard.ssl.transport.pemkey_password: ${ES_ADMIN_PASSWORD}
searchguard.ssl.transport.pemtrustedcas_filepath: certs/cacert.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: false
searchguard.ssl.transport.enabled_protocols:
  - "TLSv1.2"
  - "TLSv1.3"

#
# REST Layer Security
#
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: certs/esnode.pem
searchguard.ssl.http.pemkey_filepath: certs/esnode.key
searchguard.ssl.http.pemkey_password: ${ES_ADMIN_PASSWORD}
searchguard.ssl.http.pemtrustedcas_filepath: certs/cacert.pem
searchguard.ssl.http.clientauth_mode: OPTIONAL
searchguard.ssl.http.enabled_protocols:
  - "TLSv1.2"
  - "TLSv1.3"

# Disable geoip indexing
ingest.geoip.downloader.enabled: false
# Increase the grok timeout
ingest.grok.watchdog.max_execution_time: 5s

[nils]

Good find, also great that you found a workaround. Still, I have filed an issue to fix this:

Node startup fails if there are files larger than 2 GB in the...