I am trying to use SearchGuard only for TLS on transport and HTTP. I’ve run into a strange issue where it works on a single-node cluster, but fails on startup in a multi-node cluster with a seemingly contradictory error. The two clusters are atla and atlr, respectively. You can see that the configurations should be identical by the diff between their config files below. I’ve also included the full configuration for atlr at the bottom. Why is it telling me I need to set searchguard.ssl.transport.pemkey_filepath when I clearly have it set?
$ diff playbooks/files/atl{a,r}-elasticsearch.yml
17c17
< cluster.name: atla-moloch
---
> cluster.name: atlr-moloch
70c70,73
< - atla-mlch-elastic-001.domainredacted.com
---
> - atlr-mlch-elastic-003.domainredacted.com
> - atlr-mlch-elastic-004.domainredacted.com
> - atlr-mlch-elastic-005.domainredacted.com
> - atlr-mlch-elastic-006.domainredacted.com
74c77
< discovery.zen.minimum_master_nodes: 1
---
> discovery.zen.minimum_master_nodes: 2
93,95c96,98
< searchguard.ssl.transport.pemkey_filepath: /etc/ssl/certs/domainredacted/atla-moloch.key
< searchguard.ssl.transport.pemcert_filepath: /etc/ssl/certs/domainredacted/atla-moloch.crt
< searchguard.ssl.transport.pemtrustedcas_filepath: /etc/ssl/certs/domainredacted/atla-moloch.ca
---
> searchguard.ssl.transport.pemkey_filepath: /etc/ssl/certs/domainredacted/atlr-moloch.key
> searchguard.ssl.tarnsport.pemcert_filepath: /etc/ssl/certs/domainredacted/atlr-moloch.crt
> searchguard.ssl.transport.pemtrustedcas_filepath: /etc/ssl/certs/domainredacted/atlr-moloch.ca
99,101c102,104
< searchguard.ssl.http.pemkey_filepath: /etc/ssl/certs/domainredacted/atla-moloch.key
< searchguard.ssl.http.pemcert_filepath: /etc/ssl/certs/domainredacted/atla-moloch.crt
< searchguard.ssl.http.pemtrustedcas_filepath: /etc/ssl/certs/domainredacted/atla-moloch.ca
---
> searchguard.ssl.http.pemkey_filepath: /etc/ssl/certs/domainredacted/atlr-moloch.key
> searchguard.ssl.http.pemcert_filepath: /etc/ssl/certs/domainredacted/atlr-moloch.crt
> searchguard.ssl.http.pemtrustedcas_filepath: /etc/ssl/certs/domainredacted/atlr-moloch.ca
117c120,123
< - atla-mlch-elastic-001.domainredacted.com
---
> - atlr-mlch-elastic-003.domainredacted.com
> - atlr-mlch-elastic-004.domainredacted.com
> - atlr-mlch-elastic-005.domainredacted.com
> - atlr-mlch-elastic-006.domainredacted.com
Full log:
Aug 28 10:31:51 hostnameredacted systemd[1]: elasticsearch.service failed.
Aug 28 10:36:21 hostnameredacted systemd[1]: Started Elasticsearch.
-- Subject: Unit elasticsearch.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has finished starting up.
--
-- The start-up result is done.
Aug 28 10:36:38 hostnameredacted elasticsearch[31191]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Aug 28 10:36:38 hostnameredacted elasticsearch[31191]: OpenJDK 64-Bit Server VM warning: UseAVX=2 is not supported on this CPU, setting it to UseAVX=1
Aug 28 10:36:53 hostnameredacted elasticsearch[31191]: [2019-08-28T10:36:53,337][INFO ][o.e.e.NodeEnvironment ] [hostnameredacted] using [1] data paths, mounts [[/home (/dev/mapper/centos_atlr--mlch--elastic--003-home)]], net usable_space [31.9tb], net total_space [36.3tb], types [xfs]
Aug 28 10:36:53 hostnameredacted elasticsearch[31191]: [2019-08-28T10:36:53,342][INFO ][o.e.e.NodeEnvironment ] [hostnameredacted] heap size [30.7gb], compressed ordinary object pointers [true]
Aug 28 10:36:53 hostnameredacted elasticsearch[31191]: [2019-08-28T10:36:53,430][INFO ][o.e.n.Node ] [hostnameredacted] node name [hostnameredacted], node ID [GZCfy5uAQWuAf0sIkyRC2g], cluster name [atlr-moloch]
Aug 28 10:36:53 hostnameredacted elasticsearch[31191]: [2019-08-28T10:36:53,431][INFO ][o.e.n.Node ] [hostnameredacted] version[7.3.1], pid[31191], build[oss/rpm/4749ba6/2019-08-19T20:19:25.651794Z], OS[Linux/3.10.0-957.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12.0.2/12.0.2+9]
Aug 28 10:36:53 hostnameredacted elasticsearch[31191]: [2019-08-28T10:36:53,431][INFO ][o.e.n.Node ] [hostnameredacted] JVM home [/usr/lib/jvm/java-12-openjdk-12.0.2.9-1.rolling.el7.x86_64]
Aug 28 10:36:53 hostnameredacted elasticsearch[31191]: [2019-08-28T10:36:53,432][INFO ][o.e.n.Node ] [hostnameredacted] JVM arguments [-Xms31g, -Xmx31g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-12031765025563705772, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Dio.netty.allocator.type=pooled, -XX:MaxDirectMemorySize=16642998272, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=oss, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: [2019-08-28T10:36:54,151][INFO ][c.f.s.SearchGuardPlugin ] [hostnameredacted] ES Config path is /etc/elasticsearch
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: [2019-08-28T10:36:54,203][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] [hostnameredacted] OpenSSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSL
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: [2019-08-28T10:36:54,298][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] [hostnameredacted] JVM supports TLSv1.3
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: [2019-08-28T10:36:54,299][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] [hostnameredacted] Config directory is /etc/elasticsearch/, from there the key- and truststore files are resolved relatively
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: [2019-08-28T10:36:54,380][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [hostnameredacted] uncaught exception in thread [main]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: Caused by: java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.node.Node.<init>(Node.java:314) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: ... 6 more
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: Caused by: java.lang.reflect.InvocationTargetException
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.node.Node.<init>(Node.java:314) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: ... 6 more
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: Caused by: org.elasticsearch.ElasticsearchException: searchguard.ssl.transport.keystore_filepath or searchguard.ssl.transport.pemkey_filepath must be set if transport ssl is reqested.
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:355) ~[?:?]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.<init>(DefaultSearchGuardKeyStore.java:151) ~[?:?]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.<init>(SearchGuardSSLPlugin.java:194) ~[?:?]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at com.floragunn.searchguard.SearchGuardPlugin.<init>(SearchGuardPlugin.java:212) ~[?:?]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.node.Node.<init>(Node.java:314) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 28 10:36:54 hostnameredacted elasticsearch[31191]: ... 6 more
Aug 28 10:36:57 hostnameredacted systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Aug 28 10:36:57 hostnameredacted systemd[1]: Unit elasticsearch.service entered failed state.
Aug 28 10:36:57 hostnameredacted systemd[1]: elasticsearch.service failed.
elasticsearch.yml config file:
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: atlr-moloch
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /home/elasticsearch/data
#
# Path to log files:
#
path.logs: /home/elasticsearch/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: _site_
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
discovery.zen.ping.unicast.hosts:
- atlr-mlch-elastic-003.domainredacted.com
- atlr-mlch-elastic-004.domainredacted.com
- atlr-mlch-elastic-005.domainredacted.com
- atlr-mlch-elastic-006.domainredacted.com
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
discovery.zen.minimum_master_nodes: 2
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
searchguard.ssl_only: true
searchguard.ssl.transport.pemkey_filepath: /etc/ssl/certs/domainredacted/atlr-moloch.key
searchguard.ssl.tarnsport.pemcert_filepath: /etc/ssl/certs/domainredacted/atlr-moloch.crt
searchguard.ssl.transport.pemtrustedcas_filepath: /etc/ssl/certs/domainredacted/atlr-moloch.ca
searchguard.ssl.transport.enforce_hostname_verification: true
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemkey_filepath: /etc/ssl/certs/domainredacted/atlr-moloch.key
searchguard.ssl.http.pemcert_filepath: /etc/ssl/certs/domainredacted/atlr-moloch.crt
searchguard.ssl.http.pemtrustedcas_filepath: /etc/ssl/certs/domainredacted/atlr-moloch.ca
cluster.routing.allocation.disk.watermark.low: 97%
cluster.routing.allocation.disk.watermark.high: 98%
cluster.routing.allocation.disk.watermark.flood_stage: 99%
action.search.shard_count.limit: 100000
thread_pool.write.queue_size: 2000
http.compression: true
cluster.routing.allocation.cluster_concurrent_rebalance: 10
cluster.routing.allocation.node_concurrent_recoveries: 5
cluster.routing.allocation.node_initial_primaries_recoveries: 5
indices.recovery.max_bytes_per_sec: "400mb"
cluster.max_shards_per_node: 1000000
cluster.initial_master_nodes:
- atlr-mlch-elastic-003.domainredacted.com
- atlr-mlch-elastic-004.domainredacted.com
- atlr-mlch-elastic-005.domainredacted.com
- atlr-mlch-elastic-006.domainredacted.com
Certs do exist and are readable:
# ls -l /etc/ssl/certs/domainredacted/
total 40
-rw-------. 1 elasticsearch elasticsearch 8558 Aug 28 09:52 atlr-moloch.ca
-rw-------. 1 elasticsearch elasticsearch 7517 Aug 28 09:52 atlr-moloch.crt
-rw-------. 1 elasticsearch elasticsearch 16075 Aug 28 09:52 atlr-moloch.fullchain.crt
-rw-------. 1 elasticsearch elasticsearch 1708 Aug 28 09:52 atlr-moloch.key
Why is it telling me I need this setting even though I clearly have it set?