Searchguard cluster dead after installing DLS/FLS

Anthony_Cleaves · October 10, 2017, 9:37am

So, I popped the jar into the plugins folder and rebooted ES on each node and now the cluster is stuck in a red state.

Curling the cluster health states:

Search Guard not initialized (SG11)

The logs suggest the same.

[2017-10-10T09:31:52,370][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [elastic-master-01.x-x.com] updating number_of_replicas to [1] for indices [searchguard]

[2017-10-10T09:31:52,384][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [elastic-master-01.x-x.com] [searchguard/MfRS8XLmSoOGul-OHMMaDQ] auto expanded replicas to [1]

[2017-10-10T09:31:57,063][ERROR][c.f.s.a.BackendRegistry ] Not yet initialized (you may need to run sgadmin)

[2017-10-10T09:32:17,684][WARN ][c.f.s.c.IndexBaseConfigurationRepository] index ‘searchguard’ not healthy yet, we try again … (Reason: timeout)

Re-running the sgadmin tool just hangs and eventually fails with a timeout. What logs can I look at to figure out why this has happened? This seems to happen almost every time something in a config is changed. Am I doing something wrong somewhere?

I am running sgadmin in the exact same was as I do it on cluster setup.

···

*/bin/bash /usr/share/elasticsearch/plugins/search-guard-5/tools/sgadmin.sh *

*-cd /usr/share/elasticsearch/plugins/search-guard-5/sgconfig *

*-hostname elastic-master-01.x-x.com *

*-cn x-cluster *

*-cert /etc/elasticsearch/elastic-master-01.x-x.com/elastic-admin.pem *

*-cacert /etc/elasticsearch/elastic-master-01.x-x.com/ca-bundle.pem *

*-key /etc/elasticsearch/elastic-master-01.x-x.com/elastic-admin-key.pkcs8 *

-keypass x

jkressin · October 10, 2017, 12:27pm

“rebooted ES on each node and now the cluster is stuck in a red state”

So, the first thing to figure out is why the cluster state is red. sgadmin will refuse to upload configurations if the cluster state is not yellow at least. You can force sgadmin to push the configs to cluster in red state by adding the -arc (accept red cluster state) flag:

To find out why the cluster state is red and why it happened after you added the DLS/FLS jar, please check first if you installed the jar-with-depencies. Installing the plain jar without the dependencies will not work:

"If you want to use any of the enterprise modules, simply download the respective module jar file from Maven. Whend downloading, choose jar with dependencies … "

If the module is correctly installed, you can also set Search Guard to debug level and post you logfile here for further analysis. To enable debug mode, add the following lines to conf/log4j2.properties:

logger.fg.name = com.floragunn

logger.fg.level = debug

Finally, to check whether the cluster state is red, you can temporarily disable Search Guard and perform the usual checks, as you would do without SG:

···

On Tuesday, October 10, 2017 at 11:37:43 AM UTC+2, anthony.cleaves@actual-experience.com wrote: