Search Guard Not Initiated on Elasticsearch Restart

Search Guard
1

Hi

I am using Elastic search version 6.2.4 and Search Guard 6.2.4-22.1 and I am running into an issue that when Elasticsearch restarts I have to run the sgadmin tool again. Is this normal? How can I fix this?

Thank You!

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version - 5.2.4-22.1

  • JVM version and operating system version - OpenJDK v 1.8.0_162 XUbuntu

  • Other installed Elasticsearch or Kibana plugins, if any - Kibana, X-pack monitoring, Elastalerts

2

No, absolutely not. You only need to execute it once, the config will be stored in an Elasticsearch index that of course survives restarts.

In order to help we need some more information about your setup.

  • is this a demo installation using the demo installer script?

  • is this a node node (PoC) setup or a multi node setup

  • is it a fresh install or an upgrade

  • do you execute sgadmin against a master or a data node? Should not matter, but just to make sure

  • is there anything in the ES logfiles? Can you send the logfile on debug level after you executed sgadmin? (TLS help | Security for Elasticsearch | Search Guard)

  • Any suspicious output of sgadmin?

···

On Tuesday, May 22, 2018 at 5:39:49 PM UTC+2, william.s@orangelogic.com wrote:

Hi

I am using Elastic search version 6.2.4 and Search Guard 6.2.4-22.1 and I am running into an issue that when Elasticsearch restarts I have to run the sgadmin tool again. Is this normal? How can I fix this?

Thank You!

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version - 5.2.4-22.1
  • JVM version and operating system version - OpenJDK v 1.8.0_162 XUbuntu
  • Other installed Elasticsearch or Kibana plugins, if any - Kibana, X-pack monitoring, Elastalerts
3

Hi Jochen,

This is not a demo install.

It is a single node setup.

It was a fresh install

Since it is a single node sgadmin is ran against the master.

The logfile is massive so I was not able to find anything that was suspicious.

Sometimes if I run sgadmin too soon after start up I get this:

" Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …

ERR: Timed out while waiting for a green or yellow cluster state.

  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)

  • Make also sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml

  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)

  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster"

One note that I didn’t think was relevant at first but just as a matter of full disclosure, this happens after Elastic Search crashes and I restart it. The crashes seem to be due to running out of RAM (due to Elastalert being a surprise RAM hog).

Thank you for helping,

Will

···

On Wednesday, May 23, 2018 at 12:57:49 AM UTC-7, Jochen Kressin wrote:

No, absolutely not. You only need to execute it once, the config will be stored in an Elasticsearch index that of course survives restarts.

In order to help we need some more information about your setup.

  • is this a demo installation using the demo installer script?
  • is this a node node (PoC) setup or a multi node setup
  • is it a fresh install or an upgrade
  • do you execute sgadmin against a master or a data node? Should not matter, but just to make sure
  • Any suspicious output of sgadmin?

On Tuesday, May 22, 2018 at 5:39:49 PM UTC+2, will...@orangelogic.com wrote:

Hi

I am using Elastic search version 6.2.4 and Search Guard 6.2.4-22.1 and I am running into an issue that when Elasticsearch restarts I have to run the sgadmin tool again. Is this normal? How can I fix this?

Thank You!

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version - 5.2.4-22.1
  • JVM version and operating system version - OpenJDK v 1.8.0_162 XUbuntu
  • Other installed Elasticsearch or Kibana plugins, if any - Kibana, X-pack monitoring, Elastalerts
4

So this seems to be more related to the ES crash than to Search Guard. If I understand you correctly, you only need to re-apply sgadmin after the crash, correct? We keep the SG configuration in a regular Elasticsearch index, so if it cannot be restored because the index is corrupt this might be the cause. If you just have a single node, then there are also no replica shards on other nodes.

If I did not understand this correctly and it also happens if ES did not crash, please let me know.

···

On Wednesday, May 23, 2018 at 6:34:09 PM UTC+2, william.s@orangelogic.com wrote:

Hi Jochen,

This is not a demo install.

It is a single node setup.

It was a fresh install

Since it is a single node sgadmin is ran against the master.

The logfile is massive so I was not able to find anything that was suspicious.

Sometimes if I run sgadmin too soon after start up I get this:

" Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …

ERR: Timed out while waiting for a green or yellow cluster state.

  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
  • Make also sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster"

One note that I didn’t think was relevant at first but just as a matter of full disclosure, this happens after Elastic Search crashes and I restart it. The crashes seem to be due to running out of RAM (due to Elastalert being a surprise RAM hog).

Thank you for helping,

Will

On Wednesday, May 23, 2018 at 12:57:49 AM UTC-7, Jochen Kressin wrote:

No, absolutely not. You only need to execute it once, the config will be stored in an Elasticsearch index that of course survives restarts.

In order to help we need some more information about your setup.

  • is this a demo installation using the demo installer script?
  • is this a node node (PoC) setup or a multi node setup
  • is it a fresh install or an upgrade
  • do you execute sgadmin against a master or a data node? Should not matter, but just to make sure
  • Any suspicious output of sgadmin?

On Tuesday, May 22, 2018 at 5:39:49 PM UTC+2, will...@orangelogic.com wrote:

Hi

I am using Elastic search version 6.2.4 and Search Guard 6.2.4-22.1 and I am running into an issue that when Elasticsearch restarts I have to run the sgadmin tool again. Is this normal? How can I fix this?

Thank You!

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version - 5.2.4-22.1
  • JVM version and operating system version - OpenJDK v 1.8.0_162 XUbuntu
  • Other installed Elasticsearch or Kibana plugins, if any - Kibana, X-pack monitoring, Elastalerts