Unable to do initial Search Guard configuration in Kubernetes cluster

Search Guard
1

TL;DR

Unable to do initial sgadmin load in Elastic cluster deployed via Kubernetes.

The rest of the story…

I am configuring SearchGuard for the first time in a Kubernetes cluster. My image details are:

  • Elasticsearch 7.6.0

  • Custom image, built on top of stock elasticsearch:7.6.0, with search-guard-7 plugin installed

  • Deploying via Elastic’s Helm charts (v7.6.0), overriding the image and imageTag to use my custom image.

My problem lies in doing my initial sgadmin configuration. Since this is in Kubernetes, I’m using the standalone sgadmin command, because the stock elasticsearch image doesn’t have Java installed in it (as near as I can tell), and it defaults to running as the elasticsearch user, which can’t run sgadmin.sh anyway. Therefore, I’ve built a separate image which I can deploy and remote into to do the load from, and I’ve deployed it in the same namespace.

I’m reasonably certain that I have Search Guard configured correctly, as my log keeps showing the following message:

{"type": "server", "timestamp": "2020-03-12T19:51:16,162Z", "level": "ERROR", "component": "c.f.s.a.BackendRegistry", "cluster.name": "elk-es", "node.name": "elk-es-master-0", "message": "Not yet initialized (you may need to run sgadmin)", "cluster.uuid": "4V0IYd-uTi-Aa0eRNVp-fA", "node.id": "C44ccuOHTiaX94LIqMeyfg"  }

Here is my issue:

  • Port 9300 is open in the service. When I do not have Search Guard installed, my cluster starts running without any issues, and it was running just fine under xpack.security as well.

  • When executing sgadmin.sh -h <service-name> ... I receive the following error:

WARNING: JAVA_HOME not set, will use /opt/openjdk/bin/java
Search Guard Admin v7
Will connect to <service-name>:9300
ERR: Seems there is no Elasticsearch running on <service-name>:9300 - Will exit

  • When executing the following command from my sgadmin pod: https://<service-name>:9200/_searchguard/authinfo I receive the following message: curl: (7) Failed connect to <service-name>:9200; Connection refused

  • When executing the following command from an elasticsearch pod: https://localhost:9200/_searchguard/authinfo I receive the following message: Search Guard not initialized (SG11). See https://docs.search-guard.com/latest/sgadmin which is what I would expect to see at this point.

  • When executing the following command from the elasticsearch pod: https://<service-name>:9200/_searchguard/authinfo I receive the following message: curl: (7) Failed connect to <service-name>:9200; Connection refused

It appears as if Elasticsearch isn’t listening on 0.0.0.0, except I’ve validated that it’s running with -Enetwork.host=0.0.0.0.

2

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

3

Do you have JAVA_HOME environment variable set? Check it

echo $JAVA_HOME

A tip, you can use sdkman to make Java setup and management easier.

Did you expose the Elasticsearch ports? Check TCP connection without Elasticsearch first using netcat. For example

  1. Run the netcat server on host A
nc -l -p 9200
  1. Run the netcat client on host B
nc hostA.com 9200
  1. If the connection is successful, you see a prompt, type some words there. Then go to host A, you should see the words you typed.
6

The answer to all of the above was “yes.”

I ended up making the sgtools/ folder executable by the elasticsearch user, which allowed me to do it from the localhost. At that point, I ran into an issue where it appeared from the output that everything was configured correctly, but it wasn’t configuring. The issue there ended up being that I was attempting to use the ${envbc.<variable name>} form in my yaml files to hash my passwords on the fly.

This did not work, but once I hashed them myself, using the hash.sh script, and then ran sgadmin on the master node, everything worked as expected.

Thanks!

7

@Doug_Renze I’m glad you solved the issue. Thanks for the update.

1 Like