Search Guard 7 - sgadmin initialization error when using in docker environment

indefsystems · May 10, 2019, 2:21pm

I am trying to use Search Guard 7 (35.0.0) with Elasticsearch 7.0.1.

I am creating an ELK cluster using docker swarm. I am able to build the image and start the cluster, however, when I log in to ES container to initialize sgadmin then I am getting the following error.

[root@382e4ee52e9b bin]# ./init_sg.sh
which: no java in (/usr/share/elasticsearch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin)
WARNING: JAVA_HOME not set, will use
../plugins/search-guard-7/tools/sgadmin.sh: line 25: java: command not found

The same setup works with SG6 (25.1) and ES 6.7.2.

I use a .env file for search guard, elasticsearch and kibana versions to build the docker images. I then deploy the stack in a docker swarm. Docker exec into the ES container and then initialize sgadmin using a custom init_sg.sh script copied during the docker image creation.

As I mentioned, the exact same steps work perfectly for SG6/ES 6.7.2 but fail for SG7/ES7.0.1.

Any advice on how to fix this? Thanks.

jkressin · May 10, 2019, 2:43pm

It seems strange to me that the exact setup worked with SG6. The reason for the issue is that you have not set JAVA_HOME and you also do not have java anywhere in the $PATH.

AFAIK we did not change anything here, so with the same base image and settings for Docker, it would have expected that you see the same issue also in SG6.

The relevant part of the sgadmin script:

BIN_PATH="java"

if [ -z "$JAVA_HOME" ]; then
    echo "WARNING: JAVA_HOME not set, will use $(which $BIN_PATH)"
else
    BIN_PATH="$JAVA_HOME/bin/java"
fi

According to the output, issuing

$ wich java

does not return any java executable. Which then explains

../plugins/search-guard-7/tools/sgadmin.sh: line 25: java: command not found

indefsystems · May 10, 2019, 3:06pm

Thanks for your reply @jkressin .

Here is the console output when using SG6

[ec2-user@ip-10-150-9-175 elk]$ docker exec -it 1a7d953d5ede bash
[root@1a7d953d5ede elasticsearch]# which java
/usr/bin/which: no java in (/usr/share/elasticsearch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin)
[root@1a7d953d5ede elasticsearch]# cd bin/
[root@1a7d953d5ede bin]# ./init_sg.sh
Search Guard Admin v6
Will connect to 10.150.9.175:9300 ... done
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.bouncycastle.jcajce.provider.drbg.DRBG (file:/usr/share/elasticsearch/plugins/search-guard-6/bcprov-jdk15on-1.61.jar) to constructor sun.security.provider.Sun()
WARNING: Please consider reporting this to the maintainers of org.bouncycastle.jcajce.provider.drbg.DRBG
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Elasticsearch Version: 6.7.2
Search Guard Version: 6.7.2-25.1
Connected as CN=ktelkadmin,OU=emea,O=Kuoni Tumlare,DC=kuoni,DC=int
Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
Clustername: docker-cluster
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
searchguard index does not exists, attempt to create it ... done (0-all replicas)
Populate config from /usr/share/elasticsearch/config/sg
Will update 'sg/config' with ../config/sg/sg_config.yml
   SUCC: Configuration for 'config' created or updated
Will update 'sg/roles' with ../config/sg/sg_roles.yml
   SUCC: Configuration for 'roles' created or updated
Will update 'sg/rolesmapping' with ../config/sg/sg_roles_mapping.yml
   SUCC: Configuration for 'rolesmapping' created or updated
Will update 'sg/internalusers' with ../config/sg/sg_internal_users.yml
   SUCC: Configuration for 'internalusers' created or updated
Will update 'sg/actiongroups' with ../config/sg/sg_action_groups.yml
   SUCC: Configuration for 'actiongroups' created or updated
Done with success
[root@1a7d953d5ede bin]#

As you can see. After I get into the container I issue ‘which java’ and get ‘no java in…’ message.
However, when I run init_sg.sh then it successfully initializes Search Guard.

The very same step fails for SG7 as mentioned earlier.

jkressin · May 10, 2019, 3:11pm

But can you please also check if $JAVA_HOME is eventually set in your ES/SG6 container?

indefsystems · May 10, 2019, 3:41pm

Hi,
Yes, it seems $JAVA_HOME is set within the container.

.....
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
searchguard index does not exists, attempt to create it ... done (0-all replicas)
Populate config from /usr/share/elasticsearch/config/sg
Will update 'sg/config' with ../config/sg/sg_config.yml
   SUCC: Configuration for 'config' created or updated
Will update 'sg/roles' with ../config/sg/sg_roles.yml
   SUCC: Configuration for 'roles' created or updated
Will update 'sg/rolesmapping' with ../config/sg/sg_roles_mapping.yml
   SUCC: Configuration for 'rolesmapping' created or updated
Will update 'sg/internalusers' with ../config/sg/sg_internal_users.yml
   SUCC: Configuration for 'internalusers' created or updated
Will update 'sg/actiongroups' with ../config/sg/sg_action_groups.yml
   SUCC: Configuration for 'actiongroups' created or updated
Done with success
[root@1a7d953d5ede bin]# echo $JAVA_HOME
/opt/jdk-12.0.1
[root@1a7d953d5ede bin]# which java
/usr/bin/which: no java in (/usr/share/elasticsearch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin)

Just checked SG7/ES7.0.1 container.

[root@0bde955395d5 bin]# ./init_sg.sh
which: no java in (/usr/share/elasticsearch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin)
WARNING: JAVA_HOME not set, will use
../plugins/search-guard-7/tools/sgadmin.sh: line 25: java: command not found
[root@0bde955395d5 bin]# echo $JAVA_HOME

[root@0bde955395d5 bin]# cd /opt
[root@0bde955395d5 opt]# ll
total 0

So looks this is down to how these two versions of ES are packaged?

jkressin · May 10, 2019, 4:08pm

Yes, I think so. Starting with ES7, Elastic comes bundled with a JRE based on OpenJDK 12. You can find a “jdk” folder in the ES download containing it:

./java --version
openjdk 12 2019-03-19
OpenJDK Runtime Environment (build 12+33)
OpenJDK 64-Bit Server VM (build 12+33, mixed mode, sharing)

which kind of seems odd to me since at the same time they advise against using it here:

Java 9, Java 10, and Java 12 are short term releases. We advise against using them unless you’re prepared to handle the fast release cadence this imposes.

(Support Matrix | Elastic)

indefsystems · May 10, 2019, 4:18pm

Ok, I get it now.

So for ES7 I will have to handle JDK setup within elasticsearch Dockerfile.

Many thanks for your help.

indefsystems · May 10, 2019, 11:29pm

Hi @jkressin,
I got past the earlier problem. however, now I am getting a different error while trying to initialize SG7.

ERR: Seems …/config/sg/sg_internal_users.yml is not in legacy format: …

Please see below.

   [root@10707a2c5956 bin]# ./init_sg.sh
Search Guard Admin v7
Will connect to 10.150.9.175:9300 ... done
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.bouncycastle.jcajce.provider.drbg.DRBG (file:/usr/share/elasticsearch/plugins/search-guard-7/bcprov-jdk15on-1.61.jar) to constructor sun.security.provider.Sun()
WARNING: Please consider reporting this to the maintainers of org.bouncycastle.jcajce.provider.drbg.DRBG
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Connected as CN=ktelkadmin,OU=emea,O=Kuoni Tumlare,DC=kuoni,DC=int
Elasticsearch Version: 7.0.1
Search Guard Version: 7.0.1-35.0.0
Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
Clustername: docker-cluster
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
searchguard index already exists, so we do not need to create one.
Legacy index 'searchguard' (ES 6) detected (or forced). You should migrate the configuration!
See https://docs.search-guard.com/latest/upgrading-6-7 for more details.
Populate config from /usr/share/elasticsearch/config/sg
Will update 'sg/config' with ../config/sg/sg_config.yml (legacy mode)
   SUCC: Configuration for 'config' created or updated
Will update 'sg/roles' with ../config/sg/sg_roles.yml (legacy mode)
   SUCC: Configuration for 'roles' created or updated
Will update 'sg/rolesmapping' with ../config/sg/sg_roles_mapping.yml (legacy mode)
   SUCC: Configuration for 'rolesmapping' created or updated
ERR: Seems ../config/sg/sg_internal_users.yml is not in legacy format: com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of java.util.ArrayList out of VALUE_STRING token
 at [Source: {"admin":{"readonly":true,"hash":"$2a$12$VcCDgh2NDk07JGN0rjGbM.Ad41qVR/YFJcgHp0UGns5JDymv..TOG","roles":["admin"]},"logstash":{"hash":"$2a$12$u1ShR4l4uBS3Uv59Pa2y5.1uQuZBrZtmNfqB3iM/.jL0XoV9sghS2","roles":["logstash"]},"kibanaserver":{"readonly":true,"hash":"$2a$12$4AcgAt3xwOWadA5s5blL6ev39OXDNhmOesEoo33eZtrq2N0YrU3H."},"kibanaro":{"hash":"$2a$12$JJSXNfTowz7Uu5ttXfeYpeYE0arACvcwlPBStB1F.MI7f0U9Z4DGC","roles":["kibanauser","readall"]},"readall":{"hash":"$2a$12$ae4ycwzwvLtZxwZ82RmiEunBbIPiAmGZduBAjKN0TXdwQFtCwARz2","roles":["readall"]},"snapshotrestore":{"hash":"$2y$12$DpwmetHKwgYnorbgdvORCenv4NAK8cPUg8AI6pxLCuWf/ALc0.v7W","roles":["snapshotrestore"]},"elastalert":{"hash":"$2y$12$ZsJ1bOt0MmyBe.OvZXhTC.hHuTkfMEovM6J1JDIZp/j0VuTmboIEK","roles":"elastalert"}}; line: 1, column: 751] (through reference chain: com.floragunn.searchguard.sgconf.impl.SgDynamicConfiguration["elastalert"]->com.floragunn.searchguard.sgconf.impl.v6.InternalUserV6["roles"])
Will update 'sg/actiongroups' with ../config/sg/sg_action_groups.yml (legacy mode)
   SUCC: Configuration for 'actiongroups' created or updated
ERR: cannot upload configuration, see errors above

Again this file is unchanged and works fine with SG6.
Any idea what could be wrong here? Thanks.

hsaly · May 11, 2019, 8:33pm

elastalert:
  hash: $2y$12$ZsJ1bOt0MmyBe.OvZXhTC.hHuTkfMEovM6J1JDIZp/j0VuTmboIEK
  roles: elastalert

should more look like

elastalert:
  hash: $2y$12$ZsJ1bOt0MmyBe.OvZXhTC.hHuTkfMEovM6J1JDIZp/j0VuTmboIEK
  roles:
    - elastalert

because roles is (and was ever) a list of strings

indefsystems · May 13, 2019, 10:14am

Thanks.
suggested change fixed the issue for ES7.0.1.

Mohamed · May 28, 2019, 3:16pm

Hello indefsystems

when you said “ES7 I will have to handle JDK setup within elasticsearch Dockerfile.” could you please tell me your Dockerfile after doing this ?

Thank you in advance!

system · June 18, 2019, 3:16pm

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Unable to do initial Search Guard configuration in Kubernetes cluster Search Guard	6	860	April 22, 2020
sgadmin error Search Guard	4	827	July 24, 2018
sgadmin initial "initializing" not working Search Guard	2	812	July 1, 2018
Can't initialize Search Guard Search Guard	5	1305	October 9, 2020
Not executing initial sgadmin Search Guard	3	2823	January 17, 2019

Search Guard 7 - sgadmin initialization error when using in docker environment

Related topics