Search Guard not initialized (SG11)

Error message when running the sgadmin tool :

WARNING: JAVA_HOME not set, will use /usr/bin/java
Search Guard Admin v6
Will connect to localhost:9300 … done
Elasticsearch Version: 6.2.2
Search Guard Version: 6.2.2-22.1
Connected as CN=admin-dev
Fail-fast is activated
Diagnostic trace written to: /opt/elasticsearch-6.2.2/plugins/search-guard-6/tools/sgadmin_diag_trace_2018-May-16_02-39-49.txt
Contacting elasticsearch cluster ‘elasticsearch’ …
Clustername: dcat-elasticsearch
Clusterstate: RED
Number of nodes: 5
Number of data nodes: 3
searchguard index already exists, so we do not need to create one.
ERR: searchguard index state is RED.
Populate config from /opt/elasticsearch-6.2.2/plugins/search-guard-6/sgconfig
Will update ‘sg/config’ with …/sgconfig/sg_config.yml
Will update ‘sg/config’ with …/sgconfig/sg_config.yml FAIL: Configuration for ‘config’ failed because of UnavailableShardsException[[searchguard][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[searchguard][0]] containing [index {[searchguard][sg][config], source[n/a, actual length: [3.2kb], max length: 2kb]}] and a refresh]]
Will update ‘sg/roles’ with …/sgconfig/sg_roles.yml
FAIL: Configuration for ‘roles’ failed because of UnavailableShardsException[[searchguard][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[searchguard][0]] containing [index {[searchguard][sg][roles], source[n/a, actual length: [3.6kb], max length: 2kb]}] and a refresh]]
Will update ‘sg/rolesmapping’ with …/sgconfig/sg_roles_mapping.yml
FAIL: Configuration for ‘rolesmapping’ failed because of UnavailableShardsException[[searchguard][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[searchguard][0]] containing [index {[searchguard][sg][rolesmapping], source[{“rolesmapping”:“eyJzZ19hbGxfYWNjZXNzIjp7InJlYWRvbmx5Ijp0cnVlLCJiYWNrZW5kcm9sZXMiOlsiYWRtaW4iXX0sInNnX2xvZ3N0YXNoIjp7ImJhY2tlbmRyb2xlcyI6WyJsb2dzdGFzaCJdfSwic2dfa2liYW5hX3NlcnZlciI6eyJyZWFkb25seSI6dHJ1ZSwidXNlcnMiOlsia2liYW5hc2VydmVyIl19LCJzZ19raWJhbmFfdXNlciI6eyJiYWNrZW5kcm9sZXMiOlsia2liYW5hdXNlciJdfSwic2dfcmVhZGFsbCI6eyJyZWFkb25seSI6dHJ1ZSwiYmFja2VuZHJvbGVzIjpbInJlYWRhbGwiXX0sInNnX21hbmFnZV9zbmFwc2hvdHMiOnsicmVhZG9ubHkiOnRydWUsImJhY2tlbmRyb2xlcyI6WyJzbmFwc2hvdHJlc3RvcmUiXX0sInNnX293bl9pbmRleCI6eyJ1c2VycyI6WyIqIl19fQ==”}]}] and a refresh]]
Will update ‘sg/internalusers’ with …/sgconfig/sg_internal_users.yml
FAIL: Configuration for ‘internalusers’ failed because of UnavailableShardsException[[searchguard][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[searchguard][0]] containing [index {[searchguard][sg][internalusers], source[{“internalusers”:“eyJhZG1pbiI6eyJyZWFkb25seSI6dHJ1ZSwiaGFzaCI6IiQyYSQxMiRWY0NEZ2gyTkRrMDdKR04wcmpHYk0uQWQ0MXFWUi9ZRkpjZ0hwMFVHbnM1SkR5bXYuLlRPRyIsInJvbGVzIjpbImFkbWluIl0sImF0dHJpYnV0ZXMiOnsiYXR0cmlidXRlMSI6InZhbHVlMSIsImF0dHJpYnV0ZTIiOiJ2YWx1ZTIiLCJhdHRyaWJ1dGUzIjoidmFsdWUzIn19LCJsb2dzdGFzaCI6eyJoYXNoIjoiJDJhJDEyJHUxU2hSNGw0dUJTM1V2NTlQYTJ5NS4xdVF1WkJyWnRtTmZxQjNpTS8uakwwWG9WOXNnaFMyIiwicm9sZXMiOlsibG9nc3Rhc2giXX0sImtpYmFuYXNlcnZlciI6eyJyZWFkb25seSI6dHJ1ZSwiaGFzaCI6IiQyYSQxMiQ0QWNnQXQzeHdPV2FkQTVzNWJsTDZldjM5T1hETmhtT2VzRW9vMzNlWnRycTJOMFlyVTNILiJ9LCJraWJhbmFybyI6eyJoYXNoIjoiJDJhJDEyJEpKU1hOZlRvd3o3VXU1dHRYZmVZcGVZRTBhckFDdmN3bFBCU3RCMUYuTUk3ZjBVOVo0REdDIiwicm9sZXMiOlsia2liYW5hdXNlciIsInJlYWRhbGwiXX0sInJlYWRhbGwiOnsiaGFzaCI6IiQyYSQxMiRhZTR5Y3d6d3ZMdFp4d1o4MlJtaUV1bkJiSVBpQW1HWmR1QkFqS04wVFhkd1FGdEN3QVJ6MiIsInJvbGVzIjpbInJlYWRhbGwiXX0sInNuYXBzaG90cmVzdG9yZSI6eyJoYXNoIjoiJDJ5JDEyJERwd21ldEhLd2dZbm9yYmdkdk9SQ2VudjROQUs4Y1BVZzhBSTZweExDdVdmL0FMYzAudjdXIiwicm9sZXMiOlsic25hcHNob3RyZXN0b3JlIl19fQ==”}]}] and a refresh]]
Will update ‘sg/actiongroups’ with …/sgconfig/sg_action_groups.yml
FAIL: Configuration for ‘actiongroups’ failed because of UnavailableShardsException[[searchguard][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[searchguard][0]] containing [index {[searchguard][sg][actiongroups], source[n/a, actual length: [2.6kb], max length: 2kb]}] and a refresh]]

``

Diagnostic sgadmin trace

ES client version: 6.2.2
Client properties: {java.runtime.name=Java™ SE Runtime Environment, sun.boot.library.path=/usr/java/jdk1.8.0_161/jre/lib/amd64, java.vm.version=25.161-b12, jdk.tls.rejectClientInitiatedRenegotiation=true, java.vm.vendor=Oracle Corporation, java.vendor.url=http://java.oracle.com/, path.separator=:, java.vm.name=Java HotSpot™ 64-Bit Server VM, file.encoding.pkg=sun.io, user.country=US, sun.java.launcher=SUN_STANDARD, sun.os.patch.level=unknown, java.vm.specification.name=Java Virtual Machine Specification, user.dir=/opt/elasticsearch-6.2.2/plugins/search-guard-6/tools, sg.nowarn.client=true, java.runtime.version=1.8.0_161-b12, java.awt.graphicsenv=sun.awt.X11GraphicsEnvironment, java.endorsed.dirs=/usr/java/jdk1.8.0_161/jre/lib/endorsed, os.arch=amd64, java.io.tmpdir=/tmp, line.separator=

``

All Error logs excepts my first node say the following:

Failure No shard available for [org.elasticsearch.action.get.MultiGetShardRequest@3752f493] retrieving configuration for [config] (index=searchguard)

``

The first node log say the following:

Not yet initialized (you may need to run sgadmin)

``

Command being used to run agadmin

./sgadmin.sh -cd …/sgconfig/ -icl -nhnv --diagnose --accept-red-cluster -ff -cacert …/…/…/config/-root-ca.pem -cert …/…/…/config/admin-dev.pem -key …/…/…/config/admin-dev.key

``

···

On Tuesday, May 15, 2018 at 10:56:51 PM UTC-4, Daqueshia Irvin wrote:

Error message when running the sgadmin tool :

WARNING: JAVA_HOME not set, will use /usr/bin/java
Search Guard Admin v6
Will connect to localhost:9300 … done
Elasticsearch Version: 6.2.2
Search Guard Version: 6.2.2-22.1
Connected as CN=admin-dev
Fail-fast is activated
Diagnostic trace written to: /opt/elasticsearch-6.2.2/plugins/search-guard-6/tools/sgadmin_diag_trace_2018-May-16_02-39-49.txt
Contacting elasticsearch cluster ‘elasticsearch’ …
Clustername: dcat-elasticsearch
Clusterstate: RED
Number of nodes: 5
Number of data nodes: 3
searchguard index already exists, so we do not need to create one.
ERR: searchguard index state is RED.
Populate config from /opt/elasticsearch-6.2.2/plugins/search-guard-6/sgconfig
Will update ‘sg/config’ with …/sgconfig/sg_config.yml
Will update ‘sg/config’ with …/sgconfig/sg_config.yml FAIL: Configuration for ‘config’ failed because of UnavailableShardsException[[searchguard][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[searchguard][0]] containing [index {[searchguard][sg][config], source[n/a, actual length: [3.2kb], max length: 2kb]}] and a refresh]]
Will update ‘sg/roles’ with …/sgconfig/sg_roles.yml
FAIL: Configuration for ‘roles’ failed because of UnavailableShardsException[[searchguard][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[searchguard][0]] containing [index {[searchguard][sg][roles], source[n/a, actual length: [3.6kb], max length: 2kb]}] and a refresh]]
Will update ‘sg/rolesmapping’ with …/sgconfig/sg_roles_mapping.yml
FAIL: Configuration for ‘rolesmapping’ failed because of UnavailableShardsException[[searchguard][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[searchguard][0]] containing [index {[searchguard][sg][rolesmapping], source[{“rolesmapping”:“eyJzZ19hbGxfYWNjZXNzIjp7InJlYWRvbmx5Ijp0cnVlLCJiYWNrZW5kcm9sZXMiOlsiYWRtaW4iXX0sInNnX2xvZ3N0YXNoIjp7ImJhY2tlbmRyb2xlcyI6WyJsb2dzdGFzaCJdfSwic2dfa2liYW5hX3NlcnZlciI6eyJyZWFkb25seSI6dHJ1ZSwidXNlcnMiOlsia2liYW5hc2VydmVyIl19LCJzZ19raWJhbmFfdXNlciI6eyJiYWNrZW5kcm9sZXMiOlsia2liYW5hdXNlciJdfSwic2dfcmVhZGFsbCI6eyJyZWFkb25seSI6dHJ1ZSwiYmFja2VuZHJvbGVzIjpbInJlYWRhbGwiXX0sInNnX21hbmFnZV9zbmFwc2hvdHMiOnsicmVhZG9ubHkiOnRydWUsImJhY2tlbmRyb2xlcyI6WyJzbmFwc2hvdHJlc3RvcmUiXX0sInNnX293bl9pbmRleCI6eyJ1c2VycyI6WyIqIl19fQ==”}]}] and a refresh]]
Will update ‘sg/internalusers’ with …/sgconfig/sg_internal_users.yml
FAIL: Configuration for ‘internalusers’ failed because of UnavailableShardsException[[searchguard][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[searchguard][0]] containing [index {[searchguard][sg][internalusers], source[{“internalusers”:“eyJhZG1pbiI6eyJyZWFkb25seSI6dHJ1ZSwiaGFzaCI6IiQyYSQxMiRWY0NEZ2gyTkRrMDdKR04wcmpHYk0uQWQ0MXFWUi9ZRkpjZ0hwMFVHbnM1SkR5bXYuLlRPRyIsInJvbGVzIjpbImFkbWluIl0sImF0dHJpYnV0ZXMiOnsiYXR0cmlidXRlMSI6InZhbHVlMSIsImF0dHJpYnV0ZTIiOiJ2YWx1ZTIiLCJhdHRyaWJ1dGUzIjoidmFsdWUzIn19LCJsb2dzdGFzaCI6eyJoYXNoIjoiJDJhJDEyJHUxU2hSNGw0dUJTM1V2NTlQYTJ5NS4xdVF1WkJyWnRtTmZxQjNpTS8uakwwWG9WOXNnaFMyIiwicm9sZXMiOlsibG9nc3Rhc2giXX0sImtpYmFuYXNlcnZlciI6eyJyZWFkb25seSI6dHJ1ZSwiaGFzaCI6IiQyYSQxMiQ0QWNnQXQzeHdPV2FkQTVzNWJsTDZldjM5T1hETmhtT2VzRW9vMzNlWnRycTJOMFlyVTNILiJ9LCJraWJhbmFybyI6eyJoYXNoIjoiJDJhJDEyJEpKU1hOZlRvd3o3VXU1dHRYZmVZcGVZRTBhckFDdmN3bFBCU3RCMUYuTUk3ZjBVOVo0REdDIiwicm9sZXMiOlsia2liYW5hdXNlciIsInJlYWRhbGwiXX0sInJlYWRhbGwiOnsiaGFzaCI6IiQyYSQxMiRhZTR5Y3d6d3ZMdFp4d1o4MlJtaUV1bkJiSVBpQW1HWmR1QkFqS04wVFhkd1FGdEN3QVJ6MiIsInJvbGVzIjpbInJlYWRhbGwiXX0sInNuYXBzaG90cmVzdG9yZSI6eyJoYXNoIjoiJDJ5JDEyJERwd21ldEhLd2dZbm9yYmdkdk9SQ2VudjROQUs4Y1BVZzhBSTZweExDdVdmL0FMYzAudjdXIiwicm9sZXMiOlsic25hcHNob3RyZXN0b3JlIl19fQ==”}]}] and a refresh]]
Will update ‘sg/actiongroups’ with …/sgconfig/sg_action_groups.yml
FAIL: Configuration for ‘actiongroups’ failed because of UnavailableShardsException[[searchguard][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[searchguard][0]] containing [index {[searchguard][sg][actiongroups], source[n/a, actual length: [2.6kb], max length: 2kb]}] and a refresh]]

``

Diagnostic sgadmin trace

ES client version: 6.2.2
Client properties: {java.runtime.name=Java™ SE Runtime Environment, sun.boot.library.path=/usr/java/jdk1.8.0_161/jre/lib/amd64, java.vm.version=25.161-b12, jdk.tls.rejectClientInitiatedRenegotiation=true, java.vm.vendor=Oracle Corporation, java.vendor.url=http://java.oracle.com/, path.separator=:, java.vm.name=Java HotSpot™ 64-Bit Server VM, file.encoding.pkg=sun.io, user.country=US, sun.java.launcher=SUN_STANDARD, sun.os.patch.level=unknown, java.vm.specification.name=Java Virtual Machine Specification, user.dir=/opt/elasticsearch-6.2.2/plugins/search-guard-6/tools, sg.nowarn.client=true, java.runtime.version=1.8.0_161-b12, java.awt.graphicsenv=sun.awt.X11GraphicsEnvironment, java.endorsed.dirs=/usr/java/jdk1.8.0_161/jre/lib/endorsed, os.arch=amd64, java.io.tmpdir=/tmp, line.separator=

``

This was solved by

  1. Turning off the xpack.monitoring feature

  2. Using the transient property for turning off cluster.routing.allocation.enable

···

On Tuesday, May 15, 2018 at 10:56:51 PM UTC-4, Daqueshia Irvin wrote:

Error message when running the sgadmin tool :

WARNING: JAVA_HOME not set, will use /usr/bin/java
Search Guard Admin v6
Will connect to localhost:9300 … done
Elasticsearch Version: 6.2.2
Search Guard Version: 6.2.2-22.1
Connected as CN=admin-dev
Fail-fast is activated
Diagnostic trace written to: /opt/elasticsearch-6.2.2/plugins/search-guard-6/tools/sgadmin_diag_trace_2018-May-16_02-39-49.txt
Contacting elasticsearch cluster ‘elasticsearch’ …
Clustername: dcat-elasticsearch
Clusterstate: RED
Number of nodes: 5
Number of data nodes: 3
searchguard index already exists, so we do not need to create one.
ERR: searchguard index state is RED.
Populate config from /opt/elasticsearch-6.2.2/plugins/search-guard-6/sgconfig
Will update ‘sg/config’ with …/sgconfig/sg_config.yml
Will update ‘sg/config’ with …/sgconfig/sg_config.yml FAIL: Configuration for ‘config’ failed because of UnavailableShardsException[[searchguard][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[searchguard][0]] containing [index {[searchguard][sg][config], source[n/a, actual length: [3.2kb], max length: 2kb]}] and a refresh]]
Will update ‘sg/roles’ with …/sgconfig/sg_roles.yml
FAIL: Configuration for ‘roles’ failed because of UnavailableShardsException[[searchguard][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[searchguard][0]] containing [index {[searchguard][sg][roles], source[n/a, actual length: [3.6kb], max length: 2kb]}] and a refresh]]
Will update ‘sg/rolesmapping’ with …/sgconfig/sg_roles_mapping.yml
FAIL: Configuration for ‘rolesmapping’ failed because of UnavailableShardsException[[searchguard][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[searchguard][0]] containing [index {[searchguard][sg][rolesmapping], source[{“rolesmapping”:“eyJzZ19hbGxfYWNjZXNzIjp7InJlYWRvbmx5Ijp0cnVlLCJiYWNrZW5kcm9sZXMiOlsiYWRtaW4iXX0sInNnX2xvZ3N0YXNoIjp7ImJhY2tlbmRyb2xlcyI6WyJsb2dzdGFzaCJdfSwic2dfa2liYW5hX3NlcnZlciI6eyJyZWFkb25seSI6dHJ1ZSwidXNlcnMiOlsia2liYW5hc2VydmVyIl19LCJzZ19raWJhbmFfdXNlciI6eyJiYWNrZW5kcm9sZXMiOlsia2liYW5hdXNlciJdfSwic2dfcmVhZGFsbCI6eyJyZWFkb25seSI6dHJ1ZSwiYmFja2VuZHJvbGVzIjpbInJlYWRhbGwiXX0sInNnX21hbmFnZV9zbmFwc2hvdHMiOnsicmVhZG9ubHkiOnRydWUsImJhY2tlbmRyb2xlcyI6WyJzbmFwc2hvdHJlc3RvcmUiXX0sInNnX293bl9pbmRleCI6eyJ1c2VycyI6WyIqIl19fQ==”}]}] and a refresh]]
Will update ‘sg/internalusers’ with …/sgconfig/sg_internal_users.yml
FAIL: Configuration for ‘internalusers’ failed because of UnavailableShardsException[[searchguard][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[searchguard][0]] containing [index {[searchguard][sg][internalusers], source[{“internalusers”:“eyJhZG1pbiI6eyJyZWFkb25seSI6dHJ1ZSwiaGFzaCI6IiQyYSQxMiRWY0NEZ2gyTkRrMDdKR04wcmpHYk0uQWQ0MXFWUi9ZRkpjZ0hwMFVHbnM1SkR5bXYuLlRPRyIsInJvbGVzIjpbImFkbWluIl0sImF0dHJpYnV0ZXMiOnsiYXR0cmlidXRlMSI6InZhbHVlMSIsImF0dHJpYnV0ZTIiOiJ2YWx1ZTIiLCJhdHRyaWJ1dGUzIjoidmFsdWUzIn19LCJsb2dzdGFzaCI6eyJoYXNoIjoiJDJhJDEyJHUxU2hSNGw0dUJTM1V2NTlQYTJ5NS4xdVF1WkJyWnRtTmZxQjNpTS8uakwwWG9WOXNnaFMyIiwicm9sZXMiOlsibG9nc3Rhc2giXX0sImtpYmFuYXNlcnZlciI6eyJyZWFkb25seSI6dHJ1ZSwiaGFzaCI6IiQyYSQxMiQ0QWNnQXQzeHdPV2FkQTVzNWJsTDZldjM5T1hETmhtT2VzRW9vMzNlWnRycTJOMFlyVTNILiJ9LCJraWJhbmFybyI6eyJoYXNoIjoiJDJhJDEyJEpKU1hOZlRvd3o3VXU1dHRYZmVZcGVZRTBhckFDdmN3bFBCU3RCMUYuTUk3ZjBVOVo0REdDIiwicm9sZXMiOlsia2liYW5hdXNlciIsInJlYWRhbGwiXX0sInJlYWRhbGwiOnsiaGFzaCI6IiQyYSQxMiRhZTR5Y3d6d3ZMdFp4d1o4MlJtaUV1bkJiSVBpQW1HWmR1QkFqS04wVFhkd1FGdEN3QVJ6MiIsInJvbGVzIjpbInJlYWRhbGwiXX0sInNuYXBzaG90cmVzdG9yZSI6eyJoYXNoIjoiJDJ5JDEyJERwd21ldEhLd2dZbm9yYmdkdk9SQ2VudjROQUs4Y1BVZzhBSTZweExDdVdmL0FMYzAudjdXIiwicm9sZXMiOlsic25hcHNob3RyZXN0b3JlIl19fQ==”}]}] and a refresh]]
Will update ‘sg/actiongroups’ with …/sgconfig/sg_action_groups.yml
FAIL: Configuration for ‘actiongroups’ failed because of UnavailableShardsException[[searchguard][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[searchguard][0]] containing [index {[searchguard][sg][actiongroups], source[n/a, actual length: [2.6kb], max length: 2kb]}] and a refresh]]

``

Diagnostic sgadmin trace

ES client version: 6.2.2
Client properties: {java.runtime.name=Java™ SE Runtime Environment, sun.boot.library.path=/usr/java/jdk1.8.0_161/jre/lib/amd64, java.vm.version=25.161-b12, jdk.tls.rejectClientInitiatedRenegotiation=true, java.vm.vendor=Oracle Corporation, java.vendor.url=http://java.oracle.com/, path.separator=:, java.vm.name=Java HotSpot™ 64-Bit Server VM, file.encoding.pkg=sun.io, user.country=US, sun.java.launcher=SUN_STANDARD, sun.os.patch.level=unknown, java.vm.specification.name=Java Virtual Machine Specification, user.dir=/opt/elasticsearch-6.2.2/plugins/search-guard-6/tools, sg.nowarn.client=true, java.runtime.version=1.8.0_161-b12, java.awt.graphicsenv=sun.awt.X11GraphicsEnvironment, java.endorsed.dirs=/usr/java/jdk1.8.0_161/jre/lib/endorsed, os.arch=amd64, java.io.tmpdir=/tmp, line.separator=

``