How to initiate sgadmin on RED cluster with allocation disabled and not active shards?

Grzegorz_M · July 26, 2018, 4:30pm

When asking questions, please provide the following information:

Running on Docker containers

Elasticsearch: 6.3.0

SafeGueard: 6.3.0-22.3

Enterprise module: no

JVM version : 8

I have cluster with 3 data nodes that already had some data. Following full cluster upgrade I did first stop shard allocation and then performed synced-flush and installed search guard plugins. After that I did create certificates and added them to config files. Now I need to initiate cluster and this is where issue starts. First I did start to initialize sgadmin:

/usr/share/elasticsearch/plugins/search-guard-6/tools/sgadmin.sh
-cd /usr/share/elasticsearch/config/sg/
-cert /usr/share/elasticsearch/config/sg/admin.pem
-key /usr/share/elasticsearch/config/sg/admin-pkcs8.key
-cacert /usr/share/elasticsearch/config/sg/root-ca.pem
-nhnv
-icl

and it was constantly hanging on:

Search Guard Version: 6.3.0-22.3
Connected as CN=admin,OU=test,O=test,DC=test,DC=com
Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …

Looks like even after turning off allocation and flushing my cluster status is RED so this time I did try to initiate sgadmin with:

/usr/share/elasticsearch/plugins/search-guard-6/tools/sgadmin.sh
-cd /usr/share/elasticsearch/config/sg/
-cert /usr/share/elasticsearch/config/sg/admin.pem
-key /usr/share/elasticsearch/config/sg/admin-pkcs8.key
-cacert /usr/share/elasticsearch/config/sg/root-ca.pem
-nhnv
-icl
-esa

and got this error:

Search Guard Admin v6
Will connect to localhost:9300 … done
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/usr/share/elasticsearch/plugins/search-guard-6/netty-common-4.1.16.Final.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

It did not work so my next attempt was trying to initiate sgadmin with:

/usr/share/elasticsearch/plugins/search-guard-6/tools/sgadmin.sh
-cd /usr/share/elasticsearch/config/sg/
-cert /usr/share/elasticsearch/config/sg/admin.pem
-key /usr/share/elasticsearch/config/sg/admin-pkcs8.key
-cacert /usr/share/elasticsearch/config/sg/root-ca.pem
-nhnv
-icl
-arc

and I got below errors for each stage of initalization:

Elasticsearch Version: 6.3.0
Search Guard Version: 6.3.0-22.3
Connected as CN=admin,OU=test,O=test,DC=test,DC=com
Contacting elasticsearch cluster ‘elasticsearch’ …
Clustername: te-cluster
Clusterstate: RED
Number of nodes: 4
Number of data nodes: 3
searchguard index does not exists, attempt to create it … done (0-all replicas)
Populate config from /usr/share/elasticsearch/config/sg/
Will update ‘sg/config’ with /usr/share/elasticsearch/config/sg/sg_config.yml
FAIL: Configuration for ‘config’ failed because of UnavailableShardsException[[searchguard][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[searchguard][0]] containing [index {[searchguard][sg][config], source[{“config”:“==”}]}] and a refresh]]
Will update ‘sg/roles’ with /usr/share/elasticsearch/config/sg/sg_roles.yml
FAIL: Configuration for ‘roles’ failed because of UnavailableShardsException[[searchguard][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[searchguard][0]] containing [index {[searchguard][sg][roles], source[{“roles”:“”}]}] and a refresh]]
Will update ‘sg/rolesmapping’ with /usr/share/elasticsearch/config/sg/sg_roles_mapping.yml
FAIL: Configuration for ‘rolesmapping’ failed because of UnavailableShardsException[[searchguard][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[searchguard][0]] containing [index {[searchguard][sg][rolesmapping], source[{“rolesmapping”:“==”}]}] and a refresh]]
Will update ‘sg/internalusers’ with /usr/share/elasticsearch/config/sg/sg_internal_users.yml
FAIL: Configuration for ‘internalusers’ failed because of UnavailableShardsException[[searchguard][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[searchguard][0]] containing [index {[searchguard][sg][internalusers], source[{“internalusers”:“==”}]}] and a refresh]]
Will update ‘sg/actiongroups’ with /usr/share/elasticsearch/config/sg/sg_action_groups.yml
FAIL: Configuration for ‘actiongroups’ failed because of UnavailableShardsException[[searchguard][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[searchguard][0]] containing [index {[searchguard][sg][actiongroups], source[n/a, actual length: [2.6kb], max length: 2kb]}] and a refresh]]
FAIL: Expected 4 nodes to return response, but got only 0
Done with failures

At this point I have no idea how to make cluster operational. Hopefully someone can help.

jkressin · July 26, 2018, 7:33pm

I think there are multiple issues here. First, it seems you are running Java 10:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/usr/share/elasticsearch/plugins/search-guard-6/netty-common-4.1.16.Final.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

``

Java 9 and Java 10 are non-LTS releases. Java 9 has already reached EOL, and Java 10 will reach EOL in September. The next version we will support is Java 11, which is an LTS release.

Then you said you disabled shard allocation, but your cluster state was red. The interesting thing would be to understand why the state was red in the first place. Which is probably not possible anymore I guess.

Due to the wrong Java version I guess that re-enabling shard allocation did not work. In your last call when trying to execute sgadmin with -arc the primary shard could not be created. This indicates that shard allocation is still disabled. In the scenario that you describe a better approach would be to not disable shard allocation completely, but using new_primaries:

This makes sure the primary shard can be created and thus the Search Guard index can be created.

But basically, it seems to boil down to the usage of unsupported Java 10.

···

On Thursday, July 26, 2018 at 11:28:29 AM UTC+2, Grzegorz M wrote: