Updating Internal Users Database without cluster

Hi all,
I got into a certain scenario and would like get clarification from your side.

Scenario:
I am setting up an elastic search cluster, configured the master node and before adding any other nodes in to the cluster, I have updated password of one of the user in Internal Users Database(sg_internal_users.yml). Then I ran sgadmin.sh. But the operation failed because searchguard index state is YELLOW, it seems we miss some replicas.

Error log is given below.

FAIL: Configuration for ‘config’ failed because of UnavailableShardsException[[searchguard][0] primary shard is not active Timeout: [1m]
Will update ‘roles’ with /usr/share/elasticsearch/plugins/search-guard-5/sgconfig/sg_roles.yml
FAIL: Configuration for ‘roles’ failed because of UnavailableShardsException[[searchguard][0] primary shard is not active Timeout: [1m],
Will update ‘rolesmapping’ with /usr/share/elasticsearch/plugins/search-guard-5/sgconfig/sg_roles_mapping.yml
FAIL: Configuration for ‘rolesmapping’ failed because of UnavailableShardsException[[searchguard][0] primary shard is not active Timeout: [1m],
Will update ‘internalusers’ with /usr/share/elasticsearch/plugins/search-guard-5/sgconfig/sg_internal_users.yml
FAIL: Configuration for ‘internalusers’ failed because of UnavailableShardsException[[searchguard][0] primary shard is not active Timeout: [1m],
Will update ‘actiongroups’ with /usr/share/elasticsearch/plugins/search-guard-5/sgconfig/sg_action_groups.yml
FAIL: Configuration for ‘actiongroups’ failed because of UnavailableShardsException[[searchguard][0] primary shard is not active Timeout: [1m],

I can understand that in the mentioned scenario the error is expected, but i would like to know

  1. After this error, if I add nodes in to the cluster, will there be any issues ?
  2. Do I need to re-run sgadmin.sh after adding other nodes in the cluster.?

Any help will be appreciated.
Thanks.

No, you don’t need to rerun sgadmin when you add or remove nodes. The configuration is stored in a specially secured Elasticsearch index, so new nodes are able to access it immediately.

Per default, Search Guard enables replica auto-expansion, which means a replica of the (very small) configuration index is placed on each node.

You can also configure this via sgamin: https://docs.search-guard.com/latest/search-guard-index#replica-shards

The underlying question here is why the primary shard became inactive.

@cstaley anything to add from your side?

  • There will be no errors after adding nodes (albeit Search Guard will be uninitialized)
  • You need one sgadmin run which is successful. After that all future nodes gets the config automatically

Hi,

Thank you for the valuable information.
May I bother you with some more doubts.

The node where the update has been performed is an Ingest node and no storage node has been added to the cluster. I believe this is the reason why primary shard became inactive message is shown. Since I am very new this search guard, please correct me if I am wrong.

Adding few logs, which I missed earlier.

Contacting elasticsearch cluster ‘log-collector’ and wait for YELLOW clusterstate …
Clustername: log-collector
Clusterstate: YELLOW
Number of nodes: 1
Number of data nodes: 0
searchguard index already exists, so we do not need to create one.
INFO: searchguard index state is YELLOW, it seems you miss some replicas
Populate config from /usr/share/elasticsearch/plugins/search-guard-5/sgconfig/

I have one more query.

  1. As I investigated the user password got updated in sg_internal_users.yml, but the sgadmin.sh has failed, In this situation whether the Search Guard configuration index will have old password value or the new value or the index ?
    If old value, then the file will have a updated password and the index will have old password, how to take care of such cases.?

  2. Is there any possible way where I will update only sg_internal_users.yml and the Search Guard configuration index gets updated automatically ?

Need some immediate help on this.
Thanks in advance.

To run sgadmin successfully you need at least one data node. As long there is no data node the config can not be uploaded and installed into the cluster. In case sgadmin is successfull the old config gets entirely overwritten by the new one.

To update only sg_internal_users.yml use the -f option as mentioned in the docs: https://docs.search-guard.com/latest/sgadmin-examples#apply-single-file-to-cluster-named-myclustername

Thank you for the valuable information.

I would like to know is there any curl command or sgadmin command available to read users details from Search Guard configuration index.
Through the command, I wanted to verify whether the changes I did in the internal user database is getting reflected in the Search Guard configuration index.

Thanks in advance

sgadmin -r, see https://docs.search-guard.com/latest/upgrading#back-up-the-search-guard-configuration-optional