How to initialize SG using sgctl?

Hi,

I am struggeling to find the exact command to initialize my cluster using SG FLX.

I executed ./sgctl.sh connect, but still have in my logs:

Not yet initialized (you may need to run sgctl)

The documentation leads me to here: Using sgctl | Security for Elasticsearch | Search Guard.

Can you help me please?

Also I see logs like:

.searchguard_authtokens is not yet ready

basically for all .searchguard* indices I guess.

I am implementing a hot warm architecture of 3 master, 2 hot, 2 warm nodes. Do I need to disable automatic sharding and configure replica shards manually for every internal search guard index?

@Kosmonafft As per Search Guard documentation you need to upload the configuration to initialize all .searchguard indices.

1 Like

Hi @pablo . Thanks for the response. Got it.

What about using SG in a hot/warm architecture? Seems like my cluster does not start because of unassigned shards for a range of searchguard indices. How can proceed here? Adjust the shards manually for every single searchguard index or is there another way of dealing with that?

Thanks

@Kosmonafft Could you share your current shards and indices? ( _cat/indices, _cat/shards)

When I execute:

curl https://elastic1.node.com:9200/_cat/indices -k

I got:

Search Guard not initialized (SG11)

and in the logs I see:

[2023-01-24T14:12:10,929][ERROR][c.f.s.c.ProtectedConfigIndexService] [elastic1.node.com] Index .signals_watches is not yet ready:
{"cluster_name":"es-cluster","status":"red","timed_out":true,"number_of_nodes":7,"number_of_data_nodes":4,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":10,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":13.793103448275861}
Retrying.
[2023-01-24T14:13:41,245][ERROR][c.f.s.c.ProtectedConfigIndexService] [elastic1.node.com] Index .searchguard_sessions is not yet ready:
{"cluster_name":"es-cluster","status":"red","timed_out":true,"number_of_nodes":7,"number_of_data_nodes":4,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":13.793103448275861}
Retrying.
[2023-01-24T14:14:11,333][ERROR][c.f.s.c.ProtectedConfigIndexService] [elastic1.node.com] Index .searchguard_resource_owner is not yet ready:
{"cluster_name":"es-cluster","status":"red","timed_out":true,"number_of_nodes":7,"number_of_data_nodes":4,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":13.793103448275861}

@Kosmonafft How did you start your cluster? Did data nodes start first?
Is this a new cluster?

Please share elasticsearch.yml files.

I guess I found what have caused the issue:

on my hot and warm data nodes I had configured:

node.roles: [ data_hot, ingest ] and node.roles: [ data_warm ]

I learned that I have missed that data_content role must also be added to the data nodes. So correct configuration is:

node.roles: [ data_hot, data_content, ingest ] and node.roles: [ data_warm, data_content ]

After applying these changes SG could be successfully initialized and the cluster runs.

However, since I have 2 hot nodes and 2 warm nodes:

  • How search guard will behave? how many shards it will create?
  • How should I configure the sharding for the internal indices? Should I always keep them according to the number of my hot nodes?
  • Should I apply the manual sharding configuration on every internal search guard index?
  • Or should I not bother with that at all?

Thanks for any advice

@Kosmonafft The data_content is the default tier preference when creating any index in the Elasticsearch cluster.

@Kosmonafft In regards to your questions, please find some more information in regard to SearchGuard indices and cluster configuration.