Cluster status is red due to unassigned shards .signals_* and .searchguard_*

yasvanth · March 26, 2021, 2:28pm

Hi Team,

I am spining up elasticsearch cluster with SG install on docker swarm env and cluster is red because of unassigned shard. Cluster is red even after initializing SG.

I used admin cert and key to remove unassigned shards, Is there any options or config settings to avoid this in future when setting-up fresh cluster with SG installed?

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1088  100  1088    0     0  23148      0 --:--:-- --:--:-- --:--:-- 23652
.signals_accounts              0 r UNASSIGNED
.signals_accounts              0 p UNASSIGNED
.signals_watches_trigger_state 0 p UNASSIGNED
.signals_watches_trigger_state 0 r UNASSIGNED
.searchguard_authtokens        0 r UNASSIGNED
.searchguard_authtokens        0 p UNASSIGNED
.searchguard_config_history    0 r UNASSIGNED
.searchguard_config_history    0 p UNASSIGNED
.signals_watches               0 r UNASSIGNED
.signals_watches               0 p UNASSIGNED
.signals_settings              0 p UNASSIGNED
.signals_settings              0 r UNASSIGNED
.signals_watches_state         0 r UNASSIGNED
.signals_watches_state         0 p UNASSIGNED

{“type”: “server”, “timestamp”: “2021-03-26T14:06:43,116Z”, “level”: “ERROR”, “component”: “c.f.s.c.ProtectedConfigIndexService”, “cluster.name”: “<cluster_name>”, “node.name”: “esmaster01”, “message”: “Index .searchguard_authtokens is not yet ready:\n{"cluster_name":"<cluster_name>","status":"red","timed_out":true,"number_of_nodes":5,"number_of_data_nodes":2,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":4,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":0.0}\nRetrying.”, “cluster.uuid”: “<cluster_uuid>”, “node.id”: “<node_id>” }

ES Version
“version” : {
“number” : “7.10.1”,
“build_flavor” : “default”,
“build_type” : “docker”,
“build_hash” : “1c34507e66d7db1211f66f3513706fdf548736aa”,
“build_date” : “2020-12-05T01:00:33.671820Z”,
“build_snapshot” : false,
“lucene_version” : “8.7.0”,
“minimum_wire_compatibility_version” : “6.8.0”,
“minimum_index_compatibility_version” : “6.0.0-beta1”
},
“tagline” : “You Know, for Search”
}

Steps to reproduce:

Deploy es cluster using docker swarm

Provide configuration:
Basic ES Configuration with certs

srgbnd · March 26, 2021, 2:40pm

Hi. Have only the SG indices had the unassigned shards? Maybe you did the rolling restart and forgot to re-enable the shard allocation? It is recommended to disable the allocation before the restart and re-enable when the cluster is yellow to avoid unnecessary replication Full-cluster restart and rolling restart | Elasticsearch Guide [8.4] | Elastic

yasvanth · March 26, 2021, 2:58pm

@srgbnd, This is a fresh install using docker images with SG installed in it. There was no indices created earlier. I didnt experienced this issue before signals and auth-token.

Best,
Yash

yasvanth · March 26, 2021, 4:55pm

Now cluster is stable but no indices for signals and auth-token, will check whether I can mannually add the indices for signals and auth-tokens.

yasvanth · March 26, 2021, 7:44pm

Update to my previous issue, I did upgrade from 7.10.1 to 7.10.2 and search-guard SecurityIndices are created.

If we are setting up cluster using docker-swarm with SG installed, issue occurs because we couldn’t override sharding allocation setting on start-up. Once cluster is up,

Initialize SG,
Remove unassigned shards
Disable sharding allocation
Upgrade the cluster to next minor version so SG installation will re-deploy all the required security indices.

Would be great if SG installs security-indices after initialization when there is no existing search-guard indices. Can you guide me if there is a way to do this which I am not aware.

marc · March 26, 2021, 10:01pm

@yasvanth it sounds like you have this instance under control, but if you can still reproduce the original issue it would be helpful if you could call this ES endpoint:

GET /_cluster/allocation/explain

while you still have all the UNASSIGNED shards, and share the results here. Since the searchguard index won’t be initialized at that point, you’d have to make this call using the admin cert and key (as you did before).

yasvanth · March 29, 2021, 9:52am

@marc , Yes, I can spin-up small cluster with single node today and extract the details that has been requested.

yasvanth · March 29, 2021, 10:43am

@marc, I got the response below when I spuned single node cluster with SG installed.

{
  "index" : ".signals_settings",
  "shard" : 0,
  "primary" : true,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "ALLOCATION_FAILED",
    "at" : "2021-03-29T10:35:21.977Z",
    "failed_allocation_attempts" : 1,
    "details" : "failed shard on node [xtteOv-oQZm0xM3TGHcwzQ]: master {esmaster01-test}{xtteOv-oQZm0xM3TGHcwzQ}{3IG9fTReR4K5Bra2KZmNPw}{10.0.4.3}{10.0.4.3:9300}{dm}{xpack.installed=true, transform.node=false} has not removed previously failed shard. resending shard failure",
    "last_allocation_status" : "awaiting_info"
  },
  "can_allocate" : "awaiting_info",
  "allocate_explanation" : "cannot allocate because information about existing shard data is still being retrieved from some of the nodes",
  "node_allocation_decisions" : [
    {
      "node_id" : "xtteOv-oQZm0xM3TGHcwzQ",
      "node_name" : "esmaster01-test",
      "transport_address" : "10.0.4.3:9300",
      "node_attributes" : {
        "xpack.installed" : "true",
        "transform.node" : "false"
      },
      "node_decision" : "yes"
    }
  ]
}

marc · March 29, 2021, 4:40pm

@yasvanth could you please test whether a “retry” would finally allocate the failed shards:

POST /_cluster/reroute?retry_failed=true&explain=true

Also, are you planning to use the Signals functionality ? If you’re not, as a workaround, we can explicitly disable Signals - in elasticsearch.yml:

signals.enabled: false

we’d have to disable it once the SG plugin has been installed (otherwise ES will reject this value while starting up), but before the first time you start the ES node with SG already installed (so that the signals indices have not been created) - do you think you can introduce that in the middle of your set up sequence?

yasvanth · March 30, 2021, 9:28am

@marc I wanted to use signals, even if I disable signals then issue persists with “.searchguard_authtoken” indices.

Output of “retrying” allocation is provided below and initiallized SG after retrying but no luck.

system · April 20, 2021, 9:28am

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Inconsistent cluster state during 6-7 Upgrade Search Guard	19	545	March 10, 2022
Remove/move/delete .signals_watches_trigger_state Signals Alerting	3	447	December 21, 2020
Deleting searchguard/signals indexes and recreating Search Guard	2	477	July 2, 2021
Receiving signals_unavailable_exception Signals Alerting	9	1477	February 15, 2021
Signals - watch with query on data from remote cluster Signals Alerting	6	756	May 15, 2020

Cluster status is red due to unassigned shards .signals_* and .searchguard_*

Related Topics