I have a cluster that runs on two physical machines, each of which hosts three VMs that store data. A few weeks ago I had some trouble that involved the a precipitous reboot of one of the two physical systems. I can log into the system with existing accounts, but any attempt to update the internal users produces this:
Clustername: elasticsearch
Clusterstate: GREEN
Number of nodes: 8
Number of data nodes: 6
searchguard index already exists, so we do not need to create one.
Populate config from /usr/share/elasticsearch/plugins/search-guard-6/sgconfig/
Will update ‘sg/config’ with /usr/share/elasticsearch/plugins/search-guard-6/sgconfig/sg_config.yml
FAIL: Configuration for ‘config’ failed because of ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
Will update ‘sg/roles’ with /usr/share/elasticsearch/plugins/search-guard-6/sgconfig/sg_roles.yml
FAIL: Configuration for ‘roles’ failed because of ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
Will update ‘sg/rolesmapping’ with /usr/share/elasticsearch/plugins/search-guard-6/sgconfig/sg_roles_mapping.yml
FAIL: Configuration for ‘rolesmapping’ failed because of ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
Will update ‘sg/internalusers’ with /usr/share/elasticsearch/plugins/search-guard-6/sgconfig/sg_internal_users.yml
FAIL: Configuration for ‘internalusers’ failed because of ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
Will update ‘sg/actiongroups’ with /usr/share/elasticsearch/plugins/search-guard-6/sgconfig/sg_action_groups.yml
FAIL: Configuration for ‘actiongroups’ failed because of ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
Done with failures
This is the invocation that produces the above errors:
/usr/share/elasticsearch/plugins/search-guard-6/tools/sgadmin.sh -diagnose -arc -cd “/usr/share/elasticsearch/plugins/search-guard-6/sgconfig” -icl -key “/etc/elasticsearch/tundra.key” -cert “/etc/elasticsearch/tundra.pem” -cacert “/etc/elasticsearch/root-ca.pem” -nhnv
And here is the specific part of the output that covers the searchguard index.
https://gist.github.com/NetwarSystem/576ff0e7077d3276d760164f19431bdf
How do I restore access to this index so I can update it?