Cannot retrieve cluster state due to: no permissions for [cluster:monitor/health]

Search Guard
1

I have a cluster with two nodes, that keeps failing to create the initial sg users, if both elasticsearch are up from the beginning. Stopping the non-admin node, fixes the situation, but for some deployments, we cannot confirm that the admin node will be brought up before the no-admin ones.

[root@es001 /etc/elasticsearch/inst01]# /usr/share/elasticsearch/plugins/search-guard-6/tools/sgadmin.sh -cd /etc/elasticsearch/inst01/sg_config/ -icl -nhnv -arc -cacert /etc/elasticsearch/inst01/root-ca.pem -cert /etc/elasticsearch/inst01/es001admin.crt -key /etc/elasticsearch/inst01/es001.admin.key --hostname es001 --accept-red-cluster --diagnose
WARNING: JAVA_HOME not set, will use /bin/java
Search Guard Admin v6
Will connect to es001:9300 … done
Elasticsearch Version: 6.6.0
Search Guard Version: 6.6.0-24.3
Connected as C=…CN=es001-admin
Diagnostic trace written to: /etc/elasticsearch/inst01/sgadmin_diag_trace_2019-Jul-25_11-02-48.txt
Contacting elasticsearch cluster ‘elasticsearch’ …
Cannot retrieve cluster state due to: no permissions for [cluster:monitor/health] and User [name=C=…,CN=es001-admin, roles=, requestedTenant=null]. This is not an error, will keep on trying …
Root cause: ElasticsearchSecurityException[no permissions for [cluster:monitor/health] and User [name=C=…,CN=es001-admin, roles=, requestedTenant=null]] (org.elasticsearch.ElasticsearchSecurityException/org.elasticsearch.ElasticsearchSecurityException)

  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
  • Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
    [root@es001 /etc/elasticsearch/inst01]# /usr/share/elasticsearch/plugins/search-guard-6/tools/sgadmin.sh -cdetc/elasticsearch/inst01/sg_config/ -icl -nhnv -arc -cacert /etc/elasticsearch/inst01/root-ca.pem -cert /etc/elasticsearch/inst01/es001-admin.crt -key /etc/elasticsearch/inst01/es000admin.key --hostname es001 --accept-red-cluster --diagnose -w
    WARNING: JAVA_HOME not set, will use /bin/java
    Search Guard Admin v6
    Will connect to es001:9300 … done
    Elasticsearch Version: 6.6.0
    Search Guard Version: 6.6.0-24.3
    Connected as C=…,CN=es001
    {
    “whoami” : {
    “dn” : “C=…,CN=es001”,
    “is_admin” : true,
    “is_authenticated” : true,
    “is_node_certificate_request” : false
    }
    }
cluster.name: elasticsearch-lab
discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.unicast.hosts:
- es001:9300
- es002:9300
http.host: 10.10.10.1
network.host: 10.10.10.2
node.data: true
node.master: true
node.name: es001
path.data: "/var/lib/elasticsearch/inst01"
path.logs: "/var/log/elasticsearch/inst01"
searchguard.authcz.admin_dn:
- C=...,CN=es001admin
searchguard.enterprise_modules_enabled: false
searchguard.nodes_dn:
- "*CN=es001"
- "*CN=es002"
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.pemcert_filepath: es001.crt
searchguard.ssl.transport.pemkey_filepath: es001.key
searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem
searchguard.ssl.transport.resolve_hostname: false
transport.tcp.port: 9300
3

There is no such thing like an “admin node”. The elasticsearch.yml needs to be (more or less) identical for all nodes in your cluster (and Search Guard needs also to be installed on all nodes). Please elaborate what you mean with “admin node” and please post both your elasticsearch.yml for your two nodes.

4

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.