Permission of searchguard

Hi, I tried to deal with searchguard behaviour. I’ve got a filebeat. It tells me this error:


окт 23 22:25:14 suricata.eureca.ru filebeat[3035]: 2019-10-23T22:25:14.654+0300        ERROR        pipeline/output.go:100        Failed to connect to backoff(elasticsearch(https://elasticseach:9200)): Connection marked as failed because the onConnect callback failed: Error loading pipeline for fileset suricata/eve: Error querying _nodes/ingest: 403 Forbidden: {"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [cluster:monitor/nodes/info] and User [name=usr_logstash, backend_roles=[], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [cluster:monitor/nodes/info] and User [name=usr_logstash, backend_roles=[], requestedTenant=null]"},"status":403}`

I added nessesary permission to user that used for filebeat
and tried to apply it with sgadmin


# /usr/local/elasticsearch/plugins/search-guard-7/tools/sgadmin.sh -cd /usr/local/elasticsearch/sgconfig/ -ks /usr/local/elasticsearch/config/sg/admin-keystore.jks -ts /usr/local/elasticsearch/config/sg/truststore.jks -kspass password -tspass another_password -nhnv
    WARNING: JAVA_HOME not set, will use /usr/bin/java
    Search Guard Admin v7
    Will connect to localhost:9300 ... done
    Connected as CN=admin,OU=client,O=client,L=Test,C=DE
    Elasticsearch Version: 7.3.2
    Search Guard Version: 7.3.2-37.0.0
    Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
    Clustername: elasticsearch
    Clusterstate: YELLOW
    Number of nodes: 1
    Number of data nodes: 1
    searchguard index already exists, so we do not need to create one.
    Populate config from /usr/local/elasticsearch/sgconfig/
    /usr/local/elasticsearch/sgconfig/sg_action_groups.yml OK
    /usr/local/elasticsearch/sgconfig/sg_internal_users.yml OK
    /usr/local/elasticsearch/sgconfig/sg_roles.yml OK
    /usr/local/elasticsearch/sgconfig/sg_roles_mapping.yml OK
    /usr/local/elasticsearch/sgconfig/sg_config.yml OK
    /usr/local/elasticsearch/sgconfig/sg_tenants.yml OK
    Will update '_doc/config' with /usr/local/elasticsearch/sgconfig/sg_config.yml 
       FAIL: Configuration for 'config' failed because of ClusterBlockException[index [searchguard] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
    Will update '_doc/roles' with /usr/local/elasticsearch/sgconfig/sg_roles.yml 
       FAIL: Configuration for 'roles' failed because of ClusterBlockException[index [searchguard] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
    Will update '_doc/rolesmapping' with /usr/local/elasticsearch/sgconfig/sg_roles_mapping.yml 
       FAIL: Configuration for 'rolesmapping' failed because of ClusterBlockException[index [searchguard] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
    Will update '_doc/internalusers' with /usr/local/elasticsearch/sgconfig/sg_internal_users.yml 
       FAIL: Configuration for 'internalusers' failed because of ClusterBlockException[index [searchguard] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
    Will update '_doc/actiongroups' with /usr/local/elasticsearch/sgconfig/sg_action_groups.yml 
       FAIL: Configuration for 'actiongroups' failed because of ClusterBlockException[index [searchguard] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
    Will update '_doc/tenants' with /usr/local/elasticsearch/sgconfig/sg_tenants.yml 
       FAIL: Configuration for 'tenants' failed because of ClusterBlockException[index [searchguard] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
    ERR: cannot upload configuration, see errors above

when I had been exploring the google I found that I can set
the setting index.blocks.read_only_allow_delete to null and
it will help solve this problem. So when I’ve tried to do it I saw:


# curl  -u admin:password --insecure -X PUT 'https://localhost:9200/_all/_settings?pretty' -H 'Content-Type: application/json' -d '{ "index.blocks.read_only_allow_delete": null }'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "no permissions for [] and User [name=admin, backend_roles=[], requestedTenant=null]"
      }
    ],
    "type" : "security_exception",
    "reason" : "no permissions for [] and User [name=admin, backend_roles=[], requestedTenant=null]"
  },
  "status" : 403
}

I thought the admin user has appropriate permissions in sg_internal_users.yml:


admin:
  hash: xxxx
  search_guard_roles:
    - SGS_ALL_ACCESS

I've changed the elasticsearch.yml file and added:
searchguard.roles_mapping_resolution: BOTH
searchguard.restapi.roles_enabled: ["SGS_ALL_ACCESS"]

but it didn’t help, the error has stayed. It is turn out not enough.
The user that I use for filebeat is


# sg_internal_users.yml
usr_logstash:
  hash: xxxx
  search_guard_roles:
    - SG_BEATS

# sg_roles.yml
SG_BEATS:
  cluster_permissions:
      - indices:admin/template/get
      - indices:admin/template/put
      - cluster:monitor/main
      - 'indices:data/write/bulk*'
      - indices:data/write/index
      - cluster:monitor/xpack/info
      - cluster:monitor/nodes/info # I've added this rermission
  index_permissions:
      - index_patterns:
          - '*'
        allowed_actions:
          - SGS_CRUD
          - SGS_CREATE_INDEX

What I’m doing wrong?
PS: I’ve read this topic Failed to create new users and roles and @jkressin wrote example
using certificates to authenticate with curl. Maybe I shouldn’t use plain auth?
And when I execute curl I find out in elasticsearch log:


окт 24 11:09:46 logstach.local elasticsearch[5272]: [2019-10-24T11:09:46,885][WARN ][c.f.s.p.SearchGuardIndexAccessEvaluator] [node-1] indices:admin/settings/update for '_all' indices is not allowed for a regular user

I dont know what had happened, but I deal with it. I deleted searchguard index using sgadmin and flag -dci and and create it again.

Just for the sake of completeness and maybe other users that stumble upon this:

There are several errors here, but this error here:

FAIL: Configuration for 'config' failed because of ClusterBlockException[index [searchguard] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]

means that the Search Guard index was marked read-only. Maybe you hit the disk quote on your system and Elasticsearch decided to put the index in read only mode.

If this happens, then sgadmin cannot write to this index anymore, hence the error messages.

The Search Guard index contains sensitive information and is therefore protected. You can only access the index with the admin TSL certificate, the one you use with sgadmin. A regular Search Guard user, including the admin user, cannot access the Search Guard configuration index.

Think of the admin TLS certificate as the equivalent to a root user. You need to use it whenever you need to make changes to the SG index directly. You can use it with sgadmin, but you can also use it for example with curl - this will give you full access to the cluster, including the SG index.

Hope this helps!