ES: all data are not returned on Admin tenant

Search Guard
1

Hi Team,

I have ELK stack deployed using docker swarm, when I query data I couldn’t retrive all the data. I have multi-tenant kibana indices such as Admin-tenant can view all the data and individual-tenant where team/ins data are available using DLS function.

My Issue is I couldn’t view some group of data either on discover, dev-tools and elasticsearch curl query, but I could see the data on individual tenant. Data groups with lesser documents in indicies are not visible. Is there something on ES setting that I can update to retrive all the documents?

ES version : 7.10.1
Kibana: 7.10.1

FYI: I raised this with elastic too (Not all the datas are returned - Elasticsearch - Discuss the Elastic Stack)

Best,
Yash

2

Hi @yasvanth,

I’m going to try and reproduce your issue, can you provide some further details please.

Such as sg_roles.yml, sg_roles_mappings.yml, sg_tenants.yml and sg_config.yml (Please redact any sensitive details)

Is there a particular role that is failing?

Can you also provide the curl command that is failing to retrieve the complete results?

3

@sirHusky, Thanks for your reply,
I have provided all the configurations that I used to enable multi-tenancy, I never seen this issue before with SG 6.
The douments under GROUP is fewer and couldn’t access it using curl or from admin tenant (kibana) whereas data is visible on kibana under GROUP tenancy.

sg_tenant.yml

_sg_meta:
  type: "tenants"
  config_version: 2

admin:
  reserved: true
  description: "Admin tenant"

GROUP_tenant:
  reserved: true
  description: "group1 tenant"

sg_role.yml

_sg_meta:
  type: "roles"
  config_version: 2

ADMIN_ROLE:
  reserved: true
  description: "admin role"
  cluster_permissions:
    - "SGS_UNLIMITED"
  index_permissions:
    - index_patterns:
        - "*"
      allowed_actions:
        - "SGS_UNLIMITED"
    - index_patterns:
        - "?kibana*"
      allowed_actions:
        - "SGS_UNLIMITED"
  tenant_permissions:
    - tenant_patterns:
        - "admin"
      allowed_actions:
        - "SGS_KIBANA_ALL_WRITE"
        - "SGS_SIGNALS_ALL"

GROUP_ROLE:
  reserved: true
  description: "Access to GROUP members"
  cluster_permissions:
    - "SGS_CLUSTER_COMPOSITE_OPS"
  index_permissions:
    - index_patterns:
        - "indices_name-*"
      allowed_actions:
        - "SGS_INDICES_ALL"
      dls: '{ "bool" :{ "must" :{ "match" :{ "Organization" : "group value_removed_due_to_sensitive" }}}}'
    - index_patterns:
        - "?kibana*"
      allowed_actions:
        - "SGS_KIBANA_ALL_WRITE"
  tenant_permissions:
    - tenant_patterns:
        - "GROUP_tenant"
      allowed_actions:
        - "SGS_KIBANA_ALL_WRITE"

sg_role_mapping.yml

_sg_meta:
  type: "rolesmapping"
  config_version: 2

ADMIN_ROLE:
  reserved: true
  users:
  - "admin_uses"
  description: "Maps user to ADMIN_ROLE"

GROUP_ROLE:
  description: "Maps GROUP Users to GROUP_ROLE"
  reserved: true
  users:
    - "users"

sg_config.yml

_sg_meta:
  type: "config"
  config_version: 2

sg_config:
  dynamic:
      license: "License_key_removed"
      do_not_fail_on_forbidden: true
      kibana:
        multitenancy_enabled: true
        server_username: 'kibana_user'
        index: '.kibana'
      http:
        anonymous_auth_enabled: true
        xff:
          enabled: true
          internalProxies: '.*'
          remoteIpHeader:  'x-forwarded-for'
      authc:
        basic_internal_auth_domain:
          description: "Authenticate via HTTP Basic against internal users database"
          http_enabled: true
          transport_enabled: true
          order: 0
          http_authenticator:
            type: basic
            challenge: true
          authentication_backend:
            type: intern
        proxy_auth_domain:
          description: "Proxy Authentication using shibboleth SP"
          http_enabled: true
          order: 1
          http_authenticator:
            type: proxy
            challenge: false
            config:
              user_header: "x-proxy-user"
              #roles_header: "x-proxy-roles"
          authentication_backend:
            type: noop
      authz:

Curl command with admin cred:

curl  -k -XGET -u "admin_username "https://localhost:9200/index_name/_search" -H 'Content-Type: application/json' -d'{  "query": {    "match": {      "Organization": "group_values"    }  }}

Response:
{“took”:327,“timed_out”:false,“_shards”:{“total”:2,“successful”:2,“skipped”:0,“failed”:0},“hits”:{“total”:{“value”:0,“relation”:“eq”},“max_score”:null,“hits”:}}

4

@yasvanth

Can you please run the below command to check which roles the user is mapped to at login:

curl --insecure -u admin_user -XGET “https://localhost:9200/_searchguard/authinfo?pretty”

The information is under:

“sg_roles” : [
“ADMIN_ROLE”
],

Could you paste the output here?

5

@sirHusky , Yes, I do get the roles that is mapped for the user.

{
  "user" : "User [name=admin, backend_roles=[], requestedTenant=null]",
  "user_name" : "admin",
  "user_requested_tenant" : null,
  "remote_address" : "172.18.0.1:60400",
  "backend_roles" : [ ],
  "custom_attribute_names" : [ ],
  "attribute_names" : [ ],
  "sg_roles" : [
    "ADMIN_ROLE",
  ],
  "sg_tenants" : {
    "admin" : true,
  },
  "principal" : null,
  "peer_certificates" : "0",
  "sso_logout_url" : null
}
6

@yasvanth

In your previous reply you used a curl request to search index and got zero results, can you run same query using a user that produces results and paste the first part of the response?

Can you also confirm if you are using aliases?

Can you also run GET _cat/indices/index_name?v and post the results here?

It is important to note that curl request has nothing to do with tenants if you are querying normal indices. Meaning unless you are querying kibana tenant indices (.kibana_…) the results are only filtered using index permissions granted via roles. Tenants are only used to segregate kibana objects (index patterns, dashboards etc) into different ‘tenants’ on kibana, not the actual data stored on indices.

7

{“took”:13,“timed_out”:false,“_shards”:{“total”:2,“successful”:2,“skipped”:0,“failed”:0},“hits”:{“total”:{“value”:4217,“relation”:“eq”},“max_score”:7.9430285,“hits”:[…]

{“took”:23,“timed_out”:false,“_shards”:{“total”:2,“successful”:2,“skipped”:0,“failed”:0},“hits”:{“total”:{“value”:0,“relation”:“eq”},“max_score”:null,“hits”:}}

Can you also confirm if you are using aliases?

I dont have any alias

Can you also run GET _cat/indices/index_name?v and post the results here?

health status index                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   index_name-1.0.0-2019.04 4iHYw*****K-fY1Yv3zw   2   1    4367711            0      6.7gb          3.3gb
8

@yasvanth
Is the ‘group_value’ static in the above curl requests or is it changing based on user?

Also, if you run the curl request using admin_user which is mapped to ADMIN_ROLE, but without the query, do you get the full results?
Users mapped to GROUP_ROLE (running same curl) should have less results as there is a DLS enabled.

9

@sirHusky,
t
The group_value is different for each role whereas admin doesn’t have any DLS filter, can get all the document from indices. Using admin user I get most of the group data.

Also, if you run the curl request using admin_user which is mapped to ADMIN_ROLE, but without the query, do you get the full results?

When I run curl query as an admin_user I get the results but not certain group which has fewer documents.

Users mapped to GROUP_ROLE (running same curl) should have less results as there is a DLS enabled.

Yes, Admin user should able to query all group data or even certain group is quried but Admin couldn’t get the data of group with less data.

10

@yasvanth
Just to make sure I understand.

The index in question (index_name-1.0.0-2019.04) has 4,367,711 docs.

When you run the query below, how many hits are you getting back, is it the same number or less?

curl -XGET “https://localhost:9200/index_name-1.0.0-2019.04/_search” -u admin_user --insecure

11

@sirHusky ,
For Admin user, without any query

curl -XGET “https://localhost:9200/index_name-1.0.0-2019.04/_search” -u admin_user --insecure
Returns first 1000 doc
{“took”:1427,“timed_out”:false,“_shards”:{“total”:2,“successful”:2,“skipped”:0,“failed”:0},“hits”:{“total”:{“value”:10000,“relation”:“gte”},“max_score”:2.0,“hits”:[…]

When I add query I didnt get any results

curl -XGET -u admin_user --insecure “https://localhost:9200/index_name-1.0.0-2019.04/_search” -H ‘Content-Type: application/json’ -d’{ “query”: { “match”: { “Organization”: “group_value” } }}’

{“took”:1382,“timed_out”:false,“_shards”:{“total”:2,“successful”:2,“skipped”:0,“failed”:0},“hits”:{“total”:{“value”:0,“relation”:“eq”},“max_score”:null,“hits”:}}

12

@yasvanth
Could you extract 1 document which has the group_value that the admin user is not picking up using group_role user and DM me the result? (You can obfuscate the values, but keep the format)

Also, are there any errors in elasticsearch logs during execution of the failing curl query?

13

Resolved by mapping admin users to built in role: SG_ALL_ACESS

14

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.