Search Guard FLX DLS/FLS breaks Kibana multi-tenancy

trauta · November 4, 2022, 8:47am

Hi,

When using the new FLX implementation for DLS/FLS security the multi-tenancy feature in Kibana seems to be broken.

After setting use_impl: flx in sg_authz_dlsfls.yml a logged in user in Kibana gets a HTTP 500 Internal Server Error when trying to access Kibana after login.

Here are the (anonymized) Elasticsearch log messages:

[2022-11-04T09:28:08,390][WARN ][r.suppressed             ] [es-node1] path: /.kibana, params: {index=.kibana}
org.elasticsearch.transport.RemoteTransportException: [es-node2][1.2.3.4:9300][indices:admin/get]
Caused by: java.lang.IllegalStateException: Index [.kibana_109026154_USER] is not registered in com.floragunn.searchguard.enterprise.dlsfls.RoleBasedDocumentAuthorization$StatefulIndexQueries@2419ca2b
    at com.floragunn.searchguard.enterprise.dlsfls.RoleBasedDocumentAuthorization.hasDlsRestrictions(RoleBasedDocumentAuthorization.java:84) ~[?:?]
    at com.floragunn.searchguard.enterprise.dlsfls.DlsFlsValve.apply(DlsFlsValve.java:127) ~[?:?]
...

[2022-11-04T09:28:08,435][WARN ][r.suppressed             ] [es-node1] path: /.kibana_7.17.7_001, params: {index=.kibana_7.17.7_001}
org.elasticsearch.transport.RemoteTransportException: [es-node2][1.2.3.4:9300][indices:admin/create]
Caused by: java.lang.IllegalStateException: Index [.kibana_109026154_USER_7.17.7_001] is not registered in com.floragunn.searchguard.enterprise.dlsfls.RoleBasedDocumentAuthorization$StatefulIndexQueries@2419ca2b
    at com.floragunn.searchguard.enterprise.dlsfls.RoleBasedDocumentAuthorization.hasDlsRestrictions(RoleBasedDocumentAuthorization.java:84) ~[?:?]
    at com.floragunn.searchguard.enterprise.dlsfls.DlsFlsValve.apply(DlsFlsValve.java:127) ~[?:?]
..

[2022-11-04T09:28:08,470][ERROR][c.f.s.e.d.DlsFlsValve    ] [es-node1] Error while evaluating DLS/FLS privileges
java.lang.IllegalStateException: Index [.kibana_109026154_USER_7.17.7] is not registered in com.floragunn.searchguard.enterprise.dlsfls.RoleBasedDocumentAuthorization$StatefulIndexQueries@7b209981
    at com.floragunn.searchguard.enterprise.dlsfls.RoleBasedDocumentAuthorization.hasDlsRestrictions(RoleBasedDocumentAuthorization.java:84) ~[dlic-search-guard-flx-dlsfls-1.0.0-es-7.17.7.jar:1.0.0-es-7.17.7]
..

[2022-11-04T09:28:08,489][ERROR][c.f.s.f.SearchGuardFilter] [es-node1] Exception while handling indices:data/write/bulk; org.elasticsearch.action.bulk.BulkRequest/unset
java.lang.IllegalStateException: Index [.kibana_109026154_USER_7.17.7] is not registered in com.floragunn.searchguard.enterprise.dlsfls.RoleBasedDocumentAuthorization$StatefulIndexQueries@7b209981
    at com.floragunn.searchguard.enterprise.dlsfls.RoleBasedDocumentAuthorization.hasDlsRestrictions(RoleBasedDocumentAuthorization.java:84) ~[dlic-search-guard-flx-dlsfls-1.0.0-es-7.17.7.jar:1.0.0-es-7.17.7]
    at com.floragunn.searchguard.enterprise.dlsfls.DlsFlsValve.apply(DlsFlsValve.java:127) ~[dlic-search-guard-flx-dlsfls-1.0.0-es-7.17.7.jar:1.0.0-es-7.17.7]
..

[2022-11-04T09:28:08,508][WARN ][r.suppressed             ] [es-node1] path: /.kibana_7.17.7/_create/config%3A7.17.7, params: {require_alias=true, refresh=wait_for, index=.kibana_7.17.7, id=config:7.17.7, op_type=create}
java.lang.IllegalStateException: Index [.kibana_109026154_USER_7.17.7] is not registered in com.floragunn.searchguard.enterprise.dlsfls.RoleBasedDocumentAuthorization$StatefulIndexQueries@7b209981
    at com.floragunn.searchguard.enterprise.dlsfls.RoleBasedDocumentAuthorization.hasDlsRestrictions(RoleBasedDocumentAuthorization.java:84) ~[dlic-search-guard-flx-dlsfls-1.0.0-es-7.17.7.jar:1.0.0-es-7.17.7]
    at com.floragunn.searchguard.enterprise.dlsfls.DlsFlsValve.apply(DlsFlsValve.java:127) ~[dlic-search-guard-flx-dlsfls-1.0.0-es-7.17.7.jar:1.0.0-es-7.17.7]
...

If desired, I can provide the full Java stack trace.

To my understand it seems that Kibana is unable to create the hidden Kibana system indices.

It seems to do not matter if the assigned role for the user has a DLS or FLS restriction configured or not.
When reverting sg_authz_dlsfls.yml back to use_impl: legacy the user is able to access Kibana.

We excessively used DLS before upgrading to FLX without any problems.

Regards,
Alex

Elasticsearch version:
7.17.7

Server OS version:
Ubuntu 22.04.1 LTS

Kibana version (if relevant):
7.17.7

trauta · November 4, 2022, 9:11am

Hi,

also, if reverting the DLS implementation back to legacy any DLS filter that worked before upgrading to FLX does not do anything.

Here is an example of role with a DLS filter:

TEST:
  description: ""
  index_permissions:
  - allowed_actions:
    - "SGS_GET"
    - "SGS_READ"
    - "SGS_SEARCH"
    index_patterns:
    - "filebeat-*"
    fls: []
    dls: "{\"bool\":{\"must\": {\"match\": {\"agent.name\": \"test.example.com\"\
      }}}}"
    masked_fields: []
  tenant_permissions:
  - allowed_actions:
    - "SGS_KIBANA_ALL_WRITE"
    tenant_patterns:
    - "Test"
  cluster_permissions: []
  exclude_cluster_permissions: []
  exclude_index_permissions: []

This roles should restrict access to events containing agent.name: test.example.com but the user can access all events of this index, for example events containing agent.name: test2.example.com

curl ... -H "Content-type: application/json" -X GET "https://ES_HOST:9200/filebeat-*/_search?pretty" -d '
{
  "query": {
      "bool": {
          "must": {
              "match": {
                  "agent.name": "test2.example.com"
              }
          }
      }
  },
  "fields": [
      "agent.name"
  ],
  "_source": "False"
}'| jq .hits.hits[].fields | head -n 10

{
  "agent.name": [
    "test2.example.com"
  ]
}

We used DLS filter quite extensively before on our prodction cluster

Regards,
Alex

pablo · November 4, 2022, 6:16pm

@trauta Did you migrate your environment to FLX recently or this is a new deployment?
If migrated, is the migration of all nodes in the cluster completed?

trauta · November 7, 2022, 9:36am

Hi,

I’ve migrated our dev and production cluster from SG7 to FLX a few weeks ago. All nodes are running 1.0.0-es-7.17.7 of the FLX plugin.

Regards
Alex

trauta · November 10, 2022, 1:14pm

Hi,

I have some additional observations:

The multitenancy feature is only broken if the logged in user is assigned to the SGS_KIBANA_USER_NO_DEFAULT_TENANT role or when the 'Global‘ tenant is disabled via the searchguard.multitenancy.tenants.enable_global setting in the kibana.yml.

When leaving the ‘Global’ tenant is active, a user can select the other tenants without problems.

BUT: Even then, the new FLX DLS implementation is somehow broken, the user can still see events that he is not supposed to access.

The following config still allows access to all events of the filebeat index.

TEST:
  description: ""
  index_permissions:
  - allowed_actions:
    - "SGS_GET"
    - "SGS_READ"
    - "SGS_SEARCH"
    index_patterns:
    - "filebeat-*"
    fls: []
    dls: "{\"bool\":{\"must\": {\"match\": {\"agent.name\": \"test02.example.com\"\
      }}}}"
    masked_fields: []
  tenant_permissions:
  - allowed_actions:
    - "SGS_KIBANA_ALL_WRITE"
    tenant_patterns:
    - "TEST"
  cluster_permissions:
  - "SGS_CREATE_MANAGE_OWN_AUTH_TOKEN"
  exclude_cluster_permissions: []

Regards,
Alex

nils · November 16, 2022, 5:27pm

Hi Alex,

thank you for the information. We believe this is a bug and will look into this as soon as possible.

You can track the progress here:

nils · November 16, 2022, 5:38pm

I have one request: Would it be possible that you run the command ./sgctl.sh component-state and provide us the resulting output. You can also send me the output by private message.

Please keep in mind that the output of the command might be very long.

trauta · November 17, 2022, 8:10am

Hi Nils,

thank you very much for looking into this.

I will send you the (slightly anonymized) output of the component-state command from our development cluster via private message.

pablo · November 18, 2022, 4:53pm

@trauta Could you share the roles that the reported user is assigned?

Please also run the below command and share the output.

curl --insecure -u <user> -XGET https://ES_HOST:9200/_searchguard/authinfo?pretty

nils · November 18, 2022, 5:33pm

@trauta We have an experimental fix for this. Would you mind to try it whether it fixes the issues?

You can find it at https://maven.search-guard.com:443/search-guard-flx-snapshot/com/floragunn/search-guard-flx-elasticsearch-plugin/b-flx-dls-no-index-handling-SNAPSHOT/search-guard-flx-elasticsearch-plugin-b-flx-dls-no-index-handling-20221118.154911-2.zip

It is built for ES 7.17.7.

trauta · November 21, 2022, 9:30am

Hi,

here is the assigned role:

TEST:
  description: ""
  index_permissions:
  - allowed_actions:
    - "SGS_GET"
    - "SGS_READ"
    - "SGS_SEARCH"
    index_patterns:
    - "syslog-*"
    fls: []
    dls: "{\"bool\": {\"must\": {\"match\": {\"agent.name\": \"test01.example.com\"\
      }}}}"
    masked_fields: []
  tenant_permissions:
  - allowed_actions:
    - "SGS_KIBANA_ALL_WRITE"
    tenant_patterns:
    - "TEST"
  cluster_permissions:
  - "SGS_CREATE_MANAGE_OWN_AUTH_TOKEN"
  exclude_cluster_permissions: []
  exclude_index_permissions: []

And here is the _searchguard/authinfo API output:

{
  "user" : "User USER <basic/ldap> [backend_roles=[USER, it]]",
  "user_name" : "USER",
  "user_requested_tenant" : null,
  "remote_address" : "1.2.3.4:38858",
  "backend_roles" : [
    "USER",
    "it"
  ],
  "custom_attribute_names" : [ ],
  "attribute_names" : [ ],
  "sg_roles" : [
    "TEST",
    "SGS_KIBANA_USER"
  ],
  "sg_tenants" : {
    "USER" : true,
    "TEST" : true,
    "SGS_GLOBAL_TENANT" : true
  },
  "principal" : null,
  "peer_certificates" : "0",
  "cluster_name" : "dev"
}

trauta · November 23, 2022, 3:00pm

Hi @nils,

I have installed your bugfix on our dev cluster.

When the user logs in, there is no more “Internal server error” message and no error messages in the ES logs.

However, after logging in, the menu for changing tenants is no longer displayed in Kibana.

When entering the ‘Discover’ menu there is an error message saying “Error loading Discover - Forbidden”.
There are no error messages in the ES logs, but in the Kibana logs there is the following message:

{"type":"log","@timestamp":"2022-11-23T15:54:05+01:00","tags":["error","plugins","searchguard"],"pid":144903,"message":"Fail to create the index \".kibana_7.17. 7_001\" for tenant \"\", {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"Insufficient permissions\"}],\"type\":\"security_exception\",\"reason\":\"Insufficient permissions\"},\"status\":403}"}

If you add sgtenant=TEST to the Kibana URL, Kibana switches to the TEST tenant and the tenant menu is also visible again.
It seems that there are problems with the default private tenant.

Regards,
Alex

nils · November 24, 2022, 12:05pm

Hi Alex,

thank you for the update!

Could you also share your kibana.yml configuration? Especially, it would be interesting for us what tenant is displayed by default? Is it the private tenant?

Could you check whether now the DLS rules are effective?

trauta · November 24, 2022, 9:39pm

Hi,

this is the kibana.yml:

server.port: 8443
server.host: "kibana.example.com"
elasticsearch.hosts:
  - "https://es01.example.com:9200"
  - "https://es02.example.com:9200"
elasticsearch.username: "kibanaserver"
elasticsearch.password: ""
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/kibana.example.com.pem
server.ssl.key: /etc/kibana/kibana.example.com.key
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/chain.cer" ]
elasticsearch.ssl.verificationMode: full
elasticsearch.requestHeadersWhitelist: ["sgtenant", "Authorization"]
xpack:
  security.enabled: false
  spaces.enabled: false
searchguard.multitenancy.enabled: true

Unfortunately, the DLS query still does not work (tested with use_impl: "flx").

I just discovered something else strange. After logging into Kibana, the elasticsearch.log reports:

[2022-11-24T22:35:38,082][WARN ][c.f.s.a.b.RequestAuthenticationProcessor] [es01.example.com] Authentication failed for null from [request=/_searchguard/authinfo, directIpAddress=<KIBANA IP>, originatingIpAddress=<KIBANA IP>, clientCertSubject=null]

Might this have something to do with the bug?

Regards,
Alex

trauta · November 24, 2022, 9:52pm

Hi,

adding searchguard.multitenancy.tenants.preferred: ["Private"] to the kibana.yml fixes the problem with the disappearing tenant menu.

I’m not quite sure if this is really required, I thought the Private tenant is the always default tenant.
Or was this behavior maybe changed with FLX?

This does not fix the non-working DLS filter, unfortunately

Regards,
Alex

pablo · November 25, 2022, 5:15pm

@trauta The observed error could be caused by another authentication domain. What authentication domains did you configure in config.yml?

trauta · November 25, 2022, 6:02pm

Hi,

this is my sg_authc.yml:

auth_domains:
- type: "basic/ldap"
  ldap:
    idp:
      hosts:
      - "ldaps://..."
      bind_dn: "..."
      password: "..."
      tls:
        verify_hostnames: false
        trusted_cas: "#{file:...}"
    user_search:
      base_dn: "ou=people.."
      filter:
        raw: "(uid=${user.name})"
    group_search:
      base_dn: "ou=group..."
      filter:
        raw: "(uniqueMember=${dn})"
      role_name_attribute: "cn"
      recursive:
        enabled: true
    users:
      skip:
      - "admin"
      - ...
- type: "basic/internal_users_db"
- type: "clientcert"
  user_mapping:
    user_name:
      from: "$.clientcert.subject[\"CN\"]"

nils · November 29, 2022, 8:52am

We are still working on the analysis of this - I am having some more questions:

Can you please post the mapping definition of the index you are applying DLS on?
Is there any chance that you are using nested attributes in that index?

nils · November 29, 2022, 3:22pm

@trauta

We have created a snapshot with some more logging:

https://maven.search-guard.com:443/search-guard-flx-snapshot/com/floragunn/search-guard-flx-elasticsearch-plugin/b-dls-logging-SNAPSHOT/search-guard-flx-elasticsearch-plugin-b-dls-logging-20221129.151431-2.zip

Would you mind to try the DLS with this snapshot and pass the logs to us? Of course, using private messaging.

You would need to add this to log4j2.properties:

logger.dls.name = com.floragunn.searchguard.enterprise.dlsfls
logger.dls.level = trace
logger.authz.name = com.floragunn.searchguard.authz
logger.authz.level = trace

If you could also provide us once again the output of sgctl.sh component-state after having done the test, that would be very helpful as well.

Sorry for the inconveniences!

nils · December 6, 2022, 9:26am

I have just noticed that you are using a match query for the DLS query. This will not search for exact strings, but tokenize the search string and perform a fuzzy full text search. In this case, you might want to use a term query to get exact matching.

Topic		Replies	Views
Security Release: FLX 1.4.0 and Classic 53.8 Announcements	1	430	December 14, 2023
Testers wanted! Our new Kibana multi tenancy module for ES/KI 5.2.2 has arrived as first beta! Search Guard	3	397	March 30, 2017
Now available: Search Guard FLX 1.2.0 Announcements	1	706	June 23, 2023
Search Guard FLX 1.5.0 released Announcements	1	270	March 7, 2024
Search Guard v15 and Kibana Plugin v4 release - Critical security issue fixed Search Guard	0	383	August 8, 2017

Search Guard FLX DLS/FLS breaks Kibana multi-tenancy

Related topics