Search Guard FLX DLS/FLS breaks Kibana multi-tenancy

Hi,

When using the new FLX implementation for DLS/FLS security the multi-tenancy feature in Kibana seems to be broken.

After setting use_impl: flx in sg_authz_dlsfls.yml a logged in user in Kibana gets a HTTP 500 Internal Server Error when trying to access Kibana after login.

Here are the (anonymized) Elasticsearch log messages:

[2022-11-04T09:28:08,390][WARN ][r.suppressed             ] [es-node1] path: /.kibana, params: {index=.kibana}
org.elasticsearch.transport.RemoteTransportException: [es-node2][1.2.3.4:9300][indices:admin/get]
Caused by: java.lang.IllegalStateException: Index [.kibana_109026154_USER] is not registered in com.floragunn.searchguard.enterprise.dlsfls.RoleBasedDocumentAuthorization$StatefulIndexQueries@2419ca2b
    at com.floragunn.searchguard.enterprise.dlsfls.RoleBasedDocumentAuthorization.hasDlsRestrictions(RoleBasedDocumentAuthorization.java:84) ~[?:?]
    at com.floragunn.searchguard.enterprise.dlsfls.DlsFlsValve.apply(DlsFlsValve.java:127) ~[?:?]
...

[2022-11-04T09:28:08,435][WARN ][r.suppressed             ] [es-node1] path: /.kibana_7.17.7_001, params: {index=.kibana_7.17.7_001}
org.elasticsearch.transport.RemoteTransportException: [es-node2][1.2.3.4:9300][indices:admin/create]
Caused by: java.lang.IllegalStateException: Index [.kibana_109026154_USER_7.17.7_001] is not registered in com.floragunn.searchguard.enterprise.dlsfls.RoleBasedDocumentAuthorization$StatefulIndexQueries@2419ca2b
    at com.floragunn.searchguard.enterprise.dlsfls.RoleBasedDocumentAuthorization.hasDlsRestrictions(RoleBasedDocumentAuthorization.java:84) ~[?:?]
    at com.floragunn.searchguard.enterprise.dlsfls.DlsFlsValve.apply(DlsFlsValve.java:127) ~[?:?]
..

[2022-11-04T09:28:08,470][ERROR][c.f.s.e.d.DlsFlsValve    ] [es-node1] Error while evaluating DLS/FLS privileges
java.lang.IllegalStateException: Index [.kibana_109026154_USER_7.17.7] is not registered in com.floragunn.searchguard.enterprise.dlsfls.RoleBasedDocumentAuthorization$StatefulIndexQueries@7b209981
    at com.floragunn.searchguard.enterprise.dlsfls.RoleBasedDocumentAuthorization.hasDlsRestrictions(RoleBasedDocumentAuthorization.java:84) ~[dlic-search-guard-flx-dlsfls-1.0.0-es-7.17.7.jar:1.0.0-es-7.17.7]
..

[2022-11-04T09:28:08,489][ERROR][c.f.s.f.SearchGuardFilter] [es-node1] Exception while handling indices:data/write/bulk; org.elasticsearch.action.bulk.BulkRequest/unset
java.lang.IllegalStateException: Index [.kibana_109026154_USER_7.17.7] is not registered in com.floragunn.searchguard.enterprise.dlsfls.RoleBasedDocumentAuthorization$StatefulIndexQueries@7b209981
    at com.floragunn.searchguard.enterprise.dlsfls.RoleBasedDocumentAuthorization.hasDlsRestrictions(RoleBasedDocumentAuthorization.java:84) ~[dlic-search-guard-flx-dlsfls-1.0.0-es-7.17.7.jar:1.0.0-es-7.17.7]
    at com.floragunn.searchguard.enterprise.dlsfls.DlsFlsValve.apply(DlsFlsValve.java:127) ~[dlic-search-guard-flx-dlsfls-1.0.0-es-7.17.7.jar:1.0.0-es-7.17.7]
..

[2022-11-04T09:28:08,508][WARN ][r.suppressed             ] [es-node1] path: /.kibana_7.17.7/_create/config%3A7.17.7, params: {require_alias=true, refresh=wait_for, index=.kibana_7.17.7, id=config:7.17.7, op_type=create}
java.lang.IllegalStateException: Index [.kibana_109026154_USER_7.17.7] is not registered in com.floragunn.searchguard.enterprise.dlsfls.RoleBasedDocumentAuthorization$StatefulIndexQueries@7b209981
    at com.floragunn.searchguard.enterprise.dlsfls.RoleBasedDocumentAuthorization.hasDlsRestrictions(RoleBasedDocumentAuthorization.java:84) ~[dlic-search-guard-flx-dlsfls-1.0.0-es-7.17.7.jar:1.0.0-es-7.17.7]
    at com.floragunn.searchguard.enterprise.dlsfls.DlsFlsValve.apply(DlsFlsValve.java:127) ~[dlic-search-guard-flx-dlsfls-1.0.0-es-7.17.7.jar:1.0.0-es-7.17.7]
...

If desired, I can provide the full Java stack trace.

To my understand it seems that Kibana is unable to create the hidden Kibana system indices.

It seems to do not matter if the assigned role for the user has a DLS or FLS restriction configured or not.
When reverting sg_authz_dlsfls.yml back to use_impl: legacy the user is able to access Kibana.

We excessively used DLS before upgrading to FLX without any problems.

Regards,
Alex

Elasticsearch version:
7.17.7

Server OS version:
Ubuntu 22.04.1 LTS

Kibana version (if relevant):
7.17.7

Hi,

also, if reverting the DLS implementation back to legacy any DLS filter that worked before upgrading to FLX does not do anything.

Here is an example of role with a DLS filter:

TEST:
  description: ""
  index_permissions:
  - allowed_actions:
    - "SGS_GET"
    - "SGS_READ"
    - "SGS_SEARCH"
    index_patterns:
    - "filebeat-*"
    fls: []
    dls: "{\"bool\":{\"must\": {\"match\": {\"agent.name\": \"test.example.com\"\
      }}}}"
    masked_fields: []
  tenant_permissions:
  - allowed_actions:
    - "SGS_KIBANA_ALL_WRITE"
    tenant_patterns:
    - "Test"
  cluster_permissions: []
  exclude_cluster_permissions: []
  exclude_index_permissions: []

This roles should restrict access to events containing agent.name: test.example.com but the user can access all events of this index, for example events containing agent.name: test2.example.com

curl ... -H "Content-type: application/json" -X GET "https://ES_HOST:9200/filebeat-*/_search?pretty" -d '
{
  "query": {
      "bool": {
          "must": {
              "match": {
                  "agent.name": "test2.example.com"
              }
          }
      }
  },
  "fields": [
      "agent.name"
  ],
  "_source": "False"
}'| jq .hits.hits[].fields | head -n 10

{
  "agent.name": [
    "test2.example.com"
  ]
}

We used DLS filter quite extensively before on our prodction cluster :confused:

Regards,
Alex

@trauta Did you migrate your environment to FLX recently or this is a new deployment?
If migrated, is the migration of all nodes in the cluster completed?

Hi,

I’ve migrated our dev and production cluster from SG7 to FLX a few weeks ago. All nodes are running 1.0.0-es-7.17.7 of the FLX plugin.

Regards
Alex

Hi,

I have some additional observations:

The multitenancy feature is only broken if the logged in user is assigned to the SGS_KIBANA_USER_NO_DEFAULT_TENANT role or when the 'Global‘ tenant is disabled via the searchguard.multitenancy.tenants.enable_global setting in the kibana.yml.

When leaving the ‘Global’ tenant is active, a user can select the other tenants without problems.

BUT: Even then, the new FLX DLS implementation is somehow broken, the user can still see events that he is not supposed to access.

The following config still allows access to all events of the filebeat index.

TEST:
  description: ""
  index_permissions:
  - allowed_actions:
    - "SGS_GET"
    - "SGS_READ"
    - "SGS_SEARCH"
    index_patterns:
    - "filebeat-*"
    fls: []
    dls: "{\"bool\":{\"must\": {\"match\": {\"agent.name\": \"test02.example.com\"\
      }}}}"
    masked_fields: []
  tenant_permissions:
  - allowed_actions:
    - "SGS_KIBANA_ALL_WRITE"
    tenant_patterns:
    - "TEST"
  cluster_permissions:
  - "SGS_CREATE_MANAGE_OWN_AUTH_TOKEN"
  exclude_cluster_permissions: []

Regards,
Alex

Hi Alex,

thank you for the information. We believe this is a bug and will look into this as soon as possible.

You can track the progress here:

I have one request: Would it be possible that you run the command ./sgctl.sh component-state and provide us the resulting output. You can also send me the output by private message.

Please keep in mind that the output of the command might be very long.

Hi Nils,

thank you very much for looking into this.

I will send you the (slightly anonymized) output of the component-state command from our development cluster via private message.

@trauta Could you share the roles that the reported user is assigned?

Please also run the below command and share the output.

curl --insecure -u <user> -XGET https://ES_HOST:9200/_searchguard/authinfo?pretty 

@trauta We have an experimental fix for this. Would you mind to try it whether it fixes the issues?

You can find it at https://maven.search-guard.com:443/search-guard-flx-snapshot/com/floragunn/search-guard-flx-elasticsearch-plugin/b-flx-dls-no-index-handling-SNAPSHOT/search-guard-flx-elasticsearch-plugin-b-flx-dls-no-index-handling-20221118.154911-2.zip

It is built for ES 7.17.7.

Hi,

here is the assigned role:

TEST:
  description: ""
  index_permissions:
  - allowed_actions:
    - "SGS_GET"
    - "SGS_READ"
    - "SGS_SEARCH"
    index_patterns:
    - "syslog-*"
    fls: []
    dls: "{\"bool\": {\"must\": {\"match\": {\"agent.name\": \"test01.example.com\"\
      }}}}"
    masked_fields: []
  tenant_permissions:
  - allowed_actions:
    - "SGS_KIBANA_ALL_WRITE"
    tenant_patterns:
    - "TEST"
  cluster_permissions:
  - "SGS_CREATE_MANAGE_OWN_AUTH_TOKEN"
  exclude_cluster_permissions: []
  exclude_index_permissions: []

And here is the _searchguard/authinfo API output:

{
  "user" : "User USER <basic/ldap> [backend_roles=[USER, it]]",
  "user_name" : "USER",
  "user_requested_tenant" : null,
  "remote_address" : "1.2.3.4:38858",
  "backend_roles" : [
    "USER",
    "it"
  ],
  "custom_attribute_names" : [ ],
  "attribute_names" : [ ],
  "sg_roles" : [
    "TEST",
    "SGS_KIBANA_USER"
  ],
  "sg_tenants" : {
    "USER" : true,
    "TEST" : true,
    "SGS_GLOBAL_TENANT" : true
  },
  "principal" : null,
  "peer_certificates" : "0",
  "cluster_name" : "dev"
}

Hi @nils,

I have installed your bugfix on our dev cluster.

When the user logs in, there is no more “Internal server error” message and no error messages in the ES logs.

However, after logging in, the menu for changing tenants is no longer displayed in Kibana.

When entering the ‘Discover’ menu there is an error message saying “Error loading Discover - Forbidden”.
There are no error messages in the ES logs, but in the Kibana logs there is the following message:

{"type":"log","@timestamp":"2022-11-23T15:54:05+01:00","tags":["error","plugins","searchguard"],"pid":144903,"message":"Fail to create the index \".kibana_7.17. 7_001\" for tenant \"\", {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"Insufficient permissions\"}],\"type\":\"security_exception\",\"reason\":\"Insufficient permissions\"},\"status\":403}"}

If you add sgtenant=TEST to the Kibana URL, Kibana switches to the TEST tenant and the tenant menu is also visible again.
It seems that there are problems with the default private tenant.

Regards,
Alex

Hi Alex,

thank you for the update!

Could you also share your kibana.yml configuration? Especially, it would be interesting for us what tenant is displayed by default? Is it the private tenant?

Could you check whether now the DLS rules are effective?

Hi,

this is the kibana.yml:

server.port: 8443
server.host: "kibana.example.com"
elasticsearch.hosts:
  - "https://es01.example.com:9200"
  - "https://es02.example.com:9200"
elasticsearch.username: "kibanaserver"
elasticsearch.password: ""
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/kibana.example.com.pem
server.ssl.key: /etc/kibana/kibana.example.com.key
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/chain.cer" ]
elasticsearch.ssl.verificationMode: full
elasticsearch.requestHeadersWhitelist: ["sgtenant", "Authorization"]
xpack:
  security.enabled: false
  spaces.enabled: false
searchguard.multitenancy.enabled: true

Unfortunately, the DLS query still does not work (tested with use_impl: "flx").

I just discovered something else strange. After logging into Kibana, the elasticsearch.log reports:

[2022-11-24T22:35:38,082][WARN ][c.f.s.a.b.RequestAuthenticationProcessor] [es01.example.com] Authentication failed for null from [request=/_searchguard/authinfo, directIpAddress=<KIBANA IP>, originatingIpAddress=<KIBANA IP>, clientCertSubject=null]

Might this have something to do with the bug?

Regards,
Alex

Hi,

adding searchguard.multitenancy.tenants.preferred: ["Private"] to the kibana.yml fixes the problem with the disappearing tenant menu.

I’m not quite sure if this is really required, I thought the Private tenant is the always default tenant.
Or was this behavior maybe changed with FLX?

This does not fix the non-working DLS filter, unfortunately :frowning:

Regards,
Alex

@trauta The observed error could be caused by another authentication domain. What authentication domains did you configure in config.yml?

Hi,

this is my sg_authc.yml:

auth_domains:
- type: "basic/ldap"
  ldap:
    idp:
      hosts:
      - "ldaps://..."
      bind_dn: "..."
      password: "..."
      tls:
        verify_hostnames: false
        trusted_cas: "#{file:...}"
    user_search:
      base_dn: "ou=people.."
      filter:
        raw: "(uid=${user.name})"
    group_search:
      base_dn: "ou=group..."
      filter:
        raw: "(uniqueMember=${dn})"
      role_name_attribute: "cn"
      recursive:
        enabled: true
    users:
      skip:
      - "admin"
      - ...
- type: "basic/internal_users_db"
- type: "clientcert"
  user_mapping:
    user_name:
      from: "$.clientcert.subject[\"CN\"]"

We are still working on the analysis of this - I am having some more questions:

  • Can you please post the mapping definition of the index you are applying DLS on?
  • Is there any chance that you are using nested attributes in that index?

@trauta

We have created a snapshot with some more logging:

https://maven.search-guard.com:443/search-guard-flx-snapshot/com/floragunn/search-guard-flx-elasticsearch-plugin/b-dls-logging-SNAPSHOT/search-guard-flx-elasticsearch-plugin-b-dls-logging-20221129.151431-2.zip

Would you mind to try the DLS with this snapshot and pass the logs to us? Of course, using private messaging.

You would need to add this to log4j2.properties:

logger.dls.name = com.floragunn.searchguard.enterprise.dlsfls
logger.dls.level = trace
logger.authz.name = com.floragunn.searchguard.authz
logger.authz.level = trace

If you could also provide us once again the output of sgctl.sh component-state after having done the test, that would be very helpful as well.

Sorry for the inconveniences!