Search Guard incompatible with data streams?

Search Guard
1

Hi, I am running an Elasticsearch cluster with a Search Guard Scientific license.

The cluster is primarily used for log events. Up to now, daily indices were created like SERVICE-%{+YYYY.MM.dd} for example. Recently I converted all indices to data streams.

After migrating the existing indices to data streams, the existing Search Guard roles seem to be incompatible with the data streams.

On a test cluster, I have recreated following scenario for debugging purposes.

Two data streams exists:

  • logstash-SERVICE1-default
  • logstash-SERVICE2-default

sg_roles.yml:

TEST:
  reserved: false
  hidden: false
  description: ""
  cluster_permissions:
  - "SGS_CLUSTER_COMPOSITE_OPS_RO"
  index_permissions:
  - index_patterns:
    - "logstash-SERVICE1-default"
    dls: []
    fls: []
    masked_fields: []
    allowed_actions:
    - "SGS_GET"
    - "SGS_READ"
    - "SGS_SEARCH"
  tenant_permissions:
  - tenant_patterns:
    - "test"
    allowed_actions:
    - "SGS_KIBANA_ALL_READ"
    - "SGS_KIBANA_ALL_WRITE"
    - "SGS_SIGNALS_WATCH_MANAGE"
  exclude_cluster_permissions: []
  exclude_index_permissions: []
  static: false

Despite this role a user can access both data streams with the following index pattern logstash-*.

Even an exclude_index_permissions setting for the data stream logstash-SERVICE2-default does not prevent this.
Also, without any configured index permissions (includes and excludes) I can access both data stream in Kibana and via the Elasticsearch REST APIs.

Is Search Guard unable to handle data streams?
I have not found anything about this in the documentation pages.

I hope you can help me.

Regards,
Alex

Elasticsearch version:
7.17.4

Server OS version:
Ubuntu 20.04.5 LTS

Kibana version:
7.17.4

Search Guard version:
7.17.4-53.1.0

2

Hi,

a small but disturbing update:

Even if I remove every non-default role and action group, an authenticated user can access all data stream events via the _search API.

Requests to every other regular indices are blocked, but every data stream can be accessed via the logstash-* index pattern.

Direct access to a specific data stream is blocked, but once you add an * anywhere in the index string the data is retrievable.

I’m using LDAP authentication and sg_roles.yml and sg_roles_mapping.yml are empty.

This seems to be a major security flaw!
Why does Search Guard allow unrestricted access to all data streams?

Regards,
Alex

4

Hi Alex - this of course does not seem right, we are investigating and get back to you asap.

7

Hi, any updates on this?

8

Hi, we have analyzed the issue. Since data streams are an X-Pack feature in Elasticsearch 7.x, we did not officially support them. However, we already changed the implementation for Search Guard FLX, so the behavior you are describing is not present anymore in FLX, which is available for 7.17.x and the upcoming FLX for 8.x (Latest Releases | Security for Elasticsearch | Search Guard).

Having said that, we are looking into ways to mitigate the behavior also for older versions of Search Guard and will keep you posted.

9

Hi, thank you for your reply! I didn’t know that data streams are a X-Pack feature. Where did you get that information from? I don’t see any hints on this on the documentation sites.

I’ve updated our dev cluster to Search Guard FLX and I can happily confirm that the update fixes the security problem. Once I finished testing all features I will migrate our prod cluster, so we can safely use data streams again.

One more small hint, after upgrading to FLX the built-in admin user is unable to see the data streams in the ‚Index Management‘ → ‚Data Streams‘ view in Kibana. There is no error messages, it simply says that there are no existing data streams. Calling the GET /_data_stream/ API also results in an empty response, but fetching a specific existing stream like GET /_data_stream/logstash-SERVICE1-default works. This is only a small problem, but it would simplify the handling of the data streams.

10

Thank you for the report; I filed an issue in our Gitlab repo to check what’s the matter with GET /_data_steam/ API: GET /_data_stream/ API returns an empty response (#119) · Issues · search-guard / Search Guard Suite Enterprise · GitLab

12

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

14

We fixed this with Search Guard 53.5.0. Search Guard FLX for Elasticsearch 7 and 8 is not affected.