Attribute based indices filtering inside index pattern

Search Guard
1

  • Search Guard and Elasticsearch version: 6.5.1 (SG, ES & K)

  • Installed and used enterprise modules, if any: None

  • JVM version and operating system version: openjdk 1.8

  • Search Guard configuration files: see end of message

  • Elasticsearch log messages on debug level: not an ES problem

  • Other installed Elasticsearch or Kibana plugins, if any: None

We’ve been using the Elastic stack to gather data on the usage of our applications by our clients.

We collect data and store it in ES in indices that contain the client’s UID: data_0 for client 0, data_1 for client 1 etc.

In Kibana, we have an index pattern data_* that allows us to create visualizations for these data.

Say we want to give access to dashboard / visualizations to our clients, but only for their own data.

We would also like to limit the per-client configuration as we have a lot of clients. (We don’t want to re-create 1 visualisation per client, or 1 tenant per client, we don’t want to have 1 index pattern per client etc.)

One way we can do it, is to create a user per client, add a field ‘client_id’ to this user and use it in the DLS query of a generic ‘client_user’ to filter only corresponding documents.

But the first thing we tried was to only allow some indices to each client (we sorted it by indices for that purpose, and it should be much more efficient than DLS), for example ‘data_1’ for the user of client 1. We expected the index pattern ‘data_*’ in Kibana to work but only retrieve data from ‘data_1’, but instead it says:

Discover: no permissions for [indices:data/read/search] and User [name=user_1, roles=[client_user], requestedTenant=null]

And the only way to fix this message is to give READ access to ‘data_*’, thus losing the data protection of indices that do not belong to the user’s client.

Is there a way to achieve this index-level filter while keeping the same index-patterns in Kibana for two different users?

Side note: I tried to attach the config files, but I can’t find them… We’re using the GUI to set our system, but the files aren’t written in /plugins/search-guard-6/sgconfig/sg_internal_users.yml, these files only contain static data, changes made in Kibana are not reflected there, and we cannot find where they’re written, system only has one directory called ‘sgconfig’ and it’s this one. We’re running the stack as a service. The SG data is persisted without any problem.

Regards.

2

Yes, there is. Kibana uses a lot of wildcard queries when talking to Elasticsearch. The default setting in Search Guard is to fail a query with a 403 if the user does not have access to one or more indices the wildcard expression expands to. To tell search guard not to fail the query completely, but only return data from accessible indices, in sg_config.yml set:

searchguard:
dynamic:
kibana:
…
do_not_fail_on_forbidden**: true**

``

And apply the changes via sgadmin. This setting will also become the default in SG7.

Let me know if this fixed the issue. If not, please also attach the ES log file at the time when the error in Kibana arises.

···

On Wednesday, November 28, 2018 at 12:26:43 PM UTC+1, marc.zominy@hoomano.com wrote:

  • Search Guard and Elasticsearch version: 6.5.1 (SG, ES & K)
  • Installed and used enterprise modules, if any: None
  • JVM version and operating system version: openjdk 1.8
  • Search Guard configuration files: see end of message
  • Elasticsearch log messages on debug level: not an ES problem
  • Other installed Elasticsearch or Kibana plugins, if any: None

We’ve been using the Elastic stack to gather data on the usage of our applications by our clients.

We collect data and store it in ES in indices that contain the client’s UID: data_0 for client 0, data_1 for client 1 etc.

In Kibana, we have an index pattern data_* that allows us to create visualizations for these data.

Say we want to give access to dashboard / visualizations to our clients, but only for their own data.

We would also like to limit the per-client configuration as we have a lot of clients. (We don’t want to re-create 1 visualisation per client, or 1 tenant per client, we don’t want to have 1 index pattern per client etc.)

One way we can do it, is to create a user per client, add a field ‘client_id’ to this user and use it in the DLS query of a generic ‘client_user’ to filter only corresponding documents.

But the first thing we tried was to only allow some indices to each client (we sorted it by indices for that purpose, and it should be much more efficient than DLS), for example ‘data_1’ for the user of client 1. We expected the index pattern ‘data_*’ in Kibana to work but only retrieve data from ‘data_1’, but instead it says:

Discover: no permissions for [indices:data/read/search] and User [name=user_1, roles=[client_user], requestedTenant=null]

And the only way to fix this message is to give READ access to ‘data_*’, thus losing the data protection of indices that do not belong to the user’s client.

Is there a way to achieve this index-level filter while keeping the same index-patterns in Kibana for two different users?

Side note: I tried to attach the config files, but I can’t find them… We’re using the GUI to set our system, but the files aren’t written in /plugins/search-guard-6/sgconfig/sg_internal_users.yml, these files only contain static data, changes made in Kibana are not reflected there, and we cannot find where they’re written, system only has one directory called ‘sgconfig’ and it’s this one. We’re running the stack as a service. The SG data is persisted without any problem.

Regards.

3

Thank you for your answer. I have succesfully changed the mentioned setting, and the outcome has changed a bit, but it’s not helping.

EDIT: Neverming, it’s working, I had made a mistake.

Thank you very much!

4

Although I had a few troubles with the documentation that may be worth correcting:

If you use the certificates generated by the demo installation script, cd in the <ES installation directory>/plugins/search-guard-6/tools/ and execute:

./sgadmin.sh -cd ..   /sgconfig/ -icl -nhnv -cacert ../../..   /config/root-ca.pem -cert ../../..   /config/kirk.pem -key ../../../config/kirk-key.pem

Well I had used the demo script, but the above command only did this:

/usr/share/elasticsearch/plugins/search-guard-6/tools# ./sgadmin.sh --diagnose -cd …/sgconfig/ -icl -nhnv -cacert …/…/…/config/root-ca.pem -cert …/…/…/config/kirk.pem -key …/…/…/config/kirk-key.pem
WARNING: JAVA_HOME not set, will use /usr/bin/java
Search Guard Admin v6
Will connect to localhost:9300 … done
ERR: An unexpected IllegalStateException occured: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
Trace:

I tried adding –diagnose seen here: (BTW, doc is unclear about whether to used -diagnose or --diagnose)

Using the diagnose switch

If you cannot find out why sgadmin is not executing, add the --diagnose switch to gather debug information, for example

./sgadmin.sh -diagnose -cd ../sgconfig/ -cacert ... -cert ... -key ... -keypass ...

But this option did not change the output at all.

I the found out that simply …/…/…/config/root-ca.pem and the other files did not exist at this path, but simply in the current dir (config)

The command should have told me that the files didn’t exist I guess…

So the command I had to use was ultimately:

/usr/share/elasticsearch/plugins/search-guard-6/tools# ./sgadmin.sh -cd …/sgconfig/ -icl -nhnv -cacert root-ca.pem -cert kirk.pem -key kirk-key.pem

And I think I followed the usual demo installation steps, I did not move the certificates or anything.

Also, doing so removed all the previously created data (users, backend roles etc.), and reset them to defaults (content of the files), is there a way to run this command without losing existing data?

Again, thank you.

···

On Wednesday, November 28, 2018 at 4:22:11 PM UTC+1, marc....@hoomano.com wrote:

Thank you for your answer. I have succesfully changed the mentioned setting, and the outcome has changed a bit, but it’s not helping.

EDIT: Neverming, it’s working, I had made a mistake.

Thank you very much!

5

The next release contains a fix for sgadminto be more verbose about missing files etc. Thx for pointing this out.

To backup your data you can run sgadmin with the -r option. Pls refer to

···

On Wednesday, 28 November 2018 17:20:56 UTC+1, marc.zominy wrote:

Although I had a few troubles with the documentation that may be worth correcting:

If you use the certificates generated by the demo installation script, cd in the <ES installation directory>/plugins/search-guard-6/tools/ and execute:

./sgadmin.sh -cd ..   /sgconfig/ -icl -nhnv -cacert ../../..   /config/root-ca.pem -cert ../../..   /config/kirk.pem -key ../../../config/kirk-key.pem

Well I had used the demo script, but the above command only did this:

/usr/share/elasticsearch/plugins/search-guard-6/tools# ./sgadmin.sh --diagnose -cd …/sgconfig/ -icl -nhnv -cacert …/…/…/config/root-ca.pem -cert …/…/…/config/kirk.pem -key …/…/…/config/kirk-key.pem
WARNING: JAVA_HOME not set, will use /usr/bin/java
Search Guard Admin v6
Will connect to localhost:9300 … done
ERR: An unexpected IllegalStateException occured: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
Trace:

I tried adding –diagnose seen here: (BTW, doc is unclear about whether to used -diagnose or --diagnose)

Using the diagnose switch

If you cannot find out why sgadmin is not executing, add the --diagnose switch to gather debug information, for example

./sgadmin.sh -diagnose -cd ../sgconfig/ -cacert ... -cert ... -key ... -keypass ...

But this option did not change the output at all.

I the found out that simply …/…/…/config/root-ca.pem and the other files did not exist at this path, but simply in the current dir (config)

The command should have told me that the files didn’t exist I guess…

So the command I had to use was ultimately:

/usr/share/elasticsearch/plugins/search-guard-6/tools# ./sgadmin.sh -cd …/sgconfig/ -icl -nhnv -cacert root-ca.pem -cert kirk.pem -key kirk-key.pem

And I think I followed the usual demo installation steps, I did not move the certificates or anything.

Also, doing so removed all the previously created data (users, backend roles etc.), and reset them to defaults (content of the files), is there a way to run this command without losing existing data?

Again, thank you.

On Wednesday, November 28, 2018 at 4:22:11 PM UTC+1, marc....@hoomano.com wrote:

Thank you for your answer. I have succesfully changed the mentioned setting, and the outcome has changed a bit, but it’s not helping.

EDIT: Neverming, it’s working, I had made a mistake.

Thank you very much!