Index patterns are set for all tenants in Kibana Multitenancy

Search Guard
1

Hi,

I am running search guard 20.1 on elasticsearch/kibana 6.1.1

I want to set up an index pattern A for tenant A and an index pattern B for tenant B.

But then i can find index pattern A and B in both tenants. When i set up index pattern A in A, i find it in all other tenant.

What are the configuration issues ?

Here is what I added in my kibana.yml config

searchguard.multitenancy.enabled: true
searchguard.multitenancy.tenants.enable_global: false
searchguard.multitenancy.tenants.enable_private: false

elasticsearch.requestHeadersWhitelist: [“sgtenant”, “Authorization”]

``

2

Hi Charlotte,

I think your last line should read sg_tenant instead of sgtenant in the whitelist

3

That might be a bug in the 6.1.1 line of Search Guard. Thanks for reporting, we’re checking this and get back to you soon.

“sgtenant” is actually correct. We changed that in SG6 because some proxies cannot cope with underlines in header names.

···

On Friday, January 26, 2018 at 10:28:13 AM UTC+1, Charlotte Dupont wrote:

Hi,

I am running search guard 20.1 on elasticsearch/kibana 6.1.1

I want to set up an index pattern A for tenant A and an index pattern B for tenant B.

But then i can find index pattern A and B in both tenants. When i set up index pattern A in A, i find it in all other tenant.

What are the configuration issues ?

Here is what I added in my kibana.yml config

searchguard.multitenancy.enabled: true
searchguard.multitenancy.tenants.enable_global: false
searchguard.multitenancy.tenants.enable_private: false

elasticsearch.requestHeadersWhitelist: [“sgtenant”, “Authorization”]

``

4

Thanks, should I post a github issue ?

5

Yes please, that helps us keeping track of this issue.

···

On Thursday, February 1, 2018 at 8:44:09 AM UTC+1, Charlotte Dupont wrote:

Thanks, should I post a github issue ?

6

Hi,
After reporting the issue on Github that have been solved on last release of searchguard, i’m coming back here to get more help.
The conclusion of this issue was that my searchguard config was not correct with the docker image i used.
In fact i forgot to mentionned at the time that i already have spotted the errors in this image, and i’m using my own custom image based on it.
So the error mentionned in the github issued are already fixed, exept one.
When i use the default config of sg_kibana_server (see below) I have an error in Kibana that forbids me the access _==> No cluster-level perm match for User [name=kibana, roles=[kibana_server], requestedTenant=null] [IndexType [index=all, type=*]] [Action [[cluster:monitor/nodes/info]]] [RolesChecked [sg_kibana, sg_kibana_server, sg_public]]

sg_kibana_server:
readonly: true

cluster:

  • CLUSTER_MONITOR

  • CLUSTER_COMPOSITE_OPS

  • cluster:admin/xpack/monitoring*

  • indices:admin/template*

indices:

‘?kibana’:

‘*’:

  • INDICES_ALL

‘?reporting*’:

‘*’:

  • INDICES_ALL

‘?monitoring*’:

‘*’:

  • INDICES_ALL

``

So the only way to make it work that i have found is either to give simply sg_all_access role to user kibana, or a custom sg_kibana_server role like this :

sg_kibana_server:
readonly: true
cluster:
- ALL
- indices:admin/template*
- cluster:monitor/nodes/info
indices:
“":
"”:
- ALL

``

Even if it works like this, i suppose it is not a correct configuration. That’s why maybe i still have the duplication of index patterns in all tenants ?

7

That has to be some kind of misconfiguration since this is a pretty standard use case. So please attach the complete set of configuration files which lead to the exception you posted below:

  • all sg_*.yml files

  • elasticsearch.yml

  • kibana.yml

Just to make sure:

  • Which user/role are you using as the internal kibana server user?

  • Which user are you using to log in?

  • On the multi tenancy page in Kibana, do you see any error messaged?

  • Is there any error or warning in the kibana log/console when you start it?

···

On Tuesday, February 20, 2018 at 12:09:44 PM UTC+1, Charlotte Dupont wrote:

Hi,
After reporting the issue on Github that have been solved on last release of searchguard, i’m coming back here to get more help.
The conclusion of this issue was that my searchguard config was not correct with the docker image i used.
In fact i forgot to mentionned at the time that i already have spotted the errors in this image, and i’m using my own custom image based on it.
So the error mentionned in the github issued are already fixed, exept one.
When i use the default config of sg_kibana_server (see below) I have an error in Kibana that forbids me the access _==> No cluster-level perm match for User [name=kibana, roles=[kibana_server], requestedTenant=null] [IndexType [index=all, type=*]] [Action [[cluster:monitor/nodes/info]]] [RolesChecked [sg_kibana, sg_kibana_server, sg_public]]

sg_kibana_server:
readonly: true

cluster:

  • CLUSTER_MONITOR

  • CLUSTER_COMPOSITE_OPS

  • cluster:admin/xpack/monitoring*

  • indices:admin/template*

indices:

‘?kibana’:

‘*’:

  • INDICES_ALL

‘?reporting*’:

‘*’:

  • INDICES_ALL

‘?monitoring*’:

‘*’:

  • INDICES_ALL

``

So the only way to make it work that i have found is either to give simply sg_all_access role to user kibana, or a custom sg_kibana_server role like this :

sg_kibana_server:
readonly: true
cluster:
- ALL
- indices:admin/template*
- cluster:monitor/nodes/info
indices:
“":
"”:
- ALL

``

Even if it works like this, i suppose it is not a correct configuration. That’s why maybe i still have the duplication of index patterns in all tenants ?

8

Hi, I have attached the requested files.
You can find my answer to your questions below.

Thanks

That has to be some kind of misconfiguration since this is a pretty standard use case. So please attach the complete set of configuration files which lead to the exception you posted below:

  • all sg_*.yml files
  • elasticsearch.yml
  • kibana.yml

Just to make sure:

  • Which user/role are you using as the internal kibana server user?

user : kibana ; user role : sg_kibana_server

  • Which user are you using to log in?

sg_all_access role ( charlotte.dupont user in the sg_internal_users.yml)

  • On the multi tenancy page in Kibana, do you see any error messaged?

I cannot access kibana with the default sg_kibana_server configuration. i have attached a screenshot (kibana_log_in.jpeg file) that shows what happens after logging in on kibana
With the modified sg_kibana_server role, i can access and have no error except the index pattern duplication bug

  • Is there any error or warning in the kibana log/console when you start it?

With the default sg_kibana_server role, there is no error in the kibana logs. There is one in the elasticsearch logs :
elasticsearch_1 | [2018-02-20T14:44:37,990][INFO ][c.f.s.c.PrivilegesEvaluator] No permissions for {sg_public=[IndexType [index=.kibana-6.1, type=]], sg_kibana=[IndexType [index=.kibana-6.1, type=]], sg_kibana_server=[IndexType [index=.kibana-6.1, type=]]}
elasticsearch_1 | [2018-02-20T14:44:40,548][INFO ][c.f.s.c.PrivilegesEvaluator] No index-level perm match for User [name=kibana, roles=[kibana_server], requestedTenant=null] [IndexType [index=.kibana-6.1, type=]] [Action [[indices:admin/mappings/get]]] [RolesChecked [sg_kibana, sg_kibana_server, sg_public]]

** With the modified sg_kibana_server role, there is no error in kibana or elasticsearch logs.**

sg_config.yml (646 Bytes)

sg_internal_users.yml (709 Bytes)

sg_roles.yml (2.94 KB)

sg_roles_mapping.yml (875 Bytes)

elasticsearch.yml (1.21 KB)

kibana.yml (4.76 KB)

sg_action_groups.yml (2.34 KB)

1223×756 73 KB

···

On Tuesday, February 20, 2018 at 3:33:57 PM UTC+1, Jochen Kressin wrote:

9

The config files look good to me, but this esception puzzles me:

With the default sg_kibana_server role, there is no error in the kibana logs. There is one in the elasticsearch logs :
elasticsearch_1 | [2018-02-20T14:44:37,990][INFO ][c.f.s.c.PrivilegesEvaluator] No permissions for {sg_public=[IndexType [index=.kibana-6.1, type=]], sg_kibana=[IndexType [index=.kibana-6.1, type=]], sg_kibana_server=[IndexType [index=.kibana-6.1, type=*]]}
The index that Kibana tries to write to seems to be .kibana-6.1, not .kibana:

index=.kibana-6.1, type=*

In your kibana.yml you configure:

kibana.index: “.kibana”

Do you know there the .kibana-6.1 index comes from? Do you have any index alised configured?

···

On Tuesday, February 20, 2018 at 4:25:38 PM UTC+1, Charlotte Dupont wrote:

Hi, I have attached the requested files.
You can find my answer to your questions below.

Thanks

On Tuesday, February 20, 2018 at 3:33:57 PM UTC+1, Jochen Kressin wrote:

That has to be some kind of misconfiguration since this is a pretty standard use case. So please attach the complete set of configuration files which lead to the exception you posted below:

  • all sg_*.yml files
  • elasticsearch.yml
  • kibana.yml

Just to make sure:

  • Which user/role are you using as the internal kibana server user?

user : kibana ; user role : sg_kibana_server

  • Which user are you using to log in?

sg_all_access role ( charlotte.dupont user in the sg_internal_users.yml)

  • On the multi tenancy page in Kibana, do you see any error messaged?

I cannot access kibana with the default sg_kibana_server configuration. i have attached a screenshot (kibana_log_in.jpeg file) that shows what happens after logging in on kibana
With the modified sg_kibana_server role, i can access and have no error except the index pattern duplication bug

  • Is there any error or warning in the kibana log/console when you start it?

With the default sg_kibana_server role, there is no error in the kibana logs. There is one in the elasticsearch logs :
elasticsearch_1 | [2018-02-20T14:44:37,990][INFO ][c.f.s.c.PrivilegesEvaluator] No permissions for {sg_public=[IndexType [index=.kibana-6.1, type=]], sg_kibana=[IndexType [index=.kibana-6.1, type=]], sg_kibana_server=[IndexType [index=.kibana-6.1, type=]]}
elasticsearch_1 | [2018-02-20T14:44:40,548][INFO ][c.f.s.c.PrivilegesEvaluator] No index-level perm match for User [name=kibana, roles=[kibana_server], requestedTenant=null] [IndexType [index=.kibana-6.1, type=]] [Action [[indices:admin/mappings/get]]] [RolesChecked [sg_kibana, sg_kibana_server, sg_public]]

** With the modified sg_kibana_server role, there is no error in kibana or elasticsearch logs.**

10

Ok, I think I understand now what is going on.

I assume that your real Kibana index name is .kibana-6.1 and you have created an alias named .kibana that points to the .kibana-6.1 index.

In Search Guard, any index alias is always resolved to it’s underlying, real index name:

Which means that when you have an index called index-a and two aliased called alias-b and alias-c that point to that index, you need to grant the user permission for index-a, not for the aliases. This explains why your kibana server user lacks permissions, because the permissions in sg_roles apply for .kibana, not the real index .kibana-6.1

That also explains why multi tenancy is not working. You have set the index name in sg_config.yml to .kibana. Since the real write goes to .kibana-6.1 the multi tenancy module does not detect that this is a write operation to the Kibana index.

You have basically two possibilities:

a) Reindex the “.kibana-6.1” index to an index “.kibana”, and leave all config settings as they are.

b) Leave the index “.kibana-6.1”, but change the Kibana index name in kibana.yml and sg_config to .kibana-6.1. In addition, change the index name .kibana in your sg_roles.yml to .kibana-6.1 as well, or use a wildcard like .kibana*

···

On Wednesday, February 21, 2018 at 12:41:27 PM UTC+1, Jochen Kressin wrote:

The config files look good to me, but this esception puzzles me:

With the default sg_kibana_server role, there is no error in the kibana logs. There is one in the elasticsearch logs :
elasticsearch_1 | [2018-02-20T14:44:37,990][INFO ][c.f.s.c.PrivilegesEvaluator] No permissions for {sg_public=[IndexType [index=.kibana-6.1, type=]], sg_kibana=[IndexType [index=.kibana-6.1, type=]], sg_kibana_server=[IndexType [index=.kibana-6.1, type=*]]}
The index that Kibana tries to write to seems to be .kibana-6.1, not .kibana:

index=.kibana-6.1, type=*

In your kibana.yml you configure:

kibana.index: “.kibana”

Do you know there the .kibana-6.1 index comes from? Do you have any index alised configured?

On Tuesday, February 20, 2018 at 4:25:38 PM UTC+1, Charlotte Dupont wrote:

Hi, I have attached the requested files.
You can find my answer to your questions below.

Thanks

On Tuesday, February 20, 2018 at 3:33:57 PM UTC+1, Jochen Kressin wrote:

That has to be some kind of misconfiguration since this is a pretty standard use case. So please attach the complete set of configuration files which lead to the exception you posted below:

  • all sg_*.yml files
  • elasticsearch.yml
  • kibana.yml

Just to make sure:

  • Which user/role are you using as the internal kibana server user?

user : kibana ; user role : sg_kibana_server

  • Which user are you using to log in?

sg_all_access role ( charlotte.dupont user in the sg_internal_users.yml)

  • On the multi tenancy page in Kibana, do you see any error messaged?

I cannot access kibana with the default sg_kibana_server configuration. i have attached a screenshot (kibana_log_in.jpeg file) that shows what happens after logging in on kibana
With the modified sg_kibana_server role, i can access and have no error except the index pattern duplication bug

  • Is there any error or warning in the kibana log/console when you start it?

With the default sg_kibana_server role, there is no error in the kibana logs. There is one in the elasticsearch logs :
elasticsearch_1 | [2018-02-20T14:44:37,990][INFO ][c.f.s.c.PrivilegesEvaluator] No permissions for {sg_public=[IndexType [index=.kibana-6.1, type=]], sg_kibana=[IndexType [index=.kibana-6.1, type=]], sg_kibana_server=[IndexType [index=.kibana-6.1, type=]]}
elasticsearch_1 | [2018-02-20T14:44:40,548][INFO ][c.f.s.c.PrivilegesEvaluator] No index-level perm match for User [name=kibana, roles=[kibana_server], requestedTenant=null] [IndexType [index=.kibana-6.1, type=]] [Action [[indices:admin/mappings/get]]] [RolesChecked [sg_kibana, sg_kibana_server, sg_public]]

** With the modified sg_kibana_server role, there is no error in kibana or elasticsearch logs.**

11

Thank you for your time and analysis.

You were right, it was related to .kibana-6.1 index.

But both solutions a and b you proposed didn’t work.
The solution a just moved the problem to another index (I have the same error, but mentionning index .kibana instead of .kibana-6.1).
The solution b is not changing anything (same error).

So I found a solution by deleting completely any .kibana index (it’s fine since it’s a new kibana instance)
Now it works, and when i create an index pattern in tenant A, i have a new .kibana-tenantA index created
And the same happens for each tenant, and there is no duplication of data.

I guess it is not a very robust solution, but couldn’t find anything else…

12

That is strange, since proposed solution a) should have put your cluster in the same state as it is now by removing the .kibana index completely. But I guess it’s now tedious to have a closer look at it. Anyways, if you have more questions around this please let me know!

···

On Thursday, February 22, 2018 at 2:54:42 PM UTC+1, Charlotte Dupont wrote:

Thank you for your time and analysis.

You were right, it was related to .kibana-6.1 index.

But both solutions a and b you proposed didn’t work.
The solution a just moved the problem to another index (I have the same error, but mentionning index .kibana instead of .kibana-6.1).
The solution b is not changing anything (same error).

So I found a solution by deleting completely any .kibana index (it’s fine since it’s a new kibana instance)
Now it works, and when i create an index pattern in tenant A, i have a new .kibana-tenantA index created
And the same happens for each tenant, and there is no duplication of data.

I guess it is not a very robust solution, but couldn’t find anything else…