multi-tenant not honoring runas user

Search Guard
1

I’ve installed multi-tenant, configured it, and gotten it active. It shows up in Kibana, and I can see the Global and Private tenants. I can even see tenants that I configure for the kibana users.

However, the username displayed in the tenant module, in Kibana is not the user that’s logged in, it’s the DN of the certificate used by the Kibana user. Which means none of role based tenants are showing up - they’re based on the username (or rather, the ldap groups associated with that user), not the Kibana user.

Anyone have any ideas?

It would also be nice if multi-tenant could use certificates, not basic auth. It’s the only place left in our system that uses that.

Details:

Elasticsearch/Kibana 5.4.1

Elasticsearch Searchguard plugins:

dlic-search-guard-authbackend-ldap-5.0-7-jar-with-dependencies.jar

dlic-search-guard-module-dlsfls-5.3-6-jar-with-dependencies.jar

dlic-search-guard-module-kibana-multitenancy-5.4-3-jar-with-dependencies.jar

dlic-search-guard-rest-api-5.3-5-jar-with-dependencies.jar

search-guard-5-5.4.1-12.jar

search-guard-ssl-5.4.1-22.jar

Kibana searchguard plugin 5.4.1-3

Thank you.

···

–

Nobody believed that I could build a space station here. So I built it anyway. It sank into the vortex. So I built another one. It sank into the vortex. The third station burned down, fell over then sank into the vortex. The fourth station just vanished. And the fifth station, THAT stayed!

2

Can you pls. post your sg_config.yml, elasticsearch.yml and kibana.yml?

···

Am 21.07.2017 um 14:10 schrieb Dave Martin <darkmoon@vt.edu>:

I've installed multi-tenant, configured it, and gotten it active. It shows up in Kibana, and I can see the Global and Private tenants. I can even see tenants that I configure for the kibana users.

However, the username displayed in the tenant module, in Kibana is not the user that's logged in, it's the DN of the certificate used by the Kibana user. Which means none of role based tenants are showing up - they're based on the username (or rather, the ldap groups associated with that user), not the Kibana user.

Anyone have any ideas?

It would also be nice if multi-tenant could use certificates, not basic auth. It's the only place left in our system that uses that.

Details:
Elasticsearch/Kibana 5.4.1

Elasticsearch Searchguard plugins:
dlic-search-guard-authbackend-ldap-5.0-7-jar-with-dependencies.jar
dlic-search-guard-module-dlsfls-5.3-6-jar-with-dependencies.jar
dlic-search-guard-module-kibana-multitenancy-5.4-3-jar-with-dependencies.jar
dlic-search-guard-rest-api-5.3-5-jar-with-dependencies.jar
search-guard-5-5.4.1-12.jar
search-guard-ssl-5.4.1-22.jar

Kibana searchguard plugin 5.4.1-3

Thank you.

--
--
Nobody believed that I could build a space station here. So I built it anyway. It sank into the vortex. So I built another one. It sank into the vortex. The third station burned down, fell over then sank into the vortex. The fourth station just vanished. And the fifth station, THAT stayed!

--
You received this message because you are subscribed to the Google Groups "Search Guard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/CAA2%3Dfcqhju9Ga6BPPMsLm3_e0yXe0D%3DHLjm-x0%3Dz6GwNt1sj3g%40mail.gmail.com\.
For more options, visit https://groups.google.com/d/optout\.

3

Sorry, here you go. (slightly cleaned up for clarity)

kibana.yml (1.17 KB)

elasticsearch.yml (2.24 KB)

sg_config.yml (2.02 KB)

···

–

Nobody believed that I could build a space station here. So I built it anyway. It sank into the vortex. So I built another one. It sank into the vortex. The third station burned down, fell over then sank into the vortex. The fourth station just vanished. And the fifth station, THAT stayed!

4

Hi Dave,

I’ve had a look at the config files, I’m not 100% sure I understand the scenario you want to implement correctly.

You’ve enabled two authenticators, proxy and client cert. Since proxy comes first, I assume that you want to authenticate regular users via the proxy headers, and use a client certificate for authenticating Kibana. And ultimately you would like to use client certs only. Roles are fetched from LDAP for regular users. I’m also assuming that Kibana connects to ES directly since you point the elasticsearch.url to https://logview-dev-01.cc.vt.edu.mgt:9200.

Is this correct? If so:

Can you first check that the user that is authenticated by proxy auth is correct, and also mapped to the Search Guard role(s) correctly? You can check it by visiting the authinfo endpoint, which spits out a JSON containing information about the currently logged in user:

https://:9200/_searchguard/authinfo

It would also be helpful to get an ES log on debug level. In conf/log4j2.properties, set:

logger.sg.name = com.floragunn

logger.sg.level = debug

Once enabled, can you navigate to the tenants page, select any tenant, and send the logs?

By the way, these entries here in elasticsearch.yml have to be placed in kibana.yml:

searchguard:

multitenancy:

tenants:

enable_global: true

enable_private: true

prefered: [“Private”, “Global” ]

the last entry spelled “preferred”.

Thanks!

···

On Friday, July 21, 2017 at 2:42:12 PM UTC+2, Dave Martin wrote:

Sorry, here you go. (slightly cleaned up for clarity)

–

–

Nobody believed that I could build a space station here. So I built it anyway. It sank into the vortex. So I built another one. It sank into the vortex. The third station burned down, fell over then sank into the vortex. The fourth station just vanished. And the fifth station, THAT stayed!

5

Hi Dave,

I’ve had a look at the config files, I’m not 100% sure I understand the scenario you want to implement correctly.

Except for the tenants, this is a working config. We use a combination of proxy and ldap lookups for regular users. System users (kibanasever, etc) we have local certificates for.

You’ve enabled two authenticators, proxy and client cert. Since proxy comes first, I assume that you want to authenticate regular users via the proxy headers, and use a client certificate for authenticating Kibana. And ultimately you would like to use client certs only. Roles are fetched from LDAP for regular users. I’m also assuming that Kibana connects to ES directly since you point

the elasticsearch.url to https://logview-dev-01.cc.vt.edu.mgt:9200.

Is this correct? If so:

Can you first check that the user that is authenticated by proxy auth is correct, and also mapped to the Search Guard role(s) correctly? You can check it by visiting the authinfo endpoint, which spits out a JSON containing information about the currently logged in user:

https://:9200/_searchguard/authinfo

Sorry, I don’t think that’s exposed to anywhere that can run a browser. I can do it with a curl from the node itself, but it just gives me the data for the admin cert I’m using, even if it pass it a es-security-runas-user header. That matches the symptoms I’m seeing in Kibana - it’s not looking at the runas user. I am seeing the correct tenants for the server cert (Global and Private).

It would also be helpful to get an ES log on debug level. In conf/log4j2.properties, set:

logger.sg.name = com.floragunn

logger.sg.level = debug

Once enabled, can you navigate to the tenants page, select any tenant, and send the logs?

Sure. Here you go.

By the way, these entries here in elasticsearch.yml have to be placed in kibana.yml:

searchguard:

multitenancy:

tenants:

enable_global: true

enable_private: true

prefered: [“Private”, “Global” ]

the last entry spelled “preferred”.

Ahhh, sorry. I’ve removed it.

Thanks again.

sg_debug.log (612 KB)

···

–

Nobody believed that I could build a space station here. So I built it anyway. It sank into the vortex. So I built another one. It sank into the vortex. The third station burned down, fell over then sank into the vortex. The fourth station just vanished. And the fifth station, THAT stayed!

6

Hi Dave,

the configuration you use should work. We have created a PoC mimicking your config. We used your sg_config and kibana.yml, and ran it with our own certificates without problems. Since you also get the error when using curl, it should not be related to Kibana.

The only way we could reproduce it was:

  1. The headerWhitelist in kibana.yml was not correct, so the proxy header was not passed through correctly

  2. The certificate configured in elasticsearch.ssl.certificate is an admin certificate.

In cas 2) it’s expected behavior, since an admin certificate trumps all other credentials and basically gives you complete access to the cluster.

Can you check that this certificates:

elasticsearch.ssl.certificate: /apps/etc/certs/logview.crt

elasticsearch.ssl.key: /apps/etc/certs/logview.key

Is not configured as admin certificate in elasticsearch.yml?

···

On Friday, July 21, 2017 at 8:22:34 PM UTC+2, Dave Martin wrote:

On Fri, Jul 21, 2017 at 12:25 PM Jochen Kressin jkressin@floragunn.com wrote:

Hi Dave,

I’ve had a look at the config files, I’m not 100% sure I understand the scenario you want to implement correctly.

Except for the tenants, this is a working config. We use a combination of proxy and ldap lookups for regular users. System users (kibanasever, etc) we have local certificates for.

You’ve enabled two authenticators, proxy and client cert. Since proxy comes first, I assume that you want to authenticate regular users via the proxy headers, and use a client certificate for authenticating Kibana. And ultimately you would like to use client certs only. Roles are fetched from LDAP for regular users. I’m also assuming that Kibana connects to ES directly since you point

the elasticsearch.url to https://logview-dev-01.cc.vt.edu.mgt:9200.

Is this correct? If so:

Can you first check that the user that is authenticated by proxy auth is correct, and also mapped to the Search Guard role(s) correctly? You can check it by visiting the authinfo endpoint, which spits out a JSON containing information about the currently logged in user:

https://:9200/_searchguard/authinfo

Sorry, I don’t think that’s exposed to anywhere that can run a browser. I can do it with a curl from the node itself, but it just gives me the data for the admin cert I’m using, even if it pass it a es-security-runas-user header. That matches the symptoms I’m seeing in Kibana - it’s not looking at the runas user. I am seeing the correct tenants for the server cert (Global and Private).

It would also be helpful to get an ES log on debug level. In conf/log4j2.properties, set:

logger.sg.name = com.floragunn

logger.sg.level = debug

Once enabled, can you navigate to the tenants page, select any tenant, and send the logs?

Sure. Here you go.

By the way, these entries here in elasticsearch.yml have to be placed in kibana.yml:

searchguard:

multitenancy:

tenants:

enable_global: true

enable_private: true

prefered: [“Private”, “Global” ]

the last entry spelled “preferred”.

Ahhh, sorry. I’ve removed it.

Thanks again.

–

–

Nobody believed that I could build a space station here. So I built it anyway. It sank into the vortex. So I built another one. It sank into the vortex. The third station burned down, fell over then sank into the vortex. The fourth station just vanished. And the fifth station, THAT stayed!

7

Hi Dave,

the configuration you use should work. We have created a PoC mimicking your config. We used your sg_config and kibana.yml, and ran it with our own certificates without problems. Since you also get the error when using curl, it should not be related to Kibana.

Thank you. That’s an impressive amount of work.

The only way we could reproduce it was:

  1. The headerWhitelist in kibana.yml was not correct, so the proxy header was not passed through correctly
  1. The certificate configured in elasticsearch.ssl.certificate is an admin certificate.

In cas 2) it’s expected behavior, since an admin certificate trumps all other credentials and basically gives you complete access to the cluster.

Can you check that this certificates:

elasticsearch.ssl.certificate: /apps/etc/certs/logview.crt

elasticsearch.ssl.key: /apps/etc/certs/logview.key

Is not configured as admin certificate in elasticsearch.yml?

Admin? No, it’s not. But it is configured as a node certificate:

search-guard config

searchguard.authcz.admin_dn:

  • “C=US,DC=edu,DC=vt,ST=Virginia,L=Blacksburg,O=Virginia Polytechnic Institute and State University,OU=Middleware-Client,OU=Middleware,2.5.4.5=#130d31343735363034353335363232,CN=logmgr”

  • “C=US,DC=edu,DC=vt,ST=Virginia,L=Blacksburg,O=Virginia Polytechnic Institute and State University,OU=Middleware-Client,OU=Division of IT,2.5.4.5=#130d31343636303938353331333237,CN=logstore”

  • “C=US,DC=edu,DC=vt,ST=Virginia,L=Blacksburg,O=Virginia Polytechnic Institute and State University,OU=Middleware-Client,OU=Division of IT,2.5.4.5=#130d31343636343238353033313230,CN=logadmin”

searchguard.nodes_dn:

  • “/DC=edu,DC=vt,DC=it,OU=LAA,CN=logstore-(hot|warm|master)-dev-[0-9][0-9].cc.vt.edu.mgt/”

  • “/DC=edu,DC=vt,DC=it,OU=LAA,CN=logview-dev-[0-9][0-9].cc.vt.edu.mgt/”

  • “/O=Virginia Polytechnic Institute and State University,OU=LAA,CN=logstore-(hot|warm|master)-dev-[0-9][0-9].cc.vt.edu.mgt/”

  • “/O=Virginia Polytechnic Institute and State University,OU=LAA,CN=logview-dev-[0-9][0-9].cc.vt.edu.mgt/”

The kibana uses the cert for that machine, which because it’s also an elasticsearch node, has node privs.

Do we need to split that cert?

Thanks again.

···

–

Nobody believed that I could build a space station here. So I built it anyway. It sank into the vortex. So I built another one. It sank into the vortex. The third station burned down, fell over then sank into the vortex. The fourth station just vanished. And the fifth station, THAT stayed!

8

Hi Dave,

ah ok, that could indeed be the problem. We’ve not tested that scenario, but the intended use of the different node types is:

Node certificates: Used to identify valid Elasticsearch nodes in the cluster

Admin certificate: Grants all access to the cluster, needs to be configure in elasticsearch.yml

Client certificate: Any other valid certificate used to identify users

In your case, the user is the Kibana server. Kibana is not a node in the ES cluster, so you should indeed use a separate certificate for it. Means, simply a cert signed by your root CA, which is not a node or admin certificate.

I guess using a node certificate is the cause of your issue. In our (working) setup, we simply used a regular client cert.

···

On Monday, July 24, 2017 at 8:33:37 PM UTC+2, Dave Martin wrote:

On Mon, Jul 24, 2017 at 2:17 PM Jochen Kressin jkressin@floragunn.com wrote:

Hi Dave,

the configuration you use should work. We have created a PoC mimicking your config. We used your sg_config and kibana.yml, and ran it with our own certificates without problems. Since you also get the error when using curl, it should not be related to Kibana.

Thank you. That’s an impressive amount of work.

The only way we could reproduce it was:

  1. The headerWhitelist in kibana.yml was not correct, so the proxy header was not passed through correctly
  1. The certificate configured in elasticsearch.ssl.certificate is an admin certificate.

In cas 2) it’s expected behavior, since an admin certificate trumps all other credentials and basically gives you complete access to the cluster.

Can you check that this certificates:

elasticsearch.ssl.certificate: /apps/etc/certs/logview.crt

elasticsearch.ssl.key: /apps/etc/certs/logview.key

Is not configured as admin certificate in elasticsearch.yml?

Admin? No, it’s not. But it is configured as a node certificate:

search-guard config

searchguard.authcz.admin_dn:

  • “C=US,DC=edu,DC=vt,ST=Virginia,L=Blacksburg,O=Virginia Polytechnic Institute and State University,OU=Middleware-Client,OU=Middleware,2.5.4.5=#130d31343735363034353335363232,CN=logmgr”
  • “C=US,DC=edu,DC=vt,ST=Virginia,L=Blacksburg,O=Virginia Polytechnic Institute and State University,OU=Middleware-Client,OU=Division of IT,2.5.4.5=#130d31343636303938353331333237,CN=logstore”
  • “C=US,DC=edu,DC=vt,ST=Virginia,L=Blacksburg,O=Virginia Polytechnic Institute and State University,OU=Middleware-Client,OU=Division of IT,2.5.4.5=#130d31343636343238353033313230,CN=logadmin”

searchguard.nodes_dn:

  • “/DC=edu,DC=vt,DC=it,OU=LAA,CN=logstore-(hot|warm|master)-dev-[0-9][0-9].cc.vt.edu.mgt/”
  • “/DC=edu,DC=vt,DC=it,OU=LAA,CN=logview-dev-[0-9][0-9].cc.vt.edu.mgt/”
  • “/O=Virginia Polytechnic Institute and State University,OU=LAA,CN=logstore-(hot|warm|master)-dev-[0-9][0-9].cc.vt.edu.mgt/”
  • “/O=Virginia Polytechnic Institute and State University,OU=LAA,CN=logview-dev-[0-9][0-9].cc.vt.edu.mgt/”

The kibana uses the cert for that machine, which because it’s also an elasticsearch node, has node privs.

Do we need to split that cert?

Thanks again.

–

–

Nobody believed that I could build a space station here. So I built it anyway. It sank into the vortex. So I built another one. It sank into the vortex. The third station burned down, fell over then sank into the vortex. The fourth station just vanished. And the fifth station, THAT stayed!