[Proxy Authentication] Issue in getting headers from SSO Session results in Kibana not loading correctly (bootstrap.js unauthorized error)

[d.malacrida]

Elasticsearch version: 7.10.1

Server OS version: CentOS 8

Describe the issue:
We have deployed and ELK instance protected behind a Single Sign On (SSO) system and we are using Search Guard to let the users authenticate directly through the SSO system (thus with proxy authentication enabled).
The issue we are experiencing is that Search Guard does not appear to gather the x-proxy-user from the session header (injected by the SSO system into the session). In fact, when trying to connect to Kibana, it just loads a blank page, giving out an Unauthorized Error on the file bootstrap.js (in the next sections you can find a screenshot of the error given by the browser’s console when this happens).
However, we have also tried to inject the x-proxy-user value manually into the session, by using a browser extension. In this case, not only does Kibana load normally, but it also gathers the right username (specifically, the username passed through the SSO).

Steps to verify:
In order to verify this, we have done the following:

1. We manually inject the x-proxy-user value into the session (for example, x-proxy-user: “user”);
2. We try to connect to Kibana. After authenticating with the SSO system, Kibana starts correctly and, instead of showing the username we injected manually (in this case, “user”), it shows the username given by the SSO (in this case “davide.malacrida”).
3. However, as stated previously, if we do not inject anything manually into the session, Kibana does not appear to be loading at all.

Expected behavior:
We would expect Kibana to load correctly and gather the username by the SSO without having to inject any header manually into the session (just like it seems to be working with the x-forwarded-for session header). As of today, we have not been able to obtain this behavior without manually injecting a default value for the x-proxy-user header into the session.

Provide configuration:
elasticsearch.yml
elasticsearch.yml (4.8 KB)
sg_config.yml
sg_config.yml (10.9 KB)
kibana.yml
kibana.yml (6.3 KB)

Errors in browser console:
Error given when loading Kibana without manually injecting the x-proxy-user value:

Additional data:
We are confident this is not an SSO problem, since we can monitor the header passed by the SSO to the backend applications and these seem to be correct. We have also noticed that the x-forwarded-for header (which is also passed by the SSO system) does appear to be read correctly.
The issue has been encountered on every browser we have tried so far (Opera, Chrome, Firefox, Edge).

[srgbnd]

Hi.
Please enable the debug in Kibana, capture Elasticsearch and Kibana log, and paste it here.
kibana.yml

searchguard.auth.debug: true

Kibana Troubleshooting

I’ll try to reproduce the issue and let you know.

[d.malacrida]

Hi.

Thanks for your immediate reply. I will try to gather the required information as soon as possible.

[d.malacrida]

Hi,

I have been able to isolate the following entries from the kibana.log file, which seems to be the logs originated when experiencing the described issue:

kibana_issue.log (1.4 KB)

Please note that line 3 shows an error which seems to repeat itself every minute or so when Kibana is running. I have read something online about the kibana task manager. I have not looked into it yet but since I am not sure if it is related with our issue in any way, I have included it as well.

As for the elasticsearch logs, we have noticed that since we installed Search Guard, the files where elasticsearch used to save its logs have not been modified. I am not sure if this behavior is normal or not, maybe it just writes the logs in another file. I will look into it and get back to you when I find out more.

Thanks for your help.

[srgbnd]

Hi @d.malacrida
You have the authentication domain order conflict in sg_config.yml. The proxy_auth_domain and jwt_auth_domain have the same order=0. You shouldn’t have the duplicated order values. You may want to apply the following order instead.

proxy_auth_domain
  order: 0
basic_internal_auth_domain
  order: 1
  http_authenticator:
    type: basic
    challenge: false
jwt_auth_domain
  prder: 2
...

You use jwt_auth_domain for the SSO, right? What IDP do you use?

[d.malacrida]

Hi,

I have reviewed the sg_config.yml file, as you suggested. Now we no longer have the duplicated order values (however the issue is still present, as you probably expected).

As for your questions, the SSO would be corresponding to the proxy_auth_domain in the Search Guard configuration file, not to the jwt_auth_domain.

As for the IdP, basically in this scenario that would be the SSO system itself, since it is the one providing authentication for the user. But I’m not sure if I have correctly interpreted your question…

[srgbnd]

Hi. Describe your system connections.
I tried to reproduce your issue with the following setup: User → Proxy (Nginx) → Kibana. And I saw the x-proxy-user propagated to Kibana correctly.

Enable the verbose mode in kibana.yml logging.verbose: true, try to access Kibana and look for the requests and responses. You should see the headers. For example

kibana.example.com_1    | {"type":"response","@timestamp":"2021-01-20T17:49:26Z","tags":[],"pid":217,"method":"post","statusCode":200,"req":{"url":"/api/saved_objects/_bulk_get","method":"post","headers":{
 "x-forwarded-for":"172.16.0.254","x-proxy-user":"user2","x-proxy-roles":"hr,kibanauser","host":"kibana.example.com:5601","connection":"close","content-length":"39","kbn-version":"7.10.1","user-agent":"Mozi...

Do you see any errors in Elasticsearch? Send the Elasticsearch log too.

[d.malacrida]

Hi!

So, basically after working on this issue for a bit, we have solved it by changing some configurations on the load balancer. Now Search Guard is able to read properly the x-proxy-user attribute.

Unfortunately I am not able to articulate more on how the issue was solved because the load balancer is managed by another team. However, this would seem to suggest that the issue was not related with Search Guard in the first place. Probably it has something to do with the way it interacts with our backend systems.

So for me this issue can be marked as solved. Thanks for your help!