Keycloak Authentication and Searchguard problems

Search Guard
1

I have a problem with Keycloak Authentication and Searchguard.
It all works perfectly: I can login via keycloak accounts in Kibana, Roles are loaded, permissions are granted, etc. It’s beautiful, thank you very very much.

But sometimes in Kibana there is some weird stuff going on, I believe it’s because the autentication of Keycloak runs out? But here is what happens:

I go to the Kibana Dev Tools and run: “GET _cluster/health”, which works.

If I then come back after a couple of minutes and try to the same command (or actually any other command) again, the request just times out and ends up with a gateway timeout after 90 secs (the loadbalancer in front of Kibana has that). The Request was:

POST https://logs-db-ui-lagoon-master.ch.amazee.io/api/console/proxy?path=_cluster%2Fhealth&method=GET

On the Kibana Logs I can see:

If I then refresh the browser (F5) the next requests takes very long (~25 secs) and then the response runs me into another oAuth of Keycloak (redirect to /auth/openid/login?nextUrl=%2Fapp%2Fkibana and so forth). Keycloak does a new authentication and the kibana UI works again.

Another thing happens when instead of running another command, if I click on “Discover” the following requests are happening:

https://logs-db-ui-lagoon-master.ch.amazee.io/api/saved_objects/_find?type=index-pattern&per_page=10000

Status Code: 302 Found: /auth/openid/login?nextUrl=%2Fapi%2Fsaved_objects%2F_find%3Ftype%3Dindex-pattern%26per_page%3D10000

https://logs-db-ui-lagoon-master.ch.amazee.io/auth/openid/login?nextUrl=%2Fapi%2Fsaved_objects%2F_find%3Ftype%3Dindex-pattern%26per_page%3D10000

Status Code: 302 Found: https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=0-dtD4VHXnpcEe5fyAeRGG&scope=profile%20email%20openid

https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=0-dtD4VHXnpcEe5fyAeRGG&scope=profile%20email%20openid

Status Code: 204 No Content

<>

While the regular Authentication when Visiting Kibana fresh is:

https://logs-db-ui-lagoon-master.ch.amazee.io/app/kibana

Status Code: 302 Found: /auth/openid/login?nextUrl=%2Fapp%2Fkibana

https://logs-db-ui-lagoon-master.ch.amazee.io/auth/openid/login?nextUrl=%2Fapp%2Fkibana

Status Code: 302 Found: https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=PcBQ1ZScoeo5HTp0jqcBQg&scope=profile%20email%20openid

https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=PcBQ1ZScoeo5HTp0jqcBQg&scope=profile%20email%20openid

Status Code: 302 Found: Sign in to lagoon

https://logs-db-ui-lagoon-master.ch.amazee.io/auth/openid/login?state=PcBQ1ZScoeo5HTp0jqcBQg&session_state=f4cb01ff-58e3-4af1-adf5-64c82462f595&code=eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0xxxxxx

Status Code: 302 Found: Sign in to lagoon

https://logs-db-ui-lagoon-master.ch.amazee.io/app/kibana

Status Code: 200 OK

<>

Would be awesome to get some pointers where there maybe could be something wrong? Maybe even in Keycloak?

Thanks

  • Search Guard and Elasticsearch version:
    Elasticsearch 6.4.2 Searchguard Plugin 6.4.2-23.1, Kibana: 6.4.2 & Searchguard Plugin 6.4.2-15

  • Installed and used enterprise modules:

  • multitenancy
  • JVM version and operating system version

  • OS[Linux/3.10.0-862.14.4.el7.x86_64/amd64]

  • JVM[“Oracle Corporation”/OpenJDK 64-Bit Server VM/10.0.2/10.0.2+13]

  • Search Guard configuration files

ES: https://github.com/amazeeio/lagoon/blob/master/services/logs-db/sgconfig/sg_config.yml

Kibana: https://github.com/amazeeio/lagoon/blob/master/services/logs-db-ui/Dockerfile

  • Keycloak Config:

According to Kibana Single Sign-On with OpenID and Keycloak | Search Guard

2

Hi Michael,

Thanks for the detailed description!

First of all, the error message you describe in your gist:

I was able to reproduce this and we will have a fix available soon. Thanks for reporting!

However, I believe this is unrelated to the other problem (no changes in the Kibana UI).

As you say, it seems like the session expires. In Keycloak, there’s a setting for the lifespan of the access token: https://www.keycloak.org/docs/3.2/server_admin/topics/sessions/timeouts.html

I am a bit confused by the behaviour you are seeing though. Normally when the session expires, what would happen is this:

  1. When you navigate from e.g. Kibana Home to the Discover tab, the request to get the index pattern should return a 401 Unauthorized if the session cookie is missing or expired.

  2. You will be redirected to Keycloak in the same browser window, where you can enter your credentials

  3. Keycloak creates a new token and redirects and logs you back in in Kibana

I can’t really explain why you in 1) get a 302 Redirect instead of 401, and why on 2) you don’t see any changes in the Kibana UI.

Could you please check if you can see any errors in the console in the developer tools in this case?

You can also test by manually deleting the “searchguard_authentication” cookie in the Application tab in the Chrome developer tools and then navigate from Kibana Home to Discover. This switch doesn’t cause a full page reload, so it should be the same behaviour.

Thanks,

Mike

···

On Friday, 26 October 2018 23:31:01 UTC+2, Michael Schmid wrote:

I have a problem with Keycloak Authentication and Searchguard.
It all works perfectly: I can login via keycloak accounts in Kibana, Roles are loaded, permissions are granted, etc. It’s beautiful, thank you very very much.

But sometimes in Kibana there is some weird stuff going on, I believe it’s because the autentication of Keycloak runs out? But here is what happens:

I go to the Kibana Dev Tools and run: “GET _cluster/health”, which works.

If I then come back after a couple of minutes and try to the same command (or actually any other command) again, the request just times out and ends up with a gateway timeout after 90 secs (the loadbalancer in front of Kibana has that). The Request was:

POST https://logs-db-ui-lagoon-master.ch.amazee.io/api/console/proxy?path=_cluster%2Fhealth&method=GET

On the Kibana Logs I can see:

https://gist.github.com/Schnitzel/02aac92c50e1ee12bcc11aa637358e8d

If I then refresh the browser (F5) the next requests takes very long (~25 secs) and then the response runs me into another oAuth of Keycloak (redirect to /auth/openid/login?nextUrl=%2Fapp%2Fkibana and so forth). Keycloak does a new authentication and the kibana UI works again.

Another thing happens when instead of running another command, if I click on “Discover” the following requests are happening:

https://logs-db-ui-lagoon-master.ch.amazee.io/api/saved_objects/_find?type=index-pattern&per_page=10000

Status Code: 302 Found: /auth/openid/login?nextUrl=%2Fapi%2Fsaved_objects%2F_find%3Ftype%3Dindex-pattern%26per_page%3D10000

https://logs-db-ui-lagoon-master.ch.amazee.io/auth/openid/login?nextUrl=%2Fapi%2Fsaved_objects%2F_find%3Ftype%3Dindex-pattern%26per_page%3D10000

Status Code: 302 Found: https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=0-dtD4VHXnpcEe5fyAeRGG&scope=profile%20email%20openid

https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=0-dtD4VHXnpcEe5fyAeRGG&scope=profile%20email%20openid

Status Code: 204 No Content

<>

While the regular Authentication when Visiting Kibana fresh is:

https://logs-db-ui-lagoon-master.ch.amazee.io/app/kibana

Status Code: 302 Found: /auth/openid/login?nextUrl=%2Fapp%2Fkibana

https://logs-db-ui-lagoon-master.ch.amazee.io/auth/openid/login?nextUrl=%2Fapp%2Fkibana

Status Code: 302 Found: https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=PcBQ1ZScoeo5HTp0jqcBQg&scope=profile%20email%20openid

https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=PcBQ1ZScoeo5HTp0jqcBQg&scope=profile%20email%20openid

Status Code: 302 Found: https://logs-db-ui-lagoon-master.ch.amazee.io/auth/openid/login?state=PcBQ1ZScoeo5HTp0jqcBQg&session_state=f4cb01ff-58e3-4af1-adf5-64c82462f595&code=eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0xxxxxx

https://logs-db-ui-lagoon-master.ch.amazee.io/auth/openid/login?state=PcBQ1ZScoeo5HTp0jqcBQg&session_state=f4cb01ff-58e3-4af1-adf5-64c82462f595&code=eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0xxxxxx

Status Code: 302 Found: https://logs-db-ui-lagoon-master.ch.amazee.io/app/kibana

https://logs-db-ui-lagoon-master.ch.amazee.io/app/kibana

Status Code: 200 OK

<>

Would be awesome to get some pointers where there maybe could be something wrong? Maybe even in Keycloak?

Thanks

  • Search Guard and Elasticsearch version:
    Elasticsearch 6.4.2 Searchguard Plugin 6.4.2-23.1, Kibana: 6.4.2 & Searchguard Plugin 6.4.2-15
  • Installed and used enterprise modules:
  • multitenancy
  • JVM version and operating system version
  • OS[Linux/3.10.0-862.14.4.el7.x86_64/amd64]
  • JVM[“Oracle Corporation”/OpenJDK 64-Bit Server VM/10.0.2/10.0.2+13]
  • Search Guard configuration files

ES: https://github.com/amazeeio/lagoon/blob/master/services/logs-db/sgconfig/sg_config.yml

Kibana: https://github.com/amazeeio/lagoon/blob/master/services/logs-db-ui/Dockerfile

  • Keycloak Config:

According to https://search-guard.com/kibana-openid-keycloak/

3

Hi Mike

Thank you very much for your reply :slight_smile:

First of all, the error message you describe in your gist:
I was able to reproduce this and we will have a fix available soon. Thanks for reporting!

awesome, thanks!

Could you please check if you can see any errors in the console in the developer tools in this case?

Yes! Your Idea with the removal of the cookie allowed me to debug faster and I found the culprit:

Failed to load https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=XwYO1VwxLNS4WnbyRUKJ6x&scope=profile%20email%20openid: Response to preflight request doesn’t pass access control check: No ‘Access-Control-Allow-Origin’ header is present on the requested resource. Origin ‘https://logs-db-ui-lagoon-master.ch.amazee.io’ is therefore not allowed access. If an opaque response serves your needs, set the request’s mode to ‘no-cors’ to fetch the resource with CORS disabled.

This is the authentication flow:

  1. Remove Cookie

  2. Click on any Kibana link

  3. GET https://logs-db-ui-lagoon-master.ch.amazee.io/api/saved_objects/_find?type=index-pattern&fields=title&per_page=10000

Status Code: 302 Found: /auth/openid/login?nextUrl=%2Fapi%2Fsaved_objects%2F_find%3Ftype%3Dindex-pattern%26fields%3Dtitle%26per_page%3D10000

  1. GET Sign in to lagoon

Status Code: 302 Found: https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=IHMVUSHYRgWpz_3kBDFnFE&scope=profile%20email%20openid

  1. OPTIONS (!!) https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=IHMVUSHYRgWpz_3kBDFnFE&scope=profile%20email%20openid

Status Code: 204 No Content

And then the CORS error in the console

So it seems that the Kibana JS first does a CORS preflight check to “auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard” before actually executing the request.

The interesting thing is that this preflight check does not happen when I initially load Kibana, but that’s probably because the kibana itself is actually not loaded yet (aka the request is done by the browser itself and not the kibana javascript).

Is that authentication handled by Searchguard? Maybe we should change it to mode: “no-cors” (Using the Fetch API - Web APIs | MDN) as we expect a redirect anyway and not an actual JSON response from keycloak?

I’m thinking about adding a small nginx container in front of our keycloak that adds the CORS header all the time, but obviously it would be great to have that running with just keycloak and elk/searchguard :slight_smile:

Thank you!

···

On Sunday, October 28, 2018 at 1:53:47 PM UTC-5, Mike wrote:

Hi Michael,

Thanks for the detailed description!

First of all, the error message you describe in your gist:

I was able to reproduce this and we will have a fix available soon. Thanks for reporting!

However, I believe this is unrelated to the other problem (no changes in the Kibana UI).

As you say, it seems like the session expires. In Keycloak, there’s a setting for the lifespan of the access token: https://www.keycloak.org/docs/3.2/server_admin/topics/sessions/timeouts.html

I am a bit confused by the behaviour you are seeing though. Normally when the session expires, what would happen is this:

  1. When you navigate from e.g. Kibana Home to the Discover tab, the request to get the index pattern should return a 401 Unauthorized if the session cookie is missing or expired.
  1. You will be redirected to Keycloak in the same browser window, where you can enter your credentials
  1. Keycloak creates a new token and redirects and logs you back in in Kibana

I can’t really explain why you in 1) get a 302 Redirect instead of 401, and why on 2) you don’t see any changes in the Kibana UI.

Could you please check if you can see any errors in the console in the developer tools in this case?

You can also test by manually deleting the “searchguard_authentication” cookie in the Application tab in the Chrome developer tools and then navigate from Kibana Home to Discover. This switch doesn’t cause a full page reload, so it should be the same behaviour.

Thanks,

Mike

On Friday, 26 October 2018 23:31:01 UTC+2, Michael Schmid wrote:

I have a problem with Keycloak Authentication and Searchguard.
It all works perfectly: I can login via keycloak accounts in Kibana, Roles are loaded, permissions are granted, etc. It’s beautiful, thank you very very much.

But sometimes in Kibana there is some weird stuff going on, I believe it’s because the autentication of Keycloak runs out? But here is what happens:

I go to the Kibana Dev Tools and run: “GET _cluster/health”, which works.

If I then come back after a couple of minutes and try to the same command (or actually any other command) again, the request just times out and ends up with a gateway timeout after 90 secs (the loadbalancer in front of Kibana has that). The Request was:

POST https://logs-db-ui-lagoon-master.ch.amazee.io/api/console/proxy?path=_cluster%2Fhealth&method=GET

On the Kibana Logs I can see:

https://gist.github.com/Schnitzel/02aac92c50e1ee12bcc11aa637358e8d

If I then refresh the browser (F5) the next requests takes very long (~25 secs) and then the response runs me into another oAuth of Keycloak (redirect to /auth/openid/login?nextUrl=%2Fapp%2Fkibana and so forth). Keycloak does a new authentication and the kibana UI works again.

Another thing happens when instead of running another command, if I click on “Discover” the following requests are happening:

https://logs-db-ui-lagoon-master.ch.amazee.io/api/saved_objects/_find?type=index-pattern&per_page=10000

Status Code: 302 Found: /auth/openid/login?nextUrl=%2Fapi%2Fsaved_objects%2F_find%3Ftype%3Dindex-pattern%26per_page%3D10000

https://logs-db-ui-lagoon-master.ch.amazee.io/auth/openid/login?nextUrl=%2Fapi%2Fsaved_objects%2F_find%3Ftype%3Dindex-pattern%26per_page%3D10000

Status Code: 302 Found: https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=0-dtD4VHXnpcEe5fyAeRGG&scope=profile%20email%20openid

https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=0-dtD4VHXnpcEe5fyAeRGG&scope=profile%20email%20openid

Status Code: 204 No Content

<>

While the regular Authentication when Visiting Kibana fresh is:

https://logs-db-ui-lagoon-master.ch.amazee.io/app/kibana

Status Code: 302 Found: /auth/openid/login?nextUrl=%2Fapp%2Fkibana

https://logs-db-ui-lagoon-master.ch.amazee.io/auth/openid/login?nextUrl=%2Fapp%2Fkibana

Status Code: 302 Found: https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=PcBQ1ZScoeo5HTp0jqcBQg&scope=profile%20email%20openid

https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=PcBQ1ZScoeo5HTp0jqcBQg&scope=profile%20email%20openid

Status Code: 302 Found: https://logs-db-ui-lagoon-master.ch.amazee.io/auth/openid/login?state=PcBQ1ZScoeo5HTp0jqcBQg&session_state=f4cb01ff-58e3-4af1-adf5-64c82462f595&code=eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0xxxxxx

https://logs-db-ui-lagoon-master.ch.amazee.io/auth/openid/login?state=PcBQ1ZScoeo5HTp0jqcBQg&session_state=f4cb01ff-58e3-4af1-adf5-64c82462f595&code=eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0xxxxxx

Status Code: 302 Found: https://logs-db-ui-lagoon-master.ch.amazee.io/app/kibana

https://logs-db-ui-lagoon-master.ch.amazee.io/app/kibana

Status Code: 200 OK

<>

Would be awesome to get some pointers where there maybe could be something wrong? Maybe even in Keycloak?

Thanks

  • Search Guard and Elasticsearch version:
    Elasticsearch 6.4.2 Searchguard Plugin 6.4.2-23.1, Kibana: 6.4.2 & Searchguard Plugin 6.4.2-15
  • Installed and used enterprise modules:
  • multitenancy
  • JVM version and operating system version
  • OS[Linux/3.10.0-862.14.4.el7.x86_64/amd64]
  • JVM[“Oracle Corporation”/OpenJDK 64-Bit Server VM/10.0.2/10.0.2+13]
  • Search Guard configuration files

ES: https://github.com/amazeeio/lagoon/blob/master/services/logs-db/sgconfig/sg_config.yml

Kibana: https://github.com/amazeeio/lagoon/blob/master/services/logs-db-ui/Dockerfile

  • Keycloak Config:

According to https://search-guard.com/kibana-openid-keycloak/

4

While the CORS info is interesting, I think we need to look at the third step in your authentication flow.
In your step 3, I would really expect you to receive a status code of 401.

Then, your browser should be redirected to Keycloak, which in turn initiates a POST request back to Kibana.

It seems to me as if this flow happens through XHR-Calls in your case, hence the CORS issue…

What do the requests look like while your cookie is still valid? Do you have a lot of 302 redirects then too?

Just an idea, but maybe your server is redirecting each request for some reason?

I will have to discuss this with the team and get back to you.

Thanks

Mike

···

On Monday, 29 October 2018 15:09:37 UTC+1, Michael Schmid wrote:

Hi Mike

Thank you very much for your reply :slight_smile:

First of all, the error message you describe in your gist:
I was able to reproduce this and we will have a fix available soon. Thanks for reporting!

awesome, thanks!

Could you please check if you can see any errors in the console in the developer tools in this case?

Yes! Your Idea with the removal of the cookie allowed me to debug faster and I found the culprit:

Failed to load https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=XwYO1VwxLNS4WnbyRUKJ6x&scope=profile%20email%20openid: Response to preflight request doesn’t pass access control check: No ‘Access-Control-Allow-Origin’ header is present on the requested resource. Origin ‘https://logs-db-ui-lagoon-master.ch.amazee.io’ is therefore not allowed access. If an opaque response serves your needs, set the request’s mode to ‘no-cors’ to fetch the resource with CORS disabled.

This is the authentication flow:

  1. Remove Cookie
  1. Click on any Kibana link
  1. GET https://logs-db-ui-lagoon-master.ch.amazee.io/api/saved_objects/_find?type=index-pattern&fields=title&per_page=10000

Status Code: 302 Found: /auth/openid/login?nextUrl=%2Fapi%2Fsaved_objects%2F_find%3Ftype%3Dindex-pattern%26fields%3Dtitle%26per_page%3D10000

  1. GET https://logs-db-ui-lagoon-master.ch.amazee.io/auth/openid/login?nextUrl=%2Fapi%2Fsaved_objects%2F_find%3Ftype%3Dindex-pattern%26fields%3Dtitle%26per_page%3D10000

Status Code: 302 Found: https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=IHMVUSHYRgWpz_3kBDFnFE&scope=profile%20email%20openid

  1. OPTIONS (!!) https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=IHMVUSHYRgWpz_3kBDFnFE&scope=profile%20email%20openid

Status Code: 204 No Content

And then the CORS error in the console

So it seems that the Kibana JS first does a CORS preflight check to “auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard” before actually executing the request.

The interesting thing is that this preflight check does not happen when I initially load Kibana, but that’s probably because the kibana itself is actually not loaded yet (aka the request is done by the browser itself and not the kibana javascript).

Is that authentication handled by Searchguard? Maybe we should change it to mode: “no-cors” (https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch) as we expect a redirect anyway and not an actual JSON response from keycloak?

I’m thinking about adding a small nginx container in front of our keycloak that adds the CORS header all the time, but obviously it would be great to have that running with just keycloak and elk/searchguard :slight_smile:

Thank you!

On Sunday, October 28, 2018 at 1:53:47 PM UTC-5, Mike wrote:

Hi Michael,

Thanks for the detailed description!

First of all, the error message you describe in your gist:

I was able to reproduce this and we will have a fix available soon. Thanks for reporting!

However, I believe this is unrelated to the other problem (no changes in the Kibana UI).

As you say, it seems like the session expires. In Keycloak, there’s a setting for the lifespan of the access token: https://www.keycloak.org/docs/3.2/server_admin/topics/sessions/timeouts.html

I am a bit confused by the behaviour you are seeing though. Normally when the session expires, what would happen is this:

  1. When you navigate from e.g. Kibana Home to the Discover tab, the request to get the index pattern should return a 401 Unauthorized if the session cookie is missing or expired.
  1. You will be redirected to Keycloak in the same browser window, where you can enter your credentials
  1. Keycloak creates a new token and redirects and logs you back in in Kibana

I can’t really explain why you in 1) get a 302 Redirect instead of 401, and why on 2) you don’t see any changes in the Kibana UI.

Could you please check if you can see any errors in the console in the developer tools in this case?

You can also test by manually deleting the “searchguard_authentication” cookie in the Application tab in the Chrome developer tools and then navigate from Kibana Home to Discover. This switch doesn’t cause a full page reload, so it should be the same behaviour.

Thanks,

Mike

On Friday, 26 October 2018 23:31:01 UTC+2, Michael Schmid wrote:

I have a problem with Keycloak Authentication and Searchguard.
It all works perfectly: I can login via keycloak accounts in Kibana, Roles are loaded, permissions are granted, etc. It’s beautiful, thank you very very much.

But sometimes in Kibana there is some weird stuff going on, I believe it’s because the autentication of Keycloak runs out? But here is what happens:

I go to the Kibana Dev Tools and run: “GET _cluster/health”, which works.

If I then come back after a couple of minutes and try to the same command (or actually any other command) again, the request just times out and ends up with a gateway timeout after 90 secs (the loadbalancer in front of Kibana has that). The Request was:

POST https://logs-db-ui-lagoon-master.ch.amazee.io/api/console/proxy?path=_cluster%2Fhealth&method=GET

On the Kibana Logs I can see:

https://gist.github.com/Schnitzel/02aac92c50e1ee12bcc11aa637358e8d

If I then refresh the browser (F5) the next requests takes very long (~25 secs) and then the response runs me into another oAuth of Keycloak (redirect to /auth/openid/login?nextUrl=%2Fapp%2Fkibana and so forth). Keycloak does a new authentication and the kibana UI works again.

Another thing happens when instead of running another command, if I click on “Discover” the following requests are happening:

https://logs-db-ui-lagoon-master.ch.amazee.io/api/saved_objects/_find?type=index-pattern&per_page=10000

Status Code: 302 Found: /auth/openid/login?nextUrl=%2Fapi%2Fsaved_objects%2F_find%3Ftype%3Dindex-pattern%26per_page%3D10000

https://logs-db-ui-lagoon-master.ch.amazee.io/auth/openid/login?nextUrl=%2Fapi%2Fsaved_objects%2F_find%3Ftype%3Dindex-pattern%26per_page%3D10000

Status Code: 302 Found: https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=0-dtD4VHXnpcEe5fyAeRGG&scope=profile%20email%20openid

https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=0-dtD4VHXnpcEe5fyAeRGG&scope=profile%20email%20openid

Status Code: 204 No Content

<>

While the regular Authentication when Visiting Kibana fresh is:

https://logs-db-ui-lagoon-master.ch.amazee.io/app/kibana

Status Code: 302 Found: /auth/openid/login?nextUrl=%2Fapp%2Fkibana

https://logs-db-ui-lagoon-master.ch.amazee.io/auth/openid/login?nextUrl=%2Fapp%2Fkibana

Status Code: 302 Found: https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=PcBQ1ZScoeo5HTp0jqcBQg&scope=profile%20email%20openid

https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=PcBQ1ZScoeo5HTp0jqcBQg&scope=profile%20email%20openid

Status Code: 302 Found: https://logs-db-ui-lagoon-master.ch.amazee.io/auth/openid/login?state=PcBQ1ZScoeo5HTp0jqcBQg&session_state=f4cb01ff-58e3-4af1-adf5-64c82462f595&code=eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0xxxxxx

https://logs-db-ui-lagoon-master.ch.amazee.io/auth/openid/login?state=PcBQ1ZScoeo5HTp0jqcBQg&session_state=f4cb01ff-58e3-4af1-adf5-64c82462f595&code=eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0xxxxxx

Status Code: 302 Found: https://logs-db-ui-lagoon-master.ch.amazee.io/app/kibana

https://logs-db-ui-lagoon-master.ch.amazee.io/app/kibana

Status Code: 200 OK

<>

Would be awesome to get some pointers where there maybe could be something wrong? Maybe even in Keycloak?

Thanks

  • Search Guard and Elasticsearch version:
    Elasticsearch 6.4.2 Searchguard Plugin 6.4.2-23.1, Kibana: 6.4.2 & Searchguard Plugin 6.4.2-15
  • Installed and used enterprise modules:
  • multitenancy
  • JVM version and operating system version
  • OS[Linux/3.10.0-862.14.4.el7.x86_64/amd64]
  • JVM[“Oracle Corporation”/OpenJDK 64-Bit Server VM/10.0.2/10.0.2+13]
  • Search Guard configuration files

ES: https://github.com/amazeeio/lagoon/blob/master/services/logs-db/sgconfig/sg_config.yml

Kibana: https://github.com/amazeeio/lagoon/blob/master/services/logs-db-ui/Dockerfile

  • Keycloak Config:

According to https://search-guard.com/kibana-openid-keycloak/

5

just as a short idea: If you like I can create you an account in our kibana/keycloak so you can test yourself? Just shoot me an email at michael@amazee.io

···

On Tuesday, October 30, 2018 at 11:28:01 AM UTC-5, Mike wrote:

While the CORS info is interesting, I think we need to look at the third step in your authentication flow.
In your step 3, I would really expect you to receive a status code of 401.

Then, your browser should be redirected to Keycloak, which in turn initiates a POST request back to Kibana.

It seems to me as if this flow happens through XHR-Calls in your case, hence the CORS issue…

What do the requests look like while your cookie is still valid? Do you have a lot of 302 redirects then too?

Just an idea, but maybe your server is redirecting each request for some reason?

I will have to discuss this with the team and get back to you.

Thanks

Mike

On Monday, 29 October 2018 15:09:37 UTC+1, Michael Schmid wrote:

Hi Mike

Thank you very much for your reply :slight_smile:

First of all, the error message you describe in your gist:
I was able to reproduce this and we will have a fix available soon. Thanks for reporting!

awesome, thanks!

Could you please check if you can see any errors in the console in the developer tools in this case?

Yes! Your Idea with the removal of the cookie allowed me to debug faster and I found the culprit:

Failed to load https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=XwYO1VwxLNS4WnbyRUKJ6x&scope=profile%20email%20openid: Response to preflight request doesn’t pass access control check: No ‘Access-Control-Allow-Origin’ header is present on the requested resource. Origin ‘https://logs-db-ui-lagoon-master.ch.amazee.io’ is therefore not allowed access. If an opaque response serves your needs, set the request’s mode to ‘no-cors’ to fetch the resource with CORS disabled.

This is the authentication flow:

  1. Remove Cookie
  1. Click on any Kibana link
  1. GET https://logs-db-ui-lagoon-master.ch.amazee.io/api/saved_objects/_find?type=index-pattern&fields=title&per_page=10000

Status Code: 302 Found: /auth/openid/login?nextUrl=%2Fapi%2Fsaved_objects%2F_find%3Ftype%3Dindex-pattern%26fields%3Dtitle%26per_page%3D10000

  1. GET https://logs-db-ui-lagoon-master.ch.amazee.io/auth/openid/login?nextUrl=%2Fapi%2Fsaved_objects%2F_find%3Ftype%3Dindex-pattern%26fields%3Dtitle%26per_page%3D10000

Status Code: 302 Found: https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=IHMVUSHYRgWpz_3kBDFnFE&scope=profile%20email%20openid

  1. OPTIONS (!!) https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=IHMVUSHYRgWpz_3kBDFnFE&scope=profile%20email%20openid

Status Code: 204 No Content

And then the CORS error in the console

So it seems that the Kibana JS first does a CORS preflight check to “auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard” before actually executing the request.

The interesting thing is that this preflight check does not happen when I initially load Kibana, but that’s probably because the kibana itself is actually not loaded yet (aka the request is done by the browser itself and not the kibana javascript).

Is that authentication handled by Searchguard? Maybe we should change it to mode: “no-cors” (https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch) as we expect a redirect anyway and not an actual JSON response from keycloak?

I’m thinking about adding a small nginx container in front of our keycloak that adds the CORS header all the time, but obviously it would be great to have that running with just keycloak and elk/searchguard :slight_smile:

Thank you!

On Sunday, October 28, 2018 at 1:53:47 PM UTC-5, Mike wrote:

Hi Michael,

Thanks for the detailed description!

First of all, the error message you describe in your gist:

I was able to reproduce this and we will have a fix available soon. Thanks for reporting!

However, I believe this is unrelated to the other problem (no changes in the Kibana UI).

As you say, it seems like the session expires. In Keycloak, there’s a setting for the lifespan of the access token: https://www.keycloak.org/docs/3.2/server_admin/topics/sessions/timeouts.html

I am a bit confused by the behaviour you are seeing though. Normally when the session expires, what would happen is this:

  1. When you navigate from e.g. Kibana Home to the Discover tab, the request to get the index pattern should return a 401 Unauthorized if the session cookie is missing or expired.
  1. You will be redirected to Keycloak in the same browser window, where you can enter your credentials
  1. Keycloak creates a new token and redirects and logs you back in in Kibana

I can’t really explain why you in 1) get a 302 Redirect instead of 401, and why on 2) you don’t see any changes in the Kibana UI.

Could you please check if you can see any errors in the console in the developer tools in this case?

You can also test by manually deleting the “searchguard_authentication” cookie in the Application tab in the Chrome developer tools and then navigate from Kibana Home to Discover. This switch doesn’t cause a full page reload, so it should be the same behaviour.

Thanks,

Mike

On Friday, 26 October 2018 23:31:01 UTC+2, Michael Schmid wrote:

I have a problem with Keycloak Authentication and Searchguard.
It all works perfectly: I can login via keycloak accounts in Kibana, Roles are loaded, permissions are granted, etc. It’s beautiful, thank you very very much.

But sometimes in Kibana there is some weird stuff going on, I believe it’s because the autentication of Keycloak runs out? But here is what happens:

I go to the Kibana Dev Tools and run: “GET _cluster/health”, which works.

If I then come back after a couple of minutes and try to the same command (or actually any other command) again, the request just times out and ends up with a gateway timeout after 90 secs (the loadbalancer in front of Kibana has that). The Request was:

POST https://logs-db-ui-lagoon-master.ch.amazee.io/api/console/proxy?path=_cluster%2Fhealth&method=GET

On the Kibana Logs I can see:

https://gist.github.com/Schnitzel/02aac92c50e1ee12bcc11aa637358e8d

If I then refresh the browser (F5) the next requests takes very long (~25 secs) and then the response runs me into another oAuth of Keycloak (redirect to /auth/openid/login?nextUrl=%2Fapp%2Fkibana and so forth). Keycloak does a new authentication and the kibana UI works again.

Another thing happens when instead of running another command, if I click on “Discover” the following requests are happening:

https://logs-db-ui-lagoon-master.ch.amazee.io/api/saved_objects/_find?type=index-pattern&per_page=10000

Status Code: 302 Found: /auth/openid/login?nextUrl=%2Fapi%2Fsaved_objects%2F_find%3Ftype%3Dindex-pattern%26per_page%3D10000

https://logs-db-ui-lagoon-master.ch.amazee.io/auth/openid/login?nextUrl=%2Fapi%2Fsaved_objects%2F_find%3Ftype%3Dindex-pattern%26per_page%3D10000

Status Code: 302 Found: https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=0-dtD4VHXnpcEe5fyAeRGG&scope=profile%20email%20openid

https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=0-dtD4VHXnpcEe5fyAeRGG&scope=profile%20email%20openid

Status Code: 204 No Content

<>

While the regular Authentication when Visiting Kibana fresh is:

https://logs-db-ui-lagoon-master.ch.amazee.io/app/kibana

Status Code: 302 Found: /auth/openid/login?nextUrl=%2Fapp%2Fkibana

https://logs-db-ui-lagoon-master.ch.amazee.io/auth/openid/login?nextUrl=%2Fapp%2Fkibana

Status Code: 302 Found: https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=PcBQ1ZScoeo5HTp0jqcBQg&scope=profile%20email%20openid

https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=PcBQ1ZScoeo5HTp0jqcBQg&scope=profile%20email%20openid

Status Code: 302 Found: https://logs-db-ui-lagoon-master.ch.amazee.io/auth/openid/login?state=PcBQ1ZScoeo5HTp0jqcBQg&session_state=f4cb01ff-58e3-4af1-adf5-64c82462f595&code=eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0xxxxxx

https://logs-db-ui-lagoon-master.ch.amazee.io/auth/openid/login?state=PcBQ1ZScoeo5HTp0jqcBQg&session_state=f4cb01ff-58e3-4af1-adf5-64c82462f595&code=eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0xxxxxx

Status Code: 302 Found: https://logs-db-ui-lagoon-master.ch.amazee.io/app/kibana

https://logs-db-ui-lagoon-master.ch.amazee.io/app/kibana

Status Code: 200 OK

<>

Would be awesome to get some pointers where there maybe could be something wrong? Maybe even in Keycloak?

Thanks

  • Search Guard and Elasticsearch version:
    Elasticsearch 6.4.2 Searchguard Plugin 6.4.2-23.1, Kibana: 6.4.2 & Searchguard Plugin 6.4.2-15
  • Installed and used enterprise modules:
  • multitenancy
  • JVM version and operating system version
  • OS[Linux/3.10.0-862.14.4.el7.x86_64/amd64]
  • JVM[“Oracle Corporation”/OpenJDK 64-Bit Server VM/10.0.2/10.0.2+13]
  • Search Guard configuration files

ES: https://github.com/amazeeio/lagoon/blob/master/services/logs-db/sgconfig/sg_config.yml

Kibana: https://github.com/amazeeio/lagoon/blob/master/services/logs-db-ui/Dockerfile

  • Keycloak Config:

According to https://search-guard.com/kibana-openid-keycloak/

6

Just in case anybody else faces this issue - it seems like this behaviour was caused by a missing “application/json” value in the request’s “accept” header.
The latest release of the Search Guard Kibana plugin handles this by also taking the “Content-Type” header into account.

···

On Tuesday, 30 October 2018 18:46:48 UTC+1, Michael Schmid wrote:

just as a short idea: If you like I can create you an account in our kibana/keycloak so you can test yourself? Just shoot me an email at mic...@amazee.io

On Tuesday, October 30, 2018 at 11:28:01 AM UTC-5, Mike wrote:

While the CORS info is interesting, I think we need to look at the third step in your authentication flow.
In your step 3, I would really expect you to receive a status code of 401.

Then, your browser should be redirected to Keycloak, which in turn initiates a POST request back to Kibana.

It seems to me as if this flow happens through XHR-Calls in your case, hence the CORS issue…

What do the requests look like while your cookie is still valid? Do you have a lot of 302 redirects then too?

Just an idea, but maybe your server is redirecting each request for some reason?

I will have to discuss this with the team and get back to you.

Thanks

Mike

On Monday, 29 October 2018 15:09:37 UTC+1, Michael Schmid wrote:

Hi Mike

Thank you very much for your reply :slight_smile:

First of all, the error message you describe in your gist:
I was able to reproduce this and we will have a fix available soon. Thanks for reporting!

awesome, thanks!

Could you please check if you can see any errors in the console in the developer tools in this case?

Yes! Your Idea with the removal of the cookie allowed me to debug faster and I found the culprit:

Failed to load https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=XwYO1VwxLNS4WnbyRUKJ6x&scope=profile%20email%20openid: Response to preflight request doesn’t pass access control check: No ‘Access-Control-Allow-Origin’ header is present on the requested resource. Origin ‘https://logs-db-ui-lagoon-master.ch.amazee.io’ is therefore not allowed access. If an opaque response serves your needs, set the request’s mode to ‘no-cors’ to fetch the resource with CORS disabled.

This is the authentication flow:

  1. Remove Cookie
  1. Click on any Kibana link
  1. GET https://logs-db-ui-lagoon-master.ch.amazee.io/api/saved_objects/_find?type=index-pattern&fields=title&per_page=10000

Status Code: 302 Found: /auth/openid/login?nextUrl=%2Fapi%2Fsaved_objects%2F_find%3Ftype%3Dindex-pattern%26fields%3Dtitle%26per_page%3D10000

  1. GET https://logs-db-ui-lagoon-master.ch.amazee.io/auth/openid/login?nextUrl=%2Fapi%2Fsaved_objects%2F_find%3Ftype%3Dindex-pattern%26fields%3Dtitle%26per_page%3D10000

Status Code: 302 Found: https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=IHMVUSHYRgWpz_3kBDFnFE&scope=profile%20email%20openid

  1. OPTIONS (!!) https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=IHMVUSHYRgWpz_3kBDFnFE&scope=profile%20email%20openid

Status Code: 204 No Content

And then the CORS error in the console

So it seems that the Kibana JS first does a CORS preflight check to “auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard” before actually executing the request.

The interesting thing is that this preflight check does not happen when I initially load Kibana, but that’s probably because the kibana itself is actually not loaded yet (aka the request is done by the browser itself and not the kibana javascript).

Is that authentication handled by Searchguard? Maybe we should change it to mode: “no-cors” (https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch) as we expect a redirect anyway and not an actual JSON response from keycloak?

I’m thinking about adding a small nginx container in front of our keycloak that adds the CORS header all the time, but obviously it would be great to have that running with just keycloak and elk/searchguard :slight_smile:

Thank you!

On Sunday, October 28, 2018 at 1:53:47 PM UTC-5, Mike wrote:

Hi Michael,

Thanks for the detailed description!

First of all, the error message you describe in your gist:

I was able to reproduce this and we will have a fix available soon. Thanks for reporting!

However, I believe this is unrelated to the other problem (no changes in the Kibana UI).

As you say, it seems like the session expires. In Keycloak, there’s a setting for the lifespan of the access token: https://www.keycloak.org/docs/3.2/server_admin/topics/sessions/timeouts.html

I am a bit confused by the behaviour you are seeing though. Normally when the session expires, what would happen is this:

  1. When you navigate from e.g. Kibana Home to the Discover tab, the request to get the index pattern should return a 401 Unauthorized if the session cookie is missing or expired.
  1. You will be redirected to Keycloak in the same browser window, where you can enter your credentials
  1. Keycloak creates a new token and redirects and logs you back in in Kibana

I can’t really explain why you in 1) get a 302 Redirect instead of 401, and why on 2) you don’t see any changes in the Kibana UI.

Could you please check if you can see any errors in the console in the developer tools in this case?

You can also test by manually deleting the “searchguard_authentication” cookie in the Application tab in the Chrome developer tools and then navigate from Kibana Home to Discover. This switch doesn’t cause a full page reload, so it should be the same behaviour.

Thanks,

Mike

On Friday, 26 October 2018 23:31:01 UTC+2, Michael Schmid wrote:

I have a problem with Keycloak Authentication and Searchguard.
It all works perfectly: I can login via keycloak accounts in Kibana, Roles are loaded, permissions are granted, etc. It’s beautiful, thank you very very much.

But sometimes in Kibana there is some weird stuff going on, I believe it’s because the autentication of Keycloak runs out? But here is what happens:

I go to the Kibana Dev Tools and run: “GET _cluster/health”, which works.

If I then come back after a couple of minutes and try to the same command (or actually any other command) again, the request just times out and ends up with a gateway timeout after 90 secs (the loadbalancer in front of Kibana has that). The Request was:

POST https://logs-db-ui-lagoon-master.ch.amazee.io/api/console/proxy?path=_cluster%2Fhealth&method=GET

On the Kibana Logs I can see:

https://gist.github.com/Schnitzel/02aac92c50e1ee12bcc11aa637358e8d

If I then refresh the browser (F5) the next requests takes very long (~25 secs) and then the response runs me into another oAuth of Keycloak (redirect to /auth/openid/login?nextUrl=%2Fapp%2Fkibana and so forth). Keycloak does a new authentication and the kibana UI works again.

Another thing happens when instead of running another command, if I click on “Discover” the following requests are happening:

https://logs-db-ui-lagoon-master.ch.amazee.io/api/saved_objects/_find?type=index-pattern&per_page=10000

Status Code: 302 Found: /auth/openid/login?nextUrl=%2Fapi%2Fsaved_objects%2F_find%3Ftype%3Dindex-pattern%26per_page%3D10000

https://logs-db-ui-lagoon-master.ch.amazee.io/auth/openid/login?nextUrl=%2Fapi%2Fsaved_objects%2F_find%3Ftype%3Dindex-pattern%26per_page%3D10000

Status Code: 302 Found: https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=0-dtD4VHXnpcEe5fyAeRGG&scope=profile%20email%20openid

https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=0-dtD4VHXnpcEe5fyAeRGG&scope=profile%20email%20openid

Status Code: 204 No Content

<>

While the regular Authentication when Visiting Kibana fresh is:

https://logs-db-ui-lagoon-master.ch.amazee.io/app/kibana

Status Code: 302 Found: /auth/openid/login?nextUrl=%2Fapp%2Fkibana

https://logs-db-ui-lagoon-master.ch.amazee.io/auth/openid/login?nextUrl=%2Fapp%2Fkibana

Status Code: 302 Found: https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=PcBQ1ZScoeo5HTp0jqcBQg&scope=profile%20email%20openid

https://keycloak-lagoon-master.ch.amazee.io/auth/realms/lagoon/protocol/openid-connect/auth?client_id=searchguard&response_type=code&redirect_uri=https%3A%2F%2Flogs-db-ui-lagoon-master.ch.amazee.io%2Fauth%2Fopenid%2Flogin&state=PcBQ1ZScoeo5HTp0jqcBQg&scope=profile%20email%20openid

Status Code: 302 Found: https://logs-db-ui-lagoon-master.ch.amazee.io/auth/openid/login?state=PcBQ1ZScoeo5HTp0jqcBQg&session_state=f4cb01ff-58e3-4af1-adf5-64c82462f595&code=eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0xxxxxx

https://logs-db-ui-lagoon-master.ch.amazee.io/auth/openid/login?state=PcBQ1ZScoeo5HTp0jqcBQg&session_state=f4cb01ff-58e3-4af1-adf5-64c82462f595&code=eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0xxxxxx

Status Code: 302 Found: https://logs-db-ui-lagoon-master.ch.amazee.io/app/kibana

https://logs-db-ui-lagoon-master.ch.amazee.io/app/kibana

Status Code: 200 OK

<>

Would be awesome to get some pointers where there maybe could be something wrong? Maybe even in Keycloak?

Thanks

  • Search Guard and Elasticsearch version:
    Elasticsearch 6.4.2 Searchguard Plugin 6.4.2-23.1, Kibana: 6.4.2 & Searchguard Plugin 6.4.2-15
  • Installed and used enterprise modules:
  • multitenancy
  • JVM version and operating system version
  • OS[Linux/3.10.0-862.14.4.el7.x86_64/amd64]
  • JVM[“Oracle Corporation”/OpenJDK 64-Bit Server VM/10.0.2/10.0.2+13]
  • Search Guard configuration files

ES: https://github.com/amazeeio/lagoon/blob/master/services/logs-db/sgconfig/sg_config.yml

Kibana: https://github.com/amazeeio/lagoon/blob/master/services/logs-db-ui/Dockerfile

  • Keycloak Config:

According to https://search-guard.com/kibana-openid-keycloak/