Kibana OpenID Keycloak too many redirects

[Kosmonafft]

Hi folks,

I have some users who are not able to access kibana because their browser stops: too many redirects.

The only difference between other users who do not expirience such behavior is that they are members of quite a lot of active directory groups (30+), which are returned by keycloak in the payload (groups claim).

Could it be that search guard fails to handle too many values in the groups claim and some how tries to redirect to keycloak again which creates a loop and finaly breaks?

Many thanks for any feedback

[srgbnd]

Hi.
What Search Guard version do you use? Do you see any errors in Elasticsearch log? Please send the Elasticsearch and Kibana logs if you see any error there.

[Kosmonafft]

Thanks for the quick response.

There are no errors in elastic or kibana logs as far as I can see. And as a side note: there is not so much authentication information there in the logs even though i have set:

logger.sg.name = com.floragunn.dlic.auth.http.jwt
logger.sg.level = trace

I can justs see that the user was able to authenticate from the elastic logs

[2021-02-02T16:36:22,331][DEBUG][c.f.s.a.BackendRegistry  ] [elastic01.node.com] Check authdomain for rest internal/1 or 2 in total
[2021-02-02T16:36:22,332][DEBUG][c.f.s.a.BackendRegistry  ] [elastic01.node.com] Rest user 'User [name=kibanaserver, backend_roles=[], requestedTenant=null]' is authenticated
[2021-02-02T16:36:22,332][DEBUG][c.f.s.a.BackendRegistry  ] [elastic01.node.com] sgtenant 'null'
[2021-02-02T16:36:22,332][DEBUG][c.f.s.p.PrivilegesEvaluator] [elastic01.node.com] ### evaluate permissions for User [name=kibanaserver, backend_roles=[], requestedTenant=null] on elastic01.node.com
[2021-02-02T16:36:22,332][DEBUG][c.f.s.p.PrivilegesEvaluator] [elastic01.node.com] action: cluster:monitor/nodes/info (NodesInfoRequest)
[2021-02-02T16:36:22,333][DEBUG][c.f.s.r.IndexResolverReplacer] [elastic01.node.com] Resolve aliases, indices and types from NodesInfoRequest
[2021-02-02T16:36:22,333][DEBUG][c.f.s.p.PrivilegesEvaluator] [elastic01.node.com] requestedResolved : Resolved [aliases=[*], indices=[*], allIndices=[*], types=[*], originalRequested=[], remoteIndices=[]]
[2021-02-02T16:36:22,333][DEBUG][c.f.s.c.PrivilegesInterceptorImpl] [elastic01.node.com] raw requestedTenant: 'null'
[2021-02-02T16:36:22,333][DEBUG][c.f.s.p.PrivilegesEvaluator] [elastic01.node.com] Result from privileges interceptor for cluster perm: null
[2021-02-02T16:36:22,333][DEBUG][c.f.s.p.PrivilegesEvaluator] [elastic01.node.com] Allowed because we have cluster permissions for cluster:monitor/nodes/info
[2021-02-02T16:36:22,334][DEBUG][c.f.s.f.SearchGuardFilter] [elastic01.node.com] PrivEvalResponse [allowed=true, missingPrivileges=[], allowedFlsFields=null, maskedFields=null, queries=null]
[2021-02-02T16:36:22,848][DEBUG][c.f.s.a.BackendRegistry  ] [elastic01.node.com] Check authdomain for rest noop/0 or 2 in total
[2021-02-02T16:36:22,850][DEBUG][c.f.s.a.BackendRegistry  ] [elastic01.node.com] Rest user 'User [name=affected_user, backend_roles=[LONG LIST WITH 39 AD GROUPS], requestedTenant=null]' is authenticated
[2021-02-02T16:36:22,851][DEBUG][c.f.s.a.BackendRegistry  ] [elastic01.node.com] sgtenant 'null'
[2021-02-02T16:36:23,262][DEBUG][c.f.s.a.BackendRegistry  ] [elastic01.node.com] Check authdomain for rest noop/0 or 2 in total
[2021-02-02T16:36:23,264][DEBUG][c.f.s.a.BackendRegistry  ] [elastic01.node.com] 'org.apache.cxf.rs.security.jose.jws.JwsException: INVALID_COMPACT_JWS' extracting credentials from jwt-key-by-oidc http authenticator
org.apache.cxf.rs.security.jose.jws.JwsException: INVALID_COMPACT_JWS

So all in call I can just see:

[2021-02-02T16:36:22,850][DEBUG][c.f.s.a.BackendRegistry ] [elastic01.node.com] Rest user ‘User [name=affected_user, backend_roles=[LONG LIST WITH 39 AD GROUPS], requestedTenant=null]’ is authenticated

This INVALID_COMPACT_JWS stacktrace at the end belongs to the kibanaserver login fail since it uses basic auth I assume

I have also tried to use in kibana.yml:

searchguard.auth.debug: true

which does not work due the search guard version

My versions are: elastic and kibana 7.6.2, search guard 41.0.0

Update:

This is how it looks like in the browser:

[srgbnd]

What are the sizes of Search Guard cookies? Check it with a user that can login. You should have two cookies in your version. There is only one cookie since v7.10. For example

Also, tell me the size (bytes) of the list of 39 AD groups. You can calculate the size with JavaScript

$ node
> Buffer.byteLength("group1,group2", 'utf8');
13

or with bash script in Linux

$ adGroups="group1,group2";printf "%s" "$adGroups" | wc -c
      13

You may exceed the cookie size limit of 4096 bytes.

[srgbnd]

Currently, the Search Guard Kibana plugin can’t store a large amount of user data due to the cookie size limit of 4096 bytes. We started implementing the session-based authentication in the Search Guard backend. It will solve the problem.

[Kosmonafft]

Thank you. We are going to test it this way and report

[Kosmonafft]

Ok so these are the cookies for my user (which can log in without any issues):

these are cookies for someone who cannot login (I cannot see searchguard-authentication cookies tho):

The size of the ad grouop list is:

someone who can login without issues: 500
soemone who cannot login: 1800

[srgbnd]

Hi. Yes, it is because we exceed the cookie size limit here. Thank you for reporting this. As I said, implementing the server-side authentication will solve this. We started working on it. The feature is a big one. It will take some time. I’ll tell you when it is ready. You can follow this issue Session based authentication (#307) · Issues · search-guard / Search Guard Kibana Plugin · GitLab

[faxmodem]

We were running into this issue too, thanks for looking into this !
Will this be backported to 6 ?

[srgbnd]

Yesterday I posted a wrong reply here, I deleted it to avoid confusion. We still work on the issue.

[nils]

To add some background to this:

The cookie size limit is probably reached because the OIDC JWT token got too big. This usually happens because the user has a big amount of roles. The JWT internally stores all these roles.

Also, JWT can contain additional attributes; which attributes are part of the JWT can be usually configured in the IdP server software.

Thus, if you need a quick workaround for the issue, a possible solution would be to look whether it is possible to reduce the number of roles and/or attributes assigned to a user which is facing this problem.

Regarding the planned session feature: This will bring huge changes to the Search Guard code base; thus backporting it to Search Guard 6 would be very complicated and probably not feasible.

We are reviewing other options to reduce the cookie size in Search Guard 6; however, these will only move the limit of the amount of roles and attributes. It will be probably not possible to completely avoid any limit.

[faxmodem]

I can confirm the workaround works. We removed the unnecessary scopes for this client_id in keycloak, and this should suffice until we upgrade to 7.x.