I’m on Elasticsearch / Kibana 7.6.1 with X-Pack and SearchGuard v40.0.0.0 TRIAL.
I’m having some issues with Kibana when setting searchguard.auth.type: kerberos and I’m unable to figure it out.
Connecting directly to Elasticsearch via port 9200, authentication over Kerberos is working as expected.
Its also partially working in Kibana. I can authenticate successfully however when clicking on Discover I get an error popup: Error loading data Request to Elasticsearch failed: {"error":{"header":{"WWW-Authenticate":"Negotiate"}}}
Looking at XHR requests in Chrome developer tools, the error comes from a request to: /elasticsearch/<index>/_search? which responds with the JSON above and a 200 HTTP code which could be the culprit as I expected it to return a 401 HTTP code which would cause the browser to negotiate kerberos.
Making that request directly to Elasticsearch completes with no errors as I receive a 401 then negotiate kerberos. It seems the problem occurs when proxied through Kibana.
Switching to Basic Authentication also works because I suspect I’m sending the necessary headers on each request unlike Kerberos.
My sg_config file
authc:
kerberos_auth_domain:
http_enabled: true
transport_enabled: false
order: 6
http_authenticator:
type: kerberos # NOT FREE FOR COMMERCIAL USE
challenge: true
config:
# If true a lot of kerberos/security related debugging output will be logged to standard out
krb_debug: false
# If true then the realm will be stripped from the user name
strip_realm_from_principal: true
authentication_backend:
type: noop
basic_internal_auth_domain:
description: "Authenticate via HTTP Basic against internal users database"
http_enabled: true
transport_enabled: false
order: 4
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: intern
First, let’s do debug. In sg_config set krb_debug: true, apply the config and restart the Elasticsearch. Do you see any related errors in the Elasticsearch log? Post the log if there is any error.
Second, did you disable Kerberos reply cache? Usually, each request to Elasticsearch needs to carry a unique Kerberos ticket. If Kerberos detects a reused ticket, it assumes that the ticket was hijacked and used for a replay attack. Kibana reuses tokens for XHR requests, so at the moment the only way of making Kerberos work with Kibana is to disable the replay cache. On each node, add the following line to jvm.options:
-Dsun.security.krb5.rcache=none
The third test, in sg_config set challenge: false and see what id does
Yes, replay cache is disabled as that prevented me from logging in at all in to Kibana. With it disabled, I can login fine but anything that gets proxied to elasticsearch fails as described in my first post.
challenge: false fails because Kerberos is a challenging authenticator, with it disabled I do not receive a 401 Negotiate from either Kibana or Elasticsearch and I am unable to authenticate.
I’ve come across that link but don’t see anything new to try as Kerberos authentication seems to be working fine, its just Discovery and any browser requests which go to /elasticsearch/ are failing.
Did you set krb_debug: true and applied the sg_config? After you set the option to true, do you see any related errors in the Elasticsearch or Kibana log?
Yep I did set krb_debug to true and I see Kerberos related debug messages when logging logging in to Elasticsearch and Kibana. There are no errors in the debug output or logging files.
When clicking on Discover or anything that needs to proxy my request to Elasticsearch, then I see this error message in my elasticsearch logs:
[WARN ][HTTPBasicAuthenticator] [elk-dev] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'
However, when this 401 response is proxied to Kibana, Kibana returns a 200 HTTP code and not a 401 back to the client (or in this case my browser). This prevents clients from replying with a Kerberos ticket. This is mentioned in the article you sent previously.
The client accesses Elasticsearch and Elasticsearch returns an HTTP status code 401 (Unauthorized) and a WWW-Authenticate: Negotiate HTTP header.
The client is sending the session ticket to Elasticsearch for authentication with the HTTP request.
Unfortunately, this is an issue with Kibana itself and it seems to get worse and worse with newer Kibana releases. Your analysis is correct, the request should be routed back to the browser with 401 and WWW-Authenticate: Negotiate which would then cause the browser to get the ticket (SPNEGO).
Search Guard actually returns a 401 and WWW-Authenticate: Negotiate, which can be tested by accessing ES directly with a browser. Since you write that this works, your configuration seems to be fine, and SG works as expected.
We will look into this issue / Kibana behavior again, but if I am correct this is due to an incorrect Kibana handling of HTTP codes and headers
