Obtaining Keycloak HTTPS IdP endpoint using OpenID

I’m trying to integrate Keycloak with Search Guard using OpenID, but keep getting errors that the IdP configuration endpoint is not accessible.

I’m able to access the endpoint using this curl command:

curl -k https://keycloak/auth/realms/realm/.well-known/openid-configuration --cacert /path/to/cacert.pem --cert /path/to/cert.pem

I’m try to translate that to configuration options in the config files, but can’t seem to get it to work.

Below are relevant section of my config files:

sg_config.yml

searchguard:
  dynamic:
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
        internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
        remoteIpHeader:  'x-forwarded-for'
        proxiesHeader:   'x-forwarded-by'
    authc:
      basic_internal_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: internal
      openid_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: openid
          challenge: false
          config:
            enable_ssl: true
            verify_hostnames: false
            enable_ssl_client_auth: true
            pemtrustedcas_filepath: /path/to/cacert.pem
            pemkey_filepath: /path/to/key.pem
            pemcert_filepath: /path/to/cert.pem
            openid_connect_url: https://keycloak/auth/realms/realm/.well-known/openid-configuration
            subject_key: preferred_username
            roles_key: roles
        authentication_backend:
          type: noop

kibana.yml

xpack.spaces.enabled: false
xpack.security.enabled: false

searchguard.auth.type: "openid"
searchguard.openid.connect_url: "https://keycloak/auth/realms/realm/.well-known/openid-configuration"
searchguard.openid.client_id: "kibana"
searchguard.openid.client_secret: "xxx"
searchguard.openid.base_redirect_url: "http://localhost:5601"
searchguard.openid.root_ca: "/path/to/cacert.pem"
searchguard.openid.verify_hostnames: false

Below are the logs. I can’t get any more detailed information by changing the logging level.

kibana_1         | {"type":"log","@timestamp":"2020-04-27T14:55:04Z","tags":["status","plugin:searchguard@6.8.6-19.0","info"],"pid":1,"state":"yellow","message":"Status changed from yellow to yellow - 'searchguard.cookie.secure' is set to false, cookies are transmitted over unsecure HTTP connection. Consider using HTTPS and set this key to 'true'","prevState":"yellow","prevMsg":"Default cookie password detected, please set a password in kibana.yml by setting 'searchguard.cookie.password' (min. 32 characters)."}
    kibana_1         | {"type":"log","@timestamp":"2020-04-27T14:55:04Z","tags":["error","searchguard"],"pid":1,"message":"An error occurred while enabling session management: Error: Failed when trying to obtain the endpoints from your IdP"}
    kibana_1         | {"type":"log","@timestamp":"2020-04-27T14:55:04Z","tags":["status","plugin:searchguard@6.8.6-19.0","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - An error occurred during initialisation, please check the logs.","prevState":"yellow","prevMsg":"'searchguard.cookie.secure' is set to false, cookies are transmitted over unsecure HTTP connection. Consider using HTTPS and set this key to 'true'"}

I want to make sure my configuration is correct before digging into potentially deeper issues. Am I missing something?

The error means SG failed to obtain endpoints from the Keycloak.

I see you use /path/to/cacert.pem file as the root certificate when you call the Keycloak connect URI via curl. But in the Kibana and Elasticsearch configuration you use a different file - /path/to/ca.pem. Do these files contain the same root certificate? If not, try to use /path/to/ca.pem for the curl test.

Yes - sorry, I mistyped it in my original post. The root certificate file is the same in the curl and the yaml configuration files.

There’s an HAProxy in front of Keycloak and every time I try to connect to the IdP endpoint, the HAProxy logs an SSL handshake failure. Which makes me believe my OpenID SSL configuration for Search Guard is off.

Ok. I’ll run an environment similar to your’s and try to reproduce the issue. I’ll let you know.

Please give me the exact version of the Kibana you use. For example, Kibana v7.6.2

I’m using Kibana 6.8.6.

I was able to get past the Kibana/IdP error by adding options in the Kibana configuration file for client certificate and key. So the services are coming up okay.

Running into a similar issue now inside the Elasticsearch service. Whenever I try to access the Kibana dashboard after authenticating through Keycloak, the Elasticsearch container errors out when trying to reach the IdP OpenID configuration endpoint. Here’s the logs:

elasticsearch_1  | [2020-04-30T17:08:35,078][WARN ][c.f.s.h.HTTPBasicAuthenticator] [JVgaRIf] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'
elasticsearch_1  | [2020-04-30T17:08:35,078][DEBUG][c.f.d.a.h.j.k.SelfRefreshingKeySet] [JVgaRIf] performRefresh(YBM4SS0HTjosWTfCnT0ATn4spTDrdTZ3cGcXBgaefwo)
elasticsearch_1  | [2020-04-30T17:08:35,078][INFO ][c.f.d.a.h.j.k.SelfRefreshingKeySet] [JVgaRIf] Performing refresh 1
elasticsearch_1  | [2020-04-30T17:08:35,091][INFO ][c.f.d.a.h.j.AbstractHTTPJwtAuthenticator] [JVgaRIf] com.floragunn.dlic.auth.http.jwt.keybyoidc.AuthenticatorUnavailableException: Authentication backend failed
elasticsearch_1  | [2020-04-30T17:08:35,091][WARN ][c.f.s.a.BackendRegistry  ] [JVgaRIf] Authentication finally failed for null from 172.19.0.3:41904
elasticsearch_1  | [2020-04-30T17:08:35,091][WARN ][c.f.d.a.h.j.k.SelfRefreshingKeySet] [JVgaRIf] KeySetProvider threw error
elasticsearch_1  | com.floragunn.dlic.auth.http.jwt.keybyoidc.AuthenticatorUnavailableException: Error while getting https://keycloak/auth/realms/realm/.well-known/openid-configuration: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
elasticsearch_1  | 	at com.floragunn.dlic.auth.http.jwt.keybyoidc.KeySetRetriever.getJwksUri(KeySetRetriever.java:148) ~[dlic-search-guard-enterprise-modules-6.8.6-34.5.jar:6.8.6-34.5]
elasticsearch_1  | 	at com.floragunn.dlic.auth.http.jwt.keybyoidc.KeySetRetriever.get(KeySetRetriever.java:69) ~[dlic-search-guard-enterprise-modules-6.8.6-34.5.jar:6.8.6-34.5]
elasticsearch_1  | 	at com.floragunn.dlic.auth.http.jwt.keybyoidc.SelfRefreshingKeySet$1.run(SelfRefreshingKeySet.java:213) [dlic-search-guard-enterprise-modules-6.8.6-34.5.jar:6.8.6-34.5]
elasticsearch_1  | 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_151]
elasticsearch_1  | 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_151]
elasticsearch_1  | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
elasticsearch_1  | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
elasticsearch_1  | 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
elasticsearch_1  | Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
elasticsearch_1  | 	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
elasticsearch_1  | 	at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1959) ~[?:?]
elasticsearch_1  | 	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) ~[?:?]
elasticsearch_1  | 	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) ~[?:?]
elasticsearch_1  | 	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514) ~[?:?]
elasticsearch_1  | 	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[?:?]
elasticsearch_1  | 	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026) ~[?:?]
elasticsearch_1  | 	at sun.security.ssl.Handshaker.process_record(Handshaker.java:961) ~[?:?]
elasticsearch_1  | 	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072) ~[?:?]
elasticsearch_1  | 	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) ~[?:?]
elasticsearch_1  | 	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) ~[?:?]
elasticsearch_1  | 	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397) ~[?:?]
elasticsearch_1  | 	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394) ~[httpclient-4.5.2.jar:4.5.2]
elasticsearch_1  | 	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353) ~[httpclient-4.5.2.jar:4.5.2]
elasticsearch_1  | 	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2]
elasticsearch_1  | 	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2]
elasticsearch_1  | 	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2]
elasticsearch_1  | 	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2]
elasticsearch_1  | 	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2]
elasticsearch_1  | 	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2]
elasticsearch_1  | 	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2]
elasticsearch_1  | 	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2]
elasticsearch_1  | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2]
elasticsearch_1  | 	at com.floragunn.dlic.auth.http.jwt.keybyoidc.KeySetRetriever.getJwksUri(KeySetRetriever.java:121) ~[dlic-search-guard-enterprise-modules-6.8.6-34.5.jar:6.8.6-34.5]
elasticsearch_1  | 	... 7 more

The error means that the certificate is not trusted.

How did you create the certificates? It is recommended to use either TLSTool or the Online TLS Certificate Generator. Make sure you have the same root CA for all certificates.

Also, check all your certificates, here is the TLS troubleshooting guide.

Thanks for the link about adding a root certificate to the list of trusted certificates!

I think my problem throughout all of this is that the IdP is expected a two-way SSL authentication (server and client). I had to add the options to do that in the Kibana plugin. Now I’m having issues with the Elasticsearch service. It accepts the certificate that the server sends it, but the server then requests for the client to authenticate and the client claims it has no matching certificates.

*** ServerHelloDone
elasticsearch_1 | Warning: no suitable certificate found - continuing without client authentication
elasticsearch_1 | *** Certificate chain
elasticsearch_1 |
elasticsearch_1 | ***

Is there not a native way in the Search Guard configuration to allow for two-way SSL authentication? It seems like there are options for it (which I included in my sg_config.yml), so I’m confused as to why it isn’t catching on.

The client (SG) doesn’t find a certificate that was signed by any of the signers mentioned in CertificateRequest message of server (Keycloak) https://stackoverflow.com/a/39096286/2393924. And yes, you have it because of the mutual TLS authentication. Can you send the entire Elasticsearch log?

How did you create the certificates? Describe the whole process. Are you sure you have the same root CA for all certificates (both SG and Keycloak)? You can use sgtlsdiag to verify it.

To make the mutual TLS handshake working you need:

  1. Create a certificate for CA
  2. Create certificate A and sign it by CA certificate
  3. Create certificate B and sign it by CA certificate
  4. Then A and B can be authenticated mutually

And you still can curl with the certificates you specified in sg_config, right?

curl -k https://keycloak/auth/realms/realm/.well-known/openid-configuration --cacert /path/to/cacert.pem --cert /path/to/cert.pem

What if you put key in the curl request?

curl -k https://keycloak/auth/realms/realm/.well-known/openid-configuration --cacert /path/to/cacert.pem --cert /path/to/cert.pem --key /path/to/key.pem

I didn’t personally create the client certificates so was unsure of the exact creation process. However, using OpenSSL verifies that they are signed by the root CA.

I was able to resolve all of the handshake issues in the Elasticsearch service as well. I had to add the root CA to the Java truststore and then create a keystore for the client certificate/key and specify it in the Java options.

Thanks for your help navigating TLS configurations. The links you forwarded were helpful.

1 Like

Hi @ccohee Could you please let me know how did you added truststore and keystore. I’m also facing same issue

@manisha-tanwar yes, you need to configure both. A typical TLS config is

elasticsearch.yml

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.keystore_filepath: keystore.jks
searchguard.ssl.transport.truststore_filepath: truststore.jks
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: false
searchguard.ssl.transport.enable_openssl_if_available: true
searchguard.ssl.http.enabled: true
searchguard.ssl.http.keystore_filepath: keystore.jks
searchguard.ssl.http.truststore_filepath: truststore.jks
searchguard.ssl.http.enable_openssl_if_available: true
searchguard.ssl.http.clientauth_mode: OPTIONAL
searchguard.nodes_dn:
- CN=node1.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com
- CN=node2.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com
- CN=node3.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com
searchguard.authcz.admin_dn:
- CN=kirk.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

Also, you can use Search Guard TLS tool to create PEM certificates and use them instead. Look here https://git.floragunn.com/search-guard/search-guard-labs/-/blob/master/docs/tls.md#create-certificates