An error occurred while enabling session management: Error: Failed when trying to obtain the endpoints from your IdP

Hi there

I am configuring kibana 7.2 with Single Sign-On with OpenID and Keycloak for https and its throwing error unable to obtain IDP endpoint. q

I did the same today and it worked. Show the corresponding parts of your kibana.yml and sg_config.yml

sg_config.yml

# This is the main Search Guard configuration file where authentication 
# and authorization is defined.
# 
# You need to configure at least one authentication domain in the authc of this file.
# An authentication domain is responsible for extracting the user credentials from 
# the request and for validating them against an authentication backend like Active Directory for example. 
#
# If more than one authentication domain is configured the first one which succeeds wins. 
# If all authentication domains fail then the request is unauthenticated.
# In this case an exception is thrown and/or the HTTP status is set to 401.
# 
# After authentication authorization (authz) will be applied. There can be zero or more authorizers which collect
# the roles from a given backend for the authenticated user.
#
# Both, authc and auth can be enabled/disabled separately for REST and TRANSPORT layer. Default is true for both.
#        http_enabled: true
#        transport_enabled: true
#
# For HTTP it is possible to allow anonymous authentication. If that is the case then the HTTP authenticators try to
# find user credentials in the HTTP request. If credentials are found then the user gets regularly authenticated.
# If none can be found the user will be authenticated as an "anonymous" user. This user has always the username "sg_anonymous"
# and one role named "sg_anonymous_backendrole". 
# If you enable anonymous authentication all HTTP authenticators will not challenge.
# 
#
# Note: If you define more than one HTTP authenticators make sure to put non-challenging authenticators like "proxy" or "clientcert"
# first and the challenging one last. 
# Because it's not possible to challenge a client with two different authentication methods (for example
# Kerberos and Basic) only one can have the challenge flag set to true. You can cope with this situation
# by using pre-authentication, e.g. sending a HTTP Basic authentication header in the request.
#
# Default value of the challenge flag is true.
# 
#
# HTTP
#   basic (challenging)
#   proxy (not challenging, needs xff)
#   kerberos (challenging) NOT FREE FOR COMMERCIAL
#   clientcert (not challenging, needs https)
#   jwt (not challenging) NOT FREE FOR COMMERCIAL
#   host (not challenging) #DEPRECATED, will be removed in a future version.
#                          host based authentication is configurable in sg_roles_mapping

# Authc
#   internal
#   noop
#   ldap  NOT FREE FOR COMMERCIAL USE

# Authz
#   ldap  NOT FREE FOR COMMERCIAL USE
#   noop


# For more details pls refer to  https://docs.search-guard.com/latest/authentication-authorization

_sg_meta:
  type: "config"
  config_version: 2

sg_config:
      dynamic:
          # Set filtered_alias_mode to 'disallow' to forbid more than 2 filtered aliases per index
          # Set filtered_alias_mode to 'warn' to allow more than 2 filtered aliases per index but warns about it (default)
          # Set filtered_alias_mode to 'nowarn' to allow more than 2 filtered aliases per index silently
          #filtered_alias_mode: warn
          #do_not_fail_on_forbidden: false
          #kibana:
            # Kibana multitenancy - NOT FREE FOR COMMERCIAL USE
            # see https://github.com/floragunncom/search-guard-docs/blob/master/multitenancy.md
            # To make this work you need to install https://github.com/floragunncom/search-guard-module-kibana-multitenancy/wiki
            #multitenancy_enabled: true
            #server_username: kibanaserver
            #index: '.kibana'
          http:
            anonymous_auth_enabled: false
            xff:
              enabled: false
              internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
              #internalProxies: '.*' # trust all internal proxies, regex pattern
              #remoteIpHeader:  'x-forwarded-for'
              ###### see https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html for regex help
              ###### more information about XFF https://en.wikipedia.org/wiki/X-Forwarded-For
              ###### and here https://tools.ietf.org/html/rfc7239
              ###### and https://tomcat.apache.org/tomcat-8.0-doc/config/valve.html#Remote_IP_Valve
          authc:
            openid_auth_domain:
              http_enabled: true
              transport_enabled: true
              order: 1
              http_authenticator:
                type: openid
                challenge: false
                config:
                  subject_key: preferred_username
                  roles_key: roles
                  openid_connect_url: "https://domain_name/auth/realms/kibana-realm/.well-known/openid-configuration"
                  #enable_ssl: false
                  #verify_hosts: false
                  #enable_ssl_client_auth: false
                  #pemtrustedcas_filepath: "/home/smart/smart/CACert.crt"
                  #pemcert_filepath: "/home/smart/smart/cert_name.crt"
                  #pemkey_filepath: "/home/smart/smart/cert_name.crt.key"
              authentication_backend:
                type: noop
            kerberos_auth_domain: 
              http_enabled: false
              transport_enabled: false
              order: 6
              http_authenticator:
                type: kerberos # NOT FREE FOR COMMERCIAL USE
                challenge: true                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
                config:                                                                                                                                                                                                                                                                                                                           
                  # If true a lot of kerberos/security related debugging output will be logged to standard out
                  krb_debug: false
                  # If true then the realm will be stripped from the user name
                  strip_realm_from_principal: true
              authentication_backend:
                type: noop
            basic_internal_auth_domain: 
              description: "Authenticate via HTTP Basic against internal users database"
              http_enabled: true
              transport_enabled: true
              order: 4
              http_authenticator:
                type: basic
                challenge: true
              authentication_backend:
                type: intern
            proxy_auth_domain:
              description: "Authenticate via proxy"
              http_enabled: false
              transport_enabled: false
              order: 3
              http_authenticator:
                type: proxy
                challenge: false
                config:
                  user_header: "x-proxy-user"
                  roles_header: "x-proxy-roles"
              authentication_backend:
                type: noop
            jwt_auth_domain:
              http_enabled: false
              transport_enabled: false
              order: 2
              http_authenticator:
                type: jwt
                challenge: false
                config:
                  signing_key: |-
                    -----BEGIN PUBLIC KEY-----
                  key_conatct
                   -----END PUBLIC KEY-----
                  jwt_header: "Authorization"
                  jwt_url_parameter: jwtparam
                  roles_key: roles
                  subject_key: preferred_username
              authentication_backend:
                type: noop
            clientcert_auth_domain:
              description: "Authenticate via SSL client certificates"
              http_enabled: false
              transport_enabled: false
              order: 2
              http_authenticator:
                type: clientcert 
                config:
                  username_attribute: cn #optional, if omitted DN becomes username
                challenge: false
              authentication_backend:
                type: noop
            ldap:
              description: "Authenticate via LDAP or Active Directory"
              http_enabled: false
              transport_enabled: false
              order: 5
              http_authenticator:
                type: basic
                challenge: false
              authentication_backend:
                # LDAP authentication backend (authenticate users against a LDAP or Active Directory)
                type: ldap # NOT FREE FOR COMMERCIAL USE
                config:
                  # enable ldaps
                  enable_ssl: false
                  # enable start tls, enable_ssl should be false
                  enable_start_tls: false
                  # send client certificate
                  enable_ssl_client_auth: false
                  # verify ldap hostname
                  verify_hostnames: true
                  hosts:
                    - localhost:8389
                  bind_dn: null
                  password: null
                  userbase: 'ou=people,dc=example,dc=com'
                  # Filter to search for users (currently in the whole subtree beneath userbase)
                  # {0} is substituted with the username 
                  usersearch: '(sAMAccountName={0})'
                  # Use this attribute from the user as username (if not set then DN is used)
                  username_attribute: null
          authz:    
            roles_from_myldap:
              description: "Authorize via LDAP or Active Directory"
              http_enabled: false
              transport_enabled: false
              authorization_backend:
                # LDAP authorization backend (gather roles from a LDAP or Active Directory, you have to configure the above LDAP authentication backend settings too)
                type: ldap # NOT FREE FOR COMMERCIAL USE
                config:
                  # enable ldaps
                  enable_ssl: false
                  # enable start tls, enable_ssl should be false
                  enable_start_tls: false
                  # send client certificate
                  enable_ssl_client_auth: false
                  # verify ldap hostname
                  verify_hostnames: true
                  hosts:
                    - localhost:8389
                  bind_dn: null
                  password: null
                  rolebase: 'ou=groups,dc=example,dc=com'
                  # Filter to search for roles (currently in the whole subtree beneath rolebase)
                  # {0} is substituted with the DN of the user
                  # {1} is substituted with the username 
                  # {2} is substituted with an attribute value from user's directory entry, of the authenticated user. Use userroleattribute to specify the name of the attribute            
                  rolesearch: '(member={0})'
                  # Specify the name of the attribute which value should be substituted with {2} above
                  userroleattribute: null
                  # Roles as an attribute of the user entry
                  userrolename: disabled
                  #userrolename: memberOf
                  # The attribute in a role entry containing the name of that role, Default is "name".
                  # Can also be "dn" to use the full DN as rolename.
                  rolename: cn
                  # Resolve nested roles transitive (roles which are members of other roles and so on ...)
                  resolve_nested_roles: true
                  userbase: 'ou=people,dc=example,dc=com'
                  # Filter to search for users (currently in the whole subtree beneath userbase)
                  # {0} is substituted with the username 
                  usersearch: '(uid={0})'
                  # Skip users matching a user name, a wildcard or a regex pattern
                  #skip_users: 
                  #  - 'cn=Michael Jackson,ou*people,o=TEST'
                  #  - '/\S*/'    
            roles_from_another_ldap:
              description: "Authorize via another Active Directory"
              http_enabled: false
              transport_enabled: false
              authorization_backend:
                type: ldap # NOT FREE FOR COMMERCIAL USE
                #config goes here ...
      #    auth_failure_listeners:
      #      ip_rate_limiting:
      #        type: ip
      #        allowed_tries: 10
      #        time_window_seconds: 3600
      #        block_expiry_seconds: 600
      #        max_blocked_clients: 100000
      #        max_tracked_clients: 100000
      #      internal_authentication_backend_limiting:
      #        type: username
      #        authentication_backend: intern        
      #        allowed_tries: 10
      #        time_window_seconds: 3600
      #        block_expiry_seconds: 600
      #        max_blocked_clients: 100000
      #        max_tracked_clients: 100000

kibana.yml

    # Kibana is served by a back end server. This setting specifies the port to use.

    server.port: 5601

    # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.

    # The default is 'localhost', which usually means remote machines will not be able to connect.

    # To allow connections from remote users, set this parameter to a non-loopback address.

    server.host: "0.0.0.0"

    # Enables you to specify a path to mount Kibana at if you are running behind a proxy.

    # Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath

    # from requests it receives, and to prevent a deprecation warning at startup.

    # This setting cannot end in a slash.

    #server.basePath: ""

    # Specifies whether Kibana should rewrite requests that are prefixed with

    # `server.basePath` or require that they are rewritten by your reverse proxy.

    # This setting was effectively always `false` before Kibana 6.3 and will

    # default to `true` starting in Kibana 7.0.

    #server.rewriteBasePath: false

    # The maximum payload size in bytes for incoming server requests.

    #server.maxPayloadBytes: 1048576

    # The Kibana server's name. This is used for display purposes.

    #server.name: "your-hostname"

    # The URLs of the Elasticsearch instances to use for all your queries.
    elasticsearch.hosts: ["https://elastic_domain_name"]

    # When this setting's value is true Kibana uses the hostname specified in the server.host

    # setting. When the value of this setting is false, Kibana uses the hostname of the host

    # that connects to this Kibana instance.

    #elasticsearch.preserveHost: true

    # Kibana uses an index in Elasticsearch to store saved searches, visualizations and

    # dashboards. Kibana creates a new index if the index doesn't already exist.

    #kibana.index: ".kibana"

    # The default application to load.

    #kibana.defaultAppId: "home"

    # If your Elasticsearch is protected with basic authentication, these settings provide

    # the username and password that the Kibana server uses to perform maintenance on the Kibana

    # index at startup. Your Kibana users still need to authenticate with Elasticsearch, which

    # is proxied through the Kibana server.

    #elasticsearch.username: "user"

    #elasticsearch.password: "pass"

    # Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.

    # These settings enable SSL for outgoing requests from the Kibana server to the browser.

    #server.ssl.enabled: false

    #server.ssl.certificate: /path/to/your/server.crt

    #server.ssl.key: /path/to/your/server.key

    # Optional settings that provide the paths to the PEM-format SSL certificate and key files.

    # These files validate that your Elasticsearch backend uses the same key files.

    #elasticsearch.ssl.certificate: /path/to/your/client.crt

    #elasticsearch.ssl.key: /path/to/your/client.key

    # Optional setting that enables you to specify a path to the PEM file for the certificate

    # authority for your Elasticsearch instance.

    #elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

    # To disregard the validity of SSL certificates, change this setting's value to 'none'.

    #elasticsearch.ssl.verificationMode: full

    # Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of

    # the elasticsearch.requestTimeout setting.

    #elasticsearch.pingTimeout: 1500

    # Time in milliseconds to wait for responses from the back end or Elasticsearch. This value

    # must be a positive integer.

    #elasticsearch.requestTimeout: 30000

    # List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side

    # headers, set this value to [] (an empty list).

    #elasticsearch.requestHeadersWhitelist: [ authorization ]

    # Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten

    # by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.

    #elasticsearch.customHeaders: {}

    # Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.

    #elasticsearch.shardTimeout: 30000

    # Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.

    #elasticsearch.startupTimeout: 5000

    # Logs queries sent to Elasticsearch. Requires logging.verbose set to true.

    #elasticsearch.logQueries: false

    # Specifies the path where Kibana creates the process ID file.

    #pid.file: /var/run/kibana.pid

    # Enables you specify a file where Kibana stores log output.

    #logging.dest: stdout

    # Set the value of this setting to true to suppress all logging output.

    #logging.silent: false

    # Set the value of this setting to true to suppress all logging output other than error messages.

    #logging.quiet: false

    # Set the value of this setting to true to log all events, including system usage information

    # and all requests.

    logging.verbose: true

    # Set the interval in milliseconds to sample system and process performance

    # metrics. Minimum is 100ms. Defaults to 5000.

    #ops.interval: 5000

    # Specifies locale to be used for all localizable strings, dates and number formats.

    #i18n.locale: "en"

    elasticsearch.username: "kibanaserver"

    elasticsearch.password: "kibanaserver"

    elasticsearch.ssl.verificationMode: none

    xpack.security.enabled: false

    searchguard.multitenancy.enabled: true

    searchguard.openid.connect_url: "https://keycloak_domain_name/auth/realms/kibana-realm/.well-known/openid-configuration"

    searchguard.openid.client_id: "kibana-sso"

    searchguard.openid.client_secret: "***********************"

    searchguard.openid.base_redirect_url: "https://kibana_domain_name/"

    elasticsearch.requestHeadersWhitelist: ["Authorization", "sgtenant"]

    #searchguard.basicauth.enabled: false

    searchguard.auth.type: "openid"

    #searchguard.jwt.enabled: true

    #searchguard.jwt.url_param: jwtparam

    #searchguard.jwt.header: jwtheader

    xpack.monitoring.enabled: false

    xpack.graph.enabled: false

    xpack.ml.enabled: false

    xpack.security.enabled: false

    xpack.watcher.enabled: false

    xpack.reporting.enabled: false

    #sentinl configuration for email

    sentinl:

    es:

    hosts: ["https://elastic_domain_name"]

    settings:

    email:

    active: true

    user: EMAIL

    password: PASSWORD

    host: smtp.gmail.com

    port: 465

    ssl: true

    tls: false

    #searchguard.openid.root_ca: "/home/smart/smart/certs/CACert.crt"

    #searchguard.openid.verify_hostnames: false

Error: Failed when trying to obtain the endpoints from your IdP

It indicates that the metadata endpoint of the IDP is not reachable.

First, check if the following setting has the correct URL.

kibana.yml

searchguard.openid.connect_url: "https://keycloak_domain_name/auth/realms/kibana-realm/.well-known/openid-configuration" 

Second, go to the Kibana server and check if the URL is accessible from there. For example, you can use curl to do this.

curl -k https://keycloak_domain_name/auth/realms/kibana-realm/.well-known/openid-configuration

Do you get the response?

The elasticsearch nodes must have access to the IDP endpoint too.

sg_config.yml

openid_connect_url: "https://domain_name/auth/realms/kibana-realm/.well-known/openid-configuration"

Hi srgbnd

I have tried all the solutions which you mention. Url and curl request giving proper response from kibana, keycloak, and elastic search server.

Can you post the response here?

searchguard.openid.connect_url: “https://keycloak_domain_name/auth/realms/kibana-realm/.well-known/openid-configuration” response form ElasticSearch Server

    {
        	"issuer": "https://keycloak_domain/auth/realms/kibana-realm",
        	"authorization_endpoint": "https://keycloak_domain/auth/realms/kibana-realm/protocol/openid-connect/auth",
        	"token_endpoint": "https://keycloak_domain/auth/realms/kibana-realm/protocol/openid-connect/token",
        	"token_introspection_endpoint": "https://keycloak_domain/auth/realms/kibana-realm/protocol/openid-connect/token/introspect",
        	"userinfo_endpoint": "https://keycloak_domain/auth/realms/kibana-realm/protocol/openid-connect/userinfo",
        	"end_session_endpoint": "https://keycloak_domain/auth/realms/kibana-realm/protocol/openid-connect/logout",
        	"jwks_uri": "https://keycloak_domain/auth/realms/kibana-realm/protocol/openid-connect/certs",
        	"check_session_iframe": "https://keycloak_domain/auth/realms/kibana-realm/protocol/openid-connect/login-status-iframe.html",
        	"grant_types_supported": ["authorization_code", "implicit", "refresh_token", "password", "client_credentials"],
        	"response_types_supported": ["code", "none", "id_token", "token", "id_token token", "code id_token", "code token", "code id_token token"],
        	"subject_types_supported": ["public", "pairwise"],
        	"id_token_signing_alg_values_supported": ["ES384", "RS384", "HS256", "HS512", "ES256", "RS256", "HS384", "ES512", "RS512"],
        	"userinfo_signing_alg_values_supported": ["ES384", "RS384", "HS256", "HS512", "ES256", "RS256", "HS384", "ES512", "RS512", "none"],
        	"request_object_signing_alg_values_supported": ["none", "RS256"],
        	"response_modes_supported": ["query", "fragment", "form_post"],
        	"registration_endpoint": "https://keycloak_domain/auth/realms/kibana-realm/clients-registrations/openid-connect",
        	"token_endpoint_auth_methods_supported": ["private_key_jwt", "client_secret_basic", "client_secret_post", "client_secret_jwt"],
        	"token_endpoint_auth_signing_alg_values_supported": ["RS256"],
        	"claims_supported": ["sub", "iss", "auth_time", "name", "given_name", "family_name", "preferred_username", "email"],
        	"claim_types_supported": ["normal"],
        	"claims_parameter_supported": false,
        	"scopes_supported": ["openid", "address", "email", "offline_access", "phone", "profile", "roles", "web-origins"],
        	"request_parameter_supported": true,
        	"request_uri_parameter_supported": true,
        	"code_challenge_methods_supported": ["plain", "S256"],
        	"tls_client_certificate_bound_access_tokens": true,
        	"introspection_endpoint": "https://keycloak_domain/auth/realms/kibana-realm/protocol/openid-connect/token/introspect"

}

curl -k https://keycloak_domain_name/auth/realms/kibana-realm/.well-known/openid-configuration Response From Kibana Server !!

  {
        	"issuer": "https://keycloak_domainm/auth/realms/kibana-realm",
        	"authorization_endpoint": "https://keycloak_domainm/auth/realms/kibana-realm/protocol/openid-connect/auth",
        	"token_endpoint": "https://keycloak_domainm/auth/realms/kibana-realm/protocol/openid-connect/token",
        	"token_introspection_endpoint": "https://keycloak_domainm/auth/realms/kibana-realm/protocol/openid-connect/token/introspect",
        	"userinfo_endpoint": "https://keycloak_domainm/auth/realms/kibana-realm/protocol/openid-connect/userinfo",
        	"end_session_endpoint": "https://keycloak_domainm/auth/realms/kibana-realm/protocol/openid-connect/logout",
        	"jwks_uri": "https://keycloak_domainm/auth/realms/kibana-realm/protocol/openid-connect/certs",
        	"check_session_iframe": "https://keycloak_domainm/auth/realms/kibana-realm/protocol/openid-connect/login-status-iframe.html",
        	"grant_types_supported": ["authorization_code", "implicit", "refresh_token", "password", "client_credentials"],
        	"response_types_supported": ["code", "none", "id_token", "token", "id_token token", "code id_token", "code token", "code id_token token"],
        	"subject_types_supported": ["public", "pairwise"],
        	"id_token_signing_alg_values_supported": ["ES384", "RS384", "HS256", "HS512", "ES256", "RS256", "HS384", "ES512", "RS512"],
        	"userinfo_signing_alg_values_supported": ["ES384", "RS384", "HS256", "HS512", "ES256", "RS256", "HS384", "ES512", "RS512", "none"],
        	"request_object_signing_alg_values_supported": ["none", "RS256"],
        	"response_modes_supported": ["query", "fragment", "form_post"],
        	"registration_endpoint": "https://keycloak_domainm/auth/realms/kibana-realm/clients-registrations/openid-connect",
        	"token_endpoint_auth_methods_supported": ["private_key_jwt", "client_secret_basic", "client_secret_post", "client_secret_jwt"],
        	"token_endpoint_auth_signing_alg_values_supported": ["RS256"],
        	"claims_supported": ["sub", "iss", "auth_time", "name", "given_name", "family_name", "preferred_username", "email"],
        	"claim_types_supported": ["normal"],
        	"claims_parameter_supported": false,
        	"scopes_supported": ["openid", "address", "email", "offline_access", "phone", "profile", "roles", "web-origins"],
        	"request_parameter_supported": true,
        	"request_uri_parameter_supported": true,
        	"code_challenge_methods_supported": ["plain", "S256"],
        	"tls_client_certificate_bound_access_tokens": true,
        	"introspection_endpoint": "https://keycloak_domainm/auth/realms/kibana-realm/protocol/openid-connect/token/introspect"
        }

Ok. Let’s try to debug it.

First, Go to the Kibana server, execute the following two commands, and show me the results.

nslookup keycloak_domain_name
traceroute keycloak_domain_name

Second, while on the Kibana server go to kibana/plugins/searchguard/lib/auth/types/openid/OpenId.js. Find setupRoutes method, it should look like this https://git.floragunn.com/search-guard/search-guard-kibana-plugin/-/blob/7.2.x/lib/auth/types/openid/OpenId.js#L113 Comment it like this.

/*
setupRoutes() {
...
}
*/

Put the following method in the OpenId.js file

    async setupRoutes() {
        try {
            const {response, payload} = await Wreck.get(this.config.get('searchguard.openid.connect_url'));
            console.log('openid', 'setupRoutes', 'response', response);
            console.log('openid', 'setupRoutes', 'payload', payload);

            const parsedPayload = JSON.parse(payload.toString());

            let endPoints = {
                authorization_endpoint: parsedPayload.authorization_endpoint,
                token_endpoint: parsedPayload.token_endpoint,
                end_session_endpoint: parsedPayload.end_session_endpoint || null
            };
            console.log('openid', 'setupRoutes', 'endPoints', endPoints);

            require('./routes')(this.pluginRoot, this.server, this.kbnServer, this.APP_ROOT, this.API_ROOT, endPoints);
            console.log('openid', 'setupRoutes', 'routes registered');

        } catch (error) {
            console.error('openid', error);
            if (error ||
                error.output.statusCode < 200 ||
                error.output.statusCode > 299) {
                throw new Error('Failed when trying to obtain the endpoints from your IdP');
            }
        }
    }

Save, the Kibana should restart. Try OpenID authentication, get the Kibana log and paste it here.