Authentication failed. Please provide a new token

Search Guard
1

Hi,

I tried to configure Kibana Single Sign-On with OpenID and Keycloak. However I receive the error “Authentication failed. Please provide a new token.” in Kibana

----------------* Search Guard configuration files*----------------
openid_auth_domain:
enabled: true
order: 1
http_authenticator:
type: openid
challenge: false
config:
openid_connect_url: http://192.168.152.143:8080/auth/realms/Test/.well-known/openid-configuration
subject_key: preferred_username
roles_key: roles
enable_ssl: false
verify_hostnames: false
authentication_backend:
type: noop

------------* * elasticsearch.yml configuration file**---------------------------------
xpack.security.enabled: false

WARNING: revise all the lines below before you go into production

searchguard.ssl.transport.pemcert_filepath: esnode.pem
searchguard.ssl.transport.pemkey_filepath: esnode-key.pem
searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: esnode.pem
searchguard.ssl.http.pemkey_filepath: esnode-key.pem
searchguard.ssl.http.pemtrustedcas_filepath: root-ca.pem
searchguard.allow_unsafe_democertificates: true
searchguard.allow_default_init_sgindex: true
searchguard.authcz.admin_dn:

  • CN=kirk,OU=client,O=client,L=test, C=de

searchguard.audit.type: internal_elasticsearch
searchguard.enable_snapshot_restore_privilege: true
searchguard.check_snapshot_restore_write_privileges: true
searchguard.restapi.roles_enabled: [“SGS_ALL_ACCESS”]
cluster.routing.allocation.disk.threshold_enabled: false
cluster.name: searchguard_demo
node.max_local_storage_nodes: 3
######## End Search Guard Demo Configuration ########

----------------** kibana.yml configuration file**-----------------------

elasticsearch.hosts: “https://localhost:9200”
elasticsearch.ssl.verificationMode: none
elasticsearch.username: “admin”
elasticsearch.password: “admin”
searchguard.auth.type: “openid”

the IdP metadata endpoint

searchguard.openid.connect_url: “http://192.168.152.143:8080/auth/realms/Test/.well-known/openid-configuration”

the ID of the OpenID Connect client in your IdP

searchguard.openid.client_id: “kibana-sso”

searchguard.openid.client_secret: “25c3e23f-b7df-42c6-a640-793def4d32f0”
searchguard.openid.verify_hostnames: false

xpack.security.enabled: false
server.ssl.enabled: true
server.ssl.key: /kibana-7.3.0-windows-x86_64/kibana-7.3.0-windows-x86_64/config/certs/instance/instance.key
server.ssl.certificate: /kibana-7.3.0-windows-x86_64/kibana-7.3.0-windows-x86_64/config/certs/instance/instance.crt

3

This can have several reasons. Do you see anything in the Elasticsearch or Kibana logfiles?

Can you please set the Elasticsearch log level to debug:

And then try again? You should see an error message in the logfile, please post it here.

4

Hi,

I turned the log level to debug mode however i can’t see any significant error message in the logs.

The logs as per following:

[2019-08-14T16:07:59,526][DEBUG][c.f.s.p.PrivilegesEvaluator] [DESKTOP-BN85TH6] Aliases for .kibana_1: [.kibana=>{
“.kibana” : { }
}]
[2019-08-14T16:07:59,526][DEBUG][c.f.s.p.PrivilegesEvaluator] [DESKTOP-BN85TH6] sgr: [SGS_ALL_ACCESS, SGS_OWN_INDEX]
[2019-08-14T16:07:59,526][DEBUG][c.f.s.p.PrivilegesEvaluator] [DESKTOP-BN85TH6] Aliases for .kibana_1: [.kibana=>{
“.kibana” : { }
}]
[2019-08-14T16:07:59,527][DEBUG][c.f.s.c.PrivilegesInterceptorImpl] [DESKTOP-BN85TH6] raw requestedTenant: ‘null’
[2019-08-14T16:07:59,526][DEBUG][c.f.s.p.PrivilegesEvaluator] [DESKTOP-BN85TH6] Aliases for .kibana_1: [.kibana=>{
“.kibana” : { }
}]
[2019-08-14T16:07:59,527][DEBUG][c.f.s.p.PrivilegesEvaluator] [DESKTOP-BN85TH6] sgr2: [SGS_ALL_ACCESS, SGS_OWN_INDEX]
[2019-08-14T16:07:59,527][DEBUG][c.f.s.p.PrivilegesEvaluator] [DESKTOP-BN85TH6] .kibana is not an alias or does not have a filter
[2019-08-14T16:07:59,527][DEBUG][c.f.s.p.PrivilegesEvaluator] [DESKTOP-BN85TH6] .kibana is not an alias or does not have a filter
[2019-08-14T16:07:59,534][DEBUG][c.f.s.p.PrivilegesEvaluator] [DESKTOP-BN85TH6] Allowed because we have all indices permissions for indices:data/read/get
[2019-08-14T16:07:59,531][DEBUG][c.f.s.p.PrivilegesEvaluator] [DESKTOP-BN85TH6] Aliases for .kibana_1: [.kibana=>{
“.kibana” : { }
}]
[2019-08-14T16:07:59,534][DEBUG][c.f.s.f.SearchGuardFilter] [DESKTOP-BN85TH6] PrivEvalResponse [allowed=true, missingPrivileges=[indices:data/read/get], allowedFlsFields=null, maskedFields=null, queries=null]
[2019-08-14T16:07:59,526][DEBUG][c.f.s.p.PrivilegesEvaluator] [DESKTOP-BN85TH6] sgr2: [SGS_ALL_ACCESS, SGS_OWN_INDEX]
[2019-08-14T16:07:59,538][DEBUG][c.f.s.p.PrivilegesEvaluator] [DESKTOP-BN85TH6] Aliases for .kibana_1: [.kibana=>{
“.kibana” : { }
}]
[2019-08-14T16:07:59,539][DEBUG][c.f.s.p.PrivilegesEvaluator] [DESKTOP-BN85TH6] .kibana is not an alias or does not have a filter
[2019-08-14T16:07:59,554][DEBUG][c.f.s.p.PrivilegesEvaluator] [DESKTOP-BN85TH6] Allowed because we have all indices permissions for indices:data/read/get
[2019-08-14T16:07:59,534][DEBUG][c.f.s.p.PrivilegesEvaluator] [DESKTOP-BN85TH6] Allowed because we have all indices permissions for indices:data/read/get
[2019-08-14T16:07:59,528][DEBUG][c.f.s.p.PrivilegesEvaluator] [DESKTOP-BN85TH6] .kibana is not an alias or does not have a filter
[2019-08-14T16:07:59,536][DEBUG][c.f.s.p.PrivilegesEvaluator] [DESKTOP-BN85TH6] .kibana is not an alias or does not have a filter
[2019-08-14T16:07:59,528][DEBUG][c.f.s.c.PrivilegesInterceptorImpl] [DESKTOP-BN85TH6] request class org.elasticsearch.action.search.SearchRequest
[2019-08-14T16:07:59,556][DEBUG][c.f.s.p.PrivilegesEvaluator] [DESKTOP-BN85TH6] Allowed because we have all indices permissions for indices:data/read/search
[2019-08-14T16:07:59,555][DEBUG][c.f.s.f.SearchGuardFilter] [DESKTOP-BN85TH6] PrivEvalResponse [allowed=true, missingPrivileges=[indices:data/read/get], allowedFlsFields=null, maskedFields=null, queries=null]
[2019-08-14T16:07:59,555][DEBUG][c.f.s.f.SearchGuardFilter] [DESKTOP-BN85TH6] PrivEvalResponse [allowed=true, missingPrivileges=[indices:data/read/get], allowedFlsFields=null, maskedFields=null, queries=null]
[2019-08-14T16:07:59,547][DEBUG][c.f.s.a.BackendRegistry ] [DESKTOP-BN85TH6] Check authdomain for rest internal/4 or 1 in total
[2019-08-14T16:07:59,577][DEBUG][c.f.s.a.BackendRegistry ] [DESKTOP-BN85TH6] Check authdomain for rest internal/4 or 1 in total
[2019-08-14T16:07:59,582][DEBUG][c.f.s.a.BackendRegistry ] [DESKTOP-BN85TH6] Rest user ‘User [name=admin, backend_roles=[admin], requestedTenant=null]’ is authenticated
[2019-08-14T16:07:59,559][DEBUG][c.f.s.f.SearchGuardFilter] [DESKTOP-BN85TH6] PrivEvalResponse [allowed=true, missingPrivileges=[indices:data/read/search], allowedFlsFields=null, maskedFields=null, queries=null]
[2019-08-14T16:07:59,557][DEBUG][c.f.s.p.PrivilegesEvaluator] [DESKTOP-BN85TH6] Allowed because we have all indices permissions for indices:data/read/get
[2019-08-14T16:07:59,592][DEBUG][c.f.s.f.SearchGuardFilter] [DESKTOP-BN85TH6] PrivEvalResponse [allowed=true, missingPrivileges=[indices:data/read/get], allowedFlsFields=null, maskedFields=null, queries=null]
[2019-08-14T16:07:59,558][DEBUG][c.f.s.p.PrivilegesEvaluator] [DESKTOP-BN85TH6] Result from privileges interceptor: null
[2019-08-14T16:07:59,587][DEBUG][c.f.s.a.BackendRegistry ] [DESKTOP-BN85TH6] sgtenant ‘null’
[2019-08-14T16:07:59,577][DEBUG][c.f.s.a.BackendRegistry ] [DESKTOP-BN85TH6] Rest user ‘User [name=admin, backend_roles=[admin], requestedTenant=null]’ is authenticated
[2019-08-14T16:07:59,602][DEBUG][c.f.s.p.PrivilegesEvaluator] [DESKTOP-BN85TH6] ### evaluate permissions for User [name=admin, backend_roles=[admin], requestedTenant=null] on DESKTOP-BN85TH6
[2019-08-14T16:07:59,584][DEBUG][c.f.s.a.BackendRegistry ] [DESKTOP-BN85TH6] Check authdomain for rest internal/4 or 1 in total
[2019-08-14T16:07:59,599][DEBUG][c.f.s.p.PrivilegesEvaluator] [DESKTOP-BN85TH6] sgr2: [SGS_ALL_ACCESS, SGS_OWN_INDEX]
[2019-08-14T16:07:59,603][DEBUG][c.f.s.p.PrivilegesEvaluator] [DESKTOP-BN85TH6] action: indices:data/read/get (GetRequest)
[2019-08-14T16:07:59,606][DEBUG][c.f.s.p.PrivilegesEvaluator] [DESKTOP-BN85TH6] Aliases for .kibana_1: [.kibana=>{
“.kibana” : { }
}]
[2019-08-14T16:07:59,599][DEBUG][c.f.s.a.BackendRegistry ] [DESKTOP-BN85TH6] Check authdomain for rest internal/4 or 1 in total
[2019-08-14T16:07:59,599][DEBUG][c.f.s.a.BackendRegistry ] [DESKTOP-BN85TH6] Check authdomain for rest internal/4 or 1 in total
[2019-08-14T16:07:59,605][DEBUG][c.f.s.a.BackendRegistry ] [DESKTOP-BN85TH6] Rest user ‘User [name=admin, backend_roles=[admin], requestedTenant=null]’ is authenticated
[2019-08-14T16:07:59,607][DEBUG][c.f.s.r.IndexResolverReplacer] [DESKTOP-BN85TH6] Resolve aliases, indices and types from GetRequest
[2019-08-14T16:07:59,609][DEBUG][c.f.s.p.PrivilegesEvaluator] [DESKTOP-BN85TH6] .kibana is not an alias or does not have a filter
[2019-08-14T16:07:59,609][DEBUG][c.f.s.a.BackendRegistry ] [DESKTOP-BN85TH6] Rest user ‘User [name=admin, backend_roles=[admin], requestedTenant=null]’ is authenticated
[2019-08-14T16:07:59,603][DEBUG][c.f.s.a.BackendRegistry ] [DESKTOP-BN85TH6] sgtenant ‘null’
[2019-08-14T16:07:59,614][DEBUG][c.f.s.a.BackendRegistry ] [DESKTOP-BN85TH6] sgtenant ‘null’

5

From the log file it seems that you only have one authentication domain configured:

This is the internal auth domain which is used to authenticate the Kibana server user. It seems you did not configure the OpenID auth domain in sg_config.yml, or it is not active, could that be?

Can you please post your sg_config.yml?

6

We have this exact same issue using a ASP.NET core Identity Server. We also have the “Check authdomain for rest internal/4 or 1 in total” log entries, but we do have the OpenID auth domain configured.
We are testing the openid/searchguard integration on our local machines.

The openid auth config seems to be in effect because only the last step fails: The token endpoint on the IdP is returning a proper id_token with all the claims needed to SearchGuard, but SearchGuard still issues the “Authentication failed. Please provide a new token.” error.

Setting the

logger.sg.name = com.floragunn.dlic.auth.http.jwt
logger.sg.level = trace

in log4j2.properties as advised in Troubleshooting OpenID guidence did not have any effect at all. Setting the general logging level to Debug did, but dit not result in any information about why the id_token is not accepted.

Below our sg_config.yml

---

# This is the main Search Guard configuration file where authentication 
# and authorization is defined.
# 
# You need to configure at least one authentication domain in the authc of this file.
# An authentication domain is responsible for extracting the user credentials from 
# the request and for validating them against an authentication backend like Active Directory for example. 
#
# If more than one authentication domain is configured the first one which succeeds wins. 
# If all authentication domains fail then the request is unauthenticated.
# In this case an exception is thrown and/or the HTTP status is set to 401.
# 
# After authentication authorization (authz) will be applied. There can be zero or more authorizers which collect
# the roles from a given backend for the authenticated user.
#
# Both, authc and auth can be enabled/disabled separately for REST and TRANSPORT layer. Default is true for both.
#        http_enabled: true
#        transport_enabled: true
#
# For HTTP it is possible to allow anonymous authentication. If that is the case then the HTTP authenticators try to
# find user credentials in the HTTP request. If credentials are found then the user gets regularly authenticated.
# If none can be found the user will be authenticated as an "anonymous" user. This user has always the username "sg_anonymous"
# and one role named "sg_anonymous_backendrole". 
# If you enable anonymous authentication all HTTP authenticators will not challenge.
# 
#
# Note: If you define more than one HTTP authenticators make sure to put non-challenging authenticators like "proxy" or "clientcert"
# first and the challenging one last. 
# Because it's not possible to challenge a client with two different authentication methods (for example
# Kerberos and Basic) only one can have the challenge flag set to true. You can cope with this situation
# by using pre-authentication, e.g. sending a HTTP Basic authentication header in the request.
#
# Default value of the challenge flag is true.
# 
#
# HTTP
#   basic (challenging)
#   proxy (not challenging, needs xff)
#   kerberos (challenging) NOT FREE FOR COMMERCIAL
#   clientcert (not challenging, needs https)
#   jwt (not challenging) NOT FREE FOR COMMERCIAL
#   host (not challenging) #DEPRECATED, will be removed in a future version.
#                          host based authentication is configurable in sg_roles_mapping

# Authc
#   internal
#   noop
#   ldap  NOT FREE FOR COMMERCIAL USE

# Authz
#   ldap  NOT FREE FOR COMMERCIAL USE
#   noop


# For more details pls refer to  https://docs.search-guard.com/latest/authentication-authorization

_sg_meta:
  type: "config"
  config_version: 2

sg_config:
  dynamic:
      # Set filtered_alias_mode to 'disallow' to forbid more than 2 filtered aliases per index
      # Set filtered_alias_mode to 'warn' to allow more than 2 filtered aliases per index but warns about it (default)
      # Set filtered_alias_mode to 'nowarn' to allow more than 2 filtered aliases per index silently
      #filtered_alias_mode: warn
      #do_not_fail_on_forbidden: false
      #kibana:
        # Kibana multitenancy - NOT FREE FOR COMMERCIAL USE
        # see https://github.com/floragunncom/search-guard-docs/blob/master/multitenancy.md
        # To make this work you need to install https://github.com/floragunncom/search-guard-module-kibana-multitenancy/wiki
        #multitenancy_enabled: true
        #server_username: kibanaserver
        #index: '.kibana'
      http:
        anonymous_auth_enabled: false
        xff:
          enabled: false
          internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
          #internalProxies: '.*' # trust all internal proxies, regex pattern
          #remoteIpHeader:  'x-forwarded-for'
          ###### see https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html for regex help
          ###### more information about XFF https://en.wikipedia.org/wiki/X-Forwarded-For
          ###### and here https://tools.ietf.org/html/rfc7239
          ###### and https://tomcat.apache.org/tomcat-8.0-doc/config/valve.html#Remote_IP_Valve
      authc:
        basic_internal_auth_domain: 
          description: "Authenticate via HTTP Basic against internal users database"
          http_enabled: true
          transport_enabled: true
          order: 0
          http_authenticator:
            type: basic
            challenge: false
          authentication_backend:
            type: "internal"
        openid_auth_domain:
          http_enabled: true
          transport_enabled: true
          order: 1
          http_authenticator:
            type: openid
            challenge: false
            config:
              subject_key: preferred_username
              roles_key: roles
              openid_connect_url: "http://localhost:51331/.well-known/openid-configuration"
          authentication_backend:
            type: noop
      authz:    
        roles_from_myldap:
          description: "Authorize via LDAP or Active Directory"
          http_enabled: false
          transport_enabled: false
          authorization_backend:
            # LDAP authorization backend (gather roles from a LDAP or Active Directory, you have to configure the above LDAP authentication backend settings too)
            type: ldap # NOT FREE FOR COMMERCIAL USE
            config:
              # enable ldaps
              enable_ssl: false
              # enable start tls, enable_ssl should be false
              enable_start_tls: false
              # send client certificate
              enable_ssl_client_auth: false
              # verify ldap hostname
              verify_hostnames: true
              hosts:
                - localhost:8389
              bind_dn: null
              password: null
              rolebase: 'ou=groups,dc=example,dc=com'
              # Filter to search for roles (currently in the whole subtree beneath rolebase)
              # {0} is substituted with the DN of the user
              # {1} is substituted with the username 
              # {2} is substituted with an attribute value from user's directory entry, of the authenticated user. Use userroleattribute to specify the name of the attribute            
              rolesearch: '(member={0})'
              # Specify the name of the attribute which value should be substituted with {2} above
              userroleattribute: null
              # Roles as an attribute of the user entry
              userrolename: disabled
              #userrolename: memberOf
              # The attribute in a role entry containing the name of that role, Default is "name".
              # Can also be "dn" to use the full DN as rolename.
              rolename: cn
              # Resolve nested roles transitive (roles which are members of other roles and so on ...)
              resolve_nested_roles: true
              userbase: 'ou=people,dc=example,dc=com'
              # Filter to search for users (currently in the whole subtree beneath userbase)
              # {0} is substituted with the username 
              usersearch: '(uid={0})'
              # Skip users matching a user name, a wildcard or a regex pattern
              #skip_users: 
              #  - 'cn=Michael Jackson,ou*people,o=TEST'
              #  - '/\S*/'    
        roles_from_another_ldap:
          description: "Authorize via another Active Directory"
          http_enabled: false
          transport_enabled: false
          authorization_backend:
            type: ldap # NOT FREE FOR COMMERCIAL USE
            #config goes here ...
  #    auth_failure_listeners:
  #      ip_rate_limiting:
  #        type: ip
  #        allowed_tries: 10
  #        time_window_seconds: 3600
  #        block_expiry_seconds: 600
  #        max_blocked_clients: 100000
  #        max_tracked_clients: 100000
  #      internal_authentication_backend_limiting:
  #        type: username
  #        authentication_backend: intern        
  #        allowed_tries: 10
  #        time_window_seconds: 3600
  #        block_expiry_seconds: 600
  #        max_blocked_clients: 100000
  #        max_tracked_clients: 100000
7

Update on te preceding reply: We solved the issue by simply running the sgadmin tool:

./sgadmin.sh -cd ../sgconfig/ -icl -nhnv  
   -cacert ../../../config/root-ca.pem 
   -cert ../../../config/kirk.pem 
   -key ../../../config/kirk-key.pem

We made the mistake of thinking that since we were testing on localhost and restarting both elasticsearch and kibana completely each time we changed something, the settings would be picked up.
This indeed seemed to be the case because the user was redirected correctly to the IdP.
But that is apparently because the kibana.yml settings also contain the IdP url.

The settings in sg_config.yml need to be loaded at least once using the sgadmin tool even when restarting all services involved.

8

Just to clarify: You need to run sgadmin every time you make changes in your configs. The tool will then push the configuration changes to your running cluster. However, you do not need to run it if the service was only restarted, and no configuration changes have been made. We store all configs in a secured Elasticsearch index, so it survives any restart.

9

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.