Clone- Error loading data” error observed in kibana when keycloak token expires

shubha02 · January 11, 2021, 3:20pm

Hi, since the original thread - "Error loading data" error observed in kibana when keycloak token expires has been auto closed , created this new query.

We are using keycloak openid authentication in kibana.
When Kibana session remains idle for sometime (10mins) and keycloak token expires (token life span is 5 mins) in between that, after that if the user try to do filter on discover page based on a particular time period then the error pop up “Error loading data”.But manually refreshing the page can recover the access, and load the logs"

Blockquote
TypeError: Failed to fetch
at Fetch._callee3$ (https://ip/baseurl/port/bundles/commons.bundle.js:9:1989293)
at l (https://ip/baseurl/port/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:288:969217)
at Generator._invoke (https://ip/baseurl/port/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:288:968970)
at Generator.forEach.e. [as throw] (https://ip/baseurl/port/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:288:969574)
at asyncGeneratorStep (https://ip/baseurl/port/bundles/commons.bundle.js:9:1983787)
at _throw (https://ip/baseurl/port/bundles/commons.bundle.js:9:1984184)

When checked Elasticsearch client logs it says : “log”:“No ‘Basic Authorization’ header, send 401 and ‘WWW-Authenticate Basic’”}

The expectation is it should be redirected to the login page when the keycloak token gets expires.
Searchguard version used is : 7.8.0-43.0.0.
below error seen in elasticsearch client

{“type”:“log”,“host”:“elasticsearch-client-68cd995bbd-crf8h”,“level”:“WARN”,“systemid”:“2106a117733f42d697284fbc54927928”,“system”:“XXXX”,“time”: “2020-09-24T12:00:55.992Z”,“logger”:“c.f.s.h.HTTPBasicAuthenticator”,“timezone”:“UTC”,“marker”:"[elasticsearch-client-68cd995bbd-crf8h] ",“log”:“No ‘Basic Authorization’ header, send 401 and ‘WWW-Authenticate Basic’”}

srgbnd · January 12, 2021, 9:16am

Hi @shubha02. Thank you for reopening this.
To help us reproduce the issue send the following configurations:

kibana.yml
elasticsearch.yml
sg_config.yml

shubha02 · January 12, 2021, 11:27am

HI @srgbnd

Kibana yaml:

---
# Donot change sever name and host. This is default configuration.
server.name: kibana
server.customResponseHeaders: { "X-Frame-Options": "DENY" }
server.ssl.supportedProtocols: ["TLSv1.2"]
searchguard.cookie.secure: true
# uncomment below section for keycloak authentication and provide required correct parameters
searchguard.auth.type: "openid"
searchguard.openid.connect_url: "https://XX.XX.XXXX:XXXXX/auth/realms/master/.well-known/openid-configuration"
searchguard.openid.client_id: XXXXXXXX
searchguard.openid.client_secret: "XXXXX"
searchguard.openid.header: "Authorization"
### for kibana service on ingress port is not required
searchguard.openid.base_redirect_url: "https://XX.XX.XXXX"
### Do not change root_ca file path as this is the default mount path.
searchguard.openid.root_ca: "/XXXX/keycloak-root-ca.pem"
searchguard.openid.verify_hostnames: false

elasticsearch yaml:


cluster:
  name: ${CLUSTER_NAME}
  initial_master_nodes: ${CLUSTER_INITIAL_MASTER_NODES}

node:
  master: ${NODE_MASTER}
  data: ${NODE_DATA}
  name: ${NODE_NAME}
  ingest: ${NODE_INGEST}
  max_local_storage_nodes: ${MAX_LOCAL_STORAGE_NODES}

network.host: ${NETWORK_HOST}

path:
  data: /data/data
  logs: /data/log
  repo: ${PATH_REPO}

#bootstrap.memory_lock: true

http:
  compression: true
  cors:
    enabled: ${HTTP_CORS_ENABLE}
    allow-origin: ${HTTP_CORS_ALLOW_ORIGIN}

discovery:
  seed_hosts: ${DISCOVERY_SERVICE}

searchguard:
    ssl.transport:
        enabled: true
        enable_openssl_if_available: false
        keystore_type: ${KEYSTORE_TYPE}
        keystore_filepath: ${KEYSTORE_FILEPATH}
        keystore_password: ${KS_PWD}
        truststore_type: ${TRUSTSTORE_TYPE}
        truststore_filepath: ${TRUSTSTORE_FILEPATH}
        truststore_password: ${TS_PWD}
        enforce_hostname_verification: false
    ssl.http:
        enabled: ${HTTP_SSL}
        clientauth_mode: OPTIONAL
        enable_openssl_if_available: true
        keystore_type: ${KEYSTORE_TYPE}
        keystore_filepath: ${KEYSTORE_FILEPATH}
        keystore_password: ${KS_PWD}
        truststore_type: ${TRUSTSTORE_TYPE}
        truststore_filepath: ${TRUSTSTORE_FILEPATH}
        truststore_password: ${TS_PWD}
    authcz.admin_dn:
      - "${AUTH_ADMIN_DN}"
    enterprise_modules_enabled: false
    nodes_dn:
      - "${NODES_DN}"

searchguard yaml:

_sg_meta:
  type: "config"
  config_version: 2
sg_config:
  dynamic:
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
        internalProxies: '.+'
    authc:
      basic_internal_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: "basic"
          challenge: false   # Set this to false when keycloak authentication is enabled
        authentication_backend:
          type: "intern"
      keycloak_auth_domain:
        http_enabled: true  # Set to true to enable keycloak authentication
        transport_enabled: true
        order: 1
        http_authenticator:
          type: keycloak
          challenge: false
          config:
            username_key: preferred_username
            roles_key: roles
            keycloak_connect_url: https://XXXXXX/auth/realms/master/.well-known/openid-configuration
            client_id: XXXX
            client_secret: XXXX
            ssl_validate: false
        authentication_backend:
            type: noop
      proxy_auth_domain:
        http_enabled: false
        transport_enabled: false
        order: 3
        http_authenticator:
          type: "proxy"
          challenge: false
          config:
            user_header: "x-proxy-user"
            #roles_header: "x-proxy-roles"
        authentication_backend:
          type: "noop"
      clientcert_auth_domain:
        http_enabled: false
        transport_enabled: false
        order: 2
        http_authenticator:
          challenge: false
          type: "clientcert"
          config:
            username_attribute: "cn"
        authentication_backend:
          type: "noop"

please let us know in case any other information is required.

srgbnd · January 18, 2021, 6:55pm

I can reproduce the issue. It was added to the bug queue. I’ll let you know when it is resolved.

Mike · January 21, 2021, 1:22pm

Hi @shubha02,

I believe the behaviour we reproduced is somewhat different to the problem you’re seeing.
Just to clarify:

when you do a full page reload, you are redirected as expected to Keycloak’s login page, correct?
would if be possible for you to check the browser’s network tab for a request that triggers the error popup? You’d be looking for a requesting returning status code 401 or 302, and it would be great if I could see what the request headers are on that request. Here’s an example of the request going out when I change the time filter in Discover:
image1286×1148 124 KB

My guess is that we for some reason can’t detect that the search request is an AJAX request, and because of that we can’t redirect to the login page properly in this case.

Thanks,
Mike

shubha02 · February 4, 2021, 12:29pm

Hi,

when you do a full page reload, you are redirected as expected to Keycloak’s login page, correct? > yes, upon a full page reload , it is redirected to keycloak’s page for authentication.
when I observed this issue “Error loading data” , this is the network tab response.

Request URL: https://10.xx.xx.xx/xxxxx/auth/openid/login?nextUrl=%2Fapp%2Fkibana%23%2Fdiscover%3F_g%3D(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-15m%2Cto%3Anow))%26_a%3D(columns%3A!(_source)%2Cfilters%3A!()%2Cindex%3Add4538a0-66bf-11eb-a10c-9b7c2ee52a23%2Cinterval%3Aauto%2Cquery%3A(language%3Akuery%2Cquery%3A%27%27)%2Csort%3A!())
Request Method: GET
Status Code: 302 
Remote Address: 10.xx.xx.xx:443
Referrer Policy: strict-origin-when-cross-origin

Please let me in case you need more information.

shubha02 · February 16, 2021, 11:19am

Hi , Is there any update for this issue ?

Mike · February 17, 2021, 1:41pm

Hi @shubha02,

My apologies, I didn’t see your previous answer.

Is there any request before the one you sent that returns a 401 or 302?
There are two things I’m looking for:

Request URL should be something different than /auth/openid/login
The following request (not response) headers on that previous request: Content-Type or Accept.
In my screenshot above you can see that Content-Type is set to “application/json”

If none of those headers are set, we can’t detect an AJAX request at the moment.

Feel free to use @Mike in you answer, then I should get a notification.

Thanks, and sorry again for the late response.
Mike