TypeError: Cannot read property 'timed_out' of null and TypeError: Cannot read property 'total' of undefined

Hi guys,

We are using kibana, search guard in combination with keycloak (openid). While reading indeces in the kibana dashboards I have following errors appearing in the kibana UI:

TypeError: Cannot read property 'timed_out' of null
    at handleResponse (https://log.kibana.com/bundles/commons.bundle.js:11:4099490)
    at https://log.kibana.com/bundles/commons.bundle.js:11:4092562

and after this one:

TypeError: Cannot read property 'total' of undefined
    at tabifyAggResponse (https://log.kibana.com/bundles/commons.bundle.js:11:1527391)
    at onResults (https://log.kibana.com/bundles/58.bundle.js:3:32315)
    at https://log.kibana.com/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:94842
    at https://llog.kibana.com/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:94980
    at u.$digest (https://log.kibana.com/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:100155)
    at https://log.kibana.com/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:102127
    at Yo.completeTask (https://log.kibana.com/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:122692)
    at https://log.kibana.com/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:34257

you have an idea why it happens?

Hi.
Look in the Elasticsearch log. You should have error(s) there if it is the permission related errors. Paste the errors here. What version of SG do you use?

Thanks for the response.

There are no logs in elastic when it happens. log level is set to info

Searchguard version is: 41.0.0
elasitc is. 7.6.2

In kibana logs (runnin in openshift) I can see responses with 401:

{
“type”:“response”,
@timestamp”:“2020-09-25T13:38:52Z”,
“tags”:[

],
“pid”:1,
“method”:“post”,
“statusCode”:401,
“req”:{
“url”:"/elastic/filebeat-prod-/_search?rest_total_hits_as_int=true&ignore_unavailable=true&ignore_throttled=true&preference=1601040743379&timeout=30000ms",
“method”:“post”,
“headers”:{
“user-agent”:“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36”,
“accept”:“application/json, text/plain, /”,
“content-type”:“application/json”,
“accept-language”:“en-US,en;q=0.9,de;q=0.8,ru;q=0.7”,
“referer”:“https://log.kibana.com/app/kibana",
“sec-fetch-dest”:“empty”,
“origin”:“https://log.kibana.com”,
“kbn-version”:“7.6.2”,
“x-cluster-client-ip”:“1.2.3.4”,
“sec-fetch-site”:“same-origin”,
“sec-fetch-mode”:“cors”,
“accept-encoding”:"gzip
, deflate, br”,
“content-length”:“6021”,
“host”:“log.kibana.com”,
“x-forwarded-host”:“log.kibana.com”,
“x-forwarded-port”:“443”,
“x-forwarded-proto”:“https”,
“forwarded”:“for=1.2.3.4;host=log.kibana.com;proto=https;proto-version=”,
“x-forwarded-for”:“1.2.3.4”
},
“remoteAddress”:“4.3.2.1”,
“userAgent”:“4.3.2.1”,
“referer”:"https://log.kibana.com/app/kibana"
},
“res”:{
“statusCode”:401,
“responseTime”:23,
“contentLength”:9
},
“message”:"POST
/elastic/filebeat-prod-
/_search?rest_total_hits_as_int=true&ignore_unavailable=true&ignore_throttled=true&preference=1601040743379&timeout=30000ms 401 23ms - 9.0B"
}

It looks like during my session I lose my authentication somehow.

While reading indeces …

What do you mean? Can you post a screenshot?

Do you have the request payload for this call? You can check the browser dev tools for it. I wonder what would happen if you attach the payload to a curl request. Try to do it without Kibana to narrow down the scope of the issue.

When I open a fresh kibana session and begin to use the dashboard everything is fine. After some time (approximately 5 mins) an error appear in the right buttom corner. it states:

Cannot read property ‘timed_out’ of null

After you push the button “refresh” kibana indicates following error:

Cannot read property ‘total’ of undefined

I can’t reproduce it. Do you experience it with all indices?

After you push the button “refresh” kibana indicates following error:

Can you navigate Kibana at this point?

Do you have net::ERR_CONNECTION_REFUSED error(s) in the browser log? Show me all errors you have in the browser log. If you have any async call failed, show me the headers and the request payload of the failed call(s), get it from the network tab of the browser dev tools.

Enable debug and see what you have in the logs. Paste the logs here too.

  1. Elasticsearch.
curl -u admin:admin --insecure -X PUT "https://localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d '{
  "transient": {
    "logger.com.floragunn": "debug"
  }
}'
  1. Kibana.
    kibana.yml
searchguard.auth.debug: true

In browser console first I have:

Access to XMLHttpRequest at ‘https://sso-url.com/auth/realms/prod/protocol/openid-connect/auth?client_id=kibana01&response_type=code&redirect_uri=https%3A%2F%2Flog.kibana.com%2Fauth%2Fopenid%2Flogin&state=Ge1onj0D8PmOHPUv3nlKiN&scope=openid%20profile%20email%20address%20phone’ (redirected from ‘https://log.kibana.com/elasticsearch/filebeat-elastic-logs*/_search?rest_total_hits_as_int=true&ignore_unavailable=true&ignore_throttled=true&preference=1601548426613&timeout=30000ms’) from origin ‘https://log.kibana.com’ has been blocked by CORS policy: Response to preflight request doesn’t pass access control check: No ‘Access-Control-Allow-Origin’ header is present on the requested resource.

when I refresh the logs seen in the dashboard (by pressing the refresh button in kibana) I have

Failed to load resource: net::ERR_FAILED

kbn-ui-shared-deps.js:368 POST https://log.kibana.com/elasticsearch/filebeat-elastic-logs*/_search?rest_total_hits_as_int=true&ignore_unavailable=true&ignore_throttled=true&preference=1601548426613&timeout=30000ms 401 (Unauthorized)

Let’s debug it. Set the following properties in the log config. Restart the Elasticsearch. Try again and send me the Elasticsearch log.

elasticsearch/config/log4j2.properties

logger.sg.name = com.floragunn.dlic.auth.http.jwt
logger.sg.level = trace

To help me reproduce it, send the configs:

  1. sg_config.yml
  2. elasticsearch.yml
  3. kibana.yml
  4. Keycloak config

sg_config.yml:

basic_internal_auth_domain: 
  description: "Authenticate via HTTP Basic against internal users database"
  http_enabled: true
  transport_enabled: true
  order: 1
  http_authenticator:
    type: basic
    challenge: false
  authentication_backend:
    type: intern
openid_auth_domain:
  http_enabled: true
  transport_enabled: true
  order: 0
  http_authenticator:
    type: openid
    challenge: false
    config:
      openid_connect_url: https://keycloak-sso.com/auth/realms/prod/.well-known/openid-configuration
      subject_key: preferred_username
      roles_key: group
      skip_users:
        - kibanaro
        - kibanaserver
        - logstash
        - adminp
        - admin
        - filebeat_internal_es
        - kibanauser
        - kibana
        - readall
        - snapshotrestore 
  authentication_backend:
    type: noop        

elasticsearch.yml

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: cluster01
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node01.com
#
# Add custom attributes to the node:
#define as master eligible

node.master: True
node.data: True
node.ingest: True
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /elastic
#
# Path to log files:
#
path.logs: /logs/elastic
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: node01.com
#
# Set a custom port for HTTP:
#
http.port: 9200
#
transport.host: elastic01.azlhindp.lhind.dlh.de
transport.tcp.port: 9300
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["node01.com", "node02.com", "node03.com"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["node01.com", "node02.com", "node03.com"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------

# ---------------------------------- Security by SearchGuard -----------------------------------
#In order to use the REST management API, configure the Search Guard roles that should have access to the API. The following entry grants full access to the API for the role SGS_ALL_ACCESS
searchguard.restapi.roles_enabled: ["SGS_ALL_ACCESS"]

searchguard.ssl.transport.pemcert_filepath: cert.crt
searchguard.ssl.transport.pemkey_filepath: key.key
searchguard.ssl.transport.pemkey_password: password
searchguard.ssl.transport.pemtrustedcas_filepath: ca.crt    searchguard.ssl.transport.enforce_hostname_verification: true

#provide the list of hosts and domains who are allowed to communicate with the cluster
searchguard.nodes_dn:
- list of domains hidden

#provide the admin certificate name. This certificate must not bne used by any node. It must be dedicated certificate
searchguard.authcz.admin_dn:
  - CN=admin

searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: cert.crt
searchguard.ssl.http.pemkey_filepath: key.key
searchguard.ssl.http.pemkey_password: password
searchguard.ssl.http.pemtrustedcas_filepath:ca.crt

kibana.yml

server.name: kibana01
server.host: "0.0.0.0"
elasticsearch.hosts: "https://elastic.loadbalancer.com:9200"  
elasticsearch.username: ${ELASTICSEARCH_USERNAME}
elasticsearch.password: ${ELASTICSEARCH_PASSWORD}
xpack.security.enabled: false
searchguard.auth.type: "openid"
searchguard.openid.connect_url: https://keycloak-sso.com/auth/realms/prod/.well-known/openid-configuration
searchguard.openid.client_id: ${KEYCLOAK_CLIENT_ID}
searchguard.openid.client_secret: ${KEYCLOAK_CLIENT_SECRET}
elasticsearch.ssl:
  certificateAuthorities: /root-ca
  verificationMode: full  
searchguard.openid.base_redirect_url: https://logs.kibana.url.com

in the elasticsearch logs I can see:

[2020-10-13T19:27:44,055][INFO ][c.f.d.a.h.j.AbstractHTTPJwtAuthenticator] [node01.com] Extracting JWT token from .... failed
com.floragunn.dlic.auth.http.jwt.keybyoidc.BadCredentialsException: The token has expired

[2020-10-13T19:27:44,058][WARN ][c.f.s.h.HTTPBasicAuthenticator] [node01.com] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'

I do not have the keycloak configuration right now accessable

Is it possible to activate the desired logging without the restart? It is a productive cluster and I would not want to restart it just for changing the logging configuration

When you see the error, reload the browser. Are you redirected to the login form?
Also, find the failed call in the browser dev tools Network (see the screenshot I posted before) and show me its headers.

No I am not redirected to sso provider login form. When I reload the browser the requests are sent again as expected. when the very first error appear, the only failed request in browser network tab I have (sorry for bad formatting):

1. Request URL:

https://keycloak-sso/auth/realms/prod/protocol/openid-connect/auth?client_id=kibana01&response_type=code&redirect_uri=https%3A%2F%2Flogs.kibana.url.com%2Fauth%2Fopenid%2Flogin&state=B-GsqqDpBY5zEHCd35mWHG&scope=openid%20profile%20email%20address%20phone

  2. Referrer Policy:

strict-origin-when-cross-origin

1. Request Headers

  1. Provisional headers are shown

  2. Referer: https://logs.kibana.url.com

  3. User-Agent:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36

2. Query String Parametersview sourceview URL encoded

  1. client_id:

kibana01

  2. response_type:

code

  3. redirect_uri: https://logs.kibana.url.com/auth/openid/login

  4. state:

B-GsqqDpBY5zEHCd35mWHG

  5. scope:

openid profile email address phone

@Kosmonafft Hi, just to make sure I understand you correctly - when you reload the browser, Kibana works as expected for a little while. Is that correct?

Also, could could you please check the requests (in the network tab) that happen before the one you sent?
If you see a request with status code 302, the headers on that request would be interesting.

I suspect that the request, for some reason, is missing the headers (Content-Type: application/json) that we need to determine if it is an “ajax” request. As a result, while we do detect an expired session, we can’t catch the error and redirect to the IdP.
That would explain why it works on a full page reload (your user is redirected to the IdP and back to Kibana more or less transparently) in which case we can re-create the session.

Thanks
Mike

Thank you for helping me looking into it guys. When the described behavoir happens I see following request in the kibana log:

{"type":"response","@timestamp":"2020-10-15T09:05:57Z","tags":[],"pid":1,"method":"post","statusCode":302,"req":{"url":"/elasticsearch/metricbeat-*/_search?rest_total_hits_as_int=true&ignore_unavailable=true&ignore_throttled=true&preference=1602750775446&timeout=30000ms","method":"post","headers":{"user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36","accept":"application/json, text/plain, */*","content-type":"application/json","accept-language":"en-US,en;q=0.9,de;q=0.8,ru;q=0.7","referer":"[https://logs.kibana.url.com/app/kibana","sec-fetch-dest":"empty","origin":"https://logs.kibana.url.com","kbn-version":"7.6.2","x-cluster-client-ip":"12.34.56.78","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","accept-encoding":"gzip ](https://logs.kibana.url.com/app/kibana), deflate, br","content-length":"3476","host":"logs.kibana.url.com","x-forwarded-host":"logs.kibana.url.com","x-forwarded-port":"443","x-forwarded-proto":"https","forwarded":"for=12.34.56.78;host=logs.kibana.url.com;proto=https;proto-version=","x-forwarded-for":"12.34.56.78"},"remoteAddress":"10.10.10.10","userAgent":"10.10.10.10","referer":"[https://logs.kibana.url.com/app/kibana"},"res":{"statusCode":302,"responseTime":2,"contentLength":9},"message":"POST ](https://logs.kibana.url.com/app/kibana) /elasticsearch/metricbeat-*/_search?rest_total_hits_as_int=true&ignore_unavailable=true&ignore_throttled=true&preference=1602750775446&timeout=30000ms 302 2ms - 9.0B"}

and then this one:

{"type":"response","@timestamp":"2020-10-15T09:05:57Z","tags":[],"pid":1,"method":"get","statusCode":302,"req":{"url":"/auth/openid/login?nextUrl=%2Felasticsearch%2Fmetricbeat-*%2F_search%3Frest_total_hits_as_int%3Dtrue%26ignore_unavailable%3Dtrue%26ignore_throttled%3Dtrue%26preference%3D1602750775446%26timeout%3D30000ms","method":"get","headers":{"user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36","accept":"application/json, text/plain, */*","accept-language":"en-US,en;q=0.9,de;q=0.8,ru;q=0.7","referer":"[https://logs.kibana.url.com/app/kibana","sec-fetch-dest":"empty","kbn-version":"7.6.2","x-cluster-client-ip":"12.34.56.78","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","accept-encoding":"gzip ]([https://logs.kibana.url.com/app/kibana), deflate, br","host":"logs.kibana.url.com","x-forwarded-host":"logs.kibana.url.com","x-forwarded-port":"443","x-forwarded-proto":"https","forwarded":"for=12.34.56.78;host=logs.kibana.url.com;proto=https;proto-version=","x-forwarded-for":"12.34.56.78"},"remoteAddress":"10.20.10.20","userAgent":"10.20.10.20","referer":"[https://logs.kibana.url.com/app/kibana"},"res":{"statusCode":302,"responseTime":2,"contentLength":9},"message":"GET ](https://logs.kibana.url.com) /auth/openid/login?nextUrl=%2Felasticsearch%2Fmetricbeat-ops-devops-infra*%2F_search%3Frest_total_hits_as_int%3Dtrue%26ignore_unavailable%3Dtrue%26ignore_throttled%3Dtrue%26preference%3D1602750775446%26timeout%3D30000ms 302 2ms - 9.0B"}