TypeError: Cannot read property 'timed_out' of null and TypeError: Cannot read property 'total' of undefined

Kosmonafft · September 25, 2020, 7:52am

Hi guys,

We are using kibana, search guard in combination with keycloak (openid). While reading indeces in the kibana dashboards I have following errors appearing in the kibana UI:

TypeError: Cannot read property 'timed_out' of null
    at handleResponse (https://log.kibana.com/bundles/commons.bundle.js:11:4099490)
    at https://log.kibana.com/bundles/commons.bundle.js:11:4092562

and after this one:

TypeError: Cannot read property 'total' of undefined
    at tabifyAggResponse (https://log.kibana.com/bundles/commons.bundle.js:11:1527391)
    at onResults (https://log.kibana.com/bundles/58.bundle.js:3:32315)
    at https://log.kibana.com/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:94842
    at https://llog.kibana.com/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:94980
    at u.$digest (https://log.kibana.com/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:100155)
    at https://log.kibana.com/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:102127
    at Yo.completeTask (https://log.kibana.com/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:122692)
    at https://log.kibana.com/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:34257

you have an idea why it happens?

srgbnd · September 25, 2020, 10:26am

Hi.
Look in the Elasticsearch log. You should have error(s) there if it is the permission related errors. Paste the errors here. What version of SG do you use?

Kosmonafft · September 25, 2020, 1:45pm

Thanks for the response.

There are no logs in elastic when it happens. log level is set to info

Searchguard version is: 41.0.0
elasitc is. 7.6.2

In kibana logs (runnin in openshift) I can see responses with 401:

{
“type”:“response”,
“@timestamp”:“2020-09-25T13:38:52Z”,
“tags”:[

],
“pid”:1,
“method”:“post”,
“statusCode”:401,
“req”:{
“url”:“/elastic/filebeat-prod-/_search?rest_total_hits_as_int=true&ignore_unavailable=true&ignore_throttled=true&preference=1601040743379&timeout=30000ms",
“method”:“post”,
“headers”:{
“user-agent”:“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36”,
“accept”:“application/json, text/plain, /”,
“content-type”:“application/json”,
“accept-language”:“en-US,en;q=0.9,de;q=0.8,ru;q=0.7”,
“referer”:“https://log.kibana.com/app/kibana",
“sec-fetch-dest”:“empty”,
“origin”:“https://log.kibana.com”,
“kbn-version”:“7.6.2”,
“x-cluster-client-ip”:“1.2.3.4”,
“sec-fetch-site”:“same-origin”,
“sec-fetch-mode”:“cors”,
“accept-encoding”:"gzip , deflate, br”,
“content-length”:“6021”,
“host”:“log.kibana.com”,
“x-forwarded-host”:“log.kibana.com”,
“x-forwarded-port”:“443”,
“x-forwarded-proto”:“https”,
“forwarded”:“for=1.2.3.4;host=log.kibana.com;proto=https;proto-version=”,
“x-forwarded-for”:“1.2.3.4”
},
“remoteAddress”:“4.3.2.1”,
“userAgent”:“4.3.2.1”,
“referer”:"https://log.kibana.com/app/kibana"
},
“res”:{
“statusCode”:401,
“responseTime”:23,
“contentLength”:9
},
“message”:"POST /elastic/filebeat-prod-/_search?rest_total_hits_as_int=true&ignore_unavailable=true&ignore_throttled=true&preference=1601040743379&timeout=30000ms 401 23ms - 9.0B”
}

It looks like during my session I lose my authentication somehow.

srgbnd · September 28, 2020, 10:24am

While reading indeces …

What do you mean? Can you post a screenshot?

Do you have the request payload for this call? You can check the browser dev tools for it. I wonder what would happen if you attach the payload to a curl request. Try to do it without Kibana to narrow down the scope of the issue.

Kosmonafft · September 28, 2020, 11:29am

When I open a fresh kibana session and begin to use the dashboard everything is fine. After some time (approximately 5 mins) an error appear in the right buttom corner. it states:

Cannot read property ‘timed_out’ of null

After you push the button “refresh” kibana indicates following error:

Cannot read property ‘total’ of undefined

srgbnd · September 29, 2020, 9:50am

I can’t reproduce it. Do you experience it with all indices?

After you push the button “refresh” kibana indicates following error:

Can you navigate Kibana at this point?

Do you have net::ERR_CONNECTION_REFUSED error(s) in the browser log? Show me all errors you have in the browser log. If you have any async call failed, show me the headers and the request payload of the failed call(s), get it from the network tab of the browser dev tools.

Enable debug and see what you have in the logs. Paste the logs here too.

Elasticsearch.

curl -u admin:admin --insecure -X PUT "https://localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d '{
  "transient": {
    "logger.com.floragunn": "debug"
  }
}'

Kibana.
kibana.yml

searchguard.auth.debug: true

Kosmonafft · October 1, 2020, 10:46am

In browser console first I have:

Access to XMLHttpRequest at ‘https://sso-url.com/auth/realms/prod/protocol/openid-connect/auth?client_id=kibana01&response_type=code&redirect_uri=https%3A%2F%2Flog.kibana.com%2Fauth%2Fopenid%2Flogin&state=Ge1onj0D8PmOHPUv3nlKiN&scope=openid%20profile%20email%20address%20phone’ (redirected from ‘https://log.kibana.com/elasticsearch/filebeat-elastic-logs*/_search?rest_total_hits_as_int=true&ignore_unavailable=true&ignore_throttled=true&preference=1601548426613&timeout=30000ms’) from origin ‘https://log.kibana.com’ has been blocked by CORS policy: Response to preflight request doesn’t pass access control check: No ‘Access-Control-Allow-Origin’ header is present on the requested resource.

when I refresh the logs seen in the dashboard (by pressing the refresh button in kibana) I have

Failed to load resource: net::ERR_FAILED

kbn-ui-shared-deps.js:368 POST https://log.kibana.com/elasticsearch/filebeat-elastic-logs*/_search?rest_total_hits_as_int=true&ignore_unavailable=true&ignore_throttled=true&preference=1601548426613&timeout=30000ms 401 (Unauthorized)

srgbnd · October 2, 2020, 7:58am

Let’s debug it. Set the following properties in the log config. Restart the Elasticsearch. Try again and send me the Elasticsearch log.

elasticsearch/config/log4j2.properties

logger.sg.name = com.floragunn.dlic.auth.http.jwt
logger.sg.level = trace

To help me reproduce it, send the configs:

sg_config.yml
elasticsearch.yml
kibana.yml
Keycloak config

Kosmonafft · October 13, 2020, 5:42pm

sg_config.yml:

basic_internal_auth_domain: 
  description: "Authenticate via HTTP Basic against internal users database"
  http_enabled: true
  transport_enabled: true
  order: 1
  http_authenticator:
    type: basic
    challenge: false
  authentication_backend:
    type: intern
openid_auth_domain:
  http_enabled: true
  transport_enabled: true
  order: 0
  http_authenticator:
    type: openid
    challenge: false
    config:
      openid_connect_url: https://keycloak-sso.com/auth/realms/prod/.well-known/openid-configuration
      subject_key: preferred_username
      roles_key: group
      skip_users:
        - kibanaro
        - kibanaserver
        - logstash
        - adminp
        - admin
        - filebeat_internal_es
        - kibanauser
        - kibana
        - readall
        - snapshotrestore 
  authentication_backend:
    type: noop

elasticsearch.yml

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: cluster01
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node01.com
#
# Add custom attributes to the node:
#define as master eligible

node.master: True
node.data: True
node.ingest: True
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /elastic
#
# Path to log files:
#
path.logs: /logs/elastic
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: node01.com
#
# Set a custom port for HTTP:
#
http.port: 9200
#
transport.host: elastic01.azlhindp.lhind.dlh.de
transport.tcp.port: 9300
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["node01.com", "node02.com", "node03.com"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["node01.com", "node02.com", "node03.com"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------

# ---------------------------------- Security by SearchGuard -----------------------------------
#In order to use the REST management API, configure the Search Guard roles that should have access to the API. The following entry grants full access to the API for the role SGS_ALL_ACCESS
searchguard.restapi.roles_enabled: ["SGS_ALL_ACCESS"]

searchguard.ssl.transport.pemcert_filepath: cert.crt
searchguard.ssl.transport.pemkey_filepath: key.key
searchguard.ssl.transport.pemkey_password: password
searchguard.ssl.transport.pemtrustedcas_filepath: ca.crt    searchguard.ssl.transport.enforce_hostname_verification: true

#provide the list of hosts and domains who are allowed to communicate with the cluster
searchguard.nodes_dn:
- list of domains hidden

#provide the admin certificate name. This certificate must not bne used by any node. It must be dedicated certificate
searchguard.authcz.admin_dn:
  - CN=admin

searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: cert.crt
searchguard.ssl.http.pemkey_filepath: key.key
searchguard.ssl.http.pemkey_password: password
searchguard.ssl.http.pemtrustedcas_filepath:ca.crt

kibana.yml

server.name: kibana01
server.host: "0.0.0.0"
elasticsearch.hosts: "https://elastic.loadbalancer.com:9200"  
elasticsearch.username: ${ELASTICSEARCH_USERNAME}
elasticsearch.password: ${ELASTICSEARCH_PASSWORD}
xpack.security.enabled: false
searchguard.auth.type: "openid"
searchguard.openid.connect_url: https://keycloak-sso.com/auth/realms/prod/.well-known/openid-configuration
searchguard.openid.client_id: ${KEYCLOAK_CLIENT_ID}
searchguard.openid.client_secret: ${KEYCLOAK_CLIENT_SECRET}
elasticsearch.ssl:
  certificateAuthorities: /root-ca
  verificationMode: full  
searchguard.openid.base_redirect_url: https://logs.kibana.url.com

in the elasticsearch logs I can see:

[2020-10-13T19:27:44,055][INFO ][c.f.d.a.h.j.AbstractHTTPJwtAuthenticator] [node01.com] Extracting JWT token from .... failed
com.floragunn.dlic.auth.http.jwt.keybyoidc.BadCredentialsException: The token has expired

[2020-10-13T19:27:44,058][WARN ][c.f.s.h.HTTPBasicAuthenticator] [node01.com] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'

I do not have the keycloak configuration right now accessable

Is it possible to activate the desired logging without the restart? It is a productive cluster and I would not want to restart it just for changing the logging configuration

srgbnd · October 14, 2020, 9:16am

When you see the error, reload the browser. Are you redirected to the login form?
Also, find the failed call in the browser dev tools Network (see the screenshot I posted before) and show me its headers.

Kosmonafft · October 15, 2020, 8:05am

No I am not redirected to sso provider login form. When I reload the browser the requests are sent again as expected. when the very first error appear, the only failed request in browser network tab I have (sorry for bad formatting):

1. Request URL:

https://keycloak-sso/auth/realms/prod/protocol/openid-connect/auth?client_id=kibana01&response_type=code&redirect_uri=https%3A%2F%2Flogs.kibana.url.com%2Fauth%2Fopenid%2Flogin&state=B-GsqqDpBY5zEHCd35mWHG&scope=openid%20profile%20email%20address%20phone

  2. Referrer Policy:

strict-origin-when-cross-origin

1. Request Headers

  1. Provisional headers are shown

  2. Referer: https://logs.kibana.url.com

  3. User-Agent:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36

2. Query String Parametersview sourceview URL encoded

  1. client_id:

kibana01

  2. response_type:

code

  3. redirect_uri: https://logs.kibana.url.com/auth/openid/login

  4. state:

B-GsqqDpBY5zEHCd35mWHG

  5. scope:

openid profile email address phone

Mike · October 15, 2020, 8:57am

@Kosmonafft Hi, just to make sure I understand you correctly - when you reload the browser, Kibana works as expected for a little while. Is that correct?

Also, could could you please check the requests (in the network tab) that happen before the one you sent?
If you see a request with status code 302, the headers on that request would be interesting.

I suspect that the request, for some reason, is missing the headers (Content-Type: application/json) that we need to determine if it is an “ajax” request. As a result, while we do detect an expired session, we can’t catch the error and redirect to the IdP.
That would explain why it works on a full page reload (your user is redirected to the IdP and back to Kibana more or less transparently) in which case we can re-create the session.

Thanks
Mike

Kosmonafft · October 15, 2020, 9:10am

Thank you for helping me looking into it guys. When the described behavoir happens I see following request in the kibana log:

{"type":"response","@timestamp":"2020-10-15T09:05:57Z","tags":[],"pid":1,"method":"post","statusCode":302,"req":{"url":"/elasticsearch/metricbeat-*/_search?rest_total_hits_as_int=true&ignore_unavailable=true&ignore_throttled=true&preference=1602750775446&timeout=30000ms","method":"post","headers":{"user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36","accept":"application/json, text/plain, */*","content-type":"application/json","accept-language":"en-US,en;q=0.9,de;q=0.8,ru;q=0.7","referer":"[https://logs.kibana.url.com/app/kibana","sec-fetch-dest":"empty","origin":"https://logs.kibana.url.com","kbn-version":"7.6.2","x-cluster-client-ip":"12.34.56.78","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","accept-encoding":"gzip ](https://logs.kibana.url.com/app/kibana), deflate, br","content-length":"3476","host":"logs.kibana.url.com","x-forwarded-host":"logs.kibana.url.com","x-forwarded-port":"443","x-forwarded-proto":"https","forwarded":"for=12.34.56.78;host=logs.kibana.url.com;proto=https;proto-version=","x-forwarded-for":"12.34.56.78"},"remoteAddress":"10.10.10.10","userAgent":"10.10.10.10","referer":"[https://logs.kibana.url.com/app/kibana"},"res":{"statusCode":302,"responseTime":2,"contentLength":9},"message":"POST ](https://logs.kibana.url.com/app/kibana) /elasticsearch/metricbeat-*/_search?rest_total_hits_as_int=true&ignore_unavailable=true&ignore_throttled=true&preference=1602750775446&timeout=30000ms 302 2ms - 9.0B"}

and then this one:

{"type":"response","@timestamp":"2020-10-15T09:05:57Z","tags":[],"pid":1,"method":"get","statusCode":302,"req":{"url":"/auth/openid/login?nextUrl=%2Felasticsearch%2Fmetricbeat-*%2F_search%3Frest_total_hits_as_int%3Dtrue%26ignore_unavailable%3Dtrue%26ignore_throttled%3Dtrue%26preference%3D1602750775446%26timeout%3D30000ms","method":"get","headers":{"user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36","accept":"application/json, text/plain, */*","accept-language":"en-US,en;q=0.9,de;q=0.8,ru;q=0.7","referer":"[https://logs.kibana.url.com/app/kibana","sec-fetch-dest":"empty","kbn-version":"7.6.2","x-cluster-client-ip":"12.34.56.78","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","accept-encoding":"gzip ]([https://logs.kibana.url.com/app/kibana), deflate, br","host":"logs.kibana.url.com","x-forwarded-host":"logs.kibana.url.com","x-forwarded-port":"443","x-forwarded-proto":"https","forwarded":"for=12.34.56.78;host=logs.kibana.url.com;proto=https;proto-version=","x-forwarded-for":"12.34.56.78"},"remoteAddress":"10.20.10.20","userAgent":"10.20.10.20","referer":"[https://logs.kibana.url.com/app/kibana"},"res":{"statusCode":302,"responseTime":2,"contentLength":9},"message":"GET ](https://logs.kibana.url.com) /auth/openid/login?nextUrl=%2Felasticsearch%2Fmetricbeat-ops-devops-infra*%2F_search%3Frest_total_hits_as_int%3Dtrue%26ignore_unavailable%3Dtrue%26ignore_throttled%3Dtrue%26preference%3D1602750775446%26timeout%3D30000ms 302 2ms - 9.0B"}

srgbnd · October 26, 2020, 5:39pm

Hi @Kosmonafft
I can reproduce the errors on the session timeout. But the user is redirected to the Keycloak login page as it is expected.

Look:

What token timeouts do you have in the Keycloak? You can increase the session timeout by changing the highlighted properties.

Kosmonafft · October 28, 2020, 10:25am

The timeouts are set to 30 minutes for both, idle and max. But my session gets cut much quicker, after like 2-5 minutes or so. So it is not related to session timeouts on keycloak side I believe. Also my users losing their session even if they are actively doing something without long interruptions (like browsing logs, creating visualizations etc.) Maybe the loadbalancer is part of the problem. My kibana instance is running on openshift and it gets accessed by clients using a loadbalancer url.

srgbnd · October 28, 2020, 11:38am

Maybe the loadbalancer is part of the problem.

It can be the case. Check its settings related to the session persistence. Also, check its logs.

Kosmonafft · October 28, 2020, 4:47pm

btw when I set searchguard.auth.debug: true in kibana.yml as suggested by you I have following error during start up of kibana:

FATAL ValidationError: child "searchguard" fails because [child "auth" fails because ["debug" is not allowed]]

srgbnd · October 28, 2020, 4:59pm

The debug option was added in v7.7.0-42.0.0 index.js · v7.7.0-42.0.0 · search-guard / Search Guard Kibana Plugin · GitLab

Kosmonafft · November 2, 2020, 11:20am

Ok i could sort of fix the problem. The problem was:

I had two kibana pods running (mainly because kibana takes ridiculously long to build, so to avoid long downtimes i had two pods running). But I had not configured kibana instances properly to handle the sessions between each other. So when user switched from one instance to another he probably lost his session. Still unclear to me is: Why i had no keycloak login mask presented. Anyhow I scaled kibana down to 1 pod and now it works without any issues so far.

srgbnd · November 2, 2020, 12:28pm

Glad to hear you were able to figure it out.

kibana takes ridiculously long to build

Do you mean it takes a lot of time to start Kibana due to the optimization process? Or you build Kibana from the source code? Kibana v7.10.0 (not release yet) distribution should take less time to start because it doesn’t build the Webpack bundles and doesn’t transpile the server code for the plugins anymore. The plugins must create the bundles, transpile the code, and include all this stuff in the plugin archive.

Why i had no keycloak login mask presented.

It didn’t happen to me with one node. I’ll try with two nodes.

Topic		Replies	Views
Kibana4.1 error: Courier Fetch: Cannot read property 'timed_out' of undefined ( Search Guard) Search Guard	5	1093	September 21, 2015
Cannot read property 'call' of undefined Search Guard	7	1701	October 2, 2019
Keycloak Authentication and Searchguard problems Search Guard	5	2280	December 13, 2018
TypeError: Cannot read property 'remoteAddress' of undefined Search Guard	2	746	March 18, 2021
Timeout while trying to modify roles in Kibana App Search Guard	15	1009	May 14, 2020

TypeError: Cannot read property 'timed_out' of null and TypeError: Cannot read property 'total' of undefined

Related topics