Kibana is throwing internalServer error because of cache issue

Hi

We have two apps(Test1 and kibana) integrated with same keycloak.
When we login to Test1 app with the keycloak user and in another tab we open kibana it automatically logins,It is because of Single-sign-on(SSO) session. But when we logout from Test1 app and refresh kibana it shows “internal server error”. Even if we close the Chrome browser and try again kibana,it shows same internal server error.

But in the above scenario,if we logout from kibana directly it properly logs out without any issue and redirects to login page for re-login. This action also logs out from Test1 app.
Only when we logout from another app(Test1), kibana does not logout and shows “internal server error” on accessing it. Please refer the below logs when we get error

{"type":"error","@timestamp":"2021-01-27T05:11:06Z","tags":[],"pid":10,"level":"error","error":\{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n at HapiResponseAdapter.toInternalError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:69:19)\n at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:163:34)\n at process._tickCallback (internal/process/next_tick.js:68:7)"}
,"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":null,"query":{},"pathname":"/","path":"/","href":"/"},"message":"Internal Server Error"}
{"type":"response","@timestamp":"2021-01-27T05:11:06Z","tags":[],"pid":10,"method":"get","statusCode":500,"req":{"url":"/","method":"get","headers":

{"host":"abc.xyz.net","x-request-id":"79874e40e41ed0177308e34c08537b79","x-real-ip":"yy.yy.yy.yy","x-forwarded-for":"yy.yy.yy.yy","x-forwarded-host":"abc.xyz.net","x-forwarded-port":"443","x-forwarded-proto":"https","x-original-uri":"/","x-scheme":"https","cache-control":"max-age=0","sec-ch-ua":"\"Chromium\";v=\"88\", \"Google Chrome\";v=\"88\", \";Not A Brand\";v=\"99\"","sec-ch-ua-mobile":"?0","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"}
,"remoteAddress":"xx.xx.xx.xx","userAgent":"xx.xx.xx.xx"},"res":{"statusCode":500,"responseTime":111,"contentLength":9},"message":"GET / 500 111ms - 9.0B"}
 
 
Befor 500 InternalServer error, below errors shows up:-
 
{"type":"log","@timestamp":"2021-01-27T05:11:07Z","tags":["error","http"],"pid":10,"message":"{ Error: Authentication Exception\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n at IncomingMessage.emit (events.js:203:15)\n at endReadableNT (_stream_readable.js:1145:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)\n status: 401,\n displayName: 'AuthenticationException',\n message: 'Authentication Exception',\n path: '/.kibana/_doc/config%3A7.8.0',\n query: {},\n body: 'Authentication finally failed',\n statusCode: 401,\n response: 'Authentication finally failed',\n toString: [Function],\n toJSON: [Function],\n isBoom: true,\n isServer: false,\n data: null,\n output:\n { statusCode: 401,\n payload:\n

{ statusCode: 401,\n error: 'Unauthorized',\n message: 'Authentication Exception' }
,\n headers:\n { 'WWW-Authenticate': 'Basic realm=\"Authorization Required\"' } },\n reformat: [Function],\n [Symbol(ElasticsearchError)]: 'Elasticsearch/notAuthorized',\n [Symbol(SavedObjectsClientErrorCode)]: 'SavedObjectsClient/notAuthorized' }"}

Ideally when we logout from Test1 app it should also logout from kibana properly without any issues. Kibana should not shows the “internal server error”.

For re-login to kibana again we need to clear the cookies.

Could you please help regarding this issue ?

Hi.
What exactly is this Test1? Is it another instance of Kibana?
What Search Guard version do you use?
Please send the following config files: kibana.yml and sg_config.yml.

Also, show me the browser cookies in the following cases

  1. When user authenticated and has access both to Kibana and Test1.
  2. When user did logout from the Test1.

Thank you for reporting this. I add it to the bug queue. I’ll tell you when it is solved.

Thanks, @srgbnd
What exactly is this Test1? Is it another instance of Kibana?
No it is our application UI integrated with keycloak.

What Search Guard version do you use?
searchguard version : 7.8.0-43.0.0

kibana.yml

# Donot change sever name and host. This is default configuration.
server.name: kibana
server.host: 0.0.0.0
#increase the time to max.
elasticsearch.requestTimeout: 99999
server.customResponseHeaders: { "X-Frame-Options": "DENY" }
#Set it to 'true' to help prevent the browser from allowing unsafe scripting. If true, it  will block access to Kibana for any
csp.strict: false
#Enable server.ssl.supportedProtocols when SG is enabled.
server.ssl.supportedProtocols: ["TLSv1.2"]
#searchguard cookie can be secured by setting the below parameter to true. Uncomment it when SG is enabled.
#searchguard.cookie.secure: true
# Whitelist basic headers and multi tenancy header
##elasticsearch.requestHeadersWhitelist: [ "Authorization", "sgtenant", "x-forwarded-for", "x-proxy-user", "x-proxy-roles" ]
searchguard.auth.type: "openid"
searchguard.openid.connect_url: "https://xxxxxxxxxxxxxxxxxxx/openid-configuration"
searchguard.openid.base_redirect_url: "https://yyyyyyyyyyyyyyyyyyyyyyyyy.net"
searchguard.openid.client_id: "aaaaaaaa"
searchguard.openid.client_secret: "abc_secret"
searchguard.openid.header: "Authorization"
searchguard.openid.root_ca: "/etc/kibana/certs/keycloak-root-ca.pem"

sg_config.yaml

_sg_meta:
  type: "config"
  config_version: 2
sg_config:
  dynamic:
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
        internalProxies: '.+' # 
        remoteIpHeader:  'x-forwarded-for'
        #trustedProxies: '.+' # trust all external proxies, regex pattern
    authc:
      basic_internal_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 0 
        http_authenticator:
          type: basic
          challenge: False   
        authentication_backend:
          type: intern
      openid_auth_domain:
        #enabled: True      
        http_enabled:  True      
        transport_enabled: true
        order: 1
        http_authenticator:
          type: openid
          challenge: false
          config:
            subject_key: preferred_username
            roles_key: roles
            openid_connect_url: https://xxxxxxxxxxxxxxxxxxx/openid-configuration              
        authentication_backend:
          type: noop
      proxy_auth_domain:
        http_enabled: false
        transport_enabled: false
        order: 3
        http_authenticator:
          type: proxy
          challenge: false
          config:
            user_header: "x-proxy-user"
            #roles_header: "x-proxy-roles"
        authentication_backend:
          type: noop
          config: {}	
when login Test1 UI by passing credentials
Test1 cookies:-

Name
connect.sid
Content
s%3AOXmSlqXbzA6P8CPWhlP7mkw1m4RMdeZH.h9EQo7DO8ecbrnqurnXMuJ8cTGWPm1j8xpXxADOxr5I
Domain
yyyyyyyyyyyyyyyyyyyyyyyyyyyyy.net
Path
/
Send for
Secure same-site connections only
Accessible to script
No (HttpOnly)
Created
Wednesday, February 3, 2021 at 11:59:55 AM
Expires
When the browsing session ends

When open kibana UI in another tab
kibana cookies:-

searchguard_authentication
Name
searchguard_authentication
Content
Fe26.2**1381f0dc0ed2b5eef88f6c33845b32028d574ca05c37d86c64749f9d0e896fdd*8QqmO1cde7RIw2RdQXAnlQ*EQ2f_--C73Rxbf1SgsbJpOYTtOnKNCeUfQ1gnhgtK3PEVKnR_GtcyhUE_RxoYVpEKvhABSoNPBNK0Vd2WVM9XKZnqVyWye-XTByBVJtVnaoZr0Sdfbwz5MLqyuy1AqcGKlnAYQcgN8ZnRTLYD0C9ULt32pPMom2jK-l6k86GfdbQo5ynHmDc_aTIKXpPJcHthcdfKlHfOrEFLPIF3rSiSY7u4t6mmSwfw9gBXznaMSv7WlX9-Xkvs-K2bHq3WM8QTsst6MPEIn9M6gd2JPWgiEnx_g2DjbkwBusmyqv9WU-AzBoZ3i2aOhE8cjcb4R7tna1ie0x4PzRoD6oQgn7plzDOwOq3vpevnZiv1Tz2Lajkg7OEjX9n_d1GoCPwTZmQDUDMl4jvxRg_-B4WQwMzU-y6a4FfqRZrr-dIYtt3QfzgPNIjA9Q7DMlfwzHieN7igJa5z11ur_u7h8Xgnu3MZS-ySA54kWBDrOu-6cDOSibhNNxvhG4NEoSnOk6GKXCCGSyIu-IXx8QBRhuuzuLXR8C3Fz-PjyDbG7YvoWsnon4WXQ3BzaLTI5AtRZjDYzI-sdDaV3UNkM1G5wUQVGaRCwf3hySv1cokMa0vDx9s98L-rB_B4Xxy-k2dSLS7gFL13RMuKW6hlv3j0HmS4OLtDv-b_e4IIayKdbEt01Nq4h6Auw7qc9tmy9Z65SjCE0R4qiOtu7rlOTkiTPWC0f_zUb4F9H1T3u3iAa175R2lqURbYBybBlPgdh5UXEJD92vdt1R_RkhuP1X3cZGWO2K8lyzY3eMBEyFJVA2In3RTA4qiwE-BlleesxSViWDbOGdVma9WGDM0157cACSOpCkwEQynMNpOQTFQxMQTROE-JOmEplNZ76LD26l9FLm8c4K1LG6v5MH0fF1N6ilLdaPwU1bZ-OYfTn08wxLx-ye81F_MKVMRY1vV-uyIDD-e1PU1H4ZUGZv8f8wuOmw1th27klH-nDEdfkQEpafMR6HO0Mg-Vq7jDl97_kZ_SyOJ-lvgpmepohmD459N-jWDhM_GYqqw2_K7mVt4It7c0mQInhvlODJUXb3b-_r49OSiWvlZneJTGFKahGFkqQQTM5RAXjV2369muG7fB00Xdr98cm-i3kaYWBEjSeAOLBTKDIKcxk_TwWc8sA2oBCdZfZNnPjEE0Uw0evDNqJbYIWj2t2FUMFCsnwf4ocwYN5E_YrrfGl0MFxeA34ORq3tm6EEnhGKtuwcxa7z2uW0jdAC0J2GL2Ud2XjtGODvFUmjlcDT9qQ_-C2wxxtF-73dGBVgCFZfHIWJyM6qQBlV6PjneWgM4TNg0FsFaxJtBq3TcJ7mY8U5bx9TqzbMW0JImc3j6GkMVBFtv12ems13bI-U60fajGm11O6tbVy5BXk-wlXf8_HEJzRZU1Lu0K1xepLCdoxhIcziCyB8DgY_vkkNiUPTQPQd_gkFj2t45bPtH5atilH5RuDDZ0DtkzPHrN60jT0u6hHtc7gcrSRTWt4XrI0KRpvOdbJXGF0fQddWFpF4ofv-U_unrVHKokIgPuFBY9JA2VZFeJcReVgRuAD0uQCgp_ZwIA_lEtTlshFUnQPlBunlCHNGZNGh8wLeNcty8e8Df5fLCL19mC3OeWfhjDvVNYG0TMkfeYCdHKg5HLLSfVh8ssk8dZ2c7IpWmhxRLO9YrfD5yOF9fUZpx43Mc7YmfzFGuwgNs0fX76_rOLbk0Tu39j5uyu3dNTt7naXD9EsXfBneSoIMnZIyG7k1RyujGTQ_IY4x3hT2RFctAEm3nRjAg2il_vWw_gZ8LjRoTfndENYaKRGr_lZr-cEpPBSYirm1AJIsPzQxEQ9jJbeArgboC8dO69_5DsBapVH_jd2QIytZErxYsm7-5_Yoqe7nzvFgW3p-mhKRyQ103ap9Gp3X84g5jxCG3hGOSypzbXp8vMs0ZDQKMZKtMeCQ0YloUhV50IY1cVBuqSKocVkCXVMRya03MpREjI5GJZsWLeOlYMhoqoiEjD__2Q4Pp7JZ-jHHtC036cTsOXph9yb5_M1kkyxr6v3D98v-Nn1zYPN4Fgdz9tbHPDKVi37IzwzaXEiCVwzSvCcYX-18H80JztAg0PUAdmxQ_1eLGvKj3Dt0VpwWs_tSI37yAwJijL625L76TuMNdDrekLhavyO8T_6lzFOzb_C5ziiyHHVd0DgmoeEecZI6J-VWt6rMyw_jw7M7RBbHmVYewg6IQCuwBZohYhsh9DPI0LItrOWaRQ7_sdabVSKRCUHWoq5LLCoJoCtMubpSuTdi75kvOEcuFRdsy2U2KSclhC3YPdImrF9CiGMV22vGeWnJnkQPPJ2FQMUB4pESIVbo1I8Q40OmAqCF3yLoBwwZ66t_USHgDbJ0ra6BDIlbGzEJd8Yn7uDb0U61Pt5rNn3RAwVTLc5Zh9NA-5icEt8P0xxI-oQ2qu-eapWXKZGK0xE0FijLXD_fwHvo120ozJlQX7JFUiZplQnIHkkcafaGd9Lx7iQx7-70dXrBaBBtTEwa6Aw0Sqb4_3iHmi9VEYp9kigaYcR9AZvTO6Myfh25_fnaae024manZqN6apsiNBCpQhN5DmY8ImpwlJX3RMGiyu609rY8OyV2YGai-iTgTLThGUEhkHfqLMQ6cK6WKvA-fAc_W1YnthNqB-GqFwlxhDXO2MRO6sLm5C9YnJiHn9UIsoC_Jka7xDXKRitSz5GTT1nRnlZ8v4TcN7dhGmnvLZZG8Xoiq2Bq2CACf2wDiRLevIoLzjVgW6nQkpJOSdqLruXLyCrslXFcpJRIVom3hDadvF6nXB6snAH9lxz5HiuumBwMTNMT3fxRpGyaGclESOBqM9KbQBgclkftJ8ZJkFxfsmRN_2R-mjEHXLrqpqNxv-hxRsg**5920a5667892c18a85030198fc2e6eedff534951c2e395041202a495746c0c1c*Htff7lNXACt_tV-0ubD1UMHaw5rvxsuacp3GW41tO1I
Domain
xxxxxxxxxxxxxxxxxxxxxxxx.net
Path
/
Send for
Same-site connections only
Accessible to script
No (HttpOnly)
Created
Wednesday, February 3, 2021 at 12:00:08 PM
Expires
Wednesday, February 3, 2021 at 1:00:08 PM

searchguard_storage
Name
searchguard_storage
Content
Fe26.2**e48f3cddf20e4fee9423f768c24a15ed2bd079955272b013b2402164ca14d40c*TuVLZxRIqOy2FrdYousjsg*HR5IHLhxN5B6KSnhQDsLN3KCPvYu920zSIekZDjkukbM_HcCeeO73EI5hcxdOf2rmZwJTWyoZ_5d11QDMqyin9ZHOpQDa8nyDvLGAgjz_i99LpSTNYWfAebHW59rhWM3I53CnvwFk6dGPTDHS7HfaQg1dVhj1IERL0hJVZYyb1C-_9zcddF5hBLU2AXQ5PStnoshQc01YiMZ9pHI5lYNh57CBkdIqDOq1HGR3G1jOs7OEkaQOSssMlegkBTSzEBLifiSRe-XS-3mW8o9CHZtOF3JLpTgaxjrfqXYTZomiBGjyglANvLL6J1-af3V8OpJck-zpjABlK1jSd9u_oZ1OcTCSLCXwc6pYe4d-lG_chETMsMNM2jQTfpdYr8pbKYD13pnUiVXHpYpiiQfE806Trjlo_s2STlqwpmpOJqKbLzGu3a4yslIG0w2h6zXGKjP8fSNSRUdgrYBbsatpw1NH_tBGveUFhTUJjgOjIW9AKfv1--Xpy2PIcIsKpnQBu9ff-wL-eSeK9Ltll8LhC9-wG6I8bd_04POhl0XQPtRZsQIAzNnJ2pbJAwz9D2rIf_JiZ2IKdfUfsDCPMTz4JPsy8Md3hk0nOAPv7EJ8gcq4DzrcXH2eJPg0kHMFnJcO06q_jp3nW_jsD6jIufIQJjhFg**c195f92fed804277606cbef647130a6ce982731e578a15692454db36dfcf78f0*PnhVrGDrb4eK6mLGfJ-N76CnoU7A41HeSLIx7gVqQBU
Domain
xxxxxxxxxxxxxxxxxxxxxxxx.net
Path
/
Send for
Same-site connections only
Accessible to script
No (HttpOnly)
Created
Wednesday, February 3, 2021 at 12:00:08 PM
Expires
When the browsing session ends

Local storage
Origin
https://xxxxxxxxxxxxxxxxxxxxxxxx.net
Size on disk
145 B
Last modified
Wednesday, February 3, 2021 at 12:00:40 PM 
Cookies when user did logout from the Test1

when Test1 logout
Test1 cookies:- 

connect.sid
Name
connect.sid
Content
s%3A6XD6S_bvS5Q48XnWHLhzGDFHC2Y-Dw9t.JeugV96NwYvUpB0sXtIkNBAIl0b583Ey0wcMWMTfOm0
Domain
yyyyyyyyyyyyyyyyyyyyyyyyyyyyy.net
Path
/
Send for
Secure same-site connections only
Accessible to script
No (HttpOnly)
Created
Wednesday, February 3, 2021 at 12:12:52 PM
Expires
When the browsing session ends

Local storage
Origin
https://yyyyyyyyyyyyyyyyyyyyyyyyyyyyy.net
Size on disk
249 B

When we refresh the kibana page after logout from Test1 UI. Kibana shows the pop up asking for credentials. Please refer the attachment


When we try to relogin in kibana page it shows internal server error. After this we got stuck and we have to clear the cookies to make kibana work.