Kibanaserver user unauthorized after upgrade to v6.4.2-23

Search Guard
1

Hi Searchguard folks,

I build a customised version of Kibana (the ‘-oss’ variety) to include our own plugins and perform some customisations to the CSS. No mods are made to the main Kibana code.

Since upgrading to v6.4.2 (from v6.2.4), I’ve been having a devil of a time getting Kibana to start properly. I’m getting the following error at start time:

{“type”:“log”,“@timestamp”:“2018-10-21T01:16:32Z”,“tags”:[“status”,“plugin:elasticsearch@6.4.2”,“error”],“pid”:4973,“state”:“red”,“message”:“Status changed from yellow to red - Authentication Exception”,“prevState”:“yellow”,“prevMsg”:“Waiting for Elasticsearch”}

{“type”:“log”,“@timestamp”:“2018-10-21T01:16:32Z”,“tags”:[“warning”],“pid”:4973,“message”:“Detected an unhandled Promise rejection.\nAuthentication Exception :: {"path":"/myindex/_search","query":{"q":"type=acl","size":100},"statusCode":401,"response":"Unauthorized","wwwAuthenticateDirective":"Basic realm=\"Search Guard\""}”}

I assumed that it would be a deny happening on the Elasticsearch side (from Searchguard), but I’m not seeing any of the usual Searchguard deny logs in the elasticsearch.log file. From my reading of it, it looks like the “kibanaserver” user doesn’t have access to log in, does that sound correct to you? The funny thing is that I can perform that exact query on the commandline using the kibanaserver user’s credentials. I have disabled x-pack in the elasticsearch.yml. There’s no need to disable x-pack on the kibana side because I’m using the ‘oss’ build (plus if I leave the line in to disable x-pack, it bombs out).

Have you seen errors like these being spat out by Kibana before?

Many thanks for your time.

Cheers,

Nick

2

Maybe this poster is having the same issue? Maybe not.

https://groups.google.com/forum/#!topic/search-guard/nOIG6hadWGc

···

On Sunday, October 21, 2018 at 3:41:44 PM UTC+11, Nicholas George wrote:

Hi Searchguard folks,

I build a customised version of Kibana (the ‘-oss’ variety) to include our own plugins and perform some customisations to the CSS. No mods are made to the main Kibana code.

Since upgrading to v6.4.2 (from v6.2.4), I’ve been having a devil of a time getting Kibana to start properly. I’m getting the following error at start time:

{“type”:“log”,“@timestamp”:“2018-10-21T01:16:32Z”,“tags”:[“status”,“plugin:elasticsearch@6.4.2”,“error”],“pid”:4973,“state”:“red”,“message”:“Status changed from yellow to red - Authentication Exception”,“prevState”:“yellow”,“prevMsg”:“Waiting for Elasticsearch”}

{“type”:“log”,“@timestamp”:“2018-10-21T01:16:32Z”,“tags”:[“warning”],“pid”:4973,“message”:“Detected an unhandled Promise rejection.\nAuthentication Exception :: {"path":"/myindex/_search","query":{"q":"type=acl","size":100},"statusCode":401,"response":"Unauthorized","wwwAuthenticateDirective":"Basic realm=\"Search Guard\""}”}

I assumed that it would be a deny happening on the Elasticsearch side (from Searchguard), but I’m not seeing any of the usual Searchguard deny logs in the elasticsearch.log file. From my reading of it, it looks like the “kibanaserver” user doesn’t have access to log in, does that sound correct to you? The funny thing is that I can perform that exact query on the commandline using the kibanaserver user’s credentials. I have disabled x-pack in the elasticsearch.yml. There’s no need to disable x-pack on the kibana side because I’m using the ‘oss’ build (plus if I leave the line in to disable x-pack, it bombs out).

Have you seen errors like these being spat out by Kibana before?

Many thanks for your time.

Cheers,

Nick

3

Hi there,

a couple of questions first :wink:

  • which version of the Kibana plugin did you use before when on v6.2.4?

  • If I understand you correctly, the exception happens when starting Kibana, is this correct?

  • Do you see the exception also when your own plugin is not installed?

I don’t know the inner workings of your plugin, but here’s my take on the error messages:

If the exception happens on startup of Kibana, this would imply that the call to /myindex/_search is triggered by your plugin. The kibanaserver user needs access to the .kibana index, but apart from that Kibana is not performing any _search operation on startup. That’s why I’m guessing the call stems from your plugin.

If I’m correct so far, the call to /myindex/_search does not seem to have any HTTP Basic authentication credentials attached to it. So Search Guard responds (correctly) with a 401, prompting to provide some auth credentials. That’s also why you do not see any “denied” log entry in Elasticsearch: Returning a 401 is just part of HTTP Basic authentication flow to get the user credentials. It’s not some missing privileges, because at that point SG has not retrieved any credentials to work with yet.

I don’t know what your own plugin is doing exactly, but maybe this helps to point you in the right direction. If you have any more information, please let me know!

···

On Sunday, October 21, 2018 at 6:41:44 AM UTC+2, Nicholas George wrote:

Hi Searchguard folks,

I build a customised version of Kibana (the ‘-oss’ variety) to include our own plugins and perform some customisations to the CSS. No mods are made to the main Kibana code.

Since upgrading to v6.4.2 (from v6.2.4), I’ve been having a devil of a time getting Kibana to start properly. I’m getting the following error at start time:

{“type”:“log”,“@timestamp”:“2018-10-21T01:16:32Z”,“tags”:[“status”,“plugin:elasticsearch@6.4.2”,“error”],“pid”:4973,“state”:“red”,“message”:“Status changed from yellow to red - Authentication Exception”,“prevState”:“yellow”,“prevMsg”:“Waiting for Elasticsearch”}

{“type”:“log”,“@timestamp”:“2018-10-21T01:16:32Z”,“tags”:[“warning”],“pid”:4973,“message”:“Detected an unhandled Promise rejection.\nAuthentication Exception :: {"path":"/myindex/_search","query":{"q":"type=acl","size":100},"statusCode":401,"response":"Unauthorized","wwwAuthenticateDirective":"Basic realm=\"Search Guard\""}”}

I assumed that it would be a deny happening on the Elasticsearch side (from Searchguard), but I’m not seeing any of the usual Searchguard deny logs in the elasticsearch.log file. From my reading of it, it looks like the “kibanaserver” user doesn’t have access to log in, does that sound correct to you? The funny thing is that I can perform that exact query on the commandline using the kibanaserver user’s credentials. I have disabled x-pack in the elasticsearch.yml. There’s no need to disable x-pack on the kibana side because I’m using the ‘oss’ build (plus if I leave the line in to disable x-pack, it bombs out).

Have you seen errors like these being spat out by Kibana before?

Many thanks for your time.

Cheers,

Nick

4

Hi Jochen,

Thanks for your excellent insight.

To answer your questions:

  • Prior plugin version (23 for Elasticsearch and 13 for the Kibana plugin)

  • Yes, I get the exception when starting Kibana

  • Unfortunately, I won’t be able to answer this or anything else for a few days. I can no longer authenticate (via HTTP) with ES at all. I need to figure out what is going here.

I am unable to work on this for a couple of days, but will attempt to fully troubleshoot the issue on Friday.

I’ll be back!

Nick

···

On Tuesday, October 23, 2018 at 6:07:48 AM UTC+11, Jochen Kressin wrote:

Hi there,

a couple of questions first :wink:

  • which version of the Kibana plugin did you use before when on v6.2.4?
  • If I understand you correctly, the exception happens when starting Kibana, is this correct?
  • Do you see the exception also when your own plugin is not installed?

I don’t know the inner workings of your plugin, but here’s my take on the error messages:

If the exception happens on startup of Kibana, this would imply that the call to /myindex/_search is triggered by your plugin. The kibanaserver user needs access to the .kibana index, but apart from that Kibana is not performing any _search operation on startup. That’s why I’m guessing the call stems from your plugin.

If I’m correct so far, the call to /myindex/_search does not seem to have any HTTP Basic authentication credentials attached to it. So Search Guard responds (correctly) with a 401, prompting to provide some auth credentials. That’s also why you do not see any “denied” log entry in Elasticsearch: Returning a 401 is just part of HTTP Basic authentication flow to get the user credentials. It’s not some missing privileges, because at that point SG has not retrieved any credentials to work with yet.

I don’t know what your own plugin is doing exactly, but maybe this helps to point you in the right direction. If you have any more information, please let me know!

On Sunday, October 21, 2018 at 6:41:44 AM UTC+2, Nicholas George wrote:

Hi Searchguard folks,

I build a customised version of Kibana (the ‘-oss’ variety) to include our own plugins and perform some customisations to the CSS. No mods are made to the main Kibana code.

Since upgrading to v6.4.2 (from v6.2.4), I’ve been having a devil of a time getting Kibana to start properly. I’m getting the following error at start time:

{“type”:“log”,“@timestamp”:“2018-10-21T01:16:32Z”,“tags”:[“status”,“plugin:elasticsearch@6.4.2”,“error”],“pid”:4973,“state”:“red”,“message”:“Status changed from yellow to red - Authentication Exception”,“prevState”:“yellow”,“prevMsg”:“Waiting for Elasticsearch”}

{“type”:“log”,“@timestamp”:“2018-10-21T01:16:32Z”,“tags”:[“warning”],“pid”:4973,“message”:“Detected an unhandled Promise rejection.\nAuthentication Exception :: {"path":"/myindex/_search","query":{"q":"type=acl","size":100},"statusCode":401,"response":"Unauthorized","wwwAuthenticateDirective":"Basic realm=\"Search Guard\""}”}

I assumed that it would be a deny happening on the Elasticsearch side (from Searchguard), but I’m not seeing any of the usual Searchguard deny logs in the elasticsearch.log file. From my reading of it, it looks like the “kibanaserver” user doesn’t have access to log in, does that sound correct to you? The funny thing is that I can perform that exact query on the commandline using the kibanaserver user’s credentials. I have disabled x-pack in the elasticsearch.yml. There’s no need to disable x-pack on the kibana side because I’m using the ‘oss’ build (plus if I leave the line in to disable x-pack, it bombs out).

Have you seen errors like these being spat out by Kibana before?

Many thanks for your time.

Cheers,

Nick

5

Hi Jochen,

A quick message to let you know the issue has been sorted. It was a mistake on my part and was related to the changes in the Kibana build system as of v6.3.0. We not only use our own plugins, but build a custom Kibana and rename the package and services to deconflict with the official Kibana. More trouble than it’s worth!

Thanks again for your help.

Nick

6

Hi Nick,

thanks for letting us know and glad you worked it out!

···

On Saturday, October 27, 2018 at 11:45:53 PM UTC+2, Nicholas George wrote:

Hi Jochen,

A quick message to let you know the issue has been sorted. It was a mistake on my part and was related to the changes in the Kibana build system as of v6.3.0. We not only use our own plugins, but build a custom Kibana and rename the package and services to deconflict with the official Kibana. More trouble than it’s worth!

Thanks again for your help.

Nick