6.4.0 can't login and kibanaserver user can't access .kibana

Search Guard
1

Dear searchguard,

when upgrading a fully functional test setup from 6.3.0-22.3 to 6.4.0-23.0 I can no longer log into kibana afterwards. The browser just gets an connection reset after waiting for the 30sec tcp timeout. I use proxy authentication via local apache proxy.

It throws the strange errors about .kibana not being found but I can totally see that index:

green open .kibana gd00htRbRUOXEzcyaGLQTQ 1 1 1 0 8kb 4kb

journalctl -f kibana shows an error like this:

Sep 12 17:56:27 mes-any-logwfe-dev001 kibana[15920]: {“type”:“error”,“@timestamp”:“2018-09-12T15:56:27Z”,“tags”:[“warning”,“stats-collection”],“pid”:15920,“level”:“error”,“error”:{“message”:“[security_exception] Unexpected exception indices:data/read/search”,“name”:“Error”,“stack”:“[security_exception] Unexpected exception indices:data/read/search :: {"path":"/.kibana/_search","query":{"ignore_unavailable":true,"filter_path":"aggregations.types.buckets"},"body":"{\"size\":0,\"query\":{\"terms\":{\"type\":[\"dashboard\",\"visualization\",\"search\",\"index-pattern\",\"graph-workspace\",\"timelion-sheet\"]}},\"aggs\":{\"types\":{\"terms\":{\"field\":\"type\",\"size\":6}}}}","statusCode":500,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"Unexpected exception indices:data/read/search\"}],\"type\":\"security_exception\",\"reason\":\"Unexpected exception indices:data/read/search\"},\"status\":500}"}\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:307:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:266:7)\n at HttpConnector. (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)\n at IncomingMessage.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)\n at emitNone (events.js:111:20)\n at IncomingMessage.emit (events.js:208:7)\n at endReadableNT (_stream_readable.js:1064:12)\n at _combinedTickCallback (internal/process/next_tick.js:138:11)\n at process._tickDomainCallback (internal/process/next_tick.js:218:9)”},“message”:“[security_exception] Unexpected exception indices:data/read/search”}

I have tried to purge searchguard and the error disappeared thus posting it here.

If anything is unclear or not verbose enough please feel free to ask.

Thank you

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version

6.4.0-23.0

  • Installed and used enterprise modules, if any

none/default

  • JVM version and operating system version

java version “1.8.0_121”
Java™ SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot™ 64-Bit Server VM (build 25.121-b13, mixed mode)

Debian 8

  • Search Guard configuration files

attached

  • Elasticsearch log messages on debug level

attached

  • Other installed Elasticsearch or Kibana plugins, if any

sudo ./kibana-plugin list
searchguard@6.4.0-14

sudo ./elasticsearch-plugin list
search-guard-6

sg_config.yml (1.11 KB)

sg_roles_mapping.yml (704 Bytes)

sg_roles.yml (4.47 KB)

elasticsearch.log (20.5 KB)

2

According to the ES log file there seems to be an issue when accessing the Search Guard index:

[2018

-09-12T17:43:42,182][ERROR][c.f.s.f.SearchGuardFilter] Unexpected exception [] IndexNotFoundException[no such index]
org.elasticsearch.index.IndexNotFoundException: no such index
...

at com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.getResolvedIndexPattern(ConfigModel.java:598) ~[?:?]
at com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.access$900(ConfigModel.java:484) ~[?:?]
at com.floragunn.searchguard.sgconf.ConfigModel.impliesTypePerm(ConfigModel.java:768) ~[?:?]
at com.floragunn.searchguard.sgconf.ConfigModel.access$1100(ConfigModel.java:47) ~[?:?]
at com.floragunn.searchguard.sgconf.ConfigModel$SgRoles.get(ConfigModel.java:321) ~[?:?]


``

 Which would also explain the 30 seconds timeout you experienced. When you say:

"I have tried to purge searchguard and the error disappeared thus posting it here."

Do you mean you reinitialized the SG index and the error went away? This would then indicate some problems with the availability of the SG index primary/replica shards. How did you perform the upgrade? Was it a rolling restart? Did you set anything regarding shard allocation before performing the upgrade?

<details class='elided'>
<summary title='Show trimmed content'>&#183;&#183;&#183;</summary>

On Wednesday, September 12, 2018 at 6:01:08 PM UTC+2, benedikt.haug@gmx.de wrote:
> Dear searchguard,

> 

> when upgrading a fully functional test setup from 6.3.0-22.3 to 6.4.0-23.0 I can no longer log into kibana afterwards. The browser just gets an connection reset after waiting for the 30sec tcp timeout. I use proxy authentication via local apache proxy.

> 

> It throws the strange errors about .kibana not being found but I can totally see that index:

> green open .kibana                         gd00htRbRUOXEzcyaGLQTQ 1 1     1  0     8kb     4kb

> 

> journalctl -f kibana shows an error like this:

> 

> Sep 12 17:56:27 mes-any-logwfe-dev001 kibana[15920]: {"type":"error","@timestamp":"2018-09-12T15:56:27Z","tags":["warning","stats-collection"],"pid":15920,"level":"error","error":{"message":"[security_exception] Unexpected exception indices:data/read/search","name":"Error","stack":"[security_exception] Unexpected exception indices:data/read/search :: {\"path\":\"/.kibana/_search\",\"query\":{\"ignore_unavailable\":true,\"filter_path\":\"aggregations.types.buckets\"},\"body\":\"{\\\"size\\\":0,\\\"query\\\":{\\\"terms\\\":{\\\"type\\\":[\\\"dashboard\\\",\\\"visualization\\\",\\\"search\\\",\\\"index-pattern\\\",\\\"graph-workspace\\\",\\\"timelion-sheet\\\"]}},\\\"aggs\\\":{\\\"types\\\":{\\\"terms\\\":{\\\"field\\\":\\\"type\\\",\\\"size\\\":6}}}}\",\"statusCode\":500,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"Unexpected exception indices:data/read/search\\\"}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"Unexpected exception indices:data/read/search\\\"},\\\"status\\\":500}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:307:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:266:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)\n    at IncomingMessage.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)\n    at emitNone (events.js:111:20)\n    at IncomingMessage.emit (events.js:208:7)\n    at endReadableNT (_stream_readable.js:1064:12)\n    at _combinedTickCallback (internal/process/next_tick.js:138:11)\n    at process._tickDomainCallback (internal/process/next_tick.js:218:9)"},"message":"[security_exception] Unexpected exception indices:data/read/search"}
> 

> I have tried to purge searchguard and the error disappeared thus posting it here.

> 

> If anything is unclear or not verbose enough please feel free to ask.

> 

> Thank you

> 

> 

> When asking questions, please provide the following information:

> 

> * Search Guard and Elasticsearch version

> 

> 6.4.0-23.0

> 

> * Installed and used enterprise modules, if any

> 

> none/default

> 

> * JVM version and operating system version

> 

> java version "1.8.0_121"
> Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

> 

> Debian 8

> 

> * Search Guard configuration files

> 

> attached

> 

> * Elasticsearch log messages on debug level

> 

> attached

> 

> * Other installed Elasticsearch or Kibana plugins, if any

> sudo ./kibana-plugin list
> searchguard@6.4.0-14

> sudo ./elasticsearch-plugin list
> search-guard-6

</details>
3

Thank you for having a look!

  No, it never worked. I meant that i removed searchguard

completely by removing the plugin and it started to work without
searchguard.

  The upgrade was done by installing the new plugin version and

restarting the cluster altogether. After it didn’t work I also
tried to remove the old plugin version before installing the new
one.

I did not change the shard allocation.

The searchguard indices content is here:

  I just tried to remove the searchguard index with -dci and

recreated it afterwards but the kibana journal still looks like
this:

  Sep 17 15:54:08 mes-any-logwfe-dev001 kibana[23739]:

{“type”:“error”,“@timestamp”:“2018-09-17T13:54:08Z”,“tags”:[“warning”,“stats-collection”],“pid”:23739,“level”:“error”,“error”:{“message”:“[security_exception]
Unexpected exception
indices:data/read/search”,“name”:“Error”,“stack”:“[security_exception]
Unexpected exception indices:data/read/search ::
{"path":"/.kibana/_search","query":{"ignore_unavailable":true,"filter_path":"aggregations.types.buckets"},"body":"{\"size\":0,\"query\":{\"terms\":{\"type\":[\"dashboard\",\"visualization\",\"search\",\"index-pattern\",\"graph-workspace\",\"timelion-sheet\"]}},\"aggs\":{\"types\":{\"terms\":{\"field\":\"type\",\"size\":6}}}}","statusCode":500,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"Unexpected
exception
indices:data/read/search\"}],\"type\":\"security_exception\",\"reason\":\"Unexpected
exception
indices:data/read/search\"},\"status\":500}"}\n at
respond
(/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:307:15)\n
at checkRespForFailure
(/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:266:7)\n
at HttpConnector.
(/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)\n
at IncomingMessage.bound
(/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)\n
at emitNone (events.js:111:20)\n at IncomingMessage.emit
(events.js:208:7)\n at endReadableNT
(_stream_readable.js:1064:12)\n at _combinedTickCallback
(internal/process/next_tick.js:138:11)\n at
process._tickDomainCallback
(internal/process/next_tick.js:218:9)”},“message”:“[security_exception]
Unexpected exception indices:data/read/search”}

  As this is a dev host I could drop the .kibana index but I don't

want to do this for live environments.

Thank you for your help!

···

https://gist.github.com/gna582/da1d2439a2c51ed3e41d96fd2385bdbe
On 9/17/18 3:28 PM, Jochen Kressin
wrote:

    According to the ES log file there seems to be an

issue when accessing the Search Guard index:

[2018-09-12T17:43:42,182][ERROR][c.f.s.f.SearchGuardFilter] Unexpected exception [] IndexNotFoundException[no such index]
org.elasticsearch.index.IndexNotFoundException: no such index

…

at com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.getResolvedIndexPattern(ConfigModel.java:598) ~[?:?]
at com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.access$900(ConfigModel.java:484) ~[?:?]
at com.floragunn.searchguard.sgconf.ConfigModel.impliesTypePerm(ConfigModel.java:768) ~[?:?]
at com.floragunn.searchguard.sgconf.ConfigModel.access$1100(ConfigModel.java:47) ~[?:?]
at com.floragunn.searchguard.sgconf.ConfigModel$SgRoles.get(ConfigModel.java:321) ~[?:?]

``

       Which would also explain the 30 seconds timeout you

experienced. When you say:

      "I have tried to purge searchguard and the error

disappeared thus posting it here."

      Do you mean you reinitialized the SG index and the error

went away? This would then indicate some problems with the
availability of the SG index primary/replica shards. How did
you perform the upgrade? Was it a rolling restart? Did you set
anything regarding shard allocation before performing the
upgrade?

      On Wednesday, September 12, 2018 at 6:01:08 PM UTC+2,

wrote:

–

  You received this message because you are subscribed to the Google

Groups “Search Guard Community Forum” group.

  To unsubscribe from this group and stop receiving emails from it,

send an email to search-guard+unsubscribe@googlegroups.com.

  To post to this group, send email to search-guard@googlegroups.com.

  To view this discussion on the web visit [https://groups.google.com/d/msgid/search-guard/de3e56d5-8c8e-4de9-9d2e-4e0dfd702cbc%40googlegroups.com](https://groups.google.com/d/msgid/search-guard/de3e56d5-8c8e-4de9-9d2e-4e0dfd702cbc%40googlegroups.com?utm_medium=email&utm_source=footer).

  For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

benedikt.haug@gmx.de

Dear searchguard,

            when upgrading a fully functional test setup from

6.3.0-22.3 to 6.4.0-23.0 I can no longer log into kibana
afterwards. The browser just gets an connection reset
after waiting for the 30sec tcp timeout. I use proxy
authentication via local apache proxy.

            It throws the strange errors about .kibana not being

found but I can totally see that index:

green open .kibana
gd00htRbRUOXEzcyaGLQTQ 1 1 1 0 8kb 4kb

journalctl -f kibana shows an error like this:

            Sep 12 17:56:27 mes-any-logwfe-dev001 kibana[15920]:

{“type”:“error”,“@timestamp”:“2018-09-12T15:56:27Z”,“tags”:[“warning”,“stats-collection”],“pid”:15920,“level”:“error”,“error”:{“message”:“[security_ exception]
Unexpected exception indices:data/read/search”,“name”:“Error”,“stack”:“[ security_exception]
Unexpected exception indices:data/read/search ::
{"path":"/.kibana/_search","query":{"ignore_unavailable":true,"filter_path":"aggregations.types.buckets"},"body":"{\"size\":0,\"query\":{\"terms\":{\"type\":[\"dashboard\",\"visualization\",\"search\",\"index-pattern\",\"graph-workspace\",\"timelion-sheet\"]}},\"aggs\":{\"types\":{\"terms\":{\"field\":\"type\",\"size\":6}}}}","statusCode":500,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\" reason\":\"Unexpected
exception indices:data/read/search\"}],\"type\":\"security_exception\",\"reason\":\ \“Unexpected
exception indices:data/read/search\"}, \"status\":500}"}\n
at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/ transport.js:307:15)\n
at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/ transport.js:266:7)\n
at HttpConnector.
(/usr/share/kibana/node_modules/elasticsearch/src/lib/ connectors/http.js:159:7)\n
at IncomingMessage.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js: 729:21)\n
at emitNone (events.js:111:20)\n at
IncomingMessage.emit (events.js:208:7)\n at
endReadableNT (_stream_readable.js:1064:12)\ n at
_combinedTickCallback (internal/process/next_tick. js:138:11)\n
at process._tickDomainCallback
(internal/process/next_tick.js:218:9)”},“message”:”[ security_exception]
Unexpected exception indices:data/read/search"}

            I have tried to purge searchguard and the error

disappeared thus posting it here.

            If anything is unclear or not verbose enough please

feel free to ask.

Thank you

            When asking questions, please provide the following

information:

  • Search Guard and Elasticsearch version

6.4.0-23.0

  • Installed and used enterprise modules, if any

none/default

  • JVM version and operating system version

java version “1.8.0_121”

              Java(TM) SE Runtime Environment (build 1.8.0_121-b13)

              Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13,

mixed mode)

Debian 8

  • Search Guard configuration files

attached

  • Elasticsearch log messages on debug level

attached

            * Other installed Elasticsearch or Kibana plugins, if

any

sudo ./kibana-plugin list

sudo ./elasticsearch-plugin list

            search-guard-6

searchguard@6.4.0-14

4

To me, this does not look like a problem with the .kibana index per se. I conclude this from the error message, with is an “Unexpected Exception” with status code 500:

"statusCode":500,"
…
Unexpected exception indices:data/read/search

``

This means it is an internal Search Guard error. If the .kibana index would not be accerssible, say, because of wrong roles or permissions, you would see a different message.

The message you see on the ES side:

org.elasticsearch.index.IndexNotFoundException: no such index

``

Refers to the Search Guard index not being accessible. It roots in SG not being able to read the roles from the index:

at com.floragunn.searchguard.sgconf.ConfigModel$SgRoles.get(ConfigModel.java:321) ~[?:?]

``

That’s why I was asking for the shards, because the node does not seem to have the “searchguard” index accessible.

So after recreating the SG index, is the error message on Elasticsearch side still the same? Means, the “no such index” message? This would seem very strange since the SG index is an index like any other index on your cluster. Means, it is completely managed by Elasticsearch.

Thanks for the link to the gist - but it contains the mapping only, can you do a _search on the SG index? The actual documents in this index should be base64 encoded strings.

···

On Monday, September 17, 2018 at 3:55:50 PM UTC+2, Benedikt Haug wrote:

Thank you for having a look!

  No, it never worked. I meant that i removed searchguard

completely by removing the plugin and it started to work without
searchguard.

  The upgrade was done by installing the new plugin version and

restarting the cluster altogether. After it didn’t work I also
tried to remove the old plugin version before installing the new
one.

I did not change the shard allocation.

The searchguard indices content is here:

https://gist.github.com/gna582/da1d2439a2c51ed3e41d96fd2385bdbe

  I just tried to remove the searchguard index with -dci and

recreated it afterwards but the kibana journal still looks like
this:

Sep 17 15:54:08 mes-any-logwfe-dev001 kibana[23739]:
{“type”:“error”,“@timestamp”:“2018-09-17T13:54:08Z”,“tags”:[“warning”,“stats-collection”],“pid”:23739,“level”:“error”,“error”:{“message”:“[security_ exception]
Unexpected exception
indices:data/read/search”,“name”:“Error”,“stack”:“[ security_exception]
Unexpected exception indices:data/read/search ::
{"path":"/.kibana/_search","query":{"ignore_unavailable":true,"filter_path":"aggregations.types.buckets"},"body":"{\"size\":0,\"query\":{\"terms\":{\"type\":[\"dashboard\",\"visualization\",\"search\",\"index-pattern\",\"graph-workspace\",\"timelion-sheet\"]}},\"aggs\":{\"types\":{\"terms\":{\"field\":\"type\",\"size\":6}}}}","statusCode":500,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\" reason\":\"Unexpected
exception
indices:data/read/search\"}],\"type\":\"security_exception\",\"reason\":\ \“Unexpected
exception
indices:data/read/search\"}, \"status\":500}"}\n at
respond
(/usr/share/kibana/node_modules/elasticsearch/src/lib/ transport.js:307:15)\n
at checkRespForFailure
(/usr/share/kibana/node_modules/elasticsearch/src/lib/ transport.js:266:7)\n
at HttpConnector.
(/usr/share/kibana/node_modules/elasticsearch/src/lib/ connectors/http.js:159:7)\n
at IncomingMessage.bound
(/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js: 729:21)\n
at emitNone (events.js:111:20)\n at IncomingMessage.emit
(events.js:208:7)\n at endReadableNT
(_stream_readable.js:1064:12)\ n at _combinedTickCallback
(internal/process/next_tick. js:138:11)\n at
process._tickDomainCallback
(internal/process/next_tick.js:218:9)”},“message”:”[ security_exception]
Unexpected exception indices:data/read/search"}

  As this is a dev host I could drop the .kibana index but I don't

want to do this for live environments.

Thank you for your help!

  On 9/17/18 3:28 PM, Jochen Kressin > wrote:

    According to the ES log file there seems to be an

issue when accessing the Search Guard index:

[2018-09-12T17:43:42,182][ERROR][c.f.s.f.SearchGuardFilter] Unexpected exception [] IndexNotFoundException[no such index]
org.elasticsearch.index.IndexNotFoundException: no such index

…

at com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.getResolvedIndexPattern(ConfigModel.java:598) ~[?:?]
at com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.access$900(ConfigModel.java:484) ~[?:?]
at com.floragunn.searchguard.sgconf.ConfigModel.impliesTypePerm(ConfigModel.java:768) ~[?:?]
at com.floragunn.searchguard.sgconf.ConfigModel.access$1100(ConfigModel.java:47) ~[?:?]
at com.floragunn.searchguard.sgconf.ConfigModel$SgRoles.get(ConfigModel.java:321) ~[?:?]

``

       Which would also explain the 30 seconds timeout you

experienced. When you say:

      "I have tried to purge searchguard and the error

disappeared thus posting it here."

      Do you mean you reinitialized the SG index and the error

went away? This would then indicate some problems with the
availability of the SG index primary/replica shards. How did
you perform the upgrade? Was it a rolling restart? Did you set
anything regarding shard allocation before performing the
upgrade?

      On Wednesday, September 12, 2018 at 6:01:08 PM UTC+2, > > benedikt.haug@gmx.de wrote:

Dear searchguard,

            when upgrading a fully functional test setup from

6.3.0-22.3 to 6.4.0-23.0 I can no longer log into kibana
afterwards. The browser just gets an connection reset
after waiting for the 30sec tcp timeout. I use proxy
authentication via local apache proxy.

            It throws the strange errors about .kibana not being

found but I can totally see that index:

green open .kibana
gd00htRbRUOXEzcyaGLQTQ 1 1 1 0 8kb 4kb

journalctl -f kibana shows an error like this:

            Sep 12 17:56:27 mes-any-logwfe-dev001 kibana[15920]:

{“type”:“error”,“@timestamp”:“2018-09-12T15:56:27Z”,“tags”:[“warning”,“stats-collection”],“pid”:15920,“level”:“error”,“error”:{“message”:“[security_ exception]
Unexpected exception indices:data/read/search”,“name”:“Error”,“stack”:“[ security_exception]
Unexpected exception indices:data/read/search ::
{"path":"/.kibana/_search","query":{"ignore_unavailable":true,"filter_path":"aggregations.types.buckets"},"body":"{\"size\":0,\"query\":{\"terms\":{\"type\":[\"dashboard\",\"visualization\",\"search\",\"index-pattern\",\"graph-workspace\",\"timelion-sheet\"]}},\"aggs\":{\"types\":{\"terms\":{\"field\":\"type\",\"size\":6}}}}","statusCode":500,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\" reason\":\"Unexpected
exception indices:data/read/search\"}],\"type\":\"security_exception\",\"reason\":\ \“Unexpected
exception indices:data/read/search\"}, \"status\":500}"}\n
at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/ transport.js:307:15)\n
at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/ transport.js:266:7)\n
at HttpConnector.
(/usr/share/kibana/node_modules/elasticsearch/src/lib/ connectors/http.js:159:7)\n
at IncomingMessage.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js: 729:21)\n
at emitNone (events.js:111:20)\n at
IncomingMessage.emit (events.js:208:7)\n at
endReadableNT (_stream_readable.js:1064:12)\ n at
_combinedTickCallback (internal/process/next_tick. js:138:11)\n
at process._tickDomainCallback
(internal/process/next_tick.js:218:9)”},“message”:”[ security_exception]
Unexpected exception indices:data/read/search"}

            I have tried to purge searchguard and the error

disappeared thus posting it here.

            If anything is unclear or not verbose enough please

feel free to ask.

Thank you

            When asking questions, please provide the following

information:

  • Search Guard and Elasticsearch version

6.4.0-23.0

  • Installed and used enterprise modules, if any

none/default

  • JVM version and operating system version

java version “1.8.0_121”

              Java(TM) SE Runtime Environment (build 1.8.0_121-b13)

              Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13,

mixed mode)

Debian 8

  • Search Guard configuration files

attached

  • Elasticsearch log messages on debug level

attached

            * Other installed Elasticsearch or Kibana plugins, if

any

sudo ./kibana-plugin list

            searchguard@6.4.0-14

sudo ./elasticsearch-plugin list

            search-guard-6

–

  You received this message because you are subscribed to the Google

Groups “Search Guard Community Forum” group.

  To unsubscribe from this group and stop receiving emails from it,

send an email to search-guard+unsubscribe@googlegroups.com.

  To post to this group, send email to search-guard@googlegroups.com.

  To view this discussion on the web visit [https://groups.google.com/d/msgid/search-guard/de3e56d5-8c8e-4de9-9d2e-4e0dfd702cbc%40googlegroups.com](https://groups.google.com/d/msgid/search-guard/de3e56d5-8c8e-4de9-9d2e-4e0dfd702cbc%40googlegroups.com?utm_medium=email&utm_source=footer).

  For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
5

Sure! Here it is:

  But I can retrieve the config via sgadmin -r and it looks

identical except quotation and indention.

Thank you

···

https://gist.github.com/gna582/cdc9e628330808a23443fd71c348e250
On 9/17/18 4:15 PM, Jochen Kressin
wrote:

    To me, this does not look like a problem with the

.kibana index per se. I conclude this from the error message,
with is an “Unexpected Exception” with status code 500:

"statusCode":500,"

              ...

                                Unexpected exception

indices:data/read/search

``

      This means it is an internal Search Guard error. If the

.kibana index would not be accerssible, say, because of wrong
roles or permissions, you would see a different message.

The message you see on the ES side:

org.elasticsearch.index.IndexNotFoundException: no such index

``

      Refers to the Search Guard index not being accessible. It

roots in SG not being able to read the roles from the index:

at com.floragunn.searchguard.sgconf.ConfigModel$SgRoles.get(ConfigModel.java:321) ~[?:?]

``

      That's why I was asking for the shards, because the node does

not seem to have the “searchguard” index accessible.

      So after recreating the SG index, is the error message on

Elasticsearch side still the same? Means, the “no such index”
message? This would seem very strange since the SG index is an
index like any other index on your cluster. Means, it is
completely managed by Elasticsearch.

      Thanks for the link to the gist - but it contains the

mapping only, can you do a _search on the SG index? The actual
documents in this index should be base64 encoded strings.

      On Monday, September 17, 2018 at 3:55:50 PM UTC+2, Benedikt

Haug wrote:

Thank you for having a look!

            No, it never worked. I meant that i removed searchguard

completely by removing the plugin and it started to work
without searchguard.

            The upgrade was done by installing the new plugin

version and restarting the cluster altogether. After it
didn’t work I also tried to remove the old plugin
version before installing the new one.

I did not change the shard allocation.

The searchguard indices content is here:

https://gist.github.com/gna582/da1d2439a2c51ed3e41d96fd2385bdbe

            I just tried to remove the searchguard index with -dci

and recreated it afterwards but the kibana journal still
looks like this:

            Sep 17 15:54:08 mes-any-logwfe-dev001 kibana[23739]:

{“type”:“error”,“@timestamp”:“2018-09-17T13:54:08Z”,“tags”:[“warning”,“stats-collection”],“pid”:23739,“level”:“error”,“error”:{“message”:“[security_ exception]
Unexpected exception indices:data/read/search”,“name”:“Error”,“stack”:“[ security_exception]
Unexpected exception indices:data/read/search ::
{"path":"/.kibana/_search","query":{"ignore_unavailable":true,"filter_path":"aggregations.types.buckets"},"body":"{\"size\":0,\"query\":{\"terms\":{\"type\":[\"dashboard\",\"visualization\",\"search\",\"index-pattern\",\"graph-workspace\",\"timelion-sheet\"]}},\"aggs\":{\"types\":{\"terms\":{\"field\":\"type\",\"size\":6}}}}","statusCode":500,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\" reason\":\"Unexpected
exception
indices:data/read/search\"}],\"type\":\"security_exception\",\"reason\":\ \“Unexpected
exception indices:data/read/search\"}, \"status\":500}"}\n
at respond
(/usr/share/kibana/node_modules/elasticsearch/src/lib/ transport.js:307:15)\n
at checkRespForFailure
(/usr/share/kibana/node_modules/elasticsearch/src/lib/ transport.js:266:7)\n
at HttpConnector.
(/usr/share/kibana/node_modules/elasticsearch/src/lib/ connectors/http.js:159:7)\n
at IncomingMessage.bound
(/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js: 729:21)\n
at emitNone (events.js:111:20)\n at
IncomingMessage.emit (events.js:208:7)\n at
endReadableNT (_stream_readable.js:1064:12)\ n at
_combinedTickCallback (internal/process/next_tick. js:138:11)\n
at process._tickDomainCallback
(internal/process/next_tick.js:218:9)”},“message”:”[ security_exception]
Unexpected exception indices:data/read/search"}

            As this is a dev host I could drop the .kibana index

but I don’t want to do this for live environments.

Thank you for your help!

On 9/17/18 3:28 PM, Jochen Kressin wrote:

              According to the ES log file there seems

to be an issue when accessing the Search Guard index:

[2018-09-12T17:43:42,182][ERROR][c.f.s.f.SearchGuardFilter] Unexpected exception [] IndexNotFoundException[no such index]
org.elasticsearch.index.IndexNotFoundException: no such index

…

at com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.getResolvedIndexPattern(ConfigModel.java:598) ~[?:?]
at com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.access$900(ConfigModel.java:484) ~[?:?]
at com.floragunn.searchguard.sgconf.ConfigModel.impliesTypePerm(ConfigModel.java:768) ~[?:?]
at com.floragunn.searchguard.sgconf.ConfigModel.access$1100(ConfigModel.java:47) ~[?:?]
at com.floragunn.searchguard.sgconf.ConfigModel$SgRoles.get(ConfigModel.java:321) ~[?:?]

``

                 Which would also explain the 30 seconds timeout you

experienced. When you say:

                "I have tried to purge searchguard and the error

disappeared thus posting it here."

                Do you mean you reinitialized the SG index and

the error went away? This would then indicate some
problems with the availability of the SG index
primary/replica shards. How did you perform the
upgrade? Was it a rolling restart? Did you set
anything regarding shard allocation before
performing the upgrade?

                On Wednesday, September 12, 2018 at 6:01:08 PM

UTC+2, benedikt.haug@gmx.de
wrote:

Dear searchguard,

                      when upgrading a fully functional test

setup from 6.3.0-22.3 to 6.4.0-23.0 I can no
longer log into kibana afterwards. The browser
just gets an connection reset after waiting
for the 30sec tcp timeout. I use proxy
authentication via local apache proxy.

                      It throws the strange errors about .kibana

not being found but I can totally see that
index:

green open .kibana
gd00htRbRUOXEzcyaGLQTQ 1 1 1 0
8kb 4kb

                      journalctl -f kibana shows an error like

this:

                      Sep 12 17:56:27 mes-any-logwfe-dev001

kibana[15920]: {“type”:“error”,“@timestamp”:“2018-09-12T15:56:27Z”,“tags”:[“warning”,“stats-collection”],“pid”:15920,“level”:“error”,“error”:{“message”:“[security_ exception]
Unexpected exception
indices:data/read/search”,“name”:“Error”,“stack”:“[ security_exception]
Unexpected exception indices:data/read/search
:: {"path":"/.kibana/_search","query":{"ignore_unavailable":true,"filter_path":"aggregations.types.buckets"},"body":"{\"size\":0,\"query\":{\"terms\":{\"type\":[\"dashboard\",\"visualization\",\"search\",\"index-pattern\",\"graph-workspace\",\"timelion-sheet\"]}},\"aggs\":{\"types\":{\"terms\":{\"field\":\"type\",\"size\":6}}}}","statusCode":500,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\" reason\":\"Unexpected
exception indices:data/read/search\"}],\"type\":\"security_exception\",\"reason\":\ \“Unexpected
exception indices:data/read/search\"}, \"status\":500}"}\n
at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/ transport.js:307:15)\n
at checkRespForFailure
(/usr/share/kibana/node_modules/elasticsearch/src/lib/ transport.js:266:7)\n
at HttpConnector.
(/usr/share/kibana/node_modules/elasticsearch/src/lib/ connectors/http.js:159:7)\n
at IncomingMessage.bound
(/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js: 729:21)\n
at emitNone (events.js:111:20)\n at
IncomingMessage.emit (events.js:208:7)\n at
endReadableNT (_stream_readable.js:1064:12)\ n
at _combinedTickCallback
(internal/process/next_tick. js:138:11)\n
at process._tickDomainCallback
(internal/process/next_tick.js:218:9)”},“message”:”[ security_exception]
Unexpected exception
indices:data/read/search"}

                      I have tried to purge searchguard and the

error disappeared thus posting it here.

                      If anything is unclear or not verbose

enough please feel free to ask.

Thank you

                      When asking questions, please provide the

following information:

  • Search Guard and Elasticsearch version

6.4.0-23.0

                        * Installed and used enterprise modules,

if any

none/default

                        * JVM version and operating system

version

java version “1.8.0_121”

                        Java(TM) SE Runtime Environment (build

1.8.0_121-b13)

                        Java HotSpot(TM) 64-Bit Server VM (build

25.121-b13, mixed mode)

Debian 8

  • Search Guard configuration files

attached

                        * Elasticsearch log messages on debug

level

attached

                      * Other installed Elasticsearch or Kibana

plugins, if any

sudo ./kibana-plugin list

                      searchguard@6.4.0-14

sudo ./elasticsearch-plugin list

                      search-guard-6

–

            You received this message because you are subscribed to

the Google Groups “Search Guard Community Forum” group.

            To unsubscribe from this group and stop receiving emails

from it, send an email to search-guard+unsubscribe@googlegroups.com.

            To post to this group, send email to search-guard@googlegroups.com.

            To view this discussion on the web visit [https://groups.google.com/d/msgid/search-guard/de3e56d5-8c8e-4de9-9d2e-4e0dfd702cbc%40googlegroups.com](https://groups.google.com/d/msgid/search-guard/de3e56d5-8c8e-4de9-9d2e-4e0dfd702cbc%40googlegroups.com?utm_medium=email&utm_source=footer).

            For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

–

  You received this message because you are subscribed to the Google

Groups “Search Guard Community Forum” group.

  To unsubscribe from this group and stop receiving emails from it,

send an email to search-guard+unsubscribe@googlegroups.com.

  To post to this group, send email to search-guard@googlegroups.com.

  To view this discussion on the web visit [https://groups.google.com/d/msgid/search-guard/3c5adfac-ee43-453c-a18a-dd1d501c6ae6%40googlegroups.com](https://groups.google.com/d/msgid/search-guard/3c5adfac-ee43-453c-a18a-dd1d501c6ae6%40googlegroups.com?utm_medium=email&utm_source=footer).

  For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
6

This is pretty strange, but I would still say this is somewhat more ES related, since the underlying problem is the index not found.

So, a couple of questions:

  • are you saying that you can retrieve the configs via sgadmin -r from the running cluster?

  • are you able to access Elasticsearch directly, means without Kibana?

  • can you try to apply the configuration again with sgadmin? Any errors in the logs?

  • can you start ES with debug logging? We would need the debug outputs from ES and SG while starting the node, and when the error occurs.

To enable debug logging, in your log4j2.properties, add:

logger.searchguard.name = com.floragunn
logger.searchguard.level = debugOn Monday, September 17, 2018 at

And set the ES logger to debug as well:

logger.action.name = org.elasticsearch.action

logger.action.level = debug

That will produce a lot of output, but will hopefully help to clear things up.

4:48:29 PM UTC+2, Benedikt Haug wrote:

···

Sure! Here it is:

https://gist.github.com/gna582/cdc9e628330808a23443fd71c348e250

  But I can retrieve the config via sgadmin -r and it looks

identical except quotation and indention.

Thank you

  On 9/17/18 4:15 PM, Jochen Kressin > wrote:

    To me, this does not look like a problem with the

.kibana index per se. I conclude this from the error message,
with is an “Unexpected Exception” with status code 500:

"statusCode":500,"

              ...

                                Unexpected exception

indices:data/read/search

``

      This means it is an internal Search Guard error. If the

.kibana index would not be accerssible, say, because of wrong
roles or permissions, you would see a different message.

The message you see on the ES side:

org.elasticsearch.index.IndexNotFoundException: no such index

``

      Refers to the Search Guard index not being accessible. It

roots in SG not being able to read the roles from the index:

at com.floragunn.searchguard.sgconf.ConfigModel$SgRoles.get(ConfigModel.java:321) ~[?:?]

``

      That's why I was asking for the shards, because the node does

not seem to have the “searchguard” index accessible.

      So after recreating the SG index, is the error message on

Elasticsearch side still the same? Means, the “no such index”
message? This would seem very strange since the SG index is an
index like any other index on your cluster. Means, it is
completely managed by Elasticsearch.

      Thanks for the link to the gist - but it contains the

mapping only, can you do a _search on the SG index? The actual
documents in this index should be base64 encoded strings.

      On Monday, September 17, 2018 at 3:55:50 PM UTC+2, Benedikt > > Haug wrote:

Thank you for having a look!

            No, it never worked. I meant that i removed searchguard

completely by removing the plugin and it started to work
without searchguard.

            The upgrade was done by installing the new plugin

version and restarting the cluster altogether. After it
didn’t work I also tried to remove the old plugin
version before installing the new one.

I did not change the shard allocation.

The searchguard indices content is here:

https://gist.github.com/gna582/da1d2439a2c51ed3e41d96fd2385bdbe

            I just tried to remove the searchguard index with -dci

and recreated it afterwards but the kibana journal still
looks like this:

            Sep 17 15:54:08 mes-any-logwfe-dev001 kibana[23739]:

{“type”:“error”,“@timestamp”:“2018-09-17T13:54:08Z”,“tags”:[“warning”,“stats-collection”],“pid”:23739,“level”:“error”,“error”:{“message”:“[security_ exception]
Unexpected exception indices:data/read/search”,“name”:“Error”,“stack”:“[ security_exception]
Unexpected exception indices:data/read/search ::
{"path":"/.kibana/_search","query":{"ignore_unavailable":true,"filter_path":"aggregations.types.buckets"},"body":"{\"size\":0,\"query\":{\"terms\":{\"type\":[\"dashboard\",\"visualization\",\"search\",\"index-pattern\",\"graph-workspace\",\"timelion-sheet\"]}},\"aggs\":{\"types\":{\"terms\":{\"field\":\"type\",\"size\":6}}}}","statusCode":500,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\" reason\":\"Unexpected
exception
indices:data/read/search\"}],\"type\":\"security_exception\",\"reason\":\ \“Unexpected
exception indices:data/read/search\"}, \"status\":500}"}\n
at respond
(/usr/share/kibana/node_modules/elasticsearch/src/lib/ transport.js:307:15)\n
at checkRespForFailure
(/usr/share/kibana/node_modules/elasticsearch/src/lib/ transport.js:266:7)\n
at HttpConnector.
(/usr/share/kibana/node_modules/elasticsearch/src/lib/ connectors/http.js:159:7)\n
at IncomingMessage.bound
(/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js: 729:21)\n
at emitNone (events.js:111:20)\n at
IncomingMessage.emit (events.js:208:7)\n at
endReadableNT (_stream_readable.js:1064:12)\ n at
_combinedTickCallback (internal/process/next_tick. js:138:11)\n
at process._tickDomainCallback
(internal/process/next_tick.js:218:9)”},“message”:”[ security_exception]
Unexpected exception indices:data/read/search"}

            As this is a dev host I could drop the .kibana index

but I don’t want to do this for live environments.

Thank you for your help!

On 9/17/18 3:28 PM, Jochen Kressin wrote:

              According to the ES log file there seems

to be an issue when accessing the Search Guard index:

[2018-09-12T17:43:42,182][ERROR][c.f.s.f.SearchGuardFilter] Unexpected exception [] IndexNotFoundException[no such index]
org.elasticsearch.index.IndexNotFoundException: no such index

…

at com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.getResolvedIndexPattern(ConfigModel.java:598) ~[?:?]
at com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.access$900(ConfigModel.java:484) ~[?:?]
at com.floragunn.searchguard.sgconf.ConfigModel.impliesTypePerm(ConfigModel.java:768) ~[?:?]
at com.floragunn.searchguard.sgconf.ConfigModel.access$1100(ConfigModel.java:47) ~[?:?]
at com.floragunn.searchguard.sgconf.ConfigModel$SgRoles.get(ConfigModel.java:321) ~[?:?]

``

                 Which would also explain the 30 seconds timeout you

experienced. When you say:

                "I have tried to purge searchguard and the error

disappeared thus posting it here."

                Do you mean you reinitialized the SG index and

the error went away? This would then indicate some
problems with the availability of the SG index
primary/replica shards. How did you perform the
upgrade? Was it a rolling restart? Did you set
anything regarding shard allocation before
performing the upgrade?

                On Wednesday, September 12, 2018 at 6:01:08 PM > > > > UTC+2, benedikt.haug@gmx.de > > > >                     wrote:

Dear searchguard,

                      when upgrading a fully functional test

setup from 6.3.0-22.3 to 6.4.0-23.0 I can no
longer log into kibana afterwards. The browser
just gets an connection reset after waiting
for the 30sec tcp timeout. I use proxy
authentication via local apache proxy.

                      It throws the strange errors about .kibana

not being found but I can totally see that
index:

green open .kibana
gd00htRbRUOXEzcyaGLQTQ 1 1 1 0
8kb 4kb

                      journalctl -f kibana shows an error like

this:

                      Sep 12 17:56:27 mes-any-logwfe-dev001

kibana[15920]: {“type”:“error”,“@timestamp”:“2018-09-12T15:56:27Z”,“tags”:[“warning”,“stats-collection”],“pid”:15920,“level”:“error”,“error”:{“message”:“[security_ exception]
Unexpected exception
indices:data/read/search”,“name”:“Error”,“stack”:“[ security_exception]
Unexpected exception indices:data/read/search
:: {"path":"/.kibana/_search","query":{"ignore_unavailable":true,"filter_path":"aggregations.types.buckets"},"body":"{\"size\":0,\"query\":{\"terms\":{\"type\":[\"dashboard\",\"visualization\",\"search\",\"index-pattern\",\"graph-workspace\",\"timelion-sheet\"]}},\"aggs\":{\"types\":{\"terms\":{\"field\":\"type\",\"size\":6}}}}","statusCode":500,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\" reason\":\"Unexpected
exception indices:data/read/search\"}],\"type\":\"security_exception\",\"reason\":\ \“Unexpected
exception indices:data/read/search\"}, \"status\":500}"}\n
at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/ transport.js:307:15)\n
at checkRespForFailure
(/usr/share/kibana/node_modules/elasticsearch/src/lib/ transport.js:266:7)\n
at HttpConnector.
(/usr/share/kibana/node_modules/elasticsearch/src/lib/ connectors/http.js:159:7)\n
at IncomingMessage.bound
(/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js: 729:21)\n
at emitNone (events.js:111:20)\n at
IncomingMessage.emit (events.js:208:7)\n at
endReadableNT (_stream_readable.js:1064:12)\ n
at _combinedTickCallback
(internal/process/next_tick. js:138:11)\n
at process._tickDomainCallback
(internal/process/next_tick.js:218:9)”},“message”:”[ security_exception]
Unexpected exception
indices:data/read/search"}

                      I have tried to purge searchguard and the

error disappeared thus posting it here.

                      If anything is unclear or not verbose

enough please feel free to ask.

Thank you

                      When asking questions, please provide the

following information:

  • Search Guard and Elasticsearch version

6.4.0-23.0

                        * Installed and used enterprise modules,

if any

none/default

                        * JVM version and operating system

version

java version “1.8.0_121”

                        Java(TM) SE Runtime Environment (build

1.8.0_121-b13)

                        Java HotSpot(TM) 64-Bit Server VM (build

25.121-b13, mixed mode)

Debian 8

  • Search Guard configuration files

attached

                        * Elasticsearch log messages on debug

level

attached

                      * Other installed Elasticsearch or Kibana

plugins, if any

sudo ./kibana-plugin list

                      searchguard@6.4.0-14

sudo ./elasticsearch-plugin list

                      search-guard-6

–

            You received this message because you are subscribed to

the Google Groups “Search Guard Community Forum” group.

            To unsubscribe from this group and stop receiving emails

from it, send an email to search-guard+unsubscribe@googlegroups.com.

            To post to this group, send email to search-guard@googlegroups.com.

            To view this discussion on the web visit [https://groups.google.com/d/msgid/search-guard/de3e56d5-8c8e-4de9-9d2e-4e0dfd702cbc%40googlegroups.com](https://groups.google.com/d/msgid/search-guard/de3e56d5-8c8e-4de9-9d2e-4e0dfd702cbc%40googlegroups.com?utm_medium=email&utm_source=footer).

            For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

–

  You received this message because you are subscribed to the Google

Groups “Search Guard Community Forum” group.

  To unsubscribe from this group and stop receiving emails from it,

send an email to search-guard+unsubscribe@googlegroups.com.

  To post to this group, send email to search-guard@googlegroups.com.

  To view this discussion on the web visit [https://groups.google.com/d/msgid/search-guard/3c5adfac-ee43-453c-a18a-dd1d501c6ae6%40googlegroups.com](https://groups.google.com/d/msgid/search-guard/3c5adfac-ee43-453c-a18a-dd1d501c6ae6%40googlegroups.com?utm_medium=email&utm_source=footer).

  For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
7

Thanks for the help!

      - are you saying that you can retrieve the configs via

sgadmin -r from the running cluster?

Yes. It seems to be the same except quotation and indentation.

      - are you able to access Elasticsearch directly, means

without Kibana?

  I pulled the data from the search api via curl so yes that is

working. E.g. this would result in a whole lot of output, so i
pasted only a small portion

  curl --header "x-forwarded-for: 10.90.30.226" --user ttlko

–header “x-proxy-user: myuser” --header “x-proxy-roles: mygroup”

  ...}, "segments" : { "count" : 5, "memory_in_bytes" : 18562,

“terms_memory_in_bytes” : 14045, “stored_fields_memory_in_bytes” :
1560, “term_vectors_memory_in_bytes” : 0, “norms_memory_in_bytes”
: 0, “points_memory_in_bytes” : 689, “doc_values_memory_in_bytes”
: 2268, “index_writer_memory_in_bytes” : 0,
“version_map_memory_in_bytes” : 0, “fixed_bit_set_memory_in_bytes”
: 0 }, “request_cache” : {…

      - can you try to apply the configuration again with

sgadmin? Any errors in the logs?

Yes. No errors in the logs:

  root@mes-any-logwfe-dev001:~#

/usr/share/elasticsearch/plugins/search-guard-6/tools/sgadmin.sh
-cd /usr/share/elasticsearch/plugins/search-guard-6/sgconfig/
-cacert /etc/ssl/mailcore/certs/padminmesanylogqa.pem -cert
/etc/ssl/mailcore/certs/padminmesanylogqa.pem -h
mes-any-logwfe-dev001.qa.server.lan -cn mes_any_log -key
/etc/ssl/mailcore/certs/padminmesanylogqa.pem -nhnv

  WARNING: JAVA_HOME not set, will use /usr/bin/java

  Search Guard Admin v6

  Will connect to mes-any-logwfe-dev001.qa.server.lan:9300 ... done

  Elasticsearch Version: 6.4.0

  Search Guard Version: 6.4.0-23.0

  Connected as UID=XXXXX,CN=padminmesanylogqa,O=ClientCert

  Contacting elasticsearch cluster 'mes_any_log' and wait for YELLOW

clusterstate …

  Clustername: mes_any_log

  Clusterstate: GREEN

  Number of nodes: 3

  Number of data nodes: 2

  searchguard index already exists, so we do not need to create one.

  Populate config from

/usr/share/elasticsearch/plugins/search-guard-6/sgconfig/

  Will update 'sg/config' with

/usr/share/elasticsearch/plugins/search-guard-6/sgconfig/sg_config.yml

     SUCC: Configuration for 'config' created or updated

  Will update 'sg/roles' with

/usr/share/elasticsearch/plugins/search-guard-6/sgconfig/sg_roles.yml

     SUCC: Configuration for 'roles' created or updated

  Will update 'sg/rolesmapping' with

/usr/share/elasticsearch/plugins/search-guard-6/sgconfig/sg_roles_mapping.yml

     SUCC: Configuration for 'rolesmapping' created or updated

  Will update 'sg/internalusers' with

/usr/share/elasticsearch/plugins/search-guard-6/sgconfig/sg_internal_users.yml

     SUCC: Configuration for 'internalusers' created or updated

  Will update 'sg/actiongroups' with

/usr/share/elasticsearch/plugins/search-guard-6/sgconfig/sg_action_groups.yml

     SUCC: Configuration for 'actiongroups' created or updated

  Done with success

      - can you start ES with debug logging? We would need the

debug outputs from ES and SG while starting the node, and when
the error occurs.

To enable debug logging, in your log4j2.properties, add:

logger.searchguard.name = com.floragunn

      logger.searchguard.level = debugOn Monday, September 17, 2018

at

And set the ES logger to debug as well:

logger.action.name = org.elasticsearch.action

logger.action.level = debug

Sure! The gist is here:

Thank you!

···

https://mes-any-logwfe-dev001.qa.server.lan:9200/.monitoring-es-6-2018.09.14/_search?pretty
https://gist.github.com/gna582/354a9205457b89aa4fefa1a617b0569d

8

What is the “base version” of this installation? Means: What was the first ES/SG version you installed and started with?
Was it originally a 5.x cluster or did you start with 6.3.0? This is important because if you migrated

from 5.x. to 6.x then .kibana is converted into an alias and maybe that causes the trouble (although it seemed working before upgrading to SG 23.0)

···

On Wednesday, 12 September 2018 18:01:08 UTC+2, be....g@gmx.de wrote:

Dear searchguard,

when upgrading a fully functional test setup from 6.3.0-22.3 to 6.4.0-23.0 I can no longer log into kibana afterwards. The browser just gets an connection reset after waiting for the 30sec tcp timeout. I use proxy authentication via local apache proxy.

It throws the strange errors about .kibana not being found but I can totally see that index:

green open .kibana gd00htRbRUOXEzcyaGLQTQ 1 1 1 0 8kb 4kb

journalctl -f kibana shows an error like this:

Sep 12 17:56:27 mes-any-logwfe-dev001 kibana[15920]: {“type”:“error”,“@timestamp”:“2018-09-12T15:56:27Z”,“tags”:[“warning”,“stats-collection”],“pid”:15920,“level”:“error”,“error”:{“message”:“[security_exception] Unexpected exception indices:data/read/search”,“name”:“Error”,“stack”:“[security_exception] Unexpected exception indices:data/read/search :: {"path":"/.kibana/_search","query":{"ignore_unavailable":true,"filter_path":"aggregations.types.buckets"},"body":"{\"size\":0,\"query\":{\"terms\":{\"type\":[\"dashboard\",\"visualization\",\"search\",\"index-pattern\",\"graph-workspace\",\"timelion-sheet\"]}},\"aggs\":{\"types\":{\"terms\":{\"field\":\"type\",\"size\":6}}}}","statusCode":500,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"Unexpected exception indices:data/read/search\"}],\"type\":\"security_exception\",\"reason\":\"Unexpected exception indices:data/read/search\"},\"status\":500}"}\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:307:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:266:7)\n at HttpConnector. (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)\n at IncomingMessage.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)\n at emitNone (events.js:111:20)\n at IncomingMessage.emit (events.js:208:7)\n at endReadableNT (_stream_readable.js:1064:12)\n at _combinedTickCallback (internal/process/next_tick.js:138:11)\n at process._tickDomainCallback (internal/process/next_tick.js:218:9)”},“message”:“[security_exception] Unexpected exception indices:data/read/search”}

I have tried to purge searchguard and the error disappeared thus posting it here.

If anything is unclear or not verbose enough please feel free to ask.

Thank you

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version

6.4.0-23.0

  • Installed and used enterprise modules, if any

none/default

  • JVM version and operating system version

java version “1.8.0_121”
Java™ SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot™ 64-Bit Server VM (build 25.121-b13, mixed mode)

Debian 8

  • Search Guard configuration files

attached

  • Elasticsearch log messages on debug level

attached

  • Other installed Elasticsearch or Kibana plugins, if any

sudo ./kibana-plugin list
searchguard@6.4.0-14

sudo ./elasticsearch-plugin list
search-guard-6

9

The first SG version was definitely 6.3.0-22.3. The cluster was
deployed with 6.3.0 initially and was upgraded to 6.4.0-23.0.

···

On 9/21/18 3:34 PM, Search Guard wrote:

    What is the "base version" of this installation?

Means: What was the first ES/SG version you installed and
started with?
Was it originally a 5.x cluster or did you start with
6.3.0? This is important because if you migrated

      from 5.x. to 6.x then .kibana is converted into an alias

and maybe that causes the trouble (although it seemed working
before upgrading to SG 23.0)

      On Wednesday, 12 September 2018 18:01:08 UTC+2, wrote:

–

  You received this message because you are subscribed to the Google

Groups “Search Guard Community Forum” group.

  To unsubscribe from this group and stop receiving emails from it,

send an email to search-guard+unsubscribe@googlegroups.com.

  To post to this group, send email to search-guard@googlegroups.com.

  To view this discussion on the web visit [https://groups.google.com/d/msgid/search-guard/9e32db2b-f3b2-4e6c-95a7-98984138ec4c%40googlegroups.com](https://groups.google.com/d/msgid/search-guard/9e32db2b-f3b2-4e6c-95a7-98984138ec4c%40googlegroups.com?utm_medium=email&utm_source=footer).

  For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

be....g@gmx.de

Dear searchguard,

            when upgrading a fully functional test setup from

6.3.0-22.3 to 6.4.0-23.0 I can no longer log into kibana
afterwards. The browser just gets an connection reset
after waiting for the 30sec tcp timeout. I use proxy
authentication via local apache proxy.

            It throws the strange errors about .kibana not being

found but I can totally see that index:

green open .kibana
gd00htRbRUOXEzcyaGLQTQ 1 1 1 0 8kb 4kb

journalctl -f kibana shows an error like this:

            Sep 12 17:56:27 mes-any-logwfe-dev001 kibana[15920]:

{“type”:“error”,“@timestamp”:“2018-09-12T15:56:27Z”,“tags”:[“warning”,“stats-collection”],“pid”:15920,“level”:“error”,“error”:{“message”:“[security_ exception]
Unexpected exception indices:data/read/search”,“name”:“Error”,“stack”:“[ security_exception]
Unexpected exception indices:data/read/search ::
{"path":"/.kibana/_search","query":{"ignore_unavailable":true,"filter_path":"aggregations.types.buckets"},"body":"{\"size\":0,\"query\":{\"terms\":{\"type\":[\"dashboard\",\"visualization\",\"search\",\"index-pattern\",\"graph-workspace\",\"timelion-sheet\"]}},\"aggs\":{\"types\":{\"terms\":{\"field\":\"type\",\"size\":6}}}}","statusCode":500,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\" reason\":\"Unexpected
exception indices:data/read/search\"}],\"type\":\"security_exception\",\"reason\":\ \“Unexpected
exception indices:data/read/search\"}, \"status\":500}"}\n
at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/ transport.js:307:15)\n
at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/ transport.js:266:7)\n
at HttpConnector.
(/usr/share/kibana/node_modules/elasticsearch/src/lib/ connectors/http.js:159:7)\n
at IncomingMessage.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js: 729:21)\n
at emitNone (events.js:111:20)\n at
IncomingMessage.emit (events.js:208:7)\n at
endReadableNT (_stream_readable.js:1064:12)\ n at
_combinedTickCallback (internal/process/next_tick. js:138:11)\n
at process._tickDomainCallback
(internal/process/next_tick.js:218:9)”},“message”:”[ security_exception]
Unexpected exception indices:data/read/search"}

            I have tried to purge searchguard and the error

disappeared thus posting it here.

            If anything is unclear or not verbose enough please

feel free to ask.

Thank you

            When asking questions, please provide the following

information:

  • Search Guard and Elasticsearch version

6.4.0-23.0

  • Installed and used enterprise modules, if any

none/default

  • JVM version and operating system version

java version “1.8.0_121”

              Java(TM) SE Runtime Environment (build 1.8.0_121-b13)

              Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13,

mixed mode)

Debian 8

  • Search Guard configuration files

attached

  • Elasticsearch log messages on debug level

attached

            * Other installed Elasticsearch or Kibana plugins, if

any

sudo ./kibana-plugin list

sudo ./elasticsearch-plugin list

            search-guard-6

searchguard@6.4.0-14

10

another question with regards to sg_config.yml:

you should not have an entry like

'.kibana':
      '*':
        - INDICES_ALL

because dots are not permitted here.

Where did this entry come from?

Pls. remove this entry and try again.

Therefore we use something like:

    '?kibana':
      '*':
        - INDICES_ALL

sg_kibana_server:
  readonly: true
  cluster:
    - CLUSTER_MONITOR
    - CLUSTER_COMPOSITE_OPS
    - cluster:admin/xpack/monitoring/*
    - indices:admin/template/*
  indices:
    '?kibana':
      '*':
        - INDICES_ALL
    '.kibana':
      '*':
        - INDICES_ALL
        # - indices:admin/template/put
    '?kibana-6':
      '*':
        - INDICES_ALL
    '?reporting*':
      '*':
        - INDICES_ALL
    '?monitoring*':
      '*':
        - INDICES_ALL
···

On Wednesday, 12 September 2018 18:01:08 UTC+2, benedikt haug wrote:

Dear searchguard,

when upgrading a fully functional test setup from 6.3.0-22.3 to 6.4.0-23.0 I can no longer log into kibana afterwards. The browser just gets an connection reset after waiting for the 30sec tcp timeout. I use proxy authentication via local apache proxy.

It throws the strange errors about .kibana not being found but I can totally see that index:

green open .kibana gd00htRbRUOXEzcyaGLQTQ 1 1 1 0 8kb 4kb

journalctl -f kibana shows an error like this:

Sep 12 17:56:27 mes-any-logwfe-dev001 kibana[15920]: {“type”:“error”,“@timestamp”:“2018-09-12T15:56:27Z”,“tags”:[“warning”,“stats-collection”],“pid”:15920,“level”:“error”,“error”:{“message”:“[security_exception] Unexpected exception indices:data/read/search”,“name”:“Error”,“stack”:“[security_exception] Unexpected exception indices:data/read/search :: {"path":"/.kibana/_search","query":{"ignore_unavailable":true,"filter_path":"aggregations.types.buckets"},"body":"{\"size\":0,\"query\":{\"terms\":{\"type\":[\"dashboard\",\"visualization\",\"search\",\"index-pattern\",\"graph-workspace\",\"timelion-sheet\"]}},\"aggs\":{\"types\":{\"terms\":{\"field\":\"type\",\"size\":6}}}}","statusCode":500,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"Unexpected exception indices:data/read/search\"}],\"type\":\"security_exception\",\"reason\":\"Unexpected exception indices:data/read/search\"},\"status\":500}"}\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:307:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:266:7)\n at HttpConnector. (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)\n at IncomingMessage.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)\n at emitNone (events.js:111:20)\n at IncomingMessage.emit (events.js:208:7)\n at endReadableNT (_stream_readable.js:1064:12)\n at _combinedTickCallback (internal/process/next_tick.js:138:11)\n at process._tickDomainCallback (internal/process/next_tick.js:218:9)”},“message”:“[security_exception] Unexpected exception indices:data/read/search”}

I have tried to purge searchguard and the error disappeared thus posting it here.

If anything is unclear or not verbose enough please feel free to ask.

Thank you

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version

6.4.0-23.0

  • Installed and used enterprise modules, if any

none/default

  • JVM version and operating system version

java version “1.8.0_121”
Java™ SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot™ 64-Bit Server VM (build 25.121-b13, mixed mode)

Debian 8

  • Search Guard configuration files

attached

  • Elasticsearch log messages on debug level

attached

  • Other installed Elasticsearch or Kibana plugins, if any

sudo ./kibana-plugin list
searchguard@6.4.0-14

sudo ./elasticsearch-plugin list
search-guard-6

11

You are correct! Thank you! This solved the strange .kibana
issues. I have consequently reviewed all diffs to upstream config
so this doesn’t happen again. Sry.

  Sadly I can still not log in via proxy authentication due to: "no

xff done for class
org.elasticsearch.http.netty4.Netty4HttpRequest". The same config
works fine with 6.3.0-22.3

  I think I have found the cause of this to be the new Netty

4.1.25.Final release newly introduced in ES 6.4.0 after I realized
there is a trace debug level and had a look at the code.

  Would it be possible for you to have a look and check whether

proxy authentication is still working as intended with ES 6.4.0? I
made a Gist where I have commented what I think goes wrong in the
RemoteIpDetector.java:

Thank you for your help!

···

https://gist.github.com/gna582/cc8f33835054b71cb44d2f6cba4d8765
On 9/21/18 3:48 PM, Search Guard wrote:

another question with regards to sg_config.yml:

you should not have an entry like

'.kibana':
      '*':
        - INDICES_ALL

because dots are not permitted here.

Where did this entry come from?

Pls. remove this entry and try again.

Therefore we use something like:

    '?kibana':
      '*':
        - INDICES_ALL

sg_kibana_server:
  readonly:   true
cluster:
        - CLUSTER_MONITOR
- CLUSTER_COMPOSITE_OPS
- cluster:admin/xpack/monitoring/*
- indices:admin/template/*
indices:
    '?kibana':
      '*':
        - INDICES_ALL
    '.kibana':
      '*':
                - INDICES_ALL
# - indices:admin/template/put
    '?kibana-6':
      '*':
            - INDICES_ALL
'?reporting*':
      '*':
            - INDICES_ALL
'?monitoring*':
      '*':
        - INDICES_ALL

      On Wednesday, 12 September 2018 18:01:08 UTC+2, benedikt haug

wrote:

Dear searchguard,

            when upgrading a fully functional test setup from

6.3.0-22.3 to 6.4.0-23.0 I can no longer log into kibana
afterwards. The browser just gets an connection reset
after waiting for the 30sec tcp timeout. I use proxy
authentication via local apache proxy.

            It throws the strange errors about .kibana not being

found but I can totally see that index:

green open .kibana
gd00htRbRUOXEzcyaGLQTQ 1 1 1 0 8kb 4kb

journalctl -f kibana shows an error like this:

            Sep 12 17:56:27 mes-any-logwfe-dev001 kibana[15920]:

{“type”:“error”,“@timestamp”:“2018-09-12T15:56:27Z”,“tags”:[“warning”,“stats-collection”],“pid”:15920,“level”:“error”,“error”:{“message”:“[security_ exception]
Unexpected exception indices:data/read/search”,“name”:“Error”,“stack”:“[ security_exception]
Unexpected exception indices:data/read/search ::
{"path":"/.kibana/_search","query":{"ignore_unavailable":true,"filter_path":"aggregations.types.buckets"},"body":"{\"size\":0,\"query\":{\"terms\":{\"type\":[\"dashboard\",\"visualization\",\"search\",\"index-pattern\",\"graph-workspace\",\"timelion-sheet\"]}},\"aggs\":{\"types\":{\"terms\":{\"field\":\"type\",\"size\":6}}}}","statusCode":500,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\" reason\":\"Unexpected
exception indices:data/read/search\"}],\"type\":\"security_exception\",\"reason\":\ \“Unexpected
exception indices:data/read/search\"}, \"status\":500}"}\n
at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/ transport.js:307:15)\n
at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/ transport.js:266:7)\n
at HttpConnector.
(/usr/share/kibana/node_modules/elasticsearch/src/lib/ connectors/http.js:159:7)\n
at IncomingMessage.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js: 729:21)\n
at emitNone (events.js:111:20)\n at
IncomingMessage.emit (events.js:208:7)\n at
endReadableNT (_stream_readable.js:1064:12)\ n at
_combinedTickCallback (internal/process/next_tick. js:138:11)\n
at process._tickDomainCallback
(internal/process/next_tick.js:218:9)”},“message”:”[ security_exception]
Unexpected exception indices:data/read/search"}

            I have tried to purge searchguard and the error

disappeared thus posting it here.

            If anything is unclear or not verbose enough please

feel free to ask.

Thank you

            When asking questions, please provide the following

information:

  • Search Guard and Elasticsearch version

6.4.0-23.0

  • Installed and used enterprise modules, if any

none/default

  • JVM version and operating system version

java version “1.8.0_121”

              Java(TM) SE Runtime Environment (build 1.8.0_121-b13)

              Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13,

mixed mode)

Debian 8

  • Search Guard configuration files

attached

  • Elasticsearch log messages on debug level

attached

            * Other installed Elasticsearch or Kibana plugins, if

any

sudo ./kibana-plugin list

sudo ./elasticsearch-plugin list

            search-guard-6

–

  You received this message because you are subscribed to the Google

Groups “Search Guard Community Forum” group.

  To unsubscribe from this group and stop receiving emails from it,

send an email to search-guard+unsubscribe@googlegroups.com.

  To post to this group, send email to search-guard@googlegroups.com.

  To view this discussion on the web visit [https://groups.google.com/d/msgid/search-guard/2c95a7a4-a85f-4599-8501-a66f95fec1dd%40googlegroups.com](https://groups.google.com/d/msgid/search-guard/2c95a7a4-a85f-4599-8501-a66f95fec1dd%40googlegroups.com?utm_medium=email&utm_source=footer).

  For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

searchguard@6.4.0-14

12

We test every release thoroughly with a fully automated integration test suite also containing xff/proxy tests with nginx.

That said are you sure its nothing related to proxy/sg_config configuration? Does it work with recent 6.5.x builds?

Proxy auth is used widely spread and we received no other complaints so far.

···

On Wednesday, 26 September 2018 15:43:12 UTC+2, Benedikt Haug wrote:

  You are correct! Thank you! This solved the strange .kibana

issues. I have consequently reviewed all diffs to upstream config
so this doesn’t happen again. Sry.

  Sadly I can still not log in via proxy authentication due to: "no

xff done for class
org.elasticsearch.http.netty4. Netty4HttpRequest". The same config
works fine with 6.3.0-22.3

  I think I have found the cause of this to be the new Netty

4.1.25.Final release newly introduced in ES 6.4.0 after I realized
there is a trace debug level and had a look at the code.

  Would it be possible for you to have a look and check whether

proxy authentication is still working as intended with ES 6.4.0? I
made a Gist where I have commented what I think goes wrong in the
RemoteIpDetector.java:

https://gist.github.com/gna582/cc8f33835054b71cb44d2f6cba4d8765

Thank you for your help!

On 9/21/18 3:48 PM, Search Guard wrote:

another question with regards to sg_config.yml:

you should not have an entry like

'.kibana':
      '*':
        - INDICES_ALL

because dots are not permitted here.

Where did this entry come from?

Pls. remove this entry and try again.

Therefore we use something like:

    '?kibana':
      '*':
        - INDICES_ALL

sg_kibana_server:
  readonly:   true
cluster:
        - CLUSTER_MONITOR
- CLUSTER_COMPOSITE_OPS
- cluster:admin/xpack/    monitoring/*
- indices:admin/template/*
indices:
    '?kibana':
      '*':
        - INDICES_ALL
    '.kibana':
      '*':
                - INDICES_ALL
# - indices:admin/template/put
    '?kibana-6':
      '*':
            - INDICES_ALL
'?reporting*':
      '*':
            - INDICES_ALL
'?monitoring*':
      '*':
        - INDICES_ALL

      On Wednesday, 12 September 2018 18:01:08 UTC+2, benedikt haug > > wrote:

Dear searchguard,

            when upgrading a fully functional test setup from

6.3.0-22.3 to 6.4.0-23.0 I can no longer log into kibana
afterwards. The browser just gets an connection reset
after waiting for the 30sec tcp timeout. I use proxy
authentication via local apache proxy.

            It throws the strange errors about .kibana not being

found but I can totally see that index:

green open .kibana
gd00htRbRUOXEzcyaGLQTQ 1 1 1 0 8kb 4kb

journalctl -f kibana shows an error like this:

            Sep 12 17:56:27 mes-any-logwfe-dev001 kibana[15920]:

{“type”:“error”,“@timestamp”:“2018-09-12T15:56:27Z”,“tags”:[“warning”,“stats-collection”],“pid”:15920,“level”:“error”,“error”:{“message”:“[security_ exception]
Unexpected exception indices:data/read/search”,“name”:“Error”,“stack”:“[ security_exception]
Unexpected exception indices:data/read/search ::
{"path":"/.kibana/_search","query":{"ignore_unavailable":true,"filter_path":"aggregations.types.buckets"},"body":"{\"size\":0,\"query\":{\"terms\":{\"type\":[\"dashboard\",\"visualization\",\"search\",\"index-pattern\",\"graph-workspace\",\"timelion-sheet\"]}},\"aggs\":{\"types\":{\"terms\":{\"field\":\"type\",\"size\":6}}}}","statusCode":500,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\" reason\":\"Unexpected
exception indices:data/read/search\"}],\"type\":\"security_exception\",\"reason\":\ \“Unexpected
exception indices:data/read/search\"}, \"status\":500}"}\n
at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/ transport.js:307:15)\n
at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/ transport.js:266:7)\n
at HttpConnector.
(/usr/share/kibana/node_modules/elasticsearch/src/lib/ connectors/http.js:159:7)\n
at IncomingMessage.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js: 729:21)\n
at emitNone (events.js:111:20)\n at
IncomingMessage.emit (events.js:208:7)\n at
endReadableNT (_stream_readable.js:1064:12)\ n at
_combinedTickCallback (internal/process/next_tick. js:138:11)\n
at process._tickDomainCallback
(internal/process/next_tick.js:218:9)”},“message”:”[ security_exception]
Unexpected exception indices:data/read/search"}

            I have tried to purge searchguard and the error

disappeared thus posting it here.

            If anything is unclear or not verbose enough please

feel free to ask.

Thank you

            When asking questions, please provide the following

information:

  • Search Guard and Elasticsearch version

6.4.0-23.0

  • Installed and used enterprise modules, if any

none/default

  • JVM version and operating system version

java version “1.8.0_121”

              Java(TM) SE Runtime Environment (build 1.8.0_121-b13)

              Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13,

mixed mode)

Debian 8

  • Search Guard configuration files

attached

  • Elasticsearch log messages on debug level

attached

            * Other installed Elasticsearch or Kibana plugins, if

any

sudo ./kibana-plugin list

            searchguard@6.4.0-14

sudo ./elasticsearch-plugin list

            search-guard-6

–

  You received this message because you are subscribed to the Google

Groups “Search Guard Community Forum” group.

  To unsubscribe from this group and stop receiving emails from it,

send an email to search-guard+unsubscribe@googlegroups.com.

  To post to this group, send email to search-guard@googlegroups.com.

  To view this discussion on the web visit [https://groups.google.com/d/msgid/search-guard/2c95a7a4-a85f-4599-8501-a66f95fec1dd%40googlegroups.com](https://groups.google.com/d/msgid/search-guard/2c95a7a4-a85f-4599-8501-a66f95fec1dd%40googlegroups.com?utm_medium=email&utm_source=footer).

  For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).