Kibana Monitoring page not working after 23.0 to 23.1 upgrade

Hello

Having recently upgraded from SearchGuard 23.0 > 23.1 on Elasticsearch 6.32 - None of my admin users can view the monitoring status any longer.
(previous post: https://groups.google.com/forum/#!topic/search-guard/5kRomzJyZFk)

Trying to access via Kibana: https://SERVER:5601/app/monitoring - Displays the message “You are not authorized to access Monitoring. To use Monitoring you need the privileges granted by both the kibana_user and monitoring_user roles…”

This happens to both the inbuilt (internal) ‘admin’ user and backend roles mapped against the ‘sg_all_access’ SearchGuard role.

Reviewing sg_internal_users sg_roles_mapping and sg_roles I’m pretty Admin is OK.

I even enabled the search_guard kibana GUI and checked in that. Admin looks OK.

Previously when Elastic had permission errors, it used to output info in the logs, how ever this is not happening in this case.

Any ideas on this?

Does Kibana need a different SG plugin if I upgrade Elastic SG from 23.0 to 23.1 ?

Thanks

No, you don’t need to upgrad the Kibana plugin when moving from 23.0 to 23.1. Are the Elasticsearch logs show anything when trying to access the monitoring page?

···

On Wednesday, October 24, 2018 at 7:41:14 AM UTC+2, Cam McKenzie wrote:

Hello

Having recently upgraded from SearchGuard 23.0 > 23.1 on Elasticsearch 6.32 - None of my admin users can view the monitoring status any longer.
(previous post: https://groups.google.com/forum/#!topic/search-guard/5kRomzJyZFk)

Trying to access via Kibana: https://SERVER:5601/app/monitoring - Displays the message “You are not authorized to access Monitoring. To use Monitoring you need the privileges granted by both the kibana_user and monitoring_user roles…”

This happens to both the inbuilt (internal) ‘admin’ user and backend roles mapped against the ‘sg_all_access’ SearchGuard role.

Reviewing sg_internal_users sg_roles_mapping and sg_roles I’m pretty Admin is OK.

I even enabled the search_guard kibana GUI and checked in that. Admin looks OK.

Previously when Elastic had permission errors, it used to output info in the logs, how ever this is not happening in this case.

Any ideas on this?

Does Kibana need a different SG plugin if I upgrade Elastic SG from 23.0 to 23.1 ?

Thanks

Ok in an attempt to fix the issue I upgraded my whole stack to Elastic / Kibana 6.4.2 with SearchGuard 23.1 and SearchGuard “6.4.2-15” for Kibana

So here is what I have found out:

Unknowingly to me, when I enabled:
‘do_not_fail_on_forbidden: true’

It broke Kibana Monitoring (https://KIBANASERVER/app/monitoring)

If ‘do_not_fail_on_forbidden: true’ is set and an
administrator tries to access the monitoring they get the following screenshot:

There are no errors reported in Elasticsearch at the time of
error.

There are errors reporting in the Kibana log, relating to
the error – This is the message:

25/10/2018 13:24:49{“type”:“error”,"@timestamp":“2018-10-25T03:24:49Z”,“tags”:[“warning”,“stats-collection”],“pid”:1,“level”:“error”,“error”:{“message”:"[security_exception]
no permissions for [indices:data/read/search] and User [name=kibanaserver,
roles=[kibanaserver], requestedTenant=null]",“name”:“Error”,“stack”:"[security_exception]
no permissions for [indices:data/read/search] and User [name=kibanaserver,
roles=[kibanaserver], requestedTenant=null] :: {“path”:"/.reporting-*/_search",“query”:{“filter_path”:“hits.total,aggregations.jobTypes.buckets,aggregations.objectTypes.buckets,aggregations.layoutTypes.buckets,aggregations.statusTypes.buckets”},“body”:"{\“size\”:0,\“aggs\”:{\“jobTypes\”:{\“terms\”:{\“field\”:\“jobtype\”,\“size\”:2}},\“objectTypes\”:{\“terms\”:{\“field\”:\“meta.objectType.keyword\”,\“size\”:3}},\“layoutTypes\”:{\“terms\”:{\“field\”:\“meta.layout.keyword\”,\“size\”:3}},\“statusTypes\”:{\“terms\”:{\“field\”:\“status\”,\“size\”:4}}}}",“statusCode”:403,“response”:"{\“error\”:{\“root_cause\”:[{\“type\”:\“security_exception\”,\“reason\”:\“no
permissions for [indices:data/read/search] and User [name=kibanaserver,
roles=[kibanaserver], requestedTenant=null]\”}],\“type\”:\“security_exception\”,\“reason\”:\“no
permissions for [indices:data/read/search] and User [name=kibanaserver,
roles=[kibanaserver], requestedTenant=null]\”},\“status\”:403}"}\n
at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:307:15)\n
at checkRespForFailure
(/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:266:7)\n
at HttpConnector. (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)\n
at IncomingMessage.bound
(/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)\n
at emitNone (events.js:111:20)\n at IncomingMessage.emit (events.js:208:7)\n
at endReadableNT (_stream_readable.js:1064:12)\n at
_combinedTickCallback
(internal/process/next_tick.js:138:11)\n at
process._tickCallback
(internal/process/next_tick.js:180:9)"},“message”:"[security_exception]
no permissions for [indices:data/read/search] and User [name=kibanaserver,
roles=[kibanaserver], requestedTenant=null]"}

25/10/2018
13:24:49{“type”:“log”,"@timestamp":“2018-10-25T03:24:49Z”,“tags”:[“warning”,“stats-collection”],“pid”:1,“message”:“Unable
to fetch data from reporting collector”}

It seems DIRECTLY related to setting:
‘do_not_fail_on_forbidden: true’ because if I set it to ‘false’, the monitoring
app works correctly.

Can you confirm this bug? How can I fix it?

Thanks

Thanks for reporting this! We need to investigate the behavior here locally. At the moment I don’t see how the dnfof setting can affect the access check in the monitoring app. I would have expected it to be the other way round: Access checks fail when dnfof is false, but your test proves otherwise. I will be back once we know what is going on here.

···

On Thursday, October 25, 2018 at 7:40:13 AM UTC+2, Cam McKenzie wrote:

Ok in an attempt to fix the issue I upgraded my whole stack to Elastic / Kibana 6.4.2 with SearchGuard 23.1 and SearchGuard “6.4.2-15” for Kibana

So here is what I have found out:

Unknowingly to me, when I enabled:
‘do_not_fail_on_forbidden: true’

It broke Kibana Monitoring (https://KIBANASERVER/app/monitoring)

If ‘do_not_fail_on_forbidden: true’ is set and an
administrator tries to access the monitoring they get the following screenshot:

There are no errors reported in Elasticsearch at the time of
error.

There are errors reporting in the Kibana log, relating to
the error – This is the message:

25/10/2018 13:24:49{“type”:“error”,"@timestamp":“2018-10-25T03:24:49Z”,“tags”:[“warning”,“stats-collection”],“pid”:1,“level”:“error”,“error”:{“message”:"[security_exception]
no permissions for [indices:data/read/search] and User [name=kibanaserver,
roles=[kibanaserver], requestedTenant=null]",“name”:“Error”,“stack”:"[security_exception]
no permissions for [indices:data/read/search] and User [name=kibanaserver,
roles=[kibanaserver], requestedTenant=null] :: {“path”:"/.reporting-*/_search",“query”:{“filter_path”:“hits.total,aggregations.jobTypes.buckets,aggregations.objectTypes.buckets,aggregations.layoutTypes.buckets,aggregations.statusTypes.buckets”},“body”:"{\“size\”:0,\“aggs\”:{\“jobTypes\”:{\“terms\”:{\“field\”:\“jobtype\”,\“size\”:2}},\“objectTypes\”:{\“terms\”:{\“field\”:\“meta.objectType.keyword\”,\“size\”:3}},\“layoutTypes\”:{\“terms\”:{\“field\”:\“meta.layout.keyword\”,\“size\”:3}},\“statusTypes\”:{\“terms\”:{\“field\”:\“status\”,\“size\”:4}}}}",“statusCode”:403,“response”:"{\“error\”:{\“root_cause\”:[{\“type\”:\“security_exception\”,\“reason\”:\“no
permissions for [indices:data/read/search] and User [name=kibanaserver,
roles=[kibanaserver], requestedTenant=null]\”}],\“type\”:\“security_exception\”,\“reason\”:\“no
permissions for [indices:data/read/search] and User [name=kibanaserver,
roles=[kibanaserver], requestedTenant=null]\”},\“status\”:403}"}\n
at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:307:15)\n
at checkRespForFailure
(/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:266:7)\n
at HttpConnector. (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)\n
at IncomingMessage.bound
(/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)\n
at emitNone (events.js:111:20)\n at IncomingMessage.emit (events.js:208:7)\n
at endReadableNT (_stream_readable.js:1064:12)\n at
_combinedTickCallback
(internal/process/next_tick.js:138:11)\n at
process._tickCallback
(internal/process/next_tick.js:180:9)"},“message”:"[security_exception]
no permissions for [indices:data/read/search] and User [name=kibanaserver,
roles=[kibanaserver], requestedTenant=null]"}

25/10/2018
13:24:49{“type”:“log”,"@timestamp":“2018-10-25T03:24:49Z”,“tags”:[“warning”,“stats-collection”],“pid”:1,“message”:“Unable
to fetch data from reporting collector”}

It seems DIRECTLY related to setting:
‘do_not_fail_on_forbidden: true’ because if I set it to ‘false’, the monitoring
app works correctly.

Can you confirm this bug? How can I fix it?

Thanks

Hi Cameron,

I can confirm this is a bug that was introduced by a change in 23.1 and that surfaces depending on what indices are available on your cluster, and how monitoring is configured.

We are working on a fix at the moment which we can hopefully ship as a snapshot soon. The only known workaround at the moment is to create empty indices for the wildcard checks that Kibana performs when using the Monitoring app, like:

curl -u … -k -X PUT “https://localhost:9200/.monitoring-beats-6-2018.10.25
curl -u … -k -X PUT “https://localhost:9200/.monitoring-logstash-6-2018.10.25
curl -u … -k -X PUT “https://localhost:9200/.monitoring-kibana-6-2018.10.25
curl -u … -k -X PUT “https://localhost:9200/.reporting-2018.10.25
curl -u … -k -X PUT “https://localhost:9200/.monitoring-es-6-2018.10.25

``

I know this is not elegant, but until the snapshot has been released this seems to be the only workaround.

···

On Thursday, October 25, 2018 at 11:57:38 AM UTC+2, Jochen Kressin wrote:

Thanks for reporting this! We need to investigate the behavior here locally. At the moment I don’t see how the dnfof setting can affect the access check in the monitoring app. I would have expected it to be the other way round: Access checks fail when dnfof is false, but your test proves otherwise. I will be back once we know what is going on here.

On Thursday, October 25, 2018 at 7:40:13 AM UTC+2, Cam McKenzie wrote:

Ok in an attempt to fix the issue I upgraded my whole stack to Elastic / Kibana 6.4.2 with SearchGuard 23.1 and SearchGuard “6.4.2-15” for Kibana

So here is what I have found out:

Unknowingly to me, when I enabled:
‘do_not_fail_on_forbidden: true’

It broke Kibana Monitoring (https://KIBANASERVER/app/monitoring)

If ‘do_not_fail_on_forbidden: true’ is set and an
administrator tries to access the monitoring they get the following screenshot:

There are no errors reported in Elasticsearch at the time of
error.

There are errors reporting in the Kibana log, relating to
the error – This is the message:

25/10/2018 13:24:49{“type”:“error”,"@timestamp":“2018-10-25T03:24:49Z”,“tags”:[“warning”,“stats-collection”],“pid”:1,“level”:“error”,“error”:{“message”:"[security_exception]
no permissions for [indices:data/read/search] and User [name=kibanaserver,
roles=[kibanaserver], requestedTenant=null]",“name”:“Error”,“stack”:"[security_exception]
no permissions for [indices:data/read/search] and User [name=kibanaserver,
roles=[kibanaserver], requestedTenant=null] :: {“path”:"/.reporting-*/_search",“query”:{“filter_path”:“hits.total,aggregations.jobTypes.buckets,aggregations.objectTypes.buckets,aggregations.layoutTypes.buckets,aggregations.statusTypes.buckets”},“body”:"{\“size\”:0,\“aggs\”:{\“jobTypes\”:{\“terms\”:{\“field\”:\“jobtype\”,\“size\”:2}},\“objectTypes\”:{\“terms\”:{\“field\”:\“meta.objectType.keyword\”,\“size\”:3}},\“layoutTypes\”:{\“terms\”:{\“field\”:\“meta.layout.keyword\”,\“size\”:3}},\“statusTypes\”:{\“terms\”:{\“field\”:\“status\”,\“size\”:4}}}}",“statusCode”:403,“response”:"{\“error\”:{\“root_cause\”:[{\“type\”:\“security_exception\”,\“reason\”:\“no
permissions for [indices:data/read/search] and User [name=kibanaserver,
roles=[kibanaserver], requestedTenant=null]\”}],\“type\”:\“security_exception\”,\“reason\”:\“no
permissions for [indices:data/read/search] and User [name=kibanaserver,
roles=[kibanaserver], requestedTenant=null]\”},\“status\”:403}"}\n
at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:307:15)\n
at checkRespForFailure
(/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:266:7)\n
at HttpConnector. (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)\n
at IncomingMessage.bound
(/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)\n
at emitNone (events.js:111:20)\n at IncomingMessage.emit (events.js:208:7)\n
at endReadableNT (_stream_readable.js:1064:12)\n at
_combinedTickCallback
(internal/process/next_tick.js:138:11)\n at
process._tickCallback
(internal/process/next_tick.js:180:9)"},“message”:"[security_exception]
no permissions for [indices:data/read/search] and User [name=kibanaserver,
roles=[kibanaserver], requestedTenant=null]"}

25/10/2018
13:24:49{“type”:“log”,"@timestamp":“2018-10-25T03:24:49Z”,“tags”:[“warning”,“stats-collection”],“pid”:1,“message”:“Unable
to fetch data from reporting collector”}

It seems DIRECTLY related to setting:
‘do_not_fail_on_forbidden: true’ because if I set it to ‘false’, the monitoring
app works correctly.

Can you confirm this bug? How can I fix it?

Thanks

Can you try https://oss.sonatype.org/content/repositories/snapshots/com/floragunn/search-guard-6/6.4.2-23.2-SNAPSHOT/search-guard-6-6.4.2-23.2-20181025.200751-1.zip ?

···

Am 25.10.2018 um 20:54 schrieb Jochen Kressin <jkressin@floragunn.com>:

Hi Cameron,

I can confirm this is a bug that was introduced by a change in 23.1 and that surfaces depending on what indices are available on your cluster, and how monitoring is configured.

We are working on a fix at the moment which we can hopefully ship as a snapshot soon. The only known workaround at the moment is to create empty indices for the wildcard checks that Kibana performs when using the Monitoring app, like:

curl -u ... -k -X PUT "https://localhost:9200/.monitoring-beats-6-2018.10.25"
curl -u ... -k -X PUT "https://localhost:9200/.monitoring-logstash-6-2018.10.25"
curl -u ... -k -X PUT "https://localhost:9200/.monitoring-kibana-6-2018.10.25"
curl -u ... -k -X PUT "https://localhost:9200/.reporting-2018.10.25"
curl -u ... -k -X PUT "https://localhost:9200/.monitoring-es-6-2018.10.25"

I know this is not elegant, but until the snapshot has been released this seems to be the only workaround.

On Thursday, October 25, 2018 at 11:57:38 AM UTC+2, Jochen Kressin wrote:
Thanks for reporting this! We need to investigate the behavior here locally. At the moment I don't see how the dnfof setting can affect the access check in the monitoring app. I would have expected it to be the other way round: Access checks fail when dnfof is false, but your test proves otherwise. I will be back once we know what is going on here.

On Thursday, October 25, 2018 at 7:40:13 AM UTC+2, Cam McKenzie wrote:
Ok in an attempt to fix the issue I upgraded my whole stack to Elastic / Kibana 6.4.2 with SearchGuard 23.1 and SearchGuard "6.4.2-15" for Kibana

So here is what I have found out:

Unknowingly to me, when I enabled: ‘do_not_fail_on_forbidden: true’

It broke Kibana Monitoring (https://KIBANASERVER/app/monitoring)

If ‘do_not_fail_on_forbidden: true’ is set and an administrator tries to access the monitoring they get the following screenshot:

There are no errors reported in Elasticsearch at the time of error.

There are errors reporting in the Kibana log, relating to the error – This is the message:

25/10/2018 13:24:49{"type":"error","@timestamp":"2018-10-25T03:24:49Z","tags":["warning","stats-collection"],"pid":1,"level":"error","error":{"message":"[security_exception] no permissions for [indices:data/read/search] and User [name=kibanaserver, roles=[kibanaserver], requestedTenant=null]","name":"Error","stack":"[security_exception] no permissions for [indices:data/read/search] and User [name=kibanaserver, roles=[kibanaserver], requestedTenant=null] :: {\"path\":\"/.reporting-*/_search\",\"query\":{\"filter_path\":\"hits.total,aggregations.jobTypes.buckets,aggregations.objectTypes.buckets,aggregations.layoutTypes.buckets,aggregations.statusTypes.buckets\"},\"body\":\"{\\\"size\\\":0,\\\"aggs\\\":{\\\"jobTypes\\\":{\\\"terms\\\":{\\\"field\\\":\\\"jobtype\\\",\\\"size\\\":2}},\\\"objectTypes\\\":{\\\"terms\\\":{\\\"field\\\":\\\"meta.objectType.keyword\\\",\\\"size\\\":3}},\\\"layoutTypes\\\":{\\\"terms\\\":{\\\"field\\\":\\\"meta.layout.keyword\\\",\\\"size\\\":3}},\\\"statusTypes\\\":{\\\"terms\\\":{\\\"field\\\":\\\"status\\\",\\\"size\\\":4}}}}\",\"statusCode\":403,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"no permissions for [indices:data/read/search] and User [name=kibanaserver, roles=[kibanaserver], requestedTenant=null]\\\"}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"no permissions for [indices:data/read/search] and User [name=kibanaserver, roles=[kibanaserver], requestedTenant=null]\\\"},\\\"status\\\":403}\"}\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:307:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:266:7)\n at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)\n at IncomingMessage.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)\n at emitNone (events.js:111:20)\n at IncomingMessage.emit (events.js:208:7)\n at endReadableNT (_stream_readable.js:1064:12)\n at _combinedTickCallback (internal/process/next_tick.js:138:11)\n at process._tickCallback (internal/process/next_tick.js:180:9)"},"message":"[security_exception] no permissions for [indices:data/read/search] and User [name=kibanaserver, roles=[kibanaserver], requestedTenant=null]"}

25/10/2018 13:24:49{"type":"log","@timestamp":"2018-10-25T03:24:49Z","tags":["warning","stats-collection"],"pid":1,"message":"Unable to fetch data from reporting collector"}

It seems DIRECTLY related to setting: ‘do_not_fail_on_forbidden: true’ because if I set it to 'false', the monitoring app works correctly.

Can you confirm this bug? How can I fix it?

Thanks

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/83516836-1413-4a47-bab3-e850bb2bf538%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Sorry - only just saw this. I’ll have to try it on Monday.

···

On Friday, 26 October 2018 06:11:30 UTC+10, Search Guard wrote:

Can you try https://oss.sonatype.org/content/repositories/snapshots/com/floragunn/search-guard-6/6.4.2-23.2-SNAPSHOT/search-guard-6-6.4.2-23.2-20181025.200751-1.zip ?

Hello,

Essentially that plugin installs, but ElasticSearch fails to start with the error:

9/10/2018 10:02:01[2018-10-29T00:02:01,402][INFO ][o.e.n.Node ] [wa-knsg-elkdev3.acme.local] initializing …
29/10/2018 10:02:01[2018-10-29T00:02:01,608][INFO ][o.e.e.NodeEnvironment ] [wa-knsg-elkdev3.acme.local] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/vda7)]], net usable_space [43.4gb], net total_space [54.9gb], types [xfs]
29/10/2018 10:02:01[2018-10-29T00:02:01,608][INFO ][o.e.e.NodeEnvironment ] [wa-knsg-elkdev3.acme.local] heap size [3.9gb], compressed ordinary object pointers [true]
29/10/2018 10:02:01[2018-10-29T00:02:01,713][INFO ][o.e.n.Node ] [wa-knsg-elkdev3.acme.local] node name [wa-knsg-elkdev3.acme.local], node ID [tIIl1VSvSdiXW35CohSY1A]
29/10/2018 10:02:01[2018-10-29T00:02:01,714][INFO ][o.e.n.Node ] [wa-knsg-elkdev3.acme.local] version[6.4.2], pid[1], build[default/tar/04711c2/2018-09-26T13:34:09.098244Z], OS[Linux/3.10.0-862.9.1.el7.x86_64/amd64], JVM[“Oracle Corporation”/OpenJDK 64-Bit Server VM/10.0.2/10.0.2+13]
29/10/2018 10:02:01[2018-10-29T00:02:01,714][INFO ][o.e.n.Node ] [wa-knsg-elkdev3.acme.local] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.RIaV0zdg, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Des.cgroups.hierarchy.override=/, -Xms4g, -Xmx4g, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
29/10/2018 10:02:01[2018-10-29T00:02:01,759][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [wa-knsg-elkdev3.acme.local] uncaught exception in thread [main]
29/10/2018 10:02:01org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Could not load plugin descriptor for plugin directory [search-guard-6]
29/10/2018 10:02:01 at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01Caused by: java.lang.IllegalStateException: Could not load plugin descriptor for plugin directory [search-guard-6]
29/10/2018 10:02:01 at org.elasticsearch.plugins.PluginsService.readPluginBundle(PluginsService.java:394) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.plugins.PluginsService.findBundles(PluginsService.java:379) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.plugins.PluginsService.getPluginBundles(PluginsService.java:372) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.plugins.PluginsService.(PluginsService.java:144) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.node.Node.(Node.java:315) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.node.Node.(Node.java:256) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:213) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 … 6 more
29/10/2018 10:02:01Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/plugins/search-guard-6/plugin-descriptor.properties
29/10/2018 10:02:01 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
29/10/2018 10:02:01 at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
29/10/2018 10:02:01 at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
29/10/2018 10:02:01 at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:215) ~[?:?]
29/10/2018 10:02:01 at java.nio.file.Files.newByteChannel(Files.java:369) ~[?:?]
29/10/2018 10:02:01 at java.nio.file.Files.newByteChannel(Files.java:415) ~[?:?]
29/10/2018 10:02:01 at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) ~[?:?]
29/10/2018 10:02:01 at java.nio.file.Files.newInputStream(Files.java:154) ~[?:?]
29/10/2018 10:02:01 at org.elasticsearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:162) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.plugins.PluginsService.readPluginBundle(PluginsService.java:391) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.plugins.PluginsService.findBundles(PluginsService.java:379) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.plugins.PluginsService.getPluginBundles(PluginsService.java:372) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.plugins.PluginsService.(PluginsService.java:144) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.node.Node.(Node.java:315) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.node.Node.(Node.java:256) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:213) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 … 6 more

``

I’m guessing ’/usr/share/elasticsearch/plugins/search-guard-6/plugin-descriptor.properties' is missing from the install.

Are you able to re-add it back to the plugin?

Cannot reproduce.

This works perfect for me:
bin/elasticsearch-plugin remove search-guard-6
bin/elasticsearch-plugin install -b https://oss.sonatype.org/content/repositories/snapshots/com/floragunn/search-guard-6/6.4.2-23.2-SNAPSHOT/search-guard-6-6.4.2-23.2-20181025.200751-1.zip

Maybe you have a file permission issue? See https://www.elastic.co/guide/en/elasticsearch/plugins/current/plugin-management.html

If Elasticsearch was installed using the deb or rpm package then run (with sudo):
sudo bin/elasticsearch-plugin remove search-guard-6
sudo in/elasticsearch-plugin install -b https://oss.sonatype.org/content/repositories/snapshots/com/floragunn/search-guard-6/6.4.2-23.2-SNAPSHOT/search-guard-6-6.4.2-23.2-20181025.200751-1.zip

···

Am 29.10.2018 um 01:09 schrieb Cam McKenzie <cam.mckenzie@gmail.com>:

Hello,

Essentially that plugin installs, but ElasticSearch fails to start with the error:

9/10/2018 10:02:01[2018-10-29T00:02:01,402][INFO ][o.e.n.Node ] [wa-knsg-elkdev3.acme.local] initializing ...
29/10/2018 10:02:01[2018-10-29T00:02:01,608][INFO ][o.e.e.NodeEnvironment ] [wa-knsg-elkdev3.acme.local] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/vda7)]], net usable_space [43.4gb], net total_space [54.9gb], types [xfs]
29/10/2018 10:02:01[2018-10-29T00:02:01,608][INFO ][o.e.e.NodeEnvironment ] [wa-knsg-elkdev3.acme.local] heap size [3.9gb], compressed ordinary object pointers [true]
29/10/2018 10:02:01[2018-10-29T00:02:01,713][INFO ][o.e.n.Node ] [wa-knsg-elkdev3.acme.local] node name [wa-knsg-elkdev3.acme.local], node ID [tIIl1VSvSdiXW35CohSY1A]
29/10/2018 10:02:01[2018-10-29T00:02:01,714][INFO ][o.e.n.Node ] [wa-knsg-elkdev3.acme.local] version[6.4.2], pid[1], build[default/tar/04711c2/2018-09-26T13:34:09.098244Z], OS[Linux/3.10.0-862.9.1.el7.x86_64/amd64], JVM["Oracle Corporation"/OpenJDK 64-Bit Server VM/10.0.2/10.0.2+13]
29/10/2018 10:02:01[2018-10-29T00:02:01,714][INFO ][o.e.n.Node ] [wa-knsg-elkdev3.acme.local] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.RIaV0zdg, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Des.cgroups.hierarchy.override=/, -Xms4g, -Xmx4g, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
29/10/2018 10:02:01[2018-10-29T00:02:01,759][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [wa-knsg-elkdev3.acme.local] uncaught exception in thread [main]
29/10/2018 10:02:01org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Could not load plugin descriptor for plugin directory [search-guard-6]
29/10/2018 10:02:01 at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01Caused by: java.lang.IllegalStateException: Could not load plugin descriptor for plugin directory [search-guard-6]
29/10/2018 10:02:01 at org.elasticsearch.plugins.PluginsService.readPluginBundle(PluginsService.java:394) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.plugins.PluginsService.findBundles(PluginsService.java:379) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.plugins.PluginsService.getPluginBundles(PluginsService.java:372) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:144) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.node.Node.<init>(Node.java:315) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 ... 6 more
29/10/2018 10:02:01Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/plugins/search-guard-6/plugin-descriptor.properties
29/10/2018 10:02:01 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
29/10/2018 10:02:01 at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
29/10/2018 10:02:01 at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
29/10/2018 10:02:01 at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:215) ~[?:?]
29/10/2018 10:02:01 at java.nio.file.Files.newByteChannel(Files.java:369) ~[?:?]
29/10/2018 10:02:01 at java.nio.file.Files.newByteChannel(Files.java:415) ~[?:?]
29/10/2018 10:02:01 at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) ~[?:?]
29/10/2018 10:02:01 at java.nio.file.Files.newInputStream(Files.java:154) ~[?:?]
29/10/2018 10:02:01 at org.elasticsearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:162) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.plugins.PluginsService.readPluginBundle(PluginsService.java:391) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.plugins.PluginsService.findBundles(PluginsService.java:379) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.plugins.PluginsService.getPluginBundles(PluginsService.java:372) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:144) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.node.Node.<init>(Node.java:315) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.2.jar:6.4.2]
29/10/2018 10:02:01 ... 6 more

I'm guessing '/usr/share/elasticsearch/plugins/search-guard-6/plugin-descriptor.properties' is missing from the install.

Are you able to re-add it back to the plugin?

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/30da28cd-0ae0-4b70-8f01-3ad0cfa3d654%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hello,

Sorry it was an issue with my Docker build process, which is unusual considering it’s scripted… I digress.

It is working!

Is this snapshot stable to use in production? Should I wait for 23.2 release? When is that expected?

Thanks

Cam