ES 6.4.0 -> 6.5.2: Kibana cannot migrate index: "No cluster-level perm match for User kibanaserver"

  • ES 6.4.0 upgrading to 6.5.2; SG 23.2

  • JVM 1.8 on Windows 7 SP1

Testing on our dev cluster prior to updating QA and Production.

Unpacked latest ES 6.5.2 / Kibana 6.5.2 into parallel directories to our existing 6.4.0; checked they started up OK.

Installed Search Guard.

I’ve updated our sg_roles.yml for sg_kibana_server according to the upgrade notes. Kibana instances show creation of .kibana_1 / _2 but when they come to the reindex step a failure is thrown:

[2018-12-10T16:53:51,740][INFO
][c.f.s.p.PrivilegesEvaluator] [node1] No cluster-level perm match
for User
[name=kibanaserver, roles=, requestedTenant=null]
Resolved [aliases=, indices=[.kibana, .kibana_1], allIndices=[.kibana,
.kibana_1], types=[null, *], isAll()=false, isEmpty()=false] [Action
[indices:data/write/reindex]]
[RolesChecked [sg_kibana_server, sg_own_index]]
[2018-12-10T16:53:51,741][INFO
][c.f.s.p.PrivilegesEvaluator] [node1] No permissions for
[indices:data/write/reindex]
[2018-12-10T16:53:51,871][WARN
][o.e.t.LoggingTaskListener] [node1] 90702 failed with exception

Here’s a later log with DEBUG, showing a successful CREATE and a failing REINDEX:

[2018-12-10T17:29:26,250][TRACE][o.e.t.T.tracer ] [node1] [11791][indices:admin/create] received response from [{node2}{xTlrKLONT9iQaZLhPlu2mQ}{eTc-6o33TmmLjXKYhDh02w}{10.32.0.89}{10.32.0.89:9300}{ml.machine_memory=34263273472, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]
[2018-12-10T17:29:26,256][DEBUG][c.f.s.a.BackendRegistry ] [node1] User ‘User [name=kibanaserver, roles=, requestedTenant=null]’ is authenticated
[2018-12-10T17:29:26,256][DEBUG][c.f.s.a.BackendRegistry ] [node1] sgtenant ‘null’
[2018-12-10T17:29:26,257][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] ### evaluate permissions for User [name=kibanaserver, roles=, requestedTenant=null] on node1
[2018-12-10T17:29:26,257][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] action: indices:admin/create (CreateIndexRequest)
[2018-12-10T17:29:26,258][DEBUG][c.f.s.r.IndexResolverReplacer] [node1] Resolve aliases, indices and types from CreateIndexRequest
[2018-12-10T17:29:26,258][DEBUG][c.f.s.r.IndexResolverReplacer] [node1] No such indices for pattern .kibana_1, use raw value
[2018-12-10T17:29:26,258][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] requestedResolved : Resolved [aliases=, indices=[.kibana_1], allIndices=[.kibana_1], types=[], isAll()=false, isEmpty()=false]
[2018-12-10T17:29:26,258][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] requested [indices:admin/create] from 127.0.0.1:64107
[2018-12-10T17:29:26,258][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] requested resolved indextypes: Resolved [aliases=[], indices=[.kibana_1], allIndices=[.kibana_1], types=[
], isAll()=false, isEmpty()=false]
[2018-12-10T17:29:26,259][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] sgr: sg_kibana_server
[2018-12-10T17:29:26,259][DEBUG][c.f.s.c.PrivilegesInterceptorImpl] [node1] raw requestedTenant: ‘null’
[2018-12-10T17:29:26,259][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] Result from privileges interceptor: null
[2018-12-10T17:29:26,259][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] .kibana_1 does not exist in cluster metadata
[2018-12-10T17:29:26,260][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] Allowed because we have all indices permissions for indices:admin/create
[2018-12-10T17:29:26,260][DEBUG][c.f.s.f.SearchGuardFilter] [node1] PrivEvalResponse [allowed=true, missingPrivileges=[indices:admin/create], allowedFlsFields=null, maskedFields=null, queries=null]

[2018-12-10T17:29:26,302][DEBUG][o.e.i.IndicesService ] [node1] creating Index [[.kibana_1/yG378GmbR8CYO9vB7NDSUw]], shards [1]/[1] - reason [create index]
[2018-12-10T17:29:26,955][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] ### evaluate permissions for User [name=kibanaserver, roles=, requestedTenant=null] on node1
[2018-12-10T17:29:26,955][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] action: indices:data/write/reindex (ReindexRequest)
[2018-12-10T17:29:26,956][DEBUG][c.f.s.r.IndexResolverReplacer] [node1] Resolve aliases, indices and types from ReindexRequest
[2018-12-10T17:29:26,956][DEBUG][c.f.s.r.IndexResolverReplacer] [node1] Resolved pattern [.kibana_1] to [.kibana_1]
[2018-12-10T17:29:26,956][DEBUG][c.f.s.r.IndexResolverReplacer] [node1] Resolved pattern [.kibana] to [.kibana]
[2018-12-10T17:29:26,957][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] requestedResolved : Resolved [aliases=, indices=[.kibana, .kibana_1], allIndices=[.kibana, .kibana_1], types=[null, *], isAll()=false, isEmpty()=false]
[2018-12-10T17:29:26,957][INFO ][c.f.s.p.PrivilegesEvaluator] [node1] No cluster-level perm match for User [name=kibanaserver, roles=, requestedTenant=null] Resolved [aliases=, indices=[.kibana, .kibana_1], allIndices=[.kibana, .kibana_1], types=[null, *], isAll()=false, isEmpty()=false] [Action [indices:data/write/reindex]] [RolesChecked [sg_kibana_server, sg_own_index]]
[2018-12-10T17:29:26,958][INFO ][c.f.s.p.PrivilegesEvaluator] [node1] No permissions for [indices:data/write/reindex]
[2018-12-10T17:29:26,958][DEBUG][c.f.s.f.SearchGuardFilter] [node1] PrivEvalResponse [allowed=false, missingPrivileges=[indices:data/write/reindex], allowedFlsFields=null, maskedFields=null, queries=null]
[2018-12-10T17:29:26,959][DEBUG][c.f.s.f.SearchGuardFilter] [node1] no permissions for [indices:data/write/reindex]

I double checked the permissions for kibanaserver using the Search Guard API - these look correct to me:

$ curl -u admin:x http://localhost:9200/_searchguard/api/internalusers/kibanaserver
{“kibanaserver”:{“readonly”:“true”,“hash”:""}}

$ curl -u admin:x http://localhost:9200/_searchguard/api/rolesmapping/sg_kibana_server
{“sg_kibana_server”:{“readonly”:“true”,“users”:[“kibanaserver”]}}

$ curl -u admin:x http://localhost:9200/_searchguard/api/roles/sg_kibana_server
{

“sg_kibana_server”: {

“cluster”: [“CLUSTER_MONITOR”, “CLUSTER_COMPOSITE_OPS”,
“cluster:admin/xpack/monitoring*”,
“indices:admin/template*”, “indices:data/read/scroll*”],

“indices”: {

“?kibana-6”: {

“*”: [“INDICES_ALL”]

},

“?management-beats*”: {

“*”: [“INDICES_ALL”]

},

?kibana_*”: {

“*”: [“INDICES_ALL”]

},

“?monitoring*”: {

“*”: [“INDICES_ALL”]

},

“?reporting*”: {

“*”: [“INDICES_ALL”]

},

“*”: {

": ["indices:admin/aliases”]

},

“?kibana”: {

“*”: [“INDICES_ALL”]

},

“?tasks”: {

“*”: [“INDICES_ALL”]

}

},

“readonly”: “true”

}
}

In the end we’ve worked around this for our dev cluster by disabling Search Guard for one restart, letting Kibana completing its reindex before re-enabling - but this won’t be appropriate for our other clusters.

Are we missing something here?

Are your action groups up to date? CLUSTER_COMPOSITE_OPS should include "indices:data/write/reindex", see https://github.com/floragunncom/search-guard/blob/13e90b17509e95413e85ea67d2cf105935204dac/sgconfig/sg_action_groups.yml

···

Am 11.12.2018 um 17:50 schrieb jbeckett@ft-services.com:

* ES 6.4.0 upgrading to 6.5.2; SG 23.2
* JVM 1.8 on Windows 7 SP1

Testing on our dev cluster prior to updating QA and Production.
Unpacked latest ES 6.5.2 / Kibana 6.5.2 into parallel directories to our existing 6.4.0; checked they started up OK.
Installed Search Guard.
I've updated our sg_roles.yml for sg_kibana_server according to the upgrade notes. Kibana instances show creation of .kibana_1 / _2 but when they come to the reindex step a failure is thrown:

[2018-12-10T16:53:51,740][INFO ][c.f.s.p.PrivilegesEvaluator] [node1] No cluster-level perm match for User [name=kibanaserver, roles=, requestedTenant=null] Resolved [aliases=, indices=[.kibana, .kibana_1], allIndices=[.kibana, .kibana_1], types=[null, *], isAll()=false, isEmpty()=false] [Action [indices:data/write/reindex]] [RolesChecked [sg_kibana_server, sg_own_index]]
[2018-12-10T16:53:51,741][INFO ][c.f.s.p.PrivilegesEvaluator] [node1] No permissions for [indices:data/write/reindex]
[2018-12-10T16:53:51,871][WARN ][o.e.t.LoggingTaskListener] [node1] 90702 failed with exception

Here's a later log with DEBUG, showing a successful CREATE and a failing REINDEX:

[2018-12-10T17:29:26,250][TRACE][o.e.t.T.tracer ] [node1] [11791][indices:admin/create] received response from [{node2}{xTlrKLONT9iQaZLhPlu2mQ}{eTc-6o33TmmLjXKYhDh02w}{10.32.0.89}{10.32.0.89:9300}{ml.machine_memory=34263273472, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]
[2018-12-10T17:29:26,256][DEBUG][c.f.s.a.BackendRegistry ] [node1] User 'User [name=kibanaserver, roles=, requestedTenant=null]' is authenticated
[2018-12-10T17:29:26,256][DEBUG][c.f.s.a.BackendRegistry ] [node1] sgtenant 'null'
[2018-12-10T17:29:26,257][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] ### evaluate permissions for User [name=kibanaserver, roles=, requestedTenant=null] on node1
[2018-12-10T17:29:26,257][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] action: indices:admin/create (CreateIndexRequest)
[2018-12-10T17:29:26,258][DEBUG][c.f.s.r.IndexResolverReplacer] [node1] Resolve aliases, indices and types from CreateIndexRequest
[2018-12-10T17:29:26,258][DEBUG][c.f.s.r.IndexResolverReplacer] [node1] No such indices for pattern .kibana_1, use raw value
[2018-12-10T17:29:26,258][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] requestedResolved : Resolved [aliases=, indices=[.kibana_1], allIndices=[.kibana_1], types=, isAll()=false, isEmpty()=false]
[2018-12-10T17:29:26,258][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] requested [indices:admin/create] from 127.0.0.1:64107
[2018-12-10T17:29:26,258][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] requested resolved indextypes: Resolved [aliases=, indices=[.kibana_1], allIndices=[.kibana_1], types=, isAll()=false, isEmpty()=false]
[2018-12-10T17:29:26,259][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] sgr: sg_kibana_server
[2018-12-10T17:29:26,259][DEBUG][c.f.s.c.PrivilegesInterceptorImpl] [node1] raw requestedTenant: 'null'
[2018-12-10T17:29:26,259][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] Result from privileges interceptor: null
[2018-12-10T17:29:26,259][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] .kibana_1 does not exist in cluster metadata
[2018-12-10T17:29:26,260][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] Allowed because we have all indices permissions for indices:admin/create
[2018-12-10T17:29:26,260][DEBUG][c.f.s.f.SearchGuardFilter] [node1] PrivEvalResponse [allowed=true, missingPrivileges=[indices:admin/create], allowedFlsFields=null, maskedFields=null, queries=null]
  
[2018-12-10T17:29:26,302][DEBUG][o.e.i.IndicesService ] [node1] creating Index [[.kibana_1/yG378GmbR8CYO9vB7NDSUw]], shards [1]/[1] - reason [create index]

[2018-12-10T17:29:26,955][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] ### evaluate permissions for User [name=kibanaserver, roles=, requestedTenant=null] on node1
[2018-12-10T17:29:26,955][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] action: indices:data/write/reindex (ReindexRequest)
[2018-12-10T17:29:26,956][DEBUG][c.f.s.r.IndexResolverReplacer] [node1] Resolve aliases, indices and types from ReindexRequest
[2018-12-10T17:29:26,956][DEBUG][c.f.s.r.IndexResolverReplacer] [node1] Resolved pattern [.kibana_1] to [.kibana_1]
[2018-12-10T17:29:26,956][DEBUG][c.f.s.r.IndexResolverReplacer] [node1] Resolved pattern [.kibana] to [.kibana]
[2018-12-10T17:29:26,957][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] requestedResolved : Resolved [aliases=, indices=[.kibana, .kibana_1], allIndices=[.kibana, .kibana_1], types=[null, *], isAll()=false, isEmpty()=false]
[2018-12-10T17:29:26,957][INFO ][c.f.s.p.PrivilegesEvaluator] [node1] No cluster-level perm match for User [name=kibanaserver, roles=, requestedTenant=null] Resolved [aliases=, indices=[.kibana, .kibana_1], allIndices=[.kibana, .kibana_1], types=[null, *], isAll()=false, isEmpty()=false] [Action [indices:data/write/reindex]] [RolesChecked [sg_kibana_server, sg_own_index]]
[2018-12-10T17:29:26,958][INFO ][c.f.s.p.PrivilegesEvaluator] [node1] No permissions for [indices:data/write/reindex]
[2018-12-10T17:29:26,958][DEBUG][c.f.s.f.SearchGuardFilter] [node1] PrivEvalResponse [allowed=false, missingPrivileges=[indices:data/write/reindex], allowedFlsFields=null, maskedFields=null, queries=null]
[2018-12-10T17:29:26,959][DEBUG][c.f.s.f.SearchGuardFilter] [node1] no permissions for [indices:data/write/reindex]

I double checked the permissions for kibanaserver using the Search Guard API - these look correct to me:

$ curl -u admin:x http://localhost:9200/_searchguard/api/internalusers/kibanaserver
{"kibanaserver":{"readonly":"true","hash":""}}

$ curl -u admin:x http://localhost:9200/_searchguard/api/rolesmapping/sg_kibana_server
{"sg_kibana_server":{"readonly":"true","users":["kibanaserver"]}}

$ curl -u admin:x http://localhost:9200/_searchguard/api/roles/sg_kibana_server
{
                "sg_kibana_server": {
                                "cluster": ["CLUSTER_MONITOR", "CLUSTER_COMPOSITE_OPS", "cluster:admin/xpack/monitoring*", "indices:admin/template*", "indices:data/read/scroll*"],
                                "indices": {
                                                "?kibana-6": {
                                                                "*": ["INDICES_ALL"]
                                                },
                                                "?management-beats*": {
                                                                "*": ["INDICES_ALL"]
                                                },
                                                "?kibana_*": {
                                                                "*": ["INDICES_ALL"]
                                                },
                                                "?monitoring*": {
                                                                "*": ["INDICES_ALL"]
                                                },
                                                "?reporting*": {
                                                                "*": ["INDICES_ALL"]
                                                },
                                                "*": {
                                                                "*": ["indices:admin/aliases*"]
                                                },
                                                "?kibana": {
                                                                "*": ["INDICES_ALL"]
                                                },
                                                "?tasks": {
                                                                "*": ["INDICES_ALL"]
                                                }
                                },
                                "readonly": "true"
                }
}

In the end we've worked around this for our dev cluster by disabling Search Guard for one restart, letting Kibana completing its reindex before re-enabling - but this won't be appropriate for our other clusters.

Are we missing something here?

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/680fe687-fce6-40eb-948d-fb07534d8194%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Looks like that was it. I’ve added that, and I can run a reindex from .kibana to .kibana_xxx with the kibanaserver user, which I couldn’t before.

Two things we don’t understand though:

  • Why does the index-level wildcarded INDICES_ALL permission, which includes indices:* not give the required access

  • How were reindex requests working at all, prior to this? We’re performed many of these in the last year or so.

···

On Tuesday, December 11, 2018 at 5:17:06 PM UTC, Search Guard wrote:

Are your action groups up to date? CLUSTER_COMPOSITE_OPS should include “indices:data/write/reindex”