-
ES 6.4.0 upgrading to 6.5.2; SG 23.2
-
JVM 1.8 on Windows 7 SP1
Testing on our dev cluster prior to updating QA and Production.
Unpacked latest ES 6.5.2 / Kibana 6.5.2 into parallel directories to our existing 6.4.0; checked they started up OK.
Installed Search Guard.
I’ve updated our sg_roles.yml for sg_kibana_server according to the upgrade notes. Kibana instances show creation of .kibana_1 / _2 but when they come to the reindex step a failure is thrown:
[2018-12-10T16:53:51,740][INFO
][c.f.s.p.PrivilegesEvaluator] [node1] No cluster-level perm match
for User [name=kibanaserver, roles=, requestedTenant=null]
Resolved [aliases=, indices=[.kibana, .kibana_1], allIndices=[.kibana,
.kibana_1], types=[null, *], isAll()=false, isEmpty()=false] [Action
[indices:data/write/reindex]]
[RolesChecked [sg_kibana_server, sg_own_index]]
[2018-12-10T16:53:51,741][INFO
][c.f.s.p.PrivilegesEvaluator] [node1] No permissions for
[indices:data/write/reindex]
[2018-12-10T16:53:51,871][WARN
][o.e.t.LoggingTaskListener] [node1] 90702 failed with exception
Here’s a later log with DEBUG, showing a successful CREATE and a failing REINDEX:
[2018-12-10T17:29:26,250][TRACE][o.e.t.T.tracer ] [node1] [11791][indices:admin/create] received response from [{node2}{xTlrKLONT9iQaZLhPlu2mQ}{eTc-6o33TmmLjXKYhDh02w}{10.32.0.89}{10.32.0.89:9300}{ml.machine_memory=34263273472, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]
[2018-12-10T17:29:26,256][DEBUG][c.f.s.a.BackendRegistry ] [node1] User ‘User [name=kibanaserver, roles=, requestedTenant=null]’ is authenticated
[2018-12-10T17:29:26,256][DEBUG][c.f.s.a.BackendRegistry ] [node1] sgtenant ‘null’
[2018-12-10T17:29:26,257][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] ### evaluate permissions for User [name=kibanaserver, roles=, requestedTenant=null] on node1
[2018-12-10T17:29:26,257][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] action: indices:admin/create (CreateIndexRequest)
[2018-12-10T17:29:26,258][DEBUG][c.f.s.r.IndexResolverReplacer] [node1] Resolve aliases, indices and types from CreateIndexRequest
[2018-12-10T17:29:26,258][DEBUG][c.f.s.r.IndexResolverReplacer] [node1] No such indices for pattern .kibana_1, use raw value
[2018-12-10T17:29:26,258][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] requestedResolved : Resolved [aliases=, indices=[.kibana_1], allIndices=[.kibana_1], types=[], isAll()=false, isEmpty()=false]
[2018-12-10T17:29:26,258][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] requested [indices:admin/create] from 127.0.0.1:64107
[2018-12-10T17:29:26,258][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] requested resolved indextypes: Resolved [aliases=[], indices=[.kibana_1], allIndices=[.kibana_1], types=[], isAll()=false, isEmpty()=false]
[2018-12-10T17:29:26,259][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] sgr: sg_kibana_server
[2018-12-10T17:29:26,259][DEBUG][c.f.s.c.PrivilegesInterceptorImpl] [node1] raw requestedTenant: ‘null’
[2018-12-10T17:29:26,259][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] Result from privileges interceptor: null
[2018-12-10T17:29:26,259][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] .kibana_1 does not exist in cluster metadata
[2018-12-10T17:29:26,260][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] Allowed because we have all indices permissions for indices:admin/create
[2018-12-10T17:29:26,260][DEBUG][c.f.s.f.SearchGuardFilter] [node1] PrivEvalResponse [allowed=true, missingPrivileges=[indices:admin/create], allowedFlsFields=null, maskedFields=null, queries=null][2018-12-10T17:29:26,302][DEBUG][o.e.i.IndicesService ] [node1] creating Index [[.kibana_1/yG378GmbR8CYO9vB7NDSUw]], shards [1]/[1] - reason [create index]
[2018-12-10T17:29:26,955][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] ### evaluate permissions for User [name=kibanaserver, roles=, requestedTenant=null] on node1
[2018-12-10T17:29:26,955][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] action: indices:data/write/reindex (ReindexRequest)
[2018-12-10T17:29:26,956][DEBUG][c.f.s.r.IndexResolverReplacer] [node1] Resolve aliases, indices and types from ReindexRequest
[2018-12-10T17:29:26,956][DEBUG][c.f.s.r.IndexResolverReplacer] [node1] Resolved pattern [.kibana_1] to [.kibana_1]
[2018-12-10T17:29:26,956][DEBUG][c.f.s.r.IndexResolverReplacer] [node1] Resolved pattern [.kibana] to [.kibana]
[2018-12-10T17:29:26,957][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] requestedResolved : Resolved [aliases=, indices=[.kibana, .kibana_1], allIndices=[.kibana, .kibana_1], types=[null, *], isAll()=false, isEmpty()=false]
[2018-12-10T17:29:26,957][INFO ][c.f.s.p.PrivilegesEvaluator] [node1] No cluster-level perm match for User [name=kibanaserver, roles=, requestedTenant=null] Resolved [aliases=, indices=[.kibana, .kibana_1], allIndices=[.kibana, .kibana_1], types=[null, *], isAll()=false, isEmpty()=false] [Action [indices:data/write/reindex]] [RolesChecked [sg_kibana_server, sg_own_index]]
[2018-12-10T17:29:26,958][INFO ][c.f.s.p.PrivilegesEvaluator] [node1] No permissions for [indices:data/write/reindex]
[2018-12-10T17:29:26,958][DEBUG][c.f.s.f.SearchGuardFilter] [node1] PrivEvalResponse [allowed=false, missingPrivileges=[indices:data/write/reindex], allowedFlsFields=null, maskedFields=null, queries=null]
[2018-12-10T17:29:26,959][DEBUG][c.f.s.f.SearchGuardFilter] [node1] no permissions for [indices:data/write/reindex]
I double checked the permissions for kibanaserver using the Search Guard API - these look correct to me:
$ curl -u admin:x http://localhost:9200/_searchguard/api/internalusers/kibanaserver
{“kibanaserver”:{“readonly”:“true”,“hash”:“”}}
$ curl -u admin:x http://localhost:9200/_searchguard/api/rolesmapping/sg_kibana_server
{“sg_kibana_server”:{“readonly”:“true”,“users”:[“kibanaserver”]}}
$ curl -u admin:x http://localhost:9200/_searchguard/api/roles/sg_kibana_server
{“sg_kibana_server”: {
“cluster”: [“CLUSTER_MONITOR”, “CLUSTER_COMPOSITE_OPS”,
“cluster:admin/xpack/monitoring*”,
“indices:admin/template*”, “indices:data/read/scroll*”],“indices”: {
“?kibana-6”: {
“*”: [“INDICES_ALL”]
},
“?management-beats*”: {
“*”: [“INDICES_ALL”]
},
“?kibana_*”: {
“*”: [“INDICES_ALL”]
},
“?monitoring*”: {
“*”: [“INDICES_ALL”]
},
“?reporting*”: {
“*”: [“INDICES_ALL”]
},
“*”: {
“": ["indices:admin/aliases”]
},
“?kibana”: {
“*”: [“INDICES_ALL”]
},
“?tasks”: {
“*”: [“INDICES_ALL”]
}
},
“readonly”: “true”
}
}
In the end we’ve worked around this for our dev cluster by disabling Search Guard for one restart, letting Kibana completing its reindex before re-enabling - but this won’t be appropriate for our other clusters.
Are we missing something here?