Tenant foo is not allowed to write (user: bob)

Tenant foo is not allowed to write (user: bob)

We just upgraded from 5.3.2 to 5.6.6 and now the users can’t access RO tenants anymore.

Take for instance user ‘bob’ who has the following credentials, as reported by kibana/elasticsearch/_searchguard/authinfo:

{
  "user": "User [name=bob, roles=[]]",
  "user_name": "bob",
  "user_requested_tenant": "foo",
  "remote_address": "192.168.230.13:59340",
  "sg_roles": [
    "sg_role_site",
    "sg_role_bar",
    "sg_role_baz"
  ],
  "sg_tenants": {
    "foo": false,
    "bar": true,
    "beer": false,
    "sandbox": true,
    "baz": true,
    "deer": false,
    "bob": true,
    "fox": false
  },
  "principal": null,
  "peer_certificates": "0"
}

Given the following roles:

sg_role_site:
  tenants:
    deer: RO
    sandbox: RW
    foo: RO
    beer: RO
    bar: RO
    fox: RO
sg_role_bar:
  tenants:
    bar: RW
sg_role_baz:
  tenants:
    baz: RW
[...]

When the user ‘bob’ accesses kibana discovery using tenant ‘foo’ he gets the following error in the browser:

Error: no permissions for [indices:data/write/update] and User [name=bob, roles=[]]: [security_exception] no permissions for [indices:data/write/update] and User [name=bob, roles=[]]
exports.SavedObjectsClient</<.value/<@https://kibana.my.tld/bundles/kibana.bundle.js?v=15605:228:17056
processQueue@https://kibana.my.tld/bundles/commons.bundle.js?v=15605:38:23621
scheduleProcessQueue/<@https://kibana.my.tld/bundles/commons.bundle.js?v=15605:38:23888
$RootScopeProvider/this.$get</Scope.prototype.$eval@https://kibana.my.tld/bundles/commons.bundle.js?v=15605:39:4607
$RootScopeProvider/this.$get</Scope.prototype.$digest@https://kibana.my.tld/bundles/commons.bundle.js?v=15605:39:2343
$RootScopeProvider/this.$get</Scope.prototype.$apply@https://kibana.my.tld/bundles/commons.bundle.js?v=15605:39:5026
done@https://kibana.my.tld/bundles/commons.bundle.js?v=15605:37:25016
completeRequest@https://kibana.my.tld/bundles/commons.bundle.js?v=15605:37:28702
createHttpBackend/</xhr.onload@https://kibana.my.tld/bundles/commons.bundle.js?v=15605:37:29634

In Elasticsearch, I get merely the following in the logs:

[WARN ][c.f.s.c.PrivilegesInterceptorImpl] Tenant foo is not allowed to write (user: bob)

What are we doing wrong?
Why would doing a search require write access?

BTW, auth is using proxy module.

Digging a little further by enabling debug logging, I discovered the following:

[2018-02-08T21:48:25,201][DEBUG][c.f.s.h.HTTPProxyAuthenticator] userHeader x-authenticated-user, value bob
[2018-02-08T21:48:25,201][DEBUG][c.f.s.h.HTTPProxyAuthenticator] rolesHeader x-authenticated-group, value null
[2018-02-08T21:48:25,201][DEBUG][c.f.s.a.BackendRegistry  ] User 'bob' is in cache? true (cache size: 2)
[2018-02-08T21:48:25,201][DEBUG][c.f.s.a.BackendRegistry  ] User 'User [name=bob, roles=[]]' is authenticated
[2018-02-08T21:48:25,201][DEBUG][c.f.s.a.BackendRegistry  ] sg_tenant 'foo'
[2018-02-08T21:48:25,204][DEBUG][c.f.s.c.PrivilegesEvaluator] ### evaluate permissions for User [name=bob, roles=[]] on kibana.my.tld
[2018-02-08T21:48:25,204][DEBUG][c.f.s.c.PrivilegesEvaluator] requested indices:data/write/update from 192.168.230.13:58520
[2018-02-08T21:48:25,204][DEBUG][c.f.s.c.PrivilegesEvaluator] Resolve [.kibana] from class org.elasticsearch.action.update.UpdateRequest
[2018-02-08T21:48:25,204][DEBUG][c.f.s.c.PrivilegesEvaluator] indicesOptions IndicesOptions[id=48, ignore_unavailable=false, allow_no_indices=false, expand_wildcards_open=false, expand_wildcards_closed=false, allow_alisases_to_multiple_indices=false, forbid_closed_indices=true]
[2018-02-08T21:48:25,204][DEBUG][c.f.s.c.PrivilegesEvaluator] 1 raw indices [.kibana]
[2018-02-08T21:48:25,204][DEBUG][c.f.s.c.PrivilegesEvaluator] No date math indices found
[2018-02-08T21:48:25,204][DEBUG][c.f.s.c.PrivilegesEvaluator] Resolved [.kibana] to [.kibana]
[2018-02-08T21:48:25,204][DEBUG][c.f.s.c.PrivilegesEvaluator] pre final indices: [.kibana]
[2018-02-08T21:48:25,204][DEBUG][c.f.s.c.PrivilegesEvaluator] pre final types: [index-pattern]
[2018-02-08T21:48:25,204][DEBUG][c.f.s.c.PrivilegesEvaluator] requested resolved indextypes: [IndexType [index=.kibana, type=index-pattern]]
[2018-02-08T21:48:25,205][DEBUG][c.f.s.c.PrivilegesEvaluator] mapped roles for bob: [sg_role_site, sg_role_bar, sg_role_baz]
[2018-02-08T21:48:25,206][DEBUG][c.f.s.c.PrivilegesInterceptorImpl] raw requestedTenant: 'foo'
[2018-02-08T21:48:25,206][DEBUG][c.f.s.c.PrivilegesInterceptorImpl] requestedTenant: foo
[2018-02-08T21:48:25,206][DEBUG][c.f.s.c.PrivilegesInterceptorImpl] is user tenant: false
[2018-02-08T21:48:25,206][DEBUG][c.f.s.c.PrivilegesInterceptorImpl] request class org.elasticsearch.action.update.UpdateRequest
[2018-02-08T21:48:25,206][DEBUG][c.f.s.c.PrivilegesInterceptorImpl] type index-pattern
[2018-02-08T21:48:25,206][DEBUG][c.f.s.c.PrivilegesInterceptorImpl] id fox-*
[2018-02-08T21:48:25,206][DEBUG][c.f.s.c.PrivilegesInterceptorImpl] :true,\"readFromDocValues\":true} [...]
[...]
[c.f.s.f.SearchGuardFilter] no permissions for [indices:data/write/update][2018-02-08T21:48:25,207][DEBUG][r.suppressed             ] path: /.kibana/index-pattern/fox-*/_update, params: {refresh=wait_for, index=.kibana, id=fox-*, type=index-pattern}org.elasticsearch.ElasticsearchSecurityException: no permissions for [indices:data/write/update] and User [name=bob, roles=[]]

does that mean 5.6.6 now requires user ‘bob’ to have write permissions to the “virtually unused” ‘.kibana’ index?

Hm … this is strange, have not seen this behavior before. I guess it’s related to the update process. I need to dig deeper into this and will get back to you as soon as possible. Thanks for reporting!

···

On Thursday, February 8, 2018 at 10:08:11 PM UTC+1, Fabien Wernli wrote:

Digging a little further by enabling debug logging, I discovered the following:

[2018-02-08T21:48:25,201][DEBUG][c.f.s.h.HTTPProxyAuthenticator] userHeader x-authenticated-user, value bob
[2018-02-08T21:48:25,201][DEBUG][c.f.s.h.HTTPProxyAuthenticator] rolesHeader x-authenticated-group, value null
[2018-02-08T21:48:25,201][DEBUG][c.f.s.a.BackendRegistry  ] User 'bob' is in cache? true (cache size: 2)
[2018-02-08T21:48:25,201][DEBUG][c.f.s.a.BackendRegistry  ] User 'User [name=bob, roles=[]]' is authenticated
[2018-02-08T21:48:25,201][DEBUG][c.f.s.a.BackendRegistry  ] sg_tenant 'foo'
[2018-02-08T21:48:25,204][DEBUG][c.f.s.c.PrivilegesEvaluator] ### evaluate permissions for User [name=bob, roles=[]] on kibana.my.tld
[2018-02-08T21:48:25,204][DEBUG][c.f.s.c.PrivilegesEvaluator] requested indices:data/write/update from [192.168.230.13:58520](http://192.168.230.13:58520)
[2018-02-08T21:48:25,204][DEBUG][c.f.s.c.PrivilegesEvaluator] Resolve [.kibana] from class org.elasticsearch.action.update.UpdateRequest
[2018-02-08T21:48:25,204][DEBUG][c.f.s.c.PrivilegesEvaluator] indicesOptions IndicesOptions[id=48, ignore_unavailable=false, allow_no_indices=false, expand_wildcards_open=false, expand_wildcards_closed=false, allow_alisases_to_multiple_indices=false, forbid_closed_indices=true]
[2018-02-08T21:48:25,204][DEBUG][c.f.s.c.PrivilegesEvaluator] 1 raw indices [.kibana]
[2018-02-08T21:48:25,204][DEBUG][c.f.s.c.PrivilegesEvaluator] No date math indices found
[2018-02-08T21:48:25,204][DEBUG][c.f.s.c.PrivilegesEvaluator] Resolved [.kibana] to [.kibana]
[2018-02-08T21:48:25,204][DEBUG][c.f.s.c.PrivilegesEvaluator] pre final indices: [.kibana]
[2018-02-08T21:48:25,204][DEBUG][c.f.s.c.PrivilegesEvaluator] pre final types: [index-pattern]
[2018-02-08T21:48:25,204][DEBUG][c.f.s.c.PrivilegesEvaluator] requested resolved indextypes: [IndexType [index=.kibana, type=index-pattern]]
[2018-02-08T21:48:25,205][DEBUG][c.f.s.c.PrivilegesEvaluator] mapped roles for bob: [sg_role_site, sg_role_bar, sg_role_baz]
[2018-02-08T21:48:25,206][DEBUG][c.f.s.c.PrivilegesInterceptorImpl] raw requestedTenant: 'foo'
[2018-02-08T21:48:25,206][DEBUG][c.f.s.c.PrivilegesInterceptorImpl] requestedTenant: foo
[2018-02-08T21:48:25,206][DEBUG][c.f.s.c.PrivilegesInterceptorImpl] is user tenant: false
[2018-02-08T21:48:25,206][DEBUG][c.f.s.c.PrivilegesInterceptorImpl] request class org.elasticsearch.action.update.UpdateRequest
[2018-02-08T21:48:25,206][DEBUG][c.f.s.c.PrivilegesInterceptorImpl] type index-pattern
[2018-02-08T21:48:25,206][DEBUG][c.f.s.c.PrivilegesInterceptorImpl] id fox-*
[2018-02-08T21:48:25,206][DEBUG][c.f.s.c.PrivilegesInterceptorImpl] :true,\"readFromDocValues\":true} [...]
[...]
[c.f.s.f.SearchGuardFilter] no permissions for [indices:data/write/update][2018-02-08T21:48:25,207][DEBUG][r.suppressed             ] path: /.kibana/index-pattern/fox-*/_update, params: {refresh=wait_for, index=.kibana, id=fox-*, type=index-pattern}org.elasticsearch.ElasticsearchSecurityException: no permissions for [indices:data/write/update] and User [name=bob, roles=[]]

does that mean 5.6.6 now requires user ‘bob’ to have write permissions to the “virtually unused” ‘.kibana’ index?

Hi,

I tried a few things on my staging cluster, and it seems I found a workaround:

  1. Give user ‘bob’ RW permissions to tenant ‘foo’
  2. Let user browse to 'Kibana>Management>index-pattern(foo-*)
  3. Wait for ‘Error: Request Timeout after 30000ms’
  4. Repeat for all index patterns
  5. Switch user ‘bob’ back to RO permissions

Maybe some kind of upgrade of the kibana indices gets done (as you mentioned in the other thread) which needs RW. But once it’s done, write permissions aren’t required anymore.

I did a diff on the ‘.kibana-…foo’ index before and after modification by bob/RW, and here’s an excerpt:

Before:

{
“name”: “received_bytes”,
“type”: “number”,
“count”: 0,
“scripted”: false,
“indexed”: true,
“analyzed”: false,
“doc_values”: true,
“searchable”: false,
“aggregatable”: false
},

``

After:

{
“name”: “received_bytes”,
“type”: “number”,
“count”: 0,
“scripted”: false,
“searchable”: true,
“aggregatable”: true,
“readFromDocValues”: true
},

``

···

I’ll apply the workaround in production as soon as you tell me it’s the way to go!

We have a new symptom: editing visualizations times out or yields a 502 error in the browser

Visualization Editor: Request Timeout after 30000ms

``

In the kibana log:

POST /api/saved_objects/visualization/61352aa0-4ac0-11e7-b7b6-7576067609e9?overwrite=true 503 30011ms - 9.0B

``

However, changes made seem to be effective