Multi-tenancy kibana default index error

Aleksey_Kiselev · May 31, 2017, 11:58am

I try to use kibana multi-tenancy. With global tenant all good. But if i switch to other tenant and try to set default index pattern i have a error:

{“statusCode”:404,“error”:“Not Found”,“message”:“[document_missing_exception] [config][5.4.0]: document missing, with { index_uuid="n6YqxlkCT1mPbezpnrWa3g" & shard="0" & index=".kibana_-1180334778_testmain" }”}

``

But in elastic search server logs i found this message:

[2017-05-31T11:29:23,354][INFO ][o.e.c.m.MetaDataCreateIndexService] [Master-01] [.kibana_-1180334778_testmain] creating index, cause [api], templates , shards [1]/[1], mappings [config]
[2017-05-31T11:29:23,981][INFO ][o.e.c.r.a.AllocationService] [Master-01] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana_-1180334778_testmain][0]] …]).
[2017-05-31T11:29:24,230][INFO ][o.e.c.m.MetaDataMappingService] [Master-01] [.kibana_-1180334778_testmain/n6YqxlkCT1mPbezpnrWa3g] update_mapping [config]
[2017-05-31T11:31:06,358][INFO ][o.e.c.m.MetaDataMappingService] [Master-01] [.kibana_-1180334778_testmain/n6YqxlkCT1mPbezpnrWa3g] create_mapping [index-pattern]
[2017-05-31T11:31:08,786][INFO ][o.e.c.m.MetaDataMappingService] [Master-01] [.kibana_-1180334778_testmain/n6YqxlkCT1mPbezpnrWa3g] update_mapping [index-pattern]

``

What am I doing wrong?

jkressin · May 31, 2017, 12:04pm

Did you upgrade from a previous version by chance? Maybe you hit the following bug:

github.com/floragunncom/search-guard-module-kibana-multitenancy

Upgrading tenants fails

opened 12:17PM - 11 May 17 UTC

closed 07:19PM - 10 Jun 17 UTC

jochenkressin

Steps to reproduce: - Install ES and KI 5.3.1, with full MT support - Log i…n with a user that has some tenants - Choose the Global tenant - Create an index pattern, a visualization etc. - Choose a tenant other than Global - Create an index pattern, a visualization etc. - Upgrade ES and KI to 5.3.2 - Login again with the same user Result: When accessing the Global tenant, everything works as expected. When trying to access data the non-Global tenant, 404s are returned. Cause: I'm not 100% sure yet about the sequence of calls from KI, but here are the findings so far: When creating the indices with 5.3.1, both have one entry of type config with id 5.3.1: ``` { _index: ".kibana", _type: "config", _id: "5.3.1", _score: 1, _source: { buildNum: 14844, defaultIndex: "humanresources" } ``` After the upgrade to 5.3.2, the Global tenant has a second entry with version 5.3.2, and the same contents as the 5.3.1 one: ``` { _index: ".kibana", _type: "config", _id: "5.3.2", _score: 1, _source: { buildNum: 14849, defaultIndex: "humanresources" } } ``` This entry is missing in all other tenants, leading to 404s. There seems to be a problem with the PrivilegesInterceptor when Kibana tries to copy over the entry, or creating it new. I will investigate the calls from KI during the upgrade process. Might be related to #8

···

On Wednesday, May 31, 2017 at 1:58:06 PM UTC+2, Aleksey Kiselev wrote:

I try to use kibana multi-tenancy. With global tenant all good. But if i switch to other tenant and try to set default index pattern i have a error:

{“statusCode”:404,“error”:“Not Found”,“message”:“[document_missing_exception] [config][5.4.0]: document missing, with { index_uuid="n6YqxlkCT1mPbezpnrWa3g" & shard="0" & index=".kibana_-1180334778_testmain" }”}

``

But in elastic search server logs i found this message:

[2017-05-31T11:29:23,354][INFO ][o.e.c.m.MetaDataCreateIndexService] [Master-01] [.kibana_-1180334778_testmain] creating index, cause [api], templates , shards [1]/[1], mappings [config]
[2017-05-31T11:29:23,981][INFO ][o.e.c.r.a.AllocationService] [Master-01] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana_-1180334778_testmain][0]] …]).
[2017-05-31T11:29:24,230][INFO ][o.e.c.m.MetaDataMappingService] [Master-01] [.kibana_-1180334778_testmain/n6YqxlkCT1mPbezpnrWa3g] update_mapping [config]
[2017-05-31T11:31:06,358][INFO ][o.e.c.m.MetaDataMappingService] [Master-01] [.kibana_-1180334778_testmain/n6YqxlkCT1mPbezpnrWa3g] create_mapping [index-pattern]
[2017-05-31T11:31:08,786][INFO ][o.e.c.m.MetaDataMappingService] [Master-01] [.kibana_-1180334778_testmain/n6YqxlkCT1mPbezpnrWa3g] update_mapping [index-pattern]

``

What am I doing wrong?

Aleksey_Kiselev · May 31, 2017, 12:14pm

I think is the same error. Is this my config document:
{
“index": ".kibana-1180334778_testmain”,
“_type”: “config”,
“_id”: “5.1.2”,
“_score”: 1,
“_source”: {
“buildNum”: 14588,
“defaultIndex”: “XXXXXXX”
}
}

``

In this document version: 5.1.2. But now I have a 5.4.0 es version. I will try to change config version in that document.

···

среда, 31 мая 2017 г., 15:04:40 UTC+3 пользователь Jochen Kressin написал:

Did you upgrade from a previous version by chance? Maybe you hit the following bug:

https://github.com/floragunncom/search-guard-module-kibana-multitenancy/issues/9

On Wednesday, May 31, 2017 at 1:58:06 PM UTC+2, Aleksey Kiselev wrote:

I try to use kibana multi-tenancy. With global tenant all good. But if i switch to other tenant and try to set default index pattern i have a error:

{“statusCode”:404,“error”:“Not Found”,“message”:“[document_missing_exception] [config][5.4.0]: document missing, with { index_uuid="n6YqxlkCT1mPbezpnrWa3g" & shard="0" & index=".kibana_-1180334778_testmain" }”}

``

But in elastic search server logs i found this message:

[2017-05-31T11:29:23,354][INFO ][o.e.c.m.MetaDataCreateIndexService] [Master-01] [.kibana_-1180334778_testmain] creating index, cause [api], templates , shards [1]/[1], mappings [config]
[2017-05-31T11:29:23,981][INFO ][o.e.c.r.a.AllocationService] [Master-01] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana_-1180334778_testmain][0]] …]).
[2017-05-31T11:29:24,230][INFO ][o.e.c.m.MetaDataMappingService] [Master-01] [.kibana_-1180334778_testmain/n6YqxlkCT1mPbezpnrWa3g] update_mapping [config]
[2017-05-31T11:31:06,358][INFO ][o.e.c.m.MetaDataMappingService] [Master-01] [.kibana_-1180334778_testmain/n6YqxlkCT1mPbezpnrWa3g] create_mapping [index-pattern]
[2017-05-31T11:31:08,786][INFO ][o.e.c.m.MetaDataMappingService] [Master-01] [.kibana_-1180334778_testmain/n6YqxlkCT1mPbezpnrWa3g] update_mapping [index-pattern]

``

What am I doing wrong?

jkressin · May 31, 2017, 12:15pm

It should match the Kibana version, but you should run the same ES and KI version anyways. Yes, updating the version should work as a temporary workaround until the fix is released.

···

On Wednesday, May 31, 2017 at 2:14:01 PM UTC+2, Aleksey Kiselev wrote:

I think is the same error. Is this my config document:
{
“index": ".kibana-1180334778_testmain”,
“_type”: “config”,
“_id”: “5.1.2”,
“_score”: 1,
“_source”: {
“buildNum”: 14588,
“defaultIndex”: “XXXXXXX”
}
}

``

In this document version: 5.1.2. But now I have a 5.4.0 es version. I will try to change config version in that document.

среда, 31 мая 2017 г., 15:04:40 UTC+3 пользователь Jochen Kressin написал:

Did you upgrade from a previous version by chance? Maybe you hit the following bug:

https://github.com/floragunncom/search-guard-module-kibana-multitenancy/issues/9

On Wednesday, May 31, 2017 at 1:58:06 PM UTC+2, Aleksey Kiselev wrote:

I try to use kibana multi-tenancy. With global tenant all good. But if i switch to other tenant and try to set default index pattern i have a error:

{“statusCode”:404,“error”:“Not Found”,“message”:“[document_missing_exception] [config][5.4.0]: document missing, with { index_uuid="n6YqxlkCT1mPbezpnrWa3g" & shard="0" & index=".kibana_-1180334778_testmain" }”}

``

But in elastic search server logs i found this message:

[2017-05-31T11:29:23,354][INFO ][o.e.c.m.MetaDataCreateIndexService] [Master-01] [.kibana_-1180334778_testmain] creating index, cause [api], templates , shards [1]/[1], mappings [config]
[2017-05-31T11:29:23,981][INFO ][o.e.c.r.a.AllocationService] [Master-01] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana_-1180334778_testmain][0]] …]).
[2017-05-31T11:29:24,230][INFO ][o.e.c.m.MetaDataMappingService] [Master-01] [.kibana_-1180334778_testmain/n6YqxlkCT1mPbezpnrWa3g] update_mapping [config]
[2017-05-31T11:31:06,358][INFO ][o.e.c.m.MetaDataMappingService] [Master-01] [.kibana_-1180334778_testmain/n6YqxlkCT1mPbezpnrWa3g] create_mapping [index-pattern]
[2017-05-31T11:31:08,786][INFO ][o.e.c.m.MetaDataMappingService] [Master-01] [.kibana_-1180334778_testmain/n6YqxlkCT1mPbezpnrWa3g] update_mapping [index-pattern]

``

What am I doing wrong?

Aleksey_Kiselev · May 31, 2017, 12:24pm

After updating document version all works fine!

Really thanks for the fast answer!

···

среда, 31 мая 2017 г., 15:15:44 UTC+3 пользователь Jochen Kressin написал:

It should match the Kibana version, but you should run the same ES and KI version anyways. Yes, updating the version should work as a temporary workaround until the fix is released.

On Wednesday, May 31, 2017 at 2:14:01 PM UTC+2, Aleksey Kiselev wrote:

I think is the same error. Is this my config document:
{
“index": ".kibana-1180334778_testmain”,
“_type”: “config”,
“_id”: “5.1.2”,
“_score”: 1,
“_source”: {
“buildNum”: 14588,
“defaultIndex”: “XXXXXXX”
}
}

``

In this document version: 5.1.2. But now I have a 5.4.0 es version. I will try to change config version in that document.

среда, 31 мая 2017 г., 15:04:40 UTC+3 пользователь Jochen Kressin написал:

Did you upgrade from a previous version by chance? Maybe you hit the following bug:

https://github.com/floragunncom/search-guard-module-kibana-multitenancy/issues/9

On Wednesday, May 31, 2017 at 1:58:06 PM UTC+2, Aleksey Kiselev wrote:

I try to use kibana multi-tenancy. With global tenant all good. But if i switch to other tenant and try to set default index pattern i have a error:

{“statusCode”:404,“error”:“Not Found”,“message”:“[document_missing_exception] [config][5.4.0]: document missing, with { index_uuid="n6YqxlkCT1mPbezpnrWa3g" & shard="0" & index=".kibana_-1180334778_testmain" }”}

``

But in elastic search server logs i found this message:

[2017-05-31T11:29:23,354][INFO ][o.e.c.m.MetaDataCreateIndexService] [Master-01] [.kibana_-1180334778_testmain] creating index, cause [api], templates , shards [1]/[1], mappings [config]
[2017-05-31T11:29:23,981][INFO ][o.e.c.r.a.AllocationService] [Master-01] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana_-1180334778_testmain][0]] …]).
[2017-05-31T11:29:24,230][INFO ][o.e.c.m.MetaDataMappingService] [Master-01] [.kibana_-1180334778_testmain/n6YqxlkCT1mPbezpnrWa3g] update_mapping [config]
[2017-05-31T11:31:06,358][INFO ][o.e.c.m.MetaDataMappingService] [Master-01] [.kibana_-1180334778_testmain/n6YqxlkCT1mPbezpnrWa3g] create_mapping [index-pattern]
[2017-05-31T11:31:08,786][INFO ][o.e.c.m.MetaDataMappingService] [Master-01] [.kibana_-1180334778_testmain/n6YqxlkCT1mPbezpnrWa3g] update_mapping [index-pattern]

``

What am I doing wrong?

Rakesh · May 31, 2017, 7:35pm

I am also getting same error with version 5.2.2. Where do we update the version or is there any fix for this?

Config: Error 404 Not Found: [document_missing_exception] [config][5.2.2]: document missing, with { index_uuid=“AJzpRmu62bVeyxWXIw" & shard=“0” & index=".kibana-85117435_cadoperations” }

Aleksey_Kiselev · May 31, 2017, 9:51pm

At first step I check to global tenant and make query in Dev tools (sense):

GET .kibana/config

``

Look on all documents. In that list you should find a document with your kibana version.
At second step I’m copy document body with right version.

At third step I’m switch to other tenant and delete all .kibana/config with wrong version. Query like:

DELETE .kibana/config/5.1.2
DELETE .kibana/config/5.1.3
…

``

At next step you should put right document like:

POST .kibana/config/5.4.0
{
… document body json here …
}

``

It’s all. Try this please.

···

среда, 31 мая 2017 г., 22:35:02 UTC+3 пользователь Rakesh написал:

I am also getting same error with version 5.2.2. Where do we update the version or is there any fix for this?

Config: Error 404 Not Found: [document_missing_exception] [config][5.2.2]: document missing, with { index_uuid=“AJzpRmu62bVeyxWXIw" & shard=“0” & index=".kibana-85117435_cadoperations” }

Rakesh · June 1, 2017, 6:12pm

It worked
Thank you Aleksey!

···

searchguard_google_group · June 10, 2017, 7:25pm

fixed with multitenancy v2 released today

···

On Thursday, 1 June 2017 20:12:50 UTC+2, Rakesh wrote:

It worked
Thank you Aleksey!

Topic		Replies	Views
"document missing" error when creating new tenant Search Guard	5	729	December 14, 2017
Error creating private tenant index pattern Search Guard	10	1930	June 11, 2019
Kibana Multitenancy not working Search Guard	5	706	May 30, 2018
Index patterns are set for all tenants in Kibana Multitenancy Search Guard	11	890	February 22, 2018
Missing tenant index Search Guard	8	412	April 10, 2023

Multi-tenancy kibana default index error

Related topics