# Tenant indices migration failed on 6.8.6
**URL:** https://forum.search-guard.com/t/tenant-indices-migration-failed-on-6-8-6/1796
**Category:** Search Guard
**Created:** [April 14, 2020, 8:37am UTC](https://forum.search-guard.com/t/tenant-indices-migration-failed-on-6-8-6/1796 "2020-04-14T08:37:56Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@NetwarSystem](https://forum.search-guard.com/u/NetwarSystem)
#### Post date: [April 14, 2020, 8:37am UTC](https://forum.search-guard.com/t/tenant-indices-migration-failed-on-6-8-6/1796/1 "2020-04-14T08:37:57Z")
Hello,
I have a cluster of eight machines running Elasticsearch 6.8.6 with the appropriate Search Guard plugins for both Elasticsearch and Kibana. I have one Kibana instance configured with tenants enabled that is reachable from the internet and one without tenants that is a purely internal system.
This is the tenants enabled setup:
[https://gist.github.com/NetwarSystem/7202a2de71b64284b4540def3978553a](https://gist.github.com/NetwarSystem/7202a2de71b64284b4540def3978553a)
This is the one without tenants:
[https://gist.github.com/NetwarSystem/f837e4d5888d913fba9a62a95fbc5917](https://gist.github.com/NetwarSystem/f837e4d5888d913fba9a62a95fbc5917)
The system was working, but after a rolling restart we get this error and the tenants capable Kibana remains Yellow. The restart was for a kernel upgrade, nothing to do with Elastic, Search Guard, or Java. The non-tenants Kibana permits access, Monitoring shows all indices are green, and restart of Kibana, clearing of browser cache, and the like doesn’t fix it.
I looked for “index template” in /var/log/syslog and found these few messages, which seem to be relevant.
```auto
Apr 14 01:15:32 i01 kibana[26807]: {"type":"log","@timestamp":"2020-04-14T08:15:32Z","tags":["status","plugin:searchguard@6.8.6-19.0","info"],"pid":26807,"state":"yellow","message":"Status changed from yellow to yellow - Setting up index template.","prevState":"yellow","prevMsg":"Search Guard system routes registered."}
Apr 14 01:15:33 i01 kibana[26807]: {"type":"log","@timestamp":"2020-04-14T08:15:33Z","tags":["debug","task_manager"],"pid":26807,"message":"Not installing .kibana_task_manager index template: version 6080699 already exists."}
Apr 14 01:15:33 i01 kibana[26807]: {"type":"log","@timestamp":"2020-04-14T08:15:33Z","tags":["debug","task_manager"],"pid":26807,"message":"Not installing .kibana_task_manager index template: version 6080699 already exists."}
Apr 14 01:15:34 i01 kibana[26807]: {"type":"log","@timestamp":"2020-04-14T08:15:34Z","tags":["debug","task_manager"],"pid":26807,"message":"Not installing .kibana_task_manager index template: version 6080699 already exists."}
```
We had this once before and simply erasing all of the .kibana\_\* cleared it, but that’s not sustainable. This time around I just erased .kibana\_task\_manager and now I get this error:
But the log contents are the same
```auto
Apr 14 01:30:28 i01 kibana[20222]: {"type":"log","@timestamp":"2020-04-14T08:30:28Z","tags":["status","plugin:searchguard@6.8.6-19.0","info"],"pid":20222,"state":"yellow","message":"Status changed from yellow to yellow - Setting up index template.","prevState":"yellow","prevMsg":"Search Guard system routes registered."}
Apr 14 01:30:29 i01 kibana[20222]: {"type":"log","@timestamp":"2020-04-14T08:30:29Z","tags":["debug","task_manager"],"pid":20222,"message":"Not installing .kibana_task_manager index template: version 6080699 already exists."}
Apr 14 01:30:29 i01 kibana[20222]: {"type":"log","@timestamp":"2020-04-14T08:30:29Z","tags":["debug","task_manager"],"pid":20222,"message":"Not installing .kibana_task_manager index template: version 6080699 already exists."}
Apr 14 01:30:31 i01 kibana[20222]: {"type":"log","@timestamp":"2020-04-14T08:30:31Z","tags":["debug","task_manager"],"pid":20222,"message":"Not installing .kibana_task_manager index template: version 6080699 already exists."}
```
What do we do to correct this?
Further questions:
The .kibana indices have a single shard and no replicas. This really seems like it should be set to a shard plus five replicas, given that our system has six machines that host data. Do I misunderstand this?
After a restart of Elasticsearch our cluster manages to forget its license information. Is this is a know bug, or a unique problem for us?
---
### Author:  [@srgbnd](https://forum.search-guard.com/u/srgbnd)
#### Post date: [April 14, 2020, 7:24pm UTC](https://forum.search-guard.com/t/tenant-indices-migration-failed-on-6-8-6/1796/2 "2020-04-14T19:24:17Z")
Try to switch to the previous tenant index.
1. Stop Kibana with tenants
2. Look at Elasticsearch indices and aliases
```auto
curl -k -u admin:admin -X GET https://kibana_with_tenants:9200/_cat/indices?pretty -H 'Content-Type: application/json'
curl -k -u admin:admin -X GET https://kibana_with_tenants:9200/_cat/aliases?pretty -H 'Content-Type: application/json'
```
1. You have index and alias list similar to the following one. In my example, the name of the tenant is **trex**.
```auto
green open searchguard JdOn2tUPTMSw6CavW-UFLw 1 0 7 1 38.9kb 38.9kb
yellow open sg7-auditlog-2020.04.14 U6xc2QuVTyKq8WpGectdSQ 1 1 8 0 107.4kb 107.4kb
green open .kibana_3568561_trex_2 AvwVtZYYQdi4OvH8WONK-g 1 0 1 0 3.6kb 3.6kb
green open .kibana_3568561_trex_1 hJjxteWlRlaxU6IJEzM21Q 1 0 1 0 3.6kb 3.6kb
green open .kibana_1 SCJSkQihQiOhCAlMTXr4Ug 1 0 2 0 7kb 7kb
.kibana_3568561_trex .kibana_3568561_trex_2 - - - -
.kibana .kibana_1 - - - -
```
1. Notice that currently the system uses `.kibana_3568561_trex_2` index.
It is the new tenant index created by the saved objects migration process. See the alias pointing to this index.
```auto
.kibana_3568561_trex .kibana_3568561_trex_2 - - - -
```
1. Delete the alias. The alias is a ponter, it has no data. The data are in the indices.
[Index Aliases | Elasticsearch Guide [6.8] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/indices-aliases.html)
```auto
curl -k -u admin:admin -H 'Content-Type: application/json' -X POST /_aliases -d '{
"actions" : [
{ "remove" : { "index" : ".kibana_3568561_trex_2", "alias" : "kibana_3568561_trex" } }
]
}'
```
1. Create the alias with the sama name but for the previous tenant index.
```auto
curl -k -u admin:admin -H 'Content-Type: application/json' -X POST /_aliases -d '{
"actions" : [
{ "add" : { "index" : ".kibana_3568561_trex_1", "alias" : "kibana_3568561_trex" } }
]
}'
```
1. Start Kibana and look at the logs to see if you have any error.
Let me know.
---
### Author:  [@srgbnd](https://forum.search-guard.com/u/srgbnd)
#### Post date: [April 14, 2020, 7:29pm UTC](https://forum.search-guard.com/t/tenant-indices-migration-failed-on-6-8-6/1796/3 "2020-04-14T19:29:39Z")
I forgot one more more thing. You must delete `.kibana_3568561_trex_2`. Because the migrator will try to create a new index with the same name and fail to do this. Indices can’t be overwritten.
---
### Author:  [@srgbnd](https://forum.search-guard.com/u/srgbnd)
#### Post date: [April 14, 2020, 7:42pm UTC](https://forum.search-guard.com/t/tenant-indices-migration-failed-on-6-8-6/1796/4 "2020-04-14T19:42:34Z")
Wait, this strange message from your log
```auto
Apr 14 01:30:29 i01 kibana[20222]: {"type":"log","@timestamp":"2020-04-14T08:30:29Z","tags":["debug","task_manager"],"pid":20222,"message":"Not installing .kibana_task_manager index template: version 6080699 already exists."}
```
I just googled, there are similar issues. And the solution is
```auto
curl -XPUT -H 'Content-Type: application/json' '127.0.0.1:9200/_cluster/settings' -d '{ "persistent" : {"cluster.max_shards_per_node" : 2000}}'
```
> **[r/elasticsearch - elk stack 6.8.2 upgrade to 7.2.0; kibana won't come up](https://www.reddit.com/r/elasticsearch/comments/ceo3x5/elk_stack_682_upgrade_to_720_kibana_wont_come_up/)**
>
> 3 votes and 2 comments so far on Reddit
> **[Kibana won't start up after upgrading to 7.2.0](https://discuss.elastic.co/t/kibana-wont-start-up-after-upgrading-to-7-2-0/191128)**
>
> Hi, I'm having trouble starting up kibana after upgrading to 7.2.0; It is throwing following logs: {"type":"log","@timestamp":"2019-07-18T07:39:48Z","tags":\["reporting","debug"\],"pid":15699,"message":"Running on os \\"linux\\", distribution...
Try this solution first. And if you have problems with a tenant index migration in the future, use my instruction above.
---
### Author:  [@NetwarSystem](https://forum.search-guard.com/u/NetwarSystem)
#### Post date: [April 17, 2020, 6:31am UTC](https://forum.search-guard.com/t/tenant-indices-migration-failed-on-6-8-6/1796/5 "2020-04-17T06:31:32Z")
Tried upping the shards per node count, but we only have about 450 per. Didn’t make a difference after a Kibana restart.
> [@srgbnd](#):
>
> curl -XPUT -H ‘Content-Type: application/json’ ‘127.0.0.1:9200/\_cluster/settings’ -d ‘{ “persistent” : {“cluster.max\_shards\_per\_node” : 2000}}’
---
### Author:  [@srgbnd](https://forum.search-guard.com/u/srgbnd)
#### Post date: [April 17, 2020, 7:44am UTC](https://forum.search-guard.com/t/tenant-indices-migration-failed-on-6-8-6/1796/6 "2020-04-17T07:44:56Z")
> Setting up index template
Enable debug mode in kibana.yml `logging.verbose: false`. If SG fails to put the template you should see a detailed error, here is the error template [lib/elasticsearch/setup\_index\_template.js · es-6.8.6 · search-guard / Search Guard Kibana Plugin · GitLab](https://git.floragunn.com/search-guard/search-guard-kibana-plugin/-/blob/es-6.8.6/lib/elasticsearch/setup_index_template.js#L32)
---
### Author:  [@system](https://forum.search-guard.com/u/system)
#### Post date: [May 8, 2020, 7:45am UTC](https://forum.search-guard.com/t/tenant-indices-migration-failed-on-6-8-6/1796/7 "2020-05-08T07:45:00Z")
This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.