Tenant indices migration failed on 6.8.6

NetwarSystem · April 14, 2020, 8:37am

Hello,

I have a cluster of eight machines running Elasticsearch 6.8.6 with the appropriate Search Guard plugins for both Elasticsearch and Kibana. I have one Kibana instance configured with tenants enabled that is reachable from the internet and one without tenants that is a purely internal system.

This is the tenants enabled setup:

https://gist.github.com/NetwarSystem/7202a2de71b64284b4540def3978553a

This is the one without tenants:

https://gist.github.com/NetwarSystem/f837e4d5888d913fba9a62a95fbc5917

The system was working, but after a rolling restart we get this error and the tenants capable Kibana remains Yellow. The restart was for a kernel upgrade, nothing to do with Elastic, Search Guard, or Java. The non-tenants Kibana permits access, Monitoring shows all indices are green, and restart of Kibana, clearing of browser cache, and the like doesn’t fix it.

I looked for “index template” in /var/log/syslog and found these few messages, which seem to be relevant.

Apr 14 01:15:32 i01 kibana[26807]: {"type":"log","@timestamp":"2020-04-14T08:15:32Z","tags":["status","plugin:searchguard@6.8.6-19.0","info"],"pid":26807,"state":"yellow","message":"Status changed from yellow to yellow - Setting up index template.","prevState":"yellow","prevMsg":"Search Guard system routes registered."}

Apr 14 01:15:33 i01 kibana[26807]: {"type":"log","@timestamp":"2020-04-14T08:15:33Z","tags":["debug","task_manager"],"pid":26807,"message":"Not installing .kibana_task_manager index template: version 6080699 already exists."}

Apr 14 01:15:33 i01 kibana[26807]: {"type":"log","@timestamp":"2020-04-14T08:15:33Z","tags":["debug","task_manager"],"pid":26807,"message":"Not installing .kibana_task_manager index template: version 6080699 already exists."}

Apr 14 01:15:34 i01 kibana[26807]: {"type":"log","@timestamp":"2020-04-14T08:15:34Z","tags":["debug","task_manager"],"pid":26807,"message":"Not installing .kibana_task_manager index template: version 6080699 already exists."}

We had this once before and simply erasing all of the .kibana_* cleared it, but that’s not sustainable. This time around I just erased .kibana_task_manager and now I get this error:

But the log contents are the same

Apr 14 01:30:28 i01 kibana[20222]: {"type":"log","@timestamp":"2020-04-14T08:30:28Z","tags":["status","plugin:searchguard@6.8.6-19.0","info"],"pid":20222,"state":"yellow","message":"Status changed from yellow to yellow - Setting up index template.","prevState":"yellow","prevMsg":"Search Guard system routes registered."}

Apr 14 01:30:29 i01 kibana[20222]: {"type":"log","@timestamp":"2020-04-14T08:30:29Z","tags":["debug","task_manager"],"pid":20222,"message":"Not installing .kibana_task_manager index template: version 6080699 already exists."}

Apr 14 01:30:29 i01 kibana[20222]: {"type":"log","@timestamp":"2020-04-14T08:30:29Z","tags":["debug","task_manager"],"pid":20222,"message":"Not installing .kibana_task_manager index template: version 6080699 already exists."}

Apr 14 01:30:31 i01 kibana[20222]: {"type":"log","@timestamp":"2020-04-14T08:30:31Z","tags":["debug","task_manager"],"pid":20222,"message":"Not installing .kibana_task_manager index template: version 6080699 already exists."}

What do we do to correct this?

Further questions:

The .kibana indices have a single shard and no replicas. This really seems like it should be set to a shard plus five replicas, given that our system has six machines that host data. Do I misunderstand this?

After a restart of Elasticsearch our cluster manages to forget its license information. Is this is a know bug, or a unique problem for us?

srgbnd · April 14, 2020, 7:24pm

Try to switch to the previous tenant index.

Stop Kibana with tenants
Look at Elasticsearch indices and aliases

curl -k -u admin:admin -X GET https://kibana_with_tenants:9200/_cat/indices?pretty -H 'Content-Type: application/json'
curl -k -u admin:admin -X GET https://kibana_with_tenants:9200/_cat/aliases?pretty -H 'Content-Type: application/json'

You have index and alias list similar to the following one. In my example, the name of the tenant is trex.

green  open searchguard             JdOn2tUPTMSw6CavW-UFLw 1 0 7 1  38.9kb  38.9kb
yellow open sg7-auditlog-2020.04.14 U6xc2QuVTyKq8WpGectdSQ 1 1 8 0 107.4kb 107.4kb
green  open .kibana_3568561_trex_2  AvwVtZYYQdi4OvH8WONK-g 1 0 1 0   3.6kb   3.6kb
green  open .kibana_3568561_trex_1  hJjxteWlRlaxU6IJEzM21Q 1 0 1 0   3.6kb   3.6kb
green  open .kibana_1               SCJSkQihQiOhCAlMTXr4Ug 1 0 2 0     7kb     7kb
.kibana_3568561_trex .kibana_3568561_trex_2 - - - -
.kibana              .kibana_1              - - - -

Notice that currently the system uses .kibana_3568561_trex_2 index.
It is the new tenant index created by the saved objects migration process. See the alias pointing to this index.

.kibana_3568561_trex .kibana_3568561_trex_2 - - - -

Delete the alias. The alias is a ponter, it has no data. The data are in the indices.
Index Aliases | Elasticsearch Guide [6.8] | Elastic

curl -k -u admin:admin -H 'Content-Type: application/json' -X POST /_aliases -d '{
    "actions" : [
        { "remove" : { "index" : ".kibana_3568561_trex_2", "alias" : "kibana_3568561_trex" } }
    ]
}'

Create the alias with the sama name but for the previous tenant index.

curl -k -u admin:admin -H 'Content-Type: application/json' -X POST /_aliases -d '{
    "actions" : [
        { "add" : { "index" : ".kibana_3568561_trex_1", "alias" : "kibana_3568561_trex" } }
    ]
}'

Start Kibana and look at the logs to see if you have any error.

Let me know.

srgbnd · April 14, 2020, 7:29pm

I forgot one more more thing. You must delete .kibana_3568561_trex_2. Because the migrator will try to create a new index with the same name and fail to do this. Indices can’t be overwritten.

srgbnd · April 14, 2020, 7:42pm

Wait, this strange message from your log

Apr 14 01:30:29 i01 kibana[20222]: {"type":"log","@timestamp":"2020-04-14T08:30:29Z","tags":["debug","task_manager"],"pid":20222,"message":"Not installing .kibana_task_manager index template: version 6080699 already exists."}

I just googled, there are similar issues. And the solution is

curl -XPUT -H 'Content-Type: application/json' '127.0.0.1:9200/_cluster/settings' -d '{ "persistent" : {"cluster.max_shards_per_node" : 2000}}'

Try this solution first. And if you have problems with a tenant index migration in the future, use my instruction above.

NetwarSystem · April 17, 2020, 6:31am

Tried upping the shards per node count, but we only have about 450 per. Didn’t make a difference after a Kibana restart.

srgbnd · April 17, 2020, 7:44am

Setting up index template

Enable debug mode in kibana.yml logging.verbose: false. If SG fails to put the template you should see a detailed error, here is the error template lib/elasticsearch/setup_index_template.js · es-6.8.6 · search-guard / Search Guard Kibana Plugin · GitLab

system · May 8, 2020, 7:45am

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Kibana upgrade failed 7.8.1 to 7.9.3 Search Guard	5	778	July 16, 2021
Permissions problem with tenants in ELK 6.8.6 Search Guard	3	610	June 3, 2020
Kibana fails to migrade tenant index Search Guard	4	1001	August 28, 2020
Missing tenant index Search Guard	8	410	April 10, 2023
ES 6.4.0 -> 6.5.2: Kibana cannot migrate index: "No cluster-level perm match for User kibanaserver" Search Guard	2	1627	December 12, 2018

Tenant indices migration failed on 6.8.6

Related Topics