Tenant indices migration failed on 6.8.6

Hello,

I have a cluster of eight machines running Elasticsearch 6.8.6 with the appropriate Search Guard plugins for both Elasticsearch and Kibana. I have one Kibana instance configured with tenants enabled that is reachable from the internet and one without tenants that is a purely internal system.

This is the tenants enabled setup:

This is the one without tenants:

The system was working, but after a rolling restart we get this error and the tenants capable Kibana remains Yellow. The restart was for a kernel upgrade, nothing to do with Elastic, Search Guard, or Java. The non-tenants Kibana permits access, Monitoring shows all indices are green, and restart of Kibana, clearing of browser cache, and the like doesn’t fix it.

I looked for “index template” in /var/log/syslog and found these few messages, which seem to be relevant.

Apr 14 01:15:32 i01 kibana[26807]: {"type":"log","@timestamp":"2020-04-14T08:15:32Z","tags":["status","plugin:searchguard@6.8.6-19.0","info"],"pid":26807,"state":"yellow","message":"Status changed from yellow to yellow - Setting up index template.","prevState":"yellow","prevMsg":"Search Guard system routes registered."}

Apr 14 01:15:33 i01 kibana[26807]: {"type":"log","@timestamp":"2020-04-14T08:15:33Z","tags":["debug","task_manager"],"pid":26807,"message":"Not installing .kibana_task_manager index template: version 6080699 already exists."}

Apr 14 01:15:33 i01 kibana[26807]: {"type":"log","@timestamp":"2020-04-14T08:15:33Z","tags":["debug","task_manager"],"pid":26807,"message":"Not installing .kibana_task_manager index template: version 6080699 already exists."}

Apr 14 01:15:34 i01 kibana[26807]: {"type":"log","@timestamp":"2020-04-14T08:15:34Z","tags":["debug","task_manager"],"pid":26807,"message":"Not installing .kibana_task_manager index template: version 6080699 already exists."}

We had this once before and simply erasing all of the .kibana_* cleared it, but that’s not sustainable. This time around I just erased .kibana_task_manager and now I get this error:

But the log contents are the same

Apr 14 01:30:28 i01 kibana[20222]: {"type":"log","@timestamp":"2020-04-14T08:30:28Z","tags":["status","plugin:searchguard@6.8.6-19.0","info"],"pid":20222,"state":"yellow","message":"Status changed from yellow to yellow - Setting up index template.","prevState":"yellow","prevMsg":"Search Guard system routes registered."}

Apr 14 01:30:29 i01 kibana[20222]: {"type":"log","@timestamp":"2020-04-14T08:30:29Z","tags":["debug","task_manager"],"pid":20222,"message":"Not installing .kibana_task_manager index template: version 6080699 already exists."}

Apr 14 01:30:29 i01 kibana[20222]: {"type":"log","@timestamp":"2020-04-14T08:30:29Z","tags":["debug","task_manager"],"pid":20222,"message":"Not installing .kibana_task_manager index template: version 6080699 already exists."}

Apr 14 01:30:31 i01 kibana[20222]: {"type":"log","@timestamp":"2020-04-14T08:30:31Z","tags":["debug","task_manager"],"pid":20222,"message":"Not installing .kibana_task_manager index template: version 6080699 already exists."}

What do we do to correct this?

Further questions:

The .kibana indices have a single shard and no replicas. This really seems like it should be set to a shard plus five replicas, given that our system has six machines that host data. Do I misunderstand this?

After a restart of Elasticsearch our cluster manages to forget its license information. Is this is a know bug, or a unique problem for us?

Try to switch to the previous tenant index.

  1. Stop Kibana with tenants
  2. Look at Elasticsearch indices and aliases
curl -k -u admin:admin -X GET https://kibana_with_tenants:9200/_cat/indices?pretty -H 'Content-Type: application/json'
curl -k -u admin:admin -X GET https://kibana_with_tenants:9200/_cat/aliases?pretty -H 'Content-Type: application/json'
  1. You have index and alias list similar to the following one. In my example, the name of the tenant is trex.
green  open searchguard             JdOn2tUPTMSw6CavW-UFLw 1 0 7 1  38.9kb  38.9kb
yellow open sg7-auditlog-2020.04.14 U6xc2QuVTyKq8WpGectdSQ 1 1 8 0 107.4kb 107.4kb
green  open .kibana_3568561_trex_2  AvwVtZYYQdi4OvH8WONK-g 1 0 1 0   3.6kb   3.6kb
green  open .kibana_3568561_trex_1  hJjxteWlRlaxU6IJEzM21Q 1 0 1 0   3.6kb   3.6kb
green  open .kibana_1               SCJSkQihQiOhCAlMTXr4Ug 1 0 2 0     7kb     7kb
.kibana_3568561_trex .kibana_3568561_trex_2 - - - -
.kibana              .kibana_1              - - - -
  1. Notice that currently the system uses .kibana_3568561_trex_2 index.
    It is the new tenant index created by the saved objects migration process. See the alias pointing to this index.
.kibana_3568561_trex .kibana_3568561_trex_2 - - - -
  1. Delete the alias. The alias is a ponter, it has no data. The data are in the indices.
    https://www.elastic.co/guide/en/elasticsearch/reference/6.8/indices-aliases.html
curl -k -u admin:admin -H 'Content-Type: application/json' -X POST /_aliases -d '{
    "actions" : [
        { "remove" : { "index" : ".kibana_3568561_trex_2", "alias" : "kibana_3568561_trex" } }
    ]
}'
  1. Create the alias with the sama name but for the previous tenant index.
curl -k -u admin:admin -H 'Content-Type: application/json' -X POST /_aliases -d '{
    "actions" : [
        { "add" : { "index" : ".kibana_3568561_trex_1", "alias" : "kibana_3568561_trex" } }
    ]
}'
  1. Start Kibana and look at the logs to see if you have any error.

Let me know.

I forgot one more more thing. You must delete .kibana_3568561_trex_2. Because the migrator will try to create a new index with the same name and fail to do this. Indices can’t be overwritten.

Wait, this strange message from your log

Apr 14 01:30:29 i01 kibana[20222]: {"type":"log","@timestamp":"2020-04-14T08:30:29Z","tags":["debug","task_manager"],"pid":20222,"message":"Not installing .kibana_task_manager index template: version 6080699 already exists."}

I just googled, there are similar issues. And the solution is

curl -XPUT -H 'Content-Type: application/json' '127.0.0.1:9200/_cluster/settings' -d '{ "persistent" : {"cluster.max_shards_per_node" : 2000}}'


Try this solution first. And if you have problems with a tenant index migration in the future, use my instruction above.

1 Like

Tried upping the shards per node count, but we only have about 450 per. Didn’t make a difference after a Kibana restart.

Setting up index template

Enable debug mode in kibana.yml logging.verbose: false. If SG fails to put the template you should see a detailed error, here is the error template https://git.floragunn.com/search-guard/search-guard-kibana-plugin/-/blob/es-6.8.6/lib/elasticsearch/setup_index_template.js#L32

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.