Kibana upgrade failed 7.8.1 to 7.9.3

Search Guard
1

Elasticsearch version:
7.9.3 + sg plugin 47.0.0
Server OS version:
Centos 7
Kibana version
Kibana 7.8.1 + kibana sg plugin 43.0.0

Hi,
I want to upgrade Kibana from 7.8.1 version to 7.9.3 but unfortunately i have a problem with it. During process of tenant migration i have an error:

{“type”:“log”,“@timestamp”:“2021-06-10T19:23:22Z”,“tags”:[“status”,“plugin:searchguard@7.9.3-46.2.0”,“info”],“pid”:6,“state”:“yellow”,“message”:“Status changed from yellow to yellow - Tenant indices migration failed”,“prevState”:“yellow”,“prevMsg”:“Setting up index template.”}

I have around 100 tenants, when I turn off multitenancy and set one tenant migration works correctly.
Do you have any idea what’s the problem of that situation? Do you know how to check which index is problematic? In debug logs we didn’t find any specific info.

2

@Patryk can you increase the logging via kibana.yml with logging.verbose: true and try the migration process again, it should provide better details.
If this doesn’t provide more insight, have you tried solutions here (remapping aliases and deleting newly created indices) - do you see more errors in kibana log as a result?

3 
{"type":"log","@timestamp":"2021-06-24T19:33:33Z","tags":["info","Search Guard migration"],"pid":6,"message":"Starting tenant migration"}
{"type":"error","@timestamp":"2021-06-24T19:33:33Z","tags":["error","migration"],"pid":6,"level":"error","error":{"message":"Authorization Exception: Authorization Exception","name":"Error","stack":"Error: Authorization Exception: Authorization Exception\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n at IncomingMessage.wrapper (/usr/share/kibana/node_modules/lodash/lodash.js:4949:19)\n at IncomingMessage.emit (events.js:203:15)\n at endReadableNT (_stream_readable.js:1145:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)"},"message":"Authorization Exception: Authorization Exception"}
{"type":"error","@timestamp":"2021-06-24T19:33:33Z","tags":["error","Search Guard migration"],"pid":6,"level":"error","error":{"message":"Authorization Exception: Authorization Exception","name":"Error","stack":"Error: Authorization Exception: Authorization Exception\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n at IncomingMessage.wrapper (/usr/share/kibana/node_modules/lodash/lodash.js:4949:19)\n at IncomingMessage.emit (events.js:203:15)\n at endReadableNT (_stream_readable.js:1145:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)"},"message":"Authorization Exception: Authorization Exception"}
{"type":"log","@timestamp":"2021-06-24T19:33:33Z","tags":["status","plugin:searchguard@7.9.2-46.2.0","info"],"pid":6,"state":"yellow","message":"Status changed from yellow to yellow - Tenant indices migration failed","prevState":"yellow","prevMsg":"Setting up index template."}
{"type":"log","@timestamp":"2021-06-24T19:33:36Z","tags":["debug","metrics"],"pid":6,"message":"Refreshing metrics"}

and thats all what we have in logs(we set verbose mode)
Of course after each failed migration we delete new .kibana index and remapping alias

4

@Patryk Can you DM me your kibana.yml, elasticsearch.yml and sg_config.yml, please redact any sensitive info.
It looks like its complaining about lack of authentication in your case, but I can’t reproduce from my end yet.

Based on your error, there was a similar case here

5

We did it! but it last so long, we have more that 200 tenants so migrating of that last around 2hrs.

  1. We removed “%” from password of kibana user.
  2. In new config we changed in kibana.yml kibana user from “kibanaserver” to “kibana_system” and that was mistake. We forgot about setting in sg_config ( server_username: kibanaserver) so we stayed with “kibanaserver” user
  3. At the beggining we upgraded from 7.8.1 to 7.9.0 and later from 7.9.0 to 7.9.3
6

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.