Hi,
We had an issue where after upgrade from 5.3.2 > 5.6.6 the kibana status page was Red.
The reason was that the mapping for the kibana index was incorrect. On the master we saw this:
failed to put mappings on indices [[[.kibana/5QxU1s0qQB6xP3fYdZGXzg]]], type [timelion-sheet]
java.lang.IllegalArgumentException: mapper [hits] cannot be changed from type [long] to [integer]
``
Now I was wondering why the .kibana index was used at all, as we’re using multi-tenancy feature of SG.
We deleted the .kibana index, and it was created with fresh mapping and everything worked.
So the question is: why does kibana+multitenancy still sollicit the .kibana index?
cheers
You have to differentiate between requests that are executed in the context of the currently logged in user, and requests executed by the internal Kibana server user:
(The following is true for 5.x, the request flow has changed in 6.1.x)
When Kibana starts, it performs periodic health checks. This included checking if ES is running, which version is running, if the .kibana index exuísts and if the .kibana index needs to be upgraded. If there is no .kibana index it will be created (on Kibana startup). In other words, with KI 5.x you will always see an .kibana index, even if you don’t use it. So when you upgrade from 5.3.2 to 5.6.6 Kibana detects that the .kibana index is outdated and applies new mappings.
All these requests are performed by the internal Kibana server user, thus there is no tenant information available and the update is applied to the .kibana index. Search Guard then treats the .kibana index as a blueprint/template for the tenant-specific indices and applies all changes made by the Kibana server user to the tenant indices as well.
So I guess you would have seen the Exception also without Search Guard, since it’s triggered by the regular Kibana health checks.
Hope this helps!
P.S. With 6.x onwards the .kibana index is not created on startup, but on demand. So if you don’t use the GLOBAL tenant, you will not see the .kibana index.
···
On Thursday, February 8, 2018 at 4:56:18 PM UTC+1, Fabien Wernli wrote:
Hi,
We had an issue where after upgrade from 5.3.2 > 5.6.6 the kibana status page was Red.
The reason was that the mapping for the kibana index was incorrect. On the master we saw this:
failed to put mappings on indices [[[.kibana/5QxU1s0qQB6xP3fYdZGXzg]]], type [timelion-sheet]
java.lang.IllegalArgumentException: mapper [hits] cannot be changed from type [long] to [integer]
``
Now I was wondering why the .kibana index was used at all, as we’re using multi-tenancy feature of SG.
We deleted the .kibana index, and it was created with fresh mapping and everything worked.
So the question is: why does kibana+multitenancy still sollicit the .kibana index?
cheers