The Search Guard license information could not be loaded. Please contact your system administrator

[Fabien_Wernli]

I just upgraded from 5.6.6 to 6.3.2 following the (excellent) documentation at Upgrading from 5.x to 6.x | Security for Elasticsearch | Search Guard
Everything went smoothly until the kibana upgrade.
On the multitenancy page I’m getting the following error message:

The Search Guard license information could not be loaded. Please contact your system administrator

And kibana is only ever hitting the kibana-6 index.
I see no errors in kibana and ES log.

I’m using kibana-oss and elasticsearch-oss RPMs and the latest SG plugins

[Fabien_Wernli]

Aug 2 10:35:29 node42 systemd[1]: Started Kibana.
Aug 2 10:35:29 node42 systemd[1]: Starting Kibana…
Aug 2 10:35:32 node42 kibana[15375]: Status changed from uninitialized to green - Ready
Aug 2 10:35:32 node42 kibana[15375]: Status changed from uninitialized to yellow - Waiting for Elasticsearch
Aug 2 10:35:33 node42 kibana[15375]: Status changed from uninitialized to green - Ready
Aug 2 10:35:34 node42 kibana[15375]: Status changed from uninitialized to yellow - Search Guard copy JWT params disabled
Aug 2 10:35:34 node42 kibana[15375]: Status changed from yellow to yellow - Search Guard multitenancy registered. This is an Enterprise feature.
Aug 2 10:35:34 node42 kibana[15375]: Status changed from yellow to yellow - Routes for Search Guard configuration GUI registered. This is an Enterprise feature.
Aug 2 10:35:34 node42 kibana[15375]: Status changed from yellow to yellow - Search Guard system routes registered.
Aug 2 10:35:34 node42 kibana[15375]: Status changed from uninitialized to green - Ready
Aug 2 10:35:34 node42 kibana[15375]: Status changed from uninitialized to green - Ready
Aug 2 10:35:34 node42 kibana[15375]: Server running at http://127.0.0.1:8200
Aug 2 10:35:34 node42 kibana[15375]: Status changed from yellow to green - Ready
Aug 2 10:35:34 node42 kibana[15375]: Status changed from yellow to yellow - Setting up index template.
Aug 2 10:35:34 node42 kibana[15375]: Status changed from yellow to green - Search Guard plugin initialised.
Aug 2 10:36:03 node42 kibana[15375]: GET /app/searchguard-multitenancy 200 358ms - 9.0B
Aug 2 10:36:03 node42 kibana[15375]: GET /bundles/app/searchguard-multitenancy/bootstrap.js 304 29ms - 9.0B
Aug 2 10:36:03 node42 kibana[15375]: GET /ui/favicons/favicon.ico 304 12ms - 9.0B
Aug 2 10:36:03 node42 kibana[15375]: GET /bundles/commons.style.css 304 24ms - 9.0B
Aug 2 10:36:03 node42 kibana[15375]: GET /bundles/vendors.style.css 304 23ms - 9.0B
Aug 2 10:36:03 node42 kibana[15375]: GET /bundles/searchguard-multitenancy.style.css 304 12ms - 9.0B
Aug 2 10:36:03 node42 kibana[15375]: GET /bundles/vendors.bundle.js 304 64ms - 9.0B
Aug 2 10:36:03 node42 kibana[15375]: GET /bundles/commons.bundle.js 304 17ms - 9.0B
Aug 2 10:36:03 node42 kibana[15375]: GET /bundles/searchguard-multitenancy.bundle.js 304 7ms - 9.0B
Aug 2 10:36:04 node42 kibana[15375]: GET /plugins/kibana/assets/discover.svg 304 11ms - 9.0B
Aug 2 10:36:04 node42 kibana[15375]: GET /api/v1/auth/authinfo 200 13ms - 9.0B
Aug 2 10:36:04 node42 kibana[15375]: GET /plugins/kibana/assets/dashboard.svg 304 12ms - 9.0B
Aug 2 10:36:04 node42 kibana[15375]: GET /plugins/kibana/assets/wrench.svg 304 7ms - 9.0B
Aug 2 10:36:04 node42 kibana[15375]: GET /plugins/timelion/icon.svg 304 5ms - 9.0B
Aug 2 10:36:04 node42 kibana[15375]: GET /plugins/kibana/assets/visualize.svg 304 5ms - 9.0B
Aug 2 10:36:04 node42 kibana[15375]: GET /plugins/kibana/assets/settings.svg 304 5ms - 9.0B
Aug 2 10:36:04 node42 kibana[15375]: GET /plugins/searchguard/assets/networking.svg 304 7ms - 9.0B
Aug 2 10:36:04 node42 kibana[15375]: GET /plugins/kibana/assets/play-circle.svg 304 6ms - 9.0B
Aug 2 10:36:04 node42 kibana[15375]: GET /bundles/ebdca7741674eca4e1fadeca157f3ae6.svg 304 7ms - 9.0B
Aug 2 10:36:04 node42 kibana[15375]: GET /api/v1/multitenancy/info 200 13ms - 9.0B
Aug 2 10:36:04 node42 kibana[15375]: GET /api/v1/auth/authinfo 200 12ms - 9.0B
Aug 2 10:36:04 node42 kibana[15375]: GET /api/v1/multitenancy/tenant 200 6ms - 9.0B

``

[jkressin]

Is this only happening on the multi-tenancy page? That would be weird because the element that prints out the warning is a shared component. So you should see this message also on the login page (if basic auth is used).

It usually means that Kibana cannot reach ES/SG, but since you are able to log in and use Kibana, this does not seem to be the case here.

With the dev console in your browser, can you check the requests on the multi-tenancy page? One of them seems to fail, would be good to know which and what the return value is.

Also, from the machine where Kibana is running, are you able to curl /_searchguard/license? This is the endpoint that the Kibana plugin is calling.

···

On Thursday, August 2, 2018 at 10:41:52 AM UTC+2, Fabien Wernli wrote:

Aug 2 10:35:29 node42 systemd[1]: Started Kibana.
Aug 2 10:35:29 node42 systemd[1]: Starting Kibana…
Aug 2 10:35:32 node42 kibana[15375]: Status changed from uninitialized to green - Ready
Aug 2 10:35:32 node42 kibana[15375]: Status changed from uninitialized to yellow - Waiting for Elasticsearch
Aug 2 10:35:33 node42 kibana[15375]: Status changed from uninitialized to green - Ready
Aug 2 10:35:34 node42 kibana[15375]: Status changed from uninitialized to yellow - Search Guard copy JWT params disabled
Aug 2 10:35:34 node42 kibana[15375]: Status changed from yellow to yellow - Search Guard multitenancy registered. This is an Enterprise feature.
Aug 2 10:35:34 node42 kibana[15375]: Status changed from yellow to yellow - Routes for Search Guard configuration GUI registered. This is an Enterprise feature.
Aug 2 10:35:34 node42 kibana[15375]: Status changed from yellow to yellow - Search Guard system routes registered.
Aug 2 10:35:34 node42 kibana[15375]: Status changed from uninitialized to green - Ready
Aug 2 10:35:34 node42 kibana[15375]: Status changed from uninitialized to green - Ready
Aug 2 10:35:34 node42 kibana[15375]: Server running at http://127.0.0.1:8200
Aug 2 10:35:34 node42 kibana[15375]: Status changed from yellow to green - Ready
Aug 2 10:35:34 node42 kibana[15375]: Status changed from yellow to yellow - Setting up index template.
Aug 2 10:35:34 node42 kibana[15375]: Status changed from yellow to green - Search Guard plugin initialised.
Aug 2 10:36:03 node42 kibana[15375]: GET /app/searchguard-multitenancy 200 358ms - 9.0B
Aug 2 10:36:03 node42 kibana[15375]: GET /bundles/app/searchguard-multitenancy/bootstrap.js 304 29ms - 9.0B
Aug 2 10:36:03 node42 kibana[15375]: GET /ui/favicons/favicon.ico 304 12ms - 9.0B
Aug 2 10:36:03 node42 kibana[15375]: GET /bundles/commons.style.css 304 24ms - 9.0B
Aug 2 10:36:03 node42 kibana[15375]: GET /bundles/vendors.style.css 304 23ms - 9.0B
Aug 2 10:36:03 node42 kibana[15375]: GET /bundles/searchguard-multitenancy.style.css 304 12ms - 9.0B
Aug 2 10:36:03 node42 kibana[15375]: GET /bundles/vendors.bundle.js 304 64ms - 9.0B
Aug 2 10:36:03 node42 kibana[15375]: GET /bundles/commons.bundle.js 304 17ms - 9.0B
Aug 2 10:36:03 node42 kibana[15375]: GET /bundles/searchguard-multitenancy.bundle.js 304 7ms - 9.0B
Aug 2 10:36:04 node42 kibana[15375]: GET /plugins/kibana/assets/discover.svg 304 11ms - 9.0B
Aug 2 10:36:04 node42 kibana[15375]: GET /api/v1/auth/authinfo 200 13ms - 9.0B
Aug 2 10:36:04 node42 kibana[15375]: GET /plugins/kibana/assets/dashboard.svg 304 12ms - 9.0B
Aug 2 10:36:04 node42 kibana[15375]: GET /plugins/kibana/assets/wrench.svg 304 7ms - 9.0B
Aug 2 10:36:04 node42 kibana[15375]: GET /plugins/timelion/icon.svg 304 5ms - 9.0B
Aug 2 10:36:04 node42 kibana[15375]: GET /plugins/kibana/assets/visualize.svg 304 5ms - 9.0B
Aug 2 10:36:04 node42 kibana[15375]: GET /plugins/kibana/assets/settings.svg 304 5ms - 9.0B
Aug 2 10:36:04 node42 kibana[15375]: GET /plugins/searchguard/assets/networking.svg 304 7ms - 9.0B
Aug 2 10:36:04 node42 kibana[15375]: GET /plugins/kibana/assets/play-circle.svg 304 6ms - 9.0B
Aug 2 10:36:04 node42 kibana[15375]: GET /bundles/ebdca7741674eca4e1fadeca157f3ae6.svg 304 7ms - 9.0B
Aug 2 10:36:04 node42 kibana[15375]: GET /api/v1/multitenancy/info 200 13ms - 9.0B
Aug 2 10:36:04 node42 kibana[15375]: GET /api/v1/auth/authinfo 200 12ms - 9.0B
Aug 2 10:36:04 node42 kibana[15375]: GET /api/v1/multitenancy/tenant 200 6ms - 9.0B

``

[Fabien_Wernli]

yes, only on that page. we’re using proxy auth.
querying /_searchguard/license on localhost:9200 works fine.

for an unknown reason the error message now disappeared magically,
but I am still hitting the .kibana-6 index no matter what tenant I choose

[Fabien_Wernli]

In the console the only potentially relevant error I see is the following:

Public-Key-Pins: The certificate used by the site was not issued by a certificate in the default root certificate store. To prevent accidental breakage, the specified header was ignored.[Learn More]

``

[jkressin]

If you’re stuck on the .kibana-6 index then it means the tenant HTTP header is not passed correctly to SG. Because proxy + multi tenancy is part of our integration tests I don’t assume we have a bug here. Since you’re upgrading from 5 → 6 the only reason I can think of is that the MT header is not whitelisted. Due to some proxies having problems with header names that contain underscores, the header name was changed from sg_tenant to sgtenant. Can you check kibana.yml if *sgtenant *is whitelisted?

Regarding the Heisenbug error message: This is just a wild guess, but we had the case where the customer forgot to upgrade some nodes in the cluster, i.e. they were still running on ES/SG 5. If a request was routed to one of these nodes by the proxy, the customer would see the error message. If they were routed to the SG6 nodes, everything was fine. Maybe you had something similar?

I will also create an issue so that our plugin checks the whitelisted headers and issues an error or warning message when the tenant header is not whitelisted.

···

On Thursday, August 2, 2018 at 2:21:07 PM UTC+2, Fabien Wernli wrote:

In the console the only potentially relevant error I see is the following:

Public-Key-Pins: The certificate used by the site was not issued by a certificate in the default root certificate store. To prevent accidental breakage, the specified header was ignored.[Learn More]

``

[Fabien_Wernli]

the error message comes back after restarting kibana

[Fabien_Wernli]

I whitelisted both sgtenant and sg_tenant headers

[Fabien_Wernli]

I also checked that all nodes appearing in /_cat/nodes are in 6.3.2

[jkressin]

Then I’m afraid we need to see the Elasticsearch logs on debug level while you switch tenants and try to access e.g. a dashboard.

Also, when querying /_searchguard/license: Do you see any entry “modules mismatch” at the end of the JSON? Could you send the output if this endpoint so we can check the exact versions of the modules?

If possible can you also send the sg_config that you downloaded via sgadmin?

You can post them here and remove any sensitive information or send them to info@search-guard.com.

···

On Thursday, August 2, 2018 at 2:46:54 PM UTC+2, Fabien Wernli wrote:

I whitelisted both sgtenant and sg_tenant headers

[Fabien_Wernli]

So afer deleting all kibana indices, it seems the correct kibana index is used: it is being created from scratch.
Then I get to create an index-pattern, but that fails with the following message in kibana:

Could not locate that index-pattern (id: 9cd56ce0-970a-11e8-82d8-53bd24ea4c3c), click here to re-create it

``

the following API:

/api/saved_objects/_find?type=index-pattern&fields=title&per_page=10000&page=1

``

returns:

{“page”:1,“per_page”:10000,“total”:1,“saved_objects”:[{“id”:“9cd56ce0-970a-11e8-82d8-53bd24ea4c3c”,“type”:“index-pattern”,“version”:1,“attributes”:{“title”:“sys*”}}]}

``

Also, I can search it:

Curl /.kibana_-1737370602_systeme/_search?pretty
{
“took” : 4,
“timed_out” : false,
“_shards” : {
“total” : 1,
“successful” : 1,
“skipped” : 0,
“failed” : 0
},
“hits” : {
“total” : 2,
“max_score” : 1.0,
“hits” : [
{
“index" : ".kibana-1737370602_systeme”,
“_type” : “doc”,
“_id” : “index-pattern:9cd56ce0-970a-11e8-82d8-53bd24ea4c3c”,
“_score” : 1.0,
“_source” : {
“type” : “index-pattern”,
“updated_at” : “2018-08-03T10:47:27.150Z”,
“index-pattern” : {
“title” : “sys*”,
“timeFieldName” : “@timestamp”
}
}
},
{
“index" : ".kibana-1737370602_systeme”,
“_type” : “doc”,
“_id” : “config:6.3.2”,
“_score” : 1.0,
“_source” : {
“type” : “config”,
“updated_at” : “2018-08-03T10:47:29.506Z”,
“config” : {
“buildNum” : 17307,
“defaultIndex” : “9cd56ce0-970a-11e8-82d8-53bd24ea4c3c”
}
}
}
]
}
}

When I hit the discover tab however, nothing comes up but a blank page.

``

[jkressin]

This is pretty strange - I need to try to reproduce it to debug it. Is this happening every time with every tenent? Does the global tenant behave differently?

One question: You said that after you deleted the Kibana indices the tenant selection seemed to work again. Did you upgrade all existing tenant indices to the new KI 6 format as described here?

https://www.elastic.co/guide/en/kibana/current/migrating-6.0-index.html

···

On Friday, August 3, 2018 at 12:58:19 PM UTC+2, Fabien Wernli wrote:

So afer deleting all kibana indices, it seems the correct kibana index is used: it is being created from scratch.
Then I get to create an index-pattern, but that fails with the following message in kibana:

Could not locate that index-pattern (id: 9cd56ce0-970a-11e8-82d8-53bd24ea4c3c), click here to re-create it

``

the following API:

/api/saved_objects/_find?type=index-pattern&fields=title&per_page=10000&page=1

``

returns:

{“page”:1,“per_page”:10000,“total”:1,“saved_objects”:[{“id”:“9cd56ce0-970a-11e8-82d8-53bd24ea4c3c”,“type”:“index-pattern”,“version”:1,“attributes”:{“title”:“sys*”}}]}

``

Also, I can search it:

Curl /.kibana_-1737370602_systeme/_search?pretty
{
“took” : 4,
“timed_out” : false,
“_shards” : {
“total” : 1,
“successful” : 1,
“skipped” : 0,
“failed” : 0
},
“hits” : {
“total” : 2,
“max_score” : 1.0,
“hits” : [
{
“index" : ".kibana-1737370602_systeme”,
“_type” : “doc”,
“_id” : “index-pattern:9cd56ce0-970a-11e8-82d8-53bd24ea4c3c”,
“_score” : 1.0,
“_source” : {
“type” : “index-pattern”,
“updated_at” : “2018-08-03T10:47:27.150Z”,
“index-pattern” : {
“title” : “sys*”,
“timeFieldName” : “@timestamp”
}
}
},
{
“index" : ".kibana-1737370602_systeme”,
“_type” : “doc”,
“_id” : “config:6.3.2”,
“_score” : 1.0,
“_source” : {
“type” : “config”,
“updated_at” : “2018-08-03T10:47:29.506Z”,
“config” : {
“buildNum” : 17307,
“defaultIndex” : “9cd56ce0-970a-11e8-82d8-53bd24ea4c3c”
}
}
}
]
}
}

When I hit the discover tab however, nothing comes up but a blank page.

``

[Fabien_Wernli]

same for all tenants.
we disactivate the global tenant, but I can try with it.

as for the tenant selection, it works, but I still have the error message about the license.
and yes, I upgraded all indices to k6 format.

I just tried using kibana basic auth, and it doesn’t change a thing

[Fabien_Wernli]

Jochen, everything seems to fall into place when enabling the Global tenant.
I don’t see the “could not locate index pattern” anymore.

Is it possible that even if in tenant “foo”, some of the calls like “find index-pattern” still make it erroneously to .kibana instead of .kibana_foo ?

[jkressin]

Thanks for that hint, this is really helpful. Although I still don’t understand why this is happening, at least I think I can explain the effects.

There has been a major change in how the Kibana index and, more importantly, the mapping is created in 6.2.x (I think). Before that, Kibana would check on startup if the .kibana index is present. If not, it created it and put the mapping as well. Since 6.2.x the index is created upon first access, and the mapping is applied via a template.

If the mapping is not correct, then some of the Kibana calls seem to fail. This is what you experienced. So I think the mappings are not applied correctly to the tenant indices when the global tenant is not present. I will create a ticket for that and hopefully, we can provide a fix soon.

Most customers have the global tenant enabled, that’s why we did not experience this issue. So thanks for your help and analysis here, much appreciated!

···

On Friday, August 3, 2018 at 3:29:13 PM UTC+2, Fabien Wernli wrote:

Jochen, everything seems to fall into place when enabling the Global tenant.
I don’t see the “could not locate index pattern” anymore.

Is it possible that even if in tenant “foo”, some of the calls like “find index-pattern” still make it erroneously to .kibana instead of .kibana_foo ?

[Fabien_Wernli]

I see two templates in ES:

• kibana_index_template:.kibana_*
• kibana_index_template:.kibana

Is the latter created by kibana, and the former by the searchguard plugin?

[Fabien_Wernli]

The latter seems to be created only if global tenant is enabled.

[Fabien_Wernli]

I don’t think it’s the mapping:

New experiment:

0. stop kibana
1. remove all kibana related templates
2. remove all kibana indices
3. disable global tenant
4. start kibana
   ===> kibana creates ES template “kibana_index_template:.kibana_*”
5. choose tenant “foo”, create index pattern “foo-*” (note the index-pattern name)
   ===> kibana index created, but kibana can’t find index uuid
6. stop kibana
7. enable global tenant
8. start kibana
   ===> kibana creates ES template “kibana_index_template:.kibana”
9. choose global tenant, create index pattern "bar-" (different from foo-)
10. choose tenant “foo”
    ===> it works

Note that both index patterns (in global and foo tenant) are different (content and uuid).
So why does kibana somehow need global .kibana index while in tenant foo?

[Fabien_Wernli]

So as a conclusion I think I had two separate issues:

1. As you said, the mapping of my reindexed kibana indices ended up wrong

2. There’s something wrong when there is no .kibana index (global tenant)

3. is a bug, and 1. is my fault: I believe I reindexed fefore running kibana6, so the template was not in place