Unable to login to Kibana after 7.10.1 upgrade

[Doug_Renze]

TL;DR

Unable to login to Kibana via SAML after 7.10.1 upgrade.

Issue

I’m unable to login to Kibana, after deploying v7.10.1, with Search Guard installed, similar to this post. However, unlike this post, redeploying did not resolve the issue. Fortunately, this is my test environment, but I would like to move to production, ASAP.

Details are:

• Elasticsearch version: 7.10.1, with Search Guard 48.0.0
• Kibana version: 7.10.1, with Kibana Plugin 48.0.0
• Using standard Elastic container images, with Search Guard installed.

I’m receiving the following error:

I’ve attached copies of config files and logs. Note that these config files work without any issues under previous versions, including 7.9.1, which I’m currently using in production.

elasticsearch.log (21.3 KB) elasticsearch.yml (1.9 KB) kibana.log (268.2 KB) kibana.yml (1.6 KB) sg_config.yml (4.0 KB)

[srgbnd]

Hi. Have you tried to clear the browser cookies? In the 7.10.0 we had to change the cookie. Thus you may have the cookies conflict.

I see SAML auth, failed to authorize: Error: The session cookie is absent. error in the Kibana log. Show me the browser cookies.

[Doug_Renze]

@srgbnd

Hi. Have you tried to clear the browser cookies? In the 7.10.0 we had to change the cookie. Thus you may have the cookies conflict.

That was my initial thought too, so I tried to open and authenticate from an incognito window, which I can usually do. That didn’t work. However, when I cleared all browser cookies from my main window and restarted, I’m still getting the same issue.

@srgbnd

I see SAML auth, failed to authorize: Error: The session cookie is absent. error in the Kibana log. Show me the browser cookies.

Here’s a screenshot of the browser cookies from the incognito window for a failed connection to the non-working instance:

I realize you said that the cookies had been updated in the latest version, but here’s a screenshot of the cookies from the version that still works correctly:

[srgbnd]

What does happen when you click the “Back to Kibana Home” button?
What was your upgrade procedure for the Search Guard Kibana plugin?
What IDP do you use?

[Doug_Renze]

@srgbnd

What does happen when you click the “Back to Kibana Home” button?

It attempts to go to the Kibana home, then ultimately ends up at the same screen.

@srgbnd

What was your upgrade procedure for the Search Guard Kibana plugin?

We’re deploying via Kubernetes, so I wrote a Dockerfile, which pulls copies of the Elasticsearch and Kibana images published by Elastic, and then installs Search Guard into the image. At that point, I deploy.

I just rebuilt and redeployed, and confirmed that I’m using the correct version of SG with Elasticsearch & Kibana. No luck.

[srgbnd]

I see the LDAP authz error in the Elasticsearch log.

"Caused by: org.elasticsearch.ElasticsearchSecurityException: No user admin found",
"at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.fillRoles(LDAPAuthorizationBackend.java:747) ~[dlic-search-guard-suite-security-7.10.1-48.0.0.jar:7.10.1-48.0.0]",

Do you have the admin user on the LDAP server? Does connection to LDAP server work? Look at the LDAP server logs. Do you see anything suspicious in the LDAP logs?

Are you able to fetch SAML metadata_url and entity_id data from the Elasticsearch server?

curl $SG_SAML_METADATA_URL
curl $SG_SAML_IDP_ENTITY_ID
curl $SG_SAML_SP_ENTITY_ID

I put some extra logs to the code and built the SG Kibana plugin. Install it and try again. Record the Kibana logs and send the logs to me. The build: build - Google Drive

[Doug_Renze]

Sorry for taking so long to respond. I’m not able to download files from Google Drive, as we have it blocked in our organization. Any other way you can get the plugin to me with the add’l logs?

@srgbnd

Do you have the admin user on the LDAP server? Does connection to LDAP server work? Look at the LDAP server logs. Do you see anything suspicious in the LDAP logs?

Connection to the LDAP server works. When I do the following command against Elasticsearch using an LDAP user with appropriate permissions, I get the expected output.

❯ curl -ks -u $ELK_USER:$ELK_PASSWORD https://elasticsearch.example.com:9200/_cluster/health\?pretty\=true
{
  "cluster_name" : "elk-es",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 14,
  "number_of_data_nodes" : 9,
  "active_primary_shards" : 274,
  "active_shards" : 555,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

However, I’ll need to touch base with my LDAP admin to have him check those logs.

@srgbnd

Are you able to fetch SAML metadata_url and entity_id data from the Elasticsearch server?

curl $SG_SAML_METADATA_URL
curl $SG_SAML_IDP_ENTITY_ID
curl $SG_SAML_SP_ENTITY_ID

I initially was not able to hit SG_SAML_IDP_ENTITY_ID, but have that issue resolved. However, I’m still getting the same issues.

[srgbnd]

I’ve uploaded the custom build to the official Search Guard Maven repo. You can download it from here https://maven.search-guard.com//search-guard-kibana-plugin-snapshot/my-test-SNAPSHOT/search-guard-kibana-plugin-7.10.1-master-SNAPSHOT-EXTRA-LOGGING-TEST-PURPOSES-ONLY-TO-BE-DELETED.zip

[srgbnd]

Hi. Try to put the following in the kibana.yml and let me know

searchguard.cookie.isSameSite: None
searchguard.cookie.secure: true

[Doug_Renze]

I assume that searchguard.cookie.isSameSite: None is a string, not a data type, correct? IIRC, there is no None data type, just want to confirm…

[Doug_Renze]

@srgbnd

Hi. Try to put the following in the kibana.yml and let me know

searchguard.cookie.isSameSite: None
searchguard.cookie.secure: true

That did it. So my questions are:

• What was the issue?
• If I need to redeploy my 7.9.x instance before I can upgrade it to 7.10.1, will these additional config settings negatively impact my 7.9.x instance?

[srgbnd]

What was the issue?

It is required to set the isSameSite=None to enable Kibana to send the cookie in a third-party context. And in case of SAML, the cookie is sent in a third-party (cross-site) context: Kibana → SAML (IDP) → Kibana → User.

I updated the documentation Kibana SAML | Security for Elasticsearch | Search Guard

More references:

SameSite cookies explained

https://tools.ietf.org/html/draft-west-cookie-incrementalism-00

If I need to redeploy my 7.9.x instance before I can upgrade it to 7.10.1, will these additional config settings negatively impact my 7.9.x instance?

If the users use the latest browser version, and your SAML IDP and Kibana hosts are in different domains, you need these settings to make SAML work. The setting is present in 7.9.x index.js · es-7.9.1 · search-guard / search-guard-kibana-plugin · GitLab

[Doug_Renze]

@srgbnd

It is required to set the isSameSite=None to enable Kibana to send the cookie in a third-party context. And in case of SAML, the cookie is sent in a third-party (cross-site) context: Kibana → SAML (IDP) → Kibana → User.

I updated the documentation Kibana SAML | Security for Elasticsearch | Search Guard

Thanks for the add’l info and for updating the documentation.

@srgbnd

If the users use the latest browser version, and your SAML IDP and Kibana hosts are in different domains, you need these settings to make SAML work. The setting is present in 7.9.x index.js · es-7.9.1 · search-guard / search-guard-kibana-plugin · GitLab

Neither SAML configuration nor domains changed, so I’m surprised I didn’t have this issue until 7.10.1 then.

Thanks!

[srgbnd]

@Doug_Renze

Neither SAML configuration nor domains changed, so I’m surprised I didn’t have this issue until 7.10.1 then.

Do you use the latest version of Chrome or Firefox? What are the domain names of Kibana and SAML IDP server? I don’t need the exact names, just invent something that has similar formatting. For example, kibana.ent.com and saml-idp.com.