Silent failures when creating users with SG REST API

Hello

We’re creating users with internal users REST API, while the API responses with success, a portion of users are not created at all, or can not login with the chosen password. No error/warning, only INFO logs from SG when creating the users.

Context

  • user creation and login is done by automation software process, human error is not involved
  • passwords are alphanumeric, auto generated.
  • for the same user, running the same process would fix the issue, but even the 2nd attempt is not successful all the times, 3rd or 4th attempt succeeds
  • REST API is called in small batches (around 5-10 concurrent requests)
  • requests are load balanced to multiple API servers, ElasticSearch cluster is healthy.
  • Nginx handles requests and does SSL termination, proxies to SG REST API

Here are some Nginx logs (usernames are masked in this post)

root@kibana1.internal ~ # grep -F ‘24/Jan/2019:09:56’ /var/log/nginx/kibana.*.log | grep -P ‘PUT .+searchguard’

[24/Jan/2019:09:56:19 +0100] “PUT /rest/_searchguard/api/internalusers/********* HTTP/1.1” 200 55 “-” “python-requests/2.19.1”

[24/Jan/2019:09:56:20 +0100] “PUT /rest/_searchguard/api/internalusers/********* HTTP/1.1” 200 59 “-” “python-requests/2.19.1”

[24/Jan/2019:09:56:20 +0100] “PUT /rest/_searchguard/api/internalusers/********* HTTP/1.1” 200 53 “-” “python-requests/2.19.1”

[24/Jan/2019:09:56:21 +0100] “PUT /rest/_searchguard/api/internalusers/********* HTTP/1.1” 200 58 “-” “python-requests/2.19.1”

[24/Jan/2019:09:56:21 +0100] “PUT /rest/_searchguard/api/internalusers/********* HTTP/1.1” 201 65 “-” “python-requests/2.19.1”

[24/Jan/2019:09:56:21 +0100] “PUT /rest/_searchguard/api/internalusers/********* HTTP/1.1” 201 65 “-” “python-requests/2.19.1”

[24/Jan/2019:09:56:34 +0100] “PUT /rest/_searchguard/api/internalusers/********* HTTP/1.1” 200 56 “-” “python-requests/2.19.1”

Both users created with requests above with 201 responses, had issues logging in. I requested for both users again. but the GET call to API only returned 1 of the users, the other was not created.

System info:

  • Debian GNU/Linux 8.10 (jessie)
  • Java:
  • OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-1~bpo8+1-b11)
  • OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)
  • ES and Kibana: 6.2.3
  • Kibana SG plugin: 6.2.3-14
  • SG modules:
  • Audit Logging (enterprise, 6.2.3.32-0)
  • Document and Field-level security (enterprise, 6.2.3.32-0)
  • HTTP Basic Authenticator (6.2.3.23-0)
  • Internal users authentication (6.2.3.23-0)
  • Kibana Multitenancy (enterprise, 6.2.3.32-0)
  • REST Management API (enterprise, 6.2.3.32-0)

solutions/hints are much appreciated.

Can you file an issue on github please?

···

Am 24.01.2019 um 11:57 schrieb Farzad Ghanei <farzad.ghanei@byte.nl>:

Hello

We're creating users with internal users REST API, while the API responses with success, a portion of users are not created at all, or can not login with the chosen password. No error/warning, only INFO logs from SG when creating the users.

Context
  • user creation and login is done by automation software process, human error is not involved
  • passwords are alphanumeric, auto generated.
  • for the same user, running the same process would fix the issue, but even the 2nd attempt is not successful all the times, 3rd or 4th attempt succeeds
  • REST API is called in small batches (around 5-10 concurrent requests)
  • requests are load balanced to multiple API servers, ElasticSearch cluster is healthy.
  • Nginx handles requests and does SSL termination, proxies to SG REST API

Here are some Nginx logs (usernames are masked in this post)

root@kibana1.internal ~ # grep -F '24/Jan/2019:09:56' /var/log/nginx/kibana.*.log | grep -P 'PUT .+searchguard'

[24/Jan/2019:09:56:19 +0100] "PUT /rest/_searchguard/api/internalusers/********* HTTP/1.1" 200 55 "-" "python-requests/2.19.1"
[24/Jan/2019:09:56:20 +0100] "PUT /rest/_searchguard/api/internalusers/********* HTTP/1.1" 200 59 "-" "python-requests/2.19.1"
[24/Jan/2019:09:56:20 +0100] "PUT /rest/_searchguard/api/internalusers/********* HTTP/1.1" 200 53 "-" "python-requests/2.19.1"
[24/Jan/2019:09:56:21 +0100] "PUT /rest/_searchguard/api/internalusers/********* HTTP/1.1" 200 58 "-" "python-requests/2.19.1"
[24/Jan/2019:09:56:21 +0100] "PUT /rest/_searchguard/api/internalusers/********* HTTP/1.1" 201 65 "-" "python-requests/2.19.1"
[24/Jan/2019:09:56:21 +0100] "PUT /rest/_searchguard/api/internalusers/********* HTTP/1.1" 201 65 "-" "python-requests/2.19.1"
[24/Jan/2019:09:56:34 +0100] "PUT /rest/_searchguard/api/internalusers/********* HTTP/1.1" 200 56 "-" "python-requests/2.19.1"

Both users created with requests above with 201 responses, had issues logging in. I requested for both users again. but the GET call to API only returned 1 of the users, the other was not created.

System info:
  • Debian GNU/Linux 8.10 (jessie)
  • Java:
    • OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-1~bpo8+1-b11)
    • OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)
  • ES and Kibana: 6.2.3
  • Kibana SG plugin: 6.2.3-14
  • SG modules:
    • Audit Logging (enterprise, 6.2.3.32-0)
    • Document and Field-level security (enterprise, 6.2.3.32-0)
    • HTTP Basic Authenticator (6.2.3.23-0)
    • Internal users authentication (6.2.3.23-0)
    • Kibana Multitenancy (enterprise, 6.2.3.32-0)
    • REST Management API (enterprise, 6.2.3.32-0)

solutions/hints are much appreciated.

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/6c7e0f31-01fc-425b-abc7-047b63800ed8%40googlegroups.com\.
For more options, visit https://groups.google.com/d/optout\.

Sure

https://github.com/floragunncom/search-guard/issues/628

Let’s follow the issue there.

···

On Thursday, January 24, 2019 at 12:26:46 PM UTC+1, Search Guard wrote:

Can you file an issue on github please?