Timeout while trying to modify roles in Kibana App

Search Guard
1

So I’m having a strange issue loading up the SearchGuard Roles section when I attempt to add or edit an entry; basically it times out and doesn’t load.

I think I’ve traced it back to a call to ‘_all/_mapping/field/*’ timing out (Source Location; I ran the call via curl and it took 35 minutes to run!

I think the reason for this calling taking that long is I’m transitioning to ECS based formatting for multiple index templates, which by default adds a ton of field mappings that might not even be added to the index at all.

Does anyone have any workarounds for this to get the GUI app to work? I’d much rather use the GUI for management but I can use sgadmin if needed.

2

Can you share the following data?

  1. Kibana version
  2. Kibana log
  3. Elasticsearch log
  4. Full curl command you run
  5. The fields mapping you got
3
  • 7.6.0

{“type”:“response”,“@timestamp”:“2020-03-02T09:13:55-06:00”,“tags”:,“pid”:4488,“method”:“get”,“statusCode”:200,“req”:{“url”:“/bundles/kbn-ui-shared-deps/icon.save-js.js”,“method”:“get”,“headers”:{“cache-control”:“no-cache”,“connection”:“Keep-Alive”,“pragma”:“no-cache”,“accept”:“/”,“accept-encoding”:“gzip, deflate, br”,“accept-language”:“en-US,en;q=0.9”,“host”:“127.0.0.1:5601”,“max-forwards”:“10”,“referer”:“https://elk.my.domain/app/searchguard-configuration",“user-agent”:"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36”,“sec-fetch-dest”:“script”,“dnt”:“1”,“sec-fetch-site”:“same-origin”,“sec-fetch-mode”:“no-cors”,“x-original-url”:“/bundles/kbn-ui-shared-deps/icon.save-js.js”,“x-forwarded-for”:“192.168.92.8:61465”,“x-arr-ssl”:“256|256|C=US, S=MI, L=Ann Arbor, O=Internet2, OU=InCommon, CN=InCommon ECC Server CA|, CN=es-web-01.my.domain”,“x-arr-log-id”:“dd5d2a69-6bca-4068-8dba-c42a0b9ba8e2”},“remoteAddress”:“127.0.0.1”,“userAgent”:“127.0.0.1”,“referer”:“https://elk.my.domain/app/searchguard-configuration"},“res”:{“statusCode”:200,“responseTime”:16,“contentLength”:9},“message”:"GET /bundles/kbn-ui-shared-deps/icon.save-js.js 200 16ms - 9.0B”}
{“type”:“response”,“@timestamp”:“2020-03-02T09:13:55-06:00”,“tags”:,“pid”:4488,“method”:“get”,“statusCode”:200,“req”:{“url”:“/api/v1/configuration/actiongroups”,“method”:“get”,“headers”:{“cache-control”:“no-cache”,“connection”:“Keep-Alive”,“pragma”:“no-cache”,“accept”:“application/json, text/plain, /”,“accept-encoding”:“gzip, deflate, br”,“accept-language”:“en-US,en;q=0.9”,“host”:“127.0.0.1:5601”,“max-forwards”:“10”,“referer”:“https://elk.my.domain/app/searchguard-configuration",“user-agent”:"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36”,“sec-fetch-dest”:“empty”,“kbn-version”:“7.6.0”,“dnt”:“1”,“sec-fetch-site”:“same-origin”,“sec-fetch-mode”:“cors”,“x-original-url”:“/api/v1/configuration/actiongroups”,“x-forwarded-for”:“192.168.92.8:61465”,“x-arr-ssl”:“256|256|C=US, S=MI, L=Ann Arbor, O=Internet2, OU=InCommon, CN=InCommon ECC Server CA|, CN=es-web-01.my.domain”,“x-arr-log-id”:“a8a90028-1c2e-4132-8bee-8de617f3dc57”},“remoteAddress”:“127.0.0.1”,“userAgent”:“127.0.0.1”,“referer”:“https://elk.my.domain/app/searchguard-configuration"},“res”:{“statusCode”:200,“responseTime”:46,“contentLength”:9},“message”:"GET /api/v1/configuration/actiongroups 200 46ms - 9.0B”}
{“type”:“log”,“@timestamp”:“2020-03-02T09:15:56-06:00”,“tags”:[“error”,“elasticsearch”,“configuration”],“pid”:4488,“message”:“Request error, retrying\nGET https://es-web-01.my.domain:9200/_all/_mapping/field/* => socket hang up”}
{“type”:“log”,“@timestamp”:“2020-03-02T09:15:56-06:00”,“tags”:[“error”,“elasticsearch”,“admin”],“pid”:4488,“message”:“Request error, retrying\nPOST https://es-web-01.my.domain:9200/.reporting-*/_search => socket hang up”}
{“type”:“response”,“@timestamp”:“2020-03-02T09:15:57-06:00”,“tags”:,“pid”:4488,“method”:“get”,“statusCode”:200,“req”:{“url”:“/bundles/kbn-ui-shared-deps/icon.inspect-js.js”,“method”:“get”,“headers”:{“cache-control”:“no-cache”,“connection”:“Keep-Alive”,“pragma”:“no-cache”,“accept”:“/”,“accept-encoding”:“gzip, deflate, br”,“accept-language”:“en-US,en;q=0.9”,“host”:“127.0.0.1:5601”,“max-forwards”:“10”,“referer”:“https://elk.my.domain/app/searchguard-configuration",“user-agent”:"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36”,“sec-fetch-dest”:“script”,“dnt”:“1”,“sec-fetch-site”:“same-origin”,“sec-fetch-mode”:“no-cors”,“x-original-url”:“/bundles/kbn-ui-shared-deps/icon.inspect-js.js”,“x-forwarded-for”:“192.168.92.8:61465”,“x-arr-ssl”:“256|256|C=US, S=MI, L=Ann Arbor, O=Internet2, OU=InCommon, CN=InCommon ECC Server CA|, CN=es-web-01.my.domain”,“x-arr-log-id”:“889da052-28df-4656-b1f0-54298c6a253e”},“remoteAddress”:“127.0.0.1”,“userAgent”:“127.0.0.1”,“referer”:“https://elk.my.domain/app/searchguard-configuration"},“res”:{“statusCode”:200,“responseTime”:17,“contentLength”:9},“message”:"GET /bundles/kbn-ui-shared-deps/icon.inspect-js.js 200 17ms - 9.0B”}
{“type”:“response”,“@timestamp”:“2020-03-02T09:15:57-06:00”,“tags”:,“pid”:4488,“method”:“get”,“statusCode”:200,“req”:{“url”:“/bundles/kbn-ui-shared-deps/icon.iInCircle-js.js”,“method”:“get”,“headers”:{“cache-control”:“no-cache”,“connection”:“Keep-Alive”,“pragma”:“no-cache”,“accept”:“/”,“accept-encoding”:“gzip, deflate, br”,“accept-language”:“en-US,en;q=0.9”,“host”:“127.0.0.1:5601”,“max-forwards”:“10”,“referer”:“https://elk.my.domain/app/searchguard-configuration",“user-agent”:"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36”,“sec-fetch-dest”:“script”,“dnt”:“1”,“sec-fetch-site”:“same-origin”,“sec-fetch-mode”:“no-cors”,“x-original-url”:“/bundles/kbn-ui-shared-deps/icon.iInCircle-js.js”,“x-forwarded-for”:“192.168.92.8:61465”,“x-arr-ssl”:“256|256|C=US, S=MI, L=Ann Arbor, O=Internet2, OU=InCommon, CN=InCommon ECC Server CA|, CN=es-web-01.my.domain”,“x-arr-log-id”:“707a8dda-a941-4cdd-9635-c07b3bd8b507”},“remoteAddress”:“127.0.0.1”,“userAgent”:“127.0.0.1”,“referer”:“https://elk.my.domain/app/searchguard-configuration"},“res”:{“statusCode”:200,“responseTime”:16,“contentLength”:9},“message”:"GET /bundles/kbn-ui-shared-deps/icon.iInCircle-js.js 200 16ms - 9.0B”}
{“type”:“log”,“@timestamp”:“2020-03-02T09:16:00-06:00”,“tags”:[“error”,“elasticsearch”,“data”],“pid”:4488,“message”:“Request error, retrying\nGET https://es-web-01.my.domain:9200/_cluster/settings?include_defaults=true => socket hang up”}

  • There was no output on my coordinating node’s elasticsearch log
  • curl -X GET -u “{Removed}:{Removed}” “https://elk.my.domain:9200/_all/_mapping/field/*”
  • I outputted the command to a file from curl and it is 339MB, so that might be a bit too much text :slight_smile:
4

I believe something wrong with your mapping. Normally the call should take a fraction of second and give mappings as the output. For example, it took me 0.053s to get the mapping fields:

$ time curl -k -u admin:admin -X GET https://localhost:9200/_all/_mapping/field/* | cut -c 1-800
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  129k  100  129k    0     0  3603k      0 --:--:-- --:--:-- --:--:-- 3603k
{"sg7-auditlog-2020.02.24":{"mappings":{"_index":{"full_name":"_index","mapping":{}},"_feature":{"full_name":"_feature","mapping":{}},"audit_rest_request_headers.Content-Length":{"full_name":"audit_rest_request_headers.Content-Length","mapping":{"Content-Length":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"audit_cluster_name":{"full_name":"audit_cluster_name","mapping":{"audit_cluster_name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"audit_request_origin.keyword":{"full_name":"audit_request_origin.keyword","mapping":{"keyword":{"type":"keyword","ignore_above":256}}},"audit_rest_request_params.permissions.keyword":{"full_name":"audit_rest_request_params.permissions.keyword","mapping":{"keyword":{"type":"keyword","ignore_abov

real    0m0.053s
user    0m0.023s
sys     0m0.013s
5

That probably is true if you only have 1 index, but I have 1,955 indexes, 90% of which are on a HDD storage backend. About half of my indexes are now using a full filebeat ECS mapping (which I wish I could trim down with an argument to filebeat, but it doesn’t appear to be an option right now), so I anticipate my mapping sizes to be quite large.

7

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

8

@novaksam I believe we fixed this. Could you please download this test build and try in your environment? Let me know.

12

@srgbnd I was able to load the interface, though my attempt to create a new role caused kibana to crash (like the process died, not just a lockup). I’ll do some more testing :slight_smile:

13

Could you provide more information?

  1. Kibana and Elasticsearch log if there is any error
  2. Browser console log if there is any error
14

Hi @novaksam Friendly ping. Did you get time to test it some more?

15

I’ll try to get this done soon, sorry for the delay.

16

I was working on gathering the logs for you, and then I discovered that I am able to add indexes, I just need to do it before the enumeration times out, which is good enough for me :slight_smile: