Performance in Kibana with SearchGuard

Elasticsearch version:
7.10.2
Server OS version:
ubuntu-18.04
Kibana version (if relevant):
7.10.2
Browser version (if relevant):
Chrome 89.0.4389.82
SearchGuard Plugins version
49.0.0

Describe the issue:
We have big enough cluster. Without SearchGuard we get result on Discovery page for 7-8 seconds. It about 1.3M records, period is 1 hour.
When we moving on ELK with SearchGuard, it going to timeout even with 30-minutes range. We can get results only for 15-minute range, sometime, and it takes up to 30-40 seconds. At the same time 15-minutes requests works for ~ 7 second using curl to elasticsearch. So, it looks like the problem is on the side of Kibana with SG plugin.
Also when we use SG the following logs in Elasticsearch (data node):
[6:43 PM] [parent] Data too large, data for [<http_request>] would be [2097177110/1.9gb], which is larger than the limit of [1677721600/1.5gb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=2097177110/1.9gb, model_inference=0/0b, accounting=0/0b]

Our cluster size:
Elasticsearch 3 master nodes with 2gb heap
Elasticsearch 3 (tried 6) client (ingest role only) nodes with 2.5gb heap
Elasticsearch 3 data-hot nodes with 4gb heap (tried 9Gb)
200Mil records on data-hot nodes.
1.3Mil records ‘last hour’
Now we use SSD disks, but before hdd worked fast enough on ELK without SG.

Provide configuration:
elasticsearch/config/elasticsearch.yml

network.host: 0.0.0.0
cluster.name: elasticsearch
cluster.initial_master_nodes: elasticsearch-master-0
discovery.seed_hosts: elasticsearch-discovery.project
elasticsearch-discovery.project
node.name: ${HOSTNAME}
node.processors: 2
searchguard.enterprise_modules_enabled: false
searchguard.ssl.http.clientauth_mode: OPTIONAL
searchguard.ssl.transport.pemcert_filepath: certificates/node.pem
searchguard.ssl.transport.pemkey_filepath: certificates/node.key
searchguard.ssl.transport.pemtrustedcas_filepath: certificates/root-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: certificates/node_http.pem
searchguard.ssl.http.pemkey_filepath: certificates/node_http.key
searchguard.ssl.http.pemtrustedcas_filepath: certificates/root-ca.pem
searchguard.nodes_dn:
- CN=node,O=project
searchguard.authcz.admin_dn:
- CN=sgadmin,O=project
action.auto_create_index: true
xpack.ml.enabled: ${XPACK_ML_ENABLED:false}
xpack.monitoring.enabled: ${XPACK_MONITORING_ENABLED:true}
xpack.security.enabled: ${XPACK_SECURITY_ENABLED:false}
xpack.watcher.enabled: ${XPACK_WATCHER_ENABLED:false}

elasticsearch/plugins/search-guard-7/sgconfig/sg_config.yml

_sg_meta:
  type: "config"
  config_version: 2
sg_config:
  dynamic:
    filtered_alias_mode: "warn"
    disable_rest_auth: false
    disable_intertransport_auth: false
    respect_request_indices_options: false
    license: null
    auth_failure_listeners: {}
    do_not_fail_on_forbidden: true
    multi_rolespan_enabled: false
    hosts_resolver_mode: "ip-only"
    transport_userrname_attribute: null
    do_not_fail_on_forbidden_empty: true
    field_anonymization_salt2: null
    kibana:
      multitenancy_enabled: false
      server_username: "system.kibanaserver"
      index: ".kibana"
      rbac_enabled: false
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: true
        internalProxies: ".*"
        remoteIpHeader: "x-forwarded-for"
    authc:
      proxy_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          challenge: false
          type: "proxy"
          config:
            user_header: "x-proxy-user"
            roles_header: "x-proxy-roles"
        authentication_backend:
          type: "noop"
          config: {}
        skip_users: []
      clientcert_auth_domain:
        http_enabled: true
        transport_enabled: false
        order: 0
        http_authenticator:
          challenge: false
          type: "clientcert"
          config:
            username_attribute: "cn"
        authentication_backend:
          type: "noop"
          config: {}
        skip_users: []
      basic_internal_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 2
        http_authenticator:
          challenge: true
          type: "basic"
          config: {}
        authentication_backend:
          type: "intern"
          config: {}
        skip_users: []
    authz:
      roles_from_another_ldap:
        http_enabled: false
        transport_enabled: false
        authorization_backend:
          type: "ldap"
          config: {}
        skipped_users: []
      roles_from_myldap:
        http_enabled: false
        transport_enabled: false
        authorization_backend:
          type: "ldap"
          config:
            enable_ssl: false
            enable_start_tls: false
            enable_ssl_client_auth: false
            verify_hostnames: true
            hosts:
            - "localhost:8389"
            bind_dn: null
            password: null
            rolebase: "ou=groups,dc=example,dc=com"
            rolesearch: "(member={0})"
            userroleattribute: null
            userrolename: "disabled"
            rolename: "cn"
            resolve_nested_roles: true
            userbase: "ou=people,dc=example,dc=com"
            usersearch: "(uid={0})"
        skipped_users: []

kibana/config/kibana.yml (if relevant)

│ kibana.yml:                                                                                                                                                                                                                             │
│ ----                                                                                                                                                                                                                                    │
│ server.host: "0.0.0.0"                                                                                                                                                                                                                  │
│ server.rewriteBasePath: false                                                                                                                                                                                                           │
│                                                                                                                                                                                                                                         │
│ status.allowAnonymous: true                                                                                                                                                                                                             │
│ xpack.security.enabled: false                                                                                                                                                                                                           │
│ logging.quiet: false                                                                                                                                                                                                                    │
│ # Default: 30000 Time in milliseconds to wait for responses from the back end or Elasticsearch. This value must be a positive integer.                                                                                                  │
│ elasticsearch.requestTimeout: 300000                                                                                                                                                                                                    │
│ # Default: 30000 Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.                                                                                                                         │
│ elasticsearch.shardTimeout: 0                                                                                                                                                                                                           │
│ elasticsearch.hosts: "https://logging-elasticsearch-client.:9200"                                                                                                                                                            │
│ elasticsearch.username: "system.kibanaserver"                                                                                                                                                                                           │
│ elasticsearch.password: "${KIBANA_PASSWORD}"                                                                                                                                                                                            │
│                                                                                                                                                                                                                                         │
│ elasticsearch.ssl.verificationMode: "certificate"                                                                                                                                                                                       │
│ elasticsearch.ssl.certificateAuthorities: "/usr/share/kibana/config/certificates/root-ca.pem"                                                                                                                                           │
│ elasticsearch.ssl.certificate: "/usr/share/kibana/config/certificates/kibana.pem"                                                                                                                                                       │
│ elasticsearch.ssl.key: "/usr/share/kibana/config/certificates/kibana.key"                                                                                                                                                               │
│ searchguard.allow_client_certificates: true                                                                                                                                                                                             │
│                                                                                                                                                                                                                                         │
│ searchguard.auth.type: "proxycache"                                                                                                                                                                                                     │
│ searchguard.proxycache.proxy_header_ip: "127.0.0.1"                                                                                                                                                                                     │
│                                                                                                                                                                                                                                         │
│ # The header that identifies the user - (required, no default)                                                                                                                                                                          │
│ searchguard.proxycache.user_header: x-proxy-user                                                                                                                            
`# The header that identifies the user's role(s) - (required, no default)                                                                                                                                                                 searchguard.proxycache.roles_header: x-proxy-roles                                                                                                                                                                                       elasticsearch.requestHeadersWhitelist: [ "Authorization", "sgtenant", "x-forwarded-for", "x-forwarded-by", "x-proxy-user", "x-proxy-roles" ]`

Hello @trautw

Is this your first SearchGuard installation or you used different versions?

Hello.
We had tried SG for ELK 6.8 about year ago, but had performance problem.
Last time we used ELK 7.10.2 without SG.
Now security aspect is critical for production and we need to research this problem.

You’ve mentioned that curl was successful and completed in around 7 seconds.
Could you share your curl command?

Could you tell me what’s the reason for using the below auth type and not proxy?

searchguard.auth.type: “proxycache”

Here is the query for elasticsearch:

curl --insecure    --cacert /sg-certificates/root-ca.pem     --key /sg-certificates/sgadmin.key     --cert /sg-certificates/sgadmin.pem     'https://logging-elasticsearch-client.project:9200/project*/_search?pretty&size=500' '{
"query":{
"range":{
"timestamp":{
"gt": "now -15m"
}}}}' 

We used proxycache by historical reasons and planned to move to proxy in nearest release.

@pablo Can you provide any recommendation?
Do you need more additional information about our env and configuration?

I try to use SearchGuard in SSL only mode, but i get seems problem.
If i run my Elasticsearch using native SSL, i don’t gets any problems with work speed.

Hello @vasilevm

I’ve tried to repro this issue but all is working on my side. I’ve noticed that you’ve used sgadmin cert to curl ES. That will ignore all the roles settings you might have.

Did you use fls or dls in any of the roles?
Could you share your roles mappings?

Hello @pablo
We use only index level security.
My sg_roles_mapping.yml in attachment.
sg_roles_mapping.yml (673 Bytes)
sg_roles.yml (2.0 KB)