Elasticsearch version:
7.10.2
Server OS version:
ubuntu-18.04
Kibana version (if relevant):
7.10.2
Browser version (if relevant):
Chrome 89.0.4389.82
SearchGuard Plugins version
49.0.0
Describe the issue:
We have big enough cluster. Without SearchGuard we get result on Discovery page for 7-8 seconds. It about 1.3M records, period is 1 hour.
When we moving on ELK with SearchGuard, it going to timeout even with 30-minutes range. We can get results only for 15-minute range, sometime, and it takes up to 30-40 seconds. At the same time 15-minutes requests works for ~ 7 second using curl to elasticsearch. So, it looks like the problem is on the side of Kibana with SG plugin.
Also when we use SG the following logs in Elasticsearch (data node):
[6:43 PM] [parent] Data too large, data for [<http_request>] would be [2097177110/1.9gb], which is larger than the limit of [1677721600/1.5gb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=2097177110/1.9gb, model_inference=0/0b, accounting=0/0b]
Our cluster size:
Elasticsearch 3 master nodes with 2gb heap
Elasticsearch 3 (tried 6) client (ingest role only) nodes with 2.5gb heap
Elasticsearch 3 data-hot nodes with 4gb heap (tried 9Gb)
200Mil records on data-hot nodes.
1.3Mil records ‘last hour’
Now we use SSD disks, but before hdd worked fast enough on ELK without SG.
Provide configuration:
elasticsearch/config/elasticsearch.yml
network.host: 0.0.0.0
cluster.name: elasticsearch
cluster.initial_master_nodes: elasticsearch-master-0
discovery.seed_hosts: elasticsearch-discovery.project
elasticsearch-discovery.project
node.name: ${HOSTNAME}
node.processors: 2
searchguard.enterprise_modules_enabled: false
searchguard.ssl.http.clientauth_mode: OPTIONAL
searchguard.ssl.transport.pemcert_filepath: certificates/node.pem
searchguard.ssl.transport.pemkey_filepath: certificates/node.key
searchguard.ssl.transport.pemtrustedcas_filepath: certificates/root-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: certificates/node_http.pem
searchguard.ssl.http.pemkey_filepath: certificates/node_http.key
searchguard.ssl.http.pemtrustedcas_filepath: certificates/root-ca.pem
searchguard.nodes_dn:
- CN=node,O=project
searchguard.authcz.admin_dn:
- CN=sgadmin,O=project
action.auto_create_index: true
xpack.ml.enabled: ${XPACK_ML_ENABLED:false}
xpack.monitoring.enabled: ${XPACK_MONITORING_ENABLED:true}
xpack.security.enabled: ${XPACK_SECURITY_ENABLED:false}
xpack.watcher.enabled: ${XPACK_WATCHER_ENABLED:false}
elasticsearch/plugins/search-guard-7/sgconfig/sg_config.yml
_sg_meta:
type: "config"
config_version: 2
sg_config:
dynamic:
filtered_alias_mode: "warn"
disable_rest_auth: false
disable_intertransport_auth: false
respect_request_indices_options: false
license: null
auth_failure_listeners: {}
do_not_fail_on_forbidden: true
multi_rolespan_enabled: false
hosts_resolver_mode: "ip-only"
transport_userrname_attribute: null
do_not_fail_on_forbidden_empty: true
field_anonymization_salt2: null
kibana:
multitenancy_enabled: false
server_username: "system.kibanaserver"
index: ".kibana"
rbac_enabled: false
http:
anonymous_auth_enabled: false
xff:
enabled: true
internalProxies: ".*"
remoteIpHeader: "x-forwarded-for"
authc:
proxy_auth_domain:
http_enabled: true
transport_enabled: true
order: 1
http_authenticator:
challenge: false
type: "proxy"
config:
user_header: "x-proxy-user"
roles_header: "x-proxy-roles"
authentication_backend:
type: "noop"
config: {}
skip_users: []
clientcert_auth_domain:
http_enabled: true
transport_enabled: false
order: 0
http_authenticator:
challenge: false
type: "clientcert"
config:
username_attribute: "cn"
authentication_backend:
type: "noop"
config: {}
skip_users: []
basic_internal_auth_domain:
http_enabled: true
transport_enabled: true
order: 2
http_authenticator:
challenge: true
type: "basic"
config: {}
authentication_backend:
type: "intern"
config: {}
skip_users: []
authz:
roles_from_another_ldap:
http_enabled: false
transport_enabled: false
authorization_backend:
type: "ldap"
config: {}
skipped_users: []
roles_from_myldap:
http_enabled: false
transport_enabled: false
authorization_backend:
type: "ldap"
config:
enable_ssl: false
enable_start_tls: false
enable_ssl_client_auth: false
verify_hostnames: true
hosts:
- "localhost:8389"
bind_dn: null
password: null
rolebase: "ou=groups,dc=example,dc=com"
rolesearch: "(member={0})"
userroleattribute: null
userrolename: "disabled"
rolename: "cn"
resolve_nested_roles: true
userbase: "ou=people,dc=example,dc=com"
usersearch: "(uid={0})"
skipped_users: []
kibana/config/kibana.yml (if relevant)
│ kibana.yml: │
│ ---- │
│ server.host: "0.0.0.0" │
│ server.rewriteBasePath: false │
│ │
│ status.allowAnonymous: true │
│ xpack.security.enabled: false │
│ logging.quiet: false │
│ # Default: 30000 Time in milliseconds to wait for responses from the back end or Elasticsearch. This value must be a positive integer. │
│ elasticsearch.requestTimeout: 300000 │
│ # Default: 30000 Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable. │
│ elasticsearch.shardTimeout: 0 │
│ elasticsearch.hosts: "https://logging-elasticsearch-client.:9200" │
│ elasticsearch.username: "system.kibanaserver" │
│ elasticsearch.password: "${KIBANA_PASSWORD}" │
│ │
│ elasticsearch.ssl.verificationMode: "certificate" │
│ elasticsearch.ssl.certificateAuthorities: "/usr/share/kibana/config/certificates/root-ca.pem" │
│ elasticsearch.ssl.certificate: "/usr/share/kibana/config/certificates/kibana.pem" │
│ elasticsearch.ssl.key: "/usr/share/kibana/config/certificates/kibana.key" │
│ searchguard.allow_client_certificates: true │
│ │
│ searchguard.auth.type: "proxycache" │
│ searchguard.proxycache.proxy_header_ip: "127.0.0.1" │
│ │
│ # The header that identifies the user - (required, no default) │
│ searchguard.proxycache.user_header: x-proxy-user
`# The header that identifies the user's role(s) - (required, no default) searchguard.proxycache.roles_header: x-proxy-roles elasticsearch.requestHeadersWhitelist: [ "Authorization", "sgtenant", "x-forwarded-for", "x-forwarded-by", "x-proxy-user", "x-proxy-roles" ]`