Unable to Search Guard not initialized (SG11)

Search Guard
1

If you think it is a bug report or you have a technical issue, please answer the following questions. For general questions, you can delete these questions.

Elasticsearch version:
7.14.0
Server OS version:
centos 8
Kibana version (if relevant):

Browser version (if relevant):

Browser OS version (if relevant):

Describe the issue:
Unable to initialize SearchGuard

Steps to reproduce:
1.
2.
3.

Expected behavior:

Provide configuration:
elasticsearch/config/elasticsearch.yml
elasticsearch/plugins/search-guard-7/sgconfig/sg_config.yml
kibana/config/kibana.yml (if relevant)

Provide logs:
Elasticsearch
Kibana (if relevant)

Screenshots (if relevant):

Errors in browser console (if relevant):

Additional data:

2

@jbalexandre1 Can you please provide more details about the issue?
What error are you seeing?
Can you provide the configuration files elasticsearch.yml and sg_config.yml?

Did you try to run sgadmin.sh to load the configuration using below, what errors are you seeing?

./sgadmin.sh -cd "path/to/plugins/search-guard-7/sgconfig" -icl -key "path/to/config/kirk-key.pem" -cert "path/to/config/kirk.pem" -cacert "path/to/config/root-ca.pem" -nhnv

3

Elasticsearch.yml config

xpack.security.enabled: false
searchguard.enterprise_modules_enabled: false
searchguard.ssl.transport.pemcert_filepath: certs/elastic-master.crt
searchguard.ssl.transport.pemkey_filepath: certs/elastic-master.key
searchguard.ssl.transport.pemkey_password: XXXXXXXXX
searchguard.ssl.transport.pemtrustedcas_filepath: certs/internal-chain-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: true

searchguard.nodes_dn:

  • CN=*.logging.svc.cluster.local,OU=black,O=XXXXXXXXX
    searchguard.authcz.admin_dn:
  • CN=kirk,OU=black,O=XXXXXXXXX

Search Guard 6.x-25.1 | Elasticsearch Security | Search Guard

searchguard.unsupported.restapi.allow_sgconfig_modification: true

4

sg_config.yml

We use basic_internal_auth_domain

For more details pls refer to Authentication | Security for Elasticsearch | Search Guard

_sg_meta:
type: “config”
config_version: 2

sg_config:
dynamic:
# Set filtered_alias_mode to ‘disallow’ to forbid more than 2 filtered aliases per index
# Set filtered_alias_mode to ‘warn’ to allow more than 2 filtered aliases per index but warns about it (default)
# Set filtered_alias_mode to ‘nowarn’ to allow more than 2 filtered aliases per index silently
#filtered_alias_mode: warn
#do_not_fail_on_forbidden: false
#kibana:
# Kibana multitenancy - NOT FREE FOR COMMERCIAL USE
# In addition to the config options below you need to set do_not_fail_on_forbidden to true (see above).
# Kibana needs to be configured for multi tenancy as well.
# See Kibana Multitenancy | Security for Elasticsearch | Search Guard for details
#multitenancy_enabled: true
#server_username: kibanaserver
#index: ‘.kibana’
http:
anonymous_auth_enabled: false
xff:
enabled: false
internalProxies: ‘192.168.0.10|192.168.0.11’ # regex pattern
#internalProxies: ‘.’ # trust all internal proxies, regex pattern
#remoteIpHeader: ‘x-forwarded-for’
###### see Pattern (Java Platform SE 7 ) for regex help
###### more information about XFF X-Forwarded-For - Wikipedia
###### and here RFC 7239: Forwarded HTTP Extension
###### and Apache Tomcat 8 Configuration Reference (8.0.53) - The Valve Component
auth_token_provider: # NOT FREE FOR COMMERCIAL USE
# To enable using Search Guard auth tokens, you also need to enable the sg_issued_jwt_auth_domain below.
enabled: false
# JWTs produced by Search Guard are signed by default with a symmetric HMAC512 hash. For production systems,
# you must replace the value specified here by your own key. You can generate a new key for example with:
# openssl rand -base64 512 | tr ‘/+’ ‘_-’
# If you want ot use another signature algorithm, you can specify a complete JWK using the attriubute jwt_signing_key.
# Refer to the documentation for details.
jwt_signing_key_hs512: “eTDZjSqRD9Abhod9iqeGX_7o93a-eElTeXWAF6FmzQshmRIrPD-C9ET3pFjJ_IBrzmWIZDk8ig-X_PIyGmKsxNMsrU-0BNWF5gJq5xOp4rYTl8z66Tw9wr8tHLxLxgJqkLSuUCRBZvlZlQ7jNdhBBxgM-hdSSzsN1T33qdIwhrUeJ-KXI5yKUXHjoWFYb9tETbYQ4NvONowkCsXK_flp-E3F_OcKe_z5iVUszAV8QfCod1zhbya540kDejXCL6N_XMmhWJqum7UJ3hgf6DEtroPSnVpHt4iR5w9ArKK-IBgluPght03gNcoNqwz7p77TFbdOmUKF_PWy1bcdbaUoSg”
# JWTs produced by Search Guard are unencrypted by default. Set a key here to activate encryption using AES Key Wrap.
# If you want ot use another signature algorithm, you can specify a complete JWK using the attriubute jwt_encryption_key.
# Refer to the documentation for details.
#jwt_encryption_key_a256kw: “…”
# Specify the maximum time period an auth token may be valid. Omit max_validity to have keys with unlimited lifetime.
# Note that when creating auth tokens, users can specify an even shorter time period.
max_validity: “1y”
# This specifies the maximum number of valid tokens a user can have at the same time.
max_tokens_per_user: 100
authc:
kerberos_auth_domain:
http_enabled: false
transport_enabled: false
order: 6
http_authenticator:
type: kerberos # NOT FREE FOR COMMERCIAL USE
challenge: true
config:
# If true a lot of kerberos/security related debugging output will be logged to standard out
krb_debug: false
# If true then the realm will be stripped from the user name
strip_realm_from_principal: true
authentication_backend:
type: noop
basic_internal_auth_domain:
description: “Authenticate via HTTP Basic against internal users database”
http_enabled: false
transport_enabled: false
order: 4
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: internal
proxy_auth_domain:
description: “Authenticate via proxy”
http_enabled: false
transport_enabled: false
order: 3
http_authenticator:
type: proxy
challenge: false
config:
user_header: “x-proxy-user”
roles_header: “x-proxy-roles”
authentication_backend:
type: noop
jwt_auth_domain:
description: “Authenticate via Json Web Token”
http_enabled: false
transport_enabled: false
order: 0
http_authenticator:
type: jwt
challenge: false
config:
signing_key: “base64 encoded HMAC key or public RSA/ECDSA pem key”
jwt_header: “Authorization”
jwt_url_parameter: null
roles_key: null
subject_key: null
authentication_backend:
type: noop
sg_issued_jwt_auth_domain:
description: “Authenticate via Json Web Tokens issued by Search Guard”
http_enabled: false
# This auth domain is only available for HTTP
order: 1
http_authenticator:
type: sg_auth_token
challenge: false
# This auth domain automatically pulls configuration from the auth_token_provider config above
authentication_backend:
type: sg_auth_token
clientcert_auth_domain:
description: “Authenticate via SSL client certificates”
http_enabled: false
transport_enabled: false
order: 2
http_authenticator:
type: clientcert
config:
username_attribute: cn #optional, if omitted DN becomes username
challenge: false
authentication_backend:
type: noop
ldap:
description: “Authenticate via LDAP or Active Directory”
http_enabled: false
transport_enabled: false
order: 5
http_authenticator:
type: basic
challenge: false
authentication_backend:
# LDAP authentication backend (authenticate users against a LDAP or Active Directory)
type: ldap # NOT FREE FOR COMMERCIAL USE
config:
# enable ldaps
enable_ssl: false
# enable start tls, enable_ssl should be false
enable_start_tls: false
# send client certificate
enable_ssl_client_auth: false
# verify ldap hostname
verify_hostnames: true
hosts:
- localhost:8389
bind_dn: null
password: null
userbase: ‘ou=people,dc=example,dc=com’
# Filter to search for users (currently in the whole subtree beneath userbase)
# {0} is substituted with the username
usersearch: ‘(sAMAccountName={0})’
# Use this attribute from the user as username (if not set then DN is used)
username_attribute: null
authz:
roles_from_myldap:
description: “Authorize via LDAP or Active Directory”
http_enabled: false
transport_enabled: false
authorization_backend:
# LDAP authorization backend (gather roles from a LDAP or Active Directory, you have to configure the above LDAP authentication backend settings too)
type: ldap # NOT FREE FOR COMMERCIAL USE
config:
# enable ldaps
enable_ssl: false
# enable start tls, enable_ssl should be false
enable_start_tls: false
# send client certificate
enable_ssl_client_auth: false
# verify ldap hostname
verify_hostnames: true
hosts:
- localhost:8389
bind_dn: null
password: null
rolebase: ‘ou=groups,dc=example,dc=com’
# Filter to search for roles (currently in the whole subtree beneath rolebase)
# {0} is substituted with the DN of the user
# {1} is substituted with the username
# {2} is substituted with an attribute value from user’s directory entry, of the authenticated user. Use userroleattribute to specify the name of the attribute
rolesearch: ‘(member={0})’
# Specify the name of the attribute which value should be substituted with {2} above
userroleattribute: null
# Roles as an attribute of the user entry
userrolename: disabled
#userrolename: memberOf
# The attribute in a role entry containing the name of that role, Default is “name”.
# Can also be “dn” to use the full DN as rolename.
rolename: cn
# Resolve nested roles transitive (roles which are members of other roles and so on …)
resolve_nested_roles: true
userbase: ‘ou=people,dc=example,dc=com’
# Filter to search for users (currently in the whole subtree beneath userbase)
# {0} is substituted with the username
usersearch: ‘(uid={0})’
# Skip users matching a user name, a wildcard or a regex pattern
#skip_users:
# - 'cn=Michael Jackson,oupeople,o=TEST’
# - ‘/\S*/’
roles_from_another_ldap:
description: “Authorize via another Active Directory”
http_enabled: false
transport_enabled: false
authorization_backend:
type: ldap # NOT FREE FOR COMMERCIAL USE
#config goes here …

5

Yes I have run this command multiple times.

The logs I get after running the command is:

Created elasticsearch keystore in /usr/share/elasticsearch/config/elasticsearch.keystore
{“type”: “server”, “timestamp”: “2021-09-20T14:41:08,438Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “version[7.14.0], pid[6], build[default/docker/dd5a0a2acaa2045ff9624f3729fc8a6f40835aa1/2021-07-29T20:49:32.864135063Z], OS[Linux/5.4.141-67.229.amzn2.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/16.0.1/16.0.1+9]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:08,440Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:08,441Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-6526633692847773061, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms1024m, -Xmx1024m, -XX:MaxDirectMemorySize=536870912, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:14,344Z”, “level”: “INFO”, “component”: “c.f.s.SearchGuardPlugin”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “ES Config path is /usr/share/elasticsearch/config” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:14,724Z”, “level”: “INFO”, “component”: “c.f.s.s.DefaultSearchGuardKeyStore”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “JVM supports TLSv1.3” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:14,726Z”, “level”: “INFO”, “component”: “c.f.s.s.DefaultSearchGuardKeyStore”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “Config directory is /usr/share/elasticsearch/config/, from there the key- and truststore files are resolved relatively” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:15,419Z”, “level”: “INFO”, “component”: “c.f.s.s.DefaultSearchGuardKeyStore”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “TLS Transport Client Provider : JDK” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:15,419Z”, “level”: “INFO”, “component”: “c.f.s.s.DefaultSearchGuardKeyStore”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “TLS Transport Server Provider : JDK” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:15,423Z”, “level”: “INFO”, “component”: “c.f.s.s.DefaultSearchGuardKeyStore”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “TLS HTTP Provider : null” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:15,423Z”, “level”: “INFO”, “component”: “c.f.s.s.DefaultSearchGuardKeyStore”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:15,423Z”, “level”: “INFO”, “component”: “c.f.s.s.DefaultSearchGuardKeyStore”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “Enabled TLS protocols for HTTP layer : ” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,343Z”, “level”: “INFO”, “component”: “c.f.s.SearchGuardPlugin”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “Clustername: elasticsearch” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,611Z”, “level”: “INFO”, “component”: “c.f.s.s.SignalsPainlessExtension”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “Loaded script whitelist: org.elasticsearch.painless.spi.Whitelist@42c9b1ee” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,618Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [aggs-matrix-stats]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,619Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [analysis-common]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,619Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [constant-keyword]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,619Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [frozen-indices]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,619Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [ingest-common]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,620Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [ingest-geoip]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,620Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [ingest-user-agent]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,620Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [kibana]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,621Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [lang-expression]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,621Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [lang-mustache]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,621Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [lang-painless]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,621Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [mapper-extras]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,622Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [mapper-version]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,622Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [parent-join]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,622Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [percolator]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,623Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [rank-eval]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,623Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [reindex]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,623Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [repositories-metering-api]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,624Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [repository-encrypted]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,624Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [repository-url]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,624Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [runtime-fields-common]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,625Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [search-business-rules]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,625Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [searchable-snapshots]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,625Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [snapshot-repo-test-kit]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,625Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [spatial]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,626Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [transform]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,626Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [transport-netty4]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,626Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [unsigned-long]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,627Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [vectors]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,627Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [wildcard]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,627Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [x-pack-aggregate-metric]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,627Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [x-pack-analytics]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,628Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [x-pack-async]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,628Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [x-pack-async-search]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,628Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [x-pack-autoscaling]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,628Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [x-pack-ccr]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,629Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [x-pack-core]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,629Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [x-pack-data-streams]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,629Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [x-pack-deprecation]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,630Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [x-pack-enrich]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,630Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [x-pack-eql]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,630Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [x-pack-fleet]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,630Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [x-pack-graph]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,631Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [x-pack-identity-provider]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,631Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [x-pack-ilm]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,631Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [x-pack-logstash]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,631Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [x-pack-ml]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,632Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [x-pack-monitoring]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,632Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [x-pack-ql]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,632Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [x-pack-rollup]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,633Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [x-pack-security]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,633Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [x-pack-shutdown]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,633Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [x-pack-sql]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,633Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [x-pack-stack]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,634Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [x-pack-text-structure]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,634Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [x-pack-voting-only-node]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,634Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded module [x-pack-watcher]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,635Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded plugin [repository-gcs]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,635Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “loaded plugin [search-guard-7]” }
{“type”: “deprecation.elasticsearch”, “timestamp”: “2021-09-20T14:41:16,639Z”, “level”: “DEPRECATION”, “component”: “o.e.d.c.s.Settings”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “[node.ml] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.” }
{“type”: “deprecation.elasticsearch”, “timestamp”: “2021-09-20T14:41:16,657Z”, “level”: “DEPRECATION”, “component”: “o.e.d.c.s.Settings”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “[node.data] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,718Z”, “level”: “INFO”, “component”: “o.e.e.NodeEnvironment”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/xvdbn)]], net usable_space [196.5gb], net total_space [196.7gb], types [ext4]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,718Z”, “level”: “INFO”, “component”: “o.e.e.NodeEnvironment”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “heap size [1gb], compressed ordinary object pointers [true]” }
{“type”: “deprecation.elasticsearch”, “timestamp”: “2021-09-20T14:41:16,728Z”, “level”: “DEPRECATION”, “component”: “o.e.d.c.s.Settings”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “[node.master] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.” }
{“type”: “deprecation.elasticsearch”, “timestamp”: “2021-09-20T14:41:16,733Z”, “level”: “DEPRECATION”, “component”: “o.e.d.c.s.Settings”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “[node.remote_cluster_client] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.” }
{“type”: “deprecation.elasticsearch”, “timestamp”: “2021-09-20T14:41:16,735Z”, “level”: “DEPRECATION”, “component”: “o.e.d.c.s.Settings”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “[node.ingest] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:16,924Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “node name [elasticsearch-master-0], node ID [LbQQh7JSQiGkSgnPVI7ghA], cluster name [elasticsearch], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]” }
{“type”: “deprecation.elasticsearch”, “timestamp”: “2021-09-20T14:41:16,926Z”, “level”: “DEPRECATION”, “component”: “o.e.d.n.Node”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “legacy role settings [node.data, node.remote_cluster_client, node.ingest, node.master, node.ml] are deprecated, use [node.roles=[transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:27,820Z”, “level”: “INFO”, “component”: “o.e.x.m.p.l.CppLogMessageHandler”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “[controller/432] [Main.cc@117] controller (64 bit): Version 7.14.0 (Build 45f9e3d7ff96e7) Copyright (c) 2021 Elasticsearch BV” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:28,515Z”, “level”: “INFO”, “component”: “o.e.i.g.LocalDatabases”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [] and watching [/usr/share/elasticsearch/config/ingest-geoip] for changes” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:28,517Z”, “level”: “INFO”, “component”: “o.e.i.g.DatabaseRegistry”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “initialized database registry, using geoip-databases directory [/tmp/elasticsearch-6526633692847773061/geoip-databases/LbQQh7JSQiGkSgnPVI7ghA]” }
{“type”: “deprecation.elasticsearch”, “timestamp”: “2021-09-20T14:41:29,728Z”, “level”: “DEPRECATION”, “component”: “o.e.d.a.ActionModule”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “The com.floragunn.searchguard.SearchGuardPlugin plugin installs a custom REST wrapper. This functionality is deprecated and will not be possible in Elasticsearch 8.0. If this plugin is intended to provide security features for Elasticsearch then you should switch to using the built-in Elasticsearch features instead.” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:29,809Z”, “level”: “INFO”, “component”: “o.e.t.NettyAllocator”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=1mb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=4mb, heap_size=1gb}]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:29,940Z”, “level”: “INFO”, “component”: “o.e.d.DiscoveryModule”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “using discovery type [zen] and seed hosts providers [settings]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:30,924Z”, “level”: “INFO”, “component”: “o.e.g.DanglingIndicesState”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:31,748Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “initialized” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:31,749Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “starting …” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:31,765Z”, “level”: “INFO”, “component”: “o.e.x.s.c.f.PersistentCache”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “persistent cache index loaded” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:32,035Z”, “level”: “INFO”, “component”: “o.e.t.TransportService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “publish_address {10.63.107.92:9300}, bound_addresses {0.0.0.0:9300}” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:32,811Z”, “level”: “INFO”, “component”: “o.e.b.BootstrapChecks”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “bound or publishing to a non-loopback address, enforcing bootstrap checks” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:32,816Z”, “level”: “INFO”, “component”: “o.e.c.c.Coordinator”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “cluster UUID [QcF0PBS_QvCwvuFRwE7d7g]” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:36,784Z”, “level”: “INFO”, “component”: “o.e.c.s.ClusterApplierService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “master node changed {previous , current [{elasticsearch-master-2}{dgi15SE0SO60ebvBVkoH8g}{1A0H74n8QX20-sUBzr0SAw}{10.63.108.11}{10.63.108.11:9300}{cdfhilmrstw}]}, added {{elasticsearch-master-2}{dgi15SE0SO60ebvBVkoH8g}{1A0H74n8QX20-sUBzr0SAw}{10.63.108.11}{10.63.108.11:9300}{cdfhilmrstw}}, term: 7, version: 126, reason: ApplyCommitRequest{term=7, version=126, sourceNode={elasticsearch-master-2}{dgi15SE0SO60ebvBVkoH8g}{1A0H74n8QX20-sUBzr0SAw}{10.63.108.11}{10.63.108.11:9300}{cdfhilmrstw}{ml.machine_memory=2147483648, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=1073741824, transform.node=true}}” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:36,840Z”, “level”: “INFO”, “component”: “o.e.h.AbstractHttpServerTransport”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “publish_address {10.63.107.92:9200}, bound_addresses {0.0.0.0:9200}”, “cluster.uuid”: “QcF0PBS_QvCwvuFRwE7d7g”, “node.id”: “LbQQh7JSQiGkSgnPVI7ghA” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:36,841Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “started”, “cluster.uuid”: “QcF0PBS_QvCwvuFRwE7d7g”, “node.id”: “LbQQh7JSQiGkSgnPVI7ghA” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:36,841Z”, “level”: “INFO”, “component”: “c.f.s.SearchGuardPlugin”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “Node started”, “cluster.uuid”: “QcF0PBS_QvCwvuFRwE7d7g”, “node.id”: “LbQQh7JSQiGkSgnPVI7ghA” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:36,841Z”, “level”: “INFO”, “component”: “c.f.s.c.ConfigurationRepository”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “Check if searchguard index exists …”, “cluster.uuid”: “QcF0PBS_QvCwvuFRwE7d7g”, “node.id”: “LbQQh7JSQiGkSgnPVI7ghA” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:36,842Z”, “level”: “INFO”, “component”: “c.f.s.c.ConfigurationRepository”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “searchguard index does not exist yet, so no need to load config on node startup. Use sgadmin to initialize cluster”, “cluster.uuid”: “QcF0PBS_QvCwvuFRwE7d7g”, “node.id”: “LbQQh7JSQiGkSgnPVI7ghA” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:36,843Z”, “level”: “INFO”, “component”: “c.f.s.SearchGuardPlugin”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “0 Search Guard modules loaded so far: ”, “cluster.uuid”: “QcF0PBS_QvCwvuFRwE7d7g”, “node.id”: “LbQQh7JSQiGkSgnPVI7ghA” }
{“type”: “server”, “timestamp”: “2021-09-20T14:41:36,843Z”, “level”: “INFO”, “component”: “c.f.s.c.ConfigurationRepository”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “Background init thread started. Install default config?: false”, “cluster.uuid”: “QcF0PBS_QvCwvuFRwE7d7g”, “node.id”: “LbQQh7JSQiGkSgnPVI7ghA” }

7

@jbalexandre1
Do you get any “errors” when running the sgadmin.sh with needed parameters? Can you paste the output here.

Does the SG index initialise when you remove the REST layer TLS part from the elasticsearch config? Or just less logging?

8

When running once and the first time the sgadmin command with the --enable-shard-allocation flag, (REST layer TLS disabled) I get no errors:

[elasticsearch@elasticsearch-master-0 tools]$ bash ./sgadmin.sh -cd /usr/share/elasticsearch/config -cacert /usr/share/elasticsearch/config/certs/internal-chain-ca.pem -cert /usr/share/elasticsearch/config/certs/kirk.crt -key /usr/share/elasticsearch/config/certs/kirk.key -keypass XXXXXXXXXXX  -h elasticsearch-master-headless -nhnv --enable-shard-allocation
Search Guard Admin v7
Will connect to elasticsearch-master-headless:9300 ... done
Connected as CN=kirk,OU=black,O=XXXXXXXXXXX
Elasticsearch Version: 7.14.0
Search Guard Version: 7.14.0-52.2.0
Persistent and transient shard allocation enabled

When I re-run the same command but without the flag --enable-shard-allocation, I still get no error:

[elasticsearch@elasticsearch-master-0 tools]$ bash ./sgadmin.sh -cd /usr/share/elasticsearch/config -cacert /usr/share/elasticsearch/config/certs/internal-chain-ca.pem -cert /usr/share/elasticsearch/config/certs/kirk.crt -key /usr/share/elasticsearch/config/certs/kirk.key -keypass XXXXXXXXXXX  -h elasticsearch-master-headless -nhnv 
Search Guard Admin v7
Will connect to elasticsearch-master-headless:9300 ... done
Connected as CN=kirk,OU=black,O=XXXXXXXXXXX
Elasticsearch Version: 7.14.0
Search Guard Version: 7.14.0-52.2.0
Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
Clustername: elasticsearch
Clusterstate: GREEN
Number of nodes: 3
Number of data nodes: 3
searchguard index already exists, so we do not need to create one.
Populate config from /usr/share/elasticsearch/config/
/usr/share/elasticsearch/config/sg_action_groups.yml OK
/usr/share/elasticsearch/config/sg_internal_users.yml OK
/usr/share/elasticsearch/config/sg_roles.yml OK
/usr/share/elasticsearch/config/sg_roles_mapping.yml OK
/usr/share/elasticsearch/config/sg_config.yml OK
/usr/share/elasticsearch/config/sg_tenants.yml OK
/usr/share/elasticsearch/config/sg_blocks.yml OK
Will update '_doc/config' with /usr/share/elasticsearch/config/sg_config.yml 
   SUCC: Configuration for 'config' created or updated
Will update '_doc/roles' with /usr/share/elasticsearch/config/sg_roles.yml 
   SUCC: Configuration for 'roles' created or updated
Will update '_doc/rolesmapping' with /usr/share/elasticsearch/config/sg_roles_mapping.yml 
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '_doc/internalusers' with /usr/share/elasticsearch/config/sg_internal_users.yml 
   SUCC: Configuration for 'internalusers' created or updated
Will update '_doc/actiongroups' with /usr/share/elasticsearch/config/sg_action_groups.yml 
   SUCC: Configuration for 'actiongroups' created or updated
Will update '_doc/tenants' with /usr/share/elasticsearch/config/sg_tenants.yml 
   SUCC: Configuration for 'tenants' created or updated
Will update '_doc/blocks' with /usr/share/elasticsearch/config/sg_blocks.yml 
   SUCC: Configuration for 'blocks' created or updated
Done with success

Yes, the SG index has been created as the output states.

9

The logs from the latest commands above:

Created elasticsearch keystore in /usr/share/elasticsearch/config/elasticsearch.keystore
{"type": "server", "timestamp": "2021-09-20T14:41:08,438Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "version[7.14.0], pid[6], build[default/docker/dd5a0a2acaa2045ff9624f3729fc8a6f40835aa1/2021-07-29T20:49:32.864135063Z], OS[Linux/5.4.141-67.229.amzn2.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/16.0.1/16.0.1+9]" }
{"type": "server", "timestamp": "2021-09-20T14:41:08,440Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]" }
{"type": "server", "timestamp": "2021-09-20T14:41:08,441Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-6526633692847773061, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms1024m, -Xmx1024m, -XX:MaxDirectMemorySize=536870912, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
{"type": "server", "timestamp": "2021-09-20T14:41:14,344Z", "level": "INFO", "component": "c.f.s.SearchGuardPlugin", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "ES Config path is /usr/share/elasticsearch/config" }
{"type": "server", "timestamp": "2021-09-20T14:41:14,724Z", "level": "INFO", "component": "c.f.s.s.DefaultSearchGuardKeyStore", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "JVM supports TLSv1.3" }
{"type": "server", "timestamp": "2021-09-20T14:41:14,726Z", "level": "INFO", "component": "c.f.s.s.DefaultSearchGuardKeyStore", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "Config directory is /usr/share/elasticsearch/config/, from there the key- and truststore files are resolved relatively" }
{"type": "server", "timestamp": "2021-09-20T14:41:15,419Z", "level": "INFO", "component": "c.f.s.s.DefaultSearchGuardKeyStore", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "TLS Transport Client Provider : JDK" }
{"type": "server", "timestamp": "2021-09-20T14:41:15,419Z", "level": "INFO", "component": "c.f.s.s.DefaultSearchGuardKeyStore", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "TLS Transport Server Provider : JDK" }
{"type": "server", "timestamp": "2021-09-20T14:41:15,423Z", "level": "INFO", "component": "c.f.s.s.DefaultSearchGuardKeyStore", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "TLS HTTP Provider             : null" }
{"type": "server", "timestamp": "2021-09-20T14:41:15,423Z", "level": "INFO", "component": "c.f.s.s.DefaultSearchGuardKeyStore", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]" }
{"type": "server", "timestamp": "2021-09-20T14:41:15,423Z", "level": "INFO", "component": "c.f.s.s.DefaultSearchGuardKeyStore", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "Enabled TLS protocols for HTTP layer      : []" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,343Z", "level": "INFO", "component": "c.f.s.SearchGuardPlugin", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "Clustername: elasticsearch" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,611Z", "level": "INFO", "component": "c.f.s.s.SignalsPainlessExtension", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "Loaded script whitelist: org.elasticsearch.painless.spi.Whitelist@42c9b1ee" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,618Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [aggs-matrix-stats]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,619Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [analysis-common]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,619Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [constant-keyword]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,619Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [frozen-indices]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,619Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [ingest-common]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,620Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [ingest-geoip]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,620Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [ingest-user-agent]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,620Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [kibana]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,621Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [lang-expression]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,621Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [lang-mustache]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,621Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [lang-painless]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,621Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [mapper-extras]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,622Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [mapper-version]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,622Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [parent-join]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,622Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [percolator]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,623Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [rank-eval]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,623Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [reindex]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,623Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [repositories-metering-api]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,624Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [repository-encrypted]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,624Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [repository-url]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,624Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [runtime-fields-common]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,625Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [search-business-rules]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,625Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [searchable-snapshots]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,625Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [snapshot-repo-test-kit]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,625Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [spatial]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,626Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [transform]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,626Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [transport-netty4]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,626Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [unsigned-long]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,627Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [vectors]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,627Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [wildcard]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,627Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-aggregate-metric]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,627Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-analytics]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,628Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-async]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,628Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-async-search]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,628Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-autoscaling]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,628Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-ccr]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,629Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-core]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,629Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-data-streams]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,629Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-deprecation]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,630Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-enrich]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,630Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-eql]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,630Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-fleet]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,630Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-graph]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,631Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-identity-provider]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,631Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-ilm]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,631Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-logstash]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,631Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-ml]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,632Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-monitoring]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,632Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-ql]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,632Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-rollup]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,633Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-security]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,633Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-shutdown]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,633Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-sql]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,633Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-stack]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,634Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-text-structure]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,634Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-voting-only-node]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,634Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-watcher]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,635Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded plugin [repository-gcs]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,635Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded plugin [search-guard-7]" }
{"type": "deprecation.elasticsearch", "timestamp": "2021-09-20T14:41:16,639Z", "level": "DEPRECATION", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[node.ml] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version." }
{"type": "deprecation.elasticsearch", "timestamp": "2021-09-20T14:41:16,657Z", "level": "DEPRECATION", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[node.data] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version." }
{"type": "server", "timestamp": "2021-09-20T14:41:16,718Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/xvdbn)]], net usable_space [196.5gb], net total_space [196.7gb], types [ext4]" }
{"type": "server", "timestamp": "2021-09-20T14:41:16,718Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "heap size [1gb], compressed ordinary object pointers [true]" }
{"type": "deprecation.elasticsearch", "timestamp": "2021-09-20T14:41:16,728Z", "level": "DEPRECATION", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[node.master] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version." }
{"type": "deprecation.elasticsearch", "timestamp": "2021-09-20T14:41:16,733Z", "level": "DEPRECATION", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[node.remote_cluster_client] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version." }
{"type": "deprecation.elasticsearch", "timestamp": "2021-09-20T14:41:16,735Z", "level": "DEPRECATION", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[node.ingest] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version." }
{"type": "server", "timestamp": "2021-09-20T14:41:16,924Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "node name [elasticsearch-master-0], node ID [LbQQh7JSQiGkSgnPVI7ghA], cluster name [elasticsearch], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]" }
{"type": "deprecation.elasticsearch", "timestamp": "2021-09-20T14:41:16,926Z", "level": "DEPRECATION", "component": "o.e.d.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "legacy role settings [node.data, node.remote_cluster_client, node.ingest, node.master, node.ml] are deprecated, use [node.roles=[transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]]" }
{"type": "server", "timestamp": "2021-09-20T14:41:27,820Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[controller/432] [Main.cc@117] controller (64 bit): Version 7.14.0 (Build 45f9e3d7ff96e7) Copyright (c) 2021 Elasticsearch BV" }
{"type": "server", "timestamp": "2021-09-20T14:41:28,515Z", "level": "INFO", "component": "o.e.i.g.LocalDatabases", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/usr/share/elasticsearch/config/ingest-geoip] for changes" }
{"type": "server", "timestamp": "2021-09-20T14:41:28,517Z", "level": "INFO", "component": "o.e.i.g.DatabaseRegistry", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "initialized database registry, using geoip-databases directory [/tmp/elasticsearch-6526633692847773061/geoip-databases/LbQQh7JSQiGkSgnPVI7ghA]" }
{"type": "deprecation.elasticsearch", "timestamp": "2021-09-20T14:41:29,728Z", "level": "DEPRECATION", "component": "o.e.d.a.ActionModule", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "The com.floragunn.searchguard.SearchGuardPlugin plugin installs a custom REST wrapper. This functionality is deprecated and will not be possible in Elasticsearch 8.0. If this plugin is intended to provide security features for Elasticsearch then you should switch to using the built-in Elasticsearch features instead." }
{"type": "server", "timestamp": "2021-09-20T14:41:29,809Z", "level": "INFO", "component": "o.e.t.NettyAllocator", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=1mb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=4mb, heap_size=1gb}]" }
{"type": "server", "timestamp": "2021-09-20T14:41:29,940Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "using discovery type [zen] and seed hosts providers [settings]" }
{"type": "server", "timestamp": "2021-09-20T14:41:30,924Z", "level": "INFO", "component": "o.e.g.DanglingIndicesState", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually" }
{"type": "server", "timestamp": "2021-09-20T14:41:31,748Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "initialized" }
{"type": "server", "timestamp": "2021-09-20T14:41:31,749Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "starting ..." }
{"type": "server", "timestamp": "2021-09-20T14:41:31,765Z", "level": "INFO", "component": "o.e.x.s.c.f.PersistentCache", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "persistent cache index loaded" }
{"type": "server", "timestamp": "2021-09-20T14:41:32,035Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "publish_address {10.63.107.92:9300}, bound_addresses {0.0.0.0:9300}" }
{"type": "server", "timestamp": "2021-09-20T14:41:32,811Z", "level": "INFO", "component": "o.e.b.BootstrapChecks", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "bound or publishing to a non-loopback address, enforcing bootstrap checks" }
{"type": "server", "timestamp": "2021-09-20T14:41:32,816Z", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "cluster UUID [QcF0PBS_QvCwvuFRwE7d7g]" }
{"type": "server", "timestamp": "2021-09-20T14:41:36,784Z", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "master node changed {previous [], current [{elasticsearch-master-2}{dgi15SE0SO60ebvBVkoH8g}{1A0H74n8QX20-sUBzr0SAw}{10.63.108.11}{10.63.108.11:9300}{cdfhilmrstw}]}, added {{elasticsearch-master-2}{dgi15SE0SO60ebvBVkoH8g}{1A0H74n8QX20-sUBzr0SAw}{10.63.108.11}{10.63.108.11:9300}{cdfhilmrstw}}, term: 7, version: 126, reason: ApplyCommitRequest{term=7, version=126, sourceNode={elasticsearch-master-2}{dgi15SE0SO60ebvBVkoH8g}{1A0H74n8QX20-sUBzr0SAw}{10.63.108.11}{10.63.108.11:9300}{cdfhilmrstw}{ml.machine_memory=2147483648, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=1073741824, transform.node=true}}" }
{"type": "server", "timestamp": "2021-09-20T14:41:36,840Z", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "publish_address {10.63.107.92:9200}, bound_addresses {0.0.0.0:9200}", "cluster.uuid": "QcF0PBS_QvCwvuFRwE7d7g", "node.id": "LbQQh7JSQiGkSgnPVI7ghA"  }
{"type": "server", "timestamp": "2021-09-20T14:41:36,841Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "started", "cluster.uuid": "QcF0PBS_QvCwvuFRwE7d7g", "node.id": "LbQQh7JSQiGkSgnPVI7ghA"  }
{"type": "server", "timestamp": "2021-09-20T14:41:36,841Z", "level": "INFO", "component": "c.f.s.SearchGuardPlugin", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "Node started", "cluster.uuid": "QcF0PBS_QvCwvuFRwE7d7g", "node.id": "LbQQh7JSQiGkSgnPVI7ghA"  }
{"type": "server", "timestamp": "2021-09-20T14:41:36,841Z", "level": "INFO", "component": "c.f.s.c.ConfigurationRepository", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "Check if searchguard index exists ...", "cluster.uuid": "QcF0PBS_QvCwvuFRwE7d7g", "node.id": "LbQQh7JSQiGkSgnPVI7ghA"  }
{"type": "server", "timestamp": "2021-09-20T14:41:36,842Z", "level": "INFO", "component": "c.f.s.c.ConfigurationRepository", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "searchguard index does not exist yet, so no need to load config on node startup. Use sgadmin to initialize cluster", "cluster.uuid": "QcF0PBS_QvCwvuFRwE7d7g", "node.id": "LbQQh7JSQiGkSgnPVI7ghA"  }
{"type": "server", "timestamp": "2021-09-20T14:41:36,843Z", "level": "INFO", "component": "c.f.s.SearchGuardPlugin", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "0 Search Guard modules loaded so far: []", "cluster.uuid": "QcF0PBS_QvCwvuFRwE7d7g", "node.id": "LbQQh7JSQiGkSgnPVI7ghA"  }
{"type": "server", "timestamp": "2021-09-20T14:41:36,843Z", "level": "INFO", "component": "c.f.s.c.ConfigurationRepository", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "Background init thread started. Install default config?: false", "cluster.uuid": "QcF0PBS_QvCwvuFRwE7d7g", "node.id": "LbQQh7JSQiGkSgnPVI7ghA"  }

My pod still aren’t ready as you can see:

$ k get po
NAME                     READY   STATUS    RESTARTS   AGE
elasticsearch-master-0   0/1     Running   0          2d4h
elasticsearch-master-1   0/1     Running   0          2d4h
elasticsearch-master-2   0/1     Running   0          2d4h

I can see in the logs some weird messages:

searchguard index does not exist yet, so no need to load config on node startup. Use sgadmin to initialize cluster

Background init thread started. Install default config?: false

gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually

The com.floragunn.searchguard.SearchGuardPlugin plugin installs a custom REST wrapper. This functionality is deprecated and will not be possible in Elasticsearch 8.0. If this plugin is intended to provide security features for Elasticsearch then you should switch to using the built-in Elasticsearch features instead

The output of sgadmin states that an index already exists. Whereas the logs states the opposite.

10

@jbalexandre1 I noticed that in your sg_config.yml you do not have basic_internal_auth_domain enabled:

basic_internal_auth_domain:
    description: “Authenticate via HTTP Basic against internal users database”
    http_enabled: false
    transport_enabled: false

This would generate error: Search Guard not initialized (SG11)

Can you enable it and see what errors you are getting?

11

Listen…

I re-install everything to have a clean environment.

Here are the steps

1- Redeploy my K8s cluster

2- Deployed ElasticSearch with SG enabled

The elasticsearch.yml file:

    xpack.security.enabled: false
    searchguard.enterprise_modules_enabled: true
    searchguard.ssl.transport.pemcert_filepath: certs/elastic-master.crt
    searchguard.ssl.transport.pemkey_filepath: certs/elastic-master.key
    searchguard.ssl.transport.pemkey_password: XXXXXXXX
    searchguard.ssl.transport.pemtrustedcas_filepath: certs/internal-chain-ca.pem
    searchguard.ssl.transport.enforce_hostname_verification: false
    searchguard.ssl.transport.resolve_hostname: true

    searchguard.nodes_dn:
    - CN=*.logging.svc.cluster.local,OU=black,O=XXXXXXXX
    searchguard.authcz.admin_dn:
    - CN=kirk,OU=black,O=XXXXXXXX

    # https://docs.search-guard.com/6.x-25/changelog-searchguard-6-x-25_1
    searchguard.unsupported.restapi.allow_sgconfig_modification: true

3- Updated and uploaded the sg*.yml file to one pod

The sg_config.yml file with both parameters to true:

_sg_meta:
  type: "config"
  config_version: 2

sg_config:
  dynamic:
      # Set filtered_alias_mode to 'disallow' to forbid more than 2 filtered aliases per index
      # Set filtered_alias_mode to 'warn' to allow more than 2 filtered aliases per index but warns about it (default)
      # Set filtered_alias_mode to 'nowarn' to allow more than 2 filtered aliases per index silently
      #filtered_alias_mode: warn
      #do_not_fail_on_forbidden: false
      #kibana:
        # Kibana multitenancy - NOT FREE FOR COMMERCIAL USE
        # In addition to the config options below you need to set do_not_fail_on_forbidden to true (see above). 
        # Kibana needs to be configured for multi tenancy as well.
        # See https://docs.search-guard.com/latest/kibana-multi-tenancy for details
        #multitenancy_enabled: true
        #server_username: kibanaserver
        #index: '.kibana'
      http:
        anonymous_auth_enabled: false
        xff:
          enabled: false
          internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
          #internalProxies: '.*' # trust all internal proxies, regex pattern
          #remoteIpHeader:  'x-forwarded-for'
          ###### see https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html for regex help
          ###### more information about XFF https://en.wikipedia.org/wiki/X-Forwarded-For
          ###### and here https://tools.ietf.org/html/rfc7239
          ###### and https://tomcat.apache.org/tomcat-8.0-doc/config/valve.html#Remote_IP_Valve          
      auth_token_provider: # NOT FREE FOR COMMERCIAL USE
        # To enable using Search Guard auth tokens, you also need to enable the sg_issued_jwt_auth_domain below.
        enabled: false
        # JWTs produced by Search Guard are signed by default with a symmetric HMAC512 hash. For production systems, 
        # you must replace the value specified here by your own key. You can generate a new key for example with:
        #  openssl rand -base64 512 | tr '/+' '_-'
        # If you want ot use another signature algorithm, you can specify a complete JWK using the attriubute jwt_signing_key.
        # Refer to the documentation for details.
        jwt_signing_key_hs512: "eTDZjSqRD9Abhod9iqeGX_7o93a-eElTeXWAF6FmzQshmRIrPD-C9ET3pFjJ_IBrzmWIZDk8ig-X_PIyGmKsxNMsrU-0BNWF5gJq5xOp4rYTl8z66Tw9wr8tHLxLxgJqkLSuUCRBZvlZlQ7jNdhBBxgM-hdSSzsN1T33qdIwhrUeJ-KXI5yKUXHjoWFYb9tETbYQ4NvONowkCsXK_flp-E3F_OcKe_z5iVUszAV8QfCod1zhbya540kDejXCL6N_XMmhWJqum7UJ3hgf6DEtroPSnVpHt4iR5w9ArKK-IBgluPght03gNcoNqwz7p77TFbdOmUKF_PWy1bcdbaUoSg"
        # JWTs produced by Search Guard are unencrypted by default. Set a key here to activate encryption using AES Key Wrap.
        # If you want ot use another signature algorithm, you can specify a complete JWK using the attriubute jwt_encryption_key.
        # Refer to the documentation for details.
        #jwt_encryption_key_a256kw: "..." 
        # Specify the maximum time period an auth token may be valid. Omit max_validity to have keys with unlimited lifetime. 
        # Note that when creating auth tokens, users can specify an even shorter time period.
        max_validity: "1y"    
        # This specifies the maximum number of valid tokens a user can have at the same time. 
        max_tokens_per_user: 100
      authc:
        kerberos_auth_domain: 
          http_enabled: false
          transport_enabled: false
          order: 6
          http_authenticator:
            type: kerberos # NOT FREE FOR COMMERCIAL USE
            challenge: true
            config:
              # If true a lot of kerberos/security related debugging output will be logged to standard out
              krb_debug: false
              # If true then the realm will be stripped from the user name
              strip_realm_from_principal: true
          authentication_backend:
            type: noop
        basic_internal_auth_domain: 
          description: "Authenticate via HTTP Basic against internal users database"
          http_enabled: true
          transport_enabled: true
          order: 4
          http_authenticator:
            type: basic
            challenge: false
          authentication_backend:
            type: internal
        proxy_auth_domain:
          description: "Authenticate via proxy"
          http_enabled: false
          transport_enabled: false
          order: 3
          http_authenticator:
            type: proxy
            challenge: false
            config:
              user_header: "x-proxy-user"
              roles_header: "x-proxy-roles"
          authentication_backend:
            type: noop
        jwt_auth_domain:
          description: "Authenticate via Json Web Token"
          http_enabled: false
          transport_enabled: false
          order: 0
          http_authenticator:
            type: jwt
            challenge: false
            config:
              signing_key: "base64 encoded HMAC key or public RSA/ECDSA pem key"
              jwt_header: "Authorization"
              jwt_url_parameter: null
              roles_key: null
              subject_key: null
          authentication_backend:
            type: noop
        sg_issued_jwt_auth_domain:
          description: "Authenticate via Json Web Tokens issued by Search Guard"
          http_enabled: false
          # This auth domain is only available for HTTP
          order: 1
          http_authenticator:
            type: sg_auth_token
            challenge: false
            # This auth domain automatically pulls configuration from the auth_token_provider config above
          authentication_backend:
            type: sg_auth_token   
        clientcert_auth_domain:
          description: "Authenticate via SSL client certificates"
          http_enabled: false
          transport_enabled: false
          order: 2
          http_authenticator:
            type: clientcert
            config:
              username_attribute: cn #optional, if omitted DN becomes username
            challenge: false
          authentication_backend:
            type: noop
        ldap:
          description: "Authenticate via LDAP or Active Directory"
          http_enabled: false
          transport_enabled: false
          order: 5
          http_authenticator:
            type: basic
            challenge: false
          authentication_backend:
            # LDAP authentication backend (authenticate users against a LDAP or Active Directory)
            type: ldap # NOT FREE FOR COMMERCIAL USE
            config:
              # enable ldaps
              enable_ssl: false
              # enable start tls, enable_ssl should be false
              enable_start_tls: false
              # send client certificate
              enable_ssl_client_auth: false
              # verify ldap hostname
              verify_hostnames: true
              hosts:
                - localhost:8389
              bind_dn: null
              password: null
              userbase: 'ou=people,dc=example,dc=com'
              # Filter to search for users (currently in the whole subtree beneath userbase)
              # {0} is substituted with the username 
              usersearch: '(sAMAccountName={0})'
              # Use this attribute from the user as username (if not set then DN is used)
              username_attribute: null
      authz:    
        roles_from_myldap:
          description: "Authorize via LDAP or Active Directory"
          http_enabled: false
          transport_enabled: false
          authorization_backend:
            # LDAP authorization backend (gather roles from a LDAP or Active Directory, you have to configure the above LDAP authentication backend settings too)
            type: ldap # NOT FREE FOR COMMERCIAL USE
            config:
              # enable ldaps
              enable_ssl: false
              # enable start tls, enable_ssl should be false
              enable_start_tls: false
              # send client certificate
              enable_ssl_client_auth: false
              # verify ldap hostname
              verify_hostnames: true
              hosts:
                - localhost:8389
              bind_dn: null
              password: null
              rolebase: 'ou=groups,dc=example,dc=com'
              # Filter to search for roles (currently in the whole subtree beneath rolebase)
              # {0} is substituted with the DN of the user
              # {1} is substituted with the username 
              # {2} is substituted with an attribute value from user's directory entry, of the authenticated user. Use userroleattribute to specify the name of the attribute            
              rolesearch: '(member={0})'
              # Specify the name of the attribute which value should be substituted with {2} above
              userroleattribute: null
              # Roles as an attribute of the user entry
              userrolename: disabled
              #userrolename: memberOf
              # The attribute in a role entry containing the name of that role, Default is "name".
              # Can also be "dn" to use the full DN as rolename.
              rolename: cn
              # Resolve nested roles transitive (roles which are members of other roles and so on ...)
              resolve_nested_roles: true
              userbase: 'ou=people,dc=example,dc=com'
              # Filter to search for users (currently in the whole subtree beneath userbase)
              # {0} is substituted with the username 
              usersearch: '(uid={0})'
              # Skip users matching a user name, a wildcard or a regex pattern
              #skip_users: 
              #  - 'cn=Michael Jackson,ou*people,o=TEST'
              #  - '/\S*/'    
        roles_from_another_ldap:
          description: "Authorize via another Active Directory"
          http_enabled: false
          transport_enabled: false
          authorization_backend:
            type: ldap # NOT FREE FOR COMMERCIAL USE

4- Ran sgadmin

[elasticsearch@elasticsearch-master-0 tools]$ bash ./sgadmin.sh -cd /usr/share/elasticsearch/config -cacert /usr/share/elasticsearch/config/certs/internal-chain-ca.pem -cert /usr/share/elasticsearch/config/certs/kirk.crt -key /usr/share/elasticsearch/config/certs/kirk.key -keypass XXXXXXXX  -h elasticsearch-master-headless -nhnv 
Search Guard Admin v7
Will connect to elasticsearch-master-headless:9300 ... done
Connected as CN=kirk,OU=black,O=XXXXXXXXX
Elasticsearch Version: 7.14.0
Search Guard Version: 7.14.0-52.2.0
Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
Clustername: elasticsearch
Clusterstate: GREEN
Number of nodes: 3
Number of data nodes: 3
searchguard index does not exists, attempt to create it ... done (0-all replicas)
Populate config from /usr/share/elasticsearch/config/
/usr/share/elasticsearch/config/sg_action_groups.yml OK
/usr/share/elasticsearch/config/sg_internal_users.yml OK
/usr/share/elasticsearch/config/sg_roles.yml OK
/usr/share/elasticsearch/config/sg_roles_mapping.yml OK
/usr/share/elasticsearch/config/sg_config.yml OK
/usr/share/elasticsearch/config/sg_tenants.yml OK
/usr/share/elasticsearch/config/sg_blocks.yml OK
Will update '_doc/config' with /usr/share/elasticsearch/config/sg_config.yml 
   SUCC: Configuration for 'config' created or updated
Will update '_doc/roles' with /usr/share/elasticsearch/config/sg_roles.yml 
   SUCC: Configuration for 'roles' created or updated
Will update '_doc/rolesmapping' with /usr/share/elasticsearch/config/sg_roles_mapping.yml 
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '_doc/internalusers' with /usr/share/elasticsearch/config/sg_internal_users.yml 
   SUCC: Configuration for 'internalusers' created or updated
Will update '_doc/actiongroups' with /usr/share/elasticsearch/config/sg_action_groups.yml 
   SUCC: Configuration for 'actiongroups' created or updated
Will update '_doc/tenants' with /usr/share/elasticsearch/config/sg_tenants.yml 
   SUCC: Configuration for 'tenants' created or updated
Will update '_doc/blocks' with /usr/share/elasticsearch/config/sg_blocks.yml 
   SUCC: Configuration for 'blocks' created or updated
Done with success
12

Actually, I see these errors in the logs:

"stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-2][10.66.104.37:9300][internal:cluster/coordination/join]",
"Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 1 does not match current term 2",
"at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:214) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:1003) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at java.util.Optional.ifPresent(Optional.java:178) ~[?:?]",
"at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:522) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.cluster.coordination.Coordinator.lambda$handleJoinRequest$7(Coordinator.java:486) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:134) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.transport.ClusterConnectionManager.connectToNode(ClusterConnectionManager.java:108) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:417) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:401) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:472) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$1(JoinHelper.java:130) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:164) ~[?:?]",
"at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:270) ~[?:?]",
"at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:136) ~[?:?]",
"at com.floragunn.searchguard.SearchGuardPlugin$7.lambda$interceptHandler$0(SearchGuardPlugin.java:690) ~[?:?]",
"at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:61) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:212) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]",
"at java.lang.Thread.run(Thread.java:831) [?:?]"] }

and

{"type": "server", "timestamp": "2021-09-23T14:19:33,663Z", "level": "ERROR", "component": "c.f.s.c.ConfigurationLoaderSG7", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "No data for internalusers while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, BLOCKS]  (index=searchguard and type=null)", "cluster.uuid": "XeM2dYi7Q-yFU2dtBP__WQ", "node.id": "HYIPqqrdTPSxwzBwo-d-NQ"  }
{"type": "server", "timestamp": "2021-09-23T14:19:33,664Z", "level": "ERROR", "component": "c.f.s.c.ConfigurationLoaderSG7", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "No data for actiongroups while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, BLOCKS]  (index=searchguard and type=null)", "cluster.uuid": "XeM2dYi7Q-yFU2dtBP__WQ", "node.id": "HYIPqqrdTPSxwzBwo-d-NQ"  }
{"type": "server", "timestamp": "2021-09-23T14:19:33,664Z", "level": "ERROR", "component": "c.f.s.c.ConfigurationLoaderSG7", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "No data for config while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, BLOCKS]  (index=searchguard and type=null)", "cluster.uuid": "XeM2dYi7Q-yFU2dtBP__WQ", "node.id": "HYIPqqrdTPSxwzBwo-d-NQ"  }
{"type": "server", "timestamp": "2021-09-23T14:19:33,664Z", "level": "ERROR", "component": "c.f.s.c.ConfigurationLoaderSG7", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "No data for roles while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, BLOCKS]  (index=searchguard and type=null)", "cluster.uuid": "XeM2dYi7Q-yFU2dtBP__WQ", "node.id": "HYIPqqrdTPSxwzBwo-d-NQ"  }
{"type": "server", "timestamp": "2021-09-23T14:19:33,664Z", "level": "ERROR", "component": "c.f.s.c.ConfigurationLoaderSG7", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "No data for rolesmapping while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, BLOCKS]  (index=searchguard and type=null)", "cluster.uuid": "XeM2dYi7Q-yFU2dtBP__WQ", "node.id": "HYIPqqrdTPSxwzBwo-d-NQ"  }
{"type": "server", "timestamp": "2021-09-23T14:19:33,667Z", "level": "ERROR", "component": "c.f.s.c.ConfigurationLoaderSG7", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "No data for tenants while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, BLOCKS]  (index=searchguard and type=null)", "cluster.uuid": "XeM2dYi7Q-yFU2dtBP__WQ", "node.id": "HYIPqqrdTPSxwzBwo-d-NQ"  }

But I don’t get the “initialization errors anymore”!

13

Now, I have another question to you.

I have 3 pods running ES.

NAME                     READY   STATUS    RESTARTS   AGE
elasticsearch-master-0   0/1     Running   0          26m
elasticsearch-master-1   0/1     Running   0          26m
elasticsearch-master-2   0/1     Running   0          26m

Obviously, they’re not in a ready state (0/1).

Do I have to repeat the same above commands on each pod or only one is sufficient?

14

By the way, is there a way that we could work this issue live, but not via this post?

It takes time (days) to test and come back, reply and wait, and so on.

Do you guys have some type of live support?

15

@jbalexandre1 no need to run on every node. Configuration is only needed to be loaded once into the security index of the cluster.

But looking at the second set of errors, it seems there are no files provided to the initialisation step. Can you verify this? And also that the certificates are at the correct location …/elasticsearch/config/certs

16

The certs are there:

[elasticsearch@elasticsearch-master-0 certs]$ pwd
/usr/share/elasticsearch/config/certs
[elasticsearch@elasticsearch-master-0 certs]$ ls -l
total 0
lrwxrwxrwx 1 root elasticsearch 25 Sep 23 14:09 elastic-master.crt -> ..data/elastic-master.crt
lrwxrwxrwx 1 root elasticsearch 30 Sep 23 14:09 elastic-master_http.crt -> ..data/elastic-master_http.crt
lrwxrwxrwx 1 root elasticsearch 30 Sep 23 14:09 elastic-master_http.key -> ..data/elastic-master_http.key
lrwxrwxrwx 1 root elasticsearch 25 Sep 23 14:09 elastic-master.key -> ..data/elastic-master.key
lrwxrwxrwx 1 root elasticsearch 28 Sep 23 14:09 internal-chain-ca.pem -> ..data/internal-chain-ca.pem
lrwxrwxrwx 1 root elasticsearch 15 Sep 23 14:09 kirk.crt -> ..data/kirk.crt
lrwxrwxrwx 1 root elasticsearch 15 Sep 23 14:09 kirk.key -> ..data/kirk.key
lrwxrwxrwx 1 root elasticsearch 16 Sep 23 14:09 spock.crt -> ..data/spock.crt
lrwxrwxrwx 1 root elasticsearch 16 Sep 23 14:09 spock.key -> ..data/spock.key

Are you refering the sg*.yml files?

[elasticsearch@elasticsearch-master-0 config]$ pwd
/usr/share/elasticsearch/config
[elasticsearch@elasticsearch-master-0 config]$ ls -l
total 92
drwxrwsrwt 3 root          elasticsearch   260 Sep 23 14:09 certs
-rw-rw---- 1 elasticsearch elasticsearch   229 Sep 23 14:09 elasticsearch.keystore
-rw-r--r-- 1 root          elasticsearch   770 Sep 23 14:09 elasticsearch.yml
-rw-rw-r-- 1 elasticsearch root           3110 Jul 29 20:47 jvm.options
drwxrwxr-x 2 elasticsearch root              6 Jul 29 20:51 jvm.options.d
-rw-rw-r-- 1 elasticsearch root          19089 Jul 29 20:51 log4j2.file.properties
-rw-rw-r-- 1 elasticsearch root          10446 Jul 29 20:53 log4j2.properties
-rw-rw-r-- 1 elasticsearch root            473 Jul 29 20:51 role_mapping.yml
-rw-rw-r-- 1 elasticsearch root            197 Jul 29 20:51 roles.yml
-rw-r--r-- 1 elasticsearch elasticsearch   450 Sep 23 14:17 sg_action_groups.yml
-rw-r--r-- 1 elasticsearch elasticsearch  1305 Sep 23 14:17 sg_blocks.yml
-rw-r--r-- 1 elasticsearch elasticsearch 13236 Sep 23 14:17 sg_config.yml
-rw-r--r-- 1 elasticsearch elasticsearch  1391 Sep 23 14:17 sg_internal_users.yml
-rw-r--r-- 1 elasticsearch elasticsearch   952 Sep 23 14:17 sg_roles_mapping.yml
-rw-r--r-- 1 elasticsearch elasticsearch  1457 Sep 23 14:17 sg_roles.yml
-rw-r--r-- 1 elasticsearch elasticsearch   393 Sep 23 14:17 sg_tenants.yml
drwxr-xr-x 2 elasticsearch elasticsearch   208 Sep 23 14:17 tmp
-rw-rw-r-- 1 elasticsearch root              0 Jul 29 20:51 users
-rw-rw-r-- 1 elasticsearch root              0 Jul 29 20:51 users_roles
17

Upon restarting ES, I found new warning and errors in the logs, the old ones don’t appear anymore:

{"type": "server", "timestamp": "2021-09-23T15:49:20,043Z", "level": "ERROR", "component": "c.f.s.a.s.SinkProvider", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "Default endpoint could not be created, auditlog will not work properly." }
{"type": "server", "timestamp": "2021-09-23T15:49:20,044Z", "level": "WARN", "component": "c.f.s.a.r.AuditMessageRouter", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "No default storage available, audit log may not work properly. Please check configuration." }
{"type": "server", "timestamp": "2021-09-23T15:49:20,044Z", "level": "INFO", "component": "c.f.s.a.i.AuditLogImpl", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "Message routing enabled: false" }
{"type": "server", "timestamp": "2021-09-23T15:49:20,045Z", "level": "WARN", "component": "c.f.s.c.ComplianceConfig", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "If you plan to use field masking pls configure searchguard.compliance.salt to be a random string of 16 chars length identical on all nodes" }

and

{"type": "server", "timestamp": "2021-09-23T15:49:26,653Z", "level": "ERROR", "component": "o.e.i.g.DatabaseRegistry", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "failed to download database [GeoLite2-ASN.mmdb]", 
"stacktrace": ["org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];",
"at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:179) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:165) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:605) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:494) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$3(TransportSearchAction.java:288) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:134) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:103) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:76) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:329) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:217) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:93) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:173) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:42) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:171) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:255) ~[?:?]",
"at com.floragunn.searchguard.filter.SearchGuardFilter.lambda$apply$0(SearchGuardFilter.java:129) ~[?:?]",
"at com.floragunn.searchguard.privileges.SpecialPrivilegesEvaluationContextProviderRegistry.provide(SpecialPrivilegesEvaluationContextProviderRegistry.java:50) ~[?:?]",
"at com.floragunn.searchguard.privileges.SpecialPrivilegesEvaluationContextProviderRegistry.lambda$provide$0(SpecialPrivilegesEvaluationContextProviderRegistry.java:42) ~[?:?]",
"at com.floragunn.searchguard.authtoken.AuthTokenService.provide(AuthTokenService.java:751) ~[?:?]",
"at com.floragunn.searchguard.privileges.SpecialPrivilegesEvaluationContextProviderRegistry.provide(SpecialPrivilegesEvaluationContextProviderRegistry.java:38) ~[?:?]",
"at com.floragunn.searchguard.privileges.SpecialPrivilegesEvaluationContextProviderRegistry.lambda$provide$0(SpecialPrivilegesEvaluationContextProviderRegistry.java:42) ~[?:?]",
"at com.floragunn.searchguard.internalauthtoken.InternalAuthTokenProvider.userAuthFromToken(InternalAuthTokenProvider.java:97) ~[?:?]",
"at com.floragunn.searchguard.privileges.SpecialPrivilegesEvaluationContextProviderRegistry.provide(SpecialPrivilegesEvaluationContextProviderRegistry.java:38) ~[?:?]",
"at com.floragunn.searchguard.privileges.SpecialPrivilegesEvaluationContextProviderRegistry.provide(SpecialPrivilegesEvaluationContextProviderRegistry.java:28) ~[?:?]",
"at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:126) ~[?:?]",
"at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:171) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:149) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:77) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:86) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:66) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:402) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:54) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.client.OriginSettingClient.doExecute(OriginSettingClient.java:40) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:402) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:390) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:534) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.ingest.geoip.DatabaseRegistry.lambda$retrieveDatabase$11(DatabaseRegistry.java:358) [ingest-geoip-7.14.0.jar:7.14.0]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:673) [elasticsearch-7.14.0.jar:7.14.0]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]",
"at java.lang.Thread.run(Thread.java:831) [?:?]"] }
{"type": "server", "timestamp": "2021-09-23T15:49:26,658Z", "level": "ERROR", "component": "o.e.i.g.DatabaseRegistry", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "failed to download database [GeoLite2-Country.mmdb]", 
"stacktrace": ["org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];",
"at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:179) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:165) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:605) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:494) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$3(TransportSearchAction.java:288) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:134) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:103) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:76) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:329) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:217) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:93) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:173) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:42) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:171) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:255) ~[?:?]",
"at com.floragunn.searchguard.filter.SearchGuardFilter.lambda$apply$0(SearchGuardFilter.java:129) ~[?:?]",
"at com.floragunn.searchguard.privileges.SpecialPrivilegesEvaluationContextProviderRegistry.provide(SpecialPrivilegesEvaluationContextProviderRegistry.java:50) ~[?:?]",
"at com.floragunn.searchguard.privileges.SpecialPrivilegesEvaluationContextProviderRegistry.lambda$provide$0(SpecialPrivilegesEvaluationContextProviderRegistry.java:42) ~[?:?]",
"at com.floragunn.searchguard.authtoken.AuthTokenService.provide(AuthTokenService.java:751) ~[?:?]",
"at com.floragunn.searchguard.privileges.SpecialPrivilegesEvaluationContextProviderRegistry.provide(SpecialPrivilegesEvaluationContextProviderRegistry.java:38) ~[?:?]",
"at com.floragunn.searchguard.privileges.SpecialPrivilegesEvaluationContextProviderRegistry.lambda$provide$0(SpecialPrivilegesEvaluationContextProviderRegistry.java:42) ~[?:?]",
"at com.floragunn.searchguard.internalauthtoken.InternalAuthTokenProvider.userAuthFromToken(InternalAuthTokenProvider.java:97) ~[?:?]",
"at com.floragunn.searchguard.privileges.SpecialPrivilegesEvaluationContextProviderRegistry.provide(SpecialPrivilegesEvaluationContextProviderRegistry.java:38) ~[?:?]",
"at com.floragunn.searchguard.privileges.SpecialPrivilegesEvaluationContextProviderRegistry.provide(SpecialPrivilegesEvaluationContextProviderRegistry.java:28) ~[?:?]",
"at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:126) ~[?:?]",
"at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:171) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:149) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:77) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:86) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:66) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:402) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:54) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.client.OriginSettingClient.doExecute(OriginSettingClient.java:40) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:402) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:390) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:534) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.ingest.geoip.DatabaseRegistry.lambda$retrieveDatabase$11(DatabaseRegistry.java:358) [ingest-geoip-7.14.0.jar:7.14.0]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:673) [elasticsearch-7.14.0.jar:7.14.0]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]",
"at java.lang.Thread.run(Thread.java:831) [?:?]"] }
{"type": "server", "timestamp": "2021-09-23T15:49:26,646Z", "level": "ERROR", "component": "o.e.i.g.DatabaseRegistry", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "failed to download database [GeoLite2-City.mmdb]", 
"stacktrace": ["org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];",
"at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:179) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:165) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:605) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:494) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$3(TransportSearchAction.java:288) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:134) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:103) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:76) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:329) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:217) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:93) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:173) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:42) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:171) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:255) ~[?:?]",
"at com.floragunn.searchguard.filter.SearchGuardFilter.lambda$apply$0(SearchGuardFilter.java:129) ~[?:?]",
"at com.floragunn.searchguard.privileges.SpecialPrivilegesEvaluationContextProviderRegistry.provide(SpecialPrivilegesEvaluationContextProviderRegistry.java:50) ~[?:?]",
"at com.floragunn.searchguard.privileges.SpecialPrivilegesEvaluationContextProviderRegistry.lambda$provide$0(SpecialPrivilegesEvaluationContextProviderRegistry.java:42) ~[?:?]",
"at com.floragunn.searchguard.authtoken.AuthTokenService.provide(AuthTokenService.java:751) ~[?:?]",
"at com.floragunn.searchguard.privileges.SpecialPrivilegesEvaluationContextProviderRegistry.provide(SpecialPrivilegesEvaluationContextProviderRegistry.java:38) ~[?:?]",
"at com.floragunn.searchguard.privileges.SpecialPrivilegesEvaluationContextProviderRegistry.lambda$provide$0(SpecialPrivilegesEvaluationContextProviderRegistry.java:42) ~[?:?]",
"at com.floragunn.searchguard.internalauthtoken.InternalAuthTokenProvider.userAuthFromToken(InternalAuthTokenProvider.java:97) ~[?:?]",
"at com.floragunn.searchguard.privileges.SpecialPrivilegesEvaluationContextProviderRegistry.provide(SpecialPrivilegesEvaluationContextProviderRegistry.java:38) ~[?:?]",
"at com.floragunn.searchguard.privileges.SpecialPrivilegesEvaluationContextProviderRegistry.provide(SpecialPrivilegesEvaluationContextProviderRegistry.java:28) ~[?:?]",
"at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:126) ~[?:?]",
"at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:171) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:149) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:77) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:86) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:66) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:402) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:54) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.client.OriginSettingClient.doExecute(OriginSettingClient.java:40) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:402) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:390) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:534) ~[elasticsearch-7.14.0.jar:7.14.0]",
"at org.elasticsearch.ingest.geoip.DatabaseRegistry.lambda$retrieveDatabase$11(DatabaseRegistry.java:358) [ingest-geoip-7.14.0.jar:7.14.0]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:673) [elasticsearch-7.14.0.jar:7.14.0]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]",
"at java.lang.Thread.run(Thread.java:831) [?:?]"] }
18

Regarding the live support question, There is enterprise support available which you can enquire about further at Contact the Search Guard team - get in touch with us

Are you able to check if the cluster is forming, by connecting to any of the nodes and running curl call to _cluster/health?pretty using admin certificate and key (kirk.pem and kirk.key). If you see 3 nodes, can you update internal_users.yml file with something obvious like user “test123”, upload the config using the same method you already used, then retrieve the config by adding option “-r” to the end of the same command. And confirm the user “test123” is present in the configuration.
This is to make sure the configuration is actually being uploaded.

19

I’ll do that. Let me get back to you.

20

Hi sirHusky!

We finally got ES+SG and KBN working. At this time, I am working the access to KBN.

I was wondering about the licensing. I got this output from running sgadmin on the server once and I’m a bit concerned of the infos in there:

{
  "sg_license" : {
    "uid" : "00000000-0000-0000-0000-000000000000",
    "type" : "TRIAL",
    "features" : [
      "COMPLIANCE"
    ],
    "issue_date" : "2021-09-23",
    "expiry_date" : "2021-11-22",
    "issued_to" : "The world",
    "issuer" : "floragunn GmbH",
    "start_date" : "2021-09-23",
    "major_version" : 7,
    "cluster_name" : "*",
    "msgs" : [ ],
    "expiry_in_days" : 60,
    "is_expired" : false,
    "is_valid" : true,
    "action" : "",
    "prod_usage" : "Yes, one cluster with all commercial features and unlimited nodes per cluster.",
    "license_required" : true,
    "allowed_node_count_per_cluster" : "unlimited"
  },

I would not want the license to expire after 60 days and render my ES+SG service unavailable.

Can you give me more infos on this and your licensing plan please?

I’m waiting for your response.

JB.

21

@jbalexandre1
You should check the feature matrix to determine if you are using any Enterprise/Compliance features.

If you are not, you can just disable the enterprise features using the line below in elasticsearch.yml and there will be no change after the 60 days.
searchguard.enterprise_modules_enabled: false

If you are using some enterprise features, you would need to contact Search Guard using this link with all the details regarding your request.

Hope this helps