Short description: unstable error during sgconfig work:
FAIL: 1 nodes reported failures. First failure is FailedNodeException[Failed node [hZm8tA43RUa5TcjRtleM9g]]; nested: RemoteTransportException[[project-logging-elasticsearch-client-676f8dbd5c-bkmmx][100.96.1.28:9300][cluster:admin/searchguard/config/update[n]]]; nested: ElasticsearchException[java.lang.NullPointerException: Cannot read field "dynamic" because "this.config" is null]; nested: NullPointerException[Cannot read field "dynamic" because "this.config" is null];
Elasticsearch version:
7.10.2
Server OS version:
ubuntu-18.04
Kibana version (if relevant):
7.10.2
Describe the issue:
We use bash script to initiate SearchGuard, do several applying like
function applyConfig(){
... check if http://elk:9200/searchguard/_doc/${1} return 404 then apply config...
/sgadmin/tools/sgadmin.sh -f /tmp/templates/${2} -t ${1} -icl -nhnv \
-cacert /certificates/root-ca.pem \
-cert /certificates/sgadmin.pem \
-key /certificates/sgadmin.key \
-h $ELASTICSEARCH_DISCOVERY_SERVICE && break
for each of type of config file:
applyConfig "config" "sg_config.yml"
applyConfig "roles" "sg_roles.yml"
applyConfig "rolesmapping" "sg_roles_mapping.yml"
applyConfig "internalusers" "sg_internal_users.yml"
applyConfig "actiongroups" "sg_action_groups.yml"
applyConfig "tenants" "sg_tenants.yml"
applyConfig "blocks" "sg_blocks.yml"
Sometime (we got it two times per 10 test clusters) this fails and second attempt to run init bash script give us the following logs:
Elastic is ready
-------------
Apply config
-------------
Template config exists in searchguard index, skip upload
-------------
Apply roles
-------------
Template roles exists in searchguard index, skip upload
-------------
Apply rolesmapping
-------------
Apply initial template rolesmapping with sgadmin.sh
Search Guard Admin v7
Will connect to project-logging-elasticsearch-discovery.project:9300 ... done
Connected as CN=sgadmin,O=project
Elasticsearch Version: 7.10.2
Search Guard Version: 7.10.2-52.3.0
Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
Clustername: elasticsearch
Clusterstate: YELLOW
Number of nodes: 3
Number of data nodes: 1
searchguard index already exists, so we do not need to create one.
Populate config from /tmp
Force type: rolesmapping
Will update '_doc/rolesmapping' with /tmp/templates/sg_roles_mapping.yml
SUCC: Configuration for 'rolesmapping' created or updated
RemoteTransportException[[project-logging-elasticsearch-client-676f8dbd5c-bkmmx][100.96.1.28:9300][cluster:admin/searchguard/config/update[n]]]; nested: ElasticsearchException[java.lang.NullPointerException: Cannot read field "dynamic" because "this.config" is null]; nested: NullPointerException[Cannot read field "dynamic" because "this.config" is null];
FAIL: Expected 3 nodes to return response, but got 2
Done with failures
Retry sgadmin.sh in 15 seconds.......
Then last error repeated until job timeout
Provide configuration:
elasticsearch/config/elasticsearch.yml
network.host: 0.0.0.0
cluster.name: elasticsearch
cluster.initial_master_nodes: -elasticsearch-master-0
discovery.seed_hosts: elasticsearch-discovery.project
elasticsearch-discovery.project
node.name: ${HOSTNAME}
node.processors: 2
searchguard.enterprise_modules_enabled: false
searchguard.ssl.http.clientauth_mode: OPTIONAL
searchguard.ssl.transport.pemcert_filepath: certificates/node.pem
searchguard.ssl.transport.pemkey_filepath: certificates/node.key
searchguard.ssl.transport.pemtrustedcas_filepath: certificates/root-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: certificates/node_http.pem
searchguard.ssl.http.pemkey_filepath: certificates/node_http.key
searchguard.ssl.http.pemtrustedcas_filepath: certificates/root-ca.pem
searchguard.nodes_dn:
- CN=node,O=project
searchguard.authcz.admin_dn:
- CN=sgadmin,O=project
action.auto_create_index: true
xpack.ml.enabled: ${XPACK_ML_ENABLED:false}
xpack.monitoring.enabled: ${XPACK_MONITORING_ENABLED:true}
xpack.security.enabled: ${XPACK_SECURITY_ENABLED:false}
xpack.watcher.enabled: ${XPACK_WATCHER_ENABLED:false}
elasticsearch/plugins/search-guard-7/sgconfig/sg_config.yml
_sg_meta:
type: "config"
config_version: 2
sg_config:
dynamic:
filtered_alias_mode: "warn"
disable_rest_auth: false
disable_intertransport_auth: false
respect_request_indices_options: false
license: null
auth_failure_listeners: {}
do_not_fail_on_forbidden: true
multi_rolespan_enabled: false
hosts_resolver_mode: "ip-only"
transport_userrname_attribute: null
do_not_fail_on_forbidden_empty: true
field_anonymization_salt2: null
kibana:
multitenancy_enabled: false
server_username: "system.kibanaserver"
index: ".kibana"
rbac_enabled: false
http:
anonymous_auth_enabled: false
xff:
enabled: true
internalProxies: ".*"
remoteIpHeader: "x-forwarded-for"
authc:
proxy_auth_domain:
http_enabled: true
transport_enabled: true
order: 1
http_authenticator:
challenge: false
type: "proxy"
config:
user_header: "x-proxy-user"
roles_header: "x-proxy-roles"
authentication_backend:
type: "noop"
config: {}
skip_users: []
clientcert_auth_domain:
http_enabled: true
transport_enabled: false
order: 0
http_authenticator:
challenge: false
type: "clientcert"
config:
username_attribute: "cn"
authentication_backend:
type: "noop"
config: {}
skip_users: []
basic_internal_auth_domain:
http_enabled: true
transport_enabled: true
order: 2
http_authenticator:
challenge: true
type: "basic"
config: {}
authentication_backend:
type: "intern"
config: {}
skip_users: []
authz:
roles_from_another_ldap:
http_enabled: false
transport_enabled: false
authorization_backend:
type: "ldap"
config: {}
skipped_users: []
roles_from_myldap:
http_enabled: false
transport_enabled: false
authorization_backend:
type: "ldap"
config:
enable_ssl: false
enable_start_tls: false
enable_ssl_client_auth: false
verify_hostnames: true
hosts:
- "localhost:8389"
bind_dn: null
password: null
rolebase: "ou=groups,dc=example,dc=com"
rolesearch: "(member={0})"
userroleattribute: null
userrolename: "disabled"
rolename: "cn"
resolve_nested_roles: true
userbase: "ou=people,dc=example,dc=com"
usersearch: "(uid={0})"
skipped_users: []
kibana/config/kibana.yml (if relevant)
Provide logs:
Elasticsearch
hard to repeat this case, we don’t have elasticsearch logs at this moment, only sgadmin logs