sgconfig hangs for 60s and exits non-cleanly

  • Search Guard and Elasticsearch version

    5-5.6.6-18 / 5.6.6

  • Installed and used enterprise modules, if any

    rest, kerberos and kibana multitenancy

  • JVM version and operating system version

openjdk8 centos7.3

  • Description

When I run sgconfig, it reads all yaml files, then hangs for 1 minute, then echoes ‘null’ and reports unclean status:

$ bash /usr/share/elasticsearch/plugins/search-guard-5/tools/sgadmin.sh -ts /etc/elasticsearch/test/truststore.jks -tspass $tspass -ks /etc/elasticsearch/test/admin.jks -kspass $kspass -cd /etc/elasticsearch/test/sgconfig -cn foo -ff -h node42 --enable-sniffing --diagnose
WARNING: JAVA_HOME not set, will use /usr/bin/java
Search Guard Admin v5
Will connect to node42:9300 … done

LICENSE NOTICE Search Guard

If you use one or more of the following features in production
make sure you have a valid Search Guard license
(See https://floragunn.com/searchguard-validate-license)

  • Kibana Multitenancy
  • LDAP authentication/authorization
  • Active Directory authentication/authorization
  • REST Management API
  • JSON Web Token (JWT) authentication/authorization
  • Kerberos authentication/authorization
  • Document- and Fieldlevel Security (DLS/FLS)
  • Auditlogging

In case of any doubt mail to sales@floragunn.com

···

###################################
Failfast is activated
Diagnostic trace written to: /home/fwernli/sgadmin_diag_trace_2018-Feb-01_15-25-31.txt
Contacting elasticsearch cluster ‘foo’ and wait for YELLOW clusterstate …
Clustername: foo
Clusterstate: GREEN
Number of nodes: 6
Number of data nodes: 3
searchguard index already exists, so we do not need to create one.
Populate config from /etc/elasticsearch/test/sgconfig/
Will update ‘config’ with /etc/elasticsearch/test/sgconfig/sg_config.yml
SUCC: Configuration for ‘config’ created or updated
Will update ‘roles’ with /etc/elasticsearch/test/sgconfig/sg_roles.yml
SUCC: Configuration for ‘roles’ created or updated
Will update ‘rolesmapping’ with /etc/elasticsearch/test/sgconfig/sg_roles_mapping.yml
SUCC: Configuration for ‘rolesmapping’ created or updated
Will update ‘internalusers’ with /etc/elasticsearch/test/sgconfig/sg_internal_users.yml
SUCC: Configuration for ‘internalusers’ created or updated
Will update ‘actiongroups’ with /etc/elasticsearch/test/sgconfig/sg_action_groups.yml
SUCC: Configuration for ‘actiongroups’ created or updated
null
Done with failures

``

There is nothing obvious in the trace file (only cluster status json).

Hi,

It turns out this happened because one of the nodes was still running an older version of ES/SG (5.3.2).
Upgrading the last node solved the issue.

Maybe sgconfig could be more verbose about the root cause?

good catch, will add a warning to sgadmin if cluster has different node versions running

···

On Thursday, 1 February 2018 16:10:18 UTC+1, Fabien Wernli wrote:

Hi,

It turns out this happened because one of the nodes was still running an older version of ES/SG (5.3.2).
Upgrading the last node solved the issue.

Maybe sgconfig could be more verbose about the root cause?