Search guard initialization fails with error none of the configured nodes are available

Search Guard
1

If you think it is a bug report or you have a technical issue, please answer the following questions. For general questions, you can delete these questions.

**Elasticsearch version: 5.5.0

Server OS version: Windows 2012

Kibana version (if relevant):

**Browser version (if relevant): IE 11

Browser OS version (if relevant):

Describe the issue:
Search guard initialization fails with the below error:

sgadmin.bat --diagnose -ks "D:/Program Files/HPE/CSA/elasticsearch-5.5.0/config/CSAKS.jks" -kst JKS -kspass xxxx -ts "D:/Program Files/HPE/CSA/elasticsearch-5.5.0/config/CSATS.jks" -tst JKS -tspass xxxx -h xxxx.com -p 9300 -cd D:/Program Files/HPE/CSA/elasticsearch-5.5.0/plugins/elasticsearch/sgconfig" -arc -icl
ClusterHealthRequest:
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{qKUaaqawSvGJCg0ZSWNpsQ}{xxxxx.com}{x.x.x.x:9300}]]
	at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347)
	at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245)
	at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
	at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:363)
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408)
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:397)
	at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:724)
	at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.health(AbstractClient.java:746)
	at com.floragunn.searchguard.tools.SearchGuardAdmin.generateDiagnoseTrace(SearchGuardAdmin.java:740)
	at com.floragunn.searchguard.tools.SearchGuardAdmin.main0(SearchGuardAdmin.java:421)
	at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:109)

The elastic search yaml entries are:

cluster.name: "csacluster"
#node.name: "node1"
#node.master: true
network.host: 0.0.0.0
transport.host: 0.0.0.0
http.port: 9201
transport.tcp.port: 9300
http.enabled: true
searchguard.ssl.transport.keystore_type: JKS
searchguard.ssl.transport.keystore_password: xxxx
searchguard.ssl.transport.truststore_type: JKS
searchguard.ssl.transport.truststore_password: xxxx
searchguard.ssl.http.keystore_type: JKS
searchguard.ssl.http.keystore_password: xxxx
searchguard.ssl.http.truststore_type: JKS
searchguard.ssl.http.truststore_password: xxxx
path.data: D:\Program Files\xxxx/elasticsearch-5.5.0/data
path.logs: D:\Program Files\xxxx/elasticsearch-5.5.0/logs
discovery.zen.ping.unicast.hosts: ['xxxx.com']
searchguard.ssl.http.enabled: true
#searchguard.ssl.transport.enabled: false
searchguard.ssl.http.keystore_filepath: D:\Program Files\xxxx/elasticsearch-5.5.0/config/CSAKS.jks  
#searchguard.ssl.http.keystore_password: xxxx
searchguard.ssl.http.truststore_filepath: D:\Program Files\xxxx\elasticsearch-5.5.0\config/cacerts
searchguard.ssl.transport.keystore_filepath: D:\Program Files\xxxx/elasticsearch-5.5.0/config/CSAKS.jks
searchguard.ssl.transport.truststore_filepath: D:\Program Files\xxxx\elasticsearch-5.5.0\config/cacerts
searchguard.ssl.transport.enforce_hostname_verification: false
bootstrap.system_call_filter: false
searchguard.authcz.admin_dn:
- CN=xxxx.com, OU=xxxx, O=xxxx, L=Newbury, ST=Berkshire, C=UK

There is only one node and search guard is installed on it. We are running the command on this host. The CN provided in the searchguard.authcz.admin_dn has the same hostname.
We get the same error even when we comment out the clustername

Steps to reproduce:

  1. Run the sgadmin command as specified above.
  2. Elastic search service is running fine
  3. The sgadmin command fails with error - none of the configured nodes are available.

Expected behavior:

Provide configuration:
elasticsearch/config/elasticsearch.yml
elasticsearch/plugins/search-guard-7/sgconfig/sg_config.yml
kibana/config/kibana.yml (if relevant)

Provide logs:
Elasticsearch
Kibana (if relevant)

Screenshots (if relevant):

Errors in browser console (if relevant):

Additional data:

2

@AshwiniMS Is there any reason why you’re running such an old version of Elasticsearch?

3

Hello,

Yes, there is a reason. We had embedded this version of Elastic search 3 years back into our legacy product. We have stopped enhancing the product, it has reached end of life but we do support our existing customer for couple more years for existing features of the product. Hence we are unable to upgrade the Elastic search as we would need to enhance our integration modules. Elastic search has been working fine for most of our customers, it is this specific env that we are facing an issue.

Thanks
Ashwini

4

Here is the sg_config file -

searchguard:
  dynamic:
    kibana:
      # Kibana multitenancy - NOT FREE FOR COMMERCIAL USE
      # see https://github.com/floragunncom/search-guard-docs/blob/master/multitenancy.md
      # To make this work you need to install https://github.com/floragunncom/search-guard-module-kibana-multitenancy/wiki
      #multitenancy_enabled: true
      #server_username: kibanaserver
      #index: '.kibana'
      #do_not_fail_on_forbidden: false
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
        internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
        #internalProxies: '.*' # trust all internal proxies, regex pattern
        remoteIpHeader:  'x-forwarded-for'
        proxiesHeader:   'x-forwarded-by'
        #trustedProxies: '.*' # trust all external proxies, regex pattern
        ###### see https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html for regex help
        ###### more information about XFF https://en.wikipedia.org/wiki/X-Forwarded-For
        ###### and here https://tools.ietf.org/html/rfc7239
        ###### and https://tomcat.apache.org/tomcat-8.0-doc/config/valve.html#Remote_IP_Valve
    authc:
      kerberos_auth_domain: 
        enabled: false
        order: 6
        http_authenticator:
          type: kerberos # NOT FREE FOR COMMERCIAL USE
          challenge: true
          config:
            # If true a lot of kerberos/security related debugging output will be logged to standard out
            krb_debug: false
            # If true then the realm will be stripped from the user name
            strip_realm_from_principal: true
        authentication_backend:
          type: noop
      basic_internal_auth_domain: 
        enabled: true
        order: 4
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: intern
      proxy_auth_domain:
        enabled: false
        order: 3
        http_authenticator:
          type: proxy
          challenge: false
          config:
            user_header: "x-proxy-user"
            roles_header: "x-proxy-roles"
        authentication_backend:
          type: noop
      host_auth_domain:
        enabled: false
        order: 1
        http_authenticator:
          type: host #DEPRECATED, will be removed in a future version
          challenge: false
        authentication_backend:
          type: noop
      jwt_auth_domain:
        enabled: false
        order: 0
        http_authenticator:
          type: jwt
          challenge: false
          config:
            signing_key: "base64 encoded key"
            jwt_header: "Authorization"
            jwt_url_parameter: null
            roles_key: null
            subject_key: null
        authentication_backend:
          type: noop
      clientcert_auth_domain:
        enabled: false
        order: 2
        http_authenticator:
          type: clientcert
          config:
            username_attribute: cn #optional, if omitted DN becomes username
          challenge: false
        authentication_backend:
          type: noop
      ldap:
        enabled: false
        order: 5
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          # LDAP authentication backend (authenticate users against a LDAP or Active Directory)
          type: ldap # NOT FREE FOR COMMERCIAL USE
          config:
            # enable ldaps
            enable_ssl: false
            # enable start tls, enable_ssl should be false
            enable_start_tls: false
            # send client certificate
            enable_ssl_client_auth: false
            # verify ldap hostname
            verify_hostnames: true
            hosts:
              - localhost:8389
            bind_dn: null
            password: null
            userbase: 'ou=people,dc=example,dc=com'
            # Filter to search for users (currently in the whole subtree beneath userbase)
            # {0} is substituted with the username 
            usersearch: '(sAMAccountName={0})'
            # Use this attribute from the user as username (if not set then DN is used)
            username_attribute: null
    authz:    
      roles_from_myldap:
        enabled: false
        authorization_backend:
          # LDAP authorization backend (gather roles from a LDAP or Active Directory, you have to configure the above LDAP authentication backend settings too)
          type: ldap # NOT FREE FOR COMMERCIAL USE
          config:
            # enable ldaps
            enable_ssl: false
            # enable start tls, enable_ssl should be false
            enable_start_tls: false
            # send client certificate
            enable_ssl_client_auth: false
            # verify ldap hostname
            verify_hostnames: true
            hosts:
              - localhost:8389
            bind_dn: null
            password: null
            rolebase: 'ou=groups,dc=example,dc=com'
            # Filter to search for roles (currently in the whole subtree beneath rolebase)
            # {0} is substituted with the DN of the user
            # {1} is substituted with the username 
            # {2} is substituted with an attribute value from user's directory entry, of the authenticated user. Use userroleattribute to specify the name of the attribute            
            rolesearch: '(member={0})'
            # Specify the name of the attribute which value should be substituted with {2} above
            userroleattribute: null
            # Roles as an attribute of the user entry
            userrolename: disabled
            #userrolename: memberOf
            # The attribute in a role entry containing the name of that role, Default is "name".
            # Can also be "dn" to use the full DN as rolename.
            rolename: cn
            # Resolve nested roles transitive (roles which are members of other roles and so on ...)
            resolve_nested_roles: true
            userbase: 'ou=people,dc=example,dc=com'
            # Filter to search for users (currently in the whole subtree beneath userbase)
            # {0} is substituted with the username 
            usersearch: '(uid={0})'
            # Skip users matching a user name, a wildcard or a regex pattern
            #skip_users: 
            #  - 'cn=Michael Jackson,ou*people,o=TEST'
            #  - '/\S*/'    
      roles_from_another_ldap:
        enabled: false
        authorization_backend:
          type: ldap # NOT FREE FOR COMMERCIAL USE
          #config goes here ...
5

@AshwiniMS The config.yml looks fine.
Try the troubleshooting described in the link below.

https://docs.search-guard.com/v5/troubleshooting-sgadmin

6

Thanks Pablo for your response.

We have verified all the options that are provided in the troubleshooting link. We have tried using : -h, -icl, -cn, -p, -nhnv, -arc . We still get the same error.

Thanks
Ashwini

7

@AshwiniMS Could you confirm that passwords for truststore and keystore configured in elasticsearch.yml are correct?

Do you see any warnings/errors in Elasticsearch logs?

8

Sorry, I was on leave for couple of days hence the delay in response.
Yes, the passwords are correct.

9

Is there anything else that we should check?
We have verified the troubleshooting steps given in sgadmin, we still can’t get this running.

10

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.