-
ES 6.2.2 / SG 6.2.2-22.1
-
Default modules from fresh install / trial license
-
JVM 1.8.0_161 on Windows 7 Pro SP1
-
Demo certs and config files
-
No other plugins
There are various other posts here with a similar sgadmin “None of the configured nodes are available” failure, but all seem to be related to custom certificates and cert config, or conflicts between node cert and admin cert. This case is on a development cluster with a fresh Search Guard install with demo certs and the standard elasticsearch.yml config lines added by install_demo_configuration.sh.
This is a 2-node cluster which has been happily running with SG 6.2.2-21.0 until we upgraded a few days ago. This condition persists through complete wipes and reinstalls of the cluster, and shows up also if I only run a single node. I’ve not yet tried a regression test against 6.2.2-21.0.
tools\sgadmin.bat -icl -key …..\config\kirk-key.pem -cert …..\config\kirk.pem -cacert …..\config\root-ca.pem -nhnv -cd d:\project.git\sgconfig
Search Guard Admin v6
Will connect to localhost:9300 … done
Unable to check whether cluster is sane: None of the configured nodes are available: [{#transport#-1}{2oijFSfIQRWnny_P5qVk2A}{localhost}{127.0.0.1:9300}]
ERR: Cannot connect to Elasticsearch. Please refer to elasticsearch logfile for more information
Trace:
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{2oijFSfIQRWnny_P5qVk2A}{localhost}{127.0.0.1:9300}]]
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245)
at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:60)
at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:371)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:405)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:394)
at com.floragunn.searchguard.tools.SearchGuardAdmin.main0(SearchGuardAdmin.java:444)
at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:123)
I’ve also tried with -cn hostname instead of -icl.
Elasticsearch DEBUG log messages are too voluminous to reproduce here, but here is everything in the startup that seems relevant to certificate management; nothing at all shows at the time of running sgadmin.
[2018-05-31T17:06:01,212][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Config directory is D:\elastic\6.2.2\elasticsearch-6.2.2\config/, from there the key- and truststore files are resolved relatively
[2018-05-31T17:06:01,212][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.transport.pemcert_filepath is esnode.pem
[2018-05-31T17:06:01,212][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved esnode.pem to D:\elastic\6.2.2\elasticsearch-6.2.2\config\esnode.pem against D:\elastic\6.2.2\elasticsearch-6.2.2\config
[2018-05-31T17:06:01,213][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.transport.pemkey_filepath is esnode-key.pem
[2018-05-31T17:06:01,213][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved esnode-key.pem to D:\elastic\6.2.2\elasticsearch-6.2.2\config\esnode-key.pem against D:\elastic\6.2.2\elasticsearch-6.2.2\config
[2018-05-31T17:06:01,213][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.transport.pemtrustedcas_filepath is root-ca.pem
[2018-05-31T17:06:01,213][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved root-ca.pem to D:\elastic\6.2.2\elasticsearch-6.2.2\config\root-ca.pem against D:\elastic\6.2.2\elasticsearch-6.2.2\config
[2018-05-31T17:06:01,271][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.http.pemtrustedcas_filepath is root-ca.pem
[2018-05-31T17:06:01,272][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved root-ca.pem to D:\elastic\6.2.2\elasticsearch-6.2.2\config\root-ca.pem against D:\elastic\6.2.2\elasticsearch-6.2.2\config
[2018-05-31T17:06:01,272][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.http.pemcert_filepath is esnode.pem
[2018-05-31T17:06:01,272][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved esnode.pem to D:\elastic\6.2.2\elasticsearch-6.2.2\config\esnode.pem against D:\elastic\6.2.2\elasticsearch-6.2.2\config
[2018-05-31T17:06:01,274][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.http.pemkey_filepath is esnode-key.pem
[2018-05-31T17:06:01,274][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved esnode-key.pem to D:\elastic\6.2.2\elasticsearch-6.2.2\config\esnode-key.pem against D:\elastic\6.2.2\elasticsearch-6.2.2\config
[2018-05-31T17:06:01,276][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Client Provider : JDK
[2018-05-31T17:06:01,278][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Server Provider : JDK
[2018-05-31T17:06:01,278][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS HTTP Provider : JDK
[2018-05-31T17:06:01,278][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] sslTransportClientProvider:JDK with ciphers […lots…]
[2018-05-31T17:06:01,279][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] sslTransportServerProvider:JDK with ciphers […lots…]
[2018-05-31T17:06:01,279][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] sslHTTPProvider:JDK with ciphers […lots…]
[2018-05-31T17:06:01,280][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for transport layer : [TLSv1.2, TLSv1.1]
[2018-05-31T17:06:01,280][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for HTTP layer : [TLSv1.2, TLSv1.1]
…
[2018-05-31T17:06:01,730][DEBUG][o.e.x.c.s.SSLService ] [devnode-1] using ssl settings [SSLConfiguration{keyConfig=[NONE], trustConfig=JDK trusted certs], cipherSuites=[[…lots…]], supportedProtocols=[[TLSv1.2, TLSv1.1, TLSv1]], sslClientAuth=[REQUIRED], verificationMode=[FULL]}]
…
[2018-05-31T17:06:04,344][DEBUG][c.f.s.c.AdminDNs ] CN=kirk,OU=client,O=client,L=test, C=de is registered as an admin dn
[2018-05-31T17:06:04,345][DEBUG][c.f.s.c.AdminDNs ] Loaded 1 admin DN’s [CN=kirk,OU=client,O=client,L=test, C=de]
[2018-05-31T17:06:04,346][DEBUG][c.f.s.c.AdminDNs ] Loaded 0 impersonation DN’s {}
[2018-05-31T17:06:04,346][DEBUG][c.f.s.c.AdminDNs ] Loaded 0 impersonation users for REST {}
I’ve also tried running the standalone sgadmin tool from another host (with the certs copied over) which gives the same result; Wireshark shows a single TCP payload from sgadmin to server and an immediate connection close from the server.
