# sgadmin "None of the configured nodes are available" with 6.2.2-22.1 **URL:** \ **Category:** Search Guard\ **Created:** [May 31, 2018, 4:47pm UTC](https://forum.search-guard.com/t/sgadmin-none-of-the-configured-nodes-are-available-with-6-2-2-22-1/932 "2018-05-31T16:47:52Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![James\_Beckett](https://avatars.discourse-cdn.com/v4/letter/j/c67d28/32.png) [@James\_Beckett](https://forum.search-guard.com/u/James_Beckett)\ **Post date:** [May 31, 2018, 4:47pm UTC](https://forum.search-guard.com/t/sgadmin-none-of-the-configured-nodes-are-available-with-6-2-2-22-1/932/1 "2018-05-31T16:47:52Z")
- ES 6.2.2 / SG 6.2.2-22.1 - Default modules from fresh install / trial license - JVM 1.8.0\_161 on Windows 7 Pro SP1 - Demo certs and config files - No other plugins There are various other posts here with a similar sgadmin “None of the configured nodes are available” failure, but all seem to be related to custom certificates and cert config, or conflicts between node cert and admin cert. This case is on a development cluster with a fresh Search Guard install with demo certs and the standard _elasticsearch.yml_ config lines added by _install\_demo\_configuration.sh_. This is a 2-node cluster which has been happily running with SG 6.2.2-21.0 until we upgraded a few days ago. This condition persists through complete wipes and reinstalls of the cluster, and shows up also if I only run a single node. I’ve not yet tried a regression test against 6.2.2-21.0. > tools\sgadmin.bat -icl -key …..\config\kirk-key.pem -cert …..\config\kirk.pem -cacert …..\config\root-ca.pem -nhnv -cd d:\project.git\sgconfig Search Guard Admin v6 Will connect to localhost:9300 … done Unable to check whether cluster is sane: None of the configured nodes are available: [{#transport#-1}{2oijFSfIQRWnny\_P5qVk2A}{localhost}{127.0.0.1:9300}] ERR: Cannot connect to Elasticsearch. Please refer to elasticsearch logfile for more information Trace: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{2oijFSfIQRWnny\_P5qVk2A}{localhost}{127.0.0.1:9300}]] ``` at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347) at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245) at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:60) at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:371) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:405) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:394) at com.floragunn.searchguard.tools.SearchGuardAdmin.main0(SearchGuardAdmin.java:444) at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:123) ``` I’ve also tried with _-cn hostname_ instead of _-icl._ Elasticsearch DEBUG log messages are too voluminous to reproduce here, but here is everything in the startup that seems relevant to certificate management; nothing at all shows at the time of running _sgadmin._ [2018-05-31T17:06:01,212][INFO][c.f.s.s.DefaultSearchGuardKeyStore] Config directory is D:\elastic\6.2.2\elasticsearch-6.2.2\config/, from there the key- and truststore files are resolved relatively [2018-05-31T17:06:01,212][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.transport.pemcert\_filepath is esnode.pem [2018-05-31T17:06:01,212][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved esnode.pem to D:\elastic\6.2.2\elasticsearch-6.2.2\config\esnode.pem against D:\elastic\6.2.2\elasticsearch-6.2.2\config [2018-05-31T17:06:01,213][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.transport.pemkey\_filepath is esnode-key.pem [2018-05-31T17:06:01,213][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved esnode-key.pem to D:\elastic\6.2.2\elasticsearch-6.2.2\config\esnode-key.pem against D:\elastic\6.2.2\elasticsearch-6.2.2\config [2018-05-31T17:06:01,213][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.transport.pemtrustedcas\_filepath is root-ca.pem [2018-05-31T17:06:01,213][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved root-ca.pem to D:\elastic\6.2.2\elasticsearch-6.2.2\config\root-ca.pem against D:\elastic\6.2.2\elasticsearch-6.2.2\config [2018-05-31T17:06:01,271][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.http.pemtrustedcas\_filepath is root-ca.pem [2018-05-31T17:06:01,272][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved root-ca.pem to D:\elastic\6.2.2\elasticsearch-6.2.2\config\root-ca.pem against D:\elastic\6.2.2\elasticsearch-6.2.2\config [2018-05-31T17:06:01,272][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.http.pemcert\_filepath is esnode.pem [2018-05-31T17:06:01,272][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved esnode.pem to D:\elastic\6.2.2\elasticsearch-6.2.2\config\esnode.pem against D:\elastic\6.2.2\elasticsearch-6.2.2\config [2018-05-31T17:06:01,274][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.http.pemkey\_filepath is esnode-key.pem [2018-05-31T17:06:01,274][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved esnode-key.pem to D:\elastic\6.2.2\elasticsearch-6.2.2\config\esnode-key.pem against D:\elastic\6.2.2\elasticsearch-6.2.2\config [2018-05-31T17:06:01,276][INFO][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Client Provider : JDK [2018-05-31T17:06:01,278][INFO][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Server Provider : JDK [2018-05-31T17:06:01,278][INFO][c.f.s.s.DefaultSearchGuardKeyStore] TLS HTTP Provider : JDK [2018-05-31T17:06:01,278][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] sslTransportClientProvider:JDK with ciphers […lots…] [2018-05-31T17:06:01,279][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] sslTransportServerProvider:JDK with ciphers […lots…] [2018-05-31T17:06:01,279][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] sslHTTPProvider:JDK with ciphers […lots…] [2018-05-31T17:06:01,280][INFO][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for transport layer : [TLSv1.2, TLSv1.1] [2018-05-31T17:06:01,280][INFO][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for HTTP layer : [TLSv1.2, TLSv1.1] … [2018-05-31T17:06:01,730][DEBUG][o.e.x.c.s.SSLService] [devnode-1] using ssl settings [SSLConfiguration{keyConfig=[NONE], trustConfig=JDK trusted certs], cipherSuites=[[…lots…]], supportedProtocols=[[TLSv1.2, TLSv1.1, TLSv1]], sslClientAuth=[REQUIRED], verificationMode=[FULL]}] … [2018-05-31T17:06:04,344][DEBUG][c.f.s.c.AdminDNs] CN=kirk,OU=client,O=client,L=test, C=de is registered as an admin dn [2018-05-31T17:06:04,345][DEBUG][c.f.s.c.AdminDNs] Loaded 1 admin DN’s [CN=kirk,OU=client,O=client,L=test, C=de] [2018-05-31T17:06:04,346][DEBUG][c.f.s.c.AdminDNs] Loaded 0 impersonation DN’s {} [2018-05-31T17:06:04,346][DEBUG][c.f.s.c.AdminDNs] Loaded 0 impersonation users for REST {} I’ve also tried running the standalone sgadmin tool from another host (with the certs copied over) which gives the same result; Wireshark shows a single TCP payload from sgadmin to server and an immediate connection close from the server. ---
**Author:** ![searchguard\_google\_group](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/searchguard_google_group/32/462_2.png) [@searchguard\_google\_group](https://forum.search-guard.com/u/searchguard_google_group)\ **Post date:** [May 31, 2018, 4:59pm UTC](https://forum.search-guard.com/t/sgadmin-none-of-the-configured-nodes-are-available-with-6-2-2-22-1/932/2 "2018-05-31T16:59:42Z")
Our demo certificates changed recently, maybe thats the reason: [https://groups.google.com/d/msgid/search-guard/cbc49abf-f378-4358-96c1-584513c6481d%40googlegroups.com](https://groups.google.com/d/msgid/search-guard/cbc49abf-f378-4358-96c1-584513c6481d%40googlegroups.com) Make sure sgadmin use the same (new) certificates! > **···** > > > Am 31.05.2018 um 18:47 schrieb James Beckett \: > > > > \* ES 6.2.2 / SG 6.2.2-22.1 > > \* Default modules from fresh install / trial license > > \* JVM 1.8.0\_161 on Windows 7 Pro SP1 > > \* Demo certs and config files > > \* No other plugins > > > > There are various other posts here with a similar sgadmin "None of the configured nodes are available" failure, but all seem to be related to custom certificates and cert config, or conflicts between node cert and admin cert. This case is on a development cluster with a fresh Search Guard install with demo certs and the standard elasticsearch.yml config lines added by install\_demo\_configuration.sh. > > > > This is a 2-node cluster which has been happily running with SG 6.2.2-21.0 until we upgraded a few days ago. This condition persists through complete wipes and reinstalls of the cluster, and shows up also if I only run a single node. I've not yet tried a regression test against 6.2.2-21.0. > > > > \> tools\sgadmin.bat -icl -key ..\..\config\kirk-key.pem -cert ..\..\config\kirk.pem -cacert ..\..\config\root-ca.pem -nhnv -cd d:\project.git\sgconfig > > Search Guard Admin v6 > > Will connect to localhost:9300 ... done > > Unable to check whether cluster is sane: None of the configured nodes are available: [{#transport#-1}{2oijFSfIQRWnny\_P5qVk2A}{localhost}{127.0.0.1:9300}] > > ERR: Cannot connect to Elasticsearch. Please refer to elasticsearch logfile for more information > > Trace: > > NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{2oijFSfIQRWnny\_P5qVk2A}{localhost}{127.0.0.1:9300}]] > >         at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347) > >         at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245) > >         at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:60) > >         at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:371) > >         at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:405) > >         at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:394) > >         at com.floragunn.searchguard.tools.SearchGuardAdmin.main0(SearchGuardAdmin.java:444) > >         at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:123) > > > > I've also tried with -cn hostname instead of -icl. > > > > Elasticsearch DEBUG log messages are too voluminous to reproduce here, but here is everything in the startup that seems relevant to certificate management; nothing at all shows at the time of running sgadmin. > > > > [2018-05-31T17:06:01,212][INFO][c.f.s.s.DefaultSearchGuardKeyStore] Config directory is D:\elastic\6.2.2\elasticsearch-6.2.2\config/, from there the key- and truststore files are resolved relatively > > [2018-05-31T17:06:01,212][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.transport.pemcert\_filepath is esnode.pem > > [2018-05-31T17:06:01,212][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved esnode.pem to D:\elastic\6.2.2\elasticsearch-6.2.2\config\esnode.pem against D:\elastic\6.2.2\elasticsearch-6.2.2\config > > [2018-05-31T17:06:01,213][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.transport.pemkey\_filepath is esnode-key.pem > > [2018-05-31T17:06:01,213][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved esnode-key.pem to D:\elastic\6.2.2\elasticsearch-6.2.2\config\esnode-key.pem against D:\elastic\6.2.2\elasticsearch-6.2.2\config > > [2018-05-31T17:06:01,213][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.transport.pemtrustedcas\_filepath is root-ca.pem > > [2018-05-31T17:06:01,213][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved root-ca.pem to D:\elastic\6.2.2\elasticsearch-6.2.2\config\root-ca.pem against D:\elastic\6.2.2\elasticsearch-6.2.2\config > > [2018-05-31T17:06:01,271][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.http.pemtrustedcas\_filepath is root-ca.pem > > [2018-05-31T17:06:01,272][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved root-ca.pem to D:\elastic\6.2.2\elasticsearch-6.2.2\config\root-ca.pem against D:\elastic\6.2.2\elasticsearch-6.2.2\config > > [2018-05-31T17:06:01,272][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.http.pemcert\_filepath is esnode.pem > > [2018-05-31T17:06:01,272][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved esnode.pem to D:\elastic\6.2.2\elasticsearch-6.2.2\config\esnode.pem against D:\elastic\6.2.2\elasticsearch-6.2.2\config > > [2018-05-31T17:06:01,274][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.http.pemkey\_filepath is esnode-key.pem > > [2018-05-31T17:06:01,274][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved esnode-key.pem to D:\elastic\6.2.2\elasticsearch-6.2.2\config\esnode-key.pem against D:\elastic\6.2.2\elasticsearch-6.2.2\config > > [2018-05-31T17:06:01,276][INFO][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Client Provider : JDK > > [2018-05-31T17:06:01,278][INFO][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Server Provider : JDK > > [2018-05-31T17:06:01,278][INFO][c.f.s.s.DefaultSearchGuardKeyStore] TLS HTTP Provider : JDK > > [2018-05-31T17:06:01,278][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] sslTransportClientProvider:JDK with ciphers [...lots...] > > [2018-05-31T17:06:01,279][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] sslTransportServerProvider:JDK with ciphers [...lots...] > > [2018-05-31T17:06:01,279][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] sslHTTPProvider:JDK with ciphers [...lots...] > > [2018-05-31T17:06:01,280][INFO][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for transport layer : [TLSv1.2, TLSv1.1] > > [2018-05-31T17:06:01,280][INFO][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for HTTP layer : [TLSv1.2, TLSv1.1] > > ... > > [2018-05-31T17:06:01,730][DEBUG][o.e.x.c.s.SSLService] [devnode-1] using ssl settings [SSLConfiguration{keyConfig=[NONE], trustConfig=JDK trusted certs], cipherSuites=[[...lots...]], supportedProtocols=[[TLSv1.2, TLSv1.1, TLSv1]], sslClientAuth=[REQUIRED], verificationMode=[FULL]}] > > ... > > [2018-05-31T17:06:04,344][DEBUG][c.f.s.c.AdminDNs] CN=kirk,OU=client,O=client,L=test, C=de is registered as an admin dn > > [2018-05-31T17:06:04,345][DEBUG][c.f.s.c.AdminDNs] Loaded 1 admin DN's [CN=kirk,OU=client,O=client,L=test, C=de] > > [2018-05-31T17:06:04,346][DEBUG][c.f.s.c.AdminDNs] Loaded 0 impersonation DN's {} > > [2018-05-31T17:06:04,346][DEBUG][c.f.s.c.AdminDNs] Loaded 0 impersonation users for REST {} > > > > I've also tried running the standalone sgadmin tool from another host (with the certs copied over) which gives the same result; Wireshark shows a single TCP payload from sgadmin to server and an immediate connection close from the server. > > > > -- > > You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com. > > To post to this group, send email to search-guard@googlegroups.com. > > To view this discussion on the web visit [https://groups.google.com/d/msgid/search-guard/fd6d89bf-6662-466e-a5c2-138484559002%40googlegroups.com\](https://groups.google.com/d/msgid/search-guard/fd6d89bf-6662-466e-a5c2-138484559002%40googlegroups.com%5C). > > For more options, visit [https://groups.google.com/d/optout\](https://groups.google.com/d/optout%5C). ---
**Author:** ![James\_Beckett](https://avatars.discourse-cdn.com/v4/letter/j/c67d28/32.png) [@James\_Beckett](https://forum.search-guard.com/u/James_Beckett)\ **Post date:** [June 4, 2018, 10:45am UTC](https://forum.search-guard.com/t/sgadmin-none-of-the-configured-nodes-are-available-with-6-2-2-22-1/932/3 "2018-06-04T10:45:46Z")
I know - that’s the only reason we moved to 22.1. sgadmin is using the same certs, as it’s running from the same install location: “sgadmin … -cert …..\config\kirk.pem …”. > **···** > > On Thursday, 31 May 2018 17:59:44 UTC+1, Search Guard wrote: > > > Our demo certificates changed recently, maybe thats the reason: [https://groups.google.com/d/msgid/search-guard/cbc49abf-f378-4358-96c1-584513c6481d%40googlegroups.com](https://groups.google.com/d/msgid/search-guard/cbc49abf-f378-4358-96c1-584513c6481d%40googlegroups.com) > > > > Make sure sgadmin use the same (new) certificates! ---
**Author:** ![jkressin](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/jkressin/32/9_2.png) [@jkressin](https://forum.search-guard.com/u/jkressin)\ **Post date:** [June 4, 2018, 1:36pm UTC](https://forum.search-guard.com/t/sgadmin-none-of-the-configured-nodes-are-available-with-6-2-2-22-1/932/4 "2018-06-04T13:36:05Z")
So just to make sure I understand correctly: You are installing SG 6.2.2-22.1 on a fresh ES 6.2.2 instance, and then the sgadmin call already fails? You are saying you are running on Windows 7, but also write you have executed the (.sh) demo installer script. Could you please explain the steps you performed to install and configure SG so we are able to reproduce your scenario. Thanks! > **···** > > On Monday, June 4, 2018 at 12:45:46 PM UTC+2, James Beckett wrote: > > > On Thursday, 31 May 2018 17:59:44 UTC+1, Search Guard wrote: > > > > > Our demo certificates changed recently, maybe thats the reason: [https://groups.google.com/d/msgid/search-guard/cbc49abf-f378-4358-96c1-584513c6481d%40googlegroups.com](https://groups.google.com/d/msgid/search-guard/cbc49abf-f378-4358-96c1-584513c6481d%40googlegroups.com) > > > > > > Make sure sgadmin use the same (new) certificates! > > > I know - that’s the only reason we moved to 22.1. > > > sgadmin is using the same certs, as it’s running from the same install location: “sgadmin … -cert …..\config\kirk.pem …”. > > > ---
**Author:** ![James\_Beckett](https://avatars.discourse-cdn.com/v4/letter/j/c67d28/32.png) [@James\_Beckett](https://forum.search-guard.com/u/James_Beckett)\ **Post date:** [June 5, 2018, 4:47pm UTC](https://forum.search-guard.com/t/sgadmin-none-of-the-configured-nodes-are-available-with-6-2-2-22-1/932/5 "2018-06-05T16:47:49Z")
Demo installer script (.sh) seems to run fine in a git bash shell. Writes out the demo certs, edits the elasticsearch.yml with sound-looking SG config, writes an sgadmin script. I'm out of office right now so can't paste my setup steps, but there's not much more to say: from memory, install 6.2.2 from zip, set cluster/node name, discovery etc, install x-pack (for monitoring) and disable x-pack security, start and verify, install SG, run demo installer with auto init disabled, (re)start ES, run sgadmin. I'm going to repeat afresh on a couple of other nodes (Win and Linux) and compare the resulting configs - will update here after. ---
**Author:** ![James\_Beckett](https://avatars.discourse-cdn.com/v4/letter/j/c67d28/32.png) [@James\_Beckett](https://forum.search-guard.com/u/James_Beckett)\ **Post date:** [June 7, 2018, 4:52pm UTC](https://forum.search-guard.com/t/sgadmin-none-of-the-configured-nodes-are-available-with-6-2-2-22-1/932/6 "2018-06-07T16:52:36Z")
SOLVED: third-party interaction ☹ > I’m going to repeat afresh on a couple of other nodes (Win and Linux) and compare the resulting configs - will update here after. After successes on other nodes, I tried a parallel install on the same node that had a problem (using ports 9205 and 9305 to keep well clear of the usual). Clean install worked fine, sgadmin perfectly happy. Recursive diff of each install directory showed nothing corrupted in install - only differences were a few elasticsearch.yml lines around zen discovery and the different paths I’d deliberately set for clustername/data/logs etc. I’ve no hair left at this point. I tried getting a bare TLS connection a few times - even that wasn’t happy: > ## openssl s\_client -msg -CAfile config.devhost/root-ca.pem -cert config.devhost/kirk.pem -key config.devhost/kirk-key.pem -connect devhost:9300 CONNECTED(00000003) 140537057156824:error:140790E5:SSL routines:ssl23\_write:ssl handshake failure:s23\_lib.c:177: > > ## no peer certificate available > > ## No client certificate CA names sent > > SSL handshake has read 0 bytes and written 305 bytes but using curl it did reply with a familiar “This is not a HTTP port” so … that’s Elasticsearch. A non-TLS-enabled Elasticsearch. **Another** Elasticsearch? A vague memory stirred that I’d at some point installed it as a service - but no, that’s not running. Check netstat - OK yes, ports 9300, **9301** and 9505 are listening. Check Process Explorer for a some zombie Java process (could it have kept running after a CMD prompt had closed? Seems unlikely.) Check the PID from netstat against the process list and oh, here’s the culprit. One of the components in [App Dynamics](https://www.appdynamics.com/), which we’ve been looking at recently, apparently contains an embedded instance of Elasticsearch - which starts up using the default 9200/9300 ports. Host has been running fine with that installed for a while - my supposition is that when I installed it, I had “my” Elasticsearch running, so it claimed ports 9201/9301. On a recent reboot, it claimed the default ports first, moving “my” instance quietly onto 9201/9301. I say “quietly” - it does of course appear in the logs, but unless you’re looking for it, it’s swamped by other output - and it’s sufficiently similar to what you’d expect that it can go un-noticed. Forcing the ports to 9200/9300 in elasticsearch.yml provokes a failure if already claimed, and I think is a practice I’m going to promote. I’m unconvinced that the standard Elasticsearch choice to try ports in a range is a good idea, but that’s no fault of Search Guard. What could perhaps be improved by Search Guard is the handling of such a failure case - sgadmin making a TLS connection to a service which turns out to be plaintext Elastic transport protocol. Something to handle the TLS library’s failure cases and report more like “this Elastic instance is not TLS-secured” - perhaps fall back to a non-TLS TransportClient, make a cluster status request and report some basics like cluster name / node name / version? Is it too much an edge case to be worth handling? “None of the configured nodes are available” provides no indication of a transport-level issue. regards, James > **···** > > On Tuesday, 5 June 2018 17:47:49 UTC+1, James Beckett wrote: ---
**Author:** ![jkressin](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/jkressin/32/9_2.png) [@jkressin](https://forum.search-guard.com/u/jkressin)\ **Post date:** [June 8, 2018, 4:24pm UTC](https://forum.search-guard.com/t/sgadmin-none-of-the-configured-nodes-are-available-with-6-2-2-22-1/932/7 "2018-06-08T16:24:34Z")
Hi James, thanks very much for your very good explanation, that makes perfect sense and is indeed _very_ hard to spot. While I do think it might be an edge case, I will open an issue for improving the error message nonetheless. Because if it does happen, it creates a lot of confusion. I will also add another ticket to add this specific case to the sgadmin Troubleshooting docs, or improve the existing docs. Thanks again for your very helpful explanation! > **···** > > On Thursday, June 7, 2018 at 6:45:25 PM UTC+2, James Beckett wrote: > > > SOLVED: third-party interaction ☹ > > > > > > On Tuesday, 5 June 2018 17:47:49 UTC+1, James Beckett wrote: > > > > > I’m going to repeat afresh on a couple of other nodes (Win and Linux) and compare the resulting configs - will update here after. > > > > > > After successes on other nodes, I tried a parallel install on the same node that had a problem (using ports 9205 and 9305 to keep well clear of the usual). Clean install worked fine, sgadmin perfectly happy. > > > Recursive diff of each install directory showed nothing corrupted in install - only differences were a few elasticsearch.yml lines around zen discovery and the different paths I’d deliberately set for clustername/data/logs etc. I’ve no hair left at this point. > > > > > > I tried getting a bare TLS connection a few times - even that wasn’t happy: > > > > > > > ## openssl s\_client -msg -CAfile config.devhost/root-ca.pem -cert config.devhost/kirk.pem -key config.devhost/kirk-key.pem -connect devhost:9300 CONNECTED(00000003) 140537057156824:error:140790E5:SSL routines:ssl23\_write:ssl handshake failure:s23\_lib.c:177: > > > > > > ## no peer certificate available > > > > > > ## No client certificate CA names sent > > > > > > SSL handshake has read 0 bytes and written 305 bytes > > > > > > but using curl it did reply with a familiar “This is not a HTTP port” so … that’s Elasticsearch. A non-TLS-enabled Elasticsearch. **Another** Elasticsearch? A vague memory stirred that I’d at some point installed it as a service - but no, that’s not running. Check netstat - OK yes, ports 9300, **9301** and 9505 are listening. Check Process Explorer for a some zombie Java process (could it have kept running after a CMD prompt had closed? Seems unlikely.) Check the PID from netstat against the process list and oh, here’s the culprit. > > > > > > One of the components in [App Dynamics](https://www.appdynamics.com/), which we’ve been looking at recently, apparently contains an embedded instance of Elasticsearch - which starts up using the default 9200/9300 ports. Host has been running fine with that installed for a while - my supposition is that when I installed it, I had “my” Elasticsearch running, so it claimed ports 9201/9301. On a recent reboot, it claimed the default ports first, moving “my” instance quietly onto 9201/9301. I say “quietly” - it does of course appear in the logs, but unless you’re looking for it, it’s swamped by other output - and it’s sufficiently similar to what you’d expect that it can go un-noticed. > > > > > > Forcing the ports to 9200/9300 in elasticsearch.yml provokes a failure if already claimed, and I think is a practice I’m going to promote. I’m unconvinced that the standard Elasticsearch choice to try ports in a range is a good idea, but that’s no fault of Search Guard. > > > > > > What could perhaps be improved by Search Guard is the handling of such a failure case - sgadmin making a TLS connection to a service which turns out to be plaintext Elastic transport protocol. Something to handle the TLS library’s failure cases and report more like “this Elastic instance is not TLS-secured” - perhaps fall back to a non-TLS TransportClient, make a cluster status request and report some basics like cluster name / node name / version? Is it too much an edge case to be worth handling? “None of the configured nodes are available” provides no indication of a transport-level issue. > > > > > > regards, > > > James