# Searchguard could not be Initialized with error: Root cause: MasterNotDiscoveredException\[null\] **URL:** \ **Category:** Search Guard\ **Created:** [July 2, 2018, 8:20am UTC](https://forum.search-guard.com/t/searchguard-could-not-be-initialized-with-error-root-cause-masternotdiscoveredexception-null/990 "2018-07-02T08:20:34Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![YS\_L](https://avatars.discourse-cdn.com/v4/letter/y/2bfe46/32.png) [@YS\_L](https://forum.search-guard.com/u/YS_L)\ **Post date:** [July 2, 2018, 8:20am UTC](https://forum.search-guard.com/t/searchguard-could-not-be-initialized-with-error-root-cause-masternotdiscoveredexception-null/990/1 "2018-07-02T08:20:34Z")
Hi, I have this error when initializing Searchguard : _./sgadmin.sh -diagnose -cd /usr/share/elasticsearch/plugins/search-guard-6/sgconfig -icl -key /etc/elasticsearch/kirk.key -keypass dsun9XyU0pL-cert /etc/elasticsearch/kirk.pem -cacert /etc/elasticsearch/root-ca.pem -nhnv_ WARNING: JAVA\_HOME not set, will use /bin/java Search Guard Admin v6 Will connect to localhost:9300 … done Elasticsearch Version: 6.2.4 Search Guard Version: 6.2.4-22.1 Connected as CN=[kirk.mysolution.com](http://kirk.nissolution.com),OU=Ops,O=mysolution Com, Inc.,DC=mysolution,DC=com Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-6/tools/sgadmin\_diag\_trace\_2018-Jul-02\_07-25-44.txt Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate … Cannot retrieve cluster state due to: null. This is not an error, will keep on trying … Root cause: MasterNotDiscoveredException[null] (org.elasticsearch.discovery.MasterNotDiscoveredException/org.elasticsearch.discovery.MasterNotDiscoveredException) - Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates) - Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml - If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file) - Add --accept-red-cluster to allow sgadmin to operate on a red cluster. Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{1KqhDyc2RQSzFOG508SZlw}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying … Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{1KqhDyc2RQSzFOG508SZlw}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException) - Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates) - Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml - If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file) - Add --accept-red-cluster to allow sgadmin to operate on a red cluster. **My elasticsearch log:** [2018-07-02T08:15:17,771][ERROR][c.f.s.t.SearchGuardRequestHandler] ElasticsearchException[Illegal parameter in http or transport request found. This means that one node is trying to connect to another with a non-node certificate (no OID or searchguard.nodes\_dn incorrect configured) or that someone is spoofing requests. Check your TLS certificate setup as described here: See [TLS help | Security for Elasticsearch | Search Guard](http://docs.search-guard.com/latest/troubleshooting-tls)] [2018-07-02T08:15:17,773][INFO][o.e.d.z.ZenDiscovery] [node-1] failed to send join request to master [{node-2}{JYLpvoLWRhC4jD-uJWMWdw}{e6CcuimNQSm9yrDW0PwACA}{172.30.0.128}{172.30.0.128:9300}], reason [RemoteTransportException[[node-2][172.30.0.128:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[node-1][172.30.0.85:9300] general node connection failure]; nested: IllegalStateException[handshake failed with {node-1}{sc-e3s2\_Qhm8d7bOBiRQhA}{3p-oZWu9Tie7-AdZ\_D75bQ}{172.30.0.85}{172.30.0.85:9300}]; nested: RemoteTransportException[[node-1][172.30.0.85:9300][internal:transport/handshake]]; nested: ElasticsearchException[Illegal parameter in http or transport request found. [2018-07-02T08:15:17,785][ERROR][c.f.s.t.SearchGuardRequestHandler] ElasticsearchException[Illegal parameter in http or transport request found. This means that one node is trying to connect to another with a non-node certificate (no OID or searchguard.nodes\_dn incorrect configured) or that someone is spoofing requests. Check your TLS certificate setup as described here: See [TLS help | Security for Elasticsearch | Search Guard](http://docs.search-guard.com/latest/troubleshooting-tls)] [2018-07-02T08:15:17,785][WARN][o.e.d.z.UnicastZenPing] [node-1] [396] failed send ping to {#zen\_unicast\_52.40.209.185\_0#}{CIfHHpV1RQKwkxmZ0SCtgg}{52.40.209.185}{52.40.209.185:9300} I am using self-signed certificates on all nodes. Any idea what caused this? Need help on this. Thanks in advanced! ---
**Author:** ![jkressin](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/jkressin/32/9_2.png) [@jkressin](https://forum.search-guard.com/u/jkressin)\ **Post date:** [July 3, 2018, 1:34pm UTC](https://forum.search-guard.com/t/searchguard-could-not-be-initialized-with-error-root-cause-masternotdiscoveredexception-null/990/2 "2018-07-03T13:34:42Z")
This means that one node is trying to connect to another with a non-node certificate (no OID or searchguard.nodes\_dn incorrect configured) or that someone is spoofing requests. Check your TLS certificate setup as described here: See [TLS help | Security for Elasticsearch | Search Guard](http://docs.search-guard.com/latest/troubleshooting-tls)]" As printed in the message: “searchguard.nodes\_dn incorrect configured” Have you checked the searchguard.nodes\_dn entry in you elasticsearch.yml? > **···** > > On Monday, July 2, 2018 at 10:01:24 AM UTC+2, YS .L wrote: > > > Hi, > > > I have this error when initializing Searchguard : > > > > > > _./sgadmin.sh -diagnose -cd /usr/share/elasticsearch/plugins/search-guard-6/sgconfig -icl -key /etc/elasticsearch/kirk.key -keypass dsun9XyU0pL-cert /etc/elasticsearch/kirk.pem -cacert /etc/elasticsearch/root-ca.pem -nhnv_ > > > > > > WARNING: JAVA\_HOME not set, will use /bin/java > > > Search Guard Admin v6 > > > Will connect to localhost:9300 … done > > > Elasticsearch Version: 6.2.4 > > > Search Guard Version: 6.2.4-22.1 > > > Connected as CN=[kirk.mysolution.com](http://kirk.nissolution.com),OU=Ops,O=mysolution Com, Inc.,DC=mysolution,DC=com > > > Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-6/tools/sgadmin\_diag\_trace\_2018-Jul-02\_07-25-44.txt > > > Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate … > > > Cannot retrieve cluster state due to: null. This is not an error, will keep on trying … > > > Root cause: MasterNotDiscoveredException[null] (org.elasticsearch.discovery.MasterNotDiscoveredException/org.elasticsearch.discovery.MasterNotDiscoveredException) > > > - Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates) > > > - Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml > > > - If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file) > > > - Add --accept-red-cluster to allow sgadmin to operate on a red cluster. > > > Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{1KqhDyc2RQSzFOG508SZlw}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying … > > > Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{1KqhDyc2RQSzFOG508SZlw}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException) > > > - Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates) > > > - Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml > > > - If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file) > > > - Add --accept-red-cluster to allow sgadmin to operate on a red cluster. > > > > > > **My elasticsearch log:** > > > > > > [2018-07-02T08:15:17,771][ERROR][c.f.s.t.SearchGuardRequestHandler] ElasticsearchException[Illegal parameter in http or transport request found. > > > This means that one node is trying to connect to another with > > > a non-node certificate (no OID or searchguard.nodes\_dn incorrect configured) or that someone > > > is spoofing requests. Check your TLS certificate setup as described here: See [TLS help | Security for Elasticsearch | Search Guard](http://docs.search-guard.com/latest/troubleshooting-tls)] > > > > > > [2018-07-02T08:15:17,773][INFO][o.e.d.z.ZenDiscovery] [node-1] failed to send join request to master [{node-2}{JYLpvoLWRhC4jD-uJWMWdw}{e6CcuimNQSm9yrDW0PwACA}{172.30.0.128}{172.30.0.128:9300}], reason [RemoteTransportException[[node-2][172.30.0.128:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[node-1][172.30.0.85:9300] general node connection failure]; nested: IllegalStateException[handshake failed with {node-1}{sc-e3s2\_Qhm8d7bOBiRQhA}{3p-oZWu9Tie7-AdZ\_D75bQ}{172.30.0.85}{172.30.0.85:9300}]; nested: RemoteTransportException[[node-1][172.30.0.85:9300][internal:transport/handshake]]; nested: ElasticsearchException[Illegal parameter in http or transport request found. > > > > > > [2018-07-02T08:15:17,785][ERROR][c.f.s.t.SearchGuardRequestHandler] ElasticsearchException[Illegal parameter in http or transport request found. > > > This means that one node is trying to connect to another with > > > a non-node certificate (no OID or searchguard.nodes\_dn incorrect configured) or that someone > > > is spoofing requests. Check your TLS certificate setup as described here: See [TLS help | Security for Elasticsearch | Search Guard](http://docs.search-guard.com/latest/troubleshooting-tls)] > > > [2018-07-02T08:15:17,785][WARN][o.e.d.z.UnicastZenPing] [node-1] [396] failed send ping to {#zen\_unicast\_52.40.209.185\_0#}{CIfHHpV1RQKwkxmZ0SCtgg}{52.40.209.185}{52.40.209.185:9300} > > > > > > > > > I am using self-signed certificates on all nodes. > > > Any idea what caused this? Need help on this. > > > > > > Thanks in advanced! > > > > > > > > > > > "[2018-07-02T08:15:17,771][ERROR][c.f.s.t.SearchGuardRequestHandler] ElasticsearchException[Illegal parameter in http or transport request found.Have you gone through the troubleshooting article as stated in the log message? ---
**Author:** ![YS\_L](https://avatars.discourse-cdn.com/v4/letter/y/2bfe46/32.png) [@YS\_L](https://forum.search-guard.com/u/YS_L)\ **Post date:** [July 4, 2018, 2:28am UTC](https://forum.search-guard.com/t/searchguard-could-not-be-initialized-with-error-root-cause-masternotdiscoveredexception-null/990/3 "2018-07-04T02:28:35Z")
Hi Jochen, You save my life again. I overlooked the ‘searchguard.nodes\_dn’ in my node.1 elasticsearch yml. Now it works perfectly I after did some modification and configuration on EC2 discovery. Thank you so much! > **···** > > On Tuesday, 3 July 2018 21:34:42 UTC+8, Jochen Kressin wrote: > > > This means that one node is trying to connect to another with > > > a non-node certificate (no OID or searchguard.nodes\_dn incorrect configured) or that someone > > > is spoofing requests. Check your TLS certificate setup as described here: See [http://docs.search-guard.com/latest/troubleshooting-tls](http://docs.search-guard.com/latest/troubleshooting-tls)]" > > > > > > > > > As printed in the message: > > > > > > “searchguard.nodes\_dn incorrect configured” > > > > > > Have you checked the searchguard.nodes\_dn entry in you elasticsearch.yml? > > > > > > On Monday, July 2, 2018 at 10:01:24 AM UTC+2, YS .L wrote: > > > > > Hi, > > > > I have this error when initializing Searchguard : > > > > > > > > _./sgadmin.sh -diagnose -cd /usr/share/elasticsearch/plugins/search-guard-6/sgconfig -icl -key /etc/elasticsearch/kirk.key -keypass dsun9XyU0pL-cert /etc/elasticsearch/kirk.pem -cacert /etc/elasticsearch/root-ca.pem -nhnv_ > > > > > > > > WARNING: JAVA\_HOME not set, will use /bin/java > > > > Search Guard Admin v6 > > > > Will connect to localhost:9300 … done > > > > Elasticsearch Version: 6.2.4 > > > > Search Guard Version: 6.2.4-22.1 > > > > Connected as CN=[kirk.mysolution.com](http://kirk.nissolution.com),OU=Ops,O=mysolution Com, Inc.,DC=mysolution,DC=com > > > > Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-6/tools/sgadmin\_diag\_trace\_2018-Jul-02\_07-25-44.txt > > > > Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate … > > > > Cannot retrieve cluster state due to: null. This is not an error, will keep on trying … > > > > Root cause: MasterNotDiscoveredException[null] (org.elasticsearch.discovery.MasterNotDiscoveredException/org.elasticsearch.discovery.MasterNotDiscoveredException) > > > > - Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates) > > > > - Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml > > > > - If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file) > > > > - Add --accept-red-cluster to allow sgadmin to operate on a red cluster. > > > > Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{1KqhDyc2RQSzFOG508SZlw}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying … > > > > Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{1KqhDyc2RQSzFOG508SZlw}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException) > > > > - Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates) > > > > - Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml > > > > - If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file) > > > > - Add --accept-red-cluster to allow sgadmin to operate on a red cluster. > > > > > > > > **My elasticsearch log:** > > > > > > > > [2018-07-02T08:15:17,771][ERROR][c.f.s.t.SearchGuardRequestHandler] ElasticsearchException[Illegal parameter in http or transport request found. > > > > This means that one node is trying to connect to another with > > > > a non-node certificate (no OID or searchguard.nodes\_dn incorrect configured) or that someone > > > > is spoofing requests. Check your TLS certificate setup as described here: See [http://docs.search-guard.com/latest/troubleshooting-tls](http://docs.search-guard.com/latest/troubleshooting-tls)] > > > > > > > > [2018-07-02T08:15:17,773][INFO][o.e.d.z.ZenDiscovery] [node-1] failed to send join request to master [{node-2}{JYLpvoLWRhC4jD-uJWMWdw}{e6CcuimNQSm9yrDW0PwACA}{172.30.0.128}{172.30.0.128:9300}], reason [RemoteTransportException[[node-2][172.30.0.128:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[node-1][[172.30.0.85:9300](http://172.30.0.85:9300)] general node connection failure]; nested: IllegalStateException[handshake failed with {node-1}{sc-e3s2\_Qhm8d7bOBiRQhA}{3p-oZWu9Tie7-AdZ\_D75bQ}{172.30.0.85}{172.30.0.85:9300}]; nested: RemoteTransportException[[node-1][172.30.0.85:9300][internal:transport/handshake]]; nested: ElasticsearchException[Illegal parameter in http or transport request found. > > > > > > > > [2018-07-02T08:15:17,785][ERROR][c.f.s.t.SearchGuardRequestHandler] ElasticsearchException[Illegal parameter in http or transport request found. > > > > This means that one node is trying to connect to another with > > > > a non-node certificate (no OID or searchguard.nodes\_dn incorrect configured) or that someone > > > > is spoofing requests. Check your TLS certificate setup as described here: See [http://docs.search-guard.com/latest/troubleshooting-tls](http://docs.search-guard.com/latest/troubleshooting-tls)] > > > > [2018-07-02T08:15:17,785][WARN][o.e.d.z.UnicastZenPing] [node-1] [396] failed send ping to {#zen\_unicast\_52.40.209.185\_0#}{CIfHHpV1RQKwkxmZ0SCtgg}{52.40.209.185}{[52.40.209.185:9300](http://52.40.209.185:9300)} > > > > > > > > > > > > I am using self-signed certificates on all nodes. > > > > Any idea what caused this? Need help on this. > > > > > > > > Thanks in advanced! > > > > > > > > > > > > > > > "[2018-07-02T08:15:17,771][ERROR][c.f.s.t.SearchGuardRequestHandler] ElasticsearchException[Illegal parameter in http or transport request found.Have you gone through the troubleshooting article as stated in the log message? ---
**Author:** ![YS\_L](https://avatars.discourse-cdn.com/v4/letter/y/2bfe46/32.png) [@YS\_L](https://forum.search-guard.com/u/YS_L)\ **Post date:** [July 4, 2018, 4:07am UTC](https://forum.search-guard.com/t/searchguard-could-not-be-initialized-with-error-root-cause-masternotdiscoveredexception-null/990/4 "2018-07-04T04:07:07Z")
Hi Jochen, One more question regarding the sgadmin. Could you help to explain this? I have node1 and node2 both with searchguard installed. node 1 and node 2 join the cluster. Let says I have created these: - internal user1 in node1 - internal user2 in node2 The configuration is done in the file sg\_internal\_users.yml and initialize with sgadmin in 2 nodes separately (node1 first and follow by node2). My kibana is pointing to node1. Can I login with user2 (initialized in node2) through kibana? As my understanding is both nodes share the same searchguard index and I supposed to be able to login user2 in kibana. Now the case is when i initialize with sgadmin on node1, I cant login user2. And after I initialize on node2, I can login user2 but no longer user1. Is the sgadmin initialization overwrites the configuration changes on other node every time it runs? If that is the case, what is the best practise? Is it better the configuration always be done on only one node? Thanks. > **···** > > On Monday, 2 July 2018 16:01:24 UTC+8, YS .L wrote: > > > Hi, > > > I have this error when initializing Searchguard : > > > > > > _./sgadmin.sh -diagnose -cd /usr/share/elasticsearch/plugins/search-guard-6/sgconfig -icl -key /etc/elasticsearch/kirk.key -keypass dsun9XyU0pL-cert /etc/elasticsearch/kirk.pem -cacert /etc/elasticsearch/root-ca.pem -nhnv_ > > > > > > WARNING: JAVA\_HOME not set, will use /bin/java > > > Search Guard Admin v6 > > > Will connect to localhost:9300 … done > > > Elasticsearch Version: 6.2.4 > > > Search Guard Version: 6.2.4-22.1 > > > Connected as CN=[kirk.mysolution.com](http://kirk.nissolution.com),OU=Ops,O=mysolution Com, Inc.,DC=mysolution,DC=com > > > Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-6/tools/sgadmin\_diag\_trace\_2018-Jul-02\_07-25-44.txt > > > Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate … > > > Cannot retrieve cluster state due to: null. This is not an error, will keep on trying … > > > Root cause: MasterNotDiscoveredException[null] (org.elasticsearch.discovery.MasterNotDiscoveredException/org.elasticsearch.discovery.MasterNotDiscoveredException) > > > - Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates) > > > - Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml > > > - If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file) > > > - Add --accept-red-cluster to allow sgadmin to operate on a red cluster. > > > Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{1KqhDyc2RQSzFOG508SZlw}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying … > > > Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{1KqhDyc2RQSzFOG508SZlw}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException) > > > - Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates) > > > - Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml > > > - If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file) > > > - Add --accept-red-cluster to allow sgadmin to operate on a red cluster. > > > > > > **My elasticsearch log:** > > > > > > [2018-07-02T08:15:17,771][ERROR][c.f.s.t.SearchGuardRequestHandler] ElasticsearchException[Illegal parameter in http or transport request found. > > > This means that one node is trying to connect to another with > > > a non-node certificate (no OID or searchguard.nodes\_dn incorrect configured) or that someone > > > is spoofing requests. Check your TLS certificate setup as described here: See [http://docs.search-guard.com/latest/troubleshooting-tls](http://docs.search-guard.com/latest/troubleshooting-tls)] > > > > > > [2018-07-02T08:15:17,773][INFO][o.e.d.z.ZenDiscovery] [node-1] failed to send join request to master [{node-2}{JYLpvoLWRhC4jD-uJWMWdw}{e6CcuimNQSm9yrDW0PwACA}{172.30.0.128}{172.30.0.128:9300}], reason [RemoteTransportException[[node-2][172.30.0.128:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[node-1][[172.30.0.85:9300](http://172.30.0.85:9300)] general node connection failure]; nested: IllegalStateException[handshake failed with {node-1}{sc-e3s2\_Qhm8d7bOBiRQhA}{3p-oZWu9Tie7-AdZ\_D75bQ}{172.30.0.85}{172.30.0.85:9300}]; nested: RemoteTransportException[[node-1][172.30.0.85:9300][internal:transport/handshake]]; nested: ElasticsearchException[Illegal parameter in http or transport request found. > > > > > > [2018-07-02T08:15:17,785][ERROR][c.f.s.t.SearchGuardRequestHandler] ElasticsearchException[Illegal parameter in http or transport request found. > > > This means that one node is trying to connect to another with > > > a non-node certificate (no OID or searchguard.nodes\_dn incorrect configured) or that someone > > > is spoofing requests. Check your TLS certificate setup as described here: See [http://docs.search-guard.com/latest/troubleshooting-tls](http://docs.search-guard.com/latest/troubleshooting-tls)] > > > [2018-07-02T08:15:17,785][WARN][o.e.d.z.UnicastZenPing] [node-1] [396] failed send ping to {#zen\_unicast\_52.40.209.185\_0#}{CIfHHpV1RQKwkxmZ0SCtgg}{52.40.209.185}{[52.40.209.185:9300](http://52.40.209.185:9300)} > > > > > > > > > I am using self-signed certificates on all nodes. > > > Any idea what caused this? Need help on this. > > > > > > Thanks in advanced! > > > > > > > > > ---
**Author:** ![jkressin](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/jkressin/32/9_2.png) [@jkressin](https://forum.search-guard.com/u/jkressin)\ **Post date:** [July 4, 2018, 7:50pm UTC](https://forum.search-guard.com/t/searchguard-could-not-be-initialized-with-error-root-cause-masternotdiscoveredexception-null/990/5 "2018-07-04T19:50:22Z")
There is only one place where the SG configuration is stored, the SG index. As with any other Elasticsearch index, it is shared amongst all nodes. It does not matter on which node you make changes, they are reflected throughout the whole cluster. This is the same as if you add or change a document in any index on Elasticsearch: The changes are immediately propagated and you can query any node for the added/changed document. Having different users on different nodes in a distributed system does not make sense: The cluster keeps a global state. Why would you want to have different configurations on different nodes? > **···** > > On Wednesday, July 4, 2018 at 5:20:28 AM UTC+2, YS .L wrote: > > > Hi Jochen, > > > One more question regarding the sgadmin. Could you help to explain this? > > > > > > I have node1 and node2 both with searchguard installed. > > > > > > node 1 and node 2 join the cluster. > > > > > > Let says I have created these: > > > - internal user1 in node1 > > - internal user2 in node2 > > The configuration is done in the file sg\_internal\_users.yml and initialize with sgadmin in 2 nodes separately (node1 first and follow by node2). > > > > > > My kibana is pointing to node1. Can I login with user2 (initialized in node2) through kibana? > > > > > > As my understanding is both nodes share the same searchguard index and I supposed to be able to login user2 in kibana. > > > Now the case is when i initialize with sgadmin on node1, I cant login user2. And after I initialize on node2, I can login user2 but no longer user1. > > > Is the sgadmin initialization overwrites the configuration changes on other node every time it runs? If that is the case, what is the best practise? > > > Is it better the configuration always be done on only one node? > > > > > > Thanks. > > > > > > On Monday, 2 July 2018 16:01:24 UTC+8, YS .L wrote: > > > > > Hi, > > > > I have this error when initializing Searchguard : > > > > > > > > _./sgadmin.sh -diagnose -cd /usr/share/elasticsearch/plugins/search-guard-6/sgconfig -icl -key /etc/elasticsearch/kirk.key -keypass dsun9XyU0pL-cert /etc/elasticsearch/kirk.pem -cacert /etc/elasticsearch/root-ca.pem -nhnv_ > > > > > > > > WARNING: JAVA\_HOME not set, will use /bin/java > > > > Search Guard Admin v6 > > > > Will connect to localhost:9300 … done > > > > Elasticsearch Version: 6.2.4 > > > > Search Guard Version: 6.2.4-22.1 > > > > Connected as CN=[kirk.mysolution.com](http://kirk.nissolution.com),OU=Ops,O=mysolution Com, Inc.,DC=mysolution,DC=com > > > > Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-6/tools/sgadmin\_diag\_trace\_2018-Jul-02\_07-25-44.txt > > > > Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate … > > > > Cannot retrieve cluster state due to: null. This is not an error, will keep on trying … > > > > Root cause: MasterNotDiscoveredException[null] (org.elasticsearch.discovery.MasterNotDiscoveredException/org.elasticsearch.discovery.MasterNotDiscoveredException) > > > > - Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates) > > > > - Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml > > > > - If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file) > > > > - Add --accept-red-cluster to allow sgadmin to operate on a red cluster. > > > > Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{1KqhDyc2RQSzFOG508SZlw}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying … > > > > Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{1KqhDyc2RQSzFOG508SZlw}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException) > > > > - Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates) > > > > - Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml > > > > - If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file) > > > > - Add --accept-red-cluster to allow sgadmin to operate on a red cluster. > > > > > > > > **My elasticsearch log:** > > > > > > > > [2018-07-02T08:15:17,771][ERROR][c.f.s.t.SearchGuardRequestHandler] ElasticsearchException[Illegal parameter in http or transport request found. > > > > This means that one node is trying to connect to another with > > > > a non-node certificate (no OID or searchguard.nodes\_dn incorrect configured) or that someone > > > > is spoofing requests. Check your TLS certificate setup as described here: See [http://docs.search-guard.com/latest/troubleshooting-tls](http://docs.search-guard.com/latest/troubleshooting-tls)] > > > > > > > > [2018-07-02T08:15:17,773][INFO][o.e.d.z.ZenDiscovery] [node-1] failed to send join request to master [{node-2}{JYLpvoLWRhC4jD-uJWMWdw}{e6CcuimNQSm9yrDW0PwACA}{172.30.0.128}{172.30.0.128:9300}], reason [RemoteTransportException[[node-2][172.30.0.128:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[node-1][[172.30.0.85:9300](http://172.30.0.85:9300)] general node connection failure]; nested: IllegalStateException[handshake failed with {node-1}{sc-e3s2\_Qhm8d7bOBiRQhA}{3p-oZWu9Tie7-AdZ\_D75bQ}{172.30.0.85}{172.30.0.85:9300}]; nested: RemoteTransportException[[node-1][172.30.0.85:9300][internal:transport/handshake]]; nested: ElasticsearchException[Illegal parameter in http or transport request found. > > > > > > > > [2018-07-02T08:15:17,785][ERROR][c.f.s.t.SearchGuardRequestHandler] ElasticsearchException[Illegal parameter in http or transport request found. > > > > This means that one node is trying to connect to another with > > > > a non-node certificate (no OID or searchguard.nodes\_dn incorrect configured) or that someone > > > > is spoofing requests. Check your TLS certificate setup as described here: See [http://docs.search-guard.com/latest/troubleshooting-tls](http://docs.search-guard.com/latest/troubleshooting-tls)] > > > > [2018-07-02T08:15:17,785][WARN][o.e.d.z.UnicastZenPing] [node-1] [396] failed send ping to {#zen\_unicast\_52.40.209.185\_0#}{CIfHHpV1RQKwkxmZ0SCtgg}{52.40.209.185}{[52.40.209.185:9300](http://52.40.209.185:9300)} > > > > > > > > > > > > I am using self-signed certificates on all nodes. > > > > Any idea what caused this? Need help on this. > > > > > > > > Thanks in advanced! > > > > > > > > > > > > ---
**Author:** ![YS\_L](https://avatars.discourse-cdn.com/v4/letter/y/2bfe46/32.png) [@YS\_L](https://forum.search-guard.com/u/YS_L)\ **Post date:** [July 5, 2018, 6:31am UTC](https://forum.search-guard.com/t/searchguard-could-not-be-initialized-with-error-root-cause-masternotdiscoveredexception-null/990/6 "2018-07-05T06:31:13Z")
Hi Jochen, I got you. Now I have better understanding on the concept. Thanks! > **···** > > On Thursday, 5 July 2018 03:50:22 UTC+8, Jochen Kressin wrote: > > > There is only one place where the SG configuration is stored, the SG index. As with any other Elasticsearch index, it is shared amongst all nodes. It does not matter on which node you make changes, they are reflected throughout the whole cluster. This is the same as if you add or change a document in any index on Elasticsearch: The changes are immediately propagated and you can query any node for the added/changed document. > > > Having different users on different nodes in a distributed system does not make sense: The cluster keeps a global state. Why would you want to have different configurations on different nodes? > > > > On Wednesday, July 4, 2018 at 5:20:28 AM UTC+2, YS .L wrote: > > > > > Hi Jochen, > > > > One more question regarding the sgadmin. Could you help to explain this? > > > > > > > > I have node1 and node2 both with searchguard installed. > > > > > > > > node 1 and node 2 join the cluster. > > > > > > > > Let says I have created these: > > > > - internal user1 in node1 > > > - internal user2 in node2 > > > The configuration is done in the file sg\_internal\_users.yml and initialize with sgadmin in 2 nodes separately (node1 first and follow by node2). > > > > > > > > My kibana is pointing to node1. Can I login with user2 (initialized in node2) through kibana? > > > > > > > > As my understanding is both nodes share the same searchguard index and I supposed to be able to login user2 in kibana. > > > > Now the case is when i initialize with sgadmin on node1, I cant login user2. And after I initialize on node2, I can login user2 but no longer user1. > > > > Is the sgadmin initialization overwrites the configuration changes on other node every time it runs? If that is the case, what is the best practise? > > > > Is it better the configuration always be done on only one node? > > > > > > > > Thanks. > > > > > > > > On Monday, 2 July 2018 16:01:24 UTC+8, YS .L wrote: > > > > > > > Hi, > > > > > I have this error when initializing Searchguard : > > > > > > > > > > _./sgadmin.sh -diagnose -cd /usr/share/elasticsearch/plugins/search-guard-6/sgconfig -icl -key /etc/elasticsearch/kirk.key -keypass dsun9XyU0pL-cert /etc/elasticsearch/kirk.pem -cacert /etc/elasticsearch/root-ca.pem -nhnv_ > > > > > > > > > > WARNING: JAVA\_HOME not set, will use /bin/java > > > > > Search Guard Admin v6 > > > > > Will connect to localhost:9300 … done > > > > > Elasticsearch Version: 6.2.4 > > > > > Search Guard Version: 6.2.4-22.1 > > > > > Connected as CN=[kirk.mysolution.com](http://kirk.nissolution.com),OU=Ops,O=mysolution Com, Inc.,DC=mysolution,DC=com > > > > > Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-6/tools/sgadmin\_diag\_trace\_2018-Jul-02\_07-25-44.txt > > > > > Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate … > > > > > Cannot retrieve cluster state due to: null. This is not an error, will keep on trying … > > > > > Root cause: MasterNotDiscoveredException[null] (org.elasticsearch.discovery.MasterNotDiscoveredException/org.elasticsearch.discovery.MasterNotDiscoveredException) > > > > > - Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates) > > > > > - Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml > > > > > - If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file) > > > > > - Add --accept-red-cluster to allow sgadmin to operate on a red cluster. > > > > > Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{1KqhDyc2RQSzFOG508SZlw}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying … > > > > > Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{1KqhDyc2RQSzFOG508SZlw}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException) > > > > > - Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates) > > > > > - Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml > > > > > - If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file) > > > > > - Add --accept-red-cluster to allow sgadmin to operate on a red cluster. > > > > > > > > > > **My elasticsearch log:** > > > > > > > > > > [2018-07-02T08:15:17,771][ERROR][c.f.s.t.SearchGuardRequestHandler] ElasticsearchException[Illegal parameter in http or transport request found. > > > > > This means that one node is trying to connect to another with > > > > > a non-node certificate (no OID or searchguard.nodes\_dn incorrect configured) or that someone > > > > > is spoofing requests. Check your TLS certificate setup as described here: See [http://docs.search-guard.com/latest/troubleshooting-tls](http://docs.search-guard.com/latest/troubleshooting-tls)] > > > > > > > > > > [2018-07-02T08:15:17,773][INFO][o.e.d.z.ZenDiscovery] [node-1] failed to send join request to master [{node-2}{JYLpvoLWRhC4jD-uJWMWdw}{e6CcuimNQSm9yrDW0PwACA}{172.30.0.128}{172.30.0.128:9300}], reason [RemoteTransportException[[node-2][172.30.0.128:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[node-1][[172.30.0.85:9300](http://172.30.0.85:9300)] general node connection failure]; nested: IllegalStateException[handshake failed with {node-1}{sc-e3s2\_Qhm8d7bOBiRQhA}{3p-oZWu9Tie7-AdZ\_D75bQ}{172.30.0.85}{172.30.0.85:9300}]; nested: RemoteTransportException[[node-1][172.30.0.85:9300][internal:transport/handshake]]; nested: ElasticsearchException[Illegal parameter in http or transport request found. > > > > > > > > > > [2018-07-02T08:15:17,785][ERROR][c.f.s.t.SearchGuardRequestHandler] ElasticsearchException[Illegal parameter in http or transport request found. > > > > > This means that one node is trying to connect to another with > > > > > a non-node certificate (no OID or searchguard.nodes\_dn incorrect configured) or that someone > > > > > is spoofing requests. Check your TLS certificate setup as described here: See [http://docs.search-guard.com/latest/troubleshooting-tls](http://docs.search-guard.com/latest/troubleshooting-tls)] > > > > > [2018-07-02T08:15:17,785][WARN][o.e.d.z.UnicastZenPing] [node-1] [396] failed send ping to {#zen\_unicast\_52.40.209.185\_0#}{CIfHHpV1RQKwkxmZ0SCtgg}{52.40.209.185}{[52.40.209.185:9300](http://52.40.209.185:9300)} > > > > > > > > > > > > > > > I am using self-signed certificates on all nodes. > > > > > Any idea what caused this? Need help on this. > > > > > > > > > > Thanks in advanced! > > > > > > > > > > > > > > >