Searchguard could not be Initialized with error: Root cause: MasterNotDiscoveredException[null]

Search Guard
1

Hi,

I have this error when initializing Searchguard :

./sgadmin.sh -diagnose -cd /usr/share/elasticsearch/plugins/search-guard-6/sgconfig -icl -key /etc/elasticsearch/kirk.key -keypass dsun9XyU0pL-cert /etc/elasticsearch/kirk.pem -cacert /etc/elasticsearch/root-ca.pem -nhnv

WARNING: JAVA_HOME not set, will use /bin/java

Search Guard Admin v6

Will connect to localhost:9300 … done

Elasticsearch Version: 6.2.4

Search Guard Version: 6.2.4-22.1

Connected as CN=kirk.mysolution.com,OU=Ops,O=mysolution Com, Inc.,DC=mysolution,DC=com

Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-6/tools/sgadmin_diag_trace_2018-Jul-02_07-25-44.txt

Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …

Cannot retrieve cluster state due to: null. This is not an error, will keep on trying …

Root cause: MasterNotDiscoveredException[null] (org.elasticsearch.discovery.MasterNotDiscoveredException/org.elasticsearch.discovery.MasterNotDiscoveredException)

  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)

  • Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml

  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)

  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{1KqhDyc2RQSzFOG508SZlw}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying …

Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{1KqhDyc2RQSzFOG508SZlw}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)

  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)

  • Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml

  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)

  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.

My elasticsearch log:

[2018-07-02T08:15:17,771][ERROR][c.f.s.t.SearchGuardRequestHandler] ElasticsearchException[Illegal parameter in http or transport request found.

This means that one node is trying to connect to another with

a non-node certificate (no OID or searchguard.nodes_dn incorrect configured) or that someone

is spoofing requests. Check your TLS certificate setup as described here: See TLS help | Security for Elasticsearch | Search Guard]

[2018-07-02T08:15:17,773][INFO ][o.e.d.z.ZenDiscovery ] [node-1] failed to send join request to master [{node-2}{JYLpvoLWRhC4jD-uJWMWdw}{e6CcuimNQSm9yrDW0PwACA}{172.30.0.128}{172.30.0.128:9300}], reason [RemoteTransportException[[node-2][172.30.0.128:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[node-1][172.30.0.85:9300] general node connection failure]; nested: IllegalStateException[handshake failed with {node-1}{sc-e3s2_Qhm8d7bOBiRQhA}{3p-oZWu9Tie7-AdZ_D75bQ}{172.30.0.85}{172.30.0.85:9300}]; nested: RemoteTransportException[[node-1][172.30.0.85:9300][internal:transport/handshake]]; nested: ElasticsearchException[Illegal parameter in http or transport request found.

[2018-07-02T08:15:17,785][ERROR][c.f.s.t.SearchGuardRequestHandler] ElasticsearchException[Illegal parameter in http or transport request found.

This means that one node is trying to connect to another with

a non-node certificate (no OID or searchguard.nodes_dn incorrect configured) or that someone

is spoofing requests. Check your TLS certificate setup as described here: See TLS help | Security for Elasticsearch | Search Guard]

[2018-07-02T08:15:17,785][WARN ][o.e.d.z.UnicastZenPing ] [node-1] [396] failed send ping to {#zen_unicast_52.40.209.185_0#}{CIfHHpV1RQKwkxmZ0SCtgg}{52.40.209.185}{52.40.209.185:9300}

I am using self-signed certificates on all nodes.

Any idea what caused this? Need help on this.

Thanks in advanced!

2

This means that one node is trying to connect to another with

a non-node certificate (no OID or searchguard.nodes_dn incorrect configured) or that someone

is spoofing requests. Check your TLS certificate setup as described here: See TLS help | Security for Elasticsearch | Search Guard]"

As printed in the message:

“searchguard.nodes_dn incorrect configured”

Have you checked the searchguard.nodes_dn entry in you elasticsearch.yml?

···

On Monday, July 2, 2018 at 10:01:24 AM UTC+2, YS .L wrote:

Hi,

I have this error when initializing Searchguard :

./sgadmin.sh -diagnose -cd /usr/share/elasticsearch/plugins/search-guard-6/sgconfig -icl -key /etc/elasticsearch/kirk.key -keypass dsun9XyU0pL-cert /etc/elasticsearch/kirk.pem -cacert /etc/elasticsearch/root-ca.pem -nhnv

WARNING: JAVA_HOME not set, will use /bin/java

Search Guard Admin v6

Will connect to localhost:9300 … done

Elasticsearch Version: 6.2.4

Search Guard Version: 6.2.4-22.1

Connected as CN=kirk.mysolution.com,OU=Ops,O=mysolution Com, Inc.,DC=mysolution,DC=com

Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-6/tools/sgadmin_diag_trace_2018-Jul-02_07-25-44.txt

Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …

Cannot retrieve cluster state due to: null. This is not an error, will keep on trying …

Root cause: MasterNotDiscoveredException[null] (org.elasticsearch.discovery.MasterNotDiscoveredException/org.elasticsearch.discovery.MasterNotDiscoveredException)

  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
  • Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{1KqhDyc2RQSzFOG508SZlw}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying …

Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{1KqhDyc2RQSzFOG508SZlw}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)

  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
  • Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.

My elasticsearch log:

[2018-07-02T08:15:17,771][ERROR][c.f.s.t.SearchGuardRequestHandler] ElasticsearchException[Illegal parameter in http or transport request found.

This means that one node is trying to connect to another with

a non-node certificate (no OID or searchguard.nodes_dn incorrect configured) or that someone

is spoofing requests. Check your TLS certificate setup as described here: See TLS help | Security for Elasticsearch | Search Guard]

[2018-07-02T08:15:17,773][INFO ][o.e.d.z.ZenDiscovery ] [node-1] failed to send join request to master [{node-2}{JYLpvoLWRhC4jD-uJWMWdw}{e6CcuimNQSm9yrDW0PwACA}{172.30.0.128}{172.30.0.128:9300}], reason [RemoteTransportException[[node-2][172.30.0.128:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[node-1][172.30.0.85:9300] general node connection failure]; nested: IllegalStateException[handshake failed with {node-1}{sc-e3s2_Qhm8d7bOBiRQhA}{3p-oZWu9Tie7-AdZ_D75bQ}{172.30.0.85}{172.30.0.85:9300}]; nested: RemoteTransportException[[node-1][172.30.0.85:9300][internal:transport/handshake]]; nested: ElasticsearchException[Illegal parameter in http or transport request found.

[2018-07-02T08:15:17,785][ERROR][c.f.s.t.SearchGuardRequestHandler] ElasticsearchException[Illegal parameter in http or transport request found.

This means that one node is trying to connect to another with

a non-node certificate (no OID or searchguard.nodes_dn incorrect configured) or that someone

is spoofing requests. Check your TLS certificate setup as described here: See TLS help | Security for Elasticsearch | Search Guard]

[2018-07-02T08:15:17,785][WARN ][o.e.d.z.UnicastZenPing ] [node-1] [396] failed send ping to {#zen_unicast_52.40.209.185_0#}{CIfHHpV1RQKwkxmZ0SCtgg}{52.40.209.185}{52.40.209.185:9300}

I am using self-signed certificates on all nodes.

Any idea what caused this? Need help on this.

Thanks in advanced!

"[2018-07-02T08:15:17,771][ERROR][c.f.s.t.SearchGuardRequestHandler] ElasticsearchException[Illegal parameter in http or transport request found.Have you gone through the troubleshooting article as stated in the log message?

3

Hi Jochen,

You save my life again. I overlooked the ‘searchguard.nodes_dn’ in my node.1 elasticsearch yml. Now it works perfectly I after did some modification and configuration on EC2 discovery.

Thank you so much!

···

On Tuesday, 3 July 2018 21:34:42 UTC+8, Jochen Kressin wrote:

This means that one node is trying to connect to another with

a non-node certificate (no OID or searchguard.nodes_dn incorrect configured) or that someone

is spoofing requests. Check your TLS certificate setup as described here: See http://docs.search-guard.com/latest/troubleshooting-tls]"

As printed in the message:

“searchguard.nodes_dn incorrect configured”

Have you checked the searchguard.nodes_dn entry in you elasticsearch.yml?

On Monday, July 2, 2018 at 10:01:24 AM UTC+2, YS .L wrote:

Hi,

I have this error when initializing Searchguard :

./sgadmin.sh -diagnose -cd /usr/share/elasticsearch/plugins/search-guard-6/sgconfig -icl -key /etc/elasticsearch/kirk.key -keypass dsun9XyU0pL-cert /etc/elasticsearch/kirk.pem -cacert /etc/elasticsearch/root-ca.pem -nhnv

WARNING: JAVA_HOME not set, will use /bin/java

Search Guard Admin v6

Will connect to localhost:9300 … done

Elasticsearch Version: 6.2.4

Search Guard Version: 6.2.4-22.1

Connected as CN=kirk.mysolution.com,OU=Ops,O=mysolution Com, Inc.,DC=mysolution,DC=com

Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-6/tools/sgadmin_diag_trace_2018-Jul-02_07-25-44.txt

Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …

Cannot retrieve cluster state due to: null. This is not an error, will keep on trying …

Root cause: MasterNotDiscoveredException[null] (org.elasticsearch.discovery.MasterNotDiscoveredException/org.elasticsearch.discovery.MasterNotDiscoveredException)

  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
  • Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{1KqhDyc2RQSzFOG508SZlw}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying …

Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{1KqhDyc2RQSzFOG508SZlw}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)

  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
  • Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.

My elasticsearch log:

[2018-07-02T08:15:17,771][ERROR][c.f.s.t.SearchGuardRequestHandler] ElasticsearchException[Illegal parameter in http or transport request found.

This means that one node is trying to connect to another with

a non-node certificate (no OID or searchguard.nodes_dn incorrect configured) or that someone

is spoofing requests. Check your TLS certificate setup as described here: See http://docs.search-guard.com/latest/troubleshooting-tls]

[2018-07-02T08:15:17,773][INFO ][o.e.d.z.ZenDiscovery ] [node-1] failed to send join request to master [{node-2}{JYLpvoLWRhC4jD-uJWMWdw}{e6CcuimNQSm9yrDW0PwACA}{172.30.0.128}{172.30.0.128:9300}], reason [RemoteTransportException[[node-2][172.30.0.128:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[node-1][172.30.0.85:9300] general node connection failure]; nested: IllegalStateException[handshake failed with {node-1}{sc-e3s2_Qhm8d7bOBiRQhA}{3p-oZWu9Tie7-AdZ_D75bQ}{172.30.0.85}{172.30.0.85:9300}]; nested: RemoteTransportException[[node-1][172.30.0.85:9300][internal:transport/handshake]]; nested: ElasticsearchException[Illegal parameter in http or transport request found.

[2018-07-02T08:15:17,785][ERROR][c.f.s.t.SearchGuardRequestHandler] ElasticsearchException[Illegal parameter in http or transport request found.

This means that one node is trying to connect to another with

a non-node certificate (no OID or searchguard.nodes_dn incorrect configured) or that someone

is spoofing requests. Check your TLS certificate setup as described here: See http://docs.search-guard.com/latest/troubleshooting-tls]

[2018-07-02T08:15:17,785][WARN ][o.e.d.z.UnicastZenPing ] [node-1] [396] failed send ping to {#zen_unicast_52.40.209.185_0#}{CIfHHpV1RQKwkxmZ0SCtgg}{52.40.209.185}{52.40.209.185:9300}

I am using self-signed certificates on all nodes.

Any idea what caused this? Need help on this.

Thanks in advanced!

"[2018-07-02T08:15:17,771][ERROR][c.f.s.t.SearchGuardRequestHandler] ElasticsearchException[Illegal parameter in http or transport request found.Have you gone through the troubleshooting article as stated in the log message?

4

Hi Jochen,

One more question regarding the sgadmin. Could you help to explain this?

I have node1 and node2 both with searchguard installed.

node 1 and node 2 join the cluster.

Let says I have created these:

  • internal user1 in node1
  • internal user2 in node2
    The configuration is done in the file sg_internal_users.yml and initialize with sgadmin in 2 nodes separately (node1 first and follow by node2).

My kibana is pointing to node1. Can I login with user2 (initialized in node2) through kibana?

As my understanding is both nodes share the same searchguard index and I supposed to be able to login user2 in kibana.

Now the case is when i initialize with sgadmin on node1, I cant login user2. And after I initialize on node2, I can login user2 but no longer user1.

Is the sgadmin initialization overwrites the configuration changes on other node every time it runs? If that is the case, what is the best practise?

Is it better the configuration always be done on only one node?

Thanks.

···

On Monday, 2 July 2018 16:01:24 UTC+8, YS .L wrote:

Hi,

I have this error when initializing Searchguard :

./sgadmin.sh -diagnose -cd /usr/share/elasticsearch/plugins/search-guard-6/sgconfig -icl -key /etc/elasticsearch/kirk.key -keypass dsun9XyU0pL-cert /etc/elasticsearch/kirk.pem -cacert /etc/elasticsearch/root-ca.pem -nhnv

WARNING: JAVA_HOME not set, will use /bin/java

Search Guard Admin v6

Will connect to localhost:9300 … done

Elasticsearch Version: 6.2.4

Search Guard Version: 6.2.4-22.1

Connected as CN=kirk.mysolution.com,OU=Ops,O=mysolution Com, Inc.,DC=mysolution,DC=com

Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-6/tools/sgadmin_diag_trace_2018-Jul-02_07-25-44.txt

Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …

Cannot retrieve cluster state due to: null. This is not an error, will keep on trying …

Root cause: MasterNotDiscoveredException[null] (org.elasticsearch.discovery.MasterNotDiscoveredException/org.elasticsearch.discovery.MasterNotDiscoveredException)

  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
  • Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{1KqhDyc2RQSzFOG508SZlw}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying …

Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{1KqhDyc2RQSzFOG508SZlw}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)

  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
  • Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.

My elasticsearch log:

[2018-07-02T08:15:17,771][ERROR][c.f.s.t.SearchGuardRequestHandler] ElasticsearchException[Illegal parameter in http or transport request found.

This means that one node is trying to connect to another with

a non-node certificate (no OID or searchguard.nodes_dn incorrect configured) or that someone

is spoofing requests. Check your TLS certificate setup as described here: See http://docs.search-guard.com/latest/troubleshooting-tls]

[2018-07-02T08:15:17,773][INFO ][o.e.d.z.ZenDiscovery ] [node-1] failed to send join request to master [{node-2}{JYLpvoLWRhC4jD-uJWMWdw}{e6CcuimNQSm9yrDW0PwACA}{172.30.0.128}{172.30.0.128:9300}], reason [RemoteTransportException[[node-2][172.30.0.128:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[node-1][172.30.0.85:9300] general node connection failure]; nested: IllegalStateException[handshake failed with {node-1}{sc-e3s2_Qhm8d7bOBiRQhA}{3p-oZWu9Tie7-AdZ_D75bQ}{172.30.0.85}{172.30.0.85:9300}]; nested: RemoteTransportException[[node-1][172.30.0.85:9300][internal:transport/handshake]]; nested: ElasticsearchException[Illegal parameter in http or transport request found.

[2018-07-02T08:15:17,785][ERROR][c.f.s.t.SearchGuardRequestHandler] ElasticsearchException[Illegal parameter in http or transport request found.

This means that one node is trying to connect to another with

a non-node certificate (no OID or searchguard.nodes_dn incorrect configured) or that someone

is spoofing requests. Check your TLS certificate setup as described here: See http://docs.search-guard.com/latest/troubleshooting-tls]

[2018-07-02T08:15:17,785][WARN ][o.e.d.z.UnicastZenPing ] [node-1] [396] failed send ping to {#zen_unicast_52.40.209.185_0#}{CIfHHpV1RQKwkxmZ0SCtgg}{52.40.209.185}{52.40.209.185:9300}

I am using self-signed certificates on all nodes.

Any idea what caused this? Need help on this.

Thanks in advanced!

5

There is only one place where the SG configuration is stored, the SG index. As with any other Elasticsearch index, it is shared amongst all nodes. It does not matter on which node you make changes, they are reflected throughout the whole cluster. This is the same as if you add or change a document in any index on Elasticsearch: The changes are immediately propagated and you can query any node for the added/changed document.

Having different users on different nodes in a distributed system does not make sense: The cluster keeps a global state. Why would you want to have different configurations on different nodes?

···

On Wednesday, July 4, 2018 at 5:20:28 AM UTC+2, YS .L wrote:

Hi Jochen,

One more question regarding the sgadmin. Could you help to explain this?

I have node1 and node2 both with searchguard installed.

node 1 and node 2 join the cluster.

Let says I have created these:

  • internal user1 in node1
  • internal user2 in node2
    The configuration is done in the file sg_internal_users.yml and initialize with sgadmin in 2 nodes separately (node1 first and follow by node2).

My kibana is pointing to node1. Can I login with user2 (initialized in node2) through kibana?

As my understanding is both nodes share the same searchguard index and I supposed to be able to login user2 in kibana.

Now the case is when i initialize with sgadmin on node1, I cant login user2. And after I initialize on node2, I can login user2 but no longer user1.

Is the sgadmin initialization overwrites the configuration changes on other node every time it runs? If that is the case, what is the best practise?

Is it better the configuration always be done on only one node?

Thanks.

On Monday, 2 July 2018 16:01:24 UTC+8, YS .L wrote:

Hi,

I have this error when initializing Searchguard :

./sgadmin.sh -diagnose -cd /usr/share/elasticsearch/plugins/search-guard-6/sgconfig -icl -key /etc/elasticsearch/kirk.key -keypass dsun9XyU0pL-cert /etc/elasticsearch/kirk.pem -cacert /etc/elasticsearch/root-ca.pem -nhnv

WARNING: JAVA_HOME not set, will use /bin/java

Search Guard Admin v6

Will connect to localhost:9300 … done

Elasticsearch Version: 6.2.4

Search Guard Version: 6.2.4-22.1

Connected as CN=kirk.mysolution.com,OU=Ops,O=mysolution Com, Inc.,DC=mysolution,DC=com

Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-6/tools/sgadmin_diag_trace_2018-Jul-02_07-25-44.txt

Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …

Cannot retrieve cluster state due to: null. This is not an error, will keep on trying …

Root cause: MasterNotDiscoveredException[null] (org.elasticsearch.discovery.MasterNotDiscoveredException/org.elasticsearch.discovery.MasterNotDiscoveredException)

  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
  • Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{1KqhDyc2RQSzFOG508SZlw}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying …

Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{1KqhDyc2RQSzFOG508SZlw}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)

  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
  • Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.

My elasticsearch log:

[2018-07-02T08:15:17,771][ERROR][c.f.s.t.SearchGuardRequestHandler] ElasticsearchException[Illegal parameter in http or transport request found.

This means that one node is trying to connect to another with

a non-node certificate (no OID or searchguard.nodes_dn incorrect configured) or that someone

is spoofing requests. Check your TLS certificate setup as described here: See http://docs.search-guard.com/latest/troubleshooting-tls]

[2018-07-02T08:15:17,773][INFO ][o.e.d.z.ZenDiscovery ] [node-1] failed to send join request to master [{node-2}{JYLpvoLWRhC4jD-uJWMWdw}{e6CcuimNQSm9yrDW0PwACA}{172.30.0.128}{172.30.0.128:9300}], reason [RemoteTransportException[[node-2][172.30.0.128:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[node-1][172.30.0.85:9300] general node connection failure]; nested: IllegalStateException[handshake failed with {node-1}{sc-e3s2_Qhm8d7bOBiRQhA}{3p-oZWu9Tie7-AdZ_D75bQ}{172.30.0.85}{172.30.0.85:9300}]; nested: RemoteTransportException[[node-1][172.30.0.85:9300][internal:transport/handshake]]; nested: ElasticsearchException[Illegal parameter in http or transport request found.

[2018-07-02T08:15:17,785][ERROR][c.f.s.t.SearchGuardRequestHandler] ElasticsearchException[Illegal parameter in http or transport request found.

This means that one node is trying to connect to another with

a non-node certificate (no OID or searchguard.nodes_dn incorrect configured) or that someone

is spoofing requests. Check your TLS certificate setup as described here: See http://docs.search-guard.com/latest/troubleshooting-tls]

[2018-07-02T08:15:17,785][WARN ][o.e.d.z.UnicastZenPing ] [node-1] [396] failed send ping to {#zen_unicast_52.40.209.185_0#}{CIfHHpV1RQKwkxmZ0SCtgg}{52.40.209.185}{52.40.209.185:9300}

I am using self-signed certificates on all nodes.

Any idea what caused this? Need help on this.

Thanks in advanced!

6

Hi Jochen,

I got you. Now I have better understanding on the concept. Thanks!

···

On Thursday, 5 July 2018 03:50:22 UTC+8, Jochen Kressin wrote:

There is only one place where the SG configuration is stored, the SG index. As with any other Elasticsearch index, it is shared amongst all nodes. It does not matter on which node you make changes, they are reflected throughout the whole cluster. This is the same as if you add or change a document in any index on Elasticsearch: The changes are immediately propagated and you can query any node for the added/changed document.

Having different users on different nodes in a distributed system does not make sense: The cluster keeps a global state. Why would you want to have different configurations on different nodes?

On Wednesday, July 4, 2018 at 5:20:28 AM UTC+2, YS .L wrote:

Hi Jochen,

One more question regarding the sgadmin. Could you help to explain this?

I have node1 and node2 both with searchguard installed.

node 1 and node 2 join the cluster.

Let says I have created these:

  • internal user1 in node1
  • internal user2 in node2
    The configuration is done in the file sg_internal_users.yml and initialize with sgadmin in 2 nodes separately (node1 first and follow by node2).

My kibana is pointing to node1. Can I login with user2 (initialized in node2) through kibana?

As my understanding is both nodes share the same searchguard index and I supposed to be able to login user2 in kibana.

Now the case is when i initialize with sgadmin on node1, I cant login user2. And after I initialize on node2, I can login user2 but no longer user1.

Is the sgadmin initialization overwrites the configuration changes on other node every time it runs? If that is the case, what is the best practise?

Is it better the configuration always be done on only one node?

Thanks.

On Monday, 2 July 2018 16:01:24 UTC+8, YS .L wrote:

Hi,

I have this error when initializing Searchguard :

./sgadmin.sh -diagnose -cd /usr/share/elasticsearch/plugins/search-guard-6/sgconfig -icl -key /etc/elasticsearch/kirk.key -keypass dsun9XyU0pL-cert /etc/elasticsearch/kirk.pem -cacert /etc/elasticsearch/root-ca.pem -nhnv

WARNING: JAVA_HOME not set, will use /bin/java

Search Guard Admin v6

Will connect to localhost:9300 … done

Elasticsearch Version: 6.2.4

Search Guard Version: 6.2.4-22.1

Connected as CN=kirk.mysolution.com,OU=Ops,O=mysolution Com, Inc.,DC=mysolution,DC=com

Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-6/tools/sgadmin_diag_trace_2018-Jul-02_07-25-44.txt

Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …

Cannot retrieve cluster state due to: null. This is not an error, will keep on trying …

Root cause: MasterNotDiscoveredException[null] (org.elasticsearch.discovery.MasterNotDiscoveredException/org.elasticsearch.discovery.MasterNotDiscoveredException)

  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
  • Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{1KqhDyc2RQSzFOG508SZlw}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying …

Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{1KqhDyc2RQSzFOG508SZlw}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)

  • Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
  • Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
  • If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
  • Add --accept-red-cluster to allow sgadmin to operate on a red cluster.

My elasticsearch log:

[2018-07-02T08:15:17,771][ERROR][c.f.s.t.SearchGuardRequestHandler] ElasticsearchException[Illegal parameter in http or transport request found.

This means that one node is trying to connect to another with

a non-node certificate (no OID or searchguard.nodes_dn incorrect configured) or that someone

is spoofing requests. Check your TLS certificate setup as described here: See http://docs.search-guard.com/latest/troubleshooting-tls]

[2018-07-02T08:15:17,773][INFO ][o.e.d.z.ZenDiscovery ] [node-1] failed to send join request to master [{node-2}{JYLpvoLWRhC4jD-uJWMWdw}{e6CcuimNQSm9yrDW0PwACA}{172.30.0.128}{172.30.0.128:9300}], reason [RemoteTransportException[[node-2][172.30.0.128:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[node-1][172.30.0.85:9300] general node connection failure]; nested: IllegalStateException[handshake failed with {node-1}{sc-e3s2_Qhm8d7bOBiRQhA}{3p-oZWu9Tie7-AdZ_D75bQ}{172.30.0.85}{172.30.0.85:9300}]; nested: RemoteTransportException[[node-1][172.30.0.85:9300][internal:transport/handshake]]; nested: ElasticsearchException[Illegal parameter in http or transport request found.

[2018-07-02T08:15:17,785][ERROR][c.f.s.t.SearchGuardRequestHandler] ElasticsearchException[Illegal parameter in http or transport request found.

This means that one node is trying to connect to another with

a non-node certificate (no OID or searchguard.nodes_dn incorrect configured) or that someone

is spoofing requests. Check your TLS certificate setup as described here: See http://docs.search-guard.com/latest/troubleshooting-tls]

[2018-07-02T08:15:17,785][WARN ][o.e.d.z.UnicastZenPing ] [node-1] [396] failed send ping to {#zen_unicast_52.40.209.185_0#}{CIfHHpV1RQKwkxmZ0SCtgg}{52.40.209.185}{52.40.209.185:9300}

I am using self-signed certificates on all nodes.

Any idea what caused this? Need help on this.

Thanks in advanced!