Nodes are not joining after enabling searchguard

Search Guard
1

2 master and data nodes are joining in cluster if i disable the search guard. not connecting if i enable search guard.

Elasticsearch Version: 6.3.0

Search Guard Version: 6.3.0.

CentOS 7

java version “1.8.0_171”

Java™ SE Runtime Environment (build 1.8.0_171-b11)

Java HotSpot™ 64-Bit Server VM (build 25.171-b11, mixed mode)

Certificates are generated using Search Guard TLS Tool

Node1 config:

···

cluster.name: cles

node.name: cl-esnode-1

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

network.host: 10.240.0.6

discovery.zen.ping.unicast.hosts: [“31.239.124.150”, “31.238.130.20”]

discovery.zen.minimum_master_nodes: 2

action.destructive_requires_name: true

xpack.security.enabled: false

searchguard.ssl.transport.pemcert_filepath: node1.pem

searchguard.ssl.transport.pemkey_filepath: node1.key

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: false

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

searchguard.enterprise_modules_enabled: false

Node2 config:

cluster.name: cles

node.name: cl-esnode-2

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

network.host: 10.240.0.9

discovery.zen.ping.unicast.hosts: [“31.239.124.150”, “31.238.130.20”]

discovery.zen.minimum_master_nodes: 2

action.destructive_requires_name: true

xpack.security.enabled: false

searchguard.ssl.transport.pemcert_filepath: node2.pem

searchguard.ssl.transport.pemkey_filepath: node2.key

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: false

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

searchguard.enterprise_modules_enabled: false

2

Please attach your Elasticsearch logfiles:

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version

  • Installed and used enterprise modules, if any

  • JVM version and operating system version

  • Search Guard configuration files

  • Elasticsearch log messages on debug level

  • Other installed Elasticsearch or Kibana plugins, if any

···

On Thursday, July 5, 2018 at 9:07:37 AM UTC+2, ganesh@customerlabs.co wrote:

2 master and data nodes are joining in cluster if i disable the search guard. not connecting if i enable search guard.

Elasticsearch Version: 6.3.0

Search Guard Version: 6.3.0.

CentOS 7

java version “1.8.0_171”

Java™ SE Runtime Environment (build 1.8.0_171-b11)

Java HotSpot™ 64-Bit Server VM (build 25.171-b11, mixed mode)

Certificates are generated using Search Guard TLS Tool

Node1 config:

cluster.name: cles

node.name: cl-esnode-1

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

network.host: 10.240.0.6

discovery.zen.ping.unicast.hosts: [“31.239.124.150”, “31.238.130.20”]

discovery.zen.minimum_master_nodes: 2

action.destructive_requires_name: true

xpack.security.enabled: false

searchguard.ssl.transport.pemcert_filepath: node1.pem

searchguard.ssl.transport.pemkey_filepath: node1.key

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: false

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

searchguard.enterprise_modules_enabled: false

Node2 config:

cluster.name: cles

node.name: cl-esnode-2

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

network.host: 10.240.0.9

discovery.zen.ping.unicast.hosts: [“31.239.124.150”, “31.238.130.20”]

discovery.zen.minimum_master_nodes: 2

action.destructive_requires_name: true

xpack.security.enabled: false

searchguard.ssl.transport.pemcert_filepath: node2.pem

searchguard.ssl.transport.pemkey_filepath: node2.key

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: false

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

searchguard.enterprise_modules_enabled: false

3

I can not attach my logs here. Its being deleted automatically. Am i missing something?
thanks

···

On Thursday, July 5, 2018 at 3:06:57 PM UTC+5:30, Jochen Kressin wrote:

Please attach your Elasticsearch logfiles:

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version
  • Installed and used enterprise modules, if any
  • JVM version and operating system version
  • Search Guard configuration files
  • Elasticsearch log messages on debug level
  • Other installed Elasticsearch or Kibana plugins, if any

On Thursday, July 5, 2018 at 9:07:37 AM UTC+2, gan...@customerlabs.co wrote:

2 master and data nodes are joining in cluster if i disable the search guard. not connecting if i enable search guard.

Elasticsearch Version: 6.3.0

Search Guard Version: 6.3.0.

CentOS 7

java version “1.8.0_171”

Java™ SE Runtime Environment (build 1.8.0_171-b11)

Java HotSpot™ 64-Bit Server VM (build 25.171-b11, mixed mode)

Certificates are generated using Search Guard TLS Tool

Node1 config:

cluster.name: cles

node.name: cl-esnode-1

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

network.host: 10.240.0.6

discovery.zen.ping.unicast.hosts: [“31.239.124.150”, “31.238.130.20”]

discovery.zen.minimum_master_nodes: 2

action.destructive_requires_name: true

xpack.security.enabled: false

searchguard.ssl.transport.pemcert_filepath: node1.pem

searchguard.ssl.transport.pemkey_filepath: node1.key

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: false

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

searchguard.enterprise_modules_enabled: false

Node2 config:

cluster.name: cles

node.name: cl-esnode-2

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

network.host: 10.240.0.9

discovery.zen.ping.unicast.hosts: [“31.239.124.150”, “31.238.130.20”]

discovery.zen.minimum_master_nodes: 2

action.destructive_requires_name: true

xpack.security.enabled: false

searchguard.ssl.transport.pemcert_filepath: node2.pem

searchguard.ssl.transport.pemkey_filepath: node2.key

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: false

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

searchguard.enterprise_modules_enabled: false

4

Please find my partial es log

[2018-07-05T10:47:49,676][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin com.floragunn.searchguard.SearchGuardPlugin

[2018-07-05T10:47:50,119][INFO ][o.e.d.DiscoveryModule ] [cl-esnode-1] using discovery type [zen]

[2018-07-05T10:47:51,229][INFO ][c.f.s.SearchGuardPlugin ] 0 Search Guard modules loaded so far:

[2018-07-05T10:47:51,230][INFO ][o.e.n.Node ] [cl-esnode-1] initialized

[2018-07-05T10:47:51,230][INFO ][o.e.n.Node ] [cl-esnode-1] starting …

[2018-07-05T10:47:51,388][INFO ][o.e.t.TransportService ] [cl-esnode-1] publish_address {10.240.0.6:9300}, bound_addresses {10.240.0.6:9300}

[2018-07-05T10:47:51,444][INFO ][o.e.b.BootstrapChecks ] [cl-esnode-1] bound or publishing to a non-loopback address, enforcing bootstrap checks

[2018-07-05T10:47:51,461][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Check if searchguard index exists …

[2018-07-05T10:47:51,469][DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [cl-esnode-1] no known master node, scheduling a retry

[2018-07-05T10:47:54,497][WARN ][o.e.d.z.ZenDiscovery ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:47:57,501][WARN ][o.e.d.z.ZenDiscovery ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:48:00,503][WARN ][o.e.d.z.ZenDiscovery ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:48:03,506][WARN ][o.e.d.z.ZenDiscovery ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:48:05,297][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [cl-esnode-1] SSL Problem Received fatal alert: certificate_unknown

javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

···

On Thursday, July 5, 2018 at 3:06:57 PM UTC+5:30, Jochen Kressin wrote:

Please attach your Elasticsearch logfiles:

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version
  • Installed and used enterprise modules, if any
  • JVM version and operating system version
  • Search Guard configuration files
  • Elasticsearch log messages on debug level
  • Other installed Elasticsearch or Kibana plugins, if any

On Thursday, July 5, 2018 at 9:07:37 AM UTC+2, gan...@customerlabs.co wrote:

2 master and data nodes are joining in cluster if i disable the search guard. not connecting if i enable search guard.

Elasticsearch Version: 6.3.0

Search Guard Version: 6.3.0.

CentOS 7

java version “1.8.0_171”

Java™ SE Runtime Environment (build 1.8.0_171-b11)

Java HotSpot™ 64-Bit Server VM (build 25.171-b11, mixed mode)

Certificates are generated using Search Guard TLS Tool

Node1 config:

cluster.name: cles

node.name: cl-esnode-1

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

network.host: 10.240.0.6

discovery.zen.ping.unicast.hosts: [“31.239.124.150”, “31.238.130.20”]

discovery.zen.minimum_master_nodes: 2

action.destructive_requires_name: true

xpack.security.enabled: false

searchguard.ssl.transport.pemcert_filepath: node1.pem

searchguard.ssl.transport.pemkey_filepath: node1.key

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: false

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

searchguard.enterprise_modules_enabled: false

Node2 config:

cluster.name: cles

node.name: cl-esnode-2

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

network.host: 10.240.0.9

discovery.zen.ping.unicast.hosts: [“31.239.124.150”, “31.238.130.20”]

discovery.zen.minimum_master_nodes: 2

action.destructive_requires_name: true

xpack.security.enabled: false

searchguard.ssl.transport.pemcert_filepath: node2.pem

searchguard.ssl.transport.pemkey_filepath: node2.key

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: false

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

searchguard.enterprise_modules_enabled: false

5

Don’t know about the deleted messages, we never delete anything here. Strange.

You see the TLS error here:

[2018-07-05T10:48:05,297][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [cl-esnode-1] SSL Problem Received fatal alert: certificate_unknown

javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

This means that the certificate could not be validated against the root CA. How were the certificates generated?

If you have problems uploading the complete logs please send them to info@search-guard.com

···

On Thursday, July 5, 2018 at 2:54:22 PM UTC+2, ganesh@customerlabs.co wrote:

Please find my partial es log

[2018-07-05T10:47:49,676][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin com.floragunn.searchguard.SearchGuardPlugin

[2018-07-05T10:47:50,119][INFO ][o.e.d.DiscoveryModule ] [cl-esnode-1] using discovery type [zen]

[2018-07-05T10:47:51,229][INFO ][c.f.s.SearchGuardPlugin ] 0 Search Guard modules loaded so far:

[2018-07-05T10:47:51,230][INFO ][o.e.n.Node ] [cl-esnode-1] initialized

[2018-07-05T10:47:51,230][INFO ][o.e.n.Node ] [cl-esnode-1] starting …

[2018-07-05T10:47:51,388][INFO ][o.e.t.TransportService ] [cl-esnode-1] publish_address {10.240.0.6:9300}, bound_addresses {10.240.0.6:9300}

[2018-07-05T10:47:51,444][INFO ][o.e.b.BootstrapChecks ] [cl-esnode-1] bound or publishing to a non-loopback address, enforcing bootstrap checks

[2018-07-05T10:47:51,461][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Check if searchguard index exists …

[2018-07-05T10:47:51,469][DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [cl-esnode-1] no known master node, scheduling a retry

[2018-07-05T10:47:54,497][WARN ][o.e.d.z.ZenDiscovery ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:47:57,501][WARN ][o.e.d.z.ZenDiscovery ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:48:00,503][WARN ][o.e.d.z.ZenDiscovery ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:48:03,506][WARN ][o.e.d.z.ZenDiscovery ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:48:05,297][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [cl-esnode-1] SSL Problem Received fatal alert: certificate_unknown

javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

On Thursday, July 5, 2018 at 3:06:57 PM UTC+5:30, Jochen Kressin wrote:

Please attach your Elasticsearch logfiles:

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version
  • Installed and used enterprise modules, if any
  • JVM version and operating system version
  • Search Guard configuration files
  • Elasticsearch log messages on debug level
  • Other installed Elasticsearch or Kibana plugins, if any

On Thursday, July 5, 2018 at 9:07:37 AM UTC+2, gan...@customerlabs.co wrote:

2 master and data nodes are joining in cluster if i disable the search guard. not connecting if i enable search guard.

Elasticsearch Version: 6.3.0

Search Guard Version: 6.3.0.

CentOS 7

java version “1.8.0_171”

Java™ SE Runtime Environment (build 1.8.0_171-b11)

Java HotSpot™ 64-Bit Server VM (build 25.171-b11, mixed mode)

Certificates are generated using Search Guard TLS Tool

Node1 config:

cluster.name: cles

node.name: cl-esnode-1

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

network.host: 10.240.0.6

discovery.zen.ping.unicast.hosts: [“31.239.124.150”, “31.238.130.20”]

discovery.zen.minimum_master_nodes: 2

action.destructive_requires_name: true

xpack.security.enabled: false

searchguard.ssl.transport.pemcert_filepath: node1.pem

searchguard.ssl.transport.pemkey_filepath: node1.key

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: false

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

searchguard.enterprise_modules_enabled: false

Node2 config:

cluster.name: cles

node.name: cl-esnode-2

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

network.host: 10.240.0.9

discovery.zen.ping.unicast.hosts: [“31.239.124.150”, “31.238.130.20”]

discovery.zen.minimum_master_nodes: 2

action.destructive_requires_name: true

xpack.security.enabled: false

searchguard.ssl.transport.pemcert_filepath: node2.pem

searchguard.ssl.transport.pemkey_filepath: node2.key

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: false

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

searchguard.enterprise_modules_enabled: false

6

Certificates are generated using sgtlstool
Followed this steps: https://docs.search-guard.com/latest/offline-tls-tool#tls-tool

./sgtlstool.sh -c …/config/tlsconfig.yml -ca -crt

Generated certificates on both nodes using same configuration

------tlsconnfig.yml------

···

Self-generated certificate authority

If you want to create a new certificate authority, you must specify its parameters here.

You can skip this section if you only want to create CSRs

ca:

root:

  dn: CN=root.ca.example.com,OU=CA,O=Example Com\, Inc.,DC=example,DC=com

  keysize: 2048

  pkPassword: changeit

  validityDays: 3650

  file: root-ca.pem

nodes:

clients:

On Thursday, July 5, 2018 at 8:42:54 PM UTC+5:30, Jochen Kressin wrote:

Don’t know about the deleted messages, we never delete anything here. Strange.

You see the TLS error here:

[2018-07-05T10:48:05,297][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [cl-esnode-1] SSL Problem Received fatal alert: certificate_unknown

javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

This means that the certificate could not be validated against the root CA. How were the certificates generated?

If you have problems uploading the complete logs please send them to in...@search-guard.com

On Thursday, July 5, 2018 at 2:54:22 PM UTC+2, gan...@customerlabs.co wrote:

Please find my partial es log

[2018-07-05T10:47:49,676][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin com.floragunn.searchguard.SearchGuardPlugin

[2018-07-05T10:47:50,119][INFO ][o.e.d.DiscoveryModule ] [cl-esnode-1] using discovery type [zen]

[2018-07-05T10:47:51,229][INFO ][c.f.s.SearchGuardPlugin ] 0 Search Guard modules loaded so far:

[2018-07-05T10:47:51,230][INFO ][o.e.n.Node ] [cl-esnode-1] initialized

[2018-07-05T10:47:51,230][INFO ][o.e.n.Node ] [cl-esnode-1] starting …

[2018-07-05T10:47:51,388][INFO ][o.e.t.TransportService ] [cl-esnode-1] publish_address {10.240.0.6:9300}, bound_addresses {10.240.0.6:9300}

[2018-07-05T10:47:51,444][INFO ][o.e.b.BootstrapChecks ] [cl-esnode-1] bound or publishing to a non-loopback address, enforcing bootstrap checks

[2018-07-05T10:47:51,461][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Check if searchguard index exists …

[2018-07-05T10:47:51,469][DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [cl-esnode-1] no known master node, scheduling a retry

[2018-07-05T10:47:54,497][WARN ][o.e.d.z.ZenDiscovery ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:47:57,501][WARN ][o.e.d.z.ZenDiscovery ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:48:00,503][WARN ][o.e.d.z.ZenDiscovery ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:48:03,506][WARN ][o.e.d.z.ZenDiscovery ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:48:05,297][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [cl-esnode-1] SSL Problem Received fatal alert: certificate_unknown

javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

On Thursday, July 5, 2018 at 3:06:57 PM UTC+5:30, Jochen Kressin wrote:

Please attach your Elasticsearch logfiles:

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version
  • Installed and used enterprise modules, if any
  • JVM version and operating system version
  • Search Guard configuration files
  • Elasticsearch log messages on debug level
  • Other installed Elasticsearch or Kibana plugins, if any

On Thursday, July 5, 2018 at 9:07:37 AM UTC+2, gan...@customerlabs.co wrote:

2 master and data nodes are joining in cluster if i disable the search guard. not connecting if i enable search guard.

Elasticsearch Version: 6.3.0

Search Guard Version: 6.3.0.

CentOS 7

java version “1.8.0_171”

Java™ SE Runtime Environment (build 1.8.0_171-b11)

Java HotSpot™ 64-Bit Server VM (build 25.171-b11, mixed mode)

Certificates are generated using Search Guard TLS Tool

Node1 config:

cluster.name: cles

node.name: cl-esnode-1

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

network.host: 10.240.0.6

discovery.zen.ping.unicast.hosts: [“31.239.124.150”, “31.238.130.20”]

discovery.zen.minimum_master_nodes: 2

action.destructive_requires_name: true

xpack.security.enabled: false

searchguard.ssl.transport.pemcert_filepath: node1.pem

searchguard.ssl.transport.pemkey_filepath: node1.key

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: false

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

searchguard.enterprise_modules_enabled: false

Node2 config:

cluster.name: cles

node.name: cl-esnode-2

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

network.host: 10.240.0.9

discovery.zen.ping.unicast.hosts: [“31.239.124.150”, “31.238.130.20”]

discovery.zen.minimum_master_nodes: 2

action.destructive_requires_name: true

xpack.security.enabled: false

searchguard.ssl.transport.pemcert_filepath: node2.pem

searchguard.ssl.transport.pemkey_filepath: node2.key

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: false

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

searchguard.enterprise_modules_enabled: false

7

TLS error only happens if both nodes are running. No errors if i stop either of a node. Certificates are working fine on both the nodes independently but not with

discovery.zen.ping.unicast.hosts: [“31.239.124.150”, “31.238.130.20”]

I have sent a logs to info@search-guard.com as i can’t attach here

Thanks

···

On Thursday, July 5, 2018 at 10:44:04 PM UTC+5:30, gan...@customerlabs.co wrote:

Certificates are generated using sgtlstool
Followed this steps: https://docs.search-guard.com/latest/offline-tls-tool#tls-tool

./sgtlstool.sh -c …/config/tlsconfig.yml -ca -crt

Generated certificates on both nodes using same configuration

------tlsconnfig.yml------

Self-generated certificate authority

If you want to create a new certificate authority, you must specify its parameters here.

You can skip this section if you only want to create CSRs

ca:

root:

  dn: CN=[root.ca.example.com](http://root.ca.example.com),OU=CA,O=Example Com\, Inc.,DC=example,DC=com

  keysize: 2048

  pkPassword: changeit

  validityDays: 3650

  file: root-ca.pem

nodes:

  • name: node1
dn: CN=[node1.example.com](http://node1.example.com),OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

dns: [node1.example.com](http://node1.example.com)

ip: 10.240.0.6
  • name: node2
dn: CN=[node2.example.com](http://node2.example.com),OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

dns: [node2.example.com](http://node2.example.com)

ip: 10.240.0.9

clients:

  • name: spock
dn: CN=[spock.example.com](http://spock.example.com),OU=Ops,O=Example Com\, Inc.,DC=example,DC=com
  • name: kirk
dn: CN=[kirk.example.com](http://kirk.example.com),OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

admin: true

On Thursday, July 5, 2018 at 8:42:54 PM UTC+5:30, Jochen Kressin wrote:

Don’t know about the deleted messages, we never delete anything here. Strange.

You see the TLS error here:

[2018-07-05T10:48:05,297][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [cl-esnode-1] SSL Problem Received fatal alert: certificate_unknown

javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

This means that the certificate could not be validated against the root CA. How were the certificates generated?

If you have problems uploading the complete logs please send them to in...@search-guard.com

On Thursday, July 5, 2018 at 2:54:22 PM UTC+2, gan...@customerlabs.co wrote:

Please find my partial es log

[2018-07-05T10:47:49,676][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin com.floragunn.searchguard.SearchGuardPlugin

[2018-07-05T10:47:50,119][INFO ][o.e.d.DiscoveryModule ] [cl-esnode-1] using discovery type [zen]

[2018-07-05T10:47:51,229][INFO ][c.f.s.SearchGuardPlugin ] 0 Search Guard modules loaded so far:

[2018-07-05T10:47:51,230][INFO ][o.e.n.Node ] [cl-esnode-1] initialized

[2018-07-05T10:47:51,230][INFO ][o.e.n.Node ] [cl-esnode-1] starting …

[2018-07-05T10:47:51,388][INFO ][o.e.t.TransportService ] [cl-esnode-1] publish_address {10.240.0.6:9300}, bound_addresses {10.240.0.6:9300}

[2018-07-05T10:47:51,444][INFO ][o.e.b.BootstrapChecks ] [cl-esnode-1] bound or publishing to a non-loopback address, enforcing bootstrap checks

[2018-07-05T10:47:51,461][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Check if searchguard index exists …

[2018-07-05T10:47:51,469][DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [cl-esnode-1] no known master node, scheduling a retry

[2018-07-05T10:47:54,497][WARN ][o.e.d.z.ZenDiscovery ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:47:57,501][WARN ][o.e.d.z.ZenDiscovery ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:48:00,503][WARN ][o.e.d.z.ZenDiscovery ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:48:03,506][WARN ][o.e.d.z.ZenDiscovery ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:48:05,297][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [cl-esnode-1] SSL Problem Received fatal alert: certificate_unknown

javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

On Thursday, July 5, 2018 at 3:06:57 PM UTC+5:30, Jochen Kressin wrote:

Please attach your Elasticsearch logfiles:

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version
  • Installed and used enterprise modules, if any
  • JVM version and operating system version
  • Search Guard configuration files
  • Elasticsearch log messages on debug level
  • Other installed Elasticsearch or Kibana plugins, if any

On Thursday, July 5, 2018 at 9:07:37 AM UTC+2, gan...@customerlabs.co wrote:

2 master and data nodes are joining in cluster if i disable the search guard. not connecting if i enable search guard.

Elasticsearch Version: 6.3.0

Search Guard Version: 6.3.0.

CentOS 7

java version “1.8.0_171”

Java™ SE Runtime Environment (build 1.8.0_171-b11)

Java HotSpot™ 64-Bit Server VM (build 25.171-b11, mixed mode)

Certificates are generated using Search Guard TLS Tool

Node1 config:

cluster.name: cles

node.name: cl-esnode-1

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

network.host: 10.240.0.6

discovery.zen.ping.unicast.hosts: [“31.239.124.150”, “31.238.130.20”]

discovery.zen.minimum_master_nodes: 2

action.destructive_requires_name: true

xpack.security.enabled: false

searchguard.ssl.transport.pemcert_filepath: node1.pem

searchguard.ssl.transport.pemkey_filepath: node1.key

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: false

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

searchguard.enterprise_modules_enabled: false

Node2 config:

cluster.name: cles

node.name: cl-esnode-2

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

network.host: 10.240.0.9

discovery.zen.ping.unicast.hosts: [“31.239.124.150”, “31.238.130.20”]

discovery.zen.minimum_master_nodes: 2

action.destructive_requires_name: true

xpack.security.enabled: false

searchguard.ssl.transport.pemcert_filepath: node2.pem

searchguard.ssl.transport.pemkey_filepath: node2.key

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: false

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

searchguard.enterprise_modules_enabled: false

8

hello
i have a problem , please help me

Plugin [search-guard-6] was built for Elasticsearch version 6.2.4 but version 6.3.0 is running

1357×537 19.7 KB

···

On Fri, Jul 6, 2018 at 11:21 AM, ganesh@customerlabs.co wrote:

TLS error only happens if both nodes are running. No errors if i stop either of a node. Certificates are working fine on both the nodes independently but not with

discovery.zen.ping.unicast.hosts: [“31.239.124.150”, “31.238.130.20”]

I have sent a logs to info@search-guard.com as i can’t attach here

Thanks

On Thursday, July 5, 2018 at 10:44:04 PM UTC+5:30, gan...@customerlabs.co wrote:

Certificates are generated using sgtlstool
Followed this steps: https://docs.search-guard.com/latest/offline-tls-tool#tls-tool

./sgtlstool.sh -c …/config/tlsconfig.yml -ca -crt

Generated certificates on both nodes using same configuration

------tlsconnfig.yml------

Self-generated certificate authority

If you want to create a new certificate authority, you must specify its parameters here.

You can skip this section if you only want to create CSRs

ca:

root:

  dn: CN=[root.ca.example.com](http://root.ca.example.com),OU=CA,O=Example Com\, Inc.,DC=example,DC=com

  keysize: 2048

  pkPassword: changeit

  validityDays: 3650

  file: root-ca.pem

nodes:

  • name: node1
dn: CN=[node1.example.com](http://node1.example.com),OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

dns: [node1.example.com](http://node1.example.com)

ip: 10.240.0.6
  • name: node2
dn: CN=[node2.example.com](http://node2.example.com),OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

dns: [node2.example.com](http://node2.example.com)

ip: 10.240.0.9

clients:

  • name: spock
dn: CN=[spock.example.com](http://spock.example.com),OU=Ops,O=Example Com\, Inc.,DC=example,DC=com
  • name: kirk
dn: CN=[kirk.example.com](http://kirk.example.com),OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

admin: true

On Thursday, July 5, 2018 at 8:42:54 PM UTC+5:30, Jochen Kressin wrote:

Don’t know about the deleted messages, we never delete anything here. Strange.

You see the TLS error here:

[2018-07-05T10:48:05,297][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [cl-esnode-1] SSL Problem Received fatal alert: certificate_unknown

javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

This means that the certificate could not be validated against the root CA. How were the certificates generated?

If you have problems uploading the complete logs please send them to in...@search-guard.com

On Thursday, July 5, 2018 at 2:54:22 PM UTC+2, gan...@customerlabs.co wrote:

Please find my partial es log

[2018-07-05T10:47:49,676][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin com.floragunn.searchguard.SearchGuardPlugin

[2018-07-05T10:47:50,119][INFO ][o.e.d.DiscoveryModule ] [cl-esnode-1] using discovery type [zen]

[2018-07-05T10:47:51,229][INFO ][c.f.s.SearchGuardPlugin ] 0 Search Guard modules loaded so far:

[2018-07-05T10:47:51,230][INFO ][o.e.n.Node ] [cl-esnode-1] initialized

[2018-07-05T10:47:51,230][INFO ][o.e.n.Node ] [cl-esnode-1] starting …

[2018-07-05T10:47:51,388][INFO ][o.e.t.TransportService ] [cl-esnode-1] publish_address {10.240.0.6:9300}, bound_addresses {10.240.0.6:9300}

[2018-07-05T10:47:51,444][INFO ][o.e.b.BootstrapChecks ] [cl-esnode-1] bound or publishing to a non-loopback address, enforcing bootstrap checks

[2018-07-05T10:47:51,461][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Check if searchguard index exists …

[2018-07-05T10:47:51,469][DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [cl-esnode-1] no known master node, scheduling a retry

[2018-07-05T10:47:54,497][WARN ][o.e.d.z.ZenDiscovery ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:47:57,501][WARN ][o.e.d.z.ZenDiscovery ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:48:00,503][WARN ][o.e.d.z.ZenDiscovery ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:48:03,506][WARN ][o.e.d.z.ZenDiscovery ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:48:05,297][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [cl-esnode-1] SSL Problem Received fatal alert: certificate_unknown

javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

On Thursday, July 5, 2018 at 3:06:57 PM UTC+5:30, Jochen Kressin wrote:

Please attach your Elasticsearch logfiles:

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version
  • Installed and used enterprise modules, if any
  • JVM version and operating system version
  • Search Guard configuration files
  • Elasticsearch log messages on debug level
  • Other installed Elasticsearch or Kibana plugins, if any

On Thursday, July 5, 2018 at 9:07:37 AM UTC+2, gan...@customerlabs.co wrote:

2 master and data nodes are joining in cluster if i disable the search guard. not connecting if i enable search guard.

Elasticsearch Version: 6.3.0

Search Guard Version: 6.3.0.

CentOS 7

java version “1.8.0_171”

Java™ SE Runtime Environment (build 1.8.0_171-b11)

Java HotSpot™ 64-Bit Server VM (build 25.171-b11, mixed mode)

Certificates are generated using Search Guard TLS Tool

Node1 config:

cluster.name: cles

node.name: cl-esnode-1

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

network.host: 10.240.0.6

discovery.zen.ping.unicast.hosts: [“31.239.124.150”, “31.238.130.20”]

discovery.zen.minimum_master_nodes: 2

action.destructive_requires_name: true

xpack.security.enabled: false

searchguard.ssl.transport.pemcert_filepath: node1.pem

searchguard.ssl.transport.pemkey_filepath: node1.key

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: false

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

searchguard.enterprise_modules_enabled: false

Node2 config:

cluster.name: cles

node.name: cl-esnode-2

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

network.host: 10.240.0.9

discovery.zen.ping.unicast.hosts: [“31.239.124.150”, “31.238.130.20”]

discovery.zen.minimum_master_nodes: 2

action.destructive_requires_name: true

xpack.security.enabled: false

searchguard.ssl.transport.pemcert_filepath: node2.pem

searchguard.ssl.transport.pemkey_filepath: node2.key

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: false

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

searchguard.enterprise_modules_enabled: false

–

You received this message because you are subscribed to the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/85aac7b7-ef8b-4af8-b048-fff6ecc79b1d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

9

Please do not hijack threads. This thread is about TLS errors.

I think the error message is quite clear. You have installed Search Guard 6.2.4 on Elasticsearch 6.3.0. Please read the documentation for installation instructions:

“Replace the version number in the examples above with the exact version number that matches your Elasticsearch installation. A plugin built for Elasticsearch 6.3.0 will not run on Elasticsearch 6.2.4 and vice versa.”

···

On Friday, July 6, 2018 at 12:43:09 PM UTC+2, Oumeyma JELLALI wrote:

hello
i have a problem , please help me

Plugin [search-guard-6] was built for Elasticsearch version 6.2.4 but version 6.3.0 is running

image.png1357×537 19.7 KB

On Fri, Jul 6, 2018 at 11:21 AM, ganesh@customerlabs.co wrote:

TLS error only happens if both nodes are running. No errors if i stop either of a node. Certificates are working fine on both the nodes independently but not with

discovery.zen.ping.unicast.hosts: [“31.239.124.150”, “31.238.130.20”]

I have sent a logs to info@search-guard.com as i can’t attach here

Thanks

On Thursday, July 5, 2018 at 10:44:04 PM UTC+5:30, gan...@customerlabs.co wrote:

Certificates are generated using sgtlstool
Followed this steps: https://docs.search-guard.com/latest/offline-tls-tool#tls-tool

./sgtlstool.sh -c …/config/tlsconfig.yml -ca -crt

Generated certificates on both nodes using same configuration

------tlsconnfig.yml------

Self-generated certificate authority

If you want to create a new certificate authority, you must specify its parameters here.

You can skip this section if you only want to create CSRs

ca:

root:

  dn: CN=[root.ca.example.com](http://root.ca.example.com),OU=CA,O=Example Com\, Inc.,DC=example,DC=com

  keysize: 2048

  pkPassword: changeit

  validityDays: 3650

  file: root-ca.pem

nodes:

  • name: node1
dn: CN=[node1.example.com](http://node1.example.com),OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

dns: [node1.example.com](http://node1.example.com)

ip: 10.240.0.6
  • name: node2
dn: CN=[node2.example.com](http://node2.example.com),OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

dns: [node2.example.com](http://node2.example.com)

ip: 10.240.0.9

clients:

  • name: spock
dn: CN=[spock.example.com](http://spock.example.com),OU=Ops,O=Example Com\, Inc.,DC=example,DC=com
  • name: kirk
dn: CN=[kirk.example.com](http://kirk.example.com),OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

admin: true

On Thursday, July 5, 2018 at 8:42:54 PM UTC+5:30, Jochen Kressin wrote:

Don’t know about the deleted messages, we never delete anything here. Strange.

You see the TLS error here:

[2018-07-05T10:48:05,297][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [cl-esnode-1] SSL Problem Received fatal alert: certificate_unknown

javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

This means that the certificate could not be validated against the root CA. How were the certificates generated?

If you have problems uploading the complete logs please send them to in...@search-guard.com

On Thursday, July 5, 2018 at 2:54:22 PM UTC+2, gan...@customerlabs.co wrote:

Please find my partial es log

[2018-07-05T10:47:49,676][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin com.floragunn.searchguard.SearchGuardPlugin

[2018-07-05T10:47:50,119][INFO ][o.e.d.DiscoveryModule ] [cl-esnode-1] using discovery type [zen]

[2018-07-05T10:47:51,229][INFO ][c.f.s.SearchGuardPlugin ] 0 Search Guard modules loaded so far:

[2018-07-05T10:47:51,230][INFO ][o.e.n.Node ] [cl-esnode-1] initialized

[2018-07-05T10:47:51,230][INFO ][o.e.n.Node ] [cl-esnode-1] starting …

[2018-07-05T10:47:51,388][INFO ][o.e.t.TransportService ] [cl-esnode-1] publish_address {10.240.0.6:9300}, bound_addresses {10.240.0.6:9300}

[2018-07-05T10:47:51,444][INFO ][o.e.b.BootstrapChecks ] [cl-esnode-1] bound or publishing to a non-loopback address, enforcing bootstrap checks

[2018-07-05T10:47:51,461][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Check if searchguard index exists …

[2018-07-05T10:47:51,469][DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [cl-esnode-1] no known master node, scheduling a retry

[2018-07-05T10:47:54,497][WARN ][o.e.d.z.ZenDiscovery ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:47:57,501][WARN ][o.e.d.z.ZenDiscovery ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:48:00,503][WARN ][o.e.d.z.ZenDiscovery ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:48:03,506][WARN ][o.e.d.z.ZenDiscovery ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:48:05,297][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [cl-esnode-1] SSL Problem Received fatal alert: certificate_unknown

javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

On Thursday, July 5, 2018 at 3:06:57 PM UTC+5:30, Jochen Kressin wrote:

Please attach your Elasticsearch logfiles:

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version
  • Installed and used enterprise modules, if any
  • JVM version and operating system version
  • Search Guard configuration files
  • Elasticsearch log messages on debug level
  • Other installed Elasticsearch or Kibana plugins, if any

On Thursday, July 5, 2018 at 9:07:37 AM UTC+2, gan...@customerlabs.co wrote:

2 master and data nodes are joining in cluster if i disable the search guard. not connecting if i enable search guard.

Elasticsearch Version: 6.3.0

Search Guard Version: 6.3.0.

CentOS 7

java version “1.8.0_171”

Java™ SE Runtime Environment (build 1.8.0_171-b11)

Java HotSpot™ 64-Bit Server VM (build 25.171-b11, mixed mode)

Certificates are generated using Search Guard TLS Tool

Node1 config:

cluster.name: cles

node.name: cl-esnode-1

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

network.host: 10.240.0.6

discovery.zen.ping.unicast.hosts: [“31.239.124.150”, “31.238.130.20”]

discovery.zen.minimum_master_nodes: 2

action.destructive_requires_name: true

xpack.security.enabled: false

searchguard.ssl.transport.pemcert_filepath: node1.pem

searchguard.ssl.transport.pemkey_filepath: node1.key

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: false

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

searchguard.enterprise_modules_enabled: false

Node2 config:

cluster.name: cles

node.name: cl-esnode-2

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

network.host: 10.240.0.9

discovery.zen.ping.unicast.hosts: [“31.239.124.150”, “31.238.130.20”]

discovery.zen.minimum_master_nodes: 2

action.destructive_requires_name: true

xpack.security.enabled: false

searchguard.ssl.transport.pemcert_filepath: node2.pem

searchguard.ssl.transport.pemkey_filepath: node2.key

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: false

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

searchguard.enterprise_modules_enabled: false

–

You received this message because you are subscribed to the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/85aac7b7-ef8b-4af8-b048-fff6ecc79b1d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

10

Hello Team,

Any finds on my issue please?

I think this is my case updated in the docs

https://docs.search-guard.com/latest/troubleshooting-tls#checking-the-ip-addresses-of-the-certificate

but i don’t find any solution to fix this.

This issue holds my cluster moving in to production. Help on this would really much appreciated.

Thanks

···

On Friday, July 6, 2018 at 2:51:34 PM UTC+5:30, gan...@customerlabs.co wrote:

TLS error only happens if both nodes are running. No errors if i stop either of a node. Certificates are working fine on both the nodes independently but not with

discovery.zen.ping.unicast.hosts: [“31.239.124.150”, “31.238.130.20”]

I have sent a logs to info@search-guard.com as i can’t attach here

Thanks

On Thursday, July 5, 2018 at 10:44:04 PM UTC+5:30, gan...@customerlabs.co wrote:

Certificates are generated using sgtlstool
Followed this steps: https://docs.search-guard.com/latest/offline-tls-tool#tls-tool

./sgtlstool.sh -c …/config/tlsconfig.yml -ca -crt

Generated certificates on both nodes using same configuration

------tlsconnfig.yml------

Self-generated certificate authority

If you want to create a new certificate authority, you must specify its parameters here.

You can skip this section if you only want to create CSRs

ca:

root:

  dn: CN=[root.ca.example.com](http://root.ca.example.com),OU=CA,O=Example Com\, Inc.,DC=example,DC=com

  keysize: 2048

  pkPassword: changeit

  validityDays: 3650

  file: root-ca.pem

nodes:

  • name: node1
dn: CN=[node1.example.com](http://node1.example.com),OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

dns: [node1.example.com](http://node1.example.com)

ip: 10.240.0.6
  • name: node2
dn: CN=[node2.example.com](http://node2.example.com),OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

dns: [node2.example.com](http://node2.example.com)

ip: 10.240.0.9

clients:

  • name: spock
dn: CN=[spock.example.com](http://spock.example.com),OU=Ops,O=Example Com\, Inc.,DC=example,DC=com
  • name: kirk
dn: CN=[kirk.example.com](http://kirk.example.com),OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

admin: true

On Thursday, July 5, 2018 at 8:42:54 PM UTC+5:30, Jochen Kressin wrote:

Don’t know about the deleted messages, we never delete anything here. Strange.

You see the TLS error here:

[2018-07-05T10:48:05,297][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [cl-esnode-1] SSL Problem Received fatal alert: certificate_unknown

javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

This means that the certificate could not be validated against the root CA. How were the certificates generated?

If you have problems uploading the complete logs please send them to in...@search-guard.com

On Thursday, July 5, 2018 at 2:54:22 PM UTC+2, gan...@customerlabs.co wrote:

Please find my partial es log

[2018-07-05T10:47:49,676][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin com.floragunn.searchguard.SearchGuardPlugin

[2018-07-05T10:47:50,119][INFO ][o.e.d.DiscoveryModule ] [cl-esnode-1] using discovery type [zen]

[2018-07-05T10:47:51,229][INFO ][c.f.s.SearchGuardPlugin ] 0 Search Guard modules loaded so far:

[2018-07-05T10:47:51,230][INFO ][o.e.n.Node ] [cl-esnode-1] initialized

[2018-07-05T10:47:51,230][INFO ][o.e.n.Node ] [cl-esnode-1] starting …

[2018-07-05T10:47:51,388][INFO ][o.e.t.TransportService ] [cl-esnode-1] publish_address {10.240.0.6:9300}, bound_addresses {10.240.0.6:9300}

[2018-07-05T10:47:51,444][INFO ][o.e.b.BootstrapChecks ] [cl-esnode-1] bound or publishing to a non-loopback address, enforcing bootstrap checks

[2018-07-05T10:47:51,461][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Check if searchguard index exists …

[2018-07-05T10:47:51,469][DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [cl-esnode-1] no known master node, scheduling a retry

[2018-07-05T10:47:54,497][WARN ][o.e.d.z.ZenDiscovery ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:47:57,501][WARN ][o.e.d.z.ZenDiscovery ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:48:00,503][WARN ][o.e.d.z.ZenDiscovery ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:48:03,506][WARN ][o.e.d.z.ZenDiscovery ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:48:05,297][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [cl-esnode-1] SSL Problem Received fatal alert: certificate_unknown

javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

On Thursday, July 5, 2018 at 3:06:57 PM UTC+5:30, Jochen Kressin wrote:

Please attach your Elasticsearch logfiles:

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version
  • Installed and used enterprise modules, if any
  • JVM version and operating system version
  • Search Guard configuration files
  • Elasticsearch log messages on debug level
  • Other installed Elasticsearch or Kibana plugins, if any

On Thursday, July 5, 2018 at 9:07:37 AM UTC+2, gan...@customerlabs.co wrote:

2 master and data nodes are joining in cluster if i disable the search guard. not connecting if i enable search guard.

Elasticsearch Version: 6.3.0

Search Guard Version: 6.3.0.

CentOS 7

java version “1.8.0_171”

Java™ SE Runtime Environment (build 1.8.0_171-b11)

Java HotSpot™ 64-Bit Server VM (build 25.171-b11, mixed mode)

Certificates are generated using Search Guard TLS Tool

Node1 config:

cluster.name: cles

node.name: cl-esnode-1

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

network.host: 10.240.0.6

discovery.zen.ping.unicast.hosts: [“31.239.124.150”, “31.238.130.20”]

discovery.zen.minimum_master_nodes: 2

action.destructive_requires_name: true

xpack.security.enabled: false

searchguard.ssl.transport.pemcert_filepath: node1.pem

searchguard.ssl.transport.pemkey_filepath: node1.key

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: false

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

searchguard.enterprise_modules_enabled: false

Node2 config:

cluster.name: cles

node.name: cl-esnode-2

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

network.host: 10.240.0.9

discovery.zen.ping.unicast.hosts: [“31.239.124.150”, “31.238.130.20”]

discovery.zen.minimum_master_nodes: 2

action.destructive_requires_name: true

xpack.security.enabled: false

searchguard.ssl.transport.pemcert_filepath: node2.pem

searchguard.ssl.transport.pemkey_filepath: node2.key

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: false

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

searchguard.enterprise_modules_enabled: false