Node clients with Letsencrypt certs isn't working.. Any help?

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version

  • Installed and used enterprise modules, if any

  • JVM version and operating system version

  • Search Guard configuration files

  • Elasticsearch log messages on debug level

  • Other installed Elasticsearch or Kibana plugins, if any

ES version - 5.6.3

Kibana version - 5.6.3

Searchguard version - 5.6.3

Java Version - 1.8.0_152

I have a cert (elastic.example.com) that has alias of all my NODE hostname (es1.example.com, es2.example.com, es3.example.com).

here is the elasticsearch.yaml file I set the OID found in elastic.example.com cert on all 3 nodes elasticsearch yaml file.

bootstrap.memory_lock: false

cluster.name: ishlab-logging

discovery.zen.minimum_master_nodes: 2

discovery.zen.ping.unicast.hosts:

network.host: x.x.x.x

node.max_local_storage_nodes: 3

node.name: es1-es-01

path.data: “/var/lib/elasticsearch/es-01”

path.logs: “/var/log/elasticsearch/es-01”

searchguard.cert.oid: 1.3.6.1.4.1.44947.1.1.1

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/fullchain.pem”

searchguard.ssl.http.pemkey_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/private.pkcs8”

searchguard.ssl.http.pemtrustedcas_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/fullchain.pem”

searchguard.ssl.transport.enforce_hostname_verification: true

searchguard.ssl.transport.pemcert_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/fullchain.pem”

searchguard.ssl.transport.pemkey_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/private.pkcs8”

searchguard.ssl.transport.pemtrustedcas_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/fullchain.pem”

thread_pool.bulk.queue_size: 2000

xpack.graph.enabled: false

xpack.ml.enabled: false

xpack.monitoring.enabled: true

xpack.monitoring.exporters.id1.auth.password: kibanaserver

xpack.monitoring.exporters.id1.auth.username: kibanaserver

xpack.monitoring.exporters.id1.host: https://ish.example.com:9200

xpack.monitoring.exporters.id1.type: http

xpack.monitoring.history.duration: 1d

xpack.security.enabled: false

xpack.watcher.enabled: false

I’m getting the following error on each node… Nodes can’t ping each other. (no OID or searchguard.nodes_dn incorrect configured).

[2018-02-09T14:24:15,219][WARN ][o.e.d.z.UnicastZenPing ] [es1-es-01] [8] failed send ping to {#zen_unicast_es1.example.com_0#}{tEuZDtj8RvC_yLILyR-ysA}{es1.example.com}{x.x.x.x:9300}

java.lang.IllegalStateException: handshake failed with {#zen_unicast_es1.example.com_0#}{tEuZDtj8RvC_yLILyR-ysA}{es1.example.com}{x.x.x.x:9300}

at org.elasticsearch.transport.TransportService.handshake(TransportService.java:403) ~[elasticsearch-5.6.3.jar:5.6.3]

at org.elasticsearch.transport.TransportService.handshake(TransportService.java:370) ~[elasticsearch-5.6.3.jar:5.6.3]

at org.elasticsearch.discovery.zen.UnicastZenPing$PingingRound.getOrConnect(UnicastZenPing.java:400) ~[elasticsearch-5.6.3.jar:5.6.3]

at org.elasticsearch.discovery.zen.UnicastZenPing$3.doRun(UnicastZenPing.java:507) [elasticsearch-5.6.3.jar:5.6.3]

at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.6.3.jar:5.6.3]

at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.3.jar:5.6.3]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]

Caused by: org.elasticsearch.transport.RemoteTransportException: [es1-es-01][x.x.x.x:9300][internal:transport/handshake]

Caused by: org.elasticsearch.ElasticsearchException: bad header found. This means typically that one node try to connect to another with a non-node certificate (no OID or searchguard.nodes_dn incorrect configured) or that someoneis spoofing requests. See https://github.com/floragunncom/search-guard-docs/blob/master/tls_node_certificates.md

Any help on how to configure this?

What is the exact content of the SAN section of your elastic.example.com certificate? Can you post it here?

···

On Friday, February 9, 2018 at 3:50:48 PM UTC+1, No Reply wrote:

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version
  • Installed and used enterprise modules, if any
  • JVM version and operating system version
  • Search Guard configuration files
  • Elasticsearch log messages on debug level
  • Other installed Elasticsearch or Kibana plugins, if any

ES version - 5.6.3

Kibana version - 5.6.3

Searchguard version - 5.6.3

Java Version - 1.8.0_152

I have a cert (elastic.example.com) that has alias of all my NODE hostname (es1.example.com, es2.example.com, es3.example.com).

here is the elasticsearch.yaml file I set the OID found in elastic.example.com cert on all 3 nodes elasticsearch yaml file.

bootstrap.memory_lock: false

cluster.name: ishlab-logging

discovery.zen.minimum_master_nodes: 2

discovery.zen.ping.unicast.hosts:

network.host: x.x.x.x

node.max_local_storage_nodes: 3

node.name: es1-es-01

path.data: “/var/lib/elasticsearch/es-01”

path.logs: “/var/log/elasticsearch/es-01”

searchguard.cert.oid: 1.3.6.1.4.1.44947.1.1.1

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/fullchain.pem

searchguard.ssl.http.pemkey_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/private.pkcs8

searchguard.ssl.http.pemtrustedcas_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/fullchain.pem

searchguard.ssl.transport.enforce_hostname_verification: true

searchguard.ssl.transport.pemcert_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/fullchain.pem

searchguard.ssl.transport.pemkey_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/private.pkcs8

searchguard.ssl.transport.pemtrustedcas_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/fullchain.pem

thread_pool.bulk.queue_size: 2000

xpack.graph.enabled: false

xpack.ml.enabled: false

xpack.monitoring.enabled: true

xpack.monitoring.exporters.id1.auth.password: kibanaserver

xpack.monitoring.exporters.id1.auth.username: kibanaserver

xpack.monitoring.exporters.id1.host: https://ish.example.com:9200

xpack.monitoring.exporters.id1.type: http

xpack.monitoring.history.duration: 1d

xpack.security.enabled: false

xpack.watcher.enabled: false

I’m getting the following error on each node… Nodes can’t ping each other. (no OID or searchguard.nodes_dn incorrect configured).

[2018-02-09T14:24:15,219][WARN ][o.e.d.z.UnicastZenPing ] [es1-es-01] [8] failed send ping to {#zen_unicast_es1.example.com_0#}{tEuZDtj8RvC_yLILyR-ysA}{es1.example.com}{x.x.x.x:9300}

java.lang.IllegalStateException: handshake failed with {#zen_unicast_es1.example.com_0#}{tEuZDtj8RvC_yLILyR-ysA}{es1.example.com}{x.x.x.x:9300}

at org.elasticsearch.transport.TransportService.handshake(TransportService.java:403) ~[elasticsearch-5.6.3.jar:5.6.3]

at org.elasticsearch.transport.TransportService.handshake(TransportService.java:370) ~[elasticsearch-5.6.3.jar:5.6.3]

at org.elasticsearch.discovery.zen.UnicastZenPing$PingingRound.getOrConnect(UnicastZenPing.java:400) ~[elasticsearch-5.6.3.jar:5.6.3]

at org.elasticsearch.discovery.zen.UnicastZenPing$3.doRun(UnicastZenPing.java:507) [elasticsearch-5.6.3.jar:5.6.3]

at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.6.3.jar:5.6.3]

at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.3.jar:5.6.3]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]

Caused by: org.elasticsearch.transport.RemoteTransportException: [es1-es-01][x.x.x.x:9300][internal:transport/handshake]

Caused by: org.elasticsearch.ElasticsearchException: bad header found. This means typically that one node try to connect to another with a non-node certificate (no OID or searchguard.nodes_dn incorrect configured) or that someoneis spoofing requests. See https://github.com/floragunncom/search-guard-docs/blob/master/tls_node_certificates.md

Any help on how to configure this?

Here are the SAN section

X509v3 Subject Alternative Name:

DNS:elastic.example.com, DNS:es1.example.com, DNS:es2.example.com, DNS:es3.example.com

···

On Friday, February 9, 2018 at 8:50:48 AM UTC-6, No Reply wrote:

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version
  • Installed and used enterprise modules, if any
  • JVM version and operating system version
  • Search Guard configuration files
  • Elasticsearch log messages on debug level
  • Other installed Elasticsearch or Kibana plugins, if any

ES version - 5.6.3

Kibana version - 5.6.3

Searchguard version - 5.6.3

Java Version - 1.8.0_152

I have a cert (elastic.example.com) that has alias of all my NODE hostname (es1.example.com, es2.example.com, es3.example.com).

here is the elasticsearch.yaml file I set the OID found in elastic.example.com cert on all 3 nodes elasticsearch yaml file.

bootstrap.memory_lock: false

cluster.name: ishlab-logging

discovery.zen.minimum_master_nodes: 2

discovery.zen.ping.unicast.hosts:

network.host: x.x.x.x

node.max_local_storage_nodes: 3

node.name: es1-es-01

path.data: “/var/lib/elasticsearch/es-01”

path.logs: “/var/log/elasticsearch/es-01”

searchguard.cert.oid: 1.3.6.1.4.1.44947.1.1.1

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/fullchain.pem

searchguard.ssl.http.pemkey_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/private.pkcs8

searchguard.ssl.http.pemtrustedcas_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/fullchain.pem

searchguard.ssl.transport.enforce_hostname_verification: true

searchguard.ssl.transport.pemcert_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/fullchain.pem

searchguard.ssl.transport.pemkey_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/private.pkcs8

searchguard.ssl.transport.pemtrustedcas_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/fullchain.pem

thread_pool.bulk.queue_size: 2000

xpack.graph.enabled: false

xpack.ml.enabled: false

xpack.monitoring.enabled: true

xpack.monitoring.exporters.id1.auth.password: kibanaserver

xpack.monitoring.exporters.id1.auth.username: kibanaserver

xpack.monitoring.exporters.id1.host: https://ish.example.com:9200

xpack.monitoring.exporters.id1.type: http

xpack.monitoring.history.duration: 1d

xpack.security.enabled: false

xpack.watcher.enabled: false

I’m getting the following error on each node… Nodes can’t ping each other. (no OID or searchguard.nodes_dn incorrect configured).

[2018-02-09T14:24:15,219][WARN ][o.e.d.z.UnicastZenPing ] [es1-es-01] [8] failed send ping to {#zen_unicast_es1.example.com_0#}{tEuZDtj8RvC_yLILyR-ysA}{es1.example.com}{x.x.x.x:9300}

java.lang.IllegalStateException: handshake failed with {#zen_unicast_es1.example.com_0#}{tEuZDtj8RvC_yLILyR-ysA}{es1.example.com}{x.x.x.x:9300}

at org.elasticsearch.transport.TransportService.handshake(TransportService.java:403) ~[elasticsearch-5.6.3.jar:5.6.3]

at org.elasticsearch.transport.TransportService.handshake(TransportService.java:370) ~[elasticsearch-5.6.3.jar:5.6.3]

at org.elasticsearch.discovery.zen.UnicastZenPing$PingingRound.getOrConnect(UnicastZenPing.java:400) ~[elasticsearch-5.6.3.jar:5.6.3]

at org.elasticsearch.discovery.zen.UnicastZenPing$3.doRun(UnicastZenPing.java:507) [elasticsearch-5.6.3.jar:5.6.3]

at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.6.3.jar:5.6.3]

at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.3.jar:5.6.3]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]

Caused by: org.elasticsearch.transport.RemoteTransportException: [es1-es-01][x.x.x.x:9300][internal:transport/handshake]

Caused by: org.elasticsearch.ElasticsearchException: bad header found. This means typically that one node try to connect to another with a non-node certificate (no OID or searchguard.nodes_dn incorrect configured) or that someoneis spoofing requests. See https://github.com/floragunncom/search-guard-docs/blob/master/tls_node_certificates.md

Any help on how to configure this?

It seems that your certificate does not contain any OID. Usually you would see something like:

SubjectAlternativeName [

DNSName: es1.example.com

DNSName: es2.example.com

OIDName: 1.2.3.4.5.5

]

Depending on the tool your are using the OID can also be called RID. But it seems the entry is missing. Where did you get the OID value “1.3.6.1.4.1.44947.1.1.1” from? Does not look like an OID to me.

As an alternative to using an OID, you can also just list the DN of your node certificate in elasticsearch.yml like:

searchguard.nodes_dn:

For finding out the actual DN of your certificate you can use tools like OpenSSL for example. Wildcards are supported here.

Another thing, it seems you configured the full certificate chain for both the pemcert_filepath and pemtrustedcas_filepath. While this may work usually you would have the root CA in pemtrustedcas_filepath only, and the node certificate and all intermediate certificates in pemcert_filepath.

···

On Friday, February 9, 2018 at 8:34:41 PM UTC+1, No Reply wrote:

Here are the SAN section

X509v3 Subject Alternative Name:

DNS:elastic.example.com, DNS:es1.example.com, DNS:es2.example.com, DNS:es3.example.com

On Friday, February 9, 2018 at 8:50:48 AM UTC-6, No Reply wrote:

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version
  • Installed and used enterprise modules, if any
  • JVM version and operating system version
  • Search Guard configuration files
  • Elasticsearch log messages on debug level
  • Other installed Elasticsearch or Kibana plugins, if any

ES version - 5.6.3

Kibana version - 5.6.3

Searchguard version - 5.6.3

Java Version - 1.8.0_152

I have a cert (elastic.example.com) that has alias of all my NODE hostname (es1.example.com, es2.example.com, es3.example.com).

here is the elasticsearch.yaml file I set the OID found in elastic.example.com cert on all 3 nodes elasticsearch yaml file.

bootstrap.memory_lock: false

cluster.name: ishlab-logging

discovery.zen.minimum_master_nodes: 2

discovery.zen.ping.unicast.hosts:

network.host: x.x.x.x

node.max_local_storage_nodes: 3

node.name: es1-es-01

path.data: “/var/lib/elasticsearch/es-01”

path.logs: “/var/log/elasticsearch/es-01”

searchguard.cert.oid: 1.3.6.1.4.1.44947.1.1.1

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/fullchain.pem

searchguard.ssl.http.pemkey_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/private.pkcs8

searchguard.ssl.http.pemtrustedcas_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/fullchain.pem

searchguard.ssl.transport.enforce_hostname_verification: true

searchguard.ssl.transport.pemcert_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/fullchain.pem

searchguard.ssl.transport.pemkey_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/private.pkcs8

searchguard.ssl.transport.pemtrustedcas_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/fullchain.pem

thread_pool.bulk.queue_size: 2000

xpack.graph.enabled: false

xpack.ml.enabled: false

xpack.monitoring.enabled: true

xpack.monitoring.exporters.id1.auth.password: kibanaserver

xpack.monitoring.exporters.id1.auth.username: kibanaserver

xpack.monitoring.exporters.id1.host: https://ish.example.com:9200

xpack.monitoring.exporters.id1.type: http

xpack.monitoring.history.duration: 1d

xpack.security.enabled: false

xpack.watcher.enabled: false

I’m getting the following error on each node… Nodes can’t ping each other. (no OID or searchguard.nodes_dn incorrect configured).

[2018-02-09T14:24:15,219][WARN ][o.e.d.z.UnicastZenPing ] [es1-es-01] [8] failed send ping to {#zen_unicast_es1.example.com_0#}{tEuZDtj8RvC_yLILyR-ysA}{es1.example.com}{x.x.x.x:9300}

java.lang.IllegalStateException: handshake failed with {#zen_unicast_es1.example.com_0#}{tEuZDtj8RvC_yLILyR-ysA}{es1.example.com}{x.x.x.x:9300}

at org.elasticsearch.transport.TransportService.handshake(TransportService.java:403) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.transport.TransportService.handshake(TransportService.java:370) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.discovery.zen.UnicastZenPing$PingingRound.getOrConnect(UnicastZenPing.java:400) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.discovery.zen.UnicastZenPing$3.doRun(UnicastZenPing.java:507) [elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.3.jar:5.6.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]

Caused by: org.elasticsearch.transport.RemoteTransportException: [es1-es-01][x.x.x.x:9300][internal:transport/handshake]

Caused by: org.elasticsearch.ElasticsearchException: bad header found. This means typically that one node try to connect to another with a non-node certificate (no OID or searchguard.nodes_dn incorrect configured) or that someoneis spoofing requests. See https://github.com/floragunncom/search-guard-docs/blob/master/tls_node_certificates.md

Any help on how to configure this?

It doesn’t list the OIDname but it does list like the following

X509v3 Subject Alternative Name:

DNS:elastic.example.com, DNS:es1.example.com, DNS:es2.example.com, DNS:es3.example.com

X509v3 Certificate Policies:

Policy: 2.23.140.1.2.1

Policy: 1.3.6.1.4.1.44947.1.1.1

CPS: http://cps.letsencrypt.org

User Notice:

Explicit Text: This Certificate may only be relied upon by Relying Parties and only in accordance with the Certificate Policy found at https://letsencrypt.org/repository/

The OID is the Policy ID. I read this from this redhat website - (https://access.redhat.com/documentation/en-US/Red_Hat_Certificate_System/8.0/html/Admin_Guide/Standard_X.509_v3_Certificate_Extensions.html)

So if I was to list the DN of your node certificate in elasticsearch.yml

searchguard.nodes_dn:

Will this work with the elastic.example.com cert?

···

On Friday, February 9, 2018 at 6:47:50 PM UTC-6, Jochen Kressin wrote:

It seems that your certificate does not contain any OID. Usually you would see something like:

SubjectAlternativeName [

DNSName: es1.example.com

DNSName: es2.example.com

OIDName: 1.2.3.4.5.5

]

Depending on the tool your are using the OID can also be called RID. But it seems the entry is missing. Where did you get the OID value “1.3.6.1.4.1.44947.1.1.1” from? Does not look like an OID to me.

As an alternative to using an OID, you can also just list the DN of your node certificate in elasticsearch.yml like:

searchguard.nodes_dn:

For finding out the actual DN of your certificate you can use tools like OpenSSL for example. Wildcards are supported here.

Another thing, it seems you configured the full certificate chain for both the pemcert_filepath and pemtrustedcas_filepath. While this may work usually you would have the root CA in pemtrustedcas_filepath only, and the node certificate and all intermediate certificates in pemcert_filepath.

On Friday, February 9, 2018 at 8:34:41 PM UTC+1, No Reply wrote:

Here are the SAN section

X509v3 Subject Alternative Name:

DNS:elastic.example.com, DNS:es1.example.com, DNS:es2.example.com, DNS:es3.example.com

On Friday, February 9, 2018 at 8:50:48 AM UTC-6, No Reply wrote:

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version
  • Installed and used enterprise modules, if any
  • JVM version and operating system version
  • Search Guard configuration files
  • Elasticsearch log messages on debug level
  • Other installed Elasticsearch or Kibana plugins, if any

ES version - 5.6.3

Kibana version - 5.6.3

Searchguard version - 5.6.3

Java Version - 1.8.0_152

I have a cert (elastic.example.com) that has alias of all my NODE hostname (es1.example.com, es2.example.com, es3.example.com).

here is the elasticsearch.yaml file I set the OID found in elastic.example.com cert on all 3 nodes elasticsearch yaml file.

bootstrap.memory_lock: false

cluster.name: ishlab-logging

discovery.zen.minimum_master_nodes: 2

discovery.zen.ping.unicast.hosts:

network.host: x.x.x.x

node.max_local_storage_nodes: 3

node.name: es1-es-01

path.data: “/var/lib/elasticsearch/es-01”

path.logs: “/var/log/elasticsearch/es-01”

searchguard.cert.oid: 1.3.6.1.4.1.44947.1.1.1

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/fullchain.pem

searchguard.ssl.http.pemkey_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/private.pkcs8

searchguard.ssl.http.pemtrustedcas_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/fullchain.pem

searchguard.ssl.transport.enforce_hostname_verification: true

searchguard.ssl.transport.pemcert_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/fullchain.pem

searchguard.ssl.transport.pemkey_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/private.pkcs8

searchguard.ssl.transport.pemtrustedcas_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/fullchain.pem

thread_pool.bulk.queue_size: 2000

xpack.graph.enabled: false

xpack.ml.enabled: false

xpack.monitoring.enabled: true

xpack.monitoring.exporters.id1.auth.password: kibanaserver

xpack.monitoring.exporters.id1.auth.username: kibanaserver

xpack.monitoring.exporters.id1.host: https://ish.example.com:9200

xpack.monitoring.exporters.id1.type: http

xpack.monitoring.history.duration: 1d

xpack.security.enabled: false

xpack.watcher.enabled: false

I’m getting the following error on each node… Nodes can’t ping each other. (no OID or searchguard.nodes_dn incorrect configured).

[2018-02-09T14:24:15,219][WARN ][o.e.d.z.UnicastZenPing ] [es1-es-01] [8] failed send ping to {#zen_unicast_es1.example.com_0#}{tEuZDtj8RvC_yLILyR-ysA}{es1.example.com}{x.x.x.x:9300}

java.lang.IllegalStateException: handshake failed with {#zen_unicast_es1.example.com_0#}{tEuZDtj8RvC_yLILyR-ysA}{es1.example.com}{x.x.x.x:9300}

at org.elasticsearch.transport.TransportService.handshake(TransportService.java:403) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.transport.TransportService.handshake(TransportService.java:370) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.discovery.zen.UnicastZenPing$PingingRound.getOrConnect(UnicastZenPing.java:400) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.discovery.zen.UnicastZenPing$3.doRun(UnicastZenPing.java:507) [elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.3.jar:5.6.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]

Caused by: org.elasticsearch.transport.RemoteTransportException: [es1-es-01][x.x.x.x:9300][internal:transport/handshake]

Caused by: org.elasticsearch.ElasticsearchException: bad header found. This means typically that one node try to connect to another with a non-node certificate (no OID or searchguard.nodes_dn incorrect configured) or that someoneis spoofing requests. See https://github.com/floragunncom/search-guard-docs/blob/master/tls_node_certificates.md

Any help on how to configure this?

The OID that Search Guard reads is in the SAN part of the certificate. It does not refer to the OIDs of the certificate extensions. So your certificate does not contain an OID in the SAN part, that’s why you need to list them in the searchguard.nodes_dn section of elasticsearch.yml.

So the setup should work. However, you need at least two certificates to run Search Guard, a node certificate and an admin certificate. As far as I know Let’s Encrypt is not able to issue simple client certificates which you could use to run sgadmin. So while the node setup may work, you would be stuck with a cluster that you cannot initialize.

If you just want to get rid of the browser warning when using self-signed certificates, we recommend using the Let’s Encrypt certificate for the REST/HTTPS, and self-signed (or from your company PKI) certs for the nodes and sgadmin.

···

On Saturday, February 10, 2018 at 6:12:57 AM UTC+1, No Reply wrote:

It doesn’t list the OIDname but it does list like the following

X509v3 Subject Alternative Name:

DNS:elastic.example.com, DNS:es1.example.com, DNS:es2.example.com, DNS:es3.example.com

X509v3 Certificate Policies:

Policy: 2.23.140.1.2.1

Policy: 1.3.6.1.4.1.44947.1.1.1

CPS: http://cps.letsencrypt.org

User Notice:

Explicit Text: This Certificate may only be relied upon by Relying Parties and only in accordance with the Certificate Policy found at https://letsencrypt.org/repository/

The OID is the Policy ID. I read this from this redhat website - (https://access.redhat.com/documentation/en-US/Red_Hat_Certificate_System/8.0/html/Admin_Guide/Standard_X.509_v3_Certificate_Extensions.html)

So if I was to list the DN of your node certificate in elasticsearch.yml

searchguard.nodes_dn:

Will this work with the elastic.example.com cert?

On Friday, February 9, 2018 at 6:47:50 PM UTC-6, Jochen Kressin wrote:

It seems that your certificate does not contain any OID. Usually you would see something like:

SubjectAlternativeName [

DNSName: es1.example.com

DNSName: es2.example.com

OIDName: 1.2.3.4.5.5

]

Depending on the tool your are using the OID can also be called RID. But it seems the entry is missing. Where did you get the OID value “1.3.6.1.4.1.44947.1.1.1” from? Does not look like an OID to me.

As an alternative to using an OID, you can also just list the DN of your node certificate in elasticsearch.yml like:

searchguard.nodes_dn:

For finding out the actual DN of your certificate you can use tools like OpenSSL for example. Wildcards are supported here.

Another thing, it seems you configured the full certificate chain for both the pemcert_filepath and pemtrustedcas_filepath. While this may work usually you would have the root CA in pemtrustedcas_filepath only, and the node certificate and all intermediate certificates in pemcert_filepath.

On Friday, February 9, 2018 at 8:34:41 PM UTC+1, No Reply wrote:

Here are the SAN section

X509v3 Subject Alternative Name:

DNS:elastic.example.com, DNS:es1.example.com, DNS:es2.example.com, DNS:es3.example.com

On Friday, February 9, 2018 at 8:50:48 AM UTC-6, No Reply wrote:

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version
  • Installed and used enterprise modules, if any
  • JVM version and operating system version
  • Search Guard configuration files
  • Elasticsearch log messages on debug level
  • Other installed Elasticsearch or Kibana plugins, if any

ES version - 5.6.3

Kibana version - 5.6.3

Searchguard version - 5.6.3

Java Version - 1.8.0_152

I have a cert (elastic.example.com) that has alias of all my NODE hostname (es1.example.com, es2.example.com, es3.example.com).

here is the elasticsearch.yaml file I set the OID found in elastic.example.com cert on all 3 nodes elasticsearch yaml file.

bootstrap.memory_lock: false

cluster.name: ishlab-logging

discovery.zen.minimum_master_nodes: 2

discovery.zen.ping.unicast.hosts:

network.host: x.x.x.x

node.max_local_storage_nodes: 3

node.name: es1-es-01

path.data: “/var/lib/elasticsearch/es-01”

path.logs: “/var/log/elasticsearch/es-01”

searchguard.cert.oid: 1.3.6.1.4.1.44947.1.1.1

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/fullchain.pem

searchguard.ssl.http.pemkey_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/private.pkcs8

searchguard.ssl.http.pemtrustedcas_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/fullchain.pem

searchguard.ssl.transport.enforce_hostname_verification: true

searchguard.ssl.transport.pemcert_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/fullchain.pem

searchguard.ssl.transport.pemkey_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/private.pkcs8

searchguard.ssl.transport.pemtrustedcas_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/fullchain.pem

thread_pool.bulk.queue_size: 2000

xpack.graph.enabled: false

xpack.ml.enabled: false

xpack.monitoring.enabled: true

xpack.monitoring.exporters.id1.auth.password: kibanaserver

xpack.monitoring.exporters.id1.auth.username: kibanaserver

xpack.monitoring.exporters.id1.host: https://ish.example.com:9200

xpack.monitoring.exporters.id1.type: http

xpack.monitoring.history.duration: 1d

xpack.security.enabled: false

xpack.watcher.enabled: false

I’m getting the following error on each node… Nodes can’t ping each other. (no OID or searchguard.nodes_dn incorrect configured).

[2018-02-09T14:24:15,219][WARN ][o.e.d.z.UnicastZenPing ] [es1-es-01] [8] failed send ping to {#zen_unicast_es1.example.com_0#}{tEuZDtj8RvC_yLILyR-ysA}{es1.example.com}{x.x.x.x:9300}

java.lang.IllegalStateException: handshake failed with {#zen_unicast_es1.example.com_0#}{tEuZDtj8RvC_yLILyR-ysA}{es1.example.com}{x.x.x.x:9300}

at org.elasticsearch.transport.TransportService.handshake(TransportService.java:403) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.transport.TransportService.handshake(TransportService.java:370) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.discovery.zen.UnicastZenPing$PingingRound.getOrConnect(UnicastZenPing.java:400) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.discovery.zen.UnicastZenPing$3.doRun(UnicastZenPing.java:507) [elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.3.jar:5.6.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]

Caused by: org.elasticsearch.transport.RemoteTransportException: [es1-es-01][x.x.x.x:9300][internal:transport/handshake]

Caused by: org.elasticsearch.ElasticsearchException: bad header found. This means typically that one node try to connect to another with a non-node certificate (no OID or searchguard.nodes_dn incorrect configured) or that someoneis spoofing requests. See https://github.com/floragunncom/search-guard-docs/blob/master/tls_node_certificates.md

Any help on how to configure this?

Hello what do you mean by using Let’s Encrypt certificate for the REST/HTTPS?

···

On Sunday, 11 February 2018 15:05:45 UTC+4, Jochen Kressin wrote:

The OID that Search Guard reads is in the SAN part of the certificate. It does not refer to the OIDs of the certificate extensions. So your certificate does not contain an OID in the SAN part, that’s why you need to list them in the searchguard.nodes_dn section of elasticsearch.yml.

So the setup should work. However, you need at least two certificates to run Search Guard, a node certificate and an admin certificate. As far as I know Let’s Encrypt is not able to issue simple client certificates which you could use to run sgadmin. So while the node setup may work, you would be stuck with a cluster that you cannot initialize.

If you just want to get rid of the browser warning when using self-signed certificates, we recommend using the Let’s Encrypt certificate for the REST/HTTPS, and self-signed (or from your company PKI) certs for the nodes and sgadmin.

On Saturday, February 10, 2018 at 6:12:57 AM UTC+1, No Reply wrote:

It doesn’t list the OIDname but it does list like the following

X509v3 Subject Alternative Name:

DNS:elastic.example.com, DNS:es1.example.com, DNS:es2.example.com, DNS:es3.example.com

X509v3 Certificate Policies:

Policy: 2.23.140.1.2.1

Policy: 1.3.6.1.4.1.44947.1.1.1

CPS: http://cps.letsencrypt.org

User Notice:

Explicit Text: This Certificate may only be relied upon by Relying Parties and only in accordance with the Certificate Policy found at https://letsencrypt.org/repository/

The OID is the Policy ID. I read this from this redhat website - (https://access.redhat.com/documentation/en-US/Red_Hat_Certificate_System/8.0/html/Admin_Guide/Standard_X.509_v3_Certificate_Extensions.html)

So if I was to list the DN of your node certificate in elasticsearch.yml

searchguard.nodes_dn:

Will this work with the elastic.example.com cert?

On Friday, February 9, 2018 at 6:47:50 PM UTC-6, Jochen Kressin wrote:

It seems that your certificate does not contain any OID. Usually you would see something like:

SubjectAlternativeName [

DNSName: es1.example.com

DNSName: es2.example.com

OIDName: 1.2.3.4.5.5

]

Depending on the tool your are using the OID can also be called RID. But it seems the entry is missing. Where did you get the OID value “1.3.6.1.4.1.44947.1.1.1” from? Does not look like an OID to me.

As an alternative to using an OID, you can also just list the DN of your node certificate in elasticsearch.yml like:

searchguard.nodes_dn:

For finding out the actual DN of your certificate you can use tools like OpenSSL for example. Wildcards are supported here.

Another thing, it seems you configured the full certificate chain for both the pemcert_filepath and pemtrustedcas_filepath. While this may work usually you would have the root CA in pemtrustedcas_filepath only, and the node certificate and all intermediate certificates in pemcert_filepath.

On Friday, February 9, 2018 at 8:34:41 PM UTC+1, No Reply wrote:

Here are the SAN section

X509v3 Subject Alternative Name:

DNS:elastic.example.com, DNS:es1.example.com, DNS:es2.example.com, DNS:es3.example.com

On Friday, February 9, 2018 at 8:50:48 AM UTC-6, No Reply wrote:

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version
  • Installed and used enterprise modules, if any
  • JVM version and operating system version
  • Search Guard configuration files
  • Elasticsearch log messages on debug level
  • Other installed Elasticsearch or Kibana plugins, if any

ES version - 5.6.3

Kibana version - 5.6.3

Searchguard version - 5.6.3

Java Version - 1.8.0_152

I have a cert (elastic.example.com) that has alias of all my NODE hostname (es1.example.com, es2.example.com, es3.example.com).

here is the elasticsearch.yaml file I set the OID found in elastic.example.com cert on all 3 nodes elasticsearch yaml file.

bootstrap.memory_lock: false

cluster.name: ishlab-logging

discovery.zen.minimum_master_nodes: 2

discovery.zen.ping.unicast.hosts:

network.host: x.x.x.x

node.max_local_storage_nodes: 3

node.name: es1-es-01

path.data: “/var/lib/elasticsearch/es-01”

path.logs: “/var/log/elasticsearch/es-01”

searchguard.cert.oid: 1.3.6.1.4.1.44947.1.1.1

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/fullchain.pem

searchguard.ssl.http.pemkey_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/private.pkcs8

searchguard.ssl.http.pemtrustedcas_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/fullchain.pem

searchguard.ssl.transport.enforce_hostname_verification: true

searchguard.ssl.transport.pemcert_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/fullchain.pem

searchguard.ssl.transport.pemkey_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/private.pkcs8

searchguard.ssl.transport.pemtrustedcas_filepath: “/etc/elasticsearch/es-01/searchguard/elastic.example.com/fullchain.pem

thread_pool.bulk.queue_size: 2000

xpack.graph.enabled: false

xpack.ml.enabled: false

xpack.monitoring.enabled: true

xpack.monitoring.exporters.id1.auth.password: kibanaserver

xpack.monitoring.exporters.id1.auth.username: kibanaserver

xpack.monitoring.exporters.id1.host: https://ish.example.com:9200

xpack.monitoring.exporters.id1.type: http

xpack.monitoring.history.duration: 1d

xpack.security.enabled: false

xpack.watcher.enabled: false

I’m getting the following error on each node… Nodes can’t ping each other. (no OID or searchguard.nodes_dn incorrect configured).

[2018-02-09T14:24:15,219][WARN ][o.e.d.z.UnicastZenPing ] [es1-es-01] [8] failed send ping to {#zen_unicast_es1.example.com_0#}{tEuZDtj8RvC_yLILyR-ysA}{es1.example.com}{x.x.x.x:9300}

java.lang.IllegalStateException: handshake failed with {#zen_unicast_es1.example.com_0#}{tEuZDtj8RvC_yLILyR-ysA}{es1.example.com}{x.x.x.x:9300}

at org.elasticsearch.transport.TransportService.handshake(TransportService.java:403) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.transport.TransportService.handshake(TransportService.java:370) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.discovery.zen.UnicastZenPing$PingingRound.getOrConnect(UnicastZenPing.java:400) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.discovery.zen.UnicastZenPing$3.doRun(UnicastZenPing.java:507) [elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.3.jar:5.6.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]

Caused by: org.elasticsearch.transport.RemoteTransportException: [es1-es-01][x.x.x.x:9300][internal:transport/handshake]

Caused by: org.elasticsearch.ElasticsearchException: bad header found. This means typically that one node try to connect to another with a non-node certificate (no OID or searchguard.nodes_dn incorrect configured) or that someoneis spoofing requests. See https://github.com/floragunncom/search-guard-docs/blob/master/tls_node_certificates.md

Any help on how to configure this?