# Empty client certificate chain error seen in all nodes

**URL:** <https://forum.search-guard.com/t/empty-client-certificate-chain-error-seen-in-all-nodes/2021>\
**Category:** Search Guard\
**Created:** [December 9, 2020, 3:53pm UTC](https://forum.search-guard.com/t/empty-client-certificate-chain-error-seen-in-all-nodes/2021 "2020-12-09T15:53:32Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![shivani.aggarwal2195](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/shivani.aggarwal2195/32/571_2.png) [@shivani.aggarwal2195](https://forum.search-guard.com/u/shivani.aggarwal2195)\
**Post date:** [December 9, 2020, 3:53pm UTC](https://forum.search-guard.com/t/empty-client-certificate-chain-error-seen-in-all-nodes/2021/1 "2020-12-09T15:53:32Z")

</div>

**Elasticsearch version:** 7.8.0  
**Searchguard version:** 7.8.0-43.0.0

**Describe the issue:**  
I use the sample scripts at [Sample PKI scripts | Security for Elasticsearch | Search Guard](https://docs.search-guard.com/latest/tls-certificates-sample-scripts) to generate certificates for searchguard.

With this elasticsearch is installed on a k8s environment as helm chart and the elasticsearch cluster works as expected. However, in some Ipv6 envs, we notice even though the cluster formation has happened, all nodes have joined the cluster and health is green, we see this exception appearing continuously in all master, data and client nodes:

```
{"type":"log","host":"elk-efkc-elk-elasticsearch-client-6f9d5b8474-4lfj8","level":"DEBUG","systemid":"2106a117733f42d697284fbc54927928","system":"elk","time": "2020-12-08T14:13:33.497Z","logger":"c.f.s.f.SearchGuardFilter","timezone":"UTC","marker":"[elk-efkc-elk-elasticsearch-client-6f9d5b8474-4lfj8] ","log":"PrivEvalResponse [allowed=true, missingPrivileges=[], allowedFlsFields=null, maskedFields=null, queries=null]"}
{"type":"log","host":"elk-efkc-elk-elasticsearch-client-6f9d5b8474-4lfj8","level":"ERROR","systemid":"2106a117733f42d697284fbc54927928","system":"elk","time": "2020-12-08T14:13:33.950Z","logger":"c.f.s.s.t.SearchGuardSSLNettyTransport","timezone":"UTC","marker":"[elk-efkc-elk-elasticsearch-client-6f9d5b8474-4lfj8] ","log":"Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Empty client certificate chain"}
javax.net.ssl.SSLHandshakeException: Empty client certificate chain
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:313) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:269) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:260) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1176) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1163) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1048) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:995) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1542) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1556) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1440) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
       at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at java.lang.Thread.run(Thread.java:834) [?:?]
{"type":"log","host":"elk-efkc-elk-elasticsearch-client-6f9d5b8474-4lfj8","level":"WARN","systemid":"2106a117733f42d697284fbc54927928","system":"elk","time": "2020-12-08T14:13:33.952Z","logger":"o.e.t.TcpTransport","timezone":"UTC","marker":"[elk-efkc-elk-elasticsearch-client-6f9d5b8474-4lfj8] ","log":"exception caught on transport layer [Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=null}], closing connection"}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:313) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:269) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:260) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1176) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1163) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1048) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:995) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1556) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1440) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
        ... 16 more

```

Client certificate based authentication, as you can see from the shared config files, is not enabled. This looks like the ssl handshake error during communication between the nodes on 9300. However, the cluster seems healthy and able to ingest data too.  
What could be the reason of this errors ?

**Provide configuration:**  
elasticsearch/config/elasticsearch.yml : [elasticsearch.yml-emptyclient](https://forum.search-guard.com/uploads/short-url/v3se8u2nlQ7s4QP75x91063WiVg.yml-emptyc) (1.5 KB)

elasticsearch/plugins/search-guard-7/sgconfig/sg\_config.yml  
[sg\_config.yml-emptyclient](https://forum.search-guard.com/uploads/short-url/rEGaNA92iqQagXWeqC64LS4tVVQ.yml-emptyc) (1.1 KB)

Edit - Not sure if the uploaded files are viewable - adding the configurations again:  
elasticsearch.yml

```
cluster:
  name: app-elk-efkc
  initial_master_nodes: elk-efkc-elk-elasticsearch-master-0,elk-efkc-elk-elasticsearch-master-1,elk-efkc-elk-elasticsearch-master-2
node:
  master: true
  data: false
  name: elk-efkc-elk-elasticsearch-master-0
  ingest: false
  remote_cluster_client: false
network.host: _global:ipv6_
path:
  data: /data/data
  logs: /data/log
  repo: /data/esbackup
http:
  compression: true
  cors:
    enabled: true
    allow-origin: *
discovery:
  seed_hosts: elk-efkc-elk-elasticsearch-discovery
searchguard:
    ssl.transport:
        enabled: true
        enable_openssl_if_available: false
        keystore_type: JKS
        keystore_filepath: /etc/elasticsearch/certs/keystore.jks
        keystore_password: changeit
        truststore_type: JKS
        truststore_filepath: /etc/elasticsearch/certs/truststore.jks
        truststore_password: changeit
        enforce_hostname_verification: false
    ssl.http:
        enabled: true
        clientauth_mode: OPTIONAL
        enable_openssl_if_available: true
        keystore_type: JKS
        keystore_filepath: /etc/elasticsearch/certs/keystore.jks
        keystore_password: changeit
        truststore_type: JKS
        truststore_filepath: /etc/elasticsearch/certs/truststore.jks
        truststore_password: changeit
    authcz.admin_dn:
      - "CN=admin,C=ELK"
    enterprise_modules_enabled: false
    ssl:
      cert_reload_enabled: true
      http.crl.validate: false

```

sg\_config.yml

```
---
_sg_meta:
  type: config
  config_version: 2
sg_config:
  dynamic:
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: true
        internalProxies: .+
        remoteIpHeader: x-forwarded-for
    authc:
      basic_internal_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: internal
      proxy_auth_domain:
        http_enabled: false
        transport_enabled: false
        order: 1
        http_authenticator:
          type: proxy
          challenge: false
          config:
            user_header: x-proxy-user
            roles_header: x-proxy-roles
        authentication_backend:
          type: noop
      clientcert_auth_domain:
        http_enabled: false
        transport_enabled: false
        order: 2
        http_authenticator:
          challenge: false
          type: clientcert
          config:
            username_attribute: cn
        authentication_backend:
          type: noop

```

---

<div class="post-metadata">

**Author:** ![srgbnd](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/srgbnd/32/506_2.png) [@srgbnd](https://forum.search-guard.com/u/srgbnd)\
**Post date:** [December 10, 2020, 3:17pm UTC](https://forum.search-guard.com/t/empty-client-certificate-chain-error-seen-in-all-nodes/2021/2 "2020-12-10T15:17:12Z")

</div>

The error says that a client didn’t send a certificate with the key. I don’t see [searchguard.nodes\_dn](https://docs.search-guard.com/latest/configuring-tls#configuring-node-certificates) in the elasticsearch configuration, you need it for inter-cluster communication. Do you use the [OID value as a SAN entry](https://docs.search-guard.com/latest/tls-in-production#using-an-oid-value-as-san-entry)?

Do you have any external service(s) that you might forget to configure with the new certificates, maybe Logstash? What does happen if you omit `clientauth_mode: OPTIONAL` option from the config?

Another thing I would try is to test the certificates by making the curl requests. First, you need to [convert the .jks key and certificate into .pem](https://docs.cloudera.com/documentation/enterprise/5-10-x/topics/cm_sg_openssl_jks.html#concept_ek3_sdl_rp). Then you do requests from each node to each node in the cluster and see what the nodes respond. Maybe you misconfigured a node. Thus if you have `n` nodes you do `n(n-1)` requests in total. For example, to test node1 and node2 from node0 do

```bash
curl -v https://node1:9300 -key key.pem -cacert cacert.pem -cert node0.pem
curl -v https://node2:9300 -key key.pem -cacert cacert.pem -cert node0.pem

```

I would recommend you to generate certificates using the [Search Guard TLS tool](https://docs.search-guard.com/latest/offline-tls-tool). You have the certificates in .pem, it is easier to test things with curl.

---

<div class="post-metadata">

**Author:** ![shivani.aggarwal2195](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/shivani.aggarwal2195/32/571_2.png) [@shivani.aggarwal2195](https://forum.search-guard.com/u/shivani.aggarwal2195)\
**Post date:** [December 13, 2020, 6:14pm UTC](https://forum.search-guard.com/t/empty-client-certificate-chain-error-seen-in-all-nodes/2021/3 "2020-12-13T18:14:25Z")

</div>

1. Yes, my certificates contain OID value `1.2.3.4.5.5` in the SAN entry, hence have not configured [searchguard.nodes\_dn](https://docs.search-guard.com/latest/configuring-tls#configuring-node-certificates).

2. Also, tried curl using certificate and key (exported from the keystore) on 9300 port from one node to other. Looks like no issues in the certificates. In addition, as my setup is in k8s env and all es nodes run as pods, same certificates are mounted in all es pods. So I don’t see a possibility where one(or some) nodes could have been configured with different certificates.

> curl -v [https://elk-efkc-elk-elasticsearch-discovery:9300](https://elk-efkc-elk-elasticsearch-discovery:9300) -I -k --key /usr/share/elasticsearch/esnode-key.pem --cert /usr/share/elasticsearch/esnode-cert.pem  
> About to connect() to elk-efkc-elk-elasticsearch-discovery port 9300 (#0)  
> Trying fd01:abcd::b763…  
> Connected to elk-efkc-elk-elasticsearch-discovery (fd01:abcd::b763) port 9300 (#0)  
> Initializing NSS with certpath: sql:/etc/pki/nssdb  
> skipping SSL peer certificate verification  
> NSS: client certificate from file  
> subject: CN=elasticsearch.logging,C=EL…  
> start date: Dec 09 03:47:14 2020 GMT  
> expire date: Dec 09 03:47:14 2022 GMT  
> common name: elasticsearch.logging  
> issuer: CN=Example Com Inc. Signing CA,OU=Example Com Inc. Signing CA,O=Example Com Inc.,DC=example,DC=com  
> SSL connection using TLS\_ECDHE\_RSA\_WITH\_AES\_256\_GCM\_SHA384  
> Server certificate:  
> subject: CN=elasticsearch.logging,C=EL…  
> start date: Dec 09 03:47:14 2020 GMT  
> expire date: Dec 09 03:47:14 2022 GMT  
> common name: elasticsearch.logging  
> issuer: CN=Example Com Inc. Signing CA,OU=Example Com Inc. Signing CA,O=Example Com Inc.,DC=example,DC=com  
> HEAD / HTTP/1.1  
> User-Agent: curl/7.29.0  
> Host: elk-efkc-elk-elasticsearch-discovery:9300  
> Accept: _/_
> 
> Connection #0 to host elk-efkc-elk-elasticsearch-discovery left intact  
> This is not an HTTP port

1. 

> [@srgbnd](#):
>
> Do you have any external service(s) that you might forget to configure with the new certificates, maybe Logstash? What does happen if you omit `clientauth_mode: OPTIONAL` option from the config?

But as per my understanding, external services contact elasticsearch only on 9200 (i.e through REST apis) but the errors here are seen on 9300 which is used for internal node-to-node communication. To rule out possibility of external services which might be sending an incorrect client certificate, I made the REST service on 9200 unreachable to external clients. Even then, the errors keep coming regularly.

```auto
Exception caught on transport layer [Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=null}

```

---

<div class="post-metadata">

**Author:** ![srgbnd](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/srgbnd/32/506_2.png) [@srgbnd](https://forum.search-guard.com/u/srgbnd)\
**Post date:** [December 14, 2020, 2:02pm UTC](https://forum.search-guard.com/t/empty-client-certificate-chain-error-seen-in-all-nodes/2021/4 "2020-12-14T14:02:38Z")

</div>

Validate the certificate chain, here is the guide [TLS help | Security for Elasticsearch | Search Guard](https://docs.search-guard.com/latest/troubleshooting-tls#tls-troubleshooting)

Make sure that

- The keystore contains the client or node certificate with its private key, and all intermediate certificates
- The truststore contains the root certificate

Show me `X509v3 extensions` part of your node certificates. For example

```auto
        X509v3 extensions:
            X509v3 Authority Key Identifier: 
                keyid:68:26:81:65:E7:44:A9:D1:DD:AC:1E:88:16:B9:9A:FF:52:01:A4:0E
                DirName:/DC=com/DC=example/O=Example Com, Inc./OU=CA/CN=root.ca.example.com
                serial:02

            X509v3 Subject Key Identifier: 
                BD:7F:05:16:D2:71:83:16:4F:81:FE:22:C4:74:22:9B:8B:6D:BA:DE
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Key Usage: critical
                Digital Signature, Non Repudiation, Key Encipherment
            X509v3 Extended Key Usage: critical
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Alternative Name: 
                DNS:node1.example.com, IP Address:10.0.2.1

```

---

<div class="post-metadata">

**Author:** ![shivani.aggarwal2195](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/shivani.aggarwal2195/32/571_2.png) [@shivani.aggarwal2195](https://forum.search-guard.com/u/shivani.aggarwal2195)\
**Post date:** [December 14, 2020, 2:11pm UTC](https://forum.search-guard.com/t/empty-client-certificate-chain-error-seen-in-all-nodes/2021/5 "2020-12-14T14:11:22Z")

</div>

Sharing the entire keystore generated (it is a chain of 3 certs: node cert, intermediate CA and rootCA)

> $ keytool -list -v -keystore keystore.jks  
> Keystore type: PKCS12  
> Keystore provider: SUN
> 
> Your keystore contains 1 entry
> 
> Alias name: elasticsearch  
> Creation date: Dec 9, 2020  
> Entry type: PrivateKeyEntry  
> Certificate chain length: 3  
> Certificate[1]:  
> Owner: CN=elasticsearch.logging, C=ELK  
> Issuer: CN=Example Com Inc. Signing CA, OU=Example Com Inc. Signing CA, O=Example Com Inc., DC=example, DC=com  
> Serial number: 1  
> Valid from: Wed Dec 09 03:47:14 UTC 2020 until: Fri Dec 09 03:47:14 UTC 2022  
> Certificate fingerprints:  
> SHA1: BC:0D:C5:8D:3B:C3:94:9C:29:FC:55:B2:53:04:92:B5:DB:64:56:AE  
> SHA256: CC:72:0C:A3:BD:78:0A:CF:4D:53:0F:7F:FA:41:BA:9F:6E:06:FD:1F:1B:D3:31:14:66:90:27:75:D8:52:DC:CC  
> Signature algorithm name: SHA256withRSA  
> Subject Public Key Algorithm: 2048-bit RSA key  
> Version: 3
> 
> Extensions:
> 
> #1: ObjectId: 2.5.29.35 Criticality=false  
> AuthorityKeyIdentifier [  
> KeyIdentifier [  
> 0000: BB 4B 31 EA 81 CA 95 E7 16 1D 61 01 75 2B 6C 05 .K1…a.u+l.  
> 0010: 74 82 01 EA t…  
> ]  
> ]
> 
> #2: ObjectId: 2.5.29.19 Criticality=false  
> BasicConstraints:[  
> CA:false  
> PathLen: undefined  
> ]
> 
> #3: ObjectId: 2.5.29.31 Criticality=false  
> CRLDistributionPoints [  
> [DistributionPoint:  
> [URIName: [https://raw.githubusercontent.com/floragunncom/unittest-assets/master/revoked.crl](https://raw.githubusercontent.com/floragunncom/unittest-assets/master/revoked.crl)]  
> ]]
> 
> #4: ObjectId: 2.5.29.37 Criticality=false  
> ExtendedKeyUsages [  
> serverAuth  
> clientAuth  
> ]
> 
> #5: ObjectId: 2.5.29.15 Criticality=true  
> KeyUsage [  
> DigitalSignature  
> Key\_Encipherment  
> ]
> 
> #6: ObjectId: 2.5.29.17 Criticality=false  
> SubjectAlternativeName [  
> DNSName: elasticsearch.logging  
> DNSName: localhost  
> IPAddress: 127.0.0.1  
> OIDName: 1.2.3.4.5.5  
> ]
> 
> #7: ObjectId: 2.5.29.14 Criticality=false  
> SubjectKeyIdentifier [  
> KeyIdentifier [  
> 0000: E0 D9 2F D2 75 39 6B 27 F1 86 A7 8D 5D 4B C6 80 …/.u9k’…]K…  
> 0010: E6 B0 4D 15 …M.  
> ]  
> ]
> 
> Certificate[2]:  
> Owner: CN=Example Com Inc. Signing CA, OU=Example Com Inc. Signing CA, O=Example Com Inc., DC=example, DC=com  
> Issuer: CN=elasticsearch. Root CA, DC=logging, DC=elasticsearch  
> Serial number: 2  
> Valid from: Wed Dec 09 03:47:12 UTC 2020 until: Mon Dec 09 03:47:12 UTC 2030  
> Certificate fingerprints:  
> SHA1: 1A:C1:2B:7F:B4:21:FA:64:73:E4:92:DA:31:43:10:4C:8E:58:B8:80  
> SHA256: 67:44:69:B2:AE:A7:5B:F6:70:63:E3:05:2C:E8:8E:CA:00:CF:A0:8D:57:0E:3C:74:62:3C:39:4E:B0:52:BF:08  
> Signature algorithm name: SHA256withRSA  
> Subject Public Key Algorithm: 2048-bit RSA key  
> Version: 3
> 
> Extensions:
> 
> #1: ObjectId: 2.5.29.35 Criticality=false  
> AuthorityKeyIdentifier [  
> KeyIdentifier [  
> 0000: DB 3F E0 5F E2 B5 AB A3 9C BD 17 60 FE B4 2C 23 .?.\_…`…,#  
> 0010: 4B EE CF 6B K…k  
> ]  
> ]
> 
> #2: ObjectId: 2.5.29.19 Criticality=true  
> BasicConstraints:[  
> CA:true  
> PathLen:0  
> ]
> 
> #3: ObjectId: 2.5.29.15 Criticality=true  
> KeyUsage [  
> Key\_CertSign  
> Crl\_Sign  
> ]
> 
> #4: ObjectId: 2.5.29.14 Criticality=false  
> SubjectKeyIdentifier [  
> KeyIdentifier [  
> 0000: BB 4B 31 EA 81 CA 95 E7 16 1D 61 01 75 2B 6C 05 .K1…a.u+l.  
> 0010: 74 82 01 EA t…  
> ]  
> ]
> 
> Certificate[3]:  
> Owner: CN=elasticsearch. Root CA, DC=logging, DC=elasticsearch  
> Issuer: CN=elasticsearch. Root CA, DC=logging, DC=elasticsearch  
> Serial number: 1  
> Valid from: Wed Dec 09 03:47:12 UTC 2020 until: Mon Dec 09 03:47:12 UTC 2030  
> Certificate fingerprints:  
> SHA1: 76:8C:E5:8A:A6:A8:1B:D4:32:EB:6D:43:D3:8A:68:0C:09:64:61:E7  
> SHA256: D8:01:99:D5:D4:DE:C0:71:41:4F:1E:6A:D3:83:34:BD:78:18:D4:1D:9B:32:BF:01:C1:CC:A8:03:CB:BD:EE:7C  
> Signature algorithm name: SHA256withRSA  
> Subject Public Key Algorithm: 2048-bit RSA key  
> Version: 3
> 
> Extensions:
> 
> #1: ObjectId: 2.5.29.35 Criticality=false  
> AuthorityKeyIdentifier [  
> KeyIdentifier [  
> 0000: DB 3F E0 5F E2 B5 AB A3 9C BD 17 60 FE B4 2C 23 .?.\_…`…,#  
> 0010: 4B EE CF 6B K…k  
> ]  
> ]
> 
> #2: ObjectId: 2.5.29.19 Criticality=true  
> BasicConstraints:[  
> CA:true  
> PathLen:2147483647  
> ]
> 
> #3: ObjectId: 2.5.29.15 Criticality=true  
> KeyUsage [  
> Key\_CertSign  
> Crl\_Sign  
> ]
> 
> #4: ObjectId: 2.5.29.14 Criticality=false  
> SubjectKeyIdentifier [  
> KeyIdentifier [  
> 0000: DB 3F E0 5F E2 B5 AB A3 9C BD 17 60 FE B4 2C 23 .?.\_…`…,#  
> 0010: 4B EE CF 6B K…k  
> ]  
> ]

The certificates are generated using sample scripts from [Sample PKI scripts | Security for Elasticsearch | Search Guard](https://docs.search-guard.com/latest/tls-certificates-sample-scripts).

---

<div class="post-metadata">

**Author:** ![nils](https://avatars.discourse-cdn.com/v4/letter/n/d6d6ee/32.png) [@nils](https://forum.search-guard.com/u/nils)\
**Post date:** [December 14, 2020, 3:50pm UTC](https://forum.search-guard.com/t/empty-client-certificate-chain-error-seen-in-all-nodes/2021/6 "2020-12-14T15:50:19Z")

</div>

As the cluster is actually green and working, I don’t think that the problem is actually connected to a configuration problem in Elasticsearch or Search Guard. Port 9300 is not reserved exclusively for internal communication, also external clients (so-called transport clients) can connect to an ES cluster on port 9300. For example, the `sgadmin` tool connects to Elasticsearch on port 9300.

So, I’d recommend to look for something like maintenance jobs or health checks which might access each node regularly on port 9300. Do your helm charts possibly set up any health checks or maintenance jobs?

You said the exception is appearing continuously. In what time intervals does it occur?

---

<div class="post-metadata">

**Author:** ![system](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/system/32/1870_2.png) [@system](https://forum.search-guard.com/u/system)\
**Post date:** [January 4, 2021, 3:50pm UTC](https://forum.search-guard.com/t/empty-client-certificate-chain-error-seen-in-all-nodes/2021/7 "2021-01-04T15:50:34Z")

</div>

Closed on January 4, 2021, 3:50pm UTC

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.