Empty client certificate chain error seen in all nodes

Search Guard
1

Elasticsearch version: 7.8.0
Searchguard version: 7.8.0-43.0.0

Describe the issue:
I use the sample scripts at Sample PKI scripts | Security for Elasticsearch | Search Guard to generate certificates for searchguard.

With this elasticsearch is installed on a k8s environment as helm chart and the elasticsearch cluster works as expected. However, in some Ipv6 envs, we notice even though the cluster formation has happened, all nodes have joined the cluster and health is green, we see this exception appearing continuously in all master, data and client nodes:

{"type":"log","host":"elk-efkc-elk-elasticsearch-client-6f9d5b8474-4lfj8","level":"DEBUG","systemid":"2106a117733f42d697284fbc54927928","system":"elk","time": "2020-12-08T14:13:33.497Z","logger":"c.f.s.f.SearchGuardFilter","timezone":"UTC","marker":"[elk-efkc-elk-elasticsearch-client-6f9d5b8474-4lfj8] ","log":"PrivEvalResponse [allowed=true, missingPrivileges=[], allowedFlsFields=null, maskedFields=null, queries=null]"}
{"type":"log","host":"elk-efkc-elk-elasticsearch-client-6f9d5b8474-4lfj8","level":"ERROR","systemid":"2106a117733f42d697284fbc54927928","system":"elk","time": "2020-12-08T14:13:33.950Z","logger":"c.f.s.s.t.SearchGuardSSLNettyTransport","timezone":"UTC","marker":"[elk-efkc-elk-elasticsearch-client-6f9d5b8474-4lfj8] ","log":"Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Empty client certificate chain"}
javax.net.ssl.SSLHandshakeException: Empty client certificate chain
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:313) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:269) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:260) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1176) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1163) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1048) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:995) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1542) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1556) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1440) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
       at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at java.lang.Thread.run(Thread.java:834) [?:?]
{"type":"log","host":"elk-efkc-elk-elasticsearch-client-6f9d5b8474-4lfj8","level":"WARN","systemid":"2106a117733f42d697284fbc54927928","system":"elk","time": "2020-12-08T14:13:33.952Z","logger":"o.e.t.TcpTransport","timezone":"UTC","marker":"[elk-efkc-elk-elasticsearch-client-6f9d5b8474-4lfj8] ","log":"exception caught on transport layer [Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=null}], closing connection"}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:313) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:269) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:260) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1176) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1163) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1048) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:995) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1556) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1440) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
        ... 16 more

Client certificate based authentication, as you can see from the shared config files, is not enabled. This looks like the ssl handshake error during communication between the nodes on 9300. However, the cluster seems healthy and able to ingest data too.
What could be the reason of this errors ?

Provide configuration:
elasticsearch/config/elasticsearch.yml : elasticsearch.yml-emptyclient (1.5 KB)

elasticsearch/plugins/search-guard-7/sgconfig/sg_config.yml
sg_config.yml-emptyclient (1.1 KB)

Edit - Not sure if the uploaded files are viewable - adding the configurations again:
elasticsearch.yml

cluster:
  name: app-elk-efkc
  initial_master_nodes: elk-efkc-elk-elasticsearch-master-0,elk-efkc-elk-elasticsearch-master-1,elk-efkc-elk-elasticsearch-master-2
node:
  master: true
  data: false
  name: elk-efkc-elk-elasticsearch-master-0
  ingest: false
  remote_cluster_client: false
network.host: _global:ipv6_
path:
  data: /data/data
  logs: /data/log
  repo: /data/esbackup
http:
  compression: true
  cors:
    enabled: true
    allow-origin: *
discovery:
  seed_hosts: elk-efkc-elk-elasticsearch-discovery
searchguard:
    ssl.transport:
        enabled: true
        enable_openssl_if_available: false
        keystore_type: JKS
        keystore_filepath: /etc/elasticsearch/certs/keystore.jks
        keystore_password: changeit
        truststore_type: JKS
        truststore_filepath: /etc/elasticsearch/certs/truststore.jks
        truststore_password: changeit
        enforce_hostname_verification: false
    ssl.http:
        enabled: true
        clientauth_mode: OPTIONAL
        enable_openssl_if_available: true
        keystore_type: JKS
        keystore_filepath: /etc/elasticsearch/certs/keystore.jks
        keystore_password: changeit
        truststore_type: JKS
        truststore_filepath: /etc/elasticsearch/certs/truststore.jks
        truststore_password: changeit
    authcz.admin_dn:
      - "CN=admin,C=ELK"
    enterprise_modules_enabled: false
    ssl:
      cert_reload_enabled: true
      http.crl.validate: false

sg_config.yml

---
_sg_meta:
  type: config
  config_version: 2
sg_config:
  dynamic:
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: true
        internalProxies: .+
        remoteIpHeader: x-forwarded-for
    authc:
      basic_internal_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: internal
      proxy_auth_domain:
        http_enabled: false
        transport_enabled: false
        order: 1
        http_authenticator:
          type: proxy
          challenge: false
          config:
            user_header: x-proxy-user
            roles_header: x-proxy-roles
        authentication_backend:
          type: noop
      clientcert_auth_domain:
        http_enabled: false
        transport_enabled: false
        order: 2
        http_authenticator:
          challenge: false
          type: clientcert
          config:
            username_attribute: cn
        authentication_backend:
          type: noop
2

The error says that a client didn’t send a certificate with the key. I don’t see searchguard.nodes_dn in the elasticsearch configuration, you need it for inter-cluster communication. Do you use the OID value as a SAN entry?

Do you have any external service(s) that you might forget to configure with the new certificates, maybe Logstash? What does happen if you omit clientauth_mode: OPTIONAL option from the config?

Another thing I would try is to test the certificates by making the curl requests. First, you need to convert the .jks key and certificate into .pem. Then you do requests from each node to each node in the cluster and see what the nodes respond. Maybe you misconfigured a node. Thus if you have n nodes you do n(n-1) requests in total. For example, to test node1 and node2 from node0 do

curl -v https://node1:9300 -key key.pem -cacert cacert.pem -cert node0.pem
curl -v https://node2:9300 -key key.pem -cacert cacert.pem -cert node0.pem

I would recommend you to generate certificates using the Search Guard TLS tool. You have the certificates in .pem, it is easier to test things with curl.

3

  1. Yes, my certificates contain OID value 1.2.3.4.5.5 in the SAN entry, hence have not configured searchguard.nodes_dn.

  2. Also, tried curl using certificate and key (exported from the keystore) on 9300 port from one node to other. Looks like no issues in the certificates. In addition, as my setup is in k8s env and all es nodes run as pods, same certificates are mounted in all es pods. So I don’t see a possibility where one(or some) nodes could have been configured with different certificates.

curl -v https://elk-efkc-elk-elasticsearch-discovery:9300 -I -k --key /usr/share/elasticsearch/esnode-key.pem --cert /usr/share/elasticsearch/esnode-cert.pem
About to connect() to elk-efkc-elk-elasticsearch-discovery port 9300 (#0)
Trying fd01:abcd::b763…
Connected to elk-efkc-elk-elasticsearch-discovery (fd01:abcd::b763) port 9300 (#0)
Initializing NSS with certpath: sql:/etc/pki/nssdb
skipping SSL peer certificate verification
NSS: client certificate from file
subject: CN=elasticsearch.logging,C=EL…
start date: Dec 09 03:47:14 2020 GMT
expire date: Dec 09 03:47:14 2022 GMT
common name: elasticsearch.logging
issuer: CN=Example Com Inc. Signing CA,OU=Example Com Inc. Signing CA,O=Example Com Inc.,DC=example,DC=com
SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
Server certificate:
subject: CN=elasticsearch.logging,C=EL…
start date: Dec 09 03:47:14 2020 GMT
expire date: Dec 09 03:47:14 2022 GMT
common name: elasticsearch.logging
issuer: CN=Example Com Inc. Signing CA,OU=Example Com Inc. Signing CA,O=Example Com Inc.,DC=example,DC=com
HEAD / HTTP/1.1
User-Agent: curl/7.29.0
Host: elk-efkc-elk-elasticsearch-discovery:9300
Accept: /

Connection #0 to host elk-efkc-elk-elasticsearch-discovery left intact
This is not an HTTP port

But as per my understanding, external services contact elasticsearch only on 9200 (i.e through REST apis) but the errors here are seen on 9300 which is used for internal node-to-node communication. To rule out possibility of external services which might be sending an incorrect client certificate, I made the REST service on 9200 unreachable to external clients. Even then, the errors keep coming regularly.

Exception caught on transport layer [Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=null}
4

Validate the certificate chain, here is the guide TLS help | Security for Elasticsearch | Search Guard

Make sure that

  • The keystore contains the client or node certificate with its private key, and all intermediate certificates
  • The truststore contains the root certificate

Show me X509v3 extensions part of your node certificates. For example

        X509v3 extensions:
            X509v3 Authority Key Identifier: 
                keyid:68:26:81:65:E7:44:A9:D1:DD:AC:1E:88:16:B9:9A:FF:52:01:A4:0E
                DirName:/DC=com/DC=example/O=Example Com, Inc./OU=CA/CN=root.ca.example.com
                serial:02

            X509v3 Subject Key Identifier: 
                BD:7F:05:16:D2:71:83:16:4F:81:FE:22:C4:74:22:9B:8B:6D:BA:DE
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Key Usage: critical
                Digital Signature, Non Repudiation, Key Encipherment
            X509v3 Extended Key Usage: critical
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Alternative Name: 
                DNS:node1.example.com, IP Address:10.0.2.1
5

Sharing the entire keystore generated (it is a chain of 3 certs: node cert, intermediate CA and rootCA)

$ keytool -list -v -keystore keystore.jks
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: elasticsearch
Creation date: Dec 9, 2020
Entry type: PrivateKeyEntry
Certificate chain length: 3
Certificate[1]:
Owner: CN=elasticsearch.logging, C=ELK
Issuer: CN=Example Com Inc. Signing CA, OU=Example Com Inc. Signing CA, O=Example Com Inc., DC=example, DC=com
Serial number: 1
Valid from: Wed Dec 09 03:47:14 UTC 2020 until: Fri Dec 09 03:47:14 UTC 2022
Certificate fingerprints:
SHA1: BC:0D:C5:8D:3B:C3:94:9C:29:FC:55:B2:53:04:92:B5:DB:64:56:AE
SHA256: CC:72:0C:A3:BD:78:0A:CF:4D:53:0F:7F:FA:41:BA:9F:6E:06:FD:1F:1B:D3:31:14:66:90:27:75:D8:52:DC:CC
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: BB 4B 31 EA 81 CA 95 E7 16 1D 61 01 75 2B 6C 05 .K1…a.u+l.
0010: 74 82 01 EA t…
]
]

#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
CA:false
PathLen: undefined
]

#3: ObjectId: 2.5.29.31 Criticality=false
CRLDistributionPoints [
[DistributionPoint:
[URIName: https://raw.githubusercontent.com/floragunncom/unittest-assets/master/revoked.crl]
]]

#4: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
serverAuth
clientAuth
]

#5: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
DigitalSignature
Key_Encipherment
]

#6: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
DNSName: elasticsearch.logging
DNSName: localhost
IPAddress: 127.0.0.1
OIDName: 1.2.3.4.5.5
]

#7: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: E0 D9 2F D2 75 39 6B 27 F1 86 A7 8D 5D 4B C6 80 …/.u9k’…]K…
0010: E6 B0 4D 15 …M.
]
]

Certificate[2]:
Owner: CN=Example Com Inc. Signing CA, OU=Example Com Inc. Signing CA, O=Example Com Inc., DC=example, DC=com
Issuer: CN=elasticsearch. Root CA, DC=logging, DC=elasticsearch
Serial number: 2
Valid from: Wed Dec 09 03:47:12 UTC 2020 until: Mon Dec 09 03:47:12 UTC 2030
Certificate fingerprints:
SHA1: 1A:C1:2B:7F:B4:21:FA:64:73:E4:92:DA:31:43:10:4C:8E:58:B8:80
SHA256: 67:44:69:B2:AE:A7:5B:F6:70:63:E3:05:2C:E8:8E:CA:00:CF:A0:8D:57:0E:3C:74:62:3C:39:4E:B0:52:BF:08
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: DB 3F E0 5F E2 B5 AB A3 9C BD 17 60 FE B4 2C 23 .?._…`…,#
0010: 4B EE CF 6B K…k
]
]

#2: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
CA:true
PathLen:0
]

#3: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
Key_CertSign
Crl_Sign
]

#4: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: BB 4B 31 EA 81 CA 95 E7 16 1D 61 01 75 2B 6C 05 .K1…a.u+l.
0010: 74 82 01 EA t…
]
]

Certificate[3]:
Owner: CN=elasticsearch. Root CA, DC=logging, DC=elasticsearch
Issuer: CN=elasticsearch. Root CA, DC=logging, DC=elasticsearch
Serial number: 1
Valid from: Wed Dec 09 03:47:12 UTC 2020 until: Mon Dec 09 03:47:12 UTC 2030
Certificate fingerprints:
SHA1: 76:8C:E5:8A:A6:A8:1B:D4:32:EB:6D:43:D3:8A:68:0C:09:64:61:E7
SHA256: D8:01:99:D5:D4:DE:C0:71:41:4F:1E:6A:D3:83:34:BD:78:18:D4:1D:9B:32:BF:01:C1:CC:A8:03:CB:BD:EE:7C
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: DB 3F E0 5F E2 B5 AB A3 9C BD 17 60 FE B4 2C 23 .?._…`…,#
0010: 4B EE CF 6B K…k
]
]

#2: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
CA:true
PathLen:2147483647
]

#3: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
Key_CertSign
Crl_Sign
]

#4: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: DB 3F E0 5F E2 B5 AB A3 9C BD 17 60 FE B4 2C 23 .?._…`…,#
0010: 4B EE CF 6B K…k
]
]

The certificates are generated using sample scripts from Sample PKI scripts | Security for Elasticsearch | Search Guard.

6

As the cluster is actually green and working, I don’t think that the problem is actually connected to a configuration problem in Elasticsearch or Search Guard. Port 9300 is not reserved exclusively for internal communication, also external clients (so-called transport clients) can connect to an ES cluster on port 9300. For example, the sgadmin tool connects to Elasticsearch on port 9300.

So, I’d recommend to look for something like maintenance jobs or health checks which might access each node regularly on port 9300. Do your helm charts possibly set up any health checks or maintenance jobs?

You said the exception is appearing continuously. In what time intervals does it occur?

7

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.