Unknown certificate

Search Guard
1

Elasticsearch version:
7.10.2

Server OS version:
windows 10 home 64bit

Kibana version (if relevant):
7.10.2

Browser version (if relevant):
Windows Chrome Version 88.0.4324.190

Browser OS version (if relevant):
Windows Chrome Version 88.0.4324.190

Describe the issue: facing certificate_unknown issue while hitting localhost:9200 on brower and trying to connect with kibana and other beats

Steps to reproduce:

  1. Install SG plugin, run install_demo_configuration.sh.
  2. Install Kibana-SG plugin, add tsl configs in kibana.yml
  3. Hit localhost:9200 on broswer or try to connect with kibana

Expected behavior:
Smooth flow with out certificate errors

Elastic.yml:
searchguard.ssl.transport.pemcert_filepath: esnode.pem
searchguard.ssl.transport.pemkey_filepath: esnode-key.pem
searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: esnode.pem
searchguard.ssl.http.pemkey_filepath: esnode-key.pem
searchguard.ssl.http.pemtrustedcas_filepath: root-ca.pem
searchguard.allow_unsafe_democertificates: true
searchguard.allow_default_init_sgindex: true
searchguard.authcz.admin_dn:

  • CN=kirk,OU=client,O=client,L=test, C=de

searchguard.audit.type: internal_elasticsearch
searchguard.enable_snapshot_restore_privilege: true
searchguard.check_snapshot_restore_write_privileges: true
searchguard.restapi.roles_enabled: [“SGS_ALL_ACCESS”]
cluster.routing.allocation.disk.threshold_enabled: false
cluster.name: searchguard_demo
node.max_local_storage_nodes: 3
xpack.security.enabled: false

Kibana.yml
elasticsearch.hosts: [“https://localhost:9200”]
elasticsearch.username: “kibanaserver”
elasticsearch.password: “kibanaserver”
elasticsearch.ssl.verificationMode: none

4

The certificate_unknown error means that the certificate can’t be verified by the system. Let’s try to hit the Elasticsearch with curl first

curl -v -k -u admin:admin https://localhost:9200

Does it work? Show me the Elasticsearch log if any.

5

Rebuilt URL to: https://localhost:9200/

  • Trying ::1…
  • TCP_NODELAY set
  • Connected to localhost (::1) port 9200 (#0)
  • schannel: SSL/TLS connection with localhost port 9200 (step 1/3)
  • schannel: disabled server certificate revocation checks
  • schannel: verifyhost setting prevents Schannel from comparing the supplied target name with the subject names in server certificates.
  • schannel: sending initial handshake data: sending 165 bytes…
  • schannel: sent initial handshake data: sent 165 bytes
  • schannel: SSL/TLS connection with localhost port 9200 (step 2/3)
  • schannel: failed to receive handshake, need more data
  • schannel: SSL/TLS connection with localhost port 9200 (step 2/3)
  • schannel: encrypted data got 1846
  • schannel: encrypted data buffer: offset 1846 length 4096
  • schannel: a client certificate has been requested
  • schannel: SSL/TLS connection with localhost port 9200 (step 2/3)
  • schannel: encrypted data buffer: offset 1846 length 4096
  • schannel: sending next handshake data: sending 133 bytes…
  • schannel: SSL/TLS connection with localhost port 9200 (step 2/3)
  • schannel: encrypted data got 51
  • schannel: encrypted data buffer: offset 51 length 4096
  • schannel: SSL/TLS handshake complete
  • schannel: SSL/TLS connection with localhost port 9200 (step 3/3)
  • schannel: stored credential handle in session cache
  • Server auth using Basic with user ‘admin’

GET / HTTP/1.1
Host: localhost:9200
Authorization: Basic YWRtaW46YWRtaW4=
User-Agent: curl/7.55.1
Accept: /

  • schannel: client wants to read 102400 bytes
  • schannel: encdata_buffer resized 103424
  • schannel: encrypted data buffer: offset 0 length 103424
  • schannel: encrypted data got 649
  • schannel: encrypted data buffer: offset 649 length 103424
  • schannel: decrypted data length: 620
  • schannel: decrypted data added: 620
  • schannel: decrypted data cached: offset 620 length 102400
  • schannel: encrypted data buffer: offset 0 length 103424
  • schannel: decrypted data buffer: offset 620 length 102400
  • schannel: schannel_recv cleanup
  • schannel: decrypted data returned 620
  • schannel: decrypted data buffer: offset 0 length 102400
    < HTTP/1.1 200 OK
    < content-type: application/json; charset=UTF-8
    < content-length: 533
    <
    {
    “name” : “HARI”,
    “cluster_name” : “searchguard_demo”,
    “cluster_uuid” : “4pIr_u3dRfafcl98JPI8IA”,
    “version” : {
    “number” : “7.10.2”,
    “build_flavor” : “default”,
    “build_type” : “zip”,
    “build_hash” : “747e1cc71def077253878a59143c1f785afa92b9”,
    “build_date” : “2021-01-13T00:42:12.435326Z”,
    “build_snapshot” : false,
    “lucene_version” : “8.7.0”,
    “minimum_wire_compatibility_version” : “6.8.0”,
    “minimum_index_compatibility_version” : “6.0.0-beta1”
    },
    “tagline” : “You Know, for Search”
    }
  • Connection #0 to host localhost left intact
6

REQUEST
curl -v -XGET \ –cacert /path/to/elasticsearch-7.10.2/config/root-ca.pem \ –cert /path/to/elasticsearch-7.10.2/config/kirk.pem \ –key /path/to/elasticsearch-7.10.2/config/kirk-key.pem \ https://localhost:9200

RESPONSE
C:\WINDOWS\system32>curl -v -XGET --cacert F:\ELK\elasticsearch-7.10.2/config/root-ca.pem --cert F:\ELK\elasticsearch-7.10.2/config/kirk.pem --key F:\ELK\elasticsearch-7.10.2/config/kirk-key.pem https://localhost:9200
Note: Unnecessary use of -X or --request, GET is already inferred.

  • Rebuilt URL to: https://localhost:9200/
  • Trying ::1…
  • TCP_NODELAY set
  • Connected to localhost (::1) port 9200 (#0)
  • schannel: SSL/TLS connection with localhost port 9200 (step 1/3)
  • schannel: checking server certificate revocation
  • schannel: sending initial handshake data: sending 174 bytes…
  • schannel: sent initial handshake data: sent 174 bytes
  • schannel: SSL/TLS connection with localhost port 9200 (step 2/3)
  • schannel: failed to receive handshake, need more data
  • schannel: SSL/TLS connection with localhost port 9200 (step 2/3)
  • schannel: encrypted data got 1846
  • schannel: encrypted data buffer: offset 1846 length 4096
  • schannel: next InitializeSecurityContext failed: SEC_E_UNTRUSTED_ROOT (0x80090325) - The certificate chain was issued by an authority that is not trusted.
  • Closing connection 0
  • schannel: shutting down SSL/TLS connection with localhost port 9200
  • schannel: clear security context handle
    curl: (77) schannel: next InitializeSecurityContext failed: SEC_E_UNTRUSTED_ROOT (0x80090325) - The certificate chain was issued by an authority that is not trusted.
7

I see the command curl -v -k -u admin:admin https://localhost:9200 worked for you. Then, I think, hitting https://localhost:9200 with a web browser from the same machine where you executed curl should work too. You should see the basic-auth dialog, put the credentials.

The curl request with certificates should work too. It works for me. Did you change the certificates in elasticsearch-7.10.2/config that were generated when you executed install_demo_configuration.sh? Is your system behind a proxy? Use curl -x if it is HTTP proxy or use --socks5 if its SOCKS proxy.

8

While hitting https://localhost:9200… i got basic auth dialog, got response to browser
{
“name” : “HARI”,
“cluster_name” : “searchguard_demo”,
“cluster_uuid” : “VKTWfCEXR-WFVIXgSKtVxQ”,
“version” : {
“number” : “7.10.2”,
“build_flavor” : “default”,
“build_type” : “zip”,
“build_hash” : “747e1cc71def077253878a59143c1f785afa92b9”,
“build_date” : “2021-01-13T00:42:12.435326Z”,
“build_snapshot” : false,
“lucene_version” : “8.7.0”,
“minimum_wire_compatibility_version” : “6.8.0”,
“minimum_index_compatibility_version” : “6.0.0-beta1”
},
“tagline” : “You Know, for Search”
}

but in elastic console i got below error

io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
Caused by: javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1647) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1615) ~[?:?]
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1781) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1070) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:896) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_191]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:282) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
… 16 more
[2021-03-09T14:51:04,547][WARN ][o.e.h.AbstractHttpServerTransport] [HARI] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=0.0.0.0/0.0.0.0:9200, remoteAddress=/0:0:0:0:0:0:0:1:51675}

Facing this error for every request either kibana or beats connection

9

Using default certificates only which are generated while executed install_demo_configuration.sh. We did’n chang them in elasticsearch-7.10.2/config. Not using any proxies.

10

Ok, you were able to get the response. The Elasticsearch API access works as expected. The error in the log is not related to your API requests.

Check all applications that talk to Elasticsearch. Do they use the correct TLS certificates? Do they use the correct port (port 9200 must be used for API)?
Do you have a cluster? If yes, check the elasticsearch.yml config on each node. For intercommunication, the nodes must use TLS certificates signed by the same root CA.

11

I can reproduce the error in the log. With Firefox, I get it only the first time after I accept the risk in the web browser. With Chrome, I get it for every web browser request. I think it is related to this Chrome bug 952778 - chromium - An open-source project to help move the web forward. - Monorail The Chrome should handle it the same way Firefox does.

Anyway, the error is because the certificate is self-signed, and the system key chain doesn’t have the root CA. The only way to solve it is to have certificates signed by a known root CA.

12

We are trying to connect elasticsearch(https://localhost:9200) with kibana and heartbeat,
we didn’t mention about certificates details in either kibana.yml or heartbeat.yml

updates in kibana.yml
elasticsearch.username: “kibanaserver”
elasticsearch.password: “kibanaserver”
elasticsearch.hosts: “https://localhost:9200”
elasticsearch.ssl.verificationMode: none

updates in heartbeat.yml
hosts: [“localhost:9200”]
protocol: “https”

are we need to mention about TLS certificates???

how to get certificates signed by a known root CA for localhost…? It’s just for POC purpose

13

The certificate_unknown error in the log is cosmetical. What doesn’t work exactly, Kibana? I see you miss some config options in your kibana.yml. A typical Kibana configuration for the basic authentication and no certificates is

# Activate basic auth
searchguard.auth.type: "basicauth"

# Configure session management
searchguard.cookie.password: <encryption key, min. 32 characters>

# Use HTTPS instead of HTTP
elasticsearch.hosts: "https://<hostname>.com:<http port>"

# Configure the Kibana internal server user
elasticsearch.username: "kibanaserver"
elasticsearch.password: "kibanaserver"

# Disable SSL verification when using self-signed demo certificates
elasticsearch.ssl.verificationMode: none

# Whitelist basic headers and multi tenancy header
elasticsearch.requestHeadersWhitelist: ["Authorization", "sgtenant"]
14

The periodical certificate_unknown error in the log might be caused by the heartbeat. Configure ssl.verification_mode: none for it. Or provide the root certificate in ssl.certificate_authorities.