# Elasticsearh/Kibana 6.4.2 + SearchGuard 23.1, sslv3 handshake failure
**URL:** https://forum.search-guard.com/t/elasticsearh-kibana-6-4-2-searchguard-23-1-sslv3-handshake-failure/1176
**Category:** Search Guard
**Created:** [November 5, 2018, 4:23am UTC](https://forum.search-guard.com/t/elasticsearh-kibana-6-4-2-searchguard-23-1-sslv3-handshake-failure/1176 "2018-11-05T04:23:04Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@Yong\_Wang](https://forum.search-guard.com/u/Yong_Wang)
#### Post date: [November 5, 2018, 4:23am UTC](https://forum.search-guard.com/t/elasticsearh-kibana-6-4-2-searchguard-23-1-sslv3-handshake-failure/1176/1 "2018-11-05T04:23:04Z")
Dear Sir or Madam:
I deployed an Elasticsearch cluster with SearchGuard by using latest release, met a ssl handshake error when want to connect Kibana to my ES cluster. It blocked me for several days, still can’t resolve it. Could you help check and give me some hints? Thanks!
I enabled the SSL encryption and Client Certification based authentication, the same config can work in my previous cluster ( ES 5.5.1 + SearchGuard + Kibana), but not work in the new cluster.
When try to open Kibana page, it will show error message “The Search Guard license information could not be loaded. Please contact your system administrator.”, the the Kibana will report error like below :
Elasticsearch ERROR: 2018-11-05T03:24:00Z
Error: Request error, retrying
GET [https://kubernetes.mycluster.com:30099/\_searchguard/license](https://kubernetes.mycluster.com:30099/_searchguard/license) =\> write EPROTO 140599130773312:error:14094410:SSL routines:ssl3\_read\_bytes:sslv3 alert handshake failure:…/deps/openssl/openssl/ssl/s3\_pkt.c:1498:SSL alert number 40
140599130773312:error:1409E0E5:SSL routines:ssl3\_write\_bytes:ssl handshake failure:…/deps/openssl/openssl/ssl/s3\_pkt.c:659:
Elasticsearch WARNING: 2018-11-05T03:24:00Z
Unable to revive connection: [https://kubernetes.mycluster.com:30099/](https://kubernetes.mycluster.com:30099/)
Elasticsearch WARNING: 2018-11-05T03:24:00Z
No living connections
But if I use the curl command with same cert/key, can get the license info correctly, like below :
\*\*Curl command output : \*\*
curl --insecure --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key --cacert elasticsearch-admin.crt.pem -XGET [https://kubernetes.mycluster.com:30099/\_searchguard/license](https://kubernetes.mycluster.com:30099/_searchguard/license)
{“\_nodes”:{“total”:9,“successful”:9,“failed”:0},“cluster\_name”:“ICES”,“sg\_license”:{“uid”:“00000000-0000-0000-0000-000000000000”,“type”:“TRIAL”,“features”:[“COMPLIANCE”],“issue\_date”:“2018-10-25”,“expiry\_date”:“2018-12-25”,“issued\_to”:“The world”,“issuer”:“floragunn GmbH”,“start\_date”:“2018-10-25”,“major\_version”:6,“cluster\_name”:“\*”,“msgs”:,“expiry\_in\_days”:52,“is\_expired”:false,“is\_valid”:true,“action”:“”,…
Could you help check and give me some hints?
**Information for debug :**
SearchGuard : 23.1
Elasticsearch : 6.4.2
Kibana : 6.4.2
Installed and used enterprise modules : N/A
JVM version and operating system version: JDK 1.8.0.171-7.b10.el7, CentOS 7
**Kibana.yml :**
server.port: 5601
server.host: “x.x.x.x”
elasticsearch.url: “[https://kubernetes.mycluster.com:30099](https://kubernetes.mycluster.com:30099)”
elasticsearch.ssl.certificate: /root/elasticsearch/k8skey/elasticsearch-admin.crt.pem
elasticsearch.ssl.key: /root/elasticsearch/k8skey/elasticsearch-admin.key
elasticsearch.ssl.keyPassphrase: “password”
elasticsearch.ssl.certificateAuthorities: [“/root/elasticsearch/ca/chain-ca.pem”]
elasticsearch.ssl.verificationMode: none
searchguard.allow\_client\_certificates: true
xpack.reporting.encryptionKey: “1234567890”
xpack.security.enabled: false
xpack.monitoring.enabled: true
xpack.graph.enabled: false
xpack.ml.enabled: false
xpack.watcher.enabled: false
searchguard.cookie.secure: false
**Elasticsearch.yml :**
[root@es-data-0 config]# more elasticsearch.yml
#search guard config
searchguard:
ssl:
transport:
enable\_openssl\_if\_available: true
enforce\_hostname\_verification: false
pemkey\_filepath: certs/elasticsearch-transport.key
pemkey\_password: password
pemcert\_filepath: certs/elasticsearch-transport.crt.pem
pemtrustedcas\_filepath: certs/chain-ca.pem
http:
enable\_openssl\_if\_available: true
enabled: true
pemkey\_filepath: certs/elasticsearch-http.key
pemkey\_password: password
pemcert\_filepath: certs/elasticsearch-http.crt.pem
pemtrustedcas\_filepath: certs/chain-ca.pem
clientauth\_mode: REQUIRE
authcz:
admin\_dn:
- “CN=elasticsearch-admin, OU=ICES, O=MYCLUSTER, C=US”
xpack.security.enabled: false
xpack.monitoring.enabled: true
xpack.graph.enabled: false
xpack.watcher.enabled: false
**Kibana output :**
root@kubernetes:/usr/share/kibana/bin# ./kibana
log [03:18:08.914] [info][status][plugin:kibana@6.4.2] Status changed from uninitialized to green - Ready
log [03:18:09.019] [info][status][plugin:elasticsearch@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [03:18:09.025] [info][status][plugin:xpack\_main@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [03:18:09.034] [info][status][plugin:searchprofiler@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [03:18:09.041] [info][status][plugin:tilemap@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [03:18:09.052] [info][status][plugin:license\_management@6.4.2] Status changed from uninitialized to green - Ready
log [03:18:09.060] [info][status][plugin:index\_management@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [03:18:09.320] [info][status][plugin:timelion@6.4.2] Status changed from uninitialized to green - Ready
log [03:18:09.340] [info][status][plugin:monitoring@6.4.2] Status changed from uninitialized to green - Ready
log [03:18:09.344] [info][status][plugin:grokdebugger@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [03:18:09.349] [info][status][plugin:dashboard\_mode@6.4.2] Status changed from uninitialized to green - Ready
log [03:18:09.353] [info][status][plugin:logstash@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [03:18:09.405] [info][status][plugin:apm@6.4.2] Status changed from uninitialized to green - Ready
log [03:18:09.429] [info][status][plugin:console@6.4.2] Status changed from uninitialized to green - Ready
log [03:18:09.434] [info][status][plugin:console\_extensions@6.4.2] Status changed from uninitialized to green - Ready
log [03:18:09.440] [info][status][plugin:notifications@6.4.2] Status changed from uninitialized to green - Ready
log [03:18:09.610] [info][status][plugin:searchguard@6.4.2-15] Status changed from uninitialized to yellow - Initialising Search Guard authentication plugin.
log [03:18:09.612] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Default cookie password detected, please set a password in kibana.yml by setting ‘searchguard.cookie.password’ (min. 32 characters).
log [03:18:09.612] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - ‘searchguard.cookie.secure’ is set to false, cookies are transmitted over unsecure HTTP connection. Consider using HTTPS and set this key to ‘true’
log [03:18:09.664] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard session management enabled.
log [03:18:09.665] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard copy JWT params disabled
log [03:18:09.666] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard multitenancy disabled
log [03:18:09.685] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Routes for Search Guard configuration GUI registered. This is an Enterprise feature.
log [03:18:09.692] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard system routes registered.
log [03:18:09.693] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to green - Search Guard plugin initialised.
log [03:18:09.699] [info][status][plugin:metrics@6.4.2] Status changed from uninitialized to green - Ready
log [03:18:10.573] [info][status][plugin:reporting@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [03:18:10.691] [info][listening][server][http] Server running at [http://9.98.173.247:5601](http://9.98.173.247:5601)
log [03:18:11.126] [info][status][plugin:elasticsearch@6.4.2] Status changed from yellow to green - Ready
log [03:18:11.233] [info][license][xpack] Imported license information from Elasticsearch for the [data] cluster: mode: basic | status: active
log [03:18:11.241] [info][status][plugin:xpack\_main@6.4.2] Status changed from yellow to green - Ready
log [03:18:11.243] [info][status][plugin:searchprofiler@6.4.2] Status changed from yellow to green - Ready
log [03:18:11.244] [info][status][plugin:tilemap@6.4.2] Status changed from yellow to green - Ready
log [03:18:11.245] [info][status][plugin:index\_management@6.4.2] Status changed from yellow to green - Ready
log [03:18:11.247] [info][status][plugin:grokdebugger@6.4.2] Status changed from yellow to green - Ready
log [03:18:11.248] [info][status][plugin:logstash@6.4.2] Status changed from yellow to green - Ready
log [03:18:11.251] [info][status][plugin:reporting@6.4.2] Status changed from yellow to green - Ready
log [03:18:11.253] [info][kibana-monitoring][monitoring-ui] Starting monitoring stats collection
log [03:18:11.475] [info][license][xpack] Imported license information from Elasticsearch for the [monitoring] cluster: mode: basic | status: active
Elasticsearch ERROR: 2018-11-05T03:24:00Z
Error: Request error, retrying
GET [https://kubernetes.mycluster.com:30099/\_searchguard/license](https://kubernetes.mycluster.com:30099/_searchguard/license) =\> write EPROTO 140599130773312:error:14094410:SSL routines:ssl3\_read\_bytes:sslv3 alert handshake failure:…/deps/openssl/openssl/ssl/s3\_pkt.c:1498:SSL alert number 40
140599130773312:error:1409E0E5:SSL routines:ssl3\_write\_bytes:ssl handshake failure:…/deps/openssl/openssl/ssl/s3\_pkt.c:659:
at Log.error (/usr/share/kibana/node\_modules/elasticsearch/src/lib/log.js:225:56)
at checkRespForFailure (/usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:258:18)
at HttpConnector. (/usr/share/kibana/node\_modules/elasticsearch/src/lib/connectors/http.js:157:7)
at ClientRequest.bound (/usr/share/kibana/node\_modules/elasticsearch/node\_modules/lodash/dist/lodash.js:729:21)
at emitOne (events.js:116:13)
at ClientRequest.emit (events.js:211:7)
at TLSSocket.socketErrorListener (\_http\_client.js:387:9)
at emitOne (events.js:116:13)
at TLSSocket.emit (events.js:211:7)
at onwriteError (\_stream\_writable.js:418:12)
at onwrite (\_stream\_writable.js:440:5)
at \_destroy (internal/streams/destroy.js:39:7)
at TLSSocket.Socket.\_destroy (net.js:564:3)
at TLSSocket.destroy (internal/streams/destroy.js:32:8)
at WriteWrap.afterWrite (net.js:866:10)
Elasticsearch WARNING: 2018-11-05T03:24:00Z
Unable to revive connection: [https://kubernetes](https://kubernetes). [mycluster.com:30099/](http://mycluster.com:30099/)
Elasticsearch WARNING: 2018-11-05T03:24:00Z
No living connections
error [03:24:00.574] Error: No Living connections
at sendReqWithConnection (/usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:225:15)
at next (/usr/share/kibana/node\_modules/elasticsearch/src/lib/connection\_pool.js:213:7)
at \_combinedTickCallback (internal/process/next\_tick.js:131:7)
at process.\_tickDomainCallback (internal/process/next\_tick.js:218:9)
\*\*Elasticsearch log : \*\*
[2018-10-29T08:48:51,625][INFO][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Info: SearchGuardLicense [uid=00000000-0000-0000-0000-000000000000, type=TRIAL, features=[COMPLIANCE], issueDate=2018-10-25, expiryDate=2018-12-25, issuedTo=The world, issuer=floragunn GmbH, startDate=2018-10-25, majorVersion=6, clusterName=\*, allowedNodeCount=2147483647, msgs=, expiresInDays=56, isExpired=false, valid=true, action=, prodUsage=Yes, one cluster with all commercial features and unlimited nodes per cluster., clusterService=org.elasticsearch.cluster.service.ClusterService@744c2d63, getMsgs()=, getExpiresInDays()=56, isExpired()=false, isValid()=true, getAction()=, getProdUsage()=Yes, one cluster with all commercial features and unlimited nodes per cluster.]
[2018-10-29T08:48:51,626][INFO][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Type: TRIAL, valid
[2018-10-29T08:48:51,626][INFO][c.f.s.c.IndexBaseConfigurationRepository] Node ‘es-client-666c88cfb5-dmw7c’ initialized
[2018-10-29T08:50:38,214][INFO][o.e.c.s.ClusterApplierService] [es-client-666c88cfb5-dmw7c] added {{es-master-8ff849f9-j5z8l}{hOYJ3NcIQLGIniyHcUy14A}{D9WeiTgCT\_CiH1T9CDx6vg}{192.168.1.50}{192.168.1.50:9300}{ml.machine\_memory=8352608256, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {es-master-8ff849f9-d7s6g}{\_8vGh29LSFWHxxo\_TZLcsQ}{iVusXa7vS8C7YHsyjXhltA}{192.168.2.45}{192.168.2.45:9300}{ml.machine\_memory=8352608256, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true} committed version [4303]])
[2018-11-02T07:13:58,878][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [es-client-666c88cfb5-dmw7c] SSL Problem error:140890C7:SSL routines:ssl3\_get\_client\_certificate:peer did not return a certificate
javax.net.ssl.SSLHandshakeException: error:140890C7:SSL routines:ssl3\_get\_client\_certificate:peer did not return a certificate
```
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1104) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1064) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1170) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
```
\*\*Certificate info : \*\*
openssl x509 -in elasticsearch-admin.crt.pem -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 10 (0xa)
Signature Algorithm: sha256WithRSAEncryption
Issuer: DC=ICES, O=MYCLUSTER, OU=ICES, CN=ICES, Signing CA
Validity
Not Before: Feb 28 04:06:01 2018 GMT
Not After : Feb 28 04:06:01 2020 GMT
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:db:15:db:b0:06:cc:de:c4:2d:aa:77:c8:08:f7:
31:20:6b:2b:78:45:00:02:89:9a:a3:7c:dc:82:3b:
65:ca:3f:31:01:17:b3:8f:19:dd:12:8a:8b:dc:fe:
ae:65:e9:d5:22:7e:93:ac:53:f8:30:98:91:92:ef:
19:8d:20:aa:2e:a9:20:68:05:61:06:ce:b6:e5:01:
b0:5a:3a:da:7c:1c:e7:e5:2d:c3:99:45:60:37:10:
…
…
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Basic Constraints:
CA:FALSE
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Subject Key Identifier:
C0:35:91:E3:2E:DC:2E:95:B9:D3:FC:86:9C:82:C5:86:DC:39:38:67
X509v3 Authority Key Identifier:
keyid:6C:B5:45:52:EF:CA:62:0E:8D:36:9B:C7:17:9E:F6:BC:D7:90:13:64
> **···**
>
> Subject: C=US, O=MYCLUSTER, OU=ICES, CN=elasticsearch-admin
---
### Author:  [@Yong\_Wang](https://forum.search-guard.com/u/Yong_Wang)
#### Post date: [November 5, 2018, 5:21am UTC](https://forum.search-guard.com/t/elasticsearh-kibana-6-4-2-searchguard-23-1-sslv3-handshake-failure/1176/2 "2018-11-05T05:21:22Z")
And openssl client also work with same certs/keys :
root@kubernetes:/usr/share/kibana/bin# openssl s\_client -connect [kubernetes.mycluster.com:30099](http://kubernetes.mycluster.com:30099) -servername x.x.x.x -cert /root/elasticsearch/k8skey/elasticsearch-admin.crt.pem -key /root/elasticsearch/k8skey/elasticsearch-admin.key -CAfile /root/elasticsearch/ca/chain-ca.pem
Enter pass phrase for /root/elasticsearch/k8skey/elasticsearch-admin.key:
CONNECTED(00000003)
depth=1 DC = ICES, O = MYCLUSTER, OU = ICES, CN = “ICES, Signing CA”
verify error:num=20:unable to get local issuer certificate
> **···**
>
> * * *
>
> Certificate chain
>
> 0 s:/C=US/O=MYCLUSTER/OU=ICES/CN=elasticsearch-http
>
> i:/DC=ICES/O=MYCLUSTER/OU=ICES/CN=ICES, Signing CA
>
> 1 s:/DC=ICES/O=MYCLUSTER M/OU=ICES/CN=ICES, Signing CA
>
> i:/DC=ICES/O=MYCLUSTER/OU=ICES/CN=ICES, Root CA
>
> * * *
>
> Server certificate
>
> -----BEGIN CERTIFICATE-----
>
> MIIDqzCCApOgAwIBAgIBATANBgkqhkiG9w0BAQsFADBOMRQwEgYKCZImiZPyLGQB
>
> GRYESUNFUzEMMAoGA1UECgwDSUJNMQ0wCwYDVQQLDARJQ0VTMRkwFwYDVQQDDBBJ
>
> …
>
> …
>
> -----END CERTIFICATE-----
>
> subject=/C=US/O=MYCLUSTER/OU=ICES/CN=elasticsearch-http
>
> issuer=/DC=ICES/O=MYCLUSTER/OU=ICES/CN=ICES, Signing CA
>
> * * *
>
> Acceptable client certificate CA names
>
> /DC=ICES/O=MYCLUSTER/OU=ICES/CN=ICES, Root CA
>
> /DC=ICES/O=MYCLUSTER/OU=ICES/CN=ICES, Signing CA
>
> Client Certificate Types: RSA sign, DSA sign, ECDSA sign
>
> Requested Signature Algorithms: RSA+SHA512:DSA+SHA512:ECDSA+SHA512:RSA+SHA384:DSA+SHA384:ECDSA+SHA384:RSA+SHA256:DSA+SHA256:ECDSA+SHA256:RSA+SHA224:DSA+SHA224:ECDSA+SHA224:RSA+SHA1:DSA+SHA1:ECDSA+SHA1
>
> Shared Requested Signature Algorithms: RSA+SHA512:DSA+SHA512:ECDSA+SHA512:RSA+SHA384:DSA+SHA384:ECDSA+SHA384:RSA+SHA256:DSA+SHA256:ECDSA+SHA256:RSA+SHA224:DSA+SHA224:ECDSA+SHA224:RSA+SHA1:DSA+SHA1:ECDSA+SHA1
>
> Peer signing digest: SHA512
>
> Server Temp Key: ECDH, P-256, 256 bits
>
> * * *
>
> SSL handshake has read 2594 bytes and written 1689 bytes
>
> * * *
>
> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-SHA256
>
> Server public key is 2048 bit
>
> Secure Renegotiation IS supported
>
> Compression: NONE
>
> Expansion: NONE
>
> No ALPN negotiated
>
> SSL-Session:
>
> Protocol : TLSv1.2
>
> Cipher : ECDHE-RSA-AES128-SHA256
>
> Session-ID: 715D57B05C520803D87671DC24DB773C954ABC881374D4EFF74B2019BC1E77E3
>
> Session-ID-ctx:
>
> Master-Key: D79B276C3E13F9F2E4D9903FB6542046155546E132EC2C406E2C20CF40317960A2EEE7573B52E814940F2A9D48C7885B
>
> Key-Arg : None
>
> PSK identity: None
>
> PSK identity hint: None
>
> SRP username: None
>
> Start Time: 1541395055
>
> Timeout : 300 (sec)
>
> Verify return code: 20 (unable to get local issuer certificate)
>
> * * *
>
> On Monday, November 5, 2018 at 12:23:04 PM UTC+8, Yong Wang wrote:
>
> > Dear Sir or Madam:
>
> >
>
> > I deployed an Elasticsearch cluster with SearchGuard by using latest release, met a ssl handshake error when want to connect Kibana to my ES cluster. It blocked me for several days, still can’t resolve it. Could you help check and give me some hints? Thanks!
>
> >
>
> > I enabled the SSL encryption and Client Certification based authentication, the same config can work in my previous cluster ( ES 5.5.1 + SearchGuard + Kibana), but not work in the new cluster.
>
> >
>
> > When try to open Kibana page, it will show error message “The Search Guard license information could not be loaded. Please contact your system administrator.”, the the Kibana will report error like below :
>
> >
>
> > Elasticsearch ERROR: 2018-11-05T03:24:00Z
>
> > Error: Request error, retrying
>
> > GET [https://kubernetes.mycluster.com:30099/\_searchguard/license](https://kubernetes.mycluster.com:30099/_searchguard/license) =\> write EPROTO 140599130773312:error:14094410:SSL routines:ssl3\_read\_bytes:sslv3 alert handshake failure:…/deps/openssl/openssl/ssl/s3\_pkt.c:1498:SSL alert number 40
>
> > 140599130773312:error:1409E0E5:SSL routines:ssl3\_write\_bytes:ssl handshake failure:…/deps/openssl/openssl/ssl/s3\_pkt.c:659:
>
> >
>
> >
>
> > Elasticsearch WARNING: 2018-11-05T03:24:00Z
>
> > Unable to revive connection: [https://kubernetes.mycluster.com:30099/](https://kubernetes.mycluster.com:30099/)
>
> >
>
> > Elasticsearch WARNING: 2018-11-05T03:24:00Z
>
> > No living connections
>
> >
>
> > But if I use the curl command with same cert/key, can get the license info correctly, like below :
>
> >
>
> > \*\*Curl command output : \*\*
>
> > curl --insecure --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key --cacert elasticsearch-admin.crt.pem -XGET [https://kubernetes.mycluster.com:30099/\_searchguard/license](https://kubernetes.mycluster.com:30099/_searchguard/license)
>
> > {“\_nodes”:{“total”:9,“successful”:9,“failed”:0},“cluster\_name”:“ICES”,“sg\_license”:{“uid”:“00000000-0000-0000-0000-000000000000”,“type”:“TRIAL”,“features”:[“COMPLIANCE”],“issue\_date”:“2018-10-25”,“expiry\_date”:“2018-12-25”,“issued\_to”:“The world”,“issuer”:“floragunn GmbH”,“start\_date”:“2018-10-25”,“major\_version”:6,“cluster\_name”:“\*”,“msgs”:,“expiry\_in\_days”:52,“is\_expired”:false,“is\_valid”:true,“action”:“”,…
>
> >
>
> >
>
> > Could you help check and give me some hints?
>
> >
>
> >
>
> > **Information for debug :**
>
> >
>
> > SearchGuard : 23.1
>
> > Elasticsearch : 6.4.2
>
> > Kibana : 6.4.2
>
> > Installed and used enterprise modules : N/A
>
> > JVM version and operating system version: JDK 1.8.0.171-7.b10.el7, CentOS 7
>
> >
>
> >
>
> > **Kibana.yml :**
>
> >
>
> > server.port: 5601
>
> > server.host: “x.x.x.x”
>
> > elasticsearch.url: “[https://kubernetes.mycluster.com:30099](https://kubernetes.mycluster.com:30099)”
>
> > elasticsearch.ssl.certificate: /root/elasticsearch/k8skey/elasticsearch-admin.crt.pem
>
> > elasticsearch.ssl.key: /root/elasticsearch/k8skey/elasticsearch-admin.key
>
> > elasticsearch.ssl.keyPassphrase: “password”
>
> > elasticsearch.ssl.certificateAuthorities: [“/root/elasticsearch/ca/chain-ca.pem”]
>
> > elasticsearch.ssl.verificationMode: none
>
> > searchguard.allow\_client\_certificates: true
>
> > xpack.reporting.encryptionKey: “1234567890”
>
> > xpack.security.enabled: false
>
> > xpack.monitoring.enabled: true
>
> > xpack.graph.enabled: false
>
> > xpack.ml.enabled: false
>
> > xpack.watcher.enabled: false
>
> > searchguard.cookie.secure: false
>
> >
>
> > **Elasticsearch.yml :**
>
> >
>
> > [root@es-data-0 config]# more elasticsearch.yml
>
> > #search guard config
>
> > searchguard:
>
> > ssl:
>
> > transport:
>
> > enable\_openssl\_if\_available: true
>
> > enforce\_hostname\_verification: false
>
> > pemkey\_filepath: certs/elasticsearch-transport.key
>
> > pemkey\_password: password
>
> > pemcert\_filepath: certs/elasticsearch-transport.crt.pem
>
> > pemtrustedcas\_filepath: certs/chain-ca.pem
>
> > http:
>
> > enable\_openssl\_if\_available: true
>
> > enabled: true
>
> > pemkey\_filepath: certs/elasticsearch-http.key
>
> > pemkey\_password: password
>
> > pemcert\_filepath: certs/elasticsearch-http.crt.pem
>
> > pemtrustedcas\_filepath: certs/chain-ca.pem
>
> > clientauth\_mode: REQUIRE
>
> > authcz:
>
> > admin\_dn:
>
> > - “CN=elasticsearch-admin, OU=ICES, O=MYCLUSTER, C=US”
>
> >
>
> > xpack.security.enabled: false
>
> > xpack.monitoring.enabled: true
>
> > xpack.graph.enabled: false
>
> > xpack.watcher.enabled: false
>
> >
>
> >
>
> >
>
> > **Kibana output :**
>
> >
>
> > root@kubernetes:/usr/share/kibana/bin# ./kibana
>
> > log [03:18:08.914] [info][status][plugin:kibana@6.4.2] Status changed from uninitialized to green - Ready
>
> > log [03:18:09.019] [info][status][plugin:elasticsearch@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > log [03:18:09.025] [info][status][plugin:xpack\_main@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > log [03:18:09.034] [info][status][plugin:searchprofiler@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > log [03:18:09.041] [info][status][plugin:tilemap@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > log [03:18:09.052] [info][status][plugin:license\_management@6.4.2] Status changed from uninitialized to green - Ready
>
> > log [03:18:09.060] [info][status][plugin:index\_management@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > log [03:18:09.320] [info][status][plugin:timelion@6.4.2] Status changed from uninitialized to green - Ready
>
> > log [03:18:09.340] [info][status][plugin:monitoring@6.4.2] Status changed from uninitialized to green - Ready
>
> > log [03:18:09.344] [info][status][plugin:grokdebugger@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > log [03:18:09.349] [info][status][plugin:dashboard\_mode@6.4.2] Status changed from uninitialized to green - Ready
>
> > log [03:18:09.353] [info][status][plugin:logstash@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > log [03:18:09.405] [info][status][plugin:apm@6.4.2] Status changed from uninitialized to green - Ready
>
> > log [03:18:09.429] [info][status][plugin:console@6.4.2] Status changed from uninitialized to green - Ready
>
> > log [03:18:09.434] [info][status][plugin:console\_extensions@6.4.2] Status changed from uninitialized to green - Ready
>
> > log [03:18:09.440] [info][status][plugin:notifications@6.4.2] Status changed from uninitialized to green - Ready
>
> > log [03:18:09.610] [info][status][plugin:searchguard@6.4.2-15] Status changed from uninitialized to yellow - Initialising Search Guard authentication plugin.
>
> > log [03:18:09.612] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Default cookie password detected, please set a password in kibana.yml by setting ‘searchguard.cookie.password’ (min. 32 characters).
>
> > log [03:18:09.612] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - ‘searchguard.cookie.secure’ is set to false, cookies are transmitted over unsecure HTTP connection. Consider using HTTPS and set this key to ‘true’
>
> > log [03:18:09.664] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard session management enabled.
>
> > log [03:18:09.665] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard copy JWT params disabled
>
> > log [03:18:09.666] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard multitenancy disabled
>
> > log [03:18:09.685] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Routes for Search Guard configuration GUI registered. This is an Enterprise feature.
>
> > log [03:18:09.692] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard system routes registered.
>
> > log [03:18:09.693] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to green - Search Guard plugin initialised.
>
> > log [03:18:09.699] [info][status][plugin:metrics@6.4.2] Status changed from uninitialized to green - Ready
>
> > log [03:18:10.573] [info][status][plugin:reporting@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > log [03:18:10.691] [info][listening][server][http] Server running at [http://9.98.173.247:5601](http://9.98.173.247:5601)
>
> > log [03:18:11.126] [info][status][plugin:elasticsearch@6.4.2] Status changed from yellow to green - Ready
>
> > log [03:18:11.233] [info][license][xpack] Imported license information from Elasticsearch for the [data] cluster: mode: basic | status: active
>
> > log [03:18:11.241] [info][status][plugin:xpack\_main@6.4.2] Status changed from yellow to green - Ready
>
> > log [03:18:11.243] [info][status][plugin:searchprofiler@6.4.2] Status changed from yellow to green - Ready
>
> > log [03:18:11.244] [info][status][plugin:tilemap@6.4.2] Status changed from yellow to green - Ready
>
> > log [03:18:11.245] [info][status][plugin:index\_management@6.4.2] Status changed from yellow to green - Ready
>
> > log [03:18:11.247] [info][status][plugin:grokdebugger@6.4.2] Status changed from yellow to green - Ready
>
> > log [03:18:11.248] [info][status][plugin:logstash@6.4.2] Status changed from yellow to green - Ready
>
> > log [03:18:11.251] [info][status][plugin:reporting@6.4.2] Status changed from yellow to green - Ready
>
> > log [03:18:11.253] [info][kibana-monitoring][monitoring-ui] Starting monitoring stats collection
>
> > log [03:18:11.475] [info][license][xpack] Imported license information from Elasticsearch for the [monitoring] cluster: mode: basic | status: active
>
> > Elasticsearch ERROR: 2018-11-05T03:24:00Z
>
> > Error: Request error, retrying
>
> > GET [https://kubernetes.mycluster.com:30099/\_searchguard/license](https://kubernetes.mycluster.com:30099/_searchguard/license) =\> write EPROTO 140599130773312:error:14094410:SSL routines:ssl3\_read\_bytes:sslv3 alert handshake failure:…/deps/openssl/openssl/ssl/s3\_pkt.c:1498:SSL alert number 40
>
> > 140599130773312:error:1409E0E5:SSL routines:ssl3\_write\_bytes:ssl handshake failure:…/deps/openssl/openssl/ssl/s3\_pkt.c:659:
>
> >
>
> > at Log.error (/usr/share/kibana/node\_modules/elasticsearch/src/lib/log.js:225:56)
>
> > at checkRespForFailure (/usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:258:18)
>
> > at HttpConnector. (/usr/share/kibana/node\_modules/elasticsearch/src/lib/connectors/http.js:157:7)
>
> > at ClientRequest.bound (/usr/share/kibana/node\_modules/elasticsearch/node\_modules/lodash/dist/lodash.js:729:21)
>
> > at emitOne (events.js:116:13)
>
> > at ClientRequest.emit (events.js:211:7)
>
> > at TLSSocket.socketErrorListener (\_http\_client.js:387:9)
>
> > at emitOne (events.js:116:13)
>
> > at TLSSocket.emit (events.js:211:7)
>
> > at onwriteError (\_stream\_writable.js:418:12)
>
> > at onwrite (\_stream\_writable.js:440:5)
>
> > at \_destroy (internal/streams/destroy.js:39:7)
>
> > at TLSSocket.Socket.\_destroy (net.js:564:3)
>
> > at TLSSocket.destroy (internal/streams/destroy.js:32:8)
>
> > at WriteWrap.afterWrite (net.js:866:10)
>
> >
>
> > Elasticsearch WARNING: 2018-11-05T03:24:00Z
>
> > Unable to revive connection: [https://kubernetes](https://kubernetes). [mycluster.com:30099/](http://mycluster.com:30099/)
>
> >
>
> > Elasticsearch WARNING: 2018-11-05T03:24:00Z
>
> > No living connections
>
> >
>
> > error [03:24:00.574] Error: No Living connections
>
> > at sendReqWithConnection (/usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:225:15)
>
> > at next (/usr/share/kibana/node\_modules/elasticsearch/src/lib/connection\_pool.js:213:7)
>
> > at \_combinedTickCallback (internal/process/next\_tick.js:131:7)
>
> > at process.\_tickDomainCallback (internal/process/next\_tick.js:218:9)
>
> >
>
> >
>
> > \*\*Elasticsearch log : \*\*
>
> >
>
> > [2018-10-29T08:48:51,625][INFO][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Info: SearchGuardLicense [uid=00000000-0000-0000-0000-000000000000, type=TRIAL, features=[COMPLIANCE], issueDate=2018-10-25, expiryDate=2018-12-25, issuedTo=The world, issuer=floragunn GmbH, startDate=2018-10-25, majorVersion=6, clusterName=\*, allowedNodeCount=2147483647, msgs=, expiresInDays=56, isExpired=false, valid=true, action=, prodUsage=Yes, one cluster with all commercial features and unlimited nodes per cluster., clusterService=org.elasticsearch.cluster.service.ClusterService@744c2d63, getMsgs()=, getExpiresInDays()=56, isExpired()=false, isValid()=true, getAction()=, getProdUsage()=Yes, one cluster with all commercial features and unlimited nodes per cluster.]
>
> > [2018-10-29T08:48:51,626][INFO][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Type: TRIAL, valid
>
> > [2018-10-29T08:48:51,626][INFO][c.f.s.c.IndexBaseConfigurationRepository] Node ‘es-client-666c88cfb5-dmw7c’ initialized
>
> > [2018-10-29T08:50:38,214][INFO][o.e.c.s.ClusterApplierService] [es-client-666c88cfb5-dmw7c] added {{es-master-8ff849f9-j5z8l}{hOYJ3NcIQLGIniyHcUy14A}{D9WeiTgCT\_CiH1T9CDx6vg}{192.168.1.50}{192.168.1.50:9300}{ml.machine\_memory=8352608256, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {es-master-8ff849f9-d7s6g}{\_8vGh29LSFWHxxo\_TZLcsQ}{iVusXa7vS8C7YHsyjXhltA}{192.168.2.45}{192.168.2.45:9300}{ml.machine\_memory=8352608256, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true} committed version [4303]])
>
> > [2018-11-02T07:13:58,878][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [es-client-666c88cfb5-dmw7c] SSL Problem error:140890C7:SSL routines:ssl3\_get\_client\_certificate:peer did not return a certificate
>
> > javax.net.ssl.SSLHandshakeException: error:140890C7:SSL routines:ssl3\_get\_client\_certificate:peer did not return a certificate
>
> > at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1104) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1064) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1170) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
>
> > at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_171]
>
> >
>
> >
>
> > \*\*Certificate info : \*\*
>
> >
>
> > openssl x509 -in elasticsearch-admin.crt.pem -noout -text
>
> > Certificate:
>
> > Data:
>
> > Version: 3 (0x2)
>
> > Serial Number: 10 (0xa)
>
> > Signature Algorithm: sha256WithRSAEncryption
>
> > Issuer: DC=ICES, O=MYCLUSTER, OU=ICES, CN=ICES, Signing CA
>
> > Validity
>
> > Not Before: Feb 28 04:06:01 2018 GMT
>
> > Not After : Feb 28 04:06:01 2020 GMT
>
> > Subject: C=US, O=MYCLUSTER, OU=ICES, CN=elasticsearch-admin
>
> > Subject Public Key Info:
>
> > Public Key Algorithm: rsaEncryption
>
> > Public-Key: (2048 bit)
>
> > Modulus:
>
> > 00:db:15:db:b0:06:cc:de:c4:2d:aa:77:c8:08:f7:
>
> > 31:20:6b:2b:78:45:00:02:89:9a:a3:7c:dc:82:3b:
>
> > 65:ca:3f:31:01:17:b3:8f:19:dd:12:8a:8b:dc:fe:
>
> > ae:65:e9:d5:22:7e:93:ac:53:f8:30:98:91:92:ef:
>
> > 19:8d:20:aa:2e:a9:20:68:05:61:06:ce:b6:e5:01:
>
> > b0:5a:3a:da:7c:1c:e7:e5:2d:c3:99:45:60:37:10:
>
> > …
>
> > …
>
> > Exponent: 65537 (0x10001)
>
> > X509v3 extensions:
>
> > X509v3 Key Usage: critical
>
> > Digital Signature, Key Encipherment
>
> > X509v3 Basic Constraints:
>
> > CA:FALSE
>
> > X509v3 Extended Key Usage:
>
> > TLS Web Server Authentication, TLS Web Client Authentication
>
> > X509v3 Subject Key Identifier:
>
> > C0:35:91:E3:2E:DC:2E:95:B9:D3:FC:86:9C:82:C5:86:DC:39:38:67
>
> > X509v3 Authority Key Identifier:
>
> > keyid:6C:B5:45:52:EF:CA:62:0E:8D:36:9B:C7:17:9E:F6:BC:D7:90:13:64
---
### Author:  [@searchguard\_google\_group](https://forum.search-guard.com/u/searchguard_google_group)
#### Post date: [November 6, 2018, 10:14pm UTC](https://forum.search-guard.com/t/elasticsearh-kibana-6-4-2-searchguard-23-1-sslv3-handshake-failure/1176/3 "2018-11-06T22:14:01Z")
Before continuing pls. read [Installing the Search Guard Kibana Plugin | Security for Elasticsearch | Search Guard](https://docs.search-guard.com/latest/kibana-plugin-installation#client-certificates-elasticsearchsslcertificate) to make sure you understand that “If the certificate is an admin certificate, this means that all actions from all users will be allowed, regardless of other authorization settings.”
Can you post the output of
curl -vvv -k --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key -XGET [https://kubernetes.mycluster.com:30099/\_searchguard/authinfo](https://kubernetes.mycluster.com:30099/_searchguard/authinfo)
and
curl -vvv --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key --cacert /root/elasticsearch/ca/chain-ca.pem -XGET [https://kubernetes.mycluster.com:30099/\_searchguard/authinfo](https://kubernetes.mycluster.com:30099/_searchguard/authinfo)
> **···**
>
> On Monday, 5 November 2018 05:23:04 UTC+1, Yong Wang wrote:
>
> > Dear Sir or Madam:
>
> >
>
> > I deployed an Elasticsearch cluster with SearchGuard by using latest release, met a ssl handshake error when want to connect Kibana to my ES cluster. It blocked me for several days, still can’t resolve it. Could you help check and give me some hints? Thanks!
>
> >
>
> > I enabled the SSL encryption and Client Certification based authentication, the same config can work in my previous cluster ( ES 5.5.1 + SearchGuard + Kibana), but not work in the new cluster.
>
> >
>
> > When try to open Kibana page, it will show error message “The Search Guard license information could not be loaded. Please contact your system administrator.”, the the Kibana will report error like below :
>
> >
>
> > Elasticsearch ERROR: 2018-11-05T03:24:00Z
>
> > Error: Request error, retrying
>
> > GET [https://kubernetes.mycluster.com:30099/\_searchguard/license](https://kubernetes.mycluster.com:30099/_searchguard/license) =\> write EPROTO 140599130773312:error:14094410:SSL routines:ssl3\_read\_bytes:sslv3 alert handshake failure:…/deps/openssl/openssl/ssl/s3\_pkt.c:1498:SSL alert number 40
>
> > 140599130773312:error:1409E0E5:SSL routines:ssl3\_write\_bytes:ssl handshake failure:…/deps/openssl/openssl/ssl/s3\_pkt.c:659:
>
> >
>
> >
>
> > Elasticsearch WARNING: 2018-11-05T03:24:00Z
>
> > Unable to revive connection: [https://kubernetes.mycluster.com:30099/](https://kubernetes.mycluster.com:30099/)
>
> >
>
> > Elasticsearch WARNING: 2018-11-05T03:24:00Z
>
> > No living connections
>
> >
>
> > But if I use the curl command with same cert/key, can get the license info correctly, like below :
>
> >
>
> > \*\*Curl command output : \*\*
>
> > curl --insecure --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key --cacert elasticsearch-admin.crt.pem -XGET [https://kubernetes.mycluster.com:30099/\_searchguard/license](https://kubernetes.mycluster.com:30099/_searchguard/license)
>
> > {“\_nodes”:{“total”:9,“successful”:9,“failed”:0},“cluster\_name”:“ICES”,“sg\_license”:{“uid”:“00000000-0000-0000-0000-000000000000”,“type”:“TRIAL”,“features”:[“COMPLIANCE”],“issue\_date”:“2018-10-25”,“expiry\_date”:“2018-12-25”,“issued\_to”:“The world”,“issuer”:“floragunn GmbH”,“start\_date”:“2018-10-25”,“major\_version”:6,“cluster\_name”:“\*”,“msgs”:,“expiry\_in\_days”:52,“is\_expired”:false,“is\_valid”:true,“action”:“”,…
>
> >
>
> >
>
> > Could you help check and give me some hints?
>
> >
>
> >
>
> > **Information for debug :**
>
> >
>
> > SearchGuard : 23.1
>
> > Elasticsearch : 6.4.2
>
> > Kibana : 6.4.2
>
> > Installed and used enterprise modules : N/A
>
> > JVM version and operating system version: JDK 1.8.0.171-7.b10.el7, CentOS 7
>
> >
>
> >
>
> > **Kibana.yml :**
>
> >
>
> > server.port: 5601
>
> > server.host: “x.x.x.x”
>
> > elasticsearch.url: “[https://kubernetes.mycluster.com:30099](https://kubernetes.mycluster.com:30099)”
>
> > elasticsearch.ssl.certificate: /root/elasticsearch/k8skey/elasticsearch-admin.crt.pem
>
> > elasticsearch.ssl.key: /root/elasticsearch/k8skey/elasticsearch-admin.key
>
> > elasticsearch.ssl.keyPassphrase: “password”
>
> > elasticsearch.ssl.certificateAuthorities: [“/root/elasticsearch/ca/chain-ca.pem”]
>
> > elasticsearch.ssl.verificationMode: none
>
> > searchguard.allow\_client\_certificates: true
>
> > xpack.reporting.encryptionKey: “1234567890”
>
> > xpack.security.enabled: false
>
> > xpack.monitoring.enabled: true
>
> > xpack.graph.enabled: false
>
> > xpack.ml.enabled: false
>
> > xpack.watcher.enabled: false
>
> > searchguard.cookie.secure: false
>
> >
>
> > **Elasticsearch.yml :**
>
> >
>
> > [root@es-data-0 config]# more elasticsearch.yml
>
> > #search guard config
>
> > searchguard:
>
> > ssl:
>
> > transport:
>
> > enable\_openssl\_if\_available: true
>
> > enforce\_hostname\_verification: false
>
> > pemkey\_filepath: certs/elasticsearch-transport.key
>
> > pemkey\_password: password
>
> > pemcert\_filepath: certs/elasticsearch-transport.crt.pem
>
> > pemtrustedcas\_filepath: certs/chain-ca.pem
>
> > http:
>
> > enable\_openssl\_if\_available: true
>
> > enabled: true
>
> > pemkey\_filepath: certs/elasticsearch-http.key
>
> > pemkey\_password: password
>
> > pemcert\_filepath: certs/elasticsearch-http.crt.pem
>
> > pemtrustedcas\_filepath: certs/chain-ca.pem
>
> > clientauth\_mode: REQUIRE
>
> > authcz:
>
> > admin\_dn:
>
> > - “CN=elasticsearch-admin, OU=ICES, O=MYCLUSTER, C=US”
>
> >
>
> > xpack.security.enabled: false
>
> > xpack.monitoring.enabled: true
>
> > xpack.graph.enabled: false
>
> > xpack.watcher.enabled: false
>
> >
>
> >
>
> >
>
> > **Kibana output :**
>
> >
>
> > root@kubernetes:/usr/share/kibana/bin# ./kibana
>
> > log [03:18:08.914] [info][status][plugin:kibana@6.4.2] Status changed from uninitialized to green - Ready
>
> > log [03:18:09.019] [info][status][plugin:elasticsearch@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > log [03:18:09.025] [info][status][plugin:xpack\_main@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > log [03:18:09.034] [info][status][plugin:searchprofiler@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > log [03:18:09.041] [info][status][plugin:tilemap@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > log [03:18:09.052] [info][status][plugin:license\_management@6.4.2] Status changed from uninitialized to green - Ready
>
> > log [03:18:09.060] [info][status][plugin:index\_management@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > log [03:18:09.320] [info][status][plugin:timelion@6.4.2] Status changed from uninitialized to green - Ready
>
> > log [03:18:09.340] [info][status][plugin:monitoring@6.4.2] Status changed from uninitialized to green - Ready
>
> > log [03:18:09.344] [info][status][plugin:grokdebugger@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > log [03:18:09.349] [info][status][plugin:dashboard\_mode@6.4.2] Status changed from uninitialized to green - Ready
>
> > log [03:18:09.353] [info][status][plugin:logstash@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > log [03:18:09.405] [info][status][plugin:apm@6.4.2] Status changed from uninitialized to green - Ready
>
> > log [03:18:09.429] [info][status][plugin:console@6.4.2] Status changed from uninitialized to green - Ready
>
> > log [03:18:09.434] [info][status][plugin:console\_extensions@6.4.2] Status changed from uninitialized to green - Ready
>
> > log [03:18:09.440] [info][status][plugin:notifications@6.4.2] Status changed from uninitialized to green - Ready
>
> > log [03:18:09.610] [info][status][plugin:searchguard@6.4.2-15] Status changed from uninitialized to yellow - Initialising Search Guard authentication plugin.
>
> > log [03:18:09.612] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Default cookie password detected, please set a password in kibana.yml by setting ‘searchguard.cookie.password’ (min. 32 characters).
>
> > log [03:18:09.612] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - ‘searchguard.cookie.secure’ is set to false, cookies are transmitted over unsecure HTTP connection. Consider using HTTPS and set this key to ‘true’
>
> > log [03:18:09.664] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard session management enabled.
>
> > log [03:18:09.665] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard copy JWT params disabled
>
> > log [03:18:09.666] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard multitenancy disabled
>
> > log [03:18:09.685] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Routes for Search Guard configuration GUI registered. This is an Enterprise feature.
>
> > log [03:18:09.692] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard system routes registered.
>
> > log [03:18:09.693] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to green - Search Guard plugin initialised.
>
> > log [03:18:09.699] [info][status][plugin:metrics@6.4.2] Status changed from uninitialized to green - Ready
>
> > log [03:18:10.573] [info][status][plugin:reporting@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > log [03:18:10.691] [info][listening][server][http] Server running at [http://9.98.173.247:5601](http://9.98.173.247:5601)
>
> > log [03:18:11.126] [info][status][plugin:elasticsearch@6.4.2] Status changed from yellow to green - Ready
>
> > log [03:18:11.233] [info][license][xpack] Imported license information from Elasticsearch for the [data] cluster: mode: basic | status: active
>
> > log [03:18:11.241] [info][status][plugin:xpack\_main@6.4.2] Status changed from yellow to green - Ready
>
> > log [03:18:11.243] [info][status][plugin:searchprofiler@6.4.2] Status changed from yellow to green - Ready
>
> > log [03:18:11.244] [info][status][plugin:tilemap@6.4.2] Status changed from yellow to green - Ready
>
> > log [03:18:11.245] [info][status][plugin:index\_management@6.4.2] Status changed from yellow to green - Ready
>
> > log [03:18:11.247] [info][status][plugin:grokdebugger@6.4.2] Status changed from yellow to green - Ready
>
> > log [03:18:11.248] [info][status][plugin:logstash@6.4.2] Status changed from yellow to green - Ready
>
> > log [03:18:11.251] [info][status][plugin:reporting@6.4.2] Status changed from yellow to green - Ready
>
> > log [03:18:11.253] [info][kibana-monitoring][monitoring-ui] Starting monitoring stats collection
>
> > log [03:18:11.475] [info][license][xpack] Imported license information from Elasticsearch for the [monitoring] cluster: mode: basic | status: active
>
> > Elasticsearch ERROR: 2018-11-05T03:24:00Z
>
> > Error: Request error, retrying
>
> > GET [https://kubernetes.mycluster.com:30099/\_searchguard/license](https://kubernetes.mycluster.com:30099/_searchguard/license) =\> write EPROTO 140599130773312:error:14094410:SSL routines:ssl3\_read\_bytes:sslv3 alert handshake failure:…/deps/openssl/openssl/ssl/s3\_pkt.c:1498:SSL alert number 40
>
> > 140599130773312:error:1409E0E5:SSL routines:ssl3\_write\_bytes:ssl handshake failure:…/deps/openssl/openssl/ssl/s3\_pkt.c:659:
>
> >
>
> > at Log.error (/usr/share/kibana/node\_modules/elasticsearch/src/lib/log.js:225:56)
>
> > at checkRespForFailure (/usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:258:18)
>
> > at HttpConnector. (/usr/share/kibana/node\_modules/elasticsearch/src/lib/connectors/http.js:157:7)
>
> > at ClientRequest.bound (/usr/share/kibana/node\_modules/elasticsearch/node\_modules/lodash/dist/lodash.js:729:21)
>
> > at emitOne (events.js:116:13)
>
> > at ClientRequest.emit (events.js:211:7)
>
> > at TLSSocket.socketErrorListener (\_http\_client.js:387:9)
>
> > at emitOne (events.js:116:13)
>
> > at TLSSocket.emit (events.js:211:7)
>
> > at onwriteError (\_stream\_writable.js:418:12)
>
> > at onwrite (\_stream\_writable.js:440:5)
>
> > at \_destroy (internal/streams/destroy.js:39:7)
>
> > at TLSSocket.Socket.\_destroy (net.js:564:3)
>
> > at TLSSocket.destroy (internal/streams/destroy.js:32:8)
>
> > at WriteWrap.afterWrite (net.js:866:10)
>
> >
>
> > Elasticsearch WARNING: 2018-11-05T03:24:00Z
>
> > Unable to revive connection: [https://kubernetes](https://kubernetes). [mycluster.com:30099/](http://mycluster.com:30099/)
>
> >
>
> > Elasticsearch WARNING: 2018-11-05T03:24:00Z
>
> > No living connections
>
> >
>
> > error [03:24:00.574] Error: No Living connections
>
> > at sendReqWithConnection (/usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:225:15)
>
> > at next (/usr/share/kibana/node\_modules/elasticsearch/src/lib/connection\_pool.js:213:7)
>
> > at \_combinedTickCallback (internal/process/next\_tick.js:131:7)
>
> > at process.\_tickDomainCallback (internal/process/next\_tick.js:218:9)
>
> >
>
> >
>
> > \*\*Elasticsearch log : \*\*
>
> >
>
> > [2018-10-29T08:48:51,625][INFO][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Info: SearchGuardLicense [uid=00000000-0000-0000-0000-000000000000, type=TRIAL, features=[COMPLIANCE], issueDate=2018-10-25, expiryDate=2018-12-25, issuedTo=The world, issuer=floragunn GmbH, startDate=2018-10-25, majorVersion=6, clusterName=\*, allowedNodeCount=2147483647, msgs=, expiresInDays=56, isExpired=false, valid=true, action=, prodUsage=Yes, one cluster with all commercial features and unlimited nodes per cluster., clusterService=org.elasticsearch.cluster.service.ClusterService@744c2d63, getMsgs()=, getExpiresInDays()=56, isExpired()=false, isValid()=true, getAction()=, getProdUsage()=Yes, one cluster with all commercial features and unlimited nodes per cluster.]
>
> > [2018-10-29T08:48:51,626][INFO][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Type: TRIAL, valid
>
> > [2018-10-29T08:48:51,626][INFO][c.f.s.c.IndexBaseConfigurationRepository] Node ‘es-client-666c88cfb5-dmw7c’ initialized
>
> > [2018-10-29T08:50:38,214][INFO][o.e.c.s.ClusterApplierService] [es-client-666c88cfb5-dmw7c] added {{es-master-8ff849f9-j5z8l}{hOYJ3NcIQLGIniyHcUy14A}{D9WeiTgCT\_CiH1T9CDx6vg}{192.168.1.50}{192.168.1.50:9300}{ml.machine\_memory=8352608256, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {es-master-8ff849f9-d7s6g}{\_8vGh29LSFWHxxo\_TZLcsQ}{iVusXa7vS8C7YHsyjXhltA}{192.168.2.45}{192.168.2.45:9300}{ml.machine\_memory=8352608256, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true} committed version [4303]])
>
> > [2018-11-02T07:13:58,878][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [es-client-666c88cfb5-dmw7c] SSL Problem error:140890C7:SSL routines:ssl3\_get\_client\_certificate:peer did not return a certificate
>
> > javax.net.ssl.SSLHandshakeException: error:140890C7:SSL routines:ssl3\_get\_client\_certificate:peer did not return a certificate
>
> > at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1104) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1064) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1170) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
>
> > at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
>
> > at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_171]
>
> >
>
> >
>
> > \*\*Certificate info : \*\*
>
> >
>
> > openssl x509 -in elasticsearch-admin.crt.pem -noout -text
>
> > Certificate:
>
> > Data:
>
> > Version: 3 (0x2)
>
> > Serial Number: 10 (0xa)
>
> > Signature Algorithm: sha256WithRSAEncryption
>
> > Issuer: DC=ICES, O=MYCLUSTER, OU=ICES, CN=ICES, Signing CA
>
> > Validity
>
> > Not Before: Feb 28 04:06:01 2018 GMT
>
> > Not After : Feb 28 04:06:01 2020 GMT
>
> > Subject: C=US, O=MYCLUSTER, OU=ICES, CN=elasticsearch-admin
>
> > Subject Public Key Info:
>
> > Public Key Algorithm: rsaEncryption
>
> > Public-Key: (2048 bit)
>
> > Modulus:
>
> > 00:db:15:db:b0:06:cc:de:c4:2d:aa:77:c8:08:f7:
>
> > 31:20:6b:2b:78:45:00:02:89:9a:a3:7c:dc:82:3b:
>
> > 65:ca:3f:31:01:17:b3:8f:19:dd:12:8a:8b:dc:fe:
>
> > ae:65:e9:d5:22:7e:93:ac:53:f8:30:98:91:92:ef:
>
> > 19:8d:20:aa:2e:a9:20:68:05:61:06:ce:b6:e5:01:
>
> > b0:5a:3a:da:7c:1c:e7:e5:2d:c3:99:45:60:37:10:
>
> > …
>
> > …
>
> > Exponent: 65537 (0x10001)
>
> > X509v3 extensions:
>
> > X509v3 Key Usage: critical
>
> > Digital Signature, Key Encipherment
>
> > X509v3 Basic Constraints:
>
> > CA:FALSE
>
> > X509v3 Extended Key Usage:
>
> > TLS Web Server Authentication, TLS Web Client Authentication
>
> > X509v3 Subject Key Identifier:
>
> > C0:35:91:E3:2E:DC:2E:95:B9:D3:FC:86:9C:82:C5:86:DC:39:38:67
>
> > X509v3 Authority Key Identifier:
>
> > keyid:6C:B5:45:52:EF:CA:62:0E:8D:36:9B:C7:17:9E:F6:BC:D7:90:13:64
---
### Author:  [@searchguard\_google\_group](https://forum.search-guard.com/u/searchguard_google_group)
#### Post date: [November 6, 2018, 10:15pm UTC](https://forum.search-guard.com/t/elasticsearh-kibana-6-4-2-searchguard-23-1-sslv3-handshake-failure/1176/4 "2018-11-06T22:15:05Z")
Your openssl command returns an error: verify error:num=20:unable to get local issuer certificate
> **···**
>
> On Monday, 5 November 2018 06:21:23 UTC+1, Yong Wang wrote:
>
> > And openssl client also work with same certs/keys :
>
> > root@kubernetes:/usr/share/kibana/bin# openssl s\_client -connect [kubernetes.mycluster.com:30099](http://kubernetes.mycluster.com:30099) -servername x.x.x.x -cert /root/elasticsearch/k8skey/elasticsearch-admin.crt.pem -key /root/elasticsearch/k8skey/elasticsearch-admin.key -CAfile /root/elasticsearch/ca/chain-ca.pem
>
> > Enter pass phrase for /root/elasticsearch/k8skey/elasticsearch-admin.key:
>
> > CONNECTED(00000003)
>
> > depth=1 DC = ICES, O = MYCLUSTER, OU = ICES, CN = “ICES, Signing CA”
>
> > verify error:num=20:unable to get local issuer certificate
>
> > * * *
>
> > Certificate chain
>
> > 0 s:/C=US/O=MYCLUSTER/OU=ICES/CN=elasticsearch-http
>
> > i:/DC=ICES/O=MYCLUSTER/OU=ICES/CN=ICES, Signing CA
>
> > 1 s:/DC=ICES/O=MYCLUSTER M/OU=ICES/CN=ICES, Signing CA
>
> > i:/DC=ICES/O=MYCLUSTER/OU=ICES/CN=ICES, Root CA
>
> > * * *
>
> > Server certificate
>
> > -----BEGIN CERTIFICATE-----
>
> > MIIDqzCCApOgAwIBAgIBATANBgkqhkiG9w0BAQsFADBOMRQwEgYKCZImiZPyLGQB
>
> > GRYESUNFUzEMMAoGA1UECgwDSUJNMQ0wCwYDVQQLDARJQ0VTMRkwFwYDVQQDDBBJ
>
> > …
>
> > …
>
> > -----END CERTIFICATE-----
>
> > subject=/C=US/O=MYCLUSTER/OU=ICES/CN=elasticsearch-http
>
> > issuer=/DC=ICES/O=MYCLUSTER/OU=ICES/CN=ICES, Signing CA
>
> > * * *
>
> > Acceptable client certificate CA names
>
> > /DC=ICES/O=MYCLUSTER/OU=ICES/CN=ICES, Root CA
>
> > /DC=ICES/O=MYCLUSTER/OU=ICES/CN=ICES, Signing CA
>
> > Client Certificate Types: RSA sign, DSA sign, ECDSA sign
>
> > Requested Signature Algorithms: RSA+SHA512:DSA+SHA512:ECDSA+SHA512:RSA+SHA384:DSA+SHA384:ECDSA+SHA384:RSA+SHA256:DSA+SHA256:ECDSA+SHA256:RSA+SHA224:DSA+SHA224:ECDSA+SHA224:RSA+SHA1:DSA+SHA1:ECDSA+SHA1
>
> > Shared Requested Signature Algorithms: RSA+SHA512:DSA+SHA512:ECDSA+SHA512:RSA+SHA384:DSA+SHA384:ECDSA+SHA384:RSA+SHA256:DSA+SHA256:ECDSA+SHA256:RSA+SHA224:DSA+SHA224:ECDSA+SHA224:RSA+SHA1:DSA+SHA1:ECDSA+SHA1
>
> > Peer signing digest: SHA512
>
> > Server Temp Key: ECDH, P-256, 256 bits
>
> > * * *
>
> > SSL handshake has read 2594 bytes and written 1689 bytes
>
> > * * *
>
> > New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-SHA256
>
> > Server public key is 2048 bit
>
> > Secure Renegotiation IS supported
>
> > Compression: NONE
>
> > Expansion: NONE
>
> > No ALPN negotiated
>
> > SSL-Session:
>
> > Protocol : TLSv1.2
>
> > Cipher : ECDHE-RSA-AES128-SHA256
>
> > Session-ID: 715D57B05C520803D87671DC24DB773C954ABC881374D4EFF74B2019BC1E77E3
>
> > Session-ID-ctx:
>
> > Master-Key: D79B276C3E13F9F2E4D9903FB6542046155546E132EC2C406E2C20CF40317960A2EEE7573B52E814940F2A9D48C7885B
>
> > Key-Arg : None
>
> > PSK identity: None
>
> > PSK identity hint: None
>
> > SRP username: None
>
> > Start Time: 1541395055
>
> > Timeout : 300 (sec)
>
> > Verify return code: 20 (unable to get local issuer certificate)
>
> > * * *
>
> > On Monday, November 5, 2018 at 12:23:04 PM UTC+8, Yong Wang wrote:
> >
> > > Dear Sir or Madam:
>
> > >
>
> > > I deployed an Elasticsearch cluster with SearchGuard by using latest release, met a ssl handshake error when want to connect Kibana to my ES cluster. It blocked me for several days, still can’t resolve it. Could you help check and give me some hints? Thanks!
>
> > >
>
> > > I enabled the SSL encryption and Client Certification based authentication, the same config can work in my previous cluster ( ES 5.5.1 + SearchGuard + Kibana), but not work in the new cluster.
>
> > >
>
> > > When try to open Kibana page, it will show error message “The Search Guard license information could not be loaded. Please contact your system administrator.”, the the Kibana will report error like below :
>
> > >
>
> > > Elasticsearch ERROR: 2018-11-05T03:24:00Z
>
> > > Error: Request error, retrying
>
> > > GET [https://kubernetes.mycluster.com:30099/\_searchguard/license](https://kubernetes.mycluster.com:30099/_searchguard/license) =\> write EPROTO 140599130773312:error:14094410:SSL routines:ssl3\_read\_bytes:sslv3 alert handshake failure:…/deps/openssl/openssl/ssl/s3\_pkt.c:1498:SSL alert number 40
>
> > > 140599130773312:error:1409E0E5:SSL routines:ssl3\_write\_bytes:ssl handshake failure:…/deps/openssl/openssl/ssl/s3\_pkt.c:659:
>
> > >
>
> > >
>
> > > Elasticsearch WARNING: 2018-11-05T03:24:00Z
>
> > > Unable to revive connection: [https://kubernetes.mycluster.com:30099/](https://kubernetes.mycluster.com:30099/)
>
> > >
>
> > > Elasticsearch WARNING: 2018-11-05T03:24:00Z
>
> > > No living connections
>
> > >
>
> > > But if I use the curl command with same cert/key, can get the license info correctly, like below :
>
> > >
>
> > > \*\*Curl command output : \*\*
>
> > > curl --insecure --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key --cacert elasticsearch-admin.crt.pem -XGET [https://kubernetes.mycluster.com:30099/\_searchguard/license](https://kubernetes.mycluster.com:30099/_searchguard/license)
>
> > > {“\_nodes”:{“total”:9,“successful”:9,“failed”:0},“cluster\_name”:“ICES”,“sg\_license”:{“uid”:“00000000-0000-0000-0000-000000000000”,“type”:“TRIAL”,“features”:[“COMPLIANCE”],“issue\_date”:“2018-10-25”,“expiry\_date”:“2018-12-25”,“issued\_to”:“The world”,“issuer”:“floragunn GmbH”,“start\_date”:“2018-10-25”,“major\_version”:6,“cluster\_name”:“\*”,“msgs”:,“expiry\_in\_days”:52,“is\_expired”:false,“is\_valid”:true,“action”:“”,…
>
> > >
>
> > >
>
> > > Could you help check and give me some hints?
>
> > >
>
> > >
>
> > > **Information for debug :**
>
> > >
>
> > > SearchGuard : 23.1
>
> > > Elasticsearch : 6.4.2
>
> > > Kibana : 6.4.2
>
> > > Installed and used enterprise modules : N/A
>
> > > JVM version and operating system version: JDK 1.8.0.171-7.b10.el7, CentOS 7
>
> > >
>
> > >
>
> > > **Kibana.yml :**
>
> > >
>
> > > server.port: 5601
>
> > > server.host: “x.x.x.x”
>
> > > elasticsearch.url: “[https://kubernetes.mycluster.com:30099](https://kubernetes.mycluster.com:30099)”
>
> > > elasticsearch.ssl.certificate: /root/elasticsearch/k8skey/elasticsearch-admin.crt.pem
>
> > > elasticsearch.ssl.key: /root/elasticsearch/k8skey/elasticsearch-admin.key
>
> > > elasticsearch.ssl.keyPassphrase: “password”
>
> > > elasticsearch.ssl.certificateAuthorities: [“/root/elasticsearch/ca/chain-ca.pem”]
>
> > > elasticsearch.ssl.verificationMode: none
>
> > > searchguard.allow\_client\_certificates: true
>
> > > xpack.reporting.encryptionKey: “1234567890”
>
> > > xpack.security.enabled: false
>
> > > xpack.monitoring.enabled: true
>
> > > xpack.graph.enabled: false
>
> > > xpack.ml.enabled: false
>
> > > xpack.watcher.enabled: false
>
> > > searchguard.cookie.secure: false
>
> > >
>
> > > **Elasticsearch.yml :**
>
> > >
>
> > > [root@es-data-0 config]# more elasticsearch.yml
>
> > > #search guard config
>
> > > searchguard:
>
> > > ssl:
>
> > > transport:
>
> > > enable\_openssl\_if\_available: true
>
> > > enforce\_hostname\_verification: false
>
> > > pemkey\_filepath: certs/elasticsearch-transport.key
>
> > > pemkey\_password: password
>
> > > pemcert\_filepath: certs/elasticsearch-transport.crt.pem
>
> > > pemtrustedcas\_filepath: certs/chain-ca.pem
>
> > > http:
>
> > > enable\_openssl\_if\_available: true
>
> > > enabled: true
>
> > > pemkey\_filepath: certs/elasticsearch-http.key
>
> > > pemkey\_password: password
>
> > > pemcert\_filepath: certs/elasticsearch-http.crt.pem
>
> > > pemtrustedcas\_filepath: certs/chain-ca.pem
>
> > > clientauth\_mode: REQUIRE
>
> > > authcz:
>
> > > admin\_dn:
>
> > > - “CN=elasticsearch-admin, OU=ICES, O=MYCLUSTER, C=US”
>
> > >
>
> > > xpack.security.enabled: false
>
> > > xpack.monitoring.enabled: true
>
> > > xpack.graph.enabled: false
>
> > > xpack.watcher.enabled: false
>
> > >
>
> > >
>
> > >
>
> > > **Kibana output :**
>
> > >
>
> > > root@kubernetes:/usr/share/kibana/bin# ./kibana
>
> > > log [03:18:08.914] [info][status][plugin:kibana@6.4.2] Status changed from uninitialized to green - Ready
>
> > > log [03:18:09.019] [info][status][plugin:elasticsearch@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > > log [03:18:09.025] [info][status][plugin:xpack\_main@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > > log [03:18:09.034] [info][status][plugin:searchprofiler@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > > log [03:18:09.041] [info][status][plugin:tilemap@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > > log [03:18:09.052] [info][status][plugin:license\_management@6.4.2] Status changed from uninitialized to green - Ready
>
> > > log [03:18:09.060] [info][status][plugin:index\_management@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > > log [03:18:09.320] [info][status][plugin:timelion@6.4.2] Status changed from uninitialized to green - Ready
>
> > > log [03:18:09.340] [info][status][plugin:monitoring@6.4.2] Status changed from uninitialized to green - Ready
>
> > > log [03:18:09.344] [info][status][plugin:grokdebugger@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > > log [03:18:09.349] [info][status][plugin:dashboard\_mode@6.4.2] Status changed from uninitialized to green - Ready
>
> > > log [03:18:09.353] [info][status][plugin:logstash@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > > log [03:18:09.405] [info][status][plugin:apm@6.4.2] Status changed from uninitialized to green - Ready
>
> > > log [03:18:09.429] [info][status][plugin:console@6.4.2] Status changed from uninitialized to green - Ready
>
> > > log [03:18:09.434] [info][status][plugin:console\_extensions@6.4.2] Status changed from uninitialized to green - Ready
>
> > > log [03:18:09.440] [info][status][plugin:notifications@6.4.2] Status changed from uninitialized to green - Ready
>
> > > log [03:18:09.610] [info][status][plugin:searchguard@6.4.2-15] Status changed from uninitialized to yellow - Initialising Search Guard authentication plugin.
>
> > > log [03:18:09.612] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Default cookie password detected, please set a password in kibana.yml by setting ‘searchguard.cookie.password’ (min. 32 characters).
>
> > > log [03:18:09.612] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - ‘searchguard.cookie.secure’ is set to false, cookies are transmitted over unsecure HTTP connection. Consider using HTTPS and set this key to ‘true’
>
> > > log [03:18:09.664] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard session management enabled.
>
> > > log [03:18:09.665] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard copy JWT params disabled
>
> > > log [03:18:09.666] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard multitenancy disabled
>
> > > log [03:18:09.685] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Routes for Search Guard configuration GUI registered. This is an Enterprise feature.
>
> > > log [03:18:09.692] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard system routes registered.
>
> > > log [03:18:09.693] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to green - Search Guard plugin initialised.
>
> > > log [03:18:09.699] [info][status][plugin:metrics@6.4.2] Status changed from uninitialized to green - Ready
>
> > > log [03:18:10.573] [info][status][plugin:reporting@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > > log [03:18:10.691] [info][listening][server][http] Server running at [http://9.98.173.247:5601](http://9.98.173.247:5601)
>
> > > log [03:18:11.126] [info][status][plugin:elasticsearch@6.4.2] Status changed from yellow to green - Ready
>
> > > log [03:18:11.233] [info][license][xpack] Imported license information from Elasticsearch for the [data] cluster: mode: basic | status: active
>
> > > log [03:18:11.241] [info][status][plugin:xpack\_main@6.4.2] Status changed from yellow to green - Ready
>
> > > log [03:18:11.243] [info][status][plugin:searchprofiler@6.4.2] Status changed from yellow to green - Ready
>
> > > log [03:18:11.244] [info][status][plugin:tilemap@6.4.2] Status changed from yellow to green - Ready
>
> > > log [03:18:11.245] [info][status][plugin:index\_management@6.4.2] Status changed from yellow to green - Ready
>
> > > log [03:18:11.247] [info][status][plugin:grokdebugger@6.4.2] Status changed from yellow to green - Ready
>
> > > log [03:18:11.248] [info][status][plugin:logstash@6.4.2] Status changed from yellow to green - Ready
>
> > > log [03:18:11.251] [info][status][plugin:reporting@6.4.2] Status changed from yellow to green - Ready
>
> > > log [03:18:11.253] [info][kibana-monitoring][monitoring-ui] Starting monitoring stats collection
>
> > > log [03:18:11.475] [info][license][xpack] Imported license information from Elasticsearch for the [monitoring] cluster: mode: basic | status: active
>
> > > Elasticsearch ERROR: 2018-11-05T03:24:00Z
>
> > > Error: Request error, retrying
>
> > > GET [https://kubernetes.mycluster.com:30099/\_searchguard/license](https://kubernetes.mycluster.com:30099/_searchguard/license) =\> write EPROTO 140599130773312:error:14094410:SSL routines:ssl3\_read\_bytes:sslv3 alert handshake failure:…/deps/openssl/openssl/ssl/s3\_pkt.c:1498:SSL alert number 40
>
> > > 140599130773312:error:1409E0E5:SSL routines:ssl3\_write\_bytes:ssl handshake failure:…/deps/openssl/openssl/ssl/s3\_pkt.c:659:
>
> > >
>
> > > at Log.error (/usr/share/kibana/node\_modules/elasticsearch/src/lib/log.js:225:56)
>
> > > at checkRespForFailure (/usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:258:18)
>
> > > at HttpConnector. (/usr/share/kibana/node\_modules/elasticsearch/src/lib/connectors/http.js:157:7)
>
> > > at ClientRequest.bound (/usr/share/kibana/node\_modules/elasticsearch/node\_modules/lodash/dist/lodash.js:729:21)
>
> > > at emitOne (events.js:116:13)
>
> > > at ClientRequest.emit (events.js:211:7)
>
> > > at TLSSocket.socketErrorListener (\_http\_client.js:387:9)
>
> > > at emitOne (events.js:116:13)
>
> > > at TLSSocket.emit (events.js:211:7)
>
> > > at onwriteError (\_stream\_writable.js:418:12)
>
> > > at onwrite (\_stream\_writable.js:440:5)
>
> > > at \_destroy (internal/streams/destroy.js:39:7)
>
> > > at TLSSocket.Socket.\_destroy (net.js:564:3)
>
> > > at TLSSocket.destroy (internal/streams/destroy.js:32:8)
>
> > > at WriteWrap.afterWrite (net.js:866:10)
>
> > >
>
> > > Elasticsearch WARNING: 2018-11-05T03:24:00Z
>
> > > Unable to revive connection: [https://kubernetes](https://kubernetes). [mycluster.com:30099/](http://mycluster.com:30099/)
>
> > >
>
> > > Elasticsearch WARNING: 2018-11-05T03:24:00Z
>
> > > No living connections
>
> > >
>
> > > error [03:24:00.574] Error: No Living connections
>
> > > at sendReqWithConnection (/usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:225:15)
>
> > > at next (/usr/share/kibana/node\_modules/elasticsearch/src/lib/connection\_pool.js:213:7)
>
> > > at \_combinedTickCallback (internal/process/next\_tick.js:131:7)
>
> > > at process.\_tickDomainCallback (internal/process/next\_tick.js:218:9)
>
> > >
>
> > >
>
> > > \*\*Elasticsearch log : \*\*
>
> > >
>
> > > [2018-10-29T08:48:51,625][INFO][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Info: SearchGuardLicense [uid=00000000-0000-0000-0000-000000000000, type=TRIAL, features=[COMPLIANCE], issueDate=2018-10-25, expiryDate=2018-12-25, issuedTo=The world, issuer=floragunn GmbH, startDate=2018-10-25, majorVersion=6, clusterName=\*, allowedNodeCount=2147483647, msgs=, expiresInDays=56, isExpired=false, valid=true, action=, prodUsage=Yes, one cluster with all commercial features and unlimited nodes per cluster., clusterService=org.elasticsearch.cluster.service.ClusterService@744c2d63, getMsgs()=, getExpiresInDays()=56, isExpired()=false, isValid()=true, getAction()=, getProdUsage()=Yes, one cluster with all commercial features and unlimited nodes per cluster.]
>
> > > [2018-10-29T08:48:51,626][INFO][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Type: TRIAL, valid
>
> > > [2018-10-29T08:48:51,626][INFO][c.f.s.c.IndexBaseConfigurationRepository] Node ‘es-client-666c88cfb5-dmw7c’ initialized
>
> > > [2018-10-29T08:50:38,214][INFO][o.e.c.s.ClusterApplierService] [es-client-666c88cfb5-dmw7c] added {{es-master-8ff849f9-j5z8l}{hOYJ3NcIQLGIniyHcUy14A}{D9WeiTgCT\_CiH1T9CDx6vg}{192.168.1.50}{192.168.1.50:9300}{ml.machine\_memory=8352608256, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {es-master-8ff849f9-d7s6g}{\_8vGh29LSFWHxxo\_TZLcsQ}{iVusXa7vS8C7YHsyjXhltA}{192.168.2.45}{192.168.2.45:9300}{ml.machine\_memory=8352608256, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true} committed version [4303]])
>
> > > [2018-11-02T07:13:58,878][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [es-client-666c88cfb5-dmw7c] SSL Problem error:140890C7:SSL routines:ssl3\_get\_client\_certificate:peer did not return a certificate
>
> > > javax.net.ssl.SSLHandshakeException: error:140890C7:SSL routines:ssl3\_get\_client\_certificate:peer did not return a certificate
>
> > > ```
> > > at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1104) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1064) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1170) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
> > >
> > > ```
>
> > >
>
> > >
>
> > > \*\*Certificate info : \*\*
>
> > >
>
> > > openssl x509 -in elasticsearch-admin.crt.pem -noout -text
>
> > > Certificate:
>
> > > Data:
>
> > > Version: 3 (0x2)
>
> > > Serial Number: 10 (0xa)
>
> > > Signature Algorithm: sha256WithRSAEncryption
>
> > > Issuer: DC=ICES, O=MYCLUSTER, OU=ICES, CN=ICES, Signing CA
>
> > > Validity
>
> > > Not Before: Feb 28 04:06:01 2018 GMT
>
> > > Not After : Feb 28 04:06:01 2020 GMT
>
> > > Subject: C=US, O=MYCLUSTER, OU=ICES, CN=elasticsearch-admin
>
> > > Subject Public Key Info:
>
> > > Public Key Algorithm: rsaEncryption
>
> > > Public-Key: (2048 bit)
>
> > > Modulus:
>
> > > 00:db:15:db:b0:06:cc:de:c4:2d:aa:77:c8:08:f7:
>
> > > 31:20:6b:2b:78:45:00:02:89:9a:a3:7c:dc:82:3b:
>
> > > 65:ca:3f:31:01:17:b3:8f:19:dd:12:8a:8b:dc:fe:
>
> > > ae:65:e9:d5:22:7e:93:ac:53:f8:30:98:91:92:ef:
>
> > > 19:8d:20:aa:2e:a9:20:68:05:61:06:ce:b6:e5:01:
>
> > > b0:5a:3a:da:7c:1c:e7:e5:2d:c3:99:45:60:37:10:
>
> > > …
>
> > > …
>
> > > Exponent: 65537 (0x10001)
>
> > > X509v3 extensions:
>
> > > X509v3 Key Usage: critical
>
> > > Digital Signature, Key Encipherment
>
> > > X509v3 Basic Constraints:
>
> > > CA:FALSE
>
> > > X509v3 Extended Key Usage:
>
> > > TLS Web Server Authentication, TLS Web Client Authentication
>
> > > X509v3 Subject Key Identifier:
>
> > > C0:35:91:E3:2E:DC:2E:95:B9:D3:FC:86:9C:82:C5:86:DC:39:38:67
>
> > > X509v3 Authority Key Identifier:
>
> > > keyid:6C:B5:45:52:EF:CA:62:0E:8D:36:9B:C7:17:9E:F6:BC:D7:90:13:64
---
### Author:  [@Yong\_Wang](https://forum.search-guard.com/u/Yong_Wang)
#### Post date: [November 7, 2018, 4:39am UTC](https://forum.search-guard.com/t/elasticsearh-kibana-6-4-2-searchguard-23-1-sslv3-handshake-failure/1176/5 "2018-11-07T04:39:21Z")
The curl command output attached below, thanks!
**curl -vvv -k --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key -XGET [https://kubernetes.mycluster.com:30099/\_searchguard/authinfo](https://kubernetes.mycluster.com:30099/_searchguard/authinfo)**
Note: Unnecessary use of -X or --request, GET is already inferred.
- Trying 9.98.173.247…
- TCP\_NODELAY set
- Connected to [kubernetes.mycluster.com](http://kubernetes.mycluster.com) (9.98.173.247) port 30099 (#0)
- ALPN, offering h2
- ALPN, offering http/1.1
- Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
- successfully set certificate verify locations:
- CAfile: /etc/ssl/cert.pem
CApath: none
- TLSv1.2 (OUT), TLS handshake, Client hello (1):
- TLSv1.2 (IN), TLS handshake, Server hello (2):
- TLSv1.2 (IN), TLS handshake, Certificate (11):
- TLSv1.2 (IN), TLS handshake, Server key exchange (12):
- TLSv1.2 (IN), TLS handshake, Request CERT (13):
- TLSv1.2 (IN), TLS handshake, Server finished (14):
- TLSv1.2 (OUT), TLS handshake, Certificate (11):
- TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
- TLSv1.2 (OUT), TLS handshake, CERT verify (15):
- TLSv1.2 (OUT), TLS change cipher, Client hello (1):
- TLSv1.2 (OUT), TLS handshake, Finished (20):
- TLSv1.2 (IN), TLS change cipher, Client hello (1):
- TLSv1.2 (IN), TLS handshake, Finished (20):
- SSL connection using TLSv1.2 / ECDHE-RSA-AES128-SHA256
- ALPN, server did not agree to a protocol
- Server certificate:
- start date: Jun 5 02:03:03 2018 GMT
- expire date: Jun 4 02:03:03 2020 GMT
- issuer: DC=ICES; O=MYCLUSTER; OU=ICES; CN=ICES, Signing CA
- SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET /\_searchguard/authinfo HTTP/1.1
> Host: [kubernetes.mycluster.com:30099](http://kubernetes.mycluster.com:30099)
> User-Agent: curl/7.54.0
> Accept: _/_
>
\< HTTP/1.1 200 OK
\< content-type: application/json; charset=UTF-8
\< content-length: 444
\<
- Connection #0 to host [kubernetes.mycluster.com](http://kubernetes.mycluster.com) left intact
{“user”:“User [name=CN=elasticsearch-admin,OU=ICES,O=MYCLUSTER,C=US, roles=, requestedTenant=null]”,“user\_name”:“CN=elasticsearch-admin,OU=ICES,O=MYCLUSTER,C=US”,“user\_requested\_tenant”:null,“remote\_address”:null,“backend\_roles”:,“custom\_attribute\_names”:,“sg\_roles”:[“sg\_own\_index”],“sg\_tenants”:{“CN=elasticsearch-admin,OU=ICES,O=MYCLUSTER,C=US”:true},“principal”:“CN=elasticsearch-admin,OU=ICES,O=MYCLUSTER,C=US”,“peer\_certificates”:“2”,“sso\_logout\_url”:null}%
**curl -vvv --cert elasticsearch-admin.crt.pem:passw0rd --key elasticsearch-admin.key --cacert chain-ca.pem -XGET [https://kubernetes.mycluster.com:30099/\_searchguard/authinfo](https://kubernetes.mycluster.com:30099/_searchguard/authinfo)**
Note: Unnecessary use of -X or --request, GET is already inferred.
- Trying 9.98.173.247…
- TCP\_NODELAY set
- Connected to [kubernetes.mycluster.com](http://kubernetes.mycluster.com) (9.98.173.247) port 30099 (#0)
- ALPN, offering h2
- ALPN, offering http/1.1
- Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
- successfully set certificate verify locations:
- CAfile: chain-ca.pem
CApath: none
- TLSv1.2 (OUT), TLS handshake, Client hello (1):
- TLSv1.2 (IN), TLS handshake, Server hello (2):
- TLSv1.2 (IN), TLS handshake, Certificate (11):
- TLSv1.2 (OUT), TLS alert, Server hello (2):
- SSL certificate problem: unable to get local issuer certificate
- stopped the pause stream!
- Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: [https://curl.haxx.se/docs/sslcerts.html](https://curl.haxx.se/docs/sslcerts.html)
curl performs SSL certificate verification by default, using a “bundle”
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn’t adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you’d like to turn off curl’s verification of the certificate, use
the -k (or --insecure) option.
HTTPS-proxy has similar options --proxy-cacert and --proxy-insecure.
For the error message “SSL certificate problem: unable to get local issuer certificate”, if it relate to this issue? Even reported in curl and openssl command output, the handshake process still can complete and the license info can be retrieved.
> **···**
>
> - subject: C=US; O=MYCLUSTER; OU=ICES; CN=elasticsearch-http
>
> On Wed, Nov 7, 2018 at 6:14 AM Search Guard [info@search-guard.com](mailto:info@search-guard.com) wrote:
>
> > Before continuing pls. read [https://docs.search-guard.com/latest/kibana-plugin-installation#client-certificates-elasticsearchsslcertificate](https://docs.search-guard.com/latest/kibana-plugin-installation#client-certificates-elasticsearchsslcertificate) to make sure you understand that “If the certificate is an admin certificate, this means that all actions from all users will be allowed, regardless of other authorization settings.”
>
> > Can you post the output of
>
> >
>
> > curl -vvv -k --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key -XGET [https://kubernetes.mycluster](https://kubernetes.mycluster).com:30099/\_searchguard/authinfo
>
> >
>
> > and
>
> >
>
> > curl -vvv --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key --cacert /root/elasticsearch/ca/chain-ca.pem -XGET [https://kubernetes.mycluster](https://kubernetes.mycluster).com:30099/\_searchguard/authinfo
>
> > On Monday, 5 November 2018 05:23:04 UTC+1, Yong Wang wrote:
> >
> > > Dear Sir or Madam:
>
> > >
>
> > > I deployed an Elasticsearch cluster with SearchGuard by using latest release, met a ssl handshake error when want to connect Kibana to my ES cluster. It blocked me for several days, still can’t resolve it. Could you help check and give me some hints? Thanks!
>
> > >
>
> > > I enabled the SSL encryption and Client Certification based authentication, the same config can work in my previous cluster ( ES 5.5.1 + SearchGuard + Kibana), but not work in the new cluster.
>
> > >
>
> > > When try to open Kibana page, it will show error message “The Search Guard license information could not be loaded. Please contact your system administrator.”, the the Kibana will report error like below :
>
> > >
>
> > > Elasticsearch ERROR: 2018-11-05T03:24:00Z
>
> > > Error: Request error, retrying
>
> > > GET [https://kubernetes.mycluster.com:30099/\_searchguard/license](https://kubernetes.mycluster.com:30099/_searchguard/license) =\> write EPROTO 140599130773312:error:14094410:SSL routines:ssl3\_read\_bytes:sslv3 alert handshake failure:…/deps/openssl/openssl/ssl/s3\_pkt.c:1498:SSL alert number 40
>
> > > 140599130773312:error:1409E0E5:SSL routines:ssl3\_write\_bytes:ssl handshake failure:…/deps/openssl/openssl/ssl/s3\_pkt.c:659:
>
> > >
>
> > >
>
> > > Elasticsearch WARNING: 2018-11-05T03:24:00Z
>
> > > Unable to revive connection: [https://kubernetes.mycluster.com:30099/](https://kubernetes.mycluster.com:30099/)
>
> > >
>
> > > Elasticsearch WARNING: 2018-11-05T03:24:00Z
>
> > > No living connections
>
> > >
>
> > > But if I use the curl command with same cert/key, can get the license info correctly, like below :
>
> > >
>
> > > \*\*Curl command output : \*\*
>
> > > curl --insecure --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key --cacert elasticsearch-admin.crt.pem -XGET [https://kubernetes.mycluster.com:30099/\_searchguard/license](https://kubernetes.mycluster.com:30099/_searchguard/license)
>
> > > {“\_nodes”:{“total”:9,“successful”:9,“failed”:0},“cluster\_name”:“ICES”,“sg\_license”:{“uid”:“00000000-0000-0000-0000-000000000000”,“type”:“TRIAL”,“features”:[“COMPLIANCE”],“issue\_date”:“2018-10-25”,“expiry\_date”:“2018-12-25”,“issued\_to”:“The world”,“issuer”:“floragunn GmbH”,“start\_date”:“2018-10-25”,“major\_version”:6,“cluster\_name”:“\*”,“msgs”:,“expiry\_in\_days”:52,“is\_expired”:false,“is\_valid”:true,“action”:“”,…
>
> > >
>
> > >
>
> > > Could you help check and give me some hints?
>
> > >
>
> > >
>
> > > **Information for debug :**
>
> > >
>
> > > SearchGuard : 23.1
>
> > > Elasticsearch : 6.4.2
>
> > > Kibana : 6.4.2
>
> > > Installed and used enterprise modules : N/A
>
> > > JVM version and operating system version: JDK 1.8.0.171-7.b10.el7, CentOS 7
>
> > >
>
> > >
>
> > > **Kibana.yml :**
>
> > >
>
> > > server.port: 5601
>
> > > server.host: “x.x.x.x”
>
> > > elasticsearch.url: “[https://kubernetes.mycluster.com:30099](https://kubernetes.mycluster.com:30099)”
>
> > > elasticsearch.ssl.certificate: /root/elasticsearch/k8skey/elasticsearch-admin.crt.pem
>
> > > elasticsearch.ssl.key: /root/elasticsearch/k8skey/elasticsearch-admin.key
>
> > > elasticsearch.ssl.keyPassphrase: “password”
>
> > > elasticsearch.ssl.certificateAuthorities: [“/root/elasticsearch/ca/chain-ca.pem”]
>
> > > elasticsearch.ssl.verificationMode: none
>
> > > searchguard.allow\_client\_certificates: true
>
> > > xpack.reporting.encryptionKey: “1234567890”
>
> > > xpack.security.enabled: false
>
> > > xpack.monitoring.enabled: true
>
> > > xpack.graph.enabled: false
>
> > > xpack.ml.enabled: false
>
> > > xpack.watcher.enabled: false
>
> > > searchguard.cookie.secure: false
>
> > >
>
> > > **Elasticsearch.yml :**
>
> > >
>
> > > [root@es-data-0 config]# more elasticsearch.yml
>
> > > #search guard config
>
> > > searchguard:
>
> > > ssl:
>
> > > transport:
>
> > > enable\_openssl\_if\_available: true
>
> > > enforce\_hostname\_verification: false
>
> > > pemkey\_filepath: certs/elasticsearch-transport.key
>
> > > pemkey\_password: password
>
> > > pemcert\_filepath: certs/elasticsearch-transport.crt.pem
>
> > > pemtrustedcas\_filepath: certs/chain-ca.pem
>
> > > http:
>
> > > enable\_openssl\_if\_available: true
>
> > > enabled: true
>
> > > pemkey\_filepath: certs/elasticsearch-http.key
>
> > > pemkey\_password: password
>
> > > pemcert\_filepath: certs/elasticsearch-http.crt.pem
>
> > > pemtrustedcas\_filepath: certs/chain-ca.pem
>
> > > clientauth\_mode: REQUIRE
>
> > > authcz:
>
> > > admin\_dn:
>
> > > - “CN=elasticsearch-admin, OU=ICES, O=MYCLUSTER, C=US”
>
> > >
>
> > > xpack.security.enabled: false
>
> > > xpack.monitoring.enabled: true
>
> > > xpack.graph.enabled: false
>
> > > xpack.watcher.enabled: false
>
> > >
>
> > >
>
> > >
>
> > > **Kibana output :**
>
> > >
>
> > > root@kubernetes:/usr/share/kibana/bin# ./kibana
>
> > > log [03:18:08.914] [info][status][plugin:kibana@6.4.2] Status changed from uninitialized to green - Ready
>
> > > log [03:18:09.019] [info][status][plugin:elasticsearch@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > > log [03:18:09.025] [info][status][plugin:xpack\_main@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > > log [03:18:09.034] [info][status][plugin:searchprofiler@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > > log [03:18:09.041] [info][status][plugin:tilemap@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > > log [03:18:09.052] [info][status][plugin:license\_management@6.4.2] Status changed from uninitialized to green - Ready
>
> > > log [03:18:09.060] [info][status][plugin:index\_management@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > > log [03:18:09.320] [info][status][plugin:timelion@6.4.2] Status changed from uninitialized to green - Ready
>
> > > log [03:18:09.340] [info][status][plugin:monitoring@6.4.2] Status changed from uninitialized to green - Ready
>
> > > log [03:18:09.344] [info][status][plugin:grokdebugger@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > > log [03:18:09.349] [info][status][plugin:dashboard\_mode@6.4.2] Status changed from uninitialized to green - Ready
>
> > > log [03:18:09.353] [info][status][plugin:logstash@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > > log [03:18:09.405] [info][status][plugin:apm@6.4.2] Status changed from uninitialized to green - Ready
>
> > > log [03:18:09.429] [info][status][plugin:console@6.4.2] Status changed from uninitialized to green - Ready
>
> > > log [03:18:09.434] [info][status][plugin:console\_extensions@6.4.2] Status changed from uninitialized to green - Ready
>
> > > log [03:18:09.440] [info][status][plugin:notifications@6.4.2] Status changed from uninitialized to green - Ready
>
> > > log [03:18:09.610] [info][status][plugin:searchguard@6.4.2-15] Status changed from uninitialized to yellow - Initialising Search Guard authentication plugin.
>
> > > log [03:18:09.612] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Default cookie password detected, please set a password in kibana.yml by setting ‘searchguard.cookie.password’ (min. 32 characters).
>
> > > log [03:18:09.612] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - ‘searchguard.cookie.secure’ is set to false, cookies are transmitted over unsecure HTTP connection. Consider using HTTPS and set this key to ‘true’
>
> > > log [03:18:09.664] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard session management enabled.
>
> > > log [03:18:09.665] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard copy JWT params disabled
>
> > > log [03:18:09.666] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard multitenancy disabled
>
> > > log [03:18:09.685] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Routes for Search Guard configuration GUI registered. This is an Enterprise feature.
>
> > > log [03:18:09.692] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard system routes registered.
>
> > > log [03:18:09.693] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to green - Search Guard plugin initialised.
>
> > > log [03:18:09.699] [info][status][plugin:metrics@6.4.2] Status changed from uninitialized to green - Ready
>
> > > log [03:18:10.573] [info][status][plugin:reporting@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > > log [03:18:10.691] [info][listening][server][http] Server running at [http://9.98.173.247:5601](http://9.98.173.247:5601)
>
> > > log [03:18:11.126] [info][status][plugin:elasticsearch@6.4.2] Status changed from yellow to green - Ready
>
> > > log [03:18:11.233] [info][license][xpack] Imported license information from Elasticsearch for the [data] cluster: mode: basic | status: active
>
> > > log [03:18:11.241] [info][status][plugin:xpack\_main@6.4.2] Status changed from yellow to green - Ready
>
> > > log [03:18:11.243] [info][status][plugin:searchprofiler@6.4.2] Status changed from yellow to green - Ready
>
> > > log [03:18:11.244] [info][status][plugin:tilemap@6.4.2] Status changed from yellow to green - Ready
>
> > > log [03:18:11.245] [info][status][plugin:index\_management@6.4.2] Status changed from yellow to green - Ready
>
> > > log [03:18:11.247] [info][status][plugin:grokdebugger@6.4.2] Status changed from yellow to green - Ready
>
> > > log [03:18:11.248] [info][status][plugin:logstash@6.4.2] Status changed from yellow to green - Ready
>
> > > log [03:18:11.251] [info][status][plugin:reporting@6.4.2] Status changed from yellow to green - Ready
>
> > > log [03:18:11.253] [info][kibana-monitoring][monitoring-ui] Starting monitoring stats collection
>
> > > log [03:18:11.475] [info][license][xpack] Imported license information from Elasticsearch for the [monitoring] cluster: mode: basic | status: active
>
> > > Elasticsearch ERROR: 2018-11-05T03:24:00Z
>
> > > Error: Request error, retrying
>
> > > GET [https://kubernetes.mycluster.com:30099/\_searchguard/license](https://kubernetes.mycluster.com:30099/_searchguard/license) =\> write EPROTO 140599130773312:error:14094410:SSL routines:ssl3\_read\_bytes:sslv3 alert handshake failure:…/deps/openssl/openssl/ssl/s3\_pkt.c:1498:SSL alert number 40
>
> > > 140599130773312:error:1409E0E5:SSL routines:ssl3\_write\_bytes:ssl handshake failure:…/deps/openssl/openssl/ssl/s3\_pkt.c:659:
>
> > >
>
> > > at Log.error (/usr/share/kibana/node\_modules/elasticsearch/src/lib/log.js:225:56)
>
> > > at checkRespForFailure (/usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:258:18)
>
> > > at HttpConnector. (/usr/share/kibana/node\_modules/elasticsearch/src/lib/connectors/http.js:157:7)
>
> > > at ClientRequest.bound (/usr/share/kibana/node\_modules/elasticsearch/node\_modules/lodash/dist/lodash.js:729:21)
>
> > > at emitOne (events.js:116:13)
>
> > > at ClientRequest.emit (events.js:211:7)
>
> > > at TLSSocket.socketErrorListener (\_http\_client.js:387:9)
>
> > > at emitOne (events.js:116:13)
>
> > > at TLSSocket.emit (events.js:211:7)
>
> > > at onwriteError (\_stream\_writable.js:418:12)
>
> > > at onwrite (\_stream\_writable.js:440:5)
>
> > > at \_destroy (internal/streams/destroy.js:39:7)
>
> > > at TLSSocket.Socket.\_destroy (net.js:564:3)
>
> > > at TLSSocket.destroy (internal/streams/destroy.js:32:8)
>
> > > at WriteWrap.afterWrite (net.js:866:10)
>
> > >
>
> > > Elasticsearch WARNING: 2018-11-05T03:24:00Z
>
> > > Unable to revive connection: [https://kubernetes](https://kubernetes). [mycluster.com:30099/](http://mycluster.com:30099/)
>
> > >
>
> > > Elasticsearch WARNING: 2018-11-05T03:24:00Z
>
> > > No living connections
>
> > >
>
> > > error [03:24:00.574] Error: No Living connections
>
> > > at sendReqWithConnection (/usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:225:15)
>
> > > at next (/usr/share/kibana/node\_modules/elasticsearch/src/lib/connection\_pool.js:213:7)
>
> > > at \_combinedTickCallback (internal/process/next\_tick.js:131:7)
>
> > > at process.\_tickDomainCallback (internal/process/next\_tick.js:218:9)
>
> > >
>
> > >
>
> > > \*\*Elasticsearch log : \*\*
>
> > >
>
> > > [2018-10-29T08:48:51,625][INFO][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Info: SearchGuardLicense [uid=00000000-0000-0000-0000-000000000000, type=TRIAL, features=[COMPLIANCE], issueDate=2018-10-25, expiryDate=2018-12-25, issuedTo=The world, issuer=floragunn GmbH, startDate=2018-10-25, majorVersion=6, clusterName=\*, allowedNodeCount=2147483647, msgs=, expiresInDays=56, isExpired=false, valid=true, action=, prodUsage=Yes, one cluster with all commercial features and unlimited nodes per cluster., clusterService=org.elasticsearch.cluster.service.ClusterService@744c2d63, getMsgs()=, getExpiresInDays()=56, isExpired()=false, isValid()=true, getAction()=, getProdUsage()=Yes, one cluster with all commercial features and unlimited nodes per cluster.]
>
> > > [2018-10-29T08:48:51,626][INFO][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Type: TRIAL, valid
>
> > > [2018-10-29T08:48:51,626][INFO][c.f.s.c.IndexBaseConfigurationRepository] Node ‘es-client-666c88cfb5-dmw7c’ initialized
>
> > > [2018-10-29T08:50:38,214][INFO][o.e.c.s.ClusterApplierService] [es-client-666c88cfb5-dmw7c] added {{es-master-8ff849f9-j5z8l}{hOYJ3NcIQLGIniyHcUy14A}{D9WeiTgCT\_CiH1T9CDx6vg}{192.168.1.50}{192.168.1.50:9300}{ml.machine\_memory=8352608256, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {es-master-8ff849f9-d7s6g}{\_8vGh29LSFWHxxo\_TZLcsQ}{iVusXa7vS8C7YHsyjXhltA}{192.168.2.45}{192.168.2.45:9300}{ml.machine\_memory=8352608256, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true} committed version [4303]])
>
> > > [2018-11-02T07:13:58,878][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [es-client-666c88cfb5-dmw7c] SSL Problem error:140890C7:SSL routines:ssl3\_get\_client\_certificate:peer did not return a certificate
>
> > > javax.net.ssl.SSLHandshakeException: error:140890C7:SSL routines:ssl3\_get\_client\_certificate:peer did not return a certificate
>
> > > ```
> > > at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1104) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1064) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1170) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
> > >
> > > ```
>
> > > ```
> > > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
> > >
> > > ```
>
> > >
>
> > >
>
> > > \*\*Certificate info : \*\*
>
> > >
>
> > > openssl x509 -in elasticsearch-admin.crt.pem -noout -text
>
> > > Certificate:
>
> > > Data:
>
> > > Version: 3 (0x2)
>
> > > Serial Number: 10 (0xa)
>
> > > Signature Algorithm: sha256WithRSAEncryption
>
> > > Issuer: DC=ICES, O=MYCLUSTER, OU=ICES, CN=ICES, Signing CA
>
> > > Validity
>
> > > Not Before: Feb 28 04:06:01 2018 GMT
>
> > > Not After : Feb 28 04:06:01 2020 GMT
>
> > > Subject: C=US, O=MYCLUSTER, OU=ICES, CN=elasticsearch-admin
>
> > > Subject Public Key Info:
>
> > > Public Key Algorithm: rsaEncryption
>
> > > Public-Key: (2048 bit)
>
> > > Modulus:
>
> > > 00:db:15:db:b0:06:cc:de:c4:2d:aa:77:c8:08:f7:
>
> > > 31:20:6b:2b:78:45:00:02:89:9a:a3:7c:dc:82:3b:
>
> > > 65:ca:3f:31:01:17:b3:8f:19:dd:12:8a:8b:dc:fe:
>
> > > ae:65:e9:d5:22:7e:93:ac:53:f8:30:98:91:92:ef:
>
> > > 19:8d:20:aa:2e:a9:20:68:05:61:06:ce:b6:e5:01:
>
> > > b0:5a:3a:da:7c:1c:e7:e5:2d:c3:99:45:60:37:10:
>
> > > …
>
> > > …
>
> > > Exponent: 65537 (0x10001)
>
> > > X509v3 extensions:
>
> > > X509v3 Key Usage: critical
>
> > > Digital Signature, Key Encipherment
>
> > > X509v3 Basic Constraints:
>
> > > CA:FALSE
>
> > > X509v3 Extended Key Usage:
>
> > > TLS Web Server Authentication, TLS Web Client Authentication
>
> > > X509v3 Subject Key Identifier:
>
> > > C0:35:91:E3:2E:DC:2E:95:B9:D3:FC:86:9C:82:C5:86:DC:39:38:67
>
> > > X509v3 Authority Key Identifier:
>
> > > keyid:6C:B5:45:52:EF:CA:62:0E:8D:36:9B:C7:17:9E:F6:BC:D7:90:13:64
>
> > –
> >
> > You received this message because you are subscribed to the Google Groups “Search Guard Community Forum” group.
> >
> > To unsubscribe from this group and stop receiving emails from it, send an email to [search-guard+unsubscribe@googlegroups.com](mailto:search-guard+unsubscribe@googlegroups.com).
> >
> > To post to this group, send email to [search-guard@googlegroups.com](mailto:search-guard@googlegroups.com).
> >
> > To view this discussion on the web visit [https://groups.google.com/d/msgid/search-guard/15294b40-6802-4d98-94bd-331629de1c0b%40googlegroups.com](https://groups.google.com/d/msgid/search-guard/15294b40-6802-4d98-94bd-331629de1c0b%40googlegroups.com?utm_medium=email&utm_source=footer).
> >
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@searchguard\_google\_group](https://forum.search-guard.com/u/searchguard_google_group)
#### Post date: [December 12, 2018, 8:36am UTC](https://forum.search-guard.com/t/elasticsearh-kibana-6-4-2-searchguard-23-1-sslv3-handshake-failure/1176/6 "2018-12-12T08:36:49Z")
seems like the second curl does not pick up --cacert chain-ca.pem or the chain-ca.pem does not belong to ther server certificate
> **···**
>
> On Wednesday, 7 November 2018 05:39:36 UTC+1, Yong Wang wrote:
>
> > The curl command output attached below, thanks!
>
> > **curl -vvv -k --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key -XGET [https://kubernetes.mycluster.com:30099/\_searchguard/authinfo](https://kubernetes.mycluster.com:30099/_searchguard/authinfo)**
>
> >
>
> > Note: Unnecessary use of -X or --request, GET is already inferred.
>
> > - Trying 9.98.173.247…
>
> > - TCP\_NODELAY set
>
> > - Connected to [kubernetes.mycluster.com](http://kubernetes.mycluster.com) (9.98.173.247) port 30099 (#0)
>
> > - ALPN, offering h2
>
> > - ALPN, offering http/1.1
>
> > - Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
>
> > - successfully set certificate verify locations:
>
> > - CAfile: /etc/ssl/cert.pem
>
> > CApath: none
>
> > - TLSv1.2 (OUT), TLS handshake, Client hello (1):
>
> > - TLSv1.2 (IN), TLS handshake, Server hello (2):
>
> > - TLSv1.2 (IN), TLS handshake, Certificate (11):
>
> > - TLSv1.2 (IN), TLS handshake, Server key exchange (12):
>
> > - TLSv1.2 (IN), TLS handshake, Request CERT (13):
>
> > - TLSv1.2 (IN), TLS handshake, Server finished (14):
>
> > - TLSv1.2 (OUT), TLS handshake, Certificate (11):
>
> > - TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
>
> > - TLSv1.2 (OUT), TLS handshake, CERT verify (15):
>
> > - TLSv1.2 (OUT), TLS change cipher, Client hello (1):
>
> > - TLSv1.2 (OUT), TLS handshake, Finished (20):
>
> > - TLSv1.2 (IN), TLS change cipher, Client hello (1):
>
> > - TLSv1.2 (IN), TLS handshake, Finished (20):
>
> > - SSL connection using TLSv1.2 / ECDHE-RSA-AES128-SHA256
>
> > - ALPN, server did not agree to a protocol
>
> > - Server certificate:
>
> > - subject: C=US; O=MYCLUSTER; OU=ICES; CN=elasticsearch-http
>
> > - start date: Jun 5 02:03:03 2018 GMT
>
> > - expire date: Jun 4 02:03:03 2020 GMT
>
> > - issuer: DC=ICES; O=MYCLUSTER; OU=ICES; CN=ICES, Signing CA
>
> > - SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
>
> > > GET /\_searchguard/authinfo HTTP/1.1
>
> > > Host: [kubernetes.mycluster.com:30099](http://kubernetes.mycluster.com:30099)
>
> > > User-Agent: curl/7.54.0
>
> > > Accept: _/_
>
> > >
>
> > \< HTTP/1.1 200 OK
>
> > \< content-type: application/json; charset=UTF-8
>
> > \< content-length: 444
>
> > \<
>
> > - Connection #0 to host [kubernetes.mycluster.com](http://kubernetes.mycluster.com) left intact
>
> > {“user”:“User [name=CN=elasticsearch-admin,OU=ICES,O=MYCLUSTER,C=US, roles=, requestedTenant=null]”,“user\_name”:“CN=elasticsearch-admin,OU=ICES,O=MYCLUSTER,C=US”,“user\_requested\_tenant”:null,“remote\_address”:null,“backend\_roles”:,“custom\_attribute\_names”:,“sg\_roles”:[“sg\_own\_index”],“sg\_tenants”:{“CN=elasticsearch-admin,OU=ICES,O=MYCLUSTER,C=US”:true},“principal”:“CN=elasticsearch-admin,OU=ICES,O=MYCLUSTER,C=US”,“peer\_certificates”:“2”,“sso\_logout\_url”:null}%
>
> >
>
> >
>
> > **curl -vvv --cert elasticsearch-admin.crt.pem:passw0rd --key elasticsearch-admin.key --cacert chain-ca.pem -XGET [https://kubernetes.mycluster.com:30099/\_searchguard/authinfo](https://kubernetes.mycluster.com:30099/_searchguard/authinfo)**
>
> >
>
> > Note: Unnecessary use of -X or --request, GET is already inferred.
>
> > - Trying 9.98.173.247…
>
> > - TCP\_NODELAY set
>
> > - Connected to [kubernetes.mycluster.com](http://kubernetes.mycluster.com) (9.98.173.247) port 30099 (#0)
>
> > - ALPN, offering h2
>
> > - ALPN, offering http/1.1
>
> > - Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
>
> > - successfully set certificate verify locations:
>
> > - CAfile: chain-ca.pem
>
> > CApath: none
>
> > - TLSv1.2 (OUT), TLS handshake, Client hello (1):
>
> > - TLSv1.2 (IN), TLS handshake, Server hello (2):
>
> > - TLSv1.2 (IN), TLS handshake, Certificate (11):
>
> > - TLSv1.2 (OUT), TLS alert, Server hello (2):
>
> > - SSL certificate problem: unable to get local issuer certificate
>
> > - stopped the pause stream!
>
> > - Closing connection 0
>
> > curl: (60) SSL certificate problem: unable to get local issuer certificate
>
> > More details here: [https://curl.haxx.se/docs/sslcerts.html](https://curl.haxx.se/docs/sslcerts.html)
>
> >
>
> > curl performs SSL certificate verification by default, using a “bundle”
>
> > of Certificate Authority (CA) public keys (CA certs). If the default
>
> > bundle file isn’t adequate, you can specify an alternate file
>
> > using the --cacert option.
>
> > If this HTTPS server uses a certificate signed by a CA represented in
>
> > the bundle, the certificate verification probably failed due to a
>
> > problem with the certificate (it might be expired, or the name might
>
> > not match the domain name in the URL).
>
> > If you’d like to turn off curl’s verification of the certificate, use
>
> > the -k (or --insecure) option.
>
> > HTTPS-proxy has similar options --proxy-cacert and --proxy-insecure.
>
> >
>
> >
>
> > For the error message “SSL certificate problem: unable to get local issuer certificate”, if it relate to this issue? Even reported in curl and openssl command output, the handshake process still can complete and the license info can be retrieved.
>
> >
>
> > On Wed, Nov 7, 2018 at 6:14 AM Search Guard [info@search-guard.com](mailto:info@search-guard.com) wrote:
>
> > > Before continuing pls. read [https://docs.search-guard.com/latest/kibana-plugin-installation#client-certificates-elasticsearchsslcertificate](https://docs.search-guard.com/latest/kibana-plugin-installation#client-certificates-elasticsearchsslcertificate) to make sure you understand that “If the certificate is an admin certificate, this means that all actions from all users will be allowed, regardless of other authorization settings.”
>
> > > Can you post the output of
>
> > >
>
> > > curl -vvv -k --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key -XGET [https://kubernetes.mycluster](https://kubernetes.mycluster).com:30099/\_searchguard/authinfo
>
> > >
>
> > > and
>
> > >
>
> > > curl -vvv --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key --cacert /root/elasticsearch/ca/chain-ca.pem -XGET [https://kubernetes.mycluster](https://kubernetes.mycluster).com:30099/\_searchguard/authinfo
>
> > > On Monday, 5 November 2018 05:23:04 UTC+1, Yong Wang wrote:
> > >
> > > > Dear Sir or Madam:
>
> > > >
>
> > > > I deployed an Elasticsearch cluster with SearchGuard by using latest release, met a ssl handshake error when want to connect Kibana to my ES cluster. It blocked me for several days, still can’t resolve it. Could you help check and give me some hints? Thanks!
>
> > > >
>
> > > > I enabled the SSL encryption and Client Certification based authentication, the same config can work in my previous cluster ( ES 5.5.1 + SearchGuard + Kibana), but not work in the new cluster.
>
> > > >
>
> > > > When try to open Kibana page, it will show error message “The Search Guard license information could not be loaded. Please contact your system administrator.”, the the Kibana will report error like below :
>
> > > >
>
> > > > Elasticsearch ERROR: 2018-11-05T03:24:00Z
>
> > > > Error: Request error, retrying
>
> > > > GET [https://kubernetes.mycluster.com:30099/\_searchguard/license](https://kubernetes.mycluster.com:30099/_searchguard/license) =\> write EPROTO 140599130773312:error:14094410:SSL routines:ssl3\_read\_bytes:sslv3 alert handshake failure:…/deps/openssl/openssl/ssl/s3\_pkt.c:1498:SSL alert number 40
>
> > > > 140599130773312:error:1409E0E5:SSL routines:ssl3\_write\_bytes:ssl handshake failure:…/deps/openssl/openssl/ssl/s3\_pkt.c:659:
>
> > > >
>
> > > >
>
> > > > Elasticsearch WARNING: 2018-11-05T03:24:00Z
>
> > > > Unable to revive connection: [https://kubernetes.mycluster.com:30099/](https://kubernetes.mycluster.com:30099/)
>
> > > >
>
> > > > Elasticsearch WARNING: 2018-11-05T03:24:00Z
>
> > > > No living connections
>
> > > >
>
> > > > But if I use the curl command with same cert/key, can get the license info correctly, like below :
>
> > > >
>
> > > > \*\*Curl command output : \*\*
>
> > > > curl --insecure --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key --cacert elasticsearch-admin.crt.pem -XGET [https://kubernetes.mycluster.com:30099/\_searchguard/license](https://kubernetes.mycluster.com:30099/_searchguard/license)
>
> > > > {“\_nodes”:{“total”:9,“successful”:9,“failed”:0},“cluster\_name”:“ICES”,“sg\_license”:{“uid”:“00000000-0000-0000-0000-000000000000”,“type”:“TRIAL”,“features”:[“COMPLIANCE”],“issue\_date”:“2018-10-25”,“expiry\_date”:“2018-12-25”,“issued\_to”:“The world”,“issuer”:“floragunn GmbH”,“start\_date”:“2018-10-25”,“major\_version”:6,“cluster\_name”:“\*”,“msgs”:,“expiry\_in\_days”:52,“is\_expired”:false,“is\_valid”:true,“action”:“”,…
>
> > > >
>
> > > >
>
> > > > Could you help check and give me some hints?
>
> > > >
>
> > > >
>
> > > > **Information for debug :**
>
> > > >
>
> > > > SearchGuard : 23.1
>
> > > > Elasticsearch : 6.4.2
>
> > > > Kibana : 6.4.2
>
> > > > Installed and used enterprise modules : N/A
>
> > > > JVM version and operating system version: JDK 1.8.0.171-7.b10.el7, CentOS 7
>
> > > >
>
> > > >
>
> > > > **Kibana.yml :**
>
> > > >
>
> > > > server.port: 5601
>
> > > > server.host: “x.x.x.x”
>
> > > > elasticsearch.url: “[https://kubernetes.mycluster.com:30099](https://kubernetes.mycluster.com:30099)”
>
> > > > elasticsearch.ssl.certificate: /root/elasticsearch/k8skey/elasticsearch-admin.crt.pem
>
> > > > elasticsearch.ssl.key: /root/elasticsearch/k8skey/elasticsearch-admin.key
>
> > > > elasticsearch.ssl.keyPassphrase: “password”
>
> > > > elasticsearch.ssl.certificateAuthorities: [“/root/elasticsearch/ca/chain-ca.pem”]
>
> > > > elasticsearch.ssl.verificationMode: none
>
> > > > searchguard.allow\_client\_certificates: true
>
> > > > xpack.reporting.encryptionKey: “1234567890”
>
> > > > xpack.security.enabled: false
>
> > > > xpack.monitoring.enabled: true
>
> > > > xpack.graph.enabled: false
>
> > > > xpack.ml.enabled: false
>
> > > > xpack.watcher.enabled: false
>
> > > > searchguard.cookie.secure: false
>
> > > >
>
> > > > **Elasticsearch.yml :**
>
> > > >
>
> > > > [root@es-data-0 config]# more elasticsearch.yml
>
> > > > #search guard config
>
> > > > searchguard:
>
> > > > ssl:
>
> > > > transport:
>
> > > > enable\_openssl\_if\_available: true
>
> > > > enforce\_hostname\_verification: false
>
> > > > pemkey\_filepath: certs/elasticsearch-transport.key
>
> > > > pemkey\_password: password
>
> > > > pemcert\_filepath: certs/elasticsearch-transport.crt.pem
>
> > > > pemtrustedcas\_filepath: certs/chain-ca.pem
>
> > > > http:
>
> > > > enable\_openssl\_if\_available: true
>
> > > > enabled: true
>
> > > > pemkey\_filepath: certs/elasticsearch-http.key
>
> > > > pemkey\_password: password
>
> > > > pemcert\_filepath: certs/elasticsearch-http.crt.pem
>
> > > > pemtrustedcas\_filepath: certs/chain-ca.pem
>
> > > > clientauth\_mode: REQUIRE
>
> > > > authcz:
>
> > > > admin\_dn:
>
> > > > - “CN=elasticsearch-admin, OU=ICES, O=MYCLUSTER, C=US”
>
> > > >
>
> > > > xpack.security.enabled: false
>
> > > > xpack.monitoring.enabled: true
>
> > > > xpack.graph.enabled: false
>
> > > > xpack.watcher.enabled: false
>
> > > >
>
> > > >
>
> > > >
>
> > > > **Kibana output :**
>
> > > >
>
> > > > root@kubernetes:/usr/share/kibana/bin# ./kibana
>
> > > > log [03:18:08.914] [info][status][plugin:kibana@6.4.2] Status changed from uninitialized to green - Ready
>
> > > > log [03:18:09.019] [info][status][plugin:elasticsearch@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > > > log [03:18:09.025] [info][status][plugin:xpack\_main@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > > > log [03:18:09.034] [info][status][plugin:searchprofiler@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > > > log [03:18:09.041] [info][status][plugin:tilemap@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > > > log [03:18:09.052] [info][status][plugin:license\_management@6.4.2] Status changed from uninitialized to green - Ready
>
> > > > log [03:18:09.060] [info][status][plugin:index\_management@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > > > log [03:18:09.320] [info][status][plugin:timelion@6.4.2] Status changed from uninitialized to green - Ready
>
> > > > log [03:18:09.340] [info][status][plugin:monitoring@6.4.2] Status changed from uninitialized to green - Ready
>
> > > > log [03:18:09.344] [info][status][plugin:grokdebugger@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > > > log [03:18:09.349] [info][status][plugin:dashboard\_mode@6.4.2] Status changed from uninitialized to green - Ready
>
> > > > log [03:18:09.353] [info][status][plugin:logstash@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > > > log [03:18:09.405] [info][status][plugin:apm@6.4.2] Status changed from uninitialized to green - Ready
>
> > > > log [03:18:09.429] [info][status][plugin:console@6.4.2] Status changed from uninitialized to green - Ready
>
> > > > log [03:18:09.434] [info][status][plugin:console\_extensions@6.4.2] Status changed from uninitialized to green - Ready
>
> > > > log [03:18:09.440] [info][status][plugin:notifications@6.4.2] Status changed from uninitialized to green - Ready
>
> > > > log [03:18:09.610] [info][status][plugin:searchguard@6.4.2-15] Status changed from uninitialized to yellow - Initialising Search Guard authentication plugin.
>
> > > > log [03:18:09.612] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Default cookie password detected, please set a password in kibana.yml by setting ‘searchguard.cookie.password’ (min. 32 characters).
>
> > > > log [03:18:09.612] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - ‘searchguard.cookie.secure’ is set to false, cookies are transmitted over unsecure HTTP connection. Consider using HTTPS and set this key to ‘true’
>
> > > > log [03:18:09.664] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard session management enabled.
>
> > > > log [03:18:09.665] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard copy JWT params disabled
>
> > > > log [03:18:09.666] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard multitenancy disabled
>
> > > > log [03:18:09.685] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Routes for Search Guard configuration GUI registered. This is an Enterprise feature.
>
> > > > log [03:18:09.692] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard system routes registered.
>
> > > > log [03:18:09.693] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to green - Search Guard plugin initialised.
>
> > > > log [03:18:09.699] [info][status][plugin:metrics@6.4.2] Status changed from uninitialized to green - Ready
>
> > > > log [03:18:10.573] [info][status][plugin:reporting@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
>
> > > > log [03:18:10.691] [info][listening][server][http] Server running at [http://9.98.173.247:5601](http://9.98.173.247:5601)
>
> > > > log [03:18:11.126] [info][status][plugin:elasticsearch@6.4.2] Status changed from yellow to green - Ready
>
> > > > log [03:18:11.233] [info][license][xpack] Imported license information from Elasticsearch for the [data] cluster: mode: basic | status: active
>
> > > > log [03:18:11.241] [info][status][plugin:xpack\_main@6.4.2] Status changed from yellow to green - Ready
>
> > > > log [03:18:11.243] [info][status][plugin:searchprofiler@6.4.2] Status changed from yellow to green - Ready
>
> > > > log [03:18:11.244] [info][status][plugin:tilemap@6.4.2] Status changed from yellow to green - Ready
>
> > > > log [03:18:11.245] [info][status][plugin:index\_management@6.4.2] Status changed from yellow to green - Ready
>
> > > > log [03:18:11.247] [info][status][plugin:grokdebugger@6.4.2] Status changed from yellow to green - Ready
>
> > > > log [03:18:11.248] [info][status][plugin:logstash@6.4.2] Status changed from yellow to green - Ready
>
> > > > log [03:18:11.251] [info][status][plugin:reporting@6.4.2] Status changed from yellow to green - Ready
>
> > > > log [03:18:11.253] [info][kibana-monitoring][monitoring-ui] Starting monitoring stats collection
>
> > > > log [03:18:11.475] [info][license][xpack] Imported license information from Elasticsearch for the [monitoring] cluster: mode: basic | status: active
>
> > > > Elasticsearch ERROR: 2018-11-05T03:24:00Z
>
> > > > Error: Request error, retrying
>
> > > > GET [https://kubernetes.mycluster.com:30099/\_searchguard/license](https://kubernetes.mycluster.com:30099/_searchguard/license) =\> write EPROTO 140599130773312:error:14094410:SSL routines:ssl3\_read\_bytes:sslv3 alert handshake failure:…/deps/openssl/openssl/ssl/s3\_pkt.c:1498:SSL alert number 40
>
> > > > 140599130773312:error:1409E0E5:SSL routines:ssl3\_write\_bytes:ssl handshake failure:…/deps/openssl/openssl/ssl/s3\_pkt.c:659:
>
> > > >
>
> > > > at Log.error (/usr/share/kibana/node\_modules/elasticsearch/src/lib/log.js:225:56)
>
> > > > at checkRespForFailure (/usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:258:18)
>
> > > > at HttpConnector. (/usr/share/kibana/node\_modules/elasticsearch/src/lib/connectors/http.js:157:7)
>
> > > > at ClientRequest.bound (/usr/share/kibana/node\_modules/elasticsearch/node\_modules/lodash/dist/lodash.js:729:21)
>
> > > > at emitOne (events.js:116:13)
>
> > > > at ClientRequest.emit (events.js:211:7)
>
> > > > at TLSSocket.socketErrorListener (\_http\_client.js:387:9)
>
> > > > at emitOne (events.js:116:13)
>
> > > > at TLSSocket.emit (events.js:211:7)
>
> > > > at onwriteError (\_stream\_writable.js:418:12)
>
> > > > at onwrite (\_stream\_writable.js:440:5)
>
> > > > at \_destroy (internal/streams/destroy.js:39:7)
>
> > > > at TLSSocket.Socket.\_destroy (net.js:564:3)
>
> > > > at TLSSocket.destroy (internal/streams/destroy.js:32:8)
>
> > > > at WriteWrap.afterWrite (net.js:866:10)
>
> > > >
>
> > > > Elasticsearch WARNING: 2018-11-05T03:24:00Z
>
> > > > Unable to revive connection: [https://kubernetes](https://kubernetes). [mycluster.com:30099/](http://mycluster.com:30099/)
>
> > > >
>
> > > > Elasticsearch WARNING: 2018-11-05T03:24:00Z
>
> > > > No living connections
>
> > > >
>
> > > > error [03:24:00.574] Error: No Living connections
>
> > > > at sendReqWithConnection (/usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:225:15)
>
> > > > at next (/usr/share/kibana/node\_modules/elasticsearch/src/lib/connection\_pool.js:213:7)
>
> > > > at \_combinedTickCallback (internal/process/next\_tick.js:131:7)
>
> > > > at process.\_tickDomainCallback (internal/process/next\_tick.js:218:9)
>
> > > >
>
> > > >
>
> > > > \*\*Elasticsearch log : \*\*
>
> > > >
>
> > > > [2018-10-29T08:48:51,625][INFO][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Info: SearchGuardLicense [uid=00000000-0000-0000-0000-000000000000, type=TRIAL, features=[COMPLIANCE], issueDate=2018-10-25, expiryDate=2018-12-25, issuedTo=The world, issuer=floragunn GmbH, startDate=2018-10-25, majorVersion=6, clusterName=\*, allowedNodeCount=2147483647, msgs=, expiresInDays=56, isExpired=false, valid=true, action=, prodUsage=Yes, one cluster with all commercial features and unlimited nodes per cluster., clusterService=org.elasticsearch.cluster.service.ClusterService@744c2d63, getMsgs()=, getExpiresInDays()=56, isExpired()=false, isValid()=true, getAction()=, getProdUsage()=Yes, one cluster with all commercial features and unlimited nodes per cluster.]
>
> > > > [2018-10-29T08:48:51,626][INFO][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Type: TRIAL, valid
>
> > > > [2018-10-29T08:48:51,626][INFO][c.f.s.c.IndexBaseConfigurationRepository] Node ‘es-client-666c88cfb5-dmw7c’ initialized
>
> > > > [2018-10-29T08:50:38,214][INFO][o.e.c.s.ClusterApplierService] [es-client-666c88cfb5-dmw7c] added {{es-master-8ff849f9-j5z8l}{hOYJ3NcIQLGIniyHcUy14A}{D9WeiTgCT\_CiH1T9CDx6vg}{192.168.1.50}{192.168.1.50:9300}{ml.machine\_memory=8352608256, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {es-master-8ff849f9-d7s6g}{\_8vGh29LSFWHxxo\_TZLcsQ}{iVusXa7vS8C7YHsyjXhltA}{192.168.2.45}{192.168.2.45:9300}{ml.machine\_memory=8352608256, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true} committed version [4303]])
>
> > > > [2018-11-02T07:13:58,878][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [es-client-666c88cfb5-dmw7c] SSL Problem error:140890C7:SSL routines:ssl3\_get\_client\_certificate:peer did not return a certificate
>
> > > > javax.net.ssl.SSLHandshakeException: error:140890C7:SSL routines:ssl3\_get\_client\_certificate:peer did not return a certificate
>
> > > > ```
> > > > at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1104) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
> > > >
> > > > ```
>
> > > > ```
> > > > at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1064) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
> > > >
> > > > ```
>
> > > > ```
> > > > at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
> > > >
> > > > ```
>
> > > > ```
> > > > at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1170) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
> > > >
> > > > ```
>
> > > > ```
> > > > at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
> > > >
> > > > ```
>
> > > > ```
> > > > at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
> > > >
> > > > ```
>
> > > > ```
> > > > at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
> > > >
> > > > ```
>
> > > > ```
> > > > at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
> > > >
> > > > ```
>
> > > > ```
> > > > at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
> > > >
> > > > ```
>
> > > > ```
> > > > at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
> > > >
> > > > ```
>
> > > > ```
> > > > at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
> > > >
> > > > ```
>
> > > > ```
> > > > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > > >
> > > > ```
>
> > > > ```
> > > > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > > >
> > > > ```
>
> > > > ```
> > > > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > > >
> > > > ```
>
> > > > ```
> > > > at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > > >
> > > > ```
>
> > > > ```
> > > > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > > >
> > > > ```
>
> > > > ```
> > > > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > > >
> > > > ```
>
> > > > ```
> > > > at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > > >
> > > > ```
>
> > > > ```
> > > > at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > > >
> > > > ```
>
> > > > ```
> > > > at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > > >
> > > > ```
>
> > > > ```
> > > > at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > > >
> > > > ```
>
> > > > ```
> > > > at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > > >
> > > > ```
>
> > > > ```
> > > > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
> > > >
> > > > ```
>
> > > > ```
> > > > at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
> > > >
> > > > ```
>
> > > > ```
> > > > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
> > > >
> > > > ```
>
> > > >
>
> > > >
>
> > > > \*\*Certificate info : \*\*
>
> > > >
>
> > > > openssl x509 -in elasticsearch-admin.crt.pem -noout -text
>
> > > > Certificate:
>
> > > > Data:
>
> > > > Version: 3 (0x2)
>
> > > > Serial Number: 10 (0xa)
>
> > > > Signature Algorithm: sha256WithRSAEncryption
>
> > > > Issuer: DC=ICES, O=MYCLUSTER, OU=ICES, CN=ICES, Signing CA
>
> > > > Validity
>
> > > > Not Before: Feb 28 04:06:01 2018 GMT
>
> > > > Not After : Feb 28 04:06:01 2020 GMT
>
> > > > Subject: C=US, O=MYCLUSTER, OU=ICES, CN=elasticsearch-admin
>
> > > > Subject Public Key Info:
>
> > > > Public Key Algorithm: rsaEncryption
>
> > > > Public-Key: (2048 bit)
>
> > > > Modulus:
>
> > > > 00:db:15:db:b0:06:cc:de:c4:2d:aa:77:c8:08:f7:
>
> > > > 31:20:6b:2b:78:45:00:02:89:9a:a3:7c:dc:82:3b:
>
> > > > 65:ca:3f:31:01:17:b3:8f:19:dd:12:8a:8b:dc:fe:
>
> > > > ae:65:e9:d5:22:7e:93:ac:53:f8:30:98:91:92:ef:
>
> > > > 19:8d:20:aa:2e:a9:20:68:05:61:06:ce:b6:e5:01:
>
> > > > b0:5a:3a:da:7c:1c:e7:e5:2d:c3:99:45:60:37:10:
>
> > > > …
>
> > > > …
>
> > > > Exponent: 65537 (0x10001)
>
> > > > X509v3 extensions:
>
> > > > X509v3 Key Usage: critical
>
> > > > Digital Signature, Key Encipherment
>
> > > > X509v3 Basic Constraints:
>
> > > > CA:FALSE
>
> > > > X509v3 Extended Key Usage:
>
> > > > TLS Web Server Authentication, TLS Web Client Authentication
>
> > > > X509v3 Subject Key Identifier:
>
> > > > C0:35:91:E3:2E:DC:2E:95:B9:D3:FC:86:9C:82:C5:86:DC:39:38:67
>
> > > > X509v3 Authority Key Identifier:
>
> > > > keyid:6C:B5:45:52:EF:CA:62:0E:8D:36:9B:C7:17:9E:F6:BC:D7:90:13:64
>
> > > –
> > >
> > > You received this message because you are subscribed to the Google Groups “Search Guard Community Forum” group.
> > >
> > > To unsubscribe from this group and stop receiving emails from it, send an email to [search-guard+unsubscribe@googlegroups.com](mailto:search-guard+unsubscribe@googlegroups.com).
> > >
> > > To post to this group, send email to [search-guard@googlegroups.com](mailto:search-guard@googlegroups.com).
> > >
> > > To view this discussion on the web visit [https://groups.google.com/d/msgid/search-guard/15294b40-6802-4d98-94bd-331629de1c0b%40googlegroups.com](https://groups.google.com/d/msgid/search-guard/15294b40-6802-4d98-94bd-331629de1c0b%40googlegroups.com?utm_medium=email&utm_source=footer).
> > >
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).