Dear Sir or Madam:
I deployed an Elasticsearch cluster with SearchGuard by using latest release, met a ssl handshake error when want to connect Kibana to my ES cluster. It blocked me for several days, still can’t resolve it. Could you help check and give me some hints? Thanks!
I enabled the SSL encryption and Client Certification based authentication, the same config can work in my previous cluster ( ES 5.5.1 + SearchGuard + Kibana), but not work in the new cluster.
When try to open Kibana page, it will show error message “The Search Guard license information could not be loaded. Please contact your system administrator.”, the the Kibana will report error like below :
Elasticsearch ERROR: 2018-11-05T03:24:00Z
Error: Request error, retrying
GET https://kubernetes.mycluster.com:30099/_searchguard/license => write EPROTO 140599130773312:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:…/deps/openssl/openssl/ssl/s3_pkt.c:1498:SSL alert number 40
140599130773312:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:…/deps/openssl/openssl/ssl/s3_pkt.c:659:
Elasticsearch WARNING: 2018-11-05T03:24:00Z
Unable to revive connection: https://kubernetes.mycluster.com:30099/
Elasticsearch WARNING: 2018-11-05T03:24:00Z
No living connections
But if I use the curl command with same cert/key, can get the license info correctly, like below :
**Curl command output : **
curl --insecure --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key --cacert elasticsearch-admin.crt.pem -XGET https://kubernetes.mycluster.com:30099/_searchguard/license
{“_nodes”:{“total”:9,“successful”:9,“failed”:0},“cluster_name”:“ICES”,“sg_license”:{“uid”:“00000000-0000-0000-0000-000000000000”,“type”:“TRIAL”,“features”:[“COMPLIANCE”],“issue_date”:“2018-10-25”,“expiry_date”:“2018-12-25”,“issued_to”:“The world”,“issuer”:“floragunn GmbH”,“start_date”:“2018-10-25”,“major_version”:6,“cluster_name”:“*”,“msgs”:,“expiry_in_days”:52,“is_expired”:false,“is_valid”:true,“action”:“”,…
Could you help check and give me some hints?
Information for debug :
SearchGuard : 23.1
Elasticsearch : 6.4.2
Kibana : 6.4.2
Installed and used enterprise modules : N/A
JVM version and operating system version: JDK 1.8.0.171-7.b10.el7, CentOS 7
Kibana.yml :
server.port: 5601
server.host: “x.x.x.x”
elasticsearch.url: “https://kubernetes.mycluster.com:30099”
elasticsearch.ssl.certificate: /root/elasticsearch/k8skey/elasticsearch-admin.crt.pem
elasticsearch.ssl.key: /root/elasticsearch/k8skey/elasticsearch-admin.key
elasticsearch.ssl.keyPassphrase: “password”
elasticsearch.ssl.certificateAuthorities: [ “/root/elasticsearch/ca/chain-ca.pem” ]
elasticsearch.ssl.verificationMode: none
searchguard.allow_client_certificates: true
xpack.reporting.encryptionKey: “1234567890”
xpack.security.enabled: false
xpack.monitoring.enabled: true
xpack.graph.enabled: false
xpack.ml.enabled: false
xpack.watcher.enabled: false
searchguard.cookie.secure: false
Elasticsearch.yml :
[root@es-data-0 config]# more elasticsearch.yml
#search guard config
searchguard:
ssl:
transport:
enable_openssl_if_available: true
enforce_hostname_verification: false
pemkey_filepath: certs/elasticsearch-transport.key
pemkey_password: password
pemcert_filepath: certs/elasticsearch-transport.crt.pem
pemtrustedcas_filepath: certs/chain-ca.pem
http:
enable_openssl_if_available: true
enabled: true
pemkey_filepath: certs/elasticsearch-http.key
pemkey_password: password
pemcert_filepath: certs/elasticsearch-http.crt.pem
pemtrustedcas_filepath: certs/chain-ca.pem
clientauth_mode: REQUIRE
authcz:
admin_dn:
- “CN=elasticsearch-admin, OU=ICES, O=MYCLUSTER, C=US”
xpack.security.enabled: false
xpack.monitoring.enabled: true
xpack.graph.enabled: false
xpack.watcher.enabled: false
Kibana output :
root@kubernetes:/usr/share/kibana/bin# ./kibana
log [03:18:08.914] [info][status][plugin:kibana@6.4.2] Status changed from uninitialized to green - Ready
log [03:18:09.019] [info][status][plugin:elasticsearch@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [03:18:09.025] [info][status][plugin:xpack_main@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [03:18:09.034] [info][status][plugin:searchprofiler@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [03:18:09.041] [info][status][plugin:tilemap@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [03:18:09.052] [info][status][plugin:license_management@6.4.2] Status changed from uninitialized to green - Ready
log [03:18:09.060] [info][status][plugin:index_management@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [03:18:09.320] [info][status][plugin:timelion@6.4.2] Status changed from uninitialized to green - Ready
log [03:18:09.340] [info][status][plugin:monitoring@6.4.2] Status changed from uninitialized to green - Ready
log [03:18:09.344] [info][status][plugin:grokdebugger@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [03:18:09.349] [info][status][plugin:dashboard_mode@6.4.2] Status changed from uninitialized to green - Ready
log [03:18:09.353] [info][status][plugin:logstash@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [03:18:09.405] [info][status][plugin:apm@6.4.2] Status changed from uninitialized to green - Ready
log [03:18:09.429] [info][status][plugin:console@6.4.2] Status changed from uninitialized to green - Ready
log [03:18:09.434] [info][status][plugin:console_extensions@6.4.2] Status changed from uninitialized to green - Ready
log [03:18:09.440] [info][status][plugin:notifications@6.4.2] Status changed from uninitialized to green - Ready
log [03:18:09.610] [info][status][plugin:searchguard@6.4.2-15] Status changed from uninitialized to yellow - Initialising Search Guard authentication plugin.
log [03:18:09.612] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Default cookie password detected, please set a password in kibana.yml by setting ‘searchguard.cookie.password’ (min. 32 characters).
log [03:18:09.612] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - ‘searchguard.cookie.secure’ is set to false, cookies are transmitted over unsecure HTTP connection. Consider using HTTPS and set this key to ‘true’
log [03:18:09.664] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard session management enabled.
log [03:18:09.665] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard copy JWT params disabled
log [03:18:09.666] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard multitenancy disabled
log [03:18:09.685] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Routes for Search Guard configuration GUI registered. This is an Enterprise feature.
log [03:18:09.692] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard system routes registered.
log [03:18:09.693] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to green - Search Guard plugin initialised.
log [03:18:09.699] [info][status][plugin:metrics@6.4.2] Status changed from uninitialized to green - Ready
log [03:18:10.573] [info][status][plugin:reporting@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [03:18:10.691] [info][listening][server][http] Server running at http://9.98.173.247:5601
log [03:18:11.126] [info][status][plugin:elasticsearch@6.4.2] Status changed from yellow to green - Ready
log [03:18:11.233] [info][license][xpack] Imported license information from Elasticsearch for the [data] cluster: mode: basic | status: active
log [03:18:11.241] [info][status][plugin:xpack_main@6.4.2] Status changed from yellow to green - Ready
log [03:18:11.243] [info][status][plugin:searchprofiler@6.4.2] Status changed from yellow to green - Ready
log [03:18:11.244] [info][status][plugin:tilemap@6.4.2] Status changed from yellow to green - Ready
log [03:18:11.245] [info][status][plugin:index_management@6.4.2] Status changed from yellow to green - Ready
log [03:18:11.247] [info][status][plugin:grokdebugger@6.4.2] Status changed from yellow to green - Ready
log [03:18:11.248] [info][status][plugin:logstash@6.4.2] Status changed from yellow to green - Ready
log [03:18:11.251] [info][status][plugin:reporting@6.4.2] Status changed from yellow to green - Ready
log [03:18:11.253] [info][kibana-monitoring][monitoring-ui] Starting monitoring stats collection
log [03:18:11.475] [info][license][xpack] Imported license information from Elasticsearch for the [monitoring] cluster: mode: basic | status: active
Elasticsearch ERROR: 2018-11-05T03:24:00Z
Error: Request error, retrying
GET https://kubernetes.mycluster.com:30099/_searchguard/license => write EPROTO 140599130773312:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:…/deps/openssl/openssl/ssl/s3_pkt.c:1498:SSL alert number 40
140599130773312:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:…/deps/openssl/openssl/ssl/s3_pkt.c:659:
at Log.error (/usr/share/kibana/node_modules/elasticsearch/src/lib/log.js:225:56)
at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:258:18)
at HttpConnector. (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:157:7)
at ClientRequest.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)
at emitOne (events.js:116:13)
at ClientRequest.emit (events.js:211:7)
at TLSSocket.socketErrorListener (_http_client.js:387:9)
at emitOne (events.js:116:13)
at TLSSocket.emit (events.js:211:7)
at onwriteError (_stream_writable.js:418:12)
at onwrite (_stream_writable.js:440:5)
at _destroy (internal/streams/destroy.js:39:7)
at TLSSocket.Socket._destroy (net.js:564:3)
at TLSSocket.destroy (internal/streams/destroy.js:32:8)
at WriteWrap.afterWrite (net.js:866:10)
Elasticsearch WARNING: 2018-11-05T03:24:00Z
Unable to revive connection: https://kubernetes. mycluster.com:30099/
Elasticsearch WARNING: 2018-11-05T03:24:00Z
No living connections
error [03:24:00.574] Error: No Living connections
at sendReqWithConnection (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:225:15)
at next (/usr/share/kibana/node_modules/elasticsearch/src/lib/connection_pool.js:213:7)
at _combinedTickCallback (internal/process/next_tick.js:131:7)
at process._tickDomainCallback (internal/process/next_tick.js:218:9)
**Elasticsearch log : **
[2018-10-29T08:48:51,625][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Info: SearchGuardLicense [uid=00000000-0000-0000-0000-000000000000, type=TRIAL, features=[COMPLIANCE], issueDate=2018-10-25, expiryDate=2018-12-25, issuedTo=The world, issuer=floragunn GmbH, startDate=2018-10-25, majorVersion=6, clusterName=*, allowedNodeCount=2147483647, msgs=, expiresInDays=56, isExpired=false, valid=true, action=, prodUsage=Yes, one cluster with all commercial features and unlimited nodes per cluster., clusterService=org.elasticsearch.cluster.service.ClusterService@744c2d63, getMsgs()=, getExpiresInDays()=56, isExpired()=false, isValid()=true, getAction()=, getProdUsage()=Yes, one cluster with all commercial features and unlimited nodes per cluster.]
[2018-10-29T08:48:51,626][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Type: TRIAL, valid
[2018-10-29T08:48:51,626][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Node ‘es-client-666c88cfb5-dmw7c’ initialized
[2018-10-29T08:50:38,214][INFO ][o.e.c.s.ClusterApplierService] [es-client-666c88cfb5-dmw7c] added {{es-master-8ff849f9-j5z8l}{hOYJ3NcIQLGIniyHcUy14A}{D9WeiTgCT_CiH1T9CDx6vg}{192.168.1.50}{192.168.1.50:9300}{ml.machine_memory=8352608256, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {es-master-8ff849f9-d7s6g}{_8vGh29LSFWHxxo_TZLcsQ}{iVusXa7vS8C7YHsyjXhltA}{192.168.2.45}{192.168.2.45:9300}{ml.machine_memory=8352608256, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} committed version [4303]])
[2018-11-02T07:13:58,878][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [es-client-666c88cfb5-dmw7c] SSL Problem error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate
javax.net.ssl.SSLHandshakeException: error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1104) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1064) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1170) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
**Certificate info : **
openssl x509 -in elasticsearch-admin.crt.pem -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 10 (0xa)
Signature Algorithm: sha256WithRSAEncryption
Issuer: DC=ICES, O=MYCLUSTER, OU=ICES, CN=ICES, Signing CA
Validity
Not Before: Feb 28 04:06:01 2018 GMT
Not After : Feb 28 04:06:01 2020 GMT
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:db:15:db:b0:06:cc:de:c4:2d:aa:77:c8:08:f7:
31:20:6b:2b:78:45:00:02:89:9a:a3:7c:dc:82:3b:
65:ca:3f:31:01:17:b3:8f:19:dd:12:8a:8b:dc:fe:
ae:65:e9:d5:22:7e:93:ac:53:f8:30:98:91:92:ef:
19:8d:20:aa:2e:a9:20:68:05:61:06:ce:b6:e5:01:
b0:5a:3a:da:7c:1c:e7:e5:2d:c3:99:45:60:37:10:
…
…
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Basic Constraints:
CA:FALSE
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Subject Key Identifier:
C0:35:91:E3:2E:DC:2E:95:B9:D3:FC:86:9C:82:C5:86:DC:39:38:67
X509v3 Authority Key Identifier:
keyid:6C:B5:45:52:EF:CA:62:0E:8D:36:9B:C7:17:9E:F6:BC:D7:90:13:64
···
Subject: C=US, O=MYCLUSTER, OU=ICES, CN=elasticsearch-admin