Elasticsearh/Kibana 6.4.2 + SearchGuard 23.1, sslv3 handshake failure

Dear Sir or Madam:

I deployed an Elasticsearch cluster with SearchGuard by using latest release, met a ssl handshake error when want to connect Kibana to my ES cluster. It blocked me for several days, still can’t resolve it. Could you help check and give me some hints? Thanks!

I enabled the SSL encryption and Client Certification based authentication, the same config can work in my previous cluster ( ES 5.5.1 + SearchGuard + Kibana), but not work in the new cluster.

When try to open Kibana page, it will show error message “The Search Guard license information could not be loaded. Please contact your system administrator.”, the the Kibana will report error like below :

Elasticsearch ERROR: 2018-11-05T03:24:00Z

Error: Request error, retrying

GET https://kubernetes.mycluster.com:30099/_searchguard/license => write EPROTO 140599130773312:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:…/deps/openssl/openssl/ssl/s3_pkt.c:1498:SSL alert number 40

140599130773312:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:…/deps/openssl/openssl/ssl/s3_pkt.c:659:

Elasticsearch WARNING: 2018-11-05T03:24:00Z

Unable to revive connection: https://kubernetes.mycluster.com:30099/

Elasticsearch WARNING: 2018-11-05T03:24:00Z

No living connections

But if I use the curl command with same cert/key, can get the license info correctly, like below :

**Curl command output : **

curl --insecure --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key --cacert elasticsearch-admin.crt.pem -XGET https://kubernetes.mycluster.com:30099/_searchguard/license

{"_nodes":{“total”:9,“successful”:9,“failed”:0},“cluster_name”:“ICES”,“sg_license”:{“uid”:“00000000-0000-0000-0000-000000000000”,“type”:“TRIAL”,“features”:[“COMPLIANCE”],“issue_date”:“2018-10-25”,“expiry_date”:“2018-12-25”,“issued_to”:“The world”,“issuer”:“floragunn GmbH”,“start_date”:“2018-10-25”,“major_version”:6,“cluster_name”:"*",“msgs”:,“expiry_in_days”:52,“is_expired”:false,“is_valid”:true,“action”:"",…

Could you help check and give me some hints?

Information for debug :

SearchGuard : 23.1

Elasticsearch : 6.4.2

Kibana : 6.4.2

Installed and used enterprise modules : N/A

JVM version and operating system version: JDK 1.8.0.171-7.b10.el7, CentOS 7

Kibana.yml :

server.port: 5601

server.host: “x.x.x.x”

elasticsearch.url: “https://kubernetes.mycluster.com:30099

elasticsearch.ssl.certificate: /root/elasticsearch/k8skey/elasticsearch-admin.crt.pem

elasticsearch.ssl.key: /root/elasticsearch/k8skey/elasticsearch-admin.key

elasticsearch.ssl.keyPassphrase: “password”

elasticsearch.ssl.certificateAuthorities: [ “/root/elasticsearch/ca/chain-ca.pem” ]

elasticsearch.ssl.verificationMode: none

searchguard.allow_client_certificates: true

xpack.reporting.encryptionKey: “1234567890”

xpack.security.enabled: false

xpack.monitoring.enabled: true

xpack.graph.enabled: false

xpack.ml.enabled: false

xpack.watcher.enabled: false

searchguard.cookie.secure: false

Elasticsearch.yml :

[root@es-data-0 config]# more elasticsearch.yml

#search guard config

searchguard:

ssl:

transport:

enable_openssl_if_available: true

enforce_hostname_verification: false

pemkey_filepath: certs/elasticsearch-transport.key

pemkey_password: password

pemcert_filepath: certs/elasticsearch-transport.crt.pem

pemtrustedcas_filepath: certs/chain-ca.pem

http:

enable_openssl_if_available: true

enabled: true

pemkey_filepath: certs/elasticsearch-http.key

pemkey_password: password

pemcert_filepath: certs/elasticsearch-http.crt.pem

pemtrustedcas_filepath: certs/chain-ca.pem

clientauth_mode: REQUIRE

authcz:

admin_dn:

  • “CN=elasticsearch-admin, OU=ICES, O=MYCLUSTER, C=US”

xpack.security.enabled: false

xpack.monitoring.enabled: true

xpack.graph.enabled: false

xpack.watcher.enabled: false

Kibana output :

root@kubernetes:/usr/share/kibana/bin# ./kibana

log [03:18:08.914] [info][status][plugin:kibana@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.019] [info][status][plugin:elasticsearch@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.025] [info][status][plugin:xpack_main@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.034] [info][status][plugin:searchprofiler@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.041] [info][status][plugin:tilemap@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.052] [info][status][plugin:license_management@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.060] [info][status][plugin:index_management@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.320] [info][status][plugin:timelion@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.340] [info][status][plugin:monitoring@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.344] [info][status][plugin:grokdebugger@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.349] [info][status][plugin:dashboard_mode@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.353] [info][status][plugin:logstash@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.405] [info][status][plugin:apm@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.429] [info][status][plugin:console@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.434] [info][status][plugin:console_extensions@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.440] [info][status][plugin:notifications@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.610] [info][status][plugin:searchguard@6.4.2-15] Status changed from uninitialized to yellow - Initialising Search Guard authentication plugin.

log [03:18:09.612] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Default cookie password detected, please set a password in kibana.yml by setting ‘searchguard.cookie.password’ (min. 32 characters).

log [03:18:09.612] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - ‘searchguard.cookie.secure’ is set to false, cookies are transmitted over unsecure HTTP connection. Consider using HTTPS and set this key to ‘true’

log [03:18:09.664] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard session management enabled.

log [03:18:09.665] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard copy JWT params disabled

log [03:18:09.666] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard multitenancy disabled

log [03:18:09.685] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Routes for Search Guard configuration GUI registered. This is an Enterprise feature.

log [03:18:09.692] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard system routes registered.

log [03:18:09.693] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to green - Search Guard plugin initialised.

log [03:18:09.699] [info][status][plugin:metrics@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:10.573] [info][status][plugin:reporting@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:10.691] [info][listening][server][http] Server running at http://9.98.173.247:5601

log [03:18:11.126] [info][status][plugin:elasticsearch@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.233] [info][license][xpack] Imported license information from Elasticsearch for the [data] cluster: mode: basic | status: active

log [03:18:11.241] [info][status][plugin:xpack_main@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.243] [info][status][plugin:searchprofiler@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.244] [info][status][plugin:tilemap@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.245] [info][status][plugin:index_management@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.247] [info][status][plugin:grokdebugger@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.248] [info][status][plugin:logstash@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.251] [info][status][plugin:reporting@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.253] [info][kibana-monitoring][monitoring-ui] Starting monitoring stats collection

log [03:18:11.475] [info][license][xpack] Imported license information from Elasticsearch for the [monitoring] cluster: mode: basic | status: active

Elasticsearch ERROR: 2018-11-05T03:24:00Z

Error: Request error, retrying

GET https://kubernetes.mycluster.com:30099/_searchguard/license => write EPROTO 140599130773312:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:…/deps/openssl/openssl/ssl/s3_pkt.c:1498:SSL alert number 40

140599130773312:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:…/deps/openssl/openssl/ssl/s3_pkt.c:659:

at Log.error (/usr/share/kibana/node_modules/elasticsearch/src/lib/log.js:225:56)

at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:258:18)

at HttpConnector. (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:157:7)

at ClientRequest.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)

at emitOne (events.js:116:13)

at ClientRequest.emit (events.js:211:7)

at TLSSocket.socketErrorListener (_http_client.js:387:9)

at emitOne (events.js:116:13)

at TLSSocket.emit (events.js:211:7)

at onwriteError (_stream_writable.js:418:12)

at onwrite (_stream_writable.js:440:5)

at _destroy (internal/streams/destroy.js:39:7)

at TLSSocket.Socket._destroy (net.js:564:3)

at TLSSocket.destroy (internal/streams/destroy.js:32:8)

at WriteWrap.afterWrite (net.js:866:10)

Elasticsearch WARNING: 2018-11-05T03:24:00Z

Unable to revive connection: https://kubernetes. mycluster.com:30099/

Elasticsearch WARNING: 2018-11-05T03:24:00Z

No living connections

error [03:24:00.574] Error: No Living connections

at sendReqWithConnection (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:225:15)

at next (/usr/share/kibana/node_modules/elasticsearch/src/lib/connection_pool.js:213:7)

at _combinedTickCallback (internal/process/next_tick.js:131:7)

at process._tickDomainCallback (internal/process/next_tick.js:218:9)

**Elasticsearch log : **

[2018-10-29T08:48:51,625][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Info: SearchGuardLicense [uid=00000000-0000-0000-0000-000000000000, type=TRIAL, features=[COMPLIANCE], issueDate=2018-10-25, expiryDate=2018-12-25, issuedTo=The world, issuer=floragunn GmbH, startDate=2018-10-25, majorVersion=6, clusterName=*, allowedNodeCount=2147483647, msgs=, expiresInDays=56, isExpired=false, valid=true, action=, prodUsage=Yes, one cluster with all commercial features and unlimited nodes per cluster., clusterService=org.elasticsearch.cluster.service.ClusterService@744c2d63, getMsgs()=, getExpiresInDays()=56, isExpired()=false, isValid()=true, getAction()=, getProdUsage()=Yes, one cluster with all commercial features and unlimited nodes per cluster.]

[2018-10-29T08:48:51,626][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Type: TRIAL, valid

[2018-10-29T08:48:51,626][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Node ‘es-client-666c88cfb5-dmw7c’ initialized

[2018-10-29T08:50:38,214][INFO ][o.e.c.s.ClusterApplierService] [es-client-666c88cfb5-dmw7c] added {{es-master-8ff849f9-j5z8l}{hOYJ3NcIQLGIniyHcUy14A}{D9WeiTgCT_CiH1T9CDx6vg}{192.168.1.50}{192.168.1.50:9300}{ml.machine_memory=8352608256, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {es-master-8ff849f9-d7s6g}{_8vGh29LSFWHxxo_TZLcsQ}{iVusXa7vS8C7YHsyjXhltA}{192.168.2.45}{192.168.2.45:9300}{ml.machine_memory=8352608256, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} committed version [4303]])

[2018-11-02T07:13:58,878][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [es-client-666c88cfb5-dmw7c] SSL Problem error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate

javax.net.ssl.SSLHandshakeException: error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate

at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1104) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1064) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1170) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]

**Certificate info : **

openssl x509 -in elasticsearch-admin.crt.pem -noout -text

Certificate:

Data:

Version: 3 (0x2)

Serial Number: 10 (0xa)

Signature Algorithm: sha256WithRSAEncryption

Issuer: DC=ICES, O=MYCLUSTER, OU=ICES, CN=ICES, Signing CA

Validity

Not Before: Feb 28 04:06:01 2018 GMT

Not After : Feb 28 04:06:01 2020 GMT

Subject Public Key Info:

Public Key Algorithm: rsaEncryption

Public-Key: (2048 bit)

Modulus:

00:db:15:db:b0:06:cc:de:c4:2d:aa:77:c8:08:f7:

31:20:6b:2b:78:45:00:02:89:9a:a3:7c:dc:82:3b:

65:ca:3f:31:01:17:b3:8f:19:dd:12:8a:8b:dc:fe:

ae:65:e9:d5:22:7e:93:ac:53:f8:30:98:91:92:ef:

19:8d:20:aa:2e:a9:20:68:05:61:06:ce:b6:e5:01:

b0:5a:3a:da:7c:1c:e7:e5:2d:c3:99:45:60:37:10:

Exponent: 65537 (0x10001)

X509v3 extensions:

X509v3 Key Usage: critical

Digital Signature, Key Encipherment

X509v3 Basic Constraints:

CA:FALSE

X509v3 Extended Key Usage:

TLS Web Server Authentication, TLS Web Client Authentication

X509v3 Subject Key Identifier:

C0:35:91:E3:2E:DC:2E:95:B9:D3:FC:86:9C:82:C5:86:DC:39:38:67

X509v3 Authority Key Identifier:

keyid:6C:B5:45:52:EF:CA:62:0E:8D:36:9B:C7:17:9E:F6:BC:D7:90:13:64

···

Subject: C=US, O=MYCLUSTER, OU=ICES, CN=elasticsearch-admin

And openssl client also work with same certs/keys :

root@kubernetes:/usr/share/kibana/bin# openssl s_client -connect kubernetes.mycluster.com:30099 -servername x.x.x.x -cert /root/elasticsearch/k8skey/elasticsearch-admin.crt.pem -key /root/elasticsearch/k8skey/elasticsearch-admin.key -CAfile /root/elasticsearch/ca/chain-ca.pem

Enter pass phrase for /root/elasticsearch/k8skey/elasticsearch-admin.key:

CONNECTED(00000003)

depth=1 DC = ICES, O = MYCLUSTER, OU = ICES, CN = “ICES, Signing CA”

verify error:num=20:unable to get local issuer certificate

···

Certificate chain

0 s:/C=US/O=MYCLUSTER/OU=ICES/CN=elasticsearch-http

i:/DC=ICES/O=MYCLUSTER/OU=ICES/CN=ICES, Signing CA

1 s:/DC=ICES/O=MYCLUSTER M/OU=ICES/CN=ICES, Signing CA

i:/DC=ICES/O=MYCLUSTER/OU=ICES/CN=ICES, Root CA


Server certificate

-----BEGIN CERTIFICATE-----

MIIDqzCCApOgAwIBAgIBATANBgkqhkiG9w0BAQsFADBOMRQwEgYKCZImiZPyLGQB

GRYESUNFUzEMMAoGA1UECgwDSUJNMQ0wCwYDVQQLDARJQ0VTMRkwFwYDVQQDDBBJ

-----END CERTIFICATE-----

subject=/C=US/O=MYCLUSTER/OU=ICES/CN=elasticsearch-http

issuer=/DC=ICES/O=MYCLUSTER/OU=ICES/CN=ICES, Signing CA


Acceptable client certificate CA names

/DC=ICES/O=MYCLUSTER/OU=ICES/CN=ICES, Root CA

/DC=ICES/O=MYCLUSTER/OU=ICES/CN=ICES, Signing CA

Client Certificate Types: RSA sign, DSA sign, ECDSA sign

Requested Signature Algorithms: RSA+SHA512:DSA+SHA512:ECDSA+SHA512:RSA+SHA384:DSA+SHA384:ECDSA+SHA384:RSA+SHA256:DSA+SHA256:ECDSA+SHA256:RSA+SHA224:DSA+SHA224:ECDSA+SHA224:RSA+SHA1:DSA+SHA1:ECDSA+SHA1

Shared Requested Signature Algorithms: RSA+SHA512:DSA+SHA512:ECDSA+SHA512:RSA+SHA384:DSA+SHA384:ECDSA+SHA384:RSA+SHA256:DSA+SHA256:ECDSA+SHA256:RSA+SHA224:DSA+SHA224:ECDSA+SHA224:RSA+SHA1:DSA+SHA1:ECDSA+SHA1

Peer signing digest: SHA512

Server Temp Key: ECDH, P-256, 256 bits


SSL handshake has read 2594 bytes and written 1689 bytes


New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-SHA256

Server public key is 2048 bit

Secure Renegotiation IS supported

Compression: NONE

Expansion: NONE

No ALPN negotiated

SSL-Session:

Protocol : TLSv1.2

Cipher : ECDHE-RSA-AES128-SHA256

Session-ID: 715D57B05C520803D87671DC24DB773C954ABC881374D4EFF74B2019BC1E77E3

Session-ID-ctx:

Master-Key: D79B276C3E13F9F2E4D9903FB6542046155546E132EC2C406E2C20CF40317960A2EEE7573B52E814940F2A9D48C7885B

Key-Arg : None

PSK identity: None

PSK identity hint: None

SRP username: None

Start Time: 1541395055

Timeout : 300 (sec)

Verify return code: 20 (unable to get local issuer certificate)


On Monday, November 5, 2018 at 12:23:04 PM UTC+8, Yong Wang wrote:

Dear Sir or Madam:

I deployed an Elasticsearch cluster with SearchGuard by using latest release, met a ssl handshake error when want to connect Kibana to my ES cluster. It blocked me for several days, still can’t resolve it. Could you help check and give me some hints? Thanks!

I enabled the SSL encryption and Client Certification based authentication, the same config can work in my previous cluster ( ES 5.5.1 + SearchGuard + Kibana), but not work in the new cluster.

When try to open Kibana page, it will show error message “The Search Guard license information could not be loaded. Please contact your system administrator.”, the the Kibana will report error like below :

Elasticsearch ERROR: 2018-11-05T03:24:00Z

Error: Request error, retrying

GET https://kubernetes.mycluster.com:30099/_searchguard/license => write EPROTO 140599130773312:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:…/deps/openssl/openssl/ssl/s3_pkt.c:1498:SSL alert number 40

140599130773312:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:…/deps/openssl/openssl/ssl/s3_pkt.c:659:

Elasticsearch WARNING: 2018-11-05T03:24:00Z

Unable to revive connection: https://kubernetes.mycluster.com:30099/

Elasticsearch WARNING: 2018-11-05T03:24:00Z

No living connections

But if I use the curl command with same cert/key, can get the license info correctly, like below :

**Curl command output : **

curl --insecure --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key --cacert elasticsearch-admin.crt.pem -XGET https://kubernetes.mycluster.com:30099/_searchguard/license

{"_nodes":{“total”:9,“successful”:9,“failed”:0},“cluster_name”:“ICES”,“sg_license”:{“uid”:“00000000-0000-0000-0000-000000000000”,“type”:“TRIAL”,“features”:[“COMPLIANCE”],“issue_date”:“2018-10-25”,“expiry_date”:“2018-12-25”,“issued_to”:“The world”,“issuer”:“floragunn GmbH”,“start_date”:“2018-10-25”,“major_version”:6,“cluster_name”:"*",“msgs”:,“expiry_in_days”:52,“is_expired”:false,“is_valid”:true,“action”:"",…

Could you help check and give me some hints?

Information for debug :

SearchGuard : 23.1

Elasticsearch : 6.4.2

Kibana : 6.4.2

Installed and used enterprise modules : N/A

JVM version and operating system version: JDK 1.8.0.171-7.b10.el7, CentOS 7

Kibana.yml :

server.port: 5601

server.host: “x.x.x.x”

elasticsearch.url: “https://kubernetes.mycluster.com:30099

elasticsearch.ssl.certificate: /root/elasticsearch/k8skey/elasticsearch-admin.crt.pem

elasticsearch.ssl.key: /root/elasticsearch/k8skey/elasticsearch-admin.key

elasticsearch.ssl.keyPassphrase: “password”

elasticsearch.ssl.certificateAuthorities: [ “/root/elasticsearch/ca/chain-ca.pem” ]

elasticsearch.ssl.verificationMode: none

searchguard.allow_client_certificates: true

xpack.reporting.encryptionKey: “1234567890”

xpack.security.enabled: false

xpack.monitoring.enabled: true

xpack.graph.enabled: false

xpack.ml.enabled: false

xpack.watcher.enabled: false

searchguard.cookie.secure: false

Elasticsearch.yml :

[root@es-data-0 config]# more elasticsearch.yml

#search guard config

searchguard:

ssl:

transport:

enable_openssl_if_available: true

enforce_hostname_verification: false

pemkey_filepath: certs/elasticsearch-transport.key

pemkey_password: password

pemcert_filepath: certs/elasticsearch-transport.crt.pem

pemtrustedcas_filepath: certs/chain-ca.pem

http:

enable_openssl_if_available: true

enabled: true

pemkey_filepath: certs/elasticsearch-http.key

pemkey_password: password

pemcert_filepath: certs/elasticsearch-http.crt.pem

pemtrustedcas_filepath: certs/chain-ca.pem

clientauth_mode: REQUIRE

authcz:

admin_dn:

  • “CN=elasticsearch-admin, OU=ICES, O=MYCLUSTER, C=US”

xpack.security.enabled: false

xpack.monitoring.enabled: true

xpack.graph.enabled: false

xpack.watcher.enabled: false

Kibana output :

root@kubernetes:/usr/share/kibana/bin# ./kibana

log [03:18:08.914] [info][status][plugin:kibana@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.019] [info][status][plugin:elasticsearch@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.025] [info][status][plugin:xpack_main@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.034] [info][status][plugin:searchprofiler@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.041] [info][status][plugin:tilemap@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.052] [info][status][plugin:license_management@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.060] [info][status][plugin:index_management@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.320] [info][status][plugin:timelion@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.340] [info][status][plugin:monitoring@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.344] [info][status][plugin:grokdebugger@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.349] [info][status][plugin:dashboard_mode@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.353] [info][status][plugin:logstash@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.405] [info][status][plugin:apm@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.429] [info][status][plugin:console@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.434] [info][status][plugin:console_extensions@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.440] [info][status][plugin:notifications@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.610] [info][status][plugin:searchguard@6.4.2-15] Status changed from uninitialized to yellow - Initialising Search Guard authentication plugin.

log [03:18:09.612] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Default cookie password detected, please set a password in kibana.yml by setting ‘searchguard.cookie.password’ (min. 32 characters).

log [03:18:09.612] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - ‘searchguard.cookie.secure’ is set to false, cookies are transmitted over unsecure HTTP connection. Consider using HTTPS and set this key to ‘true’

log [03:18:09.664] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard session management enabled.

log [03:18:09.665] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard copy JWT params disabled

log [03:18:09.666] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard multitenancy disabled

log [03:18:09.685] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Routes for Search Guard configuration GUI registered. This is an Enterprise feature.

log [03:18:09.692] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard system routes registered.

log [03:18:09.693] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to green - Search Guard plugin initialised.

log [03:18:09.699] [info][status][plugin:metrics@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:10.573] [info][status][plugin:reporting@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:10.691] [info][listening][server][http] Server running at http://9.98.173.247:5601

log [03:18:11.126] [info][status][plugin:elasticsearch@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.233] [info][license][xpack] Imported license information from Elasticsearch for the [data] cluster: mode: basic | status: active

log [03:18:11.241] [info][status][plugin:xpack_main@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.243] [info][status][plugin:searchprofiler@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.244] [info][status][plugin:tilemap@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.245] [info][status][plugin:index_management@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.247] [info][status][plugin:grokdebugger@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.248] [info][status][plugin:logstash@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.251] [info][status][plugin:reporting@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.253] [info][kibana-monitoring][monitoring-ui] Starting monitoring stats collection

log [03:18:11.475] [info][license][xpack] Imported license information from Elasticsearch for the [monitoring] cluster: mode: basic | status: active

Elasticsearch ERROR: 2018-11-05T03:24:00Z

Error: Request error, retrying

GET https://kubernetes.mycluster.com:30099/_searchguard/license => write EPROTO 140599130773312:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:…/deps/openssl/openssl/ssl/s3_pkt.c:1498:SSL alert number 40

140599130773312:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:…/deps/openssl/openssl/ssl/s3_pkt.c:659:

at Log.error (/usr/share/kibana/node_modules/elasticsearch/src/lib/log.js:225:56)

at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:258:18)

at HttpConnector. (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:157:7)

at ClientRequest.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)

at emitOne (events.js:116:13)

at ClientRequest.emit (events.js:211:7)

at TLSSocket.socketErrorListener (_http_client.js:387:9)

at emitOne (events.js:116:13)

at TLSSocket.emit (events.js:211:7)

at onwriteError (_stream_writable.js:418:12)

at onwrite (_stream_writable.js:440:5)

at _destroy (internal/streams/destroy.js:39:7)

at TLSSocket.Socket._destroy (net.js:564:3)

at TLSSocket.destroy (internal/streams/destroy.js:32:8)

at WriteWrap.afterWrite (net.js:866:10)

Elasticsearch WARNING: 2018-11-05T03:24:00Z

Unable to revive connection: https://kubernetes. mycluster.com:30099/

Elasticsearch WARNING: 2018-11-05T03:24:00Z

No living connections

error [03:24:00.574] Error: No Living connections

at sendReqWithConnection (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:225:15)

at next (/usr/share/kibana/node_modules/elasticsearch/src/lib/connection_pool.js:213:7)

at _combinedTickCallback (internal/process/next_tick.js:131:7)

at process._tickDomainCallback (internal/process/next_tick.js:218:9)

**Elasticsearch log : **

[2018-10-29T08:48:51,625][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Info: SearchGuardLicense [uid=00000000-0000-0000-0000-000000000000, type=TRIAL, features=[COMPLIANCE], issueDate=2018-10-25, expiryDate=2018-12-25, issuedTo=The world, issuer=floragunn GmbH, startDate=2018-10-25, majorVersion=6, clusterName=*, allowedNodeCount=2147483647, msgs=, expiresInDays=56, isExpired=false, valid=true, action=, prodUsage=Yes, one cluster with all commercial features and unlimited nodes per cluster., clusterService=org.elasticsearch.cluster.service.ClusterService@744c2d63, getMsgs()=, getExpiresInDays()=56, isExpired()=false, isValid()=true, getAction()=, getProdUsage()=Yes, one cluster with all commercial features and unlimited nodes per cluster.]

[2018-10-29T08:48:51,626][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Type: TRIAL, valid

[2018-10-29T08:48:51,626][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Node ‘es-client-666c88cfb5-dmw7c’ initialized

[2018-10-29T08:50:38,214][INFO ][o.e.c.s.ClusterApplierService] [es-client-666c88cfb5-dmw7c] added {{es-master-8ff849f9-j5z8l}{hOYJ3NcIQLGIniyHcUy14A}{D9WeiTgCT_CiH1T9CDx6vg}{192.168.1.50}{192.168.1.50:9300}{ml.machine_memory=8352608256, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {es-master-8ff849f9-d7s6g}{_8vGh29LSFWHxxo_TZLcsQ}{iVusXa7vS8C7YHsyjXhltA}{192.168.2.45}{192.168.2.45:9300}{ml.machine_memory=8352608256, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} committed version [4303]])

[2018-11-02T07:13:58,878][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [es-client-666c88cfb5-dmw7c] SSL Problem error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate

javax.net.ssl.SSLHandshakeException: error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate

at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1104) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1064) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1170) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]

**Certificate info : **

openssl x509 -in elasticsearch-admin.crt.pem -noout -text

Certificate:

Data:

Version: 3 (0x2)

Serial Number: 10 (0xa)

Signature Algorithm: sha256WithRSAEncryption

Issuer: DC=ICES, O=MYCLUSTER, OU=ICES, CN=ICES, Signing CA

Validity

Not Before: Feb 28 04:06:01 2018 GMT

Not After : Feb 28 04:06:01 2020 GMT

Subject: C=US, O=MYCLUSTER, OU=ICES, CN=elasticsearch-admin

Subject Public Key Info:

Public Key Algorithm: rsaEncryption

Public-Key: (2048 bit)

Modulus:

00:db:15:db:b0:06:cc:de:c4:2d:aa:77:c8:08:f7:

31:20:6b:2b:78:45:00:02:89:9a:a3:7c:dc:82:3b:

65:ca:3f:31:01:17:b3:8f:19:dd:12:8a:8b:dc:fe:

ae:65:e9:d5:22:7e:93:ac:53:f8:30:98:91:92:ef:

19:8d:20:aa:2e:a9:20:68:05:61:06:ce:b6:e5:01:

b0:5a:3a:da:7c:1c:e7:e5:2d:c3:99:45:60:37:10:

Exponent: 65537 (0x10001)

X509v3 extensions:

X509v3 Key Usage: critical

Digital Signature, Key Encipherment

X509v3 Basic Constraints:

CA:FALSE

X509v3 Extended Key Usage:

TLS Web Server Authentication, TLS Web Client Authentication

X509v3 Subject Key Identifier:

C0:35:91:E3:2E:DC:2E:95:B9:D3:FC:86:9C:82:C5:86:DC:39:38:67

X509v3 Authority Key Identifier:

keyid:6C:B5:45:52:EF:CA:62:0E:8D:36:9B:C7:17:9E:F6:BC:D7:90:13:64

Before continuing pls. read https://docs.search-guard.com/latest/kibana-plugin-installation#client-certificates-elasticsearchsslcertificate to make sure you understand that “If the certificate is an admin certificate, this means that all actions from all users will be allowed, regardless of other authorization settings.”

Can you post the output of

curl -vvv -k --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key -XGET https://kubernetes.mycluster.com:30099/_searchguard/authinfo

and

curl -vvv --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key --cacert /root/elasticsearch/ca/chain-ca.pem -XGET https://kubernetes.mycluster.com:30099/_searchguard/authinfo

···

On Monday, 5 November 2018 05:23:04 UTC+1, Yong Wang wrote:

Dear Sir or Madam:

I deployed an Elasticsearch cluster with SearchGuard by using latest release, met a ssl handshake error when want to connect Kibana to my ES cluster. It blocked me for several days, still can’t resolve it. Could you help check and give me some hints? Thanks!

I enabled the SSL encryption and Client Certification based authentication, the same config can work in my previous cluster ( ES 5.5.1 + SearchGuard + Kibana), but not work in the new cluster.

When try to open Kibana page, it will show error message “The Search Guard license information could not be loaded. Please contact your system administrator.”, the the Kibana will report error like below :

Elasticsearch ERROR: 2018-11-05T03:24:00Z

Error: Request error, retrying

GET https://kubernetes.mycluster.com:30099/_searchguard/license => write EPROTO 140599130773312:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:…/deps/openssl/openssl/ssl/s3_pkt.c:1498:SSL alert number 40

140599130773312:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:…/deps/openssl/openssl/ssl/s3_pkt.c:659:

Elasticsearch WARNING: 2018-11-05T03:24:00Z

Unable to revive connection: https://kubernetes.mycluster.com:30099/

Elasticsearch WARNING: 2018-11-05T03:24:00Z

No living connections

But if I use the curl command with same cert/key, can get the license info correctly, like below :

**Curl command output : **

curl --insecure --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key --cacert elasticsearch-admin.crt.pem -XGET https://kubernetes.mycluster.com:30099/_searchguard/license

{"_nodes":{“total”:9,“successful”:9,“failed”:0},“cluster_name”:“ICES”,“sg_license”:{“uid”:“00000000-0000-0000-0000-000000000000”,“type”:“TRIAL”,“features”:[“COMPLIANCE”],“issue_date”:“2018-10-25”,“expiry_date”:“2018-12-25”,“issued_to”:“The world”,“issuer”:“floragunn GmbH”,“start_date”:“2018-10-25”,“major_version”:6,“cluster_name”:"*",“msgs”:,“expiry_in_days”:52,“is_expired”:false,“is_valid”:true,“action”:"",…

Could you help check and give me some hints?

Information for debug :

SearchGuard : 23.1

Elasticsearch : 6.4.2

Kibana : 6.4.2

Installed and used enterprise modules : N/A

JVM version and operating system version: JDK 1.8.0.171-7.b10.el7, CentOS 7

Kibana.yml :

server.port: 5601

server.host: “x.x.x.x”

elasticsearch.url: “https://kubernetes.mycluster.com:30099

elasticsearch.ssl.certificate: /root/elasticsearch/k8skey/elasticsearch-admin.crt.pem

elasticsearch.ssl.key: /root/elasticsearch/k8skey/elasticsearch-admin.key

elasticsearch.ssl.keyPassphrase: “password”

elasticsearch.ssl.certificateAuthorities: [ “/root/elasticsearch/ca/chain-ca.pem” ]

elasticsearch.ssl.verificationMode: none

searchguard.allow_client_certificates: true

xpack.reporting.encryptionKey: “1234567890”

xpack.security.enabled: false

xpack.monitoring.enabled: true

xpack.graph.enabled: false

xpack.ml.enabled: false

xpack.watcher.enabled: false

searchguard.cookie.secure: false

Elasticsearch.yml :

[root@es-data-0 config]# more elasticsearch.yml

#search guard config

searchguard:

ssl:

transport:

enable_openssl_if_available: true

enforce_hostname_verification: false

pemkey_filepath: certs/elasticsearch-transport.key

pemkey_password: password

pemcert_filepath: certs/elasticsearch-transport.crt.pem

pemtrustedcas_filepath: certs/chain-ca.pem

http:

enable_openssl_if_available: true

enabled: true

pemkey_filepath: certs/elasticsearch-http.key

pemkey_password: password

pemcert_filepath: certs/elasticsearch-http.crt.pem

pemtrustedcas_filepath: certs/chain-ca.pem

clientauth_mode: REQUIRE

authcz:

admin_dn:

  • “CN=elasticsearch-admin, OU=ICES, O=MYCLUSTER, C=US”

xpack.security.enabled: false

xpack.monitoring.enabled: true

xpack.graph.enabled: false

xpack.watcher.enabled: false

Kibana output :

root@kubernetes:/usr/share/kibana/bin# ./kibana

log [03:18:08.914] [info][status][plugin:kibana@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.019] [info][status][plugin:elasticsearch@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.025] [info][status][plugin:xpack_main@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.034] [info][status][plugin:searchprofiler@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.041] [info][status][plugin:tilemap@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.052] [info][status][plugin:license_management@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.060] [info][status][plugin:index_management@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.320] [info][status][plugin:timelion@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.340] [info][status][plugin:monitoring@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.344] [info][status][plugin:grokdebugger@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.349] [info][status][plugin:dashboard_mode@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.353] [info][status][plugin:logstash@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.405] [info][status][plugin:apm@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.429] [info][status][plugin:console@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.434] [info][status][plugin:console_extensions@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.440] [info][status][plugin:notifications@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.610] [info][status][plugin:searchguard@6.4.2-15] Status changed from uninitialized to yellow - Initialising Search Guard authentication plugin.

log [03:18:09.612] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Default cookie password detected, please set a password in kibana.yml by setting ‘searchguard.cookie.password’ (min. 32 characters).

log [03:18:09.612] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - ‘searchguard.cookie.secure’ is set to false, cookies are transmitted over unsecure HTTP connection. Consider using HTTPS and set this key to ‘true’

log [03:18:09.664] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard session management enabled.

log [03:18:09.665] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard copy JWT params disabled

log [03:18:09.666] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard multitenancy disabled

log [03:18:09.685] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Routes for Search Guard configuration GUI registered. This is an Enterprise feature.

log [03:18:09.692] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard system routes registered.

log [03:18:09.693] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to green - Search Guard plugin initialised.

log [03:18:09.699] [info][status][plugin:metrics@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:10.573] [info][status][plugin:reporting@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:10.691] [info][listening][server][http] Server running at http://9.98.173.247:5601

log [03:18:11.126] [info][status][plugin:elasticsearch@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.233] [info][license][xpack] Imported license information from Elasticsearch for the [data] cluster: mode: basic | status: active

log [03:18:11.241] [info][status][plugin:xpack_main@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.243] [info][status][plugin:searchprofiler@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.244] [info][status][plugin:tilemap@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.245] [info][status][plugin:index_management@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.247] [info][status][plugin:grokdebugger@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.248] [info][status][plugin:logstash@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.251] [info][status][plugin:reporting@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.253] [info][kibana-monitoring][monitoring-ui] Starting monitoring stats collection

log [03:18:11.475] [info][license][xpack] Imported license information from Elasticsearch for the [monitoring] cluster: mode: basic | status: active

Elasticsearch ERROR: 2018-11-05T03:24:00Z

Error: Request error, retrying

GET https://kubernetes.mycluster.com:30099/_searchguard/license => write EPROTO 140599130773312:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:…/deps/openssl/openssl/ssl/s3_pkt.c:1498:SSL alert number 40

140599130773312:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:…/deps/openssl/openssl/ssl/s3_pkt.c:659:

at Log.error (/usr/share/kibana/node_modules/elasticsearch/src/lib/log.js:225:56)

at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:258:18)

at HttpConnector. (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:157:7)

at ClientRequest.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)

at emitOne (events.js:116:13)

at ClientRequest.emit (events.js:211:7)

at TLSSocket.socketErrorListener (_http_client.js:387:9)

at emitOne (events.js:116:13)

at TLSSocket.emit (events.js:211:7)

at onwriteError (_stream_writable.js:418:12)

at onwrite (_stream_writable.js:440:5)

at _destroy (internal/streams/destroy.js:39:7)

at TLSSocket.Socket._destroy (net.js:564:3)

at TLSSocket.destroy (internal/streams/destroy.js:32:8)

at WriteWrap.afterWrite (net.js:866:10)

Elasticsearch WARNING: 2018-11-05T03:24:00Z

Unable to revive connection: https://kubernetes. mycluster.com:30099/

Elasticsearch WARNING: 2018-11-05T03:24:00Z

No living connections

error [03:24:00.574] Error: No Living connections

at sendReqWithConnection (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:225:15)

at next (/usr/share/kibana/node_modules/elasticsearch/src/lib/connection_pool.js:213:7)

at _combinedTickCallback (internal/process/next_tick.js:131:7)

at process._tickDomainCallback (internal/process/next_tick.js:218:9)

**Elasticsearch log : **

[2018-10-29T08:48:51,625][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Info: SearchGuardLicense [uid=00000000-0000-0000-0000-000000000000, type=TRIAL, features=[COMPLIANCE], issueDate=2018-10-25, expiryDate=2018-12-25, issuedTo=The world, issuer=floragunn GmbH, startDate=2018-10-25, majorVersion=6, clusterName=*, allowedNodeCount=2147483647, msgs=, expiresInDays=56, isExpired=false, valid=true, action=, prodUsage=Yes, one cluster with all commercial features and unlimited nodes per cluster., clusterService=org.elasticsearch.cluster.service.ClusterService@744c2d63, getMsgs()=, getExpiresInDays()=56, isExpired()=false, isValid()=true, getAction()=, getProdUsage()=Yes, one cluster with all commercial features and unlimited nodes per cluster.]

[2018-10-29T08:48:51,626][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Type: TRIAL, valid

[2018-10-29T08:48:51,626][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Node ‘es-client-666c88cfb5-dmw7c’ initialized

[2018-10-29T08:50:38,214][INFO ][o.e.c.s.ClusterApplierService] [es-client-666c88cfb5-dmw7c] added {{es-master-8ff849f9-j5z8l}{hOYJ3NcIQLGIniyHcUy14A}{D9WeiTgCT_CiH1T9CDx6vg}{192.168.1.50}{192.168.1.50:9300}{ml.machine_memory=8352608256, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {es-master-8ff849f9-d7s6g}{_8vGh29LSFWHxxo_TZLcsQ}{iVusXa7vS8C7YHsyjXhltA}{192.168.2.45}{192.168.2.45:9300}{ml.machine_memory=8352608256, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} committed version [4303]])

[2018-11-02T07:13:58,878][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [es-client-666c88cfb5-dmw7c] SSL Problem error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate

javax.net.ssl.SSLHandshakeException: error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate

at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1104) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1064) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1170) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]

**Certificate info : **

openssl x509 -in elasticsearch-admin.crt.pem -noout -text

Certificate:

Data:

Version: 3 (0x2)

Serial Number: 10 (0xa)

Signature Algorithm: sha256WithRSAEncryption

Issuer: DC=ICES, O=MYCLUSTER, OU=ICES, CN=ICES, Signing CA

Validity

Not Before: Feb 28 04:06:01 2018 GMT

Not After : Feb 28 04:06:01 2020 GMT

Subject: C=US, O=MYCLUSTER, OU=ICES, CN=elasticsearch-admin

Subject Public Key Info:

Public Key Algorithm: rsaEncryption

Public-Key: (2048 bit)

Modulus:

00:db:15:db:b0:06:cc:de:c4:2d:aa:77:c8:08:f7:

31:20:6b:2b:78:45:00:02:89:9a:a3:7c:dc:82:3b:

65:ca:3f:31:01:17:b3:8f:19:dd:12:8a:8b:dc:fe:

ae:65:e9:d5:22:7e:93:ac:53:f8:30:98:91:92:ef:

19:8d:20:aa:2e:a9:20:68:05:61:06:ce:b6:e5:01:

b0:5a:3a:da:7c:1c:e7:e5:2d:c3:99:45:60:37:10:

Exponent: 65537 (0x10001)

X509v3 extensions:

X509v3 Key Usage: critical

Digital Signature, Key Encipherment

X509v3 Basic Constraints:

CA:FALSE

X509v3 Extended Key Usage:

TLS Web Server Authentication, TLS Web Client Authentication

X509v3 Subject Key Identifier:

C0:35:91:E3:2E:DC:2E:95:B9:D3:FC:86:9C:82:C5:86:DC:39:38:67

X509v3 Authority Key Identifier:

keyid:6C:B5:45:52:EF:CA:62:0E:8D:36:9B:C7:17:9E:F6:BC:D7:90:13:64

Your openssl command returns an error: verify error:num=20:unable to get local issuer certificate

···

On Monday, 5 November 2018 06:21:23 UTC+1, Yong Wang wrote:

And openssl client also work with same certs/keys :

root@kubernetes:/usr/share/kibana/bin# openssl s_client -connect kubernetes.mycluster.com:30099 -servername x.x.x.x -cert /root/elasticsearch/k8skey/elasticsearch-admin.crt.pem -key /root/elasticsearch/k8skey/elasticsearch-admin.key -CAfile /root/elasticsearch/ca/chain-ca.pem

Enter pass phrase for /root/elasticsearch/k8skey/elasticsearch-admin.key:

CONNECTED(00000003)

depth=1 DC = ICES, O = MYCLUSTER, OU = ICES, CN = “ICES, Signing CA”

verify error:num=20:unable to get local issuer certificate


Certificate chain

0 s:/C=US/O=MYCLUSTER/OU=ICES/CN=elasticsearch-http

i:/DC=ICES/O=MYCLUSTER/OU=ICES/CN=ICES, Signing CA

1 s:/DC=ICES/O=MYCLUSTER M/OU=ICES/CN=ICES, Signing CA

i:/DC=ICES/O=MYCLUSTER/OU=ICES/CN=ICES, Root CA


Server certificate

-----BEGIN CERTIFICATE-----

MIIDqzCCApOgAwIBAgIBATANBgkqhkiG9w0BAQsFADBOMRQwEgYKCZImiZPyLGQB

GRYESUNFUzEMMAoGA1UECgwDSUJNMQ0wCwYDVQQLDARJQ0VTMRkwFwYDVQQDDBBJ

-----END CERTIFICATE-----

subject=/C=US/O=MYCLUSTER/OU=ICES/CN=elasticsearch-http

issuer=/DC=ICES/O=MYCLUSTER/OU=ICES/CN=ICES, Signing CA


Acceptable client certificate CA names

/DC=ICES/O=MYCLUSTER/OU=ICES/CN=ICES, Root CA

/DC=ICES/O=MYCLUSTER/OU=ICES/CN=ICES, Signing CA

Client Certificate Types: RSA sign, DSA sign, ECDSA sign

Requested Signature Algorithms: RSA+SHA512:DSA+SHA512:ECDSA+SHA512:RSA+SHA384:DSA+SHA384:ECDSA+SHA384:RSA+SHA256:DSA+SHA256:ECDSA+SHA256:RSA+SHA224:DSA+SHA224:ECDSA+SHA224:RSA+SHA1:DSA+SHA1:ECDSA+SHA1

Shared Requested Signature Algorithms: RSA+SHA512:DSA+SHA512:ECDSA+SHA512:RSA+SHA384:DSA+SHA384:ECDSA+SHA384:RSA+SHA256:DSA+SHA256:ECDSA+SHA256:RSA+SHA224:DSA+SHA224:ECDSA+SHA224:RSA+SHA1:DSA+SHA1:ECDSA+SHA1

Peer signing digest: SHA512

Server Temp Key: ECDH, P-256, 256 bits


SSL handshake has read 2594 bytes and written 1689 bytes


New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-SHA256

Server public key is 2048 bit

Secure Renegotiation IS supported

Compression: NONE

Expansion: NONE

No ALPN negotiated

SSL-Session:

Protocol : TLSv1.2

Cipher : ECDHE-RSA-AES128-SHA256

Session-ID: 715D57B05C520803D87671DC24DB773C954ABC881374D4EFF74B2019BC1E77E3

Session-ID-ctx:

Master-Key: D79B276C3E13F9F2E4D9903FB6542046155546E132EC2C406E2C20CF40317960A2EEE7573B52E814940F2A9D48C7885B

Key-Arg : None

PSK identity: None

PSK identity hint: None

SRP username: None

Start Time: 1541395055

Timeout : 300 (sec)

Verify return code: 20 (unable to get local issuer certificate)


On Monday, November 5, 2018 at 12:23:04 PM UTC+8, Yong Wang wrote:

Dear Sir or Madam:

I deployed an Elasticsearch cluster with SearchGuard by using latest release, met a ssl handshake error when want to connect Kibana to my ES cluster. It blocked me for several days, still can’t resolve it. Could you help check and give me some hints? Thanks!

I enabled the SSL encryption and Client Certification based authentication, the same config can work in my previous cluster ( ES 5.5.1 + SearchGuard + Kibana), but not work in the new cluster.

When try to open Kibana page, it will show error message “The Search Guard license information could not be loaded. Please contact your system administrator.”, the the Kibana will report error like below :

Elasticsearch ERROR: 2018-11-05T03:24:00Z

Error: Request error, retrying

GET https://kubernetes.mycluster.com:30099/_searchguard/license => write EPROTO 140599130773312:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:…/deps/openssl/openssl/ssl/s3_pkt.c:1498:SSL alert number 40

140599130773312:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:…/deps/openssl/openssl/ssl/s3_pkt.c:659:

Elasticsearch WARNING: 2018-11-05T03:24:00Z

Unable to revive connection: https://kubernetes.mycluster.com:30099/

Elasticsearch WARNING: 2018-11-05T03:24:00Z

No living connections

But if I use the curl command with same cert/key, can get the license info correctly, like below :

**Curl command output : **

curl --insecure --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key --cacert elasticsearch-admin.crt.pem -XGET https://kubernetes.mycluster.com:30099/_searchguard/license

{"_nodes":{“total”:9,“successful”:9,“failed”:0},“cluster_name”:“ICES”,“sg_license”:{“uid”:“00000000-0000-0000-0000-000000000000”,“type”:“TRIAL”,“features”:[“COMPLIANCE”],“issue_date”:“2018-10-25”,“expiry_date”:“2018-12-25”,“issued_to”:“The world”,“issuer”:“floragunn GmbH”,“start_date”:“2018-10-25”,“major_version”:6,“cluster_name”:"*",“msgs”:,“expiry_in_days”:52,“is_expired”:false,“is_valid”:true,“action”:"",…

Could you help check and give me some hints?

Information for debug :

SearchGuard : 23.1

Elasticsearch : 6.4.2

Kibana : 6.4.2

Installed and used enterprise modules : N/A

JVM version and operating system version: JDK 1.8.0.171-7.b10.el7, CentOS 7

Kibana.yml :

server.port: 5601

server.host: “x.x.x.x”

elasticsearch.url: “https://kubernetes.mycluster.com:30099

elasticsearch.ssl.certificate: /root/elasticsearch/k8skey/elasticsearch-admin.crt.pem

elasticsearch.ssl.key: /root/elasticsearch/k8skey/elasticsearch-admin.key

elasticsearch.ssl.keyPassphrase: “password”

elasticsearch.ssl.certificateAuthorities: [ “/root/elasticsearch/ca/chain-ca.pem” ]

elasticsearch.ssl.verificationMode: none

searchguard.allow_client_certificates: true

xpack.reporting.encryptionKey: “1234567890”

xpack.security.enabled: false

xpack.monitoring.enabled: true

xpack.graph.enabled: false

xpack.ml.enabled: false

xpack.watcher.enabled: false

searchguard.cookie.secure: false

Elasticsearch.yml :

[root@es-data-0 config]# more elasticsearch.yml

#search guard config

searchguard:

ssl:

transport:

enable_openssl_if_available: true

enforce_hostname_verification: false

pemkey_filepath: certs/elasticsearch-transport.key

pemkey_password: password

pemcert_filepath: certs/elasticsearch-transport.crt.pem

pemtrustedcas_filepath: certs/chain-ca.pem

http:

enable_openssl_if_available: true

enabled: true

pemkey_filepath: certs/elasticsearch-http.key

pemkey_password: password

pemcert_filepath: certs/elasticsearch-http.crt.pem

pemtrustedcas_filepath: certs/chain-ca.pem

clientauth_mode: REQUIRE

authcz:

admin_dn:

  • “CN=elasticsearch-admin, OU=ICES, O=MYCLUSTER, C=US”

xpack.security.enabled: false

xpack.monitoring.enabled: true

xpack.graph.enabled: false

xpack.watcher.enabled: false

Kibana output :

root@kubernetes:/usr/share/kibana/bin# ./kibana

log [03:18:08.914] [info][status][plugin:kibana@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.019] [info][status][plugin:elasticsearch@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.025] [info][status][plugin:xpack_main@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.034] [info][status][plugin:searchprofiler@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.041] [info][status][plugin:tilemap@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.052] [info][status][plugin:license_management@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.060] [info][status][plugin:index_management@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.320] [info][status][plugin:timelion@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.340] [info][status][plugin:monitoring@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.344] [info][status][plugin:grokdebugger@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.349] [info][status][plugin:dashboard_mode@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.353] [info][status][plugin:logstash@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.405] [info][status][plugin:apm@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.429] [info][status][plugin:console@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.434] [info][status][plugin:console_extensions@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.440] [info][status][plugin:notifications@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.610] [info][status][plugin:searchguard@6.4.2-15] Status changed from uninitialized to yellow - Initialising Search Guard authentication plugin.

log [03:18:09.612] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Default cookie password detected, please set a password in kibana.yml by setting ‘searchguard.cookie.password’ (min. 32 characters).

log [03:18:09.612] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - ‘searchguard.cookie.secure’ is set to false, cookies are transmitted over unsecure HTTP connection. Consider using HTTPS and set this key to ‘true’

log [03:18:09.664] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard session management enabled.

log [03:18:09.665] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard copy JWT params disabled

log [03:18:09.666] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard multitenancy disabled

log [03:18:09.685] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Routes for Search Guard configuration GUI registered. This is an Enterprise feature.

log [03:18:09.692] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard system routes registered.

log [03:18:09.693] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to green - Search Guard plugin initialised.

log [03:18:09.699] [info][status][plugin:metrics@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:10.573] [info][status][plugin:reporting@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:10.691] [info][listening][server][http] Server running at http://9.98.173.247:5601

log [03:18:11.126] [info][status][plugin:elasticsearch@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.233] [info][license][xpack] Imported license information from Elasticsearch for the [data] cluster: mode: basic | status: active

log [03:18:11.241] [info][status][plugin:xpack_main@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.243] [info][status][plugin:searchprofiler@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.244] [info][status][plugin:tilemap@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.245] [info][status][plugin:index_management@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.247] [info][status][plugin:grokdebugger@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.248] [info][status][plugin:logstash@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.251] [info][status][plugin:reporting@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.253] [info][kibana-monitoring][monitoring-ui] Starting monitoring stats collection

log [03:18:11.475] [info][license][xpack] Imported license information from Elasticsearch for the [monitoring] cluster: mode: basic | status: active

Elasticsearch ERROR: 2018-11-05T03:24:00Z

Error: Request error, retrying

GET https://kubernetes.mycluster.com:30099/_searchguard/license => write EPROTO 140599130773312:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:…/deps/openssl/openssl/ssl/s3_pkt.c:1498:SSL alert number 40

140599130773312:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:…/deps/openssl/openssl/ssl/s3_pkt.c:659:

at Log.error (/usr/share/kibana/node_modules/elasticsearch/src/lib/log.js:225:56)

at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:258:18)

at HttpConnector. (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:157:7)

at ClientRequest.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)

at emitOne (events.js:116:13)

at ClientRequest.emit (events.js:211:7)

at TLSSocket.socketErrorListener (_http_client.js:387:9)

at emitOne (events.js:116:13)

at TLSSocket.emit (events.js:211:7)

at onwriteError (_stream_writable.js:418:12)

at onwrite (_stream_writable.js:440:5)

at _destroy (internal/streams/destroy.js:39:7)

at TLSSocket.Socket._destroy (net.js:564:3)

at TLSSocket.destroy (internal/streams/destroy.js:32:8)

at WriteWrap.afterWrite (net.js:866:10)

Elasticsearch WARNING: 2018-11-05T03:24:00Z

Unable to revive connection: https://kubernetes. mycluster.com:30099/

Elasticsearch WARNING: 2018-11-05T03:24:00Z

No living connections

error [03:24:00.574] Error: No Living connections

at sendReqWithConnection (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:225:15)

at next (/usr/share/kibana/node_modules/elasticsearch/src/lib/connection_pool.js:213:7)

at _combinedTickCallback (internal/process/next_tick.js:131:7)

at process._tickDomainCallback (internal/process/next_tick.js:218:9)

**Elasticsearch log : **

[2018-10-29T08:48:51,625][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Info: SearchGuardLicense [uid=00000000-0000-0000-0000-000000000000, type=TRIAL, features=[COMPLIANCE], issueDate=2018-10-25, expiryDate=2018-12-25, issuedTo=The world, issuer=floragunn GmbH, startDate=2018-10-25, majorVersion=6, clusterName=*, allowedNodeCount=2147483647, msgs=, expiresInDays=56, isExpired=false, valid=true, action=, prodUsage=Yes, one cluster with all commercial features and unlimited nodes per cluster., clusterService=org.elasticsearch.cluster.service.ClusterService@744c2d63, getMsgs()=, getExpiresInDays()=56, isExpired()=false, isValid()=true, getAction()=, getProdUsage()=Yes, one cluster with all commercial features and unlimited nodes per cluster.]

[2018-10-29T08:48:51,626][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Type: TRIAL, valid

[2018-10-29T08:48:51,626][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Node ‘es-client-666c88cfb5-dmw7c’ initialized

[2018-10-29T08:50:38,214][INFO ][o.e.c.s.ClusterApplierService] [es-client-666c88cfb5-dmw7c] added {{es-master-8ff849f9-j5z8l}{hOYJ3NcIQLGIniyHcUy14A}{D9WeiTgCT_CiH1T9CDx6vg}{192.168.1.50}{192.168.1.50:9300}{ml.machine_memory=8352608256, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {es-master-8ff849f9-d7s6g}{_8vGh29LSFWHxxo_TZLcsQ}{iVusXa7vS8C7YHsyjXhltA}{192.168.2.45}{192.168.2.45:9300}{ml.machine_memory=8352608256, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} committed version [4303]])

[2018-11-02T07:13:58,878][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [es-client-666c88cfb5-dmw7c] SSL Problem error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate

javax.net.ssl.SSLHandshakeException: error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate

at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1104) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1064) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1170) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]

**Certificate info : **

openssl x509 -in elasticsearch-admin.crt.pem -noout -text

Certificate:

Data:

Version: 3 (0x2)

Serial Number: 10 (0xa)

Signature Algorithm: sha256WithRSAEncryption

Issuer: DC=ICES, O=MYCLUSTER, OU=ICES, CN=ICES, Signing CA

Validity

Not Before: Feb 28 04:06:01 2018 GMT

Not After : Feb 28 04:06:01 2020 GMT

Subject: C=US, O=MYCLUSTER, OU=ICES, CN=elasticsearch-admin

Subject Public Key Info:

Public Key Algorithm: rsaEncryption

Public-Key: (2048 bit)

Modulus:

00:db:15:db:b0:06:cc:de:c4:2d:aa:77:c8:08:f7:

31:20:6b:2b:78:45:00:02:89:9a:a3:7c:dc:82:3b:

65:ca:3f:31:01:17:b3:8f:19:dd:12:8a:8b:dc:fe:

ae:65:e9:d5:22:7e:93:ac:53:f8:30:98:91:92:ef:

19:8d:20:aa:2e:a9:20:68:05:61:06:ce:b6:e5:01:

b0:5a:3a:da:7c:1c:e7:e5:2d:c3:99:45:60:37:10:

Exponent: 65537 (0x10001)

X509v3 extensions:

X509v3 Key Usage: critical

Digital Signature, Key Encipherment

X509v3 Basic Constraints:

CA:FALSE

X509v3 Extended Key Usage:

TLS Web Server Authentication, TLS Web Client Authentication

X509v3 Subject Key Identifier:

C0:35:91:E3:2E:DC:2E:95:B9:D3:FC:86:9C:82:C5:86:DC:39:38:67

X509v3 Authority Key Identifier:

keyid:6C:B5:45:52:EF:CA:62:0E:8D:36:9B:C7:17:9E:F6:BC:D7:90:13:64

The curl command output attached below, thanks!

curl -vvv -k --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key -XGET https://kubernetes.mycluster.com:30099/_searchguard/authinfo

Note: Unnecessary use of -X or --request, GET is already inferred.

  • Trying 9.98.173.247…

  • TCP_NODELAY set

  • Connected to kubernetes.mycluster.com (9.98.173.247) port 30099 (#0)

  • ALPN, offering h2

  • ALPN, offering http/1.1

  • Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH

  • successfully set certificate verify locations:

  • CAfile: /etc/ssl/cert.pem

CApath: none

  • TLSv1.2 (OUT), TLS handshake, Client hello (1):

  • TLSv1.2 (IN), TLS handshake, Server hello (2):

  • TLSv1.2 (IN), TLS handshake, Certificate (11):

  • TLSv1.2 (IN), TLS handshake, Server key exchange (12):

  • TLSv1.2 (IN), TLS handshake, Request CERT (13):

  • TLSv1.2 (IN), TLS handshake, Server finished (14):

  • TLSv1.2 (OUT), TLS handshake, Certificate (11):

  • TLSv1.2 (OUT), TLS handshake, Client key exchange (16):

  • TLSv1.2 (OUT), TLS handshake, CERT verify (15):

  • TLSv1.2 (OUT), TLS change cipher, Client hello (1):

  • TLSv1.2 (OUT), TLS handshake, Finished (20):

  • TLSv1.2 (IN), TLS change cipher, Client hello (1):

  • TLSv1.2 (IN), TLS handshake, Finished (20):

  • SSL connection using TLSv1.2 / ECDHE-RSA-AES128-SHA256

  • ALPN, server did not agree to a protocol

  • Server certificate:

  • start date: Jun 5 02:03:03 2018 GMT

  • expire date: Jun 4 02:03:03 2020 GMT

  • issuer: DC=ICES; O=MYCLUSTER; OU=ICES; CN=ICES, Signing CA

  • SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.

GET /_searchguard/authinfo HTTP/1.1

Host: kubernetes.mycluster.com:30099

User-Agent: curl/7.54.0

Accept: /

< HTTP/1.1 200 OK

< content-type: application/json; charset=UTF-8

< content-length: 444

<

{“user”:“User [name=CN=elasticsearch-admin,OU=ICES,O=MYCLUSTER,C=US, roles=, requestedTenant=null]”,“user_name”:“CN=elasticsearch-admin,OU=ICES,O=MYCLUSTER,C=US”,“user_requested_tenant”:null,“remote_address”:null,“backend_roles”:,“custom_attribute_names”:,“sg_roles”:[“sg_own_index”],“sg_tenants”:{“CN=elasticsearch-admin,OU=ICES,O=MYCLUSTER,C=US”:true},“principal”:“CN=elasticsearch-admin,OU=ICES,O=MYCLUSTER,C=US”,“peer_certificates”:“2”,“sso_logout_url”:null}%

curl -vvv --cert elasticsearch-admin.crt.pem:passw0rd --key elasticsearch-admin.key --cacert chain-ca.pem -XGET https://kubernetes.mycluster.com:30099/_searchguard/authinfo

Note: Unnecessary use of -X or --request, GET is already inferred.

  • Trying 9.98.173.247…

  • TCP_NODELAY set

  • Connected to kubernetes.mycluster.com (9.98.173.247) port 30099 (#0)

  • ALPN, offering h2

  • ALPN, offering http/1.1

  • Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH

  • successfully set certificate verify locations:

  • CAfile: chain-ca.pem

CApath: none

  • TLSv1.2 (OUT), TLS handshake, Client hello (1):

  • TLSv1.2 (IN), TLS handshake, Server hello (2):

  • TLSv1.2 (IN), TLS handshake, Certificate (11):

  • TLSv1.2 (OUT), TLS alert, Server hello (2):

  • SSL certificate problem: unable to get local issuer certificate

  • stopped the pause stream!

  • Closing connection 0

curl: (60) SSL certificate problem: unable to get local issuer certificate

More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a “bundle”

of Certificate Authority (CA) public keys (CA certs). If the default

bundle file isn’t adequate, you can specify an alternate file

using the --cacert option.

If this HTTPS server uses a certificate signed by a CA represented in

the bundle, the certificate verification probably failed due to a

problem with the certificate (it might be expired, or the name might

not match the domain name in the URL).

If you’d like to turn off curl’s verification of the certificate, use

the -k (or --insecure) option.

HTTPS-proxy has similar options --proxy-cacert and --proxy-insecure.

For the error message “SSL certificate problem: unable to get local issuer certificate”, if it relate to this issue? Even reported in curl and openssl command output, the handshake process still can complete and the license info can be retrieved.

···
  • subject: C=US; O=MYCLUSTER; OU=ICES; CN=elasticsearch-http

On Wed, Nov 7, 2018 at 6:14 AM Search Guard info@search-guard.com wrote:

Before continuing pls. read https://docs.search-guard.com/latest/kibana-plugin-installation#client-certificates-elasticsearchsslcertificate to make sure you understand that “If the certificate is an admin certificate, this means that all actions from all users will be allowed, regardless of other authorization settings.”

Can you post the output of

curl -vvv -k --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key -XGET https://kubernetes.mycluster.com:30099/_searchguard/authinfo

and

curl -vvv --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key --cacert /root/elasticsearch/ca/chain-ca.pem -XGET https://kubernetes.mycluster.com:30099/_searchguard/authinfo

On Monday, 5 November 2018 05:23:04 UTC+1, Yong Wang wrote:

Dear Sir or Madam:

I deployed an Elasticsearch cluster with SearchGuard by using latest release, met a ssl handshake error when want to connect Kibana to my ES cluster. It blocked me for several days, still can’t resolve it. Could you help check and give me some hints? Thanks!

I enabled the SSL encryption and Client Certification based authentication, the same config can work in my previous cluster ( ES 5.5.1 + SearchGuard + Kibana), but not work in the new cluster.

When try to open Kibana page, it will show error message “The Search Guard license information could not be loaded. Please contact your system administrator.”, the the Kibana will report error like below :

Elasticsearch ERROR: 2018-11-05T03:24:00Z

Error: Request error, retrying

GET https://kubernetes.mycluster.com:30099/_searchguard/license => write EPROTO 140599130773312:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:…/deps/openssl/openssl/ssl/s3_pkt.c:1498:SSL alert number 40

140599130773312:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:…/deps/openssl/openssl/ssl/s3_pkt.c:659:

Elasticsearch WARNING: 2018-11-05T03:24:00Z

Unable to revive connection: https://kubernetes.mycluster.com:30099/

Elasticsearch WARNING: 2018-11-05T03:24:00Z

No living connections

But if I use the curl command with same cert/key, can get the license info correctly, like below :

**Curl command output : **

curl --insecure --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key --cacert elasticsearch-admin.crt.pem -XGET https://kubernetes.mycluster.com:30099/_searchguard/license

{"_nodes":{“total”:9,“successful”:9,“failed”:0},“cluster_name”:“ICES”,“sg_license”:{“uid”:“00000000-0000-0000-0000-000000000000”,“type”:“TRIAL”,“features”:[“COMPLIANCE”],“issue_date”:“2018-10-25”,“expiry_date”:“2018-12-25”,“issued_to”:“The world”,“issuer”:“floragunn GmbH”,“start_date”:“2018-10-25”,“major_version”:6,“cluster_name”:"*",“msgs”:,“expiry_in_days”:52,“is_expired”:false,“is_valid”:true,“action”:"",…

Could you help check and give me some hints?

Information for debug :

SearchGuard : 23.1

Elasticsearch : 6.4.2

Kibana : 6.4.2

Installed and used enterprise modules : N/A

JVM version and operating system version: JDK 1.8.0.171-7.b10.el7, CentOS 7

Kibana.yml :

server.port: 5601

server.host: “x.x.x.x”

elasticsearch.url: “https://kubernetes.mycluster.com:30099

elasticsearch.ssl.certificate: /root/elasticsearch/k8skey/elasticsearch-admin.crt.pem

elasticsearch.ssl.key: /root/elasticsearch/k8skey/elasticsearch-admin.key

elasticsearch.ssl.keyPassphrase: “password”

elasticsearch.ssl.certificateAuthorities: [ “/root/elasticsearch/ca/chain-ca.pem” ]

elasticsearch.ssl.verificationMode: none

searchguard.allow_client_certificates: true

xpack.reporting.encryptionKey: “1234567890”

xpack.security.enabled: false

xpack.monitoring.enabled: true

xpack.graph.enabled: false

xpack.ml.enabled: false

xpack.watcher.enabled: false

searchguard.cookie.secure: false

Elasticsearch.yml :

[root@es-data-0 config]# more elasticsearch.yml

#search guard config

searchguard:

ssl:

transport:

enable_openssl_if_available: true

enforce_hostname_verification: false

pemkey_filepath: certs/elasticsearch-transport.key

pemkey_password: password

pemcert_filepath: certs/elasticsearch-transport.crt.pem

pemtrustedcas_filepath: certs/chain-ca.pem

http:

enable_openssl_if_available: true

enabled: true

pemkey_filepath: certs/elasticsearch-http.key

pemkey_password: password

pemcert_filepath: certs/elasticsearch-http.crt.pem

pemtrustedcas_filepath: certs/chain-ca.pem

clientauth_mode: REQUIRE

authcz:

admin_dn:

  • “CN=elasticsearch-admin, OU=ICES, O=MYCLUSTER, C=US”

xpack.security.enabled: false

xpack.monitoring.enabled: true

xpack.graph.enabled: false

xpack.watcher.enabled: false

Kibana output :

root@kubernetes:/usr/share/kibana/bin# ./kibana

log [03:18:08.914] [info][status][plugin:kibana@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.019] [info][status][plugin:elasticsearch@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.025] [info][status][plugin:xpack_main@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.034] [info][status][plugin:searchprofiler@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.041] [info][status][plugin:tilemap@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.052] [info][status][plugin:license_management@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.060] [info][status][plugin:index_management@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.320] [info][status][plugin:timelion@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.340] [info][status][plugin:monitoring@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.344] [info][status][plugin:grokdebugger@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.349] [info][status][plugin:dashboard_mode@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.353] [info][status][plugin:logstash@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.405] [info][status][plugin:apm@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.429] [info][status][plugin:console@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.434] [info][status][plugin:console_extensions@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.440] [info][status][plugin:notifications@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.610] [info][status][plugin:searchguard@6.4.2-15] Status changed from uninitialized to yellow - Initialising Search Guard authentication plugin.

log [03:18:09.612] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Default cookie password detected, please set a password in kibana.yml by setting ‘searchguard.cookie.password’ (min. 32 characters).

log [03:18:09.612] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - ‘searchguard.cookie.secure’ is set to false, cookies are transmitted over unsecure HTTP connection. Consider using HTTPS and set this key to ‘true’

log [03:18:09.664] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard session management enabled.

log [03:18:09.665] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard copy JWT params disabled

log [03:18:09.666] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard multitenancy disabled

log [03:18:09.685] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Routes for Search Guard configuration GUI registered. This is an Enterprise feature.

log [03:18:09.692] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard system routes registered.

log [03:18:09.693] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to green - Search Guard plugin initialised.

log [03:18:09.699] [info][status][plugin:metrics@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:10.573] [info][status][plugin:reporting@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:10.691] [info][listening][server][http] Server running at http://9.98.173.247:5601

log [03:18:11.126] [info][status][plugin:elasticsearch@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.233] [info][license][xpack] Imported license information from Elasticsearch for the [data] cluster: mode: basic | status: active

log [03:18:11.241] [info][status][plugin:xpack_main@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.243] [info][status][plugin:searchprofiler@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.244] [info][status][plugin:tilemap@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.245] [info][status][plugin:index_management@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.247] [info][status][plugin:grokdebugger@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.248] [info][status][plugin:logstash@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.251] [info][status][plugin:reporting@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.253] [info][kibana-monitoring][monitoring-ui] Starting monitoring stats collection

log [03:18:11.475] [info][license][xpack] Imported license information from Elasticsearch for the [monitoring] cluster: mode: basic | status: active

Elasticsearch ERROR: 2018-11-05T03:24:00Z

Error: Request error, retrying

GET https://kubernetes.mycluster.com:30099/_searchguard/license => write EPROTO 140599130773312:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:…/deps/openssl/openssl/ssl/s3_pkt.c:1498:SSL alert number 40

140599130773312:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:…/deps/openssl/openssl/ssl/s3_pkt.c:659:

at Log.error (/usr/share/kibana/node_modules/elasticsearch/src/lib/log.js:225:56)

at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:258:18)

at HttpConnector. (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:157:7)

at ClientRequest.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)

at emitOne (events.js:116:13)

at ClientRequest.emit (events.js:211:7)

at TLSSocket.socketErrorListener (_http_client.js:387:9)

at emitOne (events.js:116:13)

at TLSSocket.emit (events.js:211:7)

at onwriteError (_stream_writable.js:418:12)

at onwrite (_stream_writable.js:440:5)

at _destroy (internal/streams/destroy.js:39:7)

at TLSSocket.Socket._destroy (net.js:564:3)

at TLSSocket.destroy (internal/streams/destroy.js:32:8)

at WriteWrap.afterWrite (net.js:866:10)

Elasticsearch WARNING: 2018-11-05T03:24:00Z

Unable to revive connection: https://kubernetes. mycluster.com:30099/

Elasticsearch WARNING: 2018-11-05T03:24:00Z

No living connections

error [03:24:00.574] Error: No Living connections

at sendReqWithConnection (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:225:15)

at next (/usr/share/kibana/node_modules/elasticsearch/src/lib/connection_pool.js:213:7)

at _combinedTickCallback (internal/process/next_tick.js:131:7)

at process._tickDomainCallback (internal/process/next_tick.js:218:9)

**Elasticsearch log : **

[2018-10-29T08:48:51,625][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Info: SearchGuardLicense [uid=00000000-0000-0000-0000-000000000000, type=TRIAL, features=[COMPLIANCE], issueDate=2018-10-25, expiryDate=2018-12-25, issuedTo=The world, issuer=floragunn GmbH, startDate=2018-10-25, majorVersion=6, clusterName=*, allowedNodeCount=2147483647, msgs=, expiresInDays=56, isExpired=false, valid=true, action=, prodUsage=Yes, one cluster with all commercial features and unlimited nodes per cluster., clusterService=org.elasticsearch.cluster.service.ClusterService@744c2d63, getMsgs()=, getExpiresInDays()=56, isExpired()=false, isValid()=true, getAction()=, getProdUsage()=Yes, one cluster with all commercial features and unlimited nodes per cluster.]

[2018-10-29T08:48:51,626][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Type: TRIAL, valid

[2018-10-29T08:48:51,626][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Node ‘es-client-666c88cfb5-dmw7c’ initialized

[2018-10-29T08:50:38,214][INFO ][o.e.c.s.ClusterApplierService] [es-client-666c88cfb5-dmw7c] added {{es-master-8ff849f9-j5z8l}{hOYJ3NcIQLGIniyHcUy14A}{D9WeiTgCT_CiH1T9CDx6vg}{192.168.1.50}{192.168.1.50:9300}{ml.machine_memory=8352608256, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {es-master-8ff849f9-d7s6g}{_8vGh29LSFWHxxo_TZLcsQ}{iVusXa7vS8C7YHsyjXhltA}{192.168.2.45}{192.168.2.45:9300}{ml.machine_memory=8352608256, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} committed version [4303]])

[2018-11-02T07:13:58,878][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [es-client-666c88cfb5-dmw7c] SSL Problem error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate

javax.net.ssl.SSLHandshakeException: error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate

at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1104) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1064) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1170) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]

**Certificate info : **

openssl x509 -in elasticsearch-admin.crt.pem -noout -text

Certificate:

Data:

Version: 3 (0x2)

Serial Number: 10 (0xa)

Signature Algorithm: sha256WithRSAEncryption

Issuer: DC=ICES, O=MYCLUSTER, OU=ICES, CN=ICES, Signing CA

Validity

Not Before: Feb 28 04:06:01 2018 GMT

Not After : Feb 28 04:06:01 2020 GMT

Subject: C=US, O=MYCLUSTER, OU=ICES, CN=elasticsearch-admin

Subject Public Key Info:

Public Key Algorithm: rsaEncryption

Public-Key: (2048 bit)

Modulus:

00:db:15:db:b0:06:cc:de:c4:2d:aa:77:c8:08:f7:

31:20:6b:2b:78:45:00:02:89:9a:a3:7c:dc:82:3b:

65:ca:3f:31:01:17:b3:8f:19:dd:12:8a:8b:dc:fe:

ae:65:e9:d5:22:7e:93:ac:53:f8:30:98:91:92:ef:

19:8d:20:aa:2e:a9:20:68:05:61:06:ce:b6:e5:01:

b0:5a:3a:da:7c:1c:e7:e5:2d:c3:99:45:60:37:10:

Exponent: 65537 (0x10001)

X509v3 extensions:

X509v3 Key Usage: critical

Digital Signature, Key Encipherment

X509v3 Basic Constraints:

CA:FALSE

X509v3 Extended Key Usage:

TLS Web Server Authentication, TLS Web Client Authentication

X509v3 Subject Key Identifier:

C0:35:91:E3:2E:DC:2E:95:B9:D3:FC:86:9C:82:C5:86:DC:39:38:67

X509v3 Authority Key Identifier:

keyid:6C:B5:45:52:EF:CA:62:0E:8D:36:9B:C7:17:9E:F6:BC:D7:90:13:64

You received this message because you are subscribed to the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/15294b40-6802-4d98-94bd-331629de1c0b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

seems like the second curl does not pick up --cacert chain-ca.pem or the chain-ca.pem does not belong to ther server certificate

···

On Wednesday, 7 November 2018 05:39:36 UTC+1, Yong Wang wrote:

The curl command output attached below, thanks!

curl -vvv -k --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key -XGET https://kubernetes.mycluster.com:30099/_searchguard/authinfo

Note: Unnecessary use of -X or --request, GET is already inferred.

  • Trying 9.98.173.247…
  • TCP_NODELAY set
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/cert.pem

CApath: none

  • TLSv1.2 (OUT), TLS handshake, Client hello (1):
  • TLSv1.2 (IN), TLS handshake, Server hello (2):
  • TLSv1.2 (IN), TLS handshake, Certificate (11):
  • TLSv1.2 (IN), TLS handshake, Server key exchange (12):
  • TLSv1.2 (IN), TLS handshake, Request CERT (13):
  • TLSv1.2 (IN), TLS handshake, Server finished (14):
  • TLSv1.2 (OUT), TLS handshake, Certificate (11):
  • TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
  • TLSv1.2 (OUT), TLS handshake, CERT verify (15):
  • TLSv1.2 (OUT), TLS change cipher, Client hello (1):
  • TLSv1.2 (OUT), TLS handshake, Finished (20):
  • TLSv1.2 (IN), TLS change cipher, Client hello (1):
  • TLSv1.2 (IN), TLS handshake, Finished (20):
  • SSL connection using TLSv1.2 / ECDHE-RSA-AES128-SHA256
  • ALPN, server did not agree to a protocol
  • Server certificate:
  • subject: C=US; O=MYCLUSTER; OU=ICES; CN=elasticsearch-http
  • start date: Jun 5 02:03:03 2018 GMT
  • expire date: Jun 4 02:03:03 2020 GMT
  • issuer: DC=ICES; O=MYCLUSTER; OU=ICES; CN=ICES, Signing CA
  • SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.

GET /_searchguard/authinfo HTTP/1.1

Host: kubernetes.mycluster.com:30099

User-Agent: curl/7.54.0

Accept: /

< HTTP/1.1 200 OK

< content-type: application/json; charset=UTF-8

< content-length: 444

<

{“user”:“User [name=CN=elasticsearch-admin,OU=ICES,O=MYCLUSTER,C=US, roles=, requestedTenant=null]”,“user_name”:“CN=elasticsearch-admin,OU=ICES,O=MYCLUSTER,C=US”,“user_requested_tenant”:null,“remote_address”:null,“backend_roles”:,“custom_attribute_names”:,“sg_roles”:[“sg_own_index”],“sg_tenants”:{“CN=elasticsearch-admin,OU=ICES,O=MYCLUSTER,C=US”:true},“principal”:“CN=elasticsearch-admin,OU=ICES,O=MYCLUSTER,C=US”,“peer_certificates”:“2”,“sso_logout_url”:null}%

curl -vvv --cert elasticsearch-admin.crt.pem:passw0rd --key elasticsearch-admin.key --cacert chain-ca.pem -XGET https://kubernetes.mycluster.com:30099/_searchguard/authinfo

Note: Unnecessary use of -X or --request, GET is already inferred.

  • Trying 9.98.173.247…
  • TCP_NODELAY set
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
  • successfully set certificate verify locations:
  • CAfile: chain-ca.pem

CApath: none

  • TLSv1.2 (OUT), TLS handshake, Client hello (1):
  • TLSv1.2 (IN), TLS handshake, Server hello (2):
  • TLSv1.2 (IN), TLS handshake, Certificate (11):
  • TLSv1.2 (OUT), TLS alert, Server hello (2):
  • SSL certificate problem: unable to get local issuer certificate
  • stopped the pause stream!
  • Closing connection 0

curl: (60) SSL certificate problem: unable to get local issuer certificate

More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a “bundle”

of Certificate Authority (CA) public keys (CA certs). If the default

bundle file isn’t adequate, you can specify an alternate file

using the --cacert option.

If this HTTPS server uses a certificate signed by a CA represented in

the bundle, the certificate verification probably failed due to a

problem with the certificate (it might be expired, or the name might

not match the domain name in the URL).

If you’d like to turn off curl’s verification of the certificate, use

the -k (or --insecure) option.

HTTPS-proxy has similar options --proxy-cacert and --proxy-insecure.

For the error message “SSL certificate problem: unable to get local issuer certificate”, if it relate to this issue? Even reported in curl and openssl command output, the handshake process still can complete and the license info can be retrieved.

On Wed, Nov 7, 2018 at 6:14 AM Search Guard info@search-guard.com wrote:

Before continuing pls. read https://docs.search-guard.com/latest/kibana-plugin-installation#client-certificates-elasticsearchsslcertificate to make sure you understand that “If the certificate is an admin certificate, this means that all actions from all users will be allowed, regardless of other authorization settings.”

Can you post the output of

curl -vvv -k --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key -XGET https://kubernetes.mycluster.com:30099/_searchguard/authinfo

and

curl -vvv --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key --cacert /root/elasticsearch/ca/chain-ca.pem -XGET https://kubernetes.mycluster.com:30099/_searchguard/authinfo

On Monday, 5 November 2018 05:23:04 UTC+1, Yong Wang wrote:

Dear Sir or Madam:

I deployed an Elasticsearch cluster with SearchGuard by using latest release, met a ssl handshake error when want to connect Kibana to my ES cluster. It blocked me for several days, still can’t resolve it. Could you help check and give me some hints? Thanks!

I enabled the SSL encryption and Client Certification based authentication, the same config can work in my previous cluster ( ES 5.5.1 + SearchGuard + Kibana), but not work in the new cluster.

When try to open Kibana page, it will show error message “The Search Guard license information could not be loaded. Please contact your system administrator.”, the the Kibana will report error like below :

Elasticsearch ERROR: 2018-11-05T03:24:00Z

Error: Request error, retrying

GET https://kubernetes.mycluster.com:30099/_searchguard/license => write EPROTO 140599130773312:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:…/deps/openssl/openssl/ssl/s3_pkt.c:1498:SSL alert number 40

140599130773312:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:…/deps/openssl/openssl/ssl/s3_pkt.c:659:

Elasticsearch WARNING: 2018-11-05T03:24:00Z

Unable to revive connection: https://kubernetes.mycluster.com:30099/

Elasticsearch WARNING: 2018-11-05T03:24:00Z

No living connections

But if I use the curl command with same cert/key, can get the license info correctly, like below :

**Curl command output : **

curl --insecure --cert elasticsearch-admin.crt.pem:password --key elasticsearch-admin.key --cacert elasticsearch-admin.crt.pem -XGET https://kubernetes.mycluster.com:30099/_searchguard/license

{"_nodes":{“total”:9,“successful”:9,“failed”:0},“cluster_name”:“ICES”,“sg_license”:{“uid”:“00000000-0000-0000-0000-000000000000”,“type”:“TRIAL”,“features”:[“COMPLIANCE”],“issue_date”:“2018-10-25”,“expiry_date”:“2018-12-25”,“issued_to”:“The world”,“issuer”:“floragunn GmbH”,“start_date”:“2018-10-25”,“major_version”:6,“cluster_name”:"*",“msgs”:,“expiry_in_days”:52,“is_expired”:false,“is_valid”:true,“action”:"",…

Could you help check and give me some hints?

Information for debug :

SearchGuard : 23.1

Elasticsearch : 6.4.2

Kibana : 6.4.2

Installed and used enterprise modules : N/A

JVM version and operating system version: JDK 1.8.0.171-7.b10.el7, CentOS 7

Kibana.yml :

server.port: 5601

server.host: “x.x.x.x”

elasticsearch.url: “https://kubernetes.mycluster.com:30099

elasticsearch.ssl.certificate: /root/elasticsearch/k8skey/elasticsearch-admin.crt.pem

elasticsearch.ssl.key: /root/elasticsearch/k8skey/elasticsearch-admin.key

elasticsearch.ssl.keyPassphrase: “password”

elasticsearch.ssl.certificateAuthorities: [ “/root/elasticsearch/ca/chain-ca.pem” ]

elasticsearch.ssl.verificationMode: none

searchguard.allow_client_certificates: true

xpack.reporting.encryptionKey: “1234567890”

xpack.security.enabled: false

xpack.monitoring.enabled: true

xpack.graph.enabled: false

xpack.ml.enabled: false

xpack.watcher.enabled: false

searchguard.cookie.secure: false

Elasticsearch.yml :

[root@es-data-0 config]# more elasticsearch.yml

#search guard config

searchguard:

ssl:

transport:

enable_openssl_if_available: true

enforce_hostname_verification: false

pemkey_filepath: certs/elasticsearch-transport.key

pemkey_password: password

pemcert_filepath: certs/elasticsearch-transport.crt.pem

pemtrustedcas_filepath: certs/chain-ca.pem

http:

enable_openssl_if_available: true

enabled: true

pemkey_filepath: certs/elasticsearch-http.key

pemkey_password: password

pemcert_filepath: certs/elasticsearch-http.crt.pem

pemtrustedcas_filepath: certs/chain-ca.pem

clientauth_mode: REQUIRE

authcz:

admin_dn:

  • “CN=elasticsearch-admin, OU=ICES, O=MYCLUSTER, C=US”

xpack.security.enabled: false

xpack.monitoring.enabled: true

xpack.graph.enabled: false

xpack.watcher.enabled: false

Kibana output :

root@kubernetes:/usr/share/kibana/bin# ./kibana

log [03:18:08.914] [info][status][plugin:kibana@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.019] [info][status][plugin:elasticsearch@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.025] [info][status][plugin:xpack_main@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.034] [info][status][plugin:searchprofiler@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.041] [info][status][plugin:tilemap@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.052] [info][status][plugin:license_management@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.060] [info][status][plugin:index_management@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.320] [info][status][plugin:timelion@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.340] [info][status][plugin:monitoring@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.344] [info][status][plugin:grokdebugger@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.349] [info][status][plugin:dashboard_mode@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.353] [info][status][plugin:logstash@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:09.405] [info][status][plugin:apm@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.429] [info][status][plugin:console@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.434] [info][status][plugin:console_extensions@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.440] [info][status][plugin:notifications@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:09.610] [info][status][plugin:searchguard@6.4.2-15] Status changed from uninitialized to yellow - Initialising Search Guard authentication plugin.

log [03:18:09.612] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Default cookie password detected, please set a password in kibana.yml by setting ‘searchguard.cookie.password’ (min. 32 characters).

log [03:18:09.612] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - ‘searchguard.cookie.secure’ is set to false, cookies are transmitted over unsecure HTTP connection. Consider using HTTPS and set this key to ‘true’

log [03:18:09.664] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard session management enabled.

log [03:18:09.665] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard copy JWT params disabled

log [03:18:09.666] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard multitenancy disabled

log [03:18:09.685] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Routes for Search Guard configuration GUI registered. This is an Enterprise feature.

log [03:18:09.692] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to yellow - Search Guard system routes registered.

log [03:18:09.693] [info][status][plugin:searchguard@6.4.2-15] Status changed from yellow to green - Search Guard plugin initialised.

log [03:18:09.699] [info][status][plugin:metrics@6.4.2] Status changed from uninitialized to green - Ready

log [03:18:10.573] [info][status][plugin:reporting@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [03:18:10.691] [info][listening][server][http] Server running at http://9.98.173.247:5601

log [03:18:11.126] [info][status][plugin:elasticsearch@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.233] [info][license][xpack] Imported license information from Elasticsearch for the [data] cluster: mode: basic | status: active

log [03:18:11.241] [info][status][plugin:xpack_main@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.243] [info][status][plugin:searchprofiler@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.244] [info][status][plugin:tilemap@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.245] [info][status][plugin:index_management@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.247] [info][status][plugin:grokdebugger@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.248] [info][status][plugin:logstash@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.251] [info][status][plugin:reporting@6.4.2] Status changed from yellow to green - Ready

log [03:18:11.253] [info][kibana-monitoring][monitoring-ui] Starting monitoring stats collection

log [03:18:11.475] [info][license][xpack] Imported license information from Elasticsearch for the [monitoring] cluster: mode: basic | status: active

Elasticsearch ERROR: 2018-11-05T03:24:00Z

Error: Request error, retrying

GET https://kubernetes.mycluster.com:30099/_searchguard/license => write EPROTO 140599130773312:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:…/deps/openssl/openssl/ssl/s3_pkt.c:1498:SSL alert number 40

140599130773312:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:…/deps/openssl/openssl/ssl/s3_pkt.c:659:

at Log.error (/usr/share/kibana/node_modules/elasticsearch/src/lib/log.js:225:56)

at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:258:18)

at HttpConnector. (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:157:7)

at ClientRequest.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)

at emitOne (events.js:116:13)

at ClientRequest.emit (events.js:211:7)

at TLSSocket.socketErrorListener (_http_client.js:387:9)

at emitOne (events.js:116:13)

at TLSSocket.emit (events.js:211:7)

at onwriteError (_stream_writable.js:418:12)

at onwrite (_stream_writable.js:440:5)

at _destroy (internal/streams/destroy.js:39:7)

at TLSSocket.Socket._destroy (net.js:564:3)

at TLSSocket.destroy (internal/streams/destroy.js:32:8)

at WriteWrap.afterWrite (net.js:866:10)

Elasticsearch WARNING: 2018-11-05T03:24:00Z

Unable to revive connection: https://kubernetes. mycluster.com:30099/

Elasticsearch WARNING: 2018-11-05T03:24:00Z

No living connections

error [03:24:00.574] Error: No Living connections

at sendReqWithConnection (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:225:15)

at next (/usr/share/kibana/node_modules/elasticsearch/src/lib/connection_pool.js:213:7)

at _combinedTickCallback (internal/process/next_tick.js:131:7)

at process._tickDomainCallback (internal/process/next_tick.js:218:9)

**Elasticsearch log : **

[2018-10-29T08:48:51,625][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Info: SearchGuardLicense [uid=00000000-0000-0000-0000-000000000000, type=TRIAL, features=[COMPLIANCE], issueDate=2018-10-25, expiryDate=2018-12-25, issuedTo=The world, issuer=floragunn GmbH, startDate=2018-10-25, majorVersion=6, clusterName=*, allowedNodeCount=2147483647, msgs=, expiresInDays=56, isExpired=false, valid=true, action=, prodUsage=Yes, one cluster with all commercial features and unlimited nodes per cluster., clusterService=org.elasticsearch.cluster.service.ClusterService@744c2d63, getMsgs()=, getExpiresInDays()=56, isExpired()=false, isValid()=true, getAction()=, getProdUsage()=Yes, one cluster with all commercial features and unlimited nodes per cluster.]

[2018-10-29T08:48:51,626][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Type: TRIAL, valid

[2018-10-29T08:48:51,626][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Node ‘es-client-666c88cfb5-dmw7c’ initialized

[2018-10-29T08:50:38,214][INFO ][o.e.c.s.ClusterApplierService] [es-client-666c88cfb5-dmw7c] added {{es-master-8ff849f9-j5z8l}{hOYJ3NcIQLGIniyHcUy14A}{D9WeiTgCT_CiH1T9CDx6vg}{192.168.1.50}{192.168.1.50:9300}{ml.machine_memory=8352608256, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {es-master-8ff849f9-d7s6g}{_8vGh29LSFWHxxo_TZLcsQ}{iVusXa7vS8C7YHsyjXhltA}{192.168.2.45}{192.168.2.45:9300}{ml.machine_memory=8352608256, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} committed version [4303]])

[2018-11-02T07:13:58,878][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [es-client-666c88cfb5-dmw7c] SSL Problem error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate

javax.net.ssl.SSLHandshakeException: error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate

at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1104) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1064) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1170) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]

**Certificate info : **

openssl x509 -in elasticsearch-admin.crt.pem -noout -text

Certificate:

Data:

Version: 3 (0x2)

Serial Number: 10 (0xa)

Signature Algorithm: sha256WithRSAEncryption

Issuer: DC=ICES, O=MYCLUSTER, OU=ICES, CN=ICES, Signing CA

Validity

Not Before: Feb 28 04:06:01 2018 GMT

Not After : Feb 28 04:06:01 2020 GMT

Subject: C=US, O=MYCLUSTER, OU=ICES, CN=elasticsearch-admin

Subject Public Key Info:

Public Key Algorithm: rsaEncryption

Public-Key: (2048 bit)

Modulus:

00:db:15:db:b0:06:cc:de:c4:2d:aa:77:c8:08:f7:

31:20:6b:2b:78:45:00:02:89:9a:a3:7c:dc:82:3b:

65:ca:3f:31:01:17:b3:8f:19:dd:12:8a:8b:dc:fe:

ae:65:e9:d5:22:7e:93:ac:53:f8:30:98:91:92:ef:

19:8d:20:aa:2e:a9:20:68:05:61:06:ce:b6:e5:01:

b0:5a:3a:da:7c:1c:e7:e5:2d:c3:99:45:60:37:10:

Exponent: 65537 (0x10001)

X509v3 extensions:

X509v3 Key Usage: critical

Digital Signature, Key Encipherment

X509v3 Basic Constraints:

CA:FALSE

X509v3 Extended Key Usage:

TLS Web Server Authentication, TLS Web Client Authentication

X509v3 Subject Key Identifier:

C0:35:91:E3:2E:DC:2E:95:B9:D3:FC:86:9C:82:C5:86:DC:39:38:67

X509v3 Authority Key Identifier:

keyid:6C:B5:45:52:EF:CA:62:0E:8D:36:9B:C7:17:9E:F6:BC:D7:90:13:64

You received this message because you are subscribed to the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/15294b40-6802-4d98-94bd-331629de1c0b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.