es 5.5 + sg 5.5 No way to initialize the cluster with sgadmin

Hello,

after several days of struggling with custom certificates I have tested the demo downloaded certificates from the official site but the problem remain exactly the same.

I have two nodes to configured in the cluster

I wanted to start from a clean environment so I deleted the node data directory from both the nodes and configured exactly in the same way, except for the node name of course.

they start correctly and they add each other to the cluster

[2018-09-25T13:16:22,312][INFO ][o.e.c.s.ClusterService ] [Abnahme-1] new_master {Abnahme-1}{Quai70juRjWxt0IEPuwLiQ}{krJAVsFIRgaFMAukc1Py6g}{10.221.118.56}{10.221.118.56:9300}, added {{Abnahme-2}{rDjr_v1dTeiWuTIvHg8VMQ}{kWIxrkgpSXm2vWdS58R4bg}{10.221.118.57}{10.221.118.57:9300},}, reason: zen-disco-elected-as-master ([1] nodes joined)[{Abnahme-2}{rDjr_v1dTeiWuTIvHg8VMQ}{kWIxrkgpSXm2vWdS58R4bg}{10.221.118.57}{10.221.118.57:9300}]
[2018-09-25T13:16:22,399][INFO ][c.f.s.h.SearchGuardHttpServerTransport] [Abnahme-1] publish_address {10.221.118.56:9201}, bound_addresses {[::]:9201}

[2018-09-25T13:16:22,329][INFO ][o.e.c.s.ClusterService ] [Abnahme-2] detected_master {Abnahme-1}{Quai70juRjWxt0IEPuwLiQ}{krJAVsFIRgaFMAukc1Py6g}{10.221.118.56}{10.221.118.56:9300}, added {{Abnahme-1}{Quai70juRjWxt0IEPuwLiQ}{krJAVsFIRgaFMAukc1Py6g}{10.221.118.56}{10.221.118.56:9300},}, reason: zen-disco-receive(from master [master {Abnahme-1}{Quai70juRjWxt0IEPuwLiQ}{krJAVsFIRgaFMAukc1Py6g}{10.221.118.56}{10.221.118.56:9300} committed version [1]])
[2018-09-25T13:16:22,403][INFO ][c.f.s.h.SearchGuardHttpServerTransport] [Abnahme-2] publish_address {10.221.118.57:9201}, bound_addresses {[::]:9201}
[2018-09-25T13:16:22,403][INFO ][o.e.n.Node ] [Abnahme-2] started

… but also configuring elasticsearch.yml with the demo certificate I am not able at all to run sgadmin.

D:\CSA\elasticsearch-5.5.0\plugins\elasticsearch\tools>sgadmin.bat -cert D:\CSA\elasticsearch-5.5.0\config\kirk.pem -key
D:\CSA\elasticsearch-5.5.0\config\kirk-key.pem -cacert D:\CSA\elasticsearch-5.5.0\config\root-ca.pem -nhnv -icl -cd D:
CSA\elasticsearch-5.5.0\plugins\elasticsearch\sgconfig --diagnose
Search Guard Admin v5
Will connect to localhost:9300 … done

LICENSE NOTICE Search Guard

If you use one or more of the following features in production
make sure you have a valid Search Guard license
(See https://floragunn.com/searchguard-validate-license)

  • Kibana Multitenancy
  • LDAP authentication/authorization
  • Active Directory authentication/authorization
  • REST Management API
  • JSON Web Token (JWT) authentication/authorization
  • Kerberos authentication/authorization
  • Document- and Fieldlevel Security (DLS/FLS)
  • Auditlogging
    In case of any doubt mail to sales@floragunn.com

sgadmin_diag_trace_2018-Sep-25_14-03-55.txt (47.3 KB)

···

###################################
Diagnostic trace written to: D:\CSA\elasticsearch-5.5.0\plugins\elasticsearch\tools\sgadmin_diag_trace_2018-Sep-25_14-03
-55.txt
Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …
Clustername: Abnahme
Clusterstate: GREEN
Number of nodes: 2
Number of data nodes: 2
ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists.
See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
Trace:
ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/flor
agunncom/search-guard-docs/blob/master/sgadmin.md]
at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportActi
on.java:64)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportActi
on.java:54)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRe
questHandler.java:178)
at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandl
er.java:192)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHan
dler.java:140)
at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1544)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1501)
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1385)
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:7
4)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1267)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at java.lang.Thread.run(Thread.java:748)

here the configuration:

======================== Elasticsearch Configuration =========================

cluster.name: “Abnahme”
node.name: “Abnahme-1”
node.master: true
network.host: 0.0.0.0
transport.host: 0.0.0.0
http.port: 9201
http.enabled: true
transport.tcp.port: 9300
path.data: D:\CSA/elasticsearch-5.5.0/data
path.logs: D:\CSA/elasticsearch-5.5.0/logs
discovery.zen.ping_timeout: 5s
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts:

  • 10.221.118.56:9300
  • 10.221.118.57:9300
    searchguard.ssl.http.enabled: true
    searchguard.ssl.http.pemcert_filepath: D:\CSA\elasticsearch-5.5.0\config\esnode.pem
    searchguard.ssl.http.pemkey_filepath: D:\CSA\elasticsearch-5.5.0\config\esnode-key.pem
    searchguard.ssl.http.pemtrustedcas_filepath: D:\CSA\elasticsearch-5.5.0\config\root-ca.pem
    searchguard.ssl.transport.pemcert_filepath: esnode.pem
    searchguard.ssl.transport.pemkey_filepath: esnode-key.pem
    searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem
    searchguard.ssl.transport.enforce_hostname_verification: false
    searchguard.ssl.transport.resolve_hostname: false
    searchguard.nodes_dn:
  • “*”

searchguard.authcz.admin_dn:

  • CN=kirk,OU=client,O=client,L=test,C=de

I have read hundreds of posts with the same problem and the solutions but it seems that no one is working for my configuration and I do not really know what to do now.

I must use specific certificates but I would like to make it working with the default one at least.

I attach the diagnose of the last command.

Thanks for your help.

I have solved the demo certificate issue.

Now I am facing a communication problem between the two nodes.

I need to use now the custom certificates

If I start the first node as single node everything is working.

When I start the second node they are not able to handshake each other

It looks like the AES256 is not supported:

[2018-09-26T10:52:51,447][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] AES-256 not supported, max key length for AES is 128 bit… That is not an issue, it just limits possible encryption strength. To enable AES 256 install ‘Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files’

I have checked the java installation directory and the two jars local_policy.jar and US_export_policy.jar are already in the lib\security directory as indicated from the documentation.

I have tried also to set the following parameters in the yml file:

searchguard.ssl.http.enabled_ciphers:

  • “TLS_DHE_RSA_WITH_AES_256_CBC_SHA”
  • “TLS_RSA_WITH_AES_256_CBC_SHA256”

searchguard.ssl.transport.enabled_ciphers:

  • “TLS_DHE_RSA_WITH_AES_256_CBC_SHA”
  • “TLS_RSA_WITH_AES_256_CBC_SHA256”

for both the servers of course, but I get this error:

[2018-09-26T10:54:48,289][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [Abnahme-1] SSL Problem no cipher suites in common
javax.net.ssl.SSLHandshakeException: no cipher suites in common
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529) ~[?:?]

I suppose the problem is in the AES256, configuration, how can I define it?

Thanks