Setting up a new 3 7.3.1 node cluster in order to test SearchGuard capabilities. Each node is acting as data, ingestor, and capable of being master. Kibana, filebeat, and metricbeat are loaded on each node and after initial installation script, all nodes stand up properly and form the desired cluster. The Elastic Cluster setup is currently working as is.
Snapshot the VMs, and started to work on adding SearchGuard by following the installation steps. I’m using the initial demo certificates to test installation and use before biting off more and working in custom certificate generation for production use. The demo certs are saved to /etc/searchguard with root:elasticsearch permissions and all config files updated following the install steps.
When issuing the final sgadmin.sh command, the tool provides the below first error output. This happens with -nhnv recommended by the troubleshooting page. The installation steps page does not show what the expected output is, so I have no idea if this error is truly a concern, but assume it is.
root@elk-172-16-5-141:/etc/searchguard# bash /usr/share/elasticsearch/plugins/search-guard-7/tools/sgadmin.sh --enable-shard-allocation -cert ./kirk.pem -key ./kirk-key.pem -cacert ./root-ca.pem
WARNING: JAVA_HOME not set, will use /usr/bin/java
Search Guard Admin v7
Will connect to localhost:9300 … done
ERR: Cannot connect to Elasticsearch. Please refer to elasticsearch logfile for more information
Trace:
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{rJmuZthvRj-WGzNKCHVQGw}{localhost}{127.0.0.1:9300}]]
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:352)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:248)
at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:57)
at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:394)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:392)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:381)
at com.floragunn.searchguard.tools.SearchGuardAdmin.execute(SearchGuardAdmin.java:510)
at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:142)
When posting in this category, please add:
-
Elasticsearch logfiles on debug level
2019-09-17T18:40:30,088][WARN ][o.e.t.TcpTransport ] [elk-172-16-5-141] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:59574}], closing connection
io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (16,3,3,1)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.36.Final.jar:4.1.36.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-common-4.1.36.Final.jar:4.1.36.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.36.Final.jar:4.1.36.Final]
at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: java.io.StreamCorruptedException: invalid internal transport message format, got (16,3,3,1)
at org.elasticsearch.transport.TcpTransport.readHeaderBuffer(TcpTransport.java:745) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.TcpTransport.readMessageLength(TcpTransport.java:731) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.netty4.Netty4SizeHeaderFrameDecoder.decode(Netty4SizeHeaderFrameDecoder.java:40) ~[transport-netty4-client-7.3.1.jar:7.3.1]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.36.Final.jar:4.1.36.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.36.Final.jar:4.1.36.Final]
… 20 more -
Your Search Guard configuration files
example config file provided by installation instructions -
Your elasticsearch.yml configuration file
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
cluster.name: ELK-Internal
node.name: ${HOSTNAME}
action.auto_create_index: .monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*
network.host: [local, site]
discovery.seed_hosts: [172.16.5.141, 172.16.5.142, 172.16.5.143]
cluster.initial_master_nodes: [172.16.5.141, 172.16.5.142, 172.16.5.143]
searchguard.ssl.transport.pemcert_filepath: /etc/searchguard/esnode.pem
searchguard.ssl.transport.pemkey_filepath: /etc/searchguard/esnode-key.pem
#searchguard.ssl.transport.pemkey_password: /etc/searchguard/
searchguard.ssl.transport.pemtrustedcas_filepath: /etc/searchguard/root-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: true
searchguard.authcz.admin_dn:
- CN=kirk,OU=client,O=client,L=test, C=de
If you are using Kibana, please also add:
- Your kibana.yml configuration file
server.host: 0.0.0.0