Invalid internal transport message format-while initializing Search-Guard

When posting in this category, please add:
On 30june 2019

  • Elasticsearch logfiles on debug level

  • Your Search Guard configuration files
    didn,t changes any config

  • Your elasticsearch.yml configuration file
    cluster.name: ABC
    node.name: node-0
    node.master: false
    node.data: false
    node.ingest: false
    search.remote.connect: false
    discovery.zen.ping.unicast.hosts: [“localhost IP”]

searchguard.disabled: true
xpack.security.enabled: false
#searchguard.ssl.http.enabled: false

---------------------------------- Search Guard SSL ----------------------------
searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.keystore_filepath: /etc/elasticsearch/keystore.jks
searchguard.ssl.transport.keystore_alias: elasticsearch
searchguard.ssl.transport.keystore_password: *********
searchguard.ssl.transport.truststore_filepath: /etc/elasticsearch/truststore.jks
searchguard.ssl.transport.truststore_alias: truststore
searchguard.ssl.transport.truststore_password: *********
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: false

---------------------------------- HTTP/REST layer SSL ----------------------------
searchguard.ssl.http.enabled: true
searchguard.ssl.http.keystore_filepath: /etc/elasticsearch/keystore.jks
searchguard.ssl.http.keystore_alias: elasticsearch
searchguard.ssl.http.keystore_password: *********
searchguard.ssl.http.truststore_filepath: /etc/elasticsearch/truststore.jks
searchguard.ssl.http.truststore_alias: truststore
searchguard.ssl.http.truststore_password: *********
searchguard.nodes_dn:

“CN=*********, OU=*********, O=*********, L=*********, ST=*********, C=*********”
searchguard.authcz.admin_dn:
“CN=*********, OU=*********, O=*********, L=*********, ST=*********, C=*********”
If you are using Kibana, please also add:

*error


when i tryed to initilize the search-guard then its show below error
command:
sudo chmod +x ./sgadmin.sh && sudo ./sgadmin.sh -h …[IP address] -cd …/sgconfig -cn abc -ts /etc/elasticsearch/truststore.jks -tspass ***** tsalias truststore -ks /etc/elasticsearch/keystore.jks -kspass ***** -nhnv

error:


[INFO ][o.e.h.n.Netty4HttpServerTransport] [node-1] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[INFO ][o.e.n.Node ] [node-1] started
[INFO ][c.f.s.SearchGuardPlugin ] 0 Search Guard modules loaded so far:
[INFO ][o.e.l.LicenseService ] [node-1] license [**********************] mode [basic] - valid
[INFO ][o.e.g.GatewayService ] [node-1] recovered [0] indices into cluster_state
[WARN ][o.e.t.n.Netty4Transport ] [node-1] exception caught on transport layer [NettyTcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:48096}], closing connection
io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (16,3,3,0)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
Caused by: java.io.StreamCorruptedException: invalid internal transport message format, got (16,3,3,0)
at org.elasticsearch.transport.TcpTransport.validateMessageHeader(TcpTransport.java:1327) ~[elasticsearch-6.4.0.jar:6.4.0]

please help me to resolve this?

Which Elasticsearch and which Search Guard version you are using?

Es 6.4And search guard :- com.floragunn:search-guard-6:6.4.0-24.3

please help me to solve this

Seems you have search guard disabled in elasticsearch.yml. Please remove searchguard.disabled: true from all your notes or set searchguard.disabled to false

after setting searchguard.disabled to false we are getting error

Journalctl logs

[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found [], but needed [-1]), pinging again
[DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [node-0] timed out while retrying [cluster:monitor/health] after failure (timeout [30s])
[DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [node-0] no known master node, scheduling a retry
: [2019-06-30T13:39:11,693][WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found [], but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found [], but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found [], but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found [], but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found [], but needed [-1]), pinging again
[o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found [], but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found [], but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found [], but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found [], but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found [], but needed [-1]), pinging again

Elastic search logs:-

[WARN ][o.e.n.Node ] [node-0] timed out while waiting for initial discovery state - timeout: 30s
[INFO ][c.f.s.h.SearchGuardHttpServerTransport] [node-0] publish_address {... :9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}, {... :9200}
[INFO ][o.e.n.Node ] [node-0] started
[INFO ][c.f.s.SearchGuardPlugin ] 4 Search Guard modules loaded so far: [Module [type=AUDITLOG, implementing class=com.floragunn.searchguard.auditlog.impl.AuditLogImpl], Module [type=DLSFLS, implementing class=com.floragunn.searchguard.configuration.SearchGuardFlsDlsIndexSearcherWrapper], Module [type=MULTITENANCY, implementing class=com.floragunn.searchguard.configuration.PrivilegesInterceptorImpl], Module [type=REST_MANAGEMENT_API, implementing class=com.floragunn.searchguard.dlic.rest.api.SearchGuardRestApiActions]]
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found [], but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found [], but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found [], but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found [], but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found [], but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found [], but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found [], but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found [], but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found [], but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found [], but needed [-1]), pinging again
[DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [node-0] timed out while retrying [indices:admin/exists] after failure (timeout [1m])
[ERROR][c.f.s.c.IndexBaseConfigurationRepository] Failure while checking MasterNotDiscoveredException[null] index searchguard
org.elasticsearch.discovery.MasterNotDiscoveredException: null
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:223) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:317) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:244) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:573) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) [elasticsearch-6.4.0.jar:6.4.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
[2019-06-30T13:28:29,474][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [node-0] no known master node, scheduling a retry

How many nodes (in total) do you have in your cluster?

If you install Search Guard the first time in your cluster you must take down all nodes, install and configure Search Guard and bring them up again.

We have 1 linux machine and installed elastic search on same machine and then installing search guard on same machine.

Total machine:1
Both thing are tried to config. On same machine

With only one node it makes no sense to have node.master: false because you need at least one master eligible node (and that’s not Search Guard related)

Thanks
i have udpated node.master to true and search.remote.connect to true and i see below error.

since my elasticsearch and seachguard installed on same liniux machine will it be possible to set up seachguard ?

node.name: node-0
node.master: true
node.data: false
node.ingest: false
search.remote.connect: true
discovery.zen.ping.unicast.hosts: ["..."]

Journalctl logs

[INFO ][c.f.s.SearchGuardPlugin ] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting ‘http.compression: true’ in elasticsearch.yml
[o.e.x.m.j.p.l.CppLogMessageHandler] [controller/9902] [Main.cc@109] controller (64 bit): Version 6.4.0 (Build cf8246175efff5) Copyright © 2018 Elasticsearch BV
[INFO ][c.f.s.a.i.AuditLogImpl ] Configured categories on rest layer to ignore: [AUTHENTICATED, GRANTED_PRIVILEGES]
[INFO ][c.f.s.a.i.AuditLogImpl ] Configured categories on transport layer to ignore: [AUTHENTICATED, GRANTED_PRIVILEGES]
[INFO ][c.f.s.a.i.AuditLogImpl ] Configured Users to ignore: [kibanaserver]
[INFO ][c.f.s.a.i.AuditLogImpl ] Configured Users to ignore for read compliance events: [kibanaserver]
[INFO ][c.f.s.a.i.AuditLogImpl ] Configured Users to ignore for write compliance events: [kibanaserver]
[ERROR][c.f.s.a.s.SinkProvider ] Default endpoint could not be created, auditlog will not work properly.
[WARN ][c.f.s.a.r.AuditMessageRouter] No default storage available, audit log may not work properly. Please check configuration.
[INFO ][c.f.s.a.i.AuditLogImpl ] Message routing enabled: false
[WARN ][c.f.s.c.ComplianceConfig ] If you plan to use field masking pls configure searchguard.compliance.salt to be a random string of 16 chars length identical on all nodes
[INFO ][c.f.s.c.ComplianceConfig ] PII configuration [auditLogPattern=null, auditLogIndex=null]: {}
[DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin com.floragunn.searchguard.SearchGuardPlugin
][INFO ][o.e.d.DiscoveryModule ] [node-0] using discovery type [zen]
[o.e.n.Node ] [node-0] initialized
[INFO ][o.e.n.Node ] [node-0] starting …
[INFO ][o.e.t.TransportService ] [node-0] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[c.f.s.c.IndexBaseConfigurationRepository] Check if searchguard index exists …
[o.e.a.a.i.e.i.TransportIndicesExistsAction] [node-0] no known master node, scheduling a retry
[INFO ][o.e.c.s.MasterService ] [node-0] zen-disco-elected-as-master ([0] nodes joined)[, ], reason: new_master {node-0}{ZyFtFnOVQfa52VTW7zTCww}{YLIiVUkiRfam_JV-jg3C6A}{localhost}{127.0.0.1:9300}{ml.machine_memory=50380750848, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[INFO ][o.e.c.s.ClusterApplierService] [node-0] new_master {node-0}{ZyFtFnOVQfa52VTW7zTCww}{YLIiVUkiRfam_JV-jg3C6A}{localhost}{127.0.0.1:9300}{ml.machine_memory=50380750848, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {node-0}{ZyFtFnOVQfa52VTW7zTCww}{YLIiVUkiRfam_JV-jg3C6A}{localhost}{127.0.0.1:9300}{ml.machine_memory=50380750848, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)[, ]]])
[INFO ][c.f.s.h.SearchGuardHttpServerTransport] [node-0] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}
[INFO ][o.e.n.Node ] [node-0] started
[INFO ][c.f.s.SearchGuardPlugin ] 4 Search Guard modules loaded so far: [Module [type=REST_MANAGEMENT_API, implementing class=com.floragunn.searchguard.dlic.rest.api.SearchGuardRestApiActions], Module [type=AUDITLOG, implementing class=com.floragunn.searchguard.auditlog.impl.AuditLogImpl], Module [type=DLSFLS, implementing class=com.floragunn.searchguard.configuration.SearchGuardFlsDlsIndexSearcherWrapper], Module [type=MULTITENANCY, implementing class=com.floragunn.searchguard.configuration.PrivilegesInterceptorImpl]]
[INFO ][o.e.l.LicenseService ] [node-0] license [9f59ccdb-7a93-41c4-95d5-ce52d34ae9fe] mode [basic] - valid
[INFO ][c.f.s.c.IndexBaseConfigurationRepository] searchguard index does not exist yet, so no need to load config on node startup. Use sgadmin to initialize cluster
[INFO ][o.e.g.GatewayService ] [node-0] recovered [0] indices into cluster_state
[ERROR][c.f.s.a.BackendRegistry ] Not yet initialized (you may need to run sgadmin)

Elastic search logs:-

[INFO ][c.f.s.SearchGuardPlugin ] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting ‘http.compression: true’ in elasticsearch.yml
[INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/9902] [Main.cc@109] controller (64 bit): Version 6.4.0 (Build cf8246175efff5) Copyright © 2018 Elasticsearch BV
[INFO ][c.f.s.a.i.AuditLogImpl ] Configured categories on rest layer to ignore: [AUTHENTICATED, GRANTED_PRIVILEGES]
[INFO ][c.f.s.a.i.AuditLogImpl ] Configured categories on transport layer to ignore: [AUTHENTICATED, GRANTED_PRIVILEGES]
[INFO ][c.f.s.a.i.AuditLogImpl ] Configured Users to ignore: [kibanaserver]
[INFO ][c.f.s.a.i.AuditLogImpl ] Configured Users to ignore for read compliance events: [kibanaserver]
[INFO ][c.f.s.a.i.AuditLogImpl ] Configured Users to ignore for write compliance events: [kibanaserver]
[ERROR][c.f.s.a.s.SinkProvider ] Default endpoint could not be created, auditlog will not work properly.
[WARN ][c.f.s.a.r.AuditMessageRouter] No default storage available, audit log may not work properly. Please check configuration.
[INFO ][c.f.s.a.i.AuditLogImpl ] Message routing enabled: false
[WARN ][c.f.s.c.ComplianceConfig ] If you plan to use field masking pls configure searchguard.compliance.salt to be a random string of 16 chars length identical on all nodes
[INFO ][c.f.s.c.ComplianceConfig ] PII configuration [auditLogPattern=null, auditLogIndex=null]: {}
[DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin com.floragunn.searchguard.SearchGuardPlugin
[INFO ][o.e.d.DiscoveryModule ] [node-0] using discovery type [zen]
[INFO ][o.e.n.Node ] [node-0] initialized
[INFO ][o.e.n.Node ] [node-0] starting …
[INFO ][o.e.t.TransportService ] [node-0] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[INFO ][c.f.s.c.IndexBaseConfigurationRepository] Check if searchguard index exists …
[DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [node-0] no known master node, scheduling a retry
[INFO ][o.e.c.s.MasterService ] [node-0] zen-disco-elected-as-master ([0] nodes joined)[, ], reason: new_master {node-0}{ZyFtFnOVQfa52VTW7zTCww}{YLIiVUkiRfam_JV-jg3C6A}{localhost}{127.0.0.1:9300}{ml.machine_memory=50380750848, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[INFO ][o.e.c.s.ClusterApplierService] [node-0] new_master {node-0}{ZyFtFnOVQfa52VTW7zTCww}{YLIiVUkiRfam_JV-jg3C6A}{localhost}{127.0.0.1:9300}{ml.machine_memory=50380750848, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {node-0}{ZyFtFnOVQfa52VTW7zTCww}{YLIiVUkiRfam_JV-jg3C6A}{localhost}{127.0.0.1:9300}{ml.machine_memory=50380750848, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)[, ]]])
[INFO ][c.f.s.h.SearchGuardHttpServerTransport] [node-0] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}
[INFO ][o.e.n.Node ] [node-0] started
[INFO ][c.f.s.SearchGuardPlugin ] 4 Search Guard modules loaded so far: [Module [type=REST_MANAGEMENT_API, implementing class=com.floragunn.searchguard.dlic.rest.api.SearchGuardRestApiActions], Module [type=AUDITLOG, implementing class=com.floragunn.searchguard.auditlog.impl.AuditLogImpl], Module [type=DLSFLS, implementing class=com.floragunn.searchguard.configuration.SearchGuardFlsDlsIndexSearcherWrapper], Module [type=MULTITENANCY, implementing class=com.floragunn.searchguard.configuration.PrivilegesInterceptorImpl]]
[INFO ][o.e.l.LicenseService ] [node-0] license [9f59ccdb-7a93-41c4-95d5-ce52d34ae9fe] mode [basic] - valid
[INFO ][c.f.s.c.IndexBaseConfigurationRepository] searchguard index does not exist yet, so no need to load config on node startup. Use sgadmin to initialize cluster
[INFO ][o.e.g.GatewayService ] [node-0] recovered [0] indices into cluster_state
[ERROR][c.f.s.a.BackendRegistry ] Not yet initialized (you may need to run sgadmin)

Again, with only one node it makes no sense to have node.data: false because you need at least one data node to store your data (and that’s not Search Guard related)

Please refer to https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html and try to get elasticsearch (and Kibana if applicable) without Search Guard fully up and running first. Then install Search Guard.

thank you

How many nodes do we need to set up search guard with elastic search ?

Is it possible to set up Elastic search and search guard on single node ?

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.