Elasticsearch cluster with safeguard plugin logging error when searchguard.disabled: true

When disable SearchgGard with searchguard.disabled: true everything works fine but elasticsearch node log some error.

Log from elasticsearch node: https://gist.github.com/hossainemruz/0d12329936367727e55ba09ef97e16f1
elasticsearch.yml: https://gist.github.com/hossainemruz/3c19905209c0226e3f45b3ec0f66dc99

Is it normal behavior?

Elasticsearch Version: 6.3
Search Gurad Version: v6

From the log message this seems like an error on the transport (inter-node) layer. Are you sure you have disabled Search Guard on all nodes in your cluster? Maybe on has SG still active (and thus uses TLS), and the others have SG disabled.

···

On Wednesday, August 15, 2018 at 11:48:56 AM UTC+2, emruz@appscode.com wrote:

When disable SearchgGard with searchguard.disabled: true everything works fine but elasticsearch node log some error.

Log from elasticsearch node: https://gist.github.com/hossainemruz/0d12329936367727e55ba09ef97e16f1
elasticsearch.yml: https://gist.github.com/hossainemruz/3c19905209c0226e3f45b3ec0f66dc99

Is it normal behavior?

Elasticsearch Version: 6.3
Search Gurad Version: v6

I have only one node in my cluster.

···

On Thursday, August 16, 2018 at 12:22:45 AM UTC+6, Jochen Kressin wrote:
From the log message this seems like an error on the transport (inter-node) layer. Are you sure you have disabled Search Guard on all nodes in your cluster? Maybe on has SG still active (and thus uses TLS), and the others have SG disabled.
On Wednesday, August 15, 2018 at 11:48:56 AM UTC+2, em...@appscode.com wrote:
When disable SearchgGard with searchguard.disabled: true everything works fine but elasticsearch node log some error.
Log from elasticsearch node: https://gist.github.com/hossainemruz/0d12329936367727e55ba09ef97e16f1
elasticsearch.yml: https://gist.github.com/hossainemruz/3c19905209c0226e3f45b3ec0f66dc99
Is it normal behavior?
Elasticsearch Version: 6.3
Search Gurad Version: v6

Strange. This line here:

[2018-08-15T07:48:20,947][WARN ][o.e.t.n.Netty4Transport ] [es-monitoring-cluster-0] exception caught on transport layer [NettyTcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:37476}], closing connection
io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (16,3,3,0)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]

``

Means that someone talks to the ES cluster on transport layer, and the message format is incorrect. Are you sure there are no other services running?

···

On Thursday, August 16, 2018 at 6:14:30 AM UTC+2, emruz@appscode.com wrote:

I have only one node in my cluster.

On Thursday, August 16, 2018 at 12:22:45 AM UTC+6, Jochen Kressin wrote:
From the log message this seems like an error on the transport (inter-node) layer. Are you sure you have disabled Search Guard on all nodes in your cluster? Maybe on has SG still active (and thus uses TLS), and the others have SG disabled.
On Wednesday, August 15, 2018 at 11:48:56 AM UTC+2, em...@appscode.com wrote:
When disable SearchgGard with searchguard.disabled: true everything works fine but elasticsearch node log some error.
Log from elasticsearch node: https://gist.github.com/hossainemruz/0d12329936367727e55ba09ef97e16f1
elasticsearch.yml: https://gist.github.com/hossainemruz/3c19905209c0226e3f45b3ec0f66dc99
Is it normal behavior?
Elasticsearch Version: 6.3
Search Gurad Version: v6