ELK+SG migration from 5.6.6 to 6.6.1 problem on Rolling upgrade

Search Guard
1

Hi :slight_smile:

  • Search Guard and Elasticsearch version
  • ELK 5.6.6 & search-guard-5-5.6.6-18
  • Installed and used enterprise modules, if any

dlic-search-guard-authbackend-ldap-5.6-11-jar-with-dependencies.jar

dlic-search-guard-module-auditlog-5.3-5-jar-with-dependencies.jar

dlic-search-guard-module-dlsfls-5.3-7-jar-with-dependencies.jar

dlic-search-guard-module-kibana-multitenancy-5.4-5-jar-with-dependencies.jar

dlic-search-guard-rest-api-5.3-6-jar-with-dependencies.jar

The case:

On our TEST ENV we have 4 data nodes, 2
masters and 1 coordinator.

All of them are in 5.6.6 with SG of
course. We are doing a rolling migration.

Today we made some test, and one of data
node was upgraded to 6.6.1 with correspond SG plugin.

Configuration was also adjusted as docs
says, and so on.

So, after starting the Node, its
connected to the rest of the cluster, and I can see him on the node list and
Shards are available again on him. **But **,
the problem is with authentication to this specific node. If our Logstash
or scripts or anybody want to connect to “6x” version of node, the node throw
errors with those logs:

ELK
NODE version 6:

[2019-03-21T15:50:38,401][ERROR][c.f.s.a.BackendRegistry
] [<>] Not yet initialized (you may need to run sgadmin)

Logstash
tried to push data to it:

2019-03-21T15:57:03,451][WARN
][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES
instance, but got an error. {:url=>“https://logstash:xxxxxx@xx.xx.xx.83:9200/”,
:error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError,
:error=>“Got response code ‘503’ contacting Elasticsearch at URL ‘https://xx.xx.xx.83:9200/’”}

And question is **What happened? Why? How can we
debug it further ? **

How we can fix it
and enable authentication again on this host when rest of the cluster are still
in version of 5 ?

Had someone this problem also ?

Any ideas ?

thanks in advance

Przemek