We have a system with Elasticsearch 6.8.6 running on a mix of Debian 10.4 and Ubuntu Budgie 20.04 hosts. Two hosts each had three VMs with data on them - three shards, there was one replica. A third host was added, a third trio of data VMs is on it, and replica count is being adjusted from one to two. Any one host should have all data in the event of some sort of loss.
When our cluster is rolling restarted it always loses track of its license. That happened again during the introduction of the new host and VM trio. This time it presented differently, requiring a login using the admin cert and the application of this solution in order to unlock the searchguard index itself.
But this is a problem we’ve not experienced in the past - monitoring information is not available.
I looked at indices, this looks like it should be working:
green open .monitoring-beats-6-2020.07.24
green open .monitoring-beats-6-2020.07.25
green open .monitoring-beats-6-2020.07.26
green open .monitoring-beats-6-2020.07.27
green open .monitoring-beats-6-2020.07.28
green open .monitoring-beats-6-2020.07.29
green open .monitoring-beats-6-2020.07.30
green open .monitoring-beats-6-2020.07.31
green open .monitoring-es-6-2020.07.24
green open .monitoring-es-6-2020.07.25
green open .monitoring-es-6-2020.07.26
green open .monitoring-es-6-2020.07.27
green open .monitoring-es-6-2020.07.28
green open .monitoring-es-6-2020.07.29
green open .monitoring-es-6-2020.07.30
green open .monitoring-es-6-2020.07.31
green open .monitoring-kibana-6-2020.07.24
green open .monitoring-kibana-6-2020.07.25
green open .monitoring-kibana-6-2020.07.26
green open .monitoring-kibana-6-2020.07.27
green open .monitoring-kibana-6-2020.07.28
green open .monitoring-kibana-6-2020.07.29
green open .monitoring-kibana-6-2020.07.30
green open .monitoring-kibana-6-2020.07.31
We have tenants in place, I checked my private one, the admin tenant, logged in with the admin account itself - don’t get the usual monitoring interface.
We run Metricbeat on all hosts and VMs - the Infrastructure tab is fine.
Give me a hint, what should I check next?