Renewing Expired Certs with Rolling Restart

Elasticsearch version:
7.4.0

Server OS version:
CentOS Linux release 7.8.2003 (Core)

Describe the issue:
The certificates that searchguard uses at the transport layer have expired for our Elasticsearch cluster. I was attempting to perform a rolling restart of the cluster and the first node I brought down failed to start back up because of the expired certificate. The rest of the cluster seems to be fine even though their certs are expired too. I have generated new certs using the sgtlstool.sh script and copied them across the cluster, however when I try to start up the node that is offline I am getting an SSL error that says “CertPathValidatorException: Path does not chain with any of the trust anchors”.

I’m hesitant to perform the full cluster restart because the cluster is up and running at the moment (minus 1 node) and if it doesn’t then the whole database will be down.

In order to use the new certificates you need to put them to all nodes and restart. Make sure you have the node certificate and the Root CA certificate on each node. HTTP and transport certificates should be different.

If you are not sure that your certificates are valid, try them in a test environment. For example, you can run a 3 node cluster in a Docker using this repository https://git.floragunn.com/search-guard/search-guard-labs/-/tree/master