Assume there’s an ES cluster with SearchGuard initialized, where searchguard index exists.
During the time between when a node joins the cluster and when the node completes loading SearchGuard configurations, an ES node does not accept any indexing requests because the node is considered to be not initialized yet.
I guess the only thing (beside retrying requests) is to use a loadbalancer which checks the health status of a node with our health check endpoint documented here Installation | Security for Elasticsearch | Search Guard (Or if not a loadbalancer maybe the client application can call the health check api on a regular basis)
I see. Good to know the health check endpoint.
We might be able to implement a custom sniffer to send requests to initialized nodes. It can be nice for SearchGuard library to provide the sniffer.
If we retry requests, can we check whether the failures can be transient for this case?
What do you exactly mean with “can we check whether the failures can be transient for this case”?
I thought it can be nice if a client can determine whether a failure can be resolved by retrying requests.
For example, an Exception class might have a flag or code that indicates whether retrying might help or not.
If a failure happens because SG is not initialized, clients might choose to retry requests until it’s initialized.
If a failure happens due to insufficient privilege, clients might choose to stop sending requests.
