watcher is pined to one node. As I so when this node left cluster watcher is not reschedule on other node.
Any workaround for this?
Is there the way to reschedule all watchers from API?
There is an API endpoint you can use to update the watch https://docs.search-guard.com/latest/elasticsearch-alerting-rest-api-watch-put Just put the same watch with the same id but have a new schedule.
The node filter setting is definitive, it is not just a preference. When you are configuring a node filter which does not match any nodes, the watches won’t be executed.
For redundancy, you can configure several nodes to match the node filter. Then, when a node leaves the cluster, the watches that were executed here will be automatically rescheduled to another node matching the node filter.
It is however not possible to specify the preferred node. For balancing, the watches will be spread over all nodes matching the node filter.