Thanks for the info about Search Guard not doing any special with regard to shard assignment. With that in mind I’ve done some experimentation and been able to recreate the seemingly impossible shard allocation without using sgadmin.sh. It’s either a bug in Elasticsearch or behaviour I don’t understand.
If I set number_of_replicas for the searchguard index to 1 then the single replica shard is allocated to a node in the datacentre that the primary shard is not in. If I then set auto_expand_replicas for the searchguard index to "0-all" a replica shard is assigned to every hot node and there is also a bunch of shards which are unassigned due to the hot/cold architecture. If I then set number_of_replicas to 5 the unassigned shard count falls to 0 and I’m left with a shard of the searchguard index assigned to every hot node despite the cluster.routing.allocation.awareness settings. I’ve also found that setting number_of_replicas to 1 and then to a number between 2 and 5 results in some, but not all, of the new shards being assigned to hot nodes despite the cluster.routing.allocation.awareness settings.
So it seems that when the number of replica shards is increased Elasticsearch 5.6.14 will sometimes assign shards to nodes they shouldn’t be assigned to. And that becomes a problem when a node with a shard that shouldn’t be assigned to it gets restarted.
The cluster will soon be upgraded to Elasticsearch 6 so I’m going to forget about it for now and see if I can receate the same behaviour with 6.