Kerberos acceptor_principal seems to be ignored

Using ES 6.8.6 and latest SG.
Using kerberos auth we noticed that the setting searchguard.kerberos.acceptor_principal seems to be ignored.
Here’s the symptom:

On node01 in elasticsearch.yml we have:

searchguard.kerberos.acceptor_keytab_filepath: sg.keytab
searchguard.kerberos.acceptor_principal: HTTP/node01.example.com@EXAMPLE.COM

sg.keytab contains:

Keytab name: FILE:/etc/elasticsearch/test/sg.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   1 08/02/2019 16:51:22 HTTP/node01.example.com@EXAMPLE.COM (aes256-cts-hmac-sha1-96)
   1 08/02/2019 16:51:22 HTTP/node01.example.com@EXAMPLE.COM (aes128-cts-hmac-sha1-96)

This is all nice and works. However, if we simply replace the keytab with another one from node02 which is identical except for the hostname, then the requests still get accepted, even without changing the acceptor principal to ...node02....

Now you have 2 nodes, with the following config, right? And you have both nodes successfully authenticated and serving requests, right?

Node 1

elasticsearch.yml

searchguard.kerberos.acceptor_keytab_filepath: sg.keytab
searchguard.kerberos.acceptor_principal: HTTP/node01.example.com@EXAMPLE.COM

sg.keytab

Keytab name: FILE:/etc/elasticsearch/test/sg.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   1 08/02/2019 16:51:22 HTTP/node02.example.com@EXAMPLE.COM (aes256-cts-hmac-sha1-96)
   1 08/02/2019 16:51:22 HTTP/node02.example.com@EXAMPLE.COM (aes128-cts-hmac-sha1-96)

Node 2

elasticsearch.yml

searchguard.kerberos.acceptor_keytab_filepath: sg.keytab
searchguard.kerberos.acceptor_principal: HTTP/node02.example.com@EXAMPLE.COM

sg.keytab

Keytab name: FILE:/etc/elasticsearch/test/sg.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   1 08/02/2019 16:51:22 HTTP/node02.example.com@EXAMPLE.COM (aes256-cts-hmac-sha1-96)
   1 08/02/2019 16:51:22 HTTP/node02.example.com@EXAMPLE.COM (aes128-cts-hmac-sha1-96)

Do you see any errors or warnings in the Authentication Server and Elasticsearch logs?

Gather some debug data to post it here. To enable the debug:

  1. Put krb_debug: true in sg_config.yml
  2. If you see no debug put the following JVM options
-Dsun.security.krb5.debug=true
-Djava.security.debug=gssloginconfig,logincontext,configparser,configfile
-Dsun.security.spnego.debug=true

yes exactly. let me check the logs

the elasticsearch logfile claims Kerberos debug is enabled on stdout so I guess I can only see it in journalctl ?

I think yes, if you use systemd to run elasticsearch.