Search Guard 6.2.3 + ELK 6.2.3 impersonation

Im attempting to upgrade from ELK 5.6.2 to ELK 6.2.3
Elasticsearch seems to have gone fine, all API actions are working as expected (search guard included), however Kibana keeps giving me a 403 for https://kibana.redact.com:4180/api/saved_objects/?type=index-pattern&per_page=10000 with error:
‘kibanaserver’
is not allowed to impersonate as ‘kibanaserver’: [security_exception] ‘kibanaserver’ is not allowed to impersonate as ‘kibanaserver’

This all worked fine under ELK 5.6.2, so Im not sure If i missed anything. (the only change to kibana.yml is searchguard.basicauth.enabled: false

ES plugin com.floragunn:search-guard-6:6.2.3-22.0
ES + kibana 6.2.3
Kibana plugin 6.2.3-11
JVM OpenJDK Runtime Environment (build 1.8.0_161-b14)
All oss

nginx.conf:
$ cat nginx.conf
events {
worker_connections 1024; ## Default: 1024
}

http {
server {
listen 5601;
server_name localhost;

    location / {
        proxy_pass https://localhost:5602/;
        proxy_set_header Authorization "Basic a2liYW5hc2VydmVyOmZvbwo=";
        proxy_set_header sg_impersonate_as kibanaserver;
        rewrite /login https://kibana.redact.com:4180/oauth2/sign_in redirect;
    }
}

}

kibana.yml:
$ cat /usr/share/kibana/config/kibana.yml
server.host: “0.0.0.0”
server.port: 5602
elasticsearch.url: “https://es.redact.com:9200
elasticsearch.username: “kibanaserver”
elasticsearch.password: “foo”
elasticsearch.ssl.verificationMode: full
logging.dest: stdout
elasticsearch.requestHeadersWhitelist: [ sg_impersonate_as, authorization ]
searchguard.basicauth.enabled: false
server.ssl.enabled: true
server.ssl.certificate: /usr/share/kibana/bundle.pem
server.ssl.key: /usr/share/kibana/key.pem

sg_internal_user.yml:
kibanaserver:
hash: $2a$hash

/etc/elasticsearch/elasticsearch.yml:
action.destructive_requires_name: true
cluster.name: redact
discovery:
zen.hosts_provider: ec2
http.max_content_length: 500mb
network.host: 0.0.0.0
node.data: true
node.master: true
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
searchguard.ssl.http.clientauth_mode: NONE
searchguard.ssl.http.enabled: true
searchguard.ssl.http.keystore_filepath: keystore.jks
searchguard.ssl.http.truststore_filepath: truststore.jks
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.keystore_filepath: keystore.jks
searchguard.ssl.transport.resolve_hostname: false
searchguard.ssl.transport.truststore_filepath: truststore.jks
xpack.graph.enabled: false
xpack.monitoring.enabled: false
xpack.security.enabled: false
xpack.watcher.enabled: false
discovery.ec2.tag.Environment: Operations
node.name: ip-10-0-3-49
searchguard.ssl.transport.truststore_password: “es”
searchguard.ssl.transport.keystore_password: “es”
searchguard.ssl.http.keystore_password: “es”
searchguard.ssl.http.truststore_password: “es”
searchguard.enable_snapshot_restore_privilege: true
searchguard.nodes_dn:

  • ‘/CN=ip.*/’
    searchguard.authcz.admin_dn:
  • CN=operations
    searchguard.authcz.impersonation_dn:
    “CN=kibanaserver”:
    • “kibanaserver”

It mostly loads: https://imgur.com/a/RUjww

Also GET api/v1/auth/authinfo (right after the 403 from first post) shows a 500: {“statusCode”:500,“error”:“Internal Server Error”,“message”:“An internal server error occurred”}

···

On Monday, April 9, 2018 at 4:29:32 PM UTC-5, marcus young wrote:

$ cat nginx.conf
events {
worker_connections 1024; ## Default: 1024
}

http {
server {
listen 5601;
server_name localhost;

    location / {
        proxy_pass [https://localhost:5602/](https://localhost:5602/);
        proxy_set_header Authorization "Basic a2liYW5hc2VydmVyOmZvbwo=";
        proxy_set_header sg_impersonate_as kibanaserver;
        rewrite /login [https://kibana.redact.com:4180/oauth2/sign_in](https://kibana.redact.com:4180/oauth2/sign_in) redirect;
    }
}

}

$ cat /usr/share/kibana/config/kibana.yml
server.host: “0.0.0.0”
server.port: 5602
elasticsearch.url: “https://es.redact.com:9200
elasticsearch.username: “kibanaserver”
elasticsearch.password: “foo”
elasticsearch.ssl.verificationMode: full
logging.dest: stdout
elasticsearch.requestHeadersWhitelist: [ sg_impersonate_as, authorization ]
searchguard.basicauth.enabled: false
server.ssl.enabled: true
server.ssl.certificate: /usr/share/kibana/bundle.pem
server.ssl.key: /usr/share/kibana/key.pem

kibanaserver:
hash: $2a$hash

action.destructive_requires_name: true
cluster.name: redact
discovery:
zen.hosts_provider: ec2
http.max_content_length: 500mb
network.host: 0.0.0.0
node.data: true
node.master: true
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
searchguard.ssl.http.clientauth_mode: NONE
searchguard.ssl.http.enabled: true
searchguard.ssl.http.keystore_filepath: keystore.jks
searchguard.ssl.http.truststore_filepath: truststore.jks
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.keystore_filepath: keystore.jks
searchguard.ssl.transport.resolve_hostname: false
searchguard.ssl.transport.truststore_filepath: truststore.jks
xpack.graph.enabled: false
xpack.monitoring.enabled: false
xpack.security.enabled: false
xpack.watcher.enabled: false
discovery.ec2.tag.Environment: Operations
node.name: ip-10-0-3-49
searchguard.ssl.transport.truststore_password: “es”
searchguard.ssl.transport.keystore_password: “es”
searchguard.ssl.http.keystore_password: “es”
searchguard.ssl.http.truststore_password: “es”
searchguard.enable_snapshot_restore_privilege: true
searchguard.nodes_dn:

  • ‘/CN=ip.*/’
    searchguard.authcz.admin_dn:
  • CN=operations
    searchguard.authcz.impersonation_dn:
    “CN=kibanaserver”:
    • “kibanaserver”

Im attempting to upgrade from ELK 5.6.2 to ELK 6.2.3
Elasticsearch seems to have gone fine, all API actions are working as expected (search guard included), however Kibana keeps giving me a 403 for https://kibana.redact.com:4180/api/saved_objects/?type=index-pattern&per_page=10000 with error:
‘kibanaserver’
is not allowed to impersonate as ‘kibanaserver’: [security_exception] ‘kibanaserver’ is not allowed to impersonate as ‘kibanaserver’

This all worked fine under ELK 5.6.2, so Im not sure If i missed anything. (the only change to kibana.yml is searchguard.basicauth.enabled: false

ES plugin com.floragunn:search-guard-6:6.2.3-22.0
ES + kibana 6.2.3
Kibana plugin 6.2.3-11
JVM OpenJDK Runtime Environment (build 1.8.0_161-b14)
All oss

nginx.conf:
kibana.yml:
sg_internal_user.yml:
/etc/elasticsearch/elasticsearch.yml:

The 500:

{
“type”: “response”,
@timestamp”: “2018-04-09T21:39:55Z”,
“tags”: ,
“pid”: 5,
“method”: “get”,
“statusCode”: 500,
“req”: {
“url”: “/api/v1/auth/authinfo”,
“method”: “get”,
“headers”: {
“sg_impersonate_as”: “kibanaserver”,
“host”: “localhost:5602”,
“connection”: “close”,
“user-agent”: “Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0”,
“accept”: “application/json, text/plain, /”,
“accept-encoding”: “gzip, deflate, br”,
“accept-language”: “en-US,en;q=0.5”,
“kbn-version”: “6.2.3”,
“referer”: “https://kibana.redact.com:4180/app/kibana”,
“x-amzn-trace-id”: “Root=1–”,
“x-forwarded-email”: "marc@redact.com",
“x-forwarded-for”: “192.168.1.220, 10.0.3.113”,
“x-forwarded-port”: “4180”,
“x-forwarded-proto”: “https”,
“x-forwarded-user”: “marc”
},
“remoteAddress”: “127.0.0.1”,
“userAgent”: “127.0.0.1”,
“referer”: “https://kibana.redact.com:4180/app/kibana
},
“res”: {
“statusCode”: 500,
“responseTime”: 22,
“contentLength”: 9
},
“message”: “GET /api/v1/auth/authinfo 500 22ms - 9.0B”
}

Bumped debug logs for searchguard, ran:
$ curl -k -s -u ‘kibanaserver:foo’ -H “sg_impersonate_as: kibanaserver” https://10.0.3.31:9200/_cat/indices | jq
{
“error”: {
“root_cause”: [
{
“type”: “security_exception”,
“reason”: “‘kibanaserver’ is not allowed to impersonate as ‘kibanaserver’”
}
],
“type”: “security_exception”,
“reason”: “‘kibanaserver’ is not allowed to impersonate as ‘kibanaserver’”
},
“status”: 403
}

only lines I see in elasticsearch logs:
[2018-04-10T00:59:40,482][DEBUG][c.f.s.a.BackendRegistry ] Try to extract auth creds from basic http authenticator
[2018-04-10T00:59:40,497][DEBUG][c.f.s.a.BackendRegistry ] kibanaserver not cached, return from internal backend directly
[2018-04-10T00:59:40,879][DEBUG][c.f.s.a.BackendRegistry ] User ‘User [name=kibanaserver, roles=, requestedTenant=null]’ is authenticated
[2018-04-10T00:59:40,880][DEBUG][c.f.s.a.BackendRegistry ] sgtenant ‘null’

Found the issue. Configuration…

was: #searchguard.authcz.impersonation_dn:

now: #searchguard.authcz.rest_impersonation_user: