ES authentication fails in kibana

Hi,

I have a kibana installed with searchguard in kubernetes, and a 3 node ES cluster on VMs

I am using the official kibana-oss docker image. There is an kubernetes nginx ingress in front of kibana interface

  • ES cluster works perfectly.

  • Application with the defined usernam/password for them working nicely

  • Curl with any user including kibanaserver works for the role defined things

  • On kibana interface i can login with the username/password of any user defined with sgadmin on the ES cluster

My problem is that i get the following errors

Error from kibana log:

{“type”:“log”,“@timestamp”:“2018-04-18T20:11:53Z”,“tags”:[“status”,“plugin:elasticsearch@6.2.2”,“error”],“pid”:8,“state”:“red”,“message”:“Status changed from yellow to red - Authentication Exception”,“prevState”:“yellow”,“prevMsg”:“Waiting for Elasticsearch”}

Error from ES:

[2018-04-18T20:18:07,559][WARN ][c.f.s.a.BackendRegistry ] Authentication finally failed for null

[2018-04-18T20:18:10,541][WARN ][c.f.s.a.BackendRegistry ] Authentication finally failed for null

How can searchguard login work on the kibana interface if kibana itself can’t connect to the ES cluster where the users are defined?

Versions:

plugin:kibana@6.2.2
Ready
plugin:elasticsearch@6.2.2
Authentication Exception
plugin:timelion@6.2.2
Ready
plugin:searchguard@6.2.2
Search Guard plugin initialised.
plugin:console@6.2.2
Ready
plugin:metrics@6.2.2
Ready

SearchGuard config

sg_config

searchguard:

dynamic:

authc:

basic_internal_auth_domain:

http_enabled: true

transport_enabled: true

order: 0

http_authenticator:

type: basic

challenge: true

authentication_backend:

type: internal

clientcert_auth_domain:

transport_enabled: false

order: 1

http_authenticator:

type: clientcert

config:

username_attribute: cn

challenge: false

authentication_backend:

type: noop

sg_internal_user

kibanaserver:

readonly: true

hash: $2y$12$AoboyrVUI2A9y1yxOiyi5yx5Ni68eQLeo2BcSCNK16d6TboO

sg_roles

sg_kibana_server:

readonly: true

cluster:

  • CLUSTER_MONITOR

  • CLUSTER_COMPOSITE_OPS

  • cluster:admin/xpack/monitoring*

  • indices:admin/template*

indices:

‘?kibana’:

‘*’:

  • INDICES_ALL

‘?reporting*’:

‘*’:

  • INDICES_ALL

‘?monitoring*’:

‘*’:

  • INDICES_ALL

sg_roles_mapping

sg_kibana_server:

readonly: true

users:

  • kibanaserver

Kibana config:

server.host: “0.0.0.0”

server.name: “gh-kibana-stage.eur.xxx.com

searchguard.cookie.secure: true

searchguard.cookie.password: “xxxxxxxxxxxx”

logging.verbose: false

elasticsearch.ssl.verificationMode: none

elasticsearch.url: “https://eurwebstageghes01.eurweb.xxx.com:9200

elasticsearch.username: “kibanaserver”

elasticsearch.password: “xxxxxxxxxxx”

ES config:

network.bind_host: 0.0.0.0

searchguard.enterprise_modules_enabled: false

searchguard.ssl.transport.pemcert_filepath: certs/eurwebstageghes01.pem

searchguard.ssl.transport.pemkey_filepath: certs/eurwebstageghes01.key

searchguard.ssl.transport.pemkey_password: xxxxx

searchguard.ssl.transport.pemtrustedcas_filepath: certs/ca.pem

searchguard.ssl.transport.enforce_hostname_verification: true

searchguard.ssl.transport.resolve_hostname: true

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: certs/eurwebstageghes01.pem

searchguard.ssl.http.pemkey_filepath: certs/eurwebstageghes01.key

searchguard.ssl.http.pemkey_password: xxxxx

searchguard.ssl.http.pemtrustedcas_filepath: certs/ca.pem

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

  • CN=see-admin,OU=XXXX,O=XXX,DC=xxxx,DC=com

searchguard.roles_mapping_resolution: MAPPING_ONLY

searchguard.restapi.roles_enabled: [“sg_all_access”]

searchguard.audit.enable_rest: true

searchguard.audit.resolve_bulk_requests: true

searchguard.audit.type: internal_elasticsearch

plse see

https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/search-guard/ZQ_-SL1tQ9k/75chW1BCAAAJ

···

Am 18.04.2018 um 22:41 schrieb Peter Horvath <peter.horvath77@gmail.com>:

Hi,

I have a kibana installed with searchguard in kubernetes, and a 3 node ES cluster on VMs
I am using the official kibana-oss docker image. There is an kubernetes nginx ingress in front of kibana interface

- ES cluster works perfectly.
- Application with the defined usernam/password for them working nicely
- Curl with any user including kibanaserver works for the role defined things
- On kibana interface i can login with the username/password of any user defined with sgadmin on the ES cluster

My problem is that i get the following errors
Error from kibana log:
{"type":"log","@timestamp":"2018-04-18T20:11:53Z","tags":["status","plugin:elasticsearch@6.2.2","error"],"pid":8,"state":"red","message":"Status changed from yellow to red - Authentication Exception","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
Error from ES:
[2018-04-18T20:18:07,559][WARN ][c.f.s.a.BackendRegistry ] Authentication finally failed for null
[2018-04-18T20:18:10,541][WARN ][c.f.s.a.BackendRegistry ] Authentication finally failed for null

How can searchguard login work on the kibana interface if kibana itself can't connect to the ES cluster where the users are defined?

Versions:
plugin:kibana@6.2.2 Ready
plugin:elasticsearch@6.2.2 Authentication Exception
plugin:timelion@6.2.2 Ready
plugin:searchguard@6.2.2 Search Guard plugin initialised.
plugin:console@6.2.2 Ready
plugin:metrics@6.2.2 Ready

SearchGuard config

sg_config
searchguard:
  dynamic:
    authc:
      basic_internal_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: internal
      clientcert_auth_domain:
        transport_enabled: false
        order: 1
        http_authenticator:
          type: clientcert
          config:
            username_attribute: cn
          challenge: false
        authentication_backend:
          type: noop

sg_internal_user
kibanaserver:
  readonly: true
  hash: $2y$12$AoboyrVUI2A9y1yxOiyi5yx5Ni68eQLeo2BcSCNK16d6TboO

sg_roles
sg_kibana_server:
  readonly: true
  cluster:
      - CLUSTER_MONITOR
      - CLUSTER_COMPOSITE_OPS
      - cluster:admin/xpack/monitoring*
      - indices:admin/template*
  indices:
    '?kibana':
      '*':
        - INDICES_ALL
    '?reporting*':
      '*':
        - INDICES_ALL
    '?monitoring*':
      '*':
        - INDICES_ALL

sg_roles_mapping
sg_kibana_server:
  readonly: true
  users:
    - kibanaserver

Kibana config:
    server.host: "0.0.0.0"
    server.name: "gh-kibana-stage.eur.xxx.com"
    searchguard.cookie.secure: true
    searchguard.cookie.password: "xxxxxxxxxxxx"
    logging.verbose: false
    elasticsearch.ssl.verificationMode: none
    elasticsearch.url: "https://eurwebstageghes01.eurweb.xxx.com:9200"
    elasticsearch.username: "kibanaserver"
    elasticsearch.password: "xxxxxxxxxxx"

ES config:
network.bind_host: 0.0.0.0
searchguard.enterprise_modules_enabled: false
searchguard.ssl.transport.pemcert_filepath: certs/eurwebstageghes01.pem
searchguard.ssl.transport.pemkey_filepath: certs/eurwebstageghes01.key
searchguard.ssl.transport.pemkey_password: xxxxx
searchguard.ssl.transport.pemtrustedcas_filepath: certs/ca.pem
searchguard.ssl.transport.enforce_hostname_verification: true
searchguard.ssl.transport.resolve_hostname: true
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: certs/eurwebstageghes01.pem
searchguard.ssl.http.pemkey_filepath: certs/eurwebstageghes01.key
searchguard.ssl.http.pemkey_password: xxxxx
searchguard.ssl.http.pemtrustedcas_filepath: certs/ca.pem
searchguard.nodes_dn:
- CN=eurwebstageghes01.eurweb.xxxx.com,OU=XXX,O=XXX,DC=xxx,DC=com
- CN=eurwebstageghes02.eurweb.xxxx.com,OU=XXX,O=XXX,DC=xxx,DC=com
- CN=eurwebstageghes03.eurweb.xxx.com,OU=XXX,O=XXX,DC=xxx,DC=com
searchguard.authcz.admin_dn:
- CN=see-admin,OU=XXXX,O=XXX,DC=xxxx,DC=com
searchguard.roles_mapping_resolution: MAPPING_ONLY
searchguard.restapi.roles_enabled: ["sg_all_access"]
searchguard.audit.enable_rest: true
searchguard.audit.resolve_bulk_requests: true
searchguard.audit.type: internal_elasticsearch

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/06657b9c-6388-48a3-a3db-5dcd8109c368%40googlegroups.com\.
For more options, visit https://groups.google.com/d/optout\.

Hi,

I’ve read that thread but it is nothing like my case.

If you check my full config you can see that searchguard login on kibana works only with the defined users not any random and i don’t have certificates setup on kibana side.

My problem is that the ES cluster plugin can’t connect

Peter

···

On 19 April 2018 at 11:53, SG info@search-guard.com wrote:

plse see

https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/search-guard/ZQ_-SL1tQ9k/75chW1BCAAAJ

Am 18.04.2018 um 22:41 schrieb Peter Horvath peter.horvath77@gmail.com:

Hi,

I have a kibana installed with searchguard in kubernetes, and a 3 node ES cluster on VMs

I am using the official kibana-oss docker image. There is an kubernetes nginx ingress in front of kibana interface

  • ES cluster works perfectly.
  • Application with the defined usernam/password for them working nicely
  • Curl with any user including kibanaserver works for the role defined things
  • On kibana interface i can login with the username/password of any user defined with sgadmin on the ES cluster

My problem is that i get the following errors

Error from kibana log:

{“type”:“log”,“@timestamp”:“2018-04-18T20:11:53Z”,“tags”:[“status”,“plugin:elasticsearch@6.2.2”,“error”],“pid”:8,“state”:“red”,“message”:“Status changed from yellow to red - Authentication Exception”,“prevState”:“yellow”,“prevMsg”:“Waiting for Elasticsearch”}

Error from ES:

[2018-04-18T20:18:07,559][WARN ][c.f.s.a.BackendRegistry ] Authentication finally failed for null

[2018-04-18T20:18:10,541][WARN ][c.f.s.a.BackendRegistry ] Authentication finally failed for null

How can searchguard login work on the kibana interface if kibana itself can’t connect to the ES cluster where the users are defined?

Versions:

plugin:kibana@6.2.2 Ready

plugin:elasticsearch@6.2.2 Authentication Exception

plugin:timelion@6.2.2 Ready

plugin:searchguard@6.2.2 Search Guard plugin initialised.

plugin:console@6.2.2 Ready

plugin:metrics@6.2.2 Ready

SearchGuard config

sg_config

searchguard:

dynamic:

authc:
  basic_internal_auth_domain:
    http_enabled: true
    transport_enabled: true
    order: 0
    http_authenticator:
      type: basic
      challenge: true
    authentication_backend:
      type: internal
  clientcert_auth_domain:
    transport_enabled: false
    order: 1
    http_authenticator:
      type: clientcert
      config:
        username_attribute: cn
      challenge: false
    authentication_backend:
      type: noop

sg_internal_user

kibanaserver:

readonly: true

hash: $2y$12$AoboyrVUI2A9y1yxOiyi5yx5Ni68eQLeo2BcSCNK16d6TboO

sg_roles

sg_kibana_server:

readonly: true

cluster:

  - CLUSTER_MONITOR
  - CLUSTER_COMPOSITE_OPS
  - cluster:admin/xpack/monitoring*
  - indices:admin/template*

indices:

'?kibana':
  '*':
    - INDICES_ALL
'?reporting*':
  '*':
    - INDICES_ALL
'?monitoring*':
  '*':
    - INDICES_ALL

sg_roles_mapping

sg_kibana_server:

readonly: true

users:

- kibanaserver

Kibana config:

server.host: "0.0.0.0"
[server.name](http://server.name): "[gh-kibana-stage.eur.xxx.com](http://gh-kibana-stage.eur.xxx.com)"
searchguard.cookie.secure: true
searchguard.cookie.password: "xxxxxxxxxxxx"
logging.verbose: false
elasticsearch.ssl.verificationMode: none
elasticsearch.url: "[https://eurwebstageghes01.eurweb.xxx.com:9200](https://eurwebstageghes01.eurweb.xxx.com:9200)"
elasticsearch.username: "kibanaserver"
elasticsearch.password: "xxxxxxxxxxx"

ES config:

network.bind_host: 0.0.0.0

searchguard.enterprise_modules_enabled: false

searchguard.ssl.transport.pemcert_filepath: certs/eurwebstageghes01.pem

searchguard.ssl.transport.pemkey_filepath: certs/eurwebstageghes01.key

searchguard.ssl.transport.pemkey_password: xxxxx

searchguard.ssl.transport.pemtrustedcas_filepath: certs/ca.pem

searchguard.ssl.transport.enforce_hostname_verification: true

searchguard.ssl.transport.resolve_hostname: true

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: certs/eurwebstageghes01.pem

searchguard.ssl.http.pemkey_filepath: certs/eurwebstageghes01.key

searchguard.ssl.http.pemkey_password: xxxxx

searchguard.ssl.http.pemtrustedcas_filepath: certs/ca.pem

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

  • CN=see-admin,OU=XXXX,O=XXX,DC=xxxx,DC=com

searchguard.roles_mapping_resolution: MAPPING_ONLY

searchguard.restapi.roles_enabled: [“sg_all_access”]

searchguard.audit.enable_rest: true

searchguard.audit.resolve_bulk_requests: true

searchguard.audit.type: internal_elasticsearch

You received this message because you are subscribed to the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/06657b9c-6388-48a3-a3db-5dcd8109c368%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/4E6A4BF2-0DDC-4F8C-9334-F17A161EB4AB%40search-guard.com.
For more options, visit https://groups.google.com/d/optout.

Sorry for the wrong link

- remove the clientcert_auth_domain from sg_config.yml if you dont need them, reload config via sgadmin
- make sure that the password for kibanaserver is correct.
- enable debug logs like described here TLS help | Security for Elasticsearch | Search Guard and post the logs
- please also post the complete kibana.yml as file attachment

···

Am 19.04.2018 um 19:10 schrieb Peter Horvath <peter.horvath77@gmail.com>:

Hi,

I've read that thread but it is nothing like my case.
If you check my full config you can see that searchguard login on kibana works only with the defined users not any random and i don't have certificates setup on kibana side.
My problem is that the ES cluster plugin can't connect

Peter

On 19 April 2018 at 11:53, SG <info@search-guard.com> wrote:
plse see

Redirecting to Google Groups

> Am 18.04.2018 um 22:41 schrieb Peter Horvath <peter.horvath77@gmail.com>:
>
> Hi,
>
> I have a kibana installed with searchguard in kubernetes, and a 3 node ES cluster on VMs
> I am using the official kibana-oss docker image. There is an kubernetes nginx ingress in front of kibana interface
>
> - ES cluster works perfectly.
> - Application with the defined usernam/password for them working nicely
> - Curl with any user including kibanaserver works for the role defined things
> - On kibana interface i can login with the username/password of any user defined with sgadmin on the ES cluster
>
> My problem is that i get the following errors
> Error from kibana log:
> {"type":"log","@timestamp":"2018-04-18T20:11:53Z","tags":["status","plugin:elasticsearch@6.2.2","error"],"pid":8,"state":"red","message":"Status changed from yellow to red - Authentication Exception","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
> Error from ES:
> [2018-04-18T20:18:07,559][WARN ][c.f.s.a.BackendRegistry ] Authentication finally failed for null
> [2018-04-18T20:18:10,541][WARN ][c.f.s.a.BackendRegistry ] Authentication finally failed for null
>
> How can searchguard login work on the kibana interface if kibana itself can't connect to the ES cluster where the users are defined?
>
> Versions:
> plugin:kibana@6.2.2 Ready
> plugin:elasticsearch@6.2.2 Authentication Exception
> plugin:timelion@6.2.2 Ready
> plugin:searchguard@6.2.2 Search Guard plugin initialised.
> plugin:console@6.2.2 Ready
> plugin:metrics@6.2.2 Ready
>
>
> SearchGuard config
>
> sg_config
> searchguard:
> dynamic:
> authc:
> basic_internal_auth_domain:
> http_enabled: true
> transport_enabled: true
> order: 0
> http_authenticator:
> type: basic
> challenge: true
> authentication_backend:
> type: internal
> clientcert_auth_domain:
> transport_enabled: false
> order: 1
> http_authenticator:
> type: clientcert
> config:
> username_attribute: cn
> challenge: false
> authentication_backend:
> type: noop
>
> sg_internal_user
> kibanaserver:
> readonly: true
> hash: $2y$12$AoboyrVUI2A9y1yxOiyi5yx5Ni68eQLeo2BcSCNK16d6TboO
>
> sg_roles
> sg_kibana_server:
> readonly: true
> cluster:
> - CLUSTER_MONITOR
> - CLUSTER_COMPOSITE_OPS
> - cluster:admin/xpack/monitoring*
> - indices:admin/template*
> indices:
> '?kibana':
> '*':
> - INDICES_ALL
> '?reporting*':
> '*':
> - INDICES_ALL
> '?monitoring*':
> '*':
> - INDICES_ALL
>
> sg_roles_mapping
> sg_kibana_server:
> readonly: true
> users:
> - kibanaserver
>
> Kibana config:
> server.host: "0.0.0.0"
> server.name: "gh-kibana-stage.eur.xxx.com"
> searchguard.cookie.secure: true
> searchguard.cookie.password: "xxxxxxxxxxxx"
> logging.verbose: false
> elasticsearch.ssl.verificationMode: none
> elasticsearch.url: "https://eurwebstageghes01.eurweb.xxx.com:9200"
> elasticsearch.username: "kibanaserver"
> elasticsearch.password: "xxxxxxxxxxx"
>
> ES config:
> network.bind_host: 0.0.0.0
> searchguard.enterprise_modules_enabled: false
> searchguard.ssl.transport.pemcert_filepath: certs/eurwebstageghes01.pem
> searchguard.ssl.transport.pemkey_filepath: certs/eurwebstageghes01.key
> searchguard.ssl.transport.pemkey_password: xxxxx
> searchguard.ssl.transport.pemtrustedcas_filepath: certs/ca.pem
> searchguard.ssl.transport.enforce_hostname_verification: true
> searchguard.ssl.transport.resolve_hostname: true
> searchguard.ssl.http.enabled: true
> searchguard.ssl.http.pemcert_filepath: certs/eurwebstageghes01.pem
> searchguard.ssl.http.pemkey_filepath: certs/eurwebstageghes01.key
> searchguard.ssl.http.pemkey_password: xxxxx
> searchguard.ssl.http.pemtrustedcas_filepath: certs/ca.pem
> searchguard.nodes_dn:
> - CN=eurwebstageghes01.eurweb.xxxx.com,OU=XXX,O=XXX,DC=xxx,DC=com
> - CN=eurwebstageghes02.eurweb.xxxx.com,OU=XXX,O=XXX,DC=xxx,DC=com
> - CN=eurwebstageghes03.eurweb.xxx.com,OU=XXX,O=XXX,DC=xxx,DC=com
> searchguard.authcz.admin_dn:
> - CN=see-admin,OU=XXXX,O=XXX,DC=xxxx,DC=com
> searchguard.roles_mapping_resolution: MAPPING_ONLY
> searchguard.restapi.roles_enabled: ["sg_all_access"]
> searchguard.audit.enable_rest: true
> searchguard.audit.resolve_bulk_requests: true
> searchguard.audit.type: internal_elasticsearch
>
>
> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
> To post to this group, send email to search-guard@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/06657b9c-6388-48a3-a3db-5dcd8109c368%40googlegroups.com\.
> For more options, visit https://groups.google.com/d/optout\.

--
You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe\.
To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/4E6A4BF2-0DDC-4F8C-9334-F17A161EB4AB%40search-guard.com\.
For more options, visit https://groups.google.com/d/optout\.

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/CAO9xhubo7pUph74OjxrX-KW2F-gJhHEtnKjWseLvfu_9un8QZw%40mail.gmail.com\.
For more options, visit https://groups.google.com/d/optout\.

Thanks! the debug mode was very useful.
My kibana config was that much i posted by the way.

I’ve tried it with " " encapsulation and without both yields the same

Kibana config:

server.host: 0.0.0.0

server.name: gh-kibana-stage.eur.xxx.com

searchguard.cookie.secure: true

searchguard.cookie.password: xxxxxxxxxxxx

logging.verbose: false

elasticsearch.ssl.verificationMode: none

elasticsearch.url: https://eurwebstageghes01.eurweb.xxx.com:9200

elasticsearch.username: kibanaserver

elasticsearch.password: xxxxxxxxxxx

So if i cat the kibana config on the kibana server and i copy paste the kibana username and password and use curl i can check health of the cluster.

elasticsearch_1 | [2018-04-19T20:06:48,971][DEBUG][c.f.s.a.BackendRegistry ] User ‘User [name=kibanaserver, roles=[kibanaserver], requestedTenant=null]’ is authenticated

elasticsearch_1 | [2018-04-19T20:06:48,971][DEBUG][c.f.s.a.BackendRegistry ] sgtenant ‘null’

elasticsearch_1 | [2018-04-19T20:06:48,971][DEBUG][c.f.s.c.PrivilegesEvaluator] ### evaluate permissions for User [name=kibanaserver, roles=[kibanaserver], requestedTenant=null] on S6VgCOD

elasticsearch_1 | [2018-04-19T20:06:48,971][DEBUG][c.f.s.c.PrivilegesEvaluator] requested cluster:monitor/health from 10.66.3.25:34102

elasticsearch_1 | [2018-04-19T20:06:48,971][DEBUG][c.f.s.c.PrivilegesEvaluator] Resolve from class org.elasticsearch.action.admin.cluster.health.ClusterHealthRequest for action cluster:monitor/health

elasticsearch_1 | [2018-04-19T20:06:48,972][DEBUG][c.f.s.c.PrivilegesEvaluator] found a match for ‘sg_kibana_server’ and cluster:monitor/health, skip other roles

But the kibana server still ends up triggering a password missmatch:

elasticsearch_1 | [2018-04-19T20:06:49,733][DEBUG][c.f.s.a.BackendRegistry ] Try to extract auth creds from basic http authenticator

elasticsearch_1 | [2018-04-19T20:06:49,733][DEBUG][c.f.s.a.BackendRegistry ] kibanaserver not cached, return from internal backend directly

elasticsearch_1 | [2018-04-19T20:06:50,207][DEBUG][c.f.s.a.BackendRegistry ] Can not authenticate kibanaserver due to com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]

elasticsearch_1 | com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]

elasticsearch_1 | at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2218) ~[guava-23.0.jar:?]

elasticsearch_1 | at com.google.common.cache.LocalCache.get(LocalCache.java:4147) ~[guava-23.0.jar:?]

Any idea?

···

On 19 April 2018 at 13:34, SG info@search-guard.com wrote:

Sorry for the wrong link

Am 19.04.2018 um 19:10 schrieb Peter Horvath peter.horvath77@gmail.com:

Hi,

I’ve read that thread but it is nothing like my case.

If you check my full config you can see that searchguard login on kibana works only with the defined users not any random and i don’t have certificates setup on kibana side.

My problem is that the ES cluster plugin can’t connect

Peter

On 19 April 2018 at 11:53, SG info@search-guard.com wrote:

plse see

https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/search-guard/ZQ_-SL1tQ9k/75chW1BCAAAJ

Am 18.04.2018 um 22:41 schrieb Peter Horvath peter.horvath77@gmail.com:

Hi,

I have a kibana installed with searchguard in kubernetes, and a 3 node ES cluster on VMs

I am using the official kibana-oss docker image. There is an kubernetes nginx ingress in front of kibana interface

  • ES cluster works perfectly.
  • Application with the defined usernam/password for them working nicely
  • Curl with any user including kibanaserver works for the role defined things
  • On kibana interface i can login with the username/password of any user defined with sgadmin on the ES cluster

My problem is that i get the following errors

Error from kibana log:

{“type”:“log”,“@timestamp”:“2018-04-18T20:11:53Z”,“tags”:[“status”,“plugin:elasticsearch@6.2.2”,“error”],“pid”:8,“state”:“red”,“message”:“Status changed from yellow to red - Authentication Exception”,“prevState”:“yellow”,“prevMsg”:“Waiting for Elasticsearch”}

Error from ES:

[2018-04-18T20:18:07,559][WARN ][c.f.s.a.BackendRegistry ] Authentication finally failed for null

[2018-04-18T20:18:10,541][WARN ][c.f.s.a.BackendRegistry ] Authentication finally failed for null

How can searchguard login work on the kibana interface if kibana itself can’t connect to the ES cluster where the users are defined?

Versions:

plugin:kibana@6.2.2 Ready

plugin:elasticsearch@6.2.2 Authentication Exception

plugin:timelion@6.2.2 Ready

plugin:searchguard@6.2.2 Search Guard plugin initialised.

plugin:console@6.2.2 Ready

plugin:metrics@6.2.2 Ready

SearchGuard config

sg_config

searchguard:

dynamic:

authc:
  basic_internal_auth_domain:
    http_enabled: true
    transport_enabled: true
    order: 0
    http_authenticator:
      type: basic
      challenge: true
    authentication_backend:
      type: internal
  clientcert_auth_domain:
    transport_enabled: false
    order: 1
    http_authenticator:
      type: clientcert
      config:
        username_attribute: cn
      challenge: false
    authentication_backend:
      type: noop

sg_internal_user

kibanaserver:

readonly: true

hash: $2y$12$AoboyrVUI2A9y1yxOiyi5yx5Ni68eQLeo2BcSCNK16d6TboO

sg_roles

sg_kibana_server:

readonly: true

cluster:

  - CLUSTER_MONITOR
  - CLUSTER_COMPOSITE_OPS
  - cluster:admin/xpack/monitoring*
  - indices:admin/template*

indices:

'?kibana':
  '*':
    - INDICES_ALL
'?reporting*':
  '*':
    - INDICES_ALL
'?monitoring*':
  '*':
    - INDICES_ALL

sg_roles_mapping

sg_kibana_server:

readonly: true

users:

- kibanaserver

Kibana config:

server.host: "0.0.0.0"
[server.name](http://server.name): "[gh-kibana-stage.eur.xxx.com](http://gh-kibana-stage.eur.xxx.com)"
searchguard.cookie.secure: true
searchguard.cookie.password: "xxxxxxxxxxxx"
logging.verbose: false
elasticsearch.ssl.verificationMode: none
elasticsearch.url: "[https://eurwebstageghes01.eurweb.xxx.com:9200](https://eurwebstageghes01.eurweb.xxx.com:9200)"
elasticsearch.username: "kibanaserver"
elasticsearch.password: "xxxxxxxxxxx"

ES config:

network.bind_host: 0.0.0.0

searchguard.enterprise_modules_enabled: false

searchguard.ssl.transport.pemcert_filepath: certs/eurwebstageghes01.pem

searchguard.ssl.transport.pemkey_filepath: certs/eurwebstageghes01.key

searchguard.ssl.transport.pemkey_password: xxxxx

searchguard.ssl.transport.pemtrustedcas_filepath: certs/ca.pem

searchguard.ssl.transport.enforce_hostname_verification: true

searchguard.ssl.transport.resolve_hostname: true

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: certs/eurwebstageghes01.pem

searchguard.ssl.http.pemkey_filepath: certs/eurwebstageghes01.key

searchguard.ssl.http.pemkey_password: xxxxx

searchguard.ssl.http.pemtrustedcas_filepath: certs/ca.pem

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

  • CN=see-admin,OU=XXXX,O=XXX,DC=xxxx,DC=com

searchguard.roles_mapping_resolution: MAPPING_ONLY

searchguard.restapi.roles_enabled: [“sg_all_access”]

searchguard.audit.enable_rest: true

searchguard.audit.resolve_bulk_requests: true

searchguard.audit.type: internal_elasticsearch

You received this message because you are subscribed to the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/06657b9c-6388-48a3-a3db-5dcd8109c368%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/4E6A4BF2-0DDC-4F8C-9334-F17A161EB4AB%40search-guard.com.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/CAO9xhubo7pUph74OjxrX-KW2F-gJhHEtnKjWseLvfu_9un8QZw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/45CB4889-26F1-4755-BB3D-59665CCD2FC0%40search-guard.com.
For more options, visit https://groups.google.com/d/optout.

I’ve also tried to upgrade the kibana stack to 6.2.3 and changed the password to short simple one in the hope that it might be too long or too complex.
None of them helped curl works kibana es plugin doesn’t.

···

On Thursday, 19 April 2018 16:15:01 UTC-4, Peter Horvath wrote:

Thanks! the debug mode was very useful.
My kibana config was that much i posted by the way.

I’ve tried it with " " encapsulation and without both yields the same

Kibana config:

server.host: 0.0.0.0

server.name: gh-kibana-stage.eur.xxx.com

searchguard.cookie.secure: true

searchguard.cookie.password: xxxxxxxxxxxx

logging.verbose: false

elasticsearch.ssl.verificationMode: none

elasticsearch.url: https://eurwebstageghes01.eurweb.xxx.com:9200

elasticsearch.username: kibanaserver

elasticsearch.password: xxxxxxxxxxx

So if i cat the kibana config on the kibana server and i copy paste the kibana username and password and use curl i can check health of the cluster.

elasticsearch_1 | [2018-04-19T20:06:48,971][DEBUG][c.f.s.a.BackendRegistry ] User ‘User [name=kibanaserver, roles=[kibanaserver], requestedTenant=null]’ is authenticated

elasticsearch_1 | [2018-04-19T20:06:48,971][DEBUG][c.f.s.a.BackendRegistry ] sgtenant ‘null’

elasticsearch_1 | [2018-04-19T20:06:48,971][DEBUG][c.f.s.c.PrivilegesEvaluator] ### evaluate permissions for User [name=kibanaserver, roles=[kibanaserver], requestedTenant=null] on S6VgCOD

elasticsearch_1 | [2018-04-19T20:06:48,971][DEBUG][c.f.s.c.PrivilegesEvaluator] requested cluster:monitor/health from 10.66.3.25:34102

elasticsearch_1 | [2018-04-19T20:06:48,971][DEBUG][c.f.s.c.PrivilegesEvaluator] Resolve from class org.elasticsearch.action.admin.cluster.health.ClusterHealthRequest for action cluster:monitor/health

elasticsearch_1 | [2018-04-19T20:06:48,972][DEBUG][c.f.s.c.PrivilegesEvaluator] found a match for ‘sg_kibana_server’ and cluster:monitor/health, skip other roles

But the kibana server still ends up triggering a password missmatch:

elasticsearch_1 | [2018-04-19T20:06:49,733][DEBUG][c.f.s.a.BackendRegistry ] Try to extract auth creds from basic http authenticator

elasticsearch_1 | [2018-04-19T20:06:49,733][DEBUG][c.f.s.a.BackendRegistry ] kibanaserver not cached, return from internal backend directly

elasticsearch_1 | [2018-04-19T20:06:50,207][DEBUG][c.f.s.a.BackendRegistry ] Can not authenticate kibanaserver due to com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]

elasticsearch_1 | com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]

elasticsearch_1 | at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2218) ~[guava-23.0.jar:?]

elasticsearch_1 | at com.google.common.cache.LocalCache.get(LocalCache.java:4147) ~[guava-23.0.jar:?]

Any idea?

On 19 April 2018 at 13:34, SG info@search-guard.com wrote:

Sorry for the wrong link

Am 19.04.2018 um 19:10 schrieb Peter Horvath peter.horvath77@gmail.com:

Hi,

I’ve read that thread but it is nothing like my case.

If you check my full config you can see that searchguard login on kibana works only with the defined users not any random and i don’t have certificates setup on kibana side.

My problem is that the ES cluster plugin can’t connect

Peter

On 19 April 2018 at 11:53, SG info@search-guard.com wrote:

plse see

https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/search-guard/ZQ_-SL1tQ9k/75chW1BCAAAJ

Am 18.04.2018 um 22:41 schrieb Peter Horvath peter.horvath77@gmail.com:

Hi,

I have a kibana installed with searchguard in kubernetes, and a 3 node ES cluster on VMs

I am using the official kibana-oss docker image. There is an kubernetes nginx ingress in front of kibana interface

  • ES cluster works perfectly.
  • Application with the defined usernam/password for them working nicely
  • Curl with any user including kibanaserver works for the role defined things
  • On kibana interface i can login with the username/password of any user defined with sgadmin on the ES cluster

My problem is that i get the following errors

Error from kibana log:

{“type”:“log”,“@timestamp”:“2018-04-18T20:11:53Z”,“tags”:[“status”,“plugin:elasticsearch@6.2.2”,“error”],“pid”:8,“state”:“red”,“message”:“Status changed from yellow to red - Authentication Exception”,“prevState”:“yellow”,“prevMsg”:“Waiting for Elasticsearch”}

Error from ES:

[2018-04-18T20:18:07,559][WARN ][c.f.s.a.BackendRegistry ] Authentication finally failed for null

[2018-04-18T20:18:10,541][WARN ][c.f.s.a.BackendRegistry ] Authentication finally failed for null

How can searchguard login work on the kibana interface if kibana itself can’t connect to the ES cluster where the users are defined?

Versions:

plugin:kibana@6.2.2 Ready

plugin:elasticsearch@6.2.2 Authentication Exception

plugin:timelion@6.2.2 Ready

plugin:searchguard@6.2.2 Search Guard plugin initialised.

plugin:console@6.2.2 Ready

plugin:metrics@6.2.2 Ready

SearchGuard config

sg_config

searchguard:

dynamic:

authc:
  basic_internal_auth_domain:
    http_enabled: true
    transport_enabled: true
    order: 0
    http_authenticator:
      type: basic
      challenge: true
    authentication_backend:
      type: internal
  clientcert_auth_domain:
    transport_enabled: false
    order: 1
    http_authenticator:
      type: clientcert
      config:
        username_attribute: cn
      challenge: false
    authentication_backend:
      type: noop

sg_internal_user

kibanaserver:

readonly: true

hash: $2y$12$AoboyrVUI2A9y1yxOiyi5yx5Ni68eQLeo2BcSCNK16d6TboO

sg_roles

sg_kibana_server:

readonly: true

cluster:

  - CLUSTER_MONITOR
  - CLUSTER_COMPOSITE_OPS
  - cluster:admin/xpack/monitoring*
  - indices:admin/template*

indices:

'?kibana':
  '*':
    - INDICES_ALL
'?reporting*':
  '*':
    - INDICES_ALL
'?monitoring*':
  '*':
    - INDICES_ALL

sg_roles_mapping

sg_kibana_server:

readonly: true

users:

- kibanaserver

Kibana config:

server.host: "0.0.0.0"
[server.name](http://server.name): "[gh-kibana-stage.eur.xxx.com](http://gh-kibana-stage.eur.xxx.com)"
searchguard.cookie.secure: true
searchguard.cookie.password: "xxxxxxxxxxxx"
logging.verbose: false
elasticsearch.ssl.verificationMode: none
elasticsearch.url: "[https://eurwebstageghes01.eurweb.xxx.com:9200](https://eurwebstageghes01.eurweb.xxx.com:9200)"
elasticsearch.username: "kibanaserver"
elasticsearch.password: "xxxxxxxxxxx"

ES config:

network.bind_host: 0.0.0.0

searchguard.enterprise_modules_enabled: false

searchguard.ssl.transport.pemcert_filepath: certs/eurwebstageghes01.pem

searchguard.ssl.transport.pemkey_filepath: certs/eurwebstageghes01.key

searchguard.ssl.transport.pemkey_password: xxxxx

searchguard.ssl.transport.pemtrustedcas_filepath: certs/ca.pem

searchguard.ssl.transport.enforce_hostname_verification: true

searchguard.ssl.transport.resolve_hostname: true

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: certs/eurwebstageghes01.pem

searchguard.ssl.http.pemkey_filepath: certs/eurwebstageghes01.key

searchguard.ssl.http.pemkey_password: xxxxx

searchguard.ssl.http.pemtrustedcas_filepath: certs/ca.pem

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

  • CN=see-admin,OU=XXXX,O=XXX,DC=xxxx,DC=com

searchguard.roles_mapping_resolution: MAPPING_ONLY

searchguard.restapi.roles_enabled: [“sg_all_access”]

searchguard.audit.enable_rest: true

searchguard.audit.resolve_bulk_requests: true

searchguard.audit.type: internal_elasticsearch

You received this message because you are subscribed to the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/06657b9c-6388-48a3-a3db-5dcd8109c368%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/4E6A4BF2-0DDC-4F8C-9334-F17A161EB4AB%40search-guard.com.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/CAO9xhubo7pUph74OjxrX-KW2F-gJhHEtnKjWseLvfu_9un8QZw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/45CB4889-26F1-4755-BB3D-59665CCD2FC0%40search-guard.com.
For more options, visit https://groups.google.com/d/optout.

This seems pretty strange.

From your first post I see that the kibanaserver user cannot connect to ES/SG:

plugin:elasticsearch@6.2.2
Authentication Exception

Which is also resembled in this error here:

elasticsearch_1 | [2018-04-19T20:06:50,207][DEBUG][c.f.s.a.BackendRegistry ] Can not authenticate kibanaserver due to com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]

elasticsearch_1 | com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]

So one might think this is just a password error. If the password works with curl, the only possibility that comes to my mind would be that the pwd is somehow trashed when it is transmitted from Kibana to ES/SG. So one thing you can do is to use the developer tools and having a look at the HTTPS calls on login. You should see one POST to:

/api/v1/auth/login

Can you check that the password used here is 1:1 the password you also use in your curl call?

···

On Thursday, April 19, 2018 at 2:46:52 PM UTC-7, Peter Horvath wrote:

I’ve also tried to upgrade the kibana stack to 6.2.3 and changed the password to short simple one in the hope that it might be too long or too complex.
None of them helped curl works kibana es plugin doesn’t.

On Thursday, 19 April 2018 16:15:01 UTC-4, Peter Horvath wrote:

Thanks! the debug mode was very useful.
My kibana config was that much i posted by the way.

I’ve tried it with " " encapsulation and without both yields the same

Kibana config:

server.host: 0.0.0.0

server.name: gh-kibana-stage.eur.xxx.com

searchguard.cookie.secure: true

searchguard.cookie.password: xxxxxxxxxxxx

logging.verbose: false

elasticsearch.ssl.verificationMode: none

elasticsearch.url: https://eurwebstageghes01.eurweb.xxx.com:9200

elasticsearch.username: kibanaserver

elasticsearch.password: xxxxxxxxxxx

So if i cat the kibana config on the kibana server and i copy paste the kibana username and password and use curl i can check health of the cluster.

elasticsearch_1 | [2018-04-19T20:06:48,971][DEBUG][c.f.s.a.BackendRegistry ] User ‘User [name=kibanaserver, roles=[kibanaserver], requestedTenant=null]’ is authenticated

elasticsearch_1 | [2018-04-19T20:06:48,971][DEBUG][c.f.s.a.BackendRegistry ] sgtenant ‘null’

elasticsearch_1 | [2018-04-19T20:06:48,971][DEBUG][c.f.s.c.PrivilegesEvaluator] ### evaluate permissions for User [name=kibanaserver, roles=[kibanaserver], requestedTenant=null] on S6VgCOD

elasticsearch_1 | [2018-04-19T20:06:48,971][DEBUG][c.f.s.c.PrivilegesEvaluator] requested cluster:monitor/health from 10.66.3.25:34102

elasticsearch_1 | [2018-04-19T20:06:48,971][DEBUG][c.f.s.c.PrivilegesEvaluator] Resolve from class org.elasticsearch.action.admin.cluster.health.ClusterHealthRequest for action cluster:monitor/health

elasticsearch_1 | [2018-04-19T20:06:48,972][DEBUG][c.f.s.c.PrivilegesEvaluator] found a match for ‘sg_kibana_server’ and cluster:monitor/health, skip other roles

But the kibana server still ends up triggering a password missmatch:

elasticsearch_1 | [2018-04-19T20:06:49,733][DEBUG][c.f.s.a.BackendRegistry ] Try to extract auth creds from basic http authenticator

elasticsearch_1 | [2018-04-19T20:06:49,733][DEBUG][c.f.s.a.BackendRegistry ] kibanaserver not cached, return from internal backend directly

elasticsearch_1 | [2018-04-19T20:06:50,207][DEBUG][c.f.s.a.BackendRegistry ] Can not authenticate kibanaserver due to com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]

elasticsearch_1 | com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]

elasticsearch_1 | at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2218) ~[guava-23.0.jar:?]

elasticsearch_1 | at com.google.common.cache.LocalCache.get(LocalCache.java:4147) ~[guava-23.0.jar:?]

Any idea?

On 19 April 2018 at 13:34, SG info@search-guard.com wrote:

Sorry for the wrong link

Am 19.04.2018 um 19:10 schrieb Peter Horvath peter.horvath77@gmail.com:

Hi,

I’ve read that thread but it is nothing like my case.

If you check my full config you can see that searchguard login on kibana works only with the defined users not any random and i don’t have certificates setup on kibana side.

My problem is that the ES cluster plugin can’t connect

Peter

On 19 April 2018 at 11:53, SG info@search-guard.com wrote:

plse see

https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/search-guard/ZQ_-SL1tQ9k/75chW1BCAAAJ

Am 18.04.2018 um 22:41 schrieb Peter Horvath peter.horvath77@gmail.com:

Hi,

I have a kibana installed with searchguard in kubernetes, and a 3 node ES cluster on VMs

I am using the official kibana-oss docker image. There is an kubernetes nginx ingress in front of kibana interface

  • ES cluster works perfectly.
  • Application with the defined usernam/password for them working nicely
  • Curl with any user including kibanaserver works for the role defined things
  • On kibana interface i can login with the username/password of any user defined with sgadmin on the ES cluster

My problem is that i get the following errors

Error from kibana log:

{“type”:“log”,“@timestamp”:“2018-04-18T20:11:53Z”,“tags”:[“status”,“plugin:elasticsearch@6.2.2”,“error”],“pid”:8,“state”:“red”,“message”:“Status changed from yellow to red - Authentication Exception”,“prevState”:“yellow”,“prevMsg”:“Waiting for Elasticsearch”}

Error from ES:

[2018-04-18T20:18:07,559][WARN ][c.f.s.a.BackendRegistry ] Authentication finally failed for null

[2018-04-18T20:18:10,541][WARN ][c.f.s.a.BackendRegistry ] Authentication finally failed for null

How can searchguard login work on the kibana interface if kibana itself can’t connect to the ES cluster where the users are defined?

Versions:

plugin:kibana@6.2.2 Ready

plugin:elasticsearch@6.2.2 Authentication Exception

plugin:timelion@6.2.2 Ready

plugin:searchguard@6.2.2 Search Guard plugin initialised.

plugin:console@6.2.2 Ready

plugin:metrics@6.2.2 Ready

SearchGuard config

sg_config

searchguard:

dynamic:

authc:
  basic_internal_auth_domain:
    http_enabled: true
    transport_enabled: true
    order: 0
    http_authenticator:
      type: basic
      challenge: true
    authentication_backend:
      type: internal
  clientcert_auth_domain:
    transport_enabled: false
    order: 1
    http_authenticator:
      type: clientcert
      config:
        username_attribute: cn
      challenge: false
    authentication_backend:
      type: noop

sg_internal_user

kibanaserver:

readonly: true

hash: $2y$12$AoboyrVUI2A9y1yxOiyi5yx5Ni68eQLeo2BcSCNK16d6TboO

sg_roles

sg_kibana_server:

readonly: true

cluster:

  - CLUSTER_MONITOR
  - CLUSTER_COMPOSITE_OPS
  - cluster:admin/xpack/monitoring*
  - indices:admin/template*

indices:

'?kibana':
  '*':
    - INDICES_ALL
'?reporting*':
  '*':
    - INDICES_ALL
'?monitoring*':
  '*':
    - INDICES_ALL

sg_roles_mapping

sg_kibana_server:

readonly: true

users:

- kibanaserver

Kibana config:

server.host: "0.0.0.0"
[server.name](http://server.name): "[gh-kibana-stage.eur.xxx.com](http://gh-kibana-stage.eur.xxx.com)"
searchguard.cookie.secure: true
searchguard.cookie.password: "xxxxxxxxxxxx"
logging.verbose: false
elasticsearch.ssl.verificationMode: none
elasticsearch.url: "[https://eurwebstageghes01.eurweb.xxx.com:9200](https://eurwebstageghes01.eurweb.xxx.com:9200)"
elasticsearch.username: "kibanaserver"
elasticsearch.password: "xxxxxxxxxxx"

ES config:

network.bind_host: 0.0.0.0

searchguard.enterprise_modules_enabled: false

searchguard.ssl.transport.pemcert_filepath: certs/eurwebstageghes01.pem

searchguard.ssl.transport.pemkey_filepath: certs/eurwebstageghes01.key

searchguard.ssl.transport.pemkey_password: xxxxx

searchguard.ssl.transport.pemtrustedcas_filepath: certs/ca.pem

searchguard.ssl.transport.enforce_hostname_verification: true

searchguard.ssl.transport.resolve_hostname: true

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: certs/eurwebstageghes01.pem

searchguard.ssl.http.pemkey_filepath: certs/eurwebstageghes01.key

searchguard.ssl.http.pemkey_password: xxxxx

searchguard.ssl.http.pemtrustedcas_filepath: certs/ca.pem

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

  • CN=see-admin,OU=XXXX,O=XXX,DC=xxxx,DC=com

searchguard.roles_mapping_resolution: MAPPING_ONLY

searchguard.restapi.roles_enabled: [“sg_all_access”]

searchguard.audit.enable_rest: true

searchguard.audit.resolve_bulk_requests: true

searchguard.audit.type: internal_elasticsearch

You received this message because you are subscribed to the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/06657b9c-6388-48a3-a3db-5dcd8109c368%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/4E6A4BF2-0DDC-4F8C-9334-F17A161EB4AB%40search-guard.com.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/CAO9xhubo7pUph74OjxrX-KW2F-gJhHEtnKjWseLvfu_9un8QZw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/45CB4889-26F1-4755-BB3D-59665CCD2FC0%40search-guard.com.
For more options, visit https://groups.google.com/d/optout.

I actually did the curl from the kibana host and 1:1 copy pasted the password from the kibana.yml and curl worked. I can as well login on the kibana gui with it. Kibana elasticsearch plugin is the only one failing.

···

On Thu, Apr 19, 2018, 21:43 Jochen Kressin jkressin@floragunn.com wrote:

This seems pretty strange.

From your first post I see that the kibanaserver user cannot connect to ES/SG:

plugin:elasticsearch@6.2.2
Authentication Exception

Which is also resembled in this error here:

elasticsearch_1 | [2018-04-19T20:06:50,207][DEBUG][c.f.s.a.BackendRegistry ] Can not authenticate kibanaserver due to com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]

elasticsearch_1 | com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]

So one might think this is just a password error. If the password works with curl, the only possibility that comes to my mind would be that the pwd is somehow trashed when it is transmitted from Kibana to ES/SG. So one thing you can do is to use the developer tools and having a look at the HTTPS calls on login. You should see one POST to:

/api/v1/auth/login

Can you check that the password used here is 1:1 the password you also use in your curl call?

On Thursday, April 19, 2018 at 2:46:52 PM UTC-7, Peter Horvath wrote:

I’ve also tried to upgrade the kibana stack to 6.2.3 and changed the password to short simple one in the hope that it might be too long or too complex.
None of them helped curl works kibana es plugin doesn’t.

On Thursday, 19 April 2018 16:15:01 UTC-4, Peter Horvath wrote:

Thanks! the debug mode was very useful.
My kibana config was that much i posted by the way.

I’ve tried it with " " encapsulation and without both yields the same

Kibana config:

server.host: 0.0.0.0

server.name: gh-kibana-stage.eur.xxx.com

searchguard.cookie.secure: true

searchguard.cookie.password: xxxxxxxxxxxx

logging.verbose: false

elasticsearch.ssl.verificationMode: none

elasticsearch.url: https://eurwebstageghes01.eurweb.xxx.com:9200

elasticsearch.username: kibanaserver

elasticsearch.password: xxxxxxxxxxx

So if i cat the kibana config on the kibana server and i copy paste the kibana username and password and use curl i can check health of the cluster.

elasticsearch_1 | [2018-04-19T20:06:48,971][DEBUG][c.f.s.a.BackendRegistry ] User ‘User [name=kibanaserver, roles=[kibanaserver], requestedTenant=null]’ is authenticated

elasticsearch_1 | [2018-04-19T20:06:48,971][DEBUG][c.f.s.a.BackendRegistry ] sgtenant ‘null’

elasticsearch_1 | [2018-04-19T20:06:48,971][DEBUG][c.f.s.c.PrivilegesEvaluator] ### evaluate permissions for User [name=kibanaserver, roles=[kibanaserver], requestedTenant=null] on S6VgCOD

elasticsearch_1 | [2018-04-19T20:06:48,971][DEBUG][c.f.s.c.PrivilegesEvaluator] requested cluster:monitor/health from 10.66.3.25:34102

elasticsearch_1 | [2018-04-19T20:06:48,971][DEBUG][c.f.s.c.PrivilegesEvaluator] Resolve from class org.elasticsearch.action.admin.cluster.health.ClusterHealthRequest for action cluster:monitor/health

elasticsearch_1 | [2018-04-19T20:06:48,972][DEBUG][c.f.s.c.PrivilegesEvaluator] found a match for ‘sg_kibana_server’ and cluster:monitor/health, skip other roles

But the kibana server still ends up triggering a password missmatch:

elasticsearch_1 | [2018-04-19T20:06:49,733][DEBUG][c.f.s.a.BackendRegistry ] Try to extract auth creds from basic http authenticator

elasticsearch_1 | [2018-04-19T20:06:49,733][DEBUG][c.f.s.a.BackendRegistry ] kibanaserver not cached, return from internal backend directly

elasticsearch_1 | [2018-04-19T20:06:50,207][DEBUG][c.f.s.a.BackendRegistry ] Can not authenticate kibanaserver due to com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]

elasticsearch_1 | com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]

elasticsearch_1 | at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2218) ~[guava-23.0.jar:?]

elasticsearch_1 | at com.google.common.cache.LocalCache.get(LocalCache.java:4147) ~[guava-23.0.jar:?]

Any idea?

On 19 April 2018 at 13:34, SG info@search-guard.com wrote:

Sorry for the wrong link

Am 19.04.2018 um 19:10 schrieb Peter Horvath peter.horvath77@gmail.com:

Hi,

I’ve read that thread but it is nothing like my case.

If you check my full config you can see that searchguard login on kibana works only with the defined users not any random and i don’t have certificates setup on kibana side.

My problem is that the ES cluster plugin can’t connect

Peter

On 19 April 2018 at 11:53, SG info@search-guard.com wrote:

plse see

https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/search-guard/ZQ_-SL1tQ9k/75chW1BCAAAJ

Am 18.04.2018 um 22:41 schrieb Peter Horvath peter.horvath77@gmail.com:

Hi,

I have a kibana installed with searchguard in kubernetes, and a 3 node ES cluster on VMs

I am using the official kibana-oss docker image. There is an kubernetes nginx ingress in front of kibana interface

  • ES cluster works perfectly.
  • Application with the defined usernam/password for them working nicely
  • Curl with any user including kibanaserver works for the role defined things
  • On kibana interface i can login with the username/password of any user defined with sgadmin on the ES cluster

My problem is that i get the following errors

Error from kibana log:

{“type”:“log”,“@timestamp”:“2018-04-18T20:11:53Z”,“tags”:[“status”,“plugin:elasticsearch@6.2.2”,“error”],“pid”:8,“state”:“red”,“message”:“Status changed from yellow to red - Authentication Exception”,“prevState”:“yellow”,“prevMsg”:“Waiting for Elasticsearch”}

Error from ES:

[2018-04-18T20:18:07,559][WARN ][c.f.s.a.BackendRegistry ] Authentication finally failed for null

[2018-04-18T20:18:10,541][WARN ][c.f.s.a.BackendRegistry ] Authentication finally failed for null

How can searchguard login work on the kibana interface if kibana itself can’t connect to the ES cluster where the users are defined?

Versions:

plugin:kibana@6.2.2 Ready

plugin:elasticsearch@6.2.2 Authentication Exception

plugin:timelion@6.2.2 Ready

plugin:searchguard@6.2.2 Search Guard plugin initialised.

plugin:console@6.2.2 Ready

plugin:metrics@6.2.2 Ready

SearchGuard config

sg_config

searchguard:

dynamic:

authc:
  basic_internal_auth_domain:
    http_enabled: true
    transport_enabled: true
    order: 0
    http_authenticator:
      type: basic
      challenge: true
    authentication_backend:
      type: internal
  clientcert_auth_domain:
    transport_enabled: false
    order: 1
    http_authenticator:
      type: clientcert
      config:
        username_attribute: cn
      challenge: false
    authentication_backend:
      type: noop

sg_internal_user

kibanaserver:

readonly: true

hash: $2y$12$AoboyrVUI2A9y1yxOiyi5yx5Ni68eQLeo2BcSCNK16d6TboO

sg_roles

sg_kibana_server:

readonly: true

cluster:

  - CLUSTER_MONITOR
  - CLUSTER_COMPOSITE_OPS
  - cluster:admin/xpack/monitoring*
  - indices:admin/template*

indices:

'?kibana':
  '*':
    - INDICES_ALL
'?reporting*':
  '*':
    - INDICES_ALL
'?monitoring*':
  '*':
    - INDICES_ALL

sg_roles_mapping

sg_kibana_server:

readonly: true

users:

- kibanaserver

Kibana config:

server.host: "0.0.0.0"
[server.name](http://server.name): "[gh-kibana-stage.eur.xxx.com](http://gh-kibana-stage.eur.xxx.com)"
searchguard.cookie.secure: true
searchguard.cookie.password: "xxxxxxxxxxxx"
logging.verbose: false
elasticsearch.ssl.verificationMode: none
elasticsearch.url: "[https://eurwebstageghes01.eurweb.xxx.com:9200](https://eurwebstageghes01.eurweb.xxx.com:9200)"
elasticsearch.username: "kibanaserver"
elasticsearch.password: "xxxxxxxxxxx"

ES config:

network.bind_host: 0.0.0.0

searchguard.enterprise_modules_enabled: false

searchguard.ssl.transport.pemcert_filepath: certs/eurwebstageghes01.pem

searchguard.ssl.transport.pemkey_filepath: certs/eurwebstageghes01.key

searchguard.ssl.transport.pemkey_password: xxxxx

searchguard.ssl.transport.pemtrustedcas_filepath: certs/ca.pem

searchguard.ssl.transport.enforce_hostname_verification: true

searchguard.ssl.transport.resolve_hostname: true

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: certs/eurwebstageghes01.pem

searchguard.ssl.http.pemkey_filepath: certs/eurwebstageghes01.key

searchguard.ssl.http.pemkey_password: xxxxx

searchguard.ssl.http.pemtrustedcas_filepath: certs/ca.pem

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

  • CN=see-admin,OU=XXXX,O=XXX,DC=xxxx,DC=com

searchguard.roles_mapping_resolution: MAPPING_ONLY

searchguard.restapi.roles_enabled: [“sg_all_access”]

searchguard.audit.enable_rest: true

searchguard.audit.resolve_bulk_requests: true

searchguard.audit.type: internal_elasticsearch

You received this message because you are subscribed to the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/06657b9c-6388-48a3-a3db-5dcd8109c368%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/4E6A4BF2-0DDC-4F8C-9334-F17A161EB4AB%40search-guard.com.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/CAO9xhubo7pUph74OjxrX-KW2F-gJhHEtnKjWseLvfu_9un8QZw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/45CB4889-26F1-4755-BB3D-59665CCD2FC0%40search-guard.com.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/de4d8cde-7acd-493d-a7a0-7d6e498fe237%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Just to conclude this thread.
The problem was a leftover ENV var which overwrote the password.

···

On 19 April 2018 at 21:46, Peter Horvath peter.horvath77@gmail.com wrote:

I actually did the curl from the kibana host and 1:1 copy pasted the password from the kibana.yml and curl worked. I can as well login on the kibana gui with it. Kibana elasticsearch plugin is the only one failing.

On Thu, Apr 19, 2018, 21:43 Jochen Kressin jkressin@floragunn.com wrote:

This seems pretty strange.

From your first post I see that the kibanaserver user cannot connect to ES/SG:

plugin:elasticsearch@6.2.2
Authentication Exception

Which is also resembled in this error here:

elasticsearch_1 | [2018-04-19T20:06:50,207][DEBUG][c.f.s.a.BackendRegistry ] Can not authenticate kibanaserver due to com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]

elasticsearch_1 | com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]

So one might think this is just a password error. If the password works with curl, the only possibility that comes to my mind would be that the pwd is somehow trashed when it is transmitted from Kibana to ES/SG. So one thing you can do is to use the developer tools and having a look at the HTTPS calls on login. You should see one POST to:

/api/v1/auth/login

Can you check that the password used here is 1:1 the password you also use in your curl call?

On Thursday, April 19, 2018 at 2:46:52 PM UTC-7, Peter Horvath wrote:

I’ve also tried to upgrade the kibana stack to 6.2.3 and changed the password to short simple one in the hope that it might be too long or too complex.
None of them helped curl works kibana es plugin doesn’t.

On Thursday, 19 April 2018 16:15:01 UTC-4, Peter Horvath wrote:

Thanks! the debug mode was very useful.
My kibana config was that much i posted by the way.

I’ve tried it with " " encapsulation and without both yields the same

Kibana config:

server.host: 0.0.0.0

server.name: gh-kibana-stage.eur.xxx.com

searchguard.cookie.secure: true

searchguard.cookie.password: xxxxxxxxxxxx

logging.verbose: false

elasticsearch.ssl.verificationMode: none

elasticsearch.url: https://eurwebstageghes01.eurweb.xxx.com:9200

elasticsearch.username: kibanaserver

elasticsearch.password: xxxxxxxxxxx

So if i cat the kibana config on the kibana server and i copy paste the kibana username and password and use curl i can check health of the cluster.

elasticsearch_1 | [2018-04-19T20:06:48,971][DEBUG][c.f.s.a.BackendRegistry ] User ‘User [name=kibanaserver, roles=[kibanaserver], requestedTenant=null]’ is authenticated

elasticsearch_1 | [2018-04-19T20:06:48,971][DEBUG][c.f.s.a.BackendRegistry ] sgtenant ‘null’

elasticsearch_1 | [2018-04-19T20:06:48,971][DEBUG][c.f.s.c.PrivilegesEvaluator] ### evaluate permissions for User [name=kibanaserver, roles=[kibanaserver], requestedTenant=null] on S6VgCOD

elasticsearch_1 | [2018-04-19T20:06:48,971][DEBUG][c.f.s.c.PrivilegesEvaluator] requested cluster:monitor/health from 10.66.3.25:34102

elasticsearch_1 | [2018-04-19T20:06:48,971][DEBUG][c.f.s.c.PrivilegesEvaluator] Resolve from class org.elasticsearch.action.admin.cluster.health.ClusterHealthRequest for action cluster:monitor/health

elasticsearch_1 | [2018-04-19T20:06:48,972][DEBUG][c.f.s.c.PrivilegesEvaluator] found a match for ‘sg_kibana_server’ and cluster:monitor/health, skip other roles

But the kibana server still ends up triggering a password missmatch:

elasticsearch_1 | [2018-04-19T20:06:49,733][DEBUG][c.f.s.a.BackendRegistry ] Try to extract auth creds from basic http authenticator

elasticsearch_1 | [2018-04-19T20:06:49,733][DEBUG][c.f.s.a.BackendRegistry ] kibanaserver not cached, return from internal backend directly

elasticsearch_1 | [2018-04-19T20:06:50,207][DEBUG][c.f.s.a.BackendRegistry ] Can not authenticate kibanaserver due to com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]

elasticsearch_1 | com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]

elasticsearch_1 | at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2218) ~[guava-23.0.jar:?]

elasticsearch_1 | at com.google.common.cache.LocalCache.get(LocalCache.java:4147) ~[guava-23.0.jar:?]

Any idea?

On 19 April 2018 at 13:34, SG info@search-guard.com wrote:

Sorry for the wrong link

Am 19.04.2018 um 19:10 schrieb Peter Horvath peter.horvath77@gmail.com:

Hi,

I’ve read that thread but it is nothing like my case.

If you check my full config you can see that searchguard login on kibana works only with the defined users not any random and i don’t have certificates setup on kibana side.

My problem is that the ES cluster plugin can’t connect

Peter

On 19 April 2018 at 11:53, SG info@search-guard.com wrote:

plse see

https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/search-guard/ZQ_-SL1tQ9k/75chW1BCAAAJ

Am 18.04.2018 um 22:41 schrieb Peter Horvath peter.horvath77@gmail.com:

Hi,

I have a kibana installed with searchguard in kubernetes, and a 3 node ES cluster on VMs

I am using the official kibana-oss docker image. There is an kubernetes nginx ingress in front of kibana interface

  • ES cluster works perfectly.
  • Application with the defined usernam/password for them working nicely
  • Curl with any user including kibanaserver works for the role defined things
  • On kibana interface i can login with the username/password of any user defined with sgadmin on the ES cluster

My problem is that i get the following errors

Error from kibana log:

{“type”:“log”,“@timestamp”:“2018-04-18T20:11:53Z”,“tags”:[“status”,“plugin:elasticsearch@6.2.2”,“error”],“pid”:8,“state”:“red”,“message”:“Status changed from yellow to red - Authentication Exception”,“prevState”:“yellow”,“prevMsg”:“Waiting for Elasticsearch”}

Error from ES:

[2018-04-18T20:18:07,559][WARN ][c.f.s.a.BackendRegistry ] Authentication finally failed for null

[2018-04-18T20:18:10,541][WARN ][c.f.s.a.BackendRegistry ] Authentication finally failed for null

How can searchguard login work on the kibana interface if kibana itself can’t connect to the ES cluster where the users are defined?

Versions:

plugin:kibana@6.2.2 Ready

plugin:elasticsearch@6.2.2 Authentication Exception

plugin:timelion@6.2.2 Ready

plugin:searchguard@6.2.2 Search Guard plugin initialised.

plugin:console@6.2.2 Ready

plugin:metrics@6.2.2 Ready

SearchGuard config

sg_config

searchguard:

dynamic:

authc:
  basic_internal_auth_domain:
    http_enabled: true
    transport_enabled: true
    order: 0
    http_authenticator:
      type: basic
      challenge: true
    authentication_backend:
      type: internal
  clientcert_auth_domain:
    transport_enabled: false
    order: 1
    http_authenticator:
      type: clientcert
      config:
        username_attribute: cn
      challenge: false
    authentication_backend:
      type: noop

sg_internal_user

kibanaserver:

readonly: true

hash: $2y$12$AoboyrVUI2A9y1yxOiyi5yx5Ni68eQLeo2BcSCNK16d6TboO

sg_roles

sg_kibana_server:

readonly: true

cluster:

  - CLUSTER_MONITOR
  - CLUSTER_COMPOSITE_OPS
  - cluster:admin/xpack/monitoring*
  - indices:admin/template*

indices:

'?kibana':
  '*':
    - INDICES_ALL
'?reporting*':
  '*':
    - INDICES_ALL
'?monitoring*':
  '*':
    - INDICES_ALL

sg_roles_mapping

sg_kibana_server:

readonly: true

users:

- kibanaserver

Kibana config:

server.host: "0.0.0.0"
[server.name](http://server.name): "[gh-kibana-stage.eur.xxx.com](http://gh-kibana-stage.eur.xxx.com)"
searchguard.cookie.secure: true
searchguard.cookie.password: "xxxxxxxxxxxx"
logging.verbose: false
elasticsearch.ssl.verificationMode: none
elasticsearch.url: "[https://eurwebstageghes01.eurweb.xxx.com:9200](https://eurwebstageghes01.eurweb.xxx.com:9200)"
elasticsearch.username: "kibanaserver"
elasticsearch.password: "xxxxxxxxxxx"

ES config:

network.bind_host: 0.0.0.0

searchguard.enterprise_modules_enabled: false

searchguard.ssl.transport.pemcert_filepath: certs/eurwebstageghes01.pem

searchguard.ssl.transport.pemkey_filepath: certs/eurwebstageghes01.key

searchguard.ssl.transport.pemkey_password: xxxxx

searchguard.ssl.transport.pemtrustedcas_filepath: certs/ca.pem

searchguard.ssl.transport.enforce_hostname_verification: true

searchguard.ssl.transport.resolve_hostname: true

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: certs/eurwebstageghes01.pem

searchguard.ssl.http.pemkey_filepath: certs/eurwebstageghes01.key

searchguard.ssl.http.pemkey_password: xxxxx

searchguard.ssl.http.pemtrustedcas_filepath: certs/ca.pem

searchguard.nodes_dn:

searchguard.authcz.admin_dn:

  • CN=see-admin,OU=XXXX,O=XXX,DC=xxxx,DC=com

searchguard.roles_mapping_resolution: MAPPING_ONLY

searchguard.restapi.roles_enabled: [“sg_all_access”]

searchguard.audit.enable_rest: true

searchguard.audit.resolve_bulk_requests: true

searchguard.audit.type: internal_elasticsearch

You received this message because you are subscribed to the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/06657b9c-6388-48a3-a3db-5dcd8109c368%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/4E6A4BF2-0DDC-4F8C-9334-F17A161EB4AB%40search-guard.com.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/CAO9xhubo7pUph74OjxrX-KW2F-gJhHEtnKjWseLvfu_9un8QZw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/45CB4889-26F1-4755-BB3D-59665CCD2FC0%40search-guard.com.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/de4d8cde-7acd-493d-a7a0-7d6e498fe237%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

so is this thread resolved?

yes thank you

···

On 23 April 2018 at 09:38, Search Guard info@search-guard.com wrote:

so is this thread resolved?

You received this message because you are subscribed to a topic in the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search-guard@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/31489aef-1b84-4f8e-a12c-bc326ff73dd2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.