Kibana Username Not Found in Elasticsearch, but cURL works

ES-Version: 6.3.2

SG-Version: 6.3.2-22.3

Enterprise Modules:

  • Kibana Multitenant

  • REST Api GUI

I run Kibana, ES in a Kubernetes Environment, I have a 5 node setup consisting of the following format:

1 Ingest Node

2 Master Nodes

2 Data Nodes

I then have a Deployment for Kibana that will connect to Elasticsearch on ‘https://elasticsearch-ingest:9200

When attempting to perform a cURL command curl -k -XGET -u kibanaserver:kibanaserver 'https://elasticsearch-ingest:9200' from the Kibana Pod I am able to get a response back with the elasticsearch information. But when I launch a kibana instance using the same credentials the ingest node cannot find the user account

My elasticsearch log (elasticerror attachment) contain the error message that pops up continuously as Kibana runs.

I have also attached my sg-config (all listed as a configmap in sg-internal.yaml attachment)

What would be the reason of why I’m getting two completely results using the same credentials?

logs-from-kibana-in-kibana-68cdd596c8-g2g7v.txt (51.8 KB)

elasticerror (12.1 KB)

sg-internal.yaml (9.2 KB)

This seems strange indeed.

First, which flavour of Elasticsearch are you running? Is it the OSS version or the one packaged with X-Pack?

Same question for Kibana, is it OSS only or the X-Pack bundle?

Can you please also post your elasticsearch.yml and the kibana.yml?

···

On Wednesday, August 22, 2018 at 3:12:36 PM UTC-5, Adrian Zygadlo wrote:

ES-Version: 6.3.2

SG-Version: 6.3.2-22.3

Enterprise Modules:

  • Kibana Multitenant
  • REST Api GUI

I run Kibana, ES in a Kubernetes Environment, I have a 5 node setup consisting of the following format:

1 Ingest Node

2 Master Nodes

2 Data Nodes

I then have a Deployment for Kibana that will connect to Elasticsearch on ‘https://elasticsearch-ingest:9200

When attempting to perform a cURL command curl -k -XGET -u kibanaserver:kibanaserver 'https://elasticsearch-ingest:9200' from the Kibana Pod I am able to get a response back with the elasticsearch information. But when I launch a kibana instance using the same credentials the ingest node cannot find the user account

My elasticsearch log (elasticerror attachment) contain the error message that pops up continuously as Kibana runs.

I have also attached my sg-config (all listed as a configmap in sg-internal.yaml attachment)

What would be the reason of why I’m getting two completely results using the same credentials?

I am running the X-Pack bundled version on both Kibana and Elasticsearch. I have also attached both of the config yml files as requested.

Thank you for looking into this.

kibana.1.yml (2.41 KB)

elasticsearch.yml (1.43 KB)

···

On Thursday, August 23, 2018 at 12:15:16 PM UTC-4, Jochen Kressin wrote:

This seems strange indeed.

First, which flavour of Elasticsearch are you running? Is it the OSS version or the one packaged with X-Pack?

Same question for Kibana, is it OSS only or the X-Pack bundle?

Can you please also post your elasticsearch.yml and the kibana.yml?

On Wednesday, August 22, 2018 at 3:12:36 PM UTC-5, Adrian Zygadlo wrote:

ES-Version: 6.3.2

SG-Version: 6.3.2-22.3

Enterprise Modules:

  • Kibana Multitenant
  • REST Api GUI

I run Kibana, ES in a Kubernetes Environment, I have a 5 node setup consisting of the following format:

1 Ingest Node

2 Master Nodes

2 Data Nodes

I then have a Deployment for Kibana that will connect to Elasticsearch on ‘https://elasticsearch-ingest:9200

When attempting to perform a cURL command curl -k -XGET -u kibanaserver:kibanaserver 'https://elasticsearch-ingest:9200' from the Kibana Pod I am able to get a response back with the elasticsearch information. But when I launch a kibana instance using the same credentials the ingest node cannot find the user account

My elasticsearch log (elasticerror attachment) contain the error message that pops up continuously as Kibana runs.

I have also attached my sg-config (all listed as a configmap in sg-internal.yaml attachment)

What would be the reason of why I’m getting two completely results using the same credentials?

Any ideas what might be going wrong?

···

On Thursday, August 23, 2018 at 12:15:16 PM UTC-4, Jochen Kressin wrote:

This seems strange indeed.

First, which flavour of Elasticsearch are you running? Is it the OSS version or the one packaged with X-Pack?

Same question for Kibana, is it OSS only or the X-Pack bundle?

Can you please also post your elasticsearch.yml and the kibana.yml?

On Wednesday, August 22, 2018 at 3:12:36 PM UTC-5, Adrian Zygadlo wrote:

ES-Version: 6.3.2

SG-Version: 6.3.2-22.3

Enterprise Modules:

  • Kibana Multitenant
  • REST Api GUI

I run Kibana, ES in a Kubernetes Environment, I have a 5 node setup consisting of the following format:

1 Ingest Node

2 Master Nodes

2 Data Nodes

I then have a Deployment for Kibana that will connect to Elasticsearch on ‘https://elasticsearch-ingest:9200

When attempting to perform a cURL command curl -k -XGET -u kibanaserver:kibanaserver 'https://elasticsearch-ingest:9200' from the Kibana Pod I am able to get a response back with the elasticsearch information. But when I launch a kibana instance using the same credentials the ingest node cannot find the user account

My elasticsearch log (elasticerror attachment) contain the error message that pops up continuously as Kibana runs.

I have also attached my sg-config (all listed as a configmap in sg-internal.yaml attachment)

What would be the reason of why I’m getting two completely results using the same credentials?

This is still strange, but I can at least state the following: All calls to Elaticsearch / Search Guard are idempotent and independent from the client that submitted the HTTP request. This means that the outcome must be the same if the input (here: the kibanaserver user) is the same. It it not possible that it sometimes works, sometimes not.

The only thing that I noticed, and this might just be a formatting or copy and paste error, is a suspicious newline in the logs whenever the “kibanaserver” username is printed. You can see it three times:

[2018-08-22T18:02:43,594][WARN ][c.f.s.a.BackendRegistry ] Authentication finally failed for kibanaserver

[2018-08-22T18:02:46,388][DEBUG][c.f.s.a.BackendRegistry ] Try to extract auth creds from basic http authenticator

[2018-08-22T18:02:46,388][DEBUG][c.f.s.a.BackendRegistry ] kibanaserver

not cached, return from internal backend directly

[2018-08-22T18:02:46,388][DEBUG][c.f.s.a.BackendRegistry ] Can not authenticate kibanaserver

due to com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[kibanaserver

not found]

``

See the three newlines here? Maybe the username gets a extra newline when it is configured in kibana.yml?

···

On Wednesday, August 29, 2018 at 8:55:30 PM UTC-4, Adrian Zygadlo wrote:

Any ideas what might be going wrong?

On Thursday, August 23, 2018 at 12:15:16 PM UTC-4, Jochen Kressin wrote:

This seems strange indeed.

First, which flavour of Elasticsearch are you running? Is it the OSS version or the one packaged with X-Pack?

Same question for Kibana, is it OSS only or the X-Pack bundle?

Can you please also post your elasticsearch.yml and the kibana.yml?

On Wednesday, August 22, 2018 at 3:12:36 PM UTC-5, Adrian Zygadlo wrote:

ES-Version: 6.3.2

SG-Version: 6.3.2-22.3

Enterprise Modules:

  • Kibana Multitenant
  • REST Api GUI

I run Kibana, ES in a Kubernetes Environment, I have a 5 node setup consisting of the following format:

1 Ingest Node

2 Master Nodes

2 Data Nodes

I then have a Deployment for Kibana that will connect to Elasticsearch on ‘https://elasticsearch-ingest:9200

When attempting to perform a cURL command curl -k -XGET -u kibanaserver:kibanaserver 'https://elasticsearch-ingest:9200' from the Kibana Pod I am able to get a response back with the elasticsearch information. But when I launch a kibana instance using the same credentials the ingest node cannot find the user account

My elasticsearch log (elasticerror attachment) contain the error message that pops up continuously as Kibana runs.

I have also attached my sg-config (all listed as a configmap in sg-internal.yaml attachment)

What would be the reason of why I’m getting two completely results using the same credentials?