Kubernetes multi-node issues

I’m struggling to understand why my single-pod in an ElasticSearch kubernetes deployment works as expected; and my scaled multi-pod ElasticSearch in kubernetes does not. Kibana exists in a separate pod in both cases.

Using the same SG/ES/K configuration and docker image version for both env’s:

Multi-Pods: 2 es-data (with persistent volumes); 2 masters; 1 ingest (“client”); 1 kibana

Single-Pod: 1 with all responsibilities and local storage; 1 kibana

Specifically the issues in the multi-node:

  1. The Kibana SearchGuard side bar menu option is not present, I understand the appearance of this option is controlled by the user logged in and the elasticsearch configuration item:

** searchguard.restapi.roles_enabled: [“sg_all_access”, “sg_kibana_server”]

  1. In kibana, under Management attempting to create an Index Pattern fails with message presented in a lower right pop-up:

** Saved object is missing Could not locate that index-pattern (id: d9229440-d7b3-11e8-ae97-a9d1b06dca43),

A few relevant things that do work in both env’s:

  • Login with different users that are defined the Elasticsearch (plugins/search-guard-6/sgconfig/sg_internal_users.yml), I’m presented with the SearchGuard login

  • The multi-pod/node ElasticSearch env works without SearchGuard, we’ve been running this for a a couple months successfully

  • Tenants defined (plugins/search-guard-6/sgconfig/sg_roles.yml) function in the Kibana UI as expected in both single and multi-pod env’s. User with limited access, only the expected tenant is available.

  • sgadmin command works expected in both env’s, I log onto a master pod and executed it, no errors… finishes with “Done with success”

es:9200/_cat/indices?v

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open sg6-auditlog-2018.10.24 czYC2f7SSNyKwRjlQA7jng 5 1 3 0 125kb 62.5kb
green open searchguard OydvEdSITbKFPuCV_m97Kw 1 1 0 6 99.7kb 38.6kb
green open .kibana_-152937574_admintenant VhuHZ6tmRDOTDiFeXiD7Zg 1 1 2 0 18.5kb 9.2kb

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version

SG: search-guard-kibana-plugin-6.4.2-15.zip

ES: 6.4.2 quay.io/pires/docker-elasticsearch:6.4.2

K: 6.4.2 docker.elastic.co/kibana/kibana:6.4.2

  • Installed and used enterprise modules, if any: None

  • JVM version and operating system version:

bash-4.4# java -version

openjdk version “1.8.0_171”

OpenJDK Runtime Environment (IcedTea 3.8.0) (Alpine 8.171.11-r0)

OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)

OS Based on: quay.io/pires/docker-elasticsearch:6.4.2

  • Search Guard configuration files: attached

  • Elasticsearch log messages on debug level: Attached are kibana and es logs at info level; if debug is required let me know

  • Other installed Elasticsearch or Kibana plugins, if any: None

Thanks, Randy

kibana.yml (5.59 KB)

sg_config.yml (9.36 KB)

sg_internal_users.yml (1.43 KB)

sg_roles.yml (6.56 KB)

sg_roles_mapping.yml (759 Bytes)

es-client-7d46689dff-wnbpj-logs.txt (14.6 KB)

es-data-0-logs.txt (12 KB)

es-data-1-logs.txt (10.5 KB)

es-master-5858d4896f-6mdws-logs.txt (25.3 KB)

es-master-5858d4896f-6wxvd-logs.txt (16.5 KB)

kibana-68ccd84494-lcwml-logs.txt (176 KB)

Pls. make sure you have not set http.enabled: false for any node (so either you should not set a specific value or set http.enabled: true).
I know that normally you want to not enable HTTP/S on the master/data nodes but we have a known bug with that yet and in addition, http.enabled is also deprecated and will be removed (by elastic because they will deprecate the transport client and so HTTP/S becomes mandatory)

This should fix issue 1)

To fix issue 2) (which is also known and will be fixed in the next release) do:

a) enable GLOBAL tenant

b) let Kibana create the .kibana index

c) disable GLOBAL tenant

Hope this helps

···

On Thursday, 25 October 2018 01:08:30 UTC+2, rho…com wrote:

I’m struggling to understand why my single-pod in an ElasticSearch kubernetes deployment works as expected; and my scaled multi-pod ElasticSearch in kubernetes does not. Kibana exists in a separate pod in both cases.

Using the same SG/ES/K configuration and docker image version for both env’s:

Multi-Pods: 2 es-data (with persistent volumes); 2 masters; 1 ingest (“client”); 1 kibana

Single-Pod: 1 with all responsibilities and local storage; 1 kibana

Specifically the issues in the multi-node:

  1. The Kibana SearchGuard side bar menu option is not present, I understand the appearance of this option is controlled by the user logged in and the elasticsearch configuration item:

** searchguard.restapi.roles_enabled: [“sg_all_access”, “sg_kibana_server”]

  1. In kibana, under Management attempting to create an Index Pattern fails with message presented in a lower right pop-up:

** Saved object is missing Could not locate that index-pattern (id: d9229440-d7b3-11e8-ae97-a9d1b06dca43),

A few relevant things that do work in both env’s:

  • Login with different users that are defined the Elasticsearch (plugins/search-guard-6/sgconfig/sg_internal_users.yml), I’m presented with the SearchGuard login
  • The multi-pod/node ElasticSearch env works without SearchGuard, we’ve been running this for a a couple months successfully
  • Tenants defined (plugins/search-guard-6/sgconfig/sg_roles.yml) function in the Kibana UI as expected in both single and multi-pod env’s. User with limited access, only the expected tenant is available.
  • sgadmin command works expected in both env’s, I log onto a master pod and executed it, no errors… finishes with “Done with success”

es:9200/_cat/indices?v

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open sg6-auditlog-2018.10.24 czYC2f7SSNyKwRjlQA7jng 5 1 3 0 125kb 62.5kb
green open searchguard OydvEdSITbKFPuCV_m97Kw 1 1 0 6 99.7kb 38.6kb
green open .kibana_-152937574_admintenant VhuHZ6tmRDOTDiFeXiD7Zg 1 1 2 0 18.5kb 9.2kb

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version

SG: search-guard-kibana-plugin-6.4.2-15.zip

ES: 6.4.2 quay.io/pires/docker-elasticsearch:6.4.2

K: 6.4.2 docker.elastic.co/kibana/kibana:6.4.2

  • Installed and used enterprise modules, if any: None
  • JVM version and operating system version:

bash-4.4# java -version

openjdk version “1.8.0_171”

OpenJDK Runtime Environment (IcedTea 3.8.0) (Alpine 8.171.11-r0)

OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)

OS Based on: quay.io/pires/docker-elasticsearch:6.4.2

  • Search Guard configuration files: attached
  • Elasticsearch log messages on debug level: Attached are kibana and es logs at info level; if debug is required let me know
  • Other installed Elasticsearch or Kibana plugins, if any: None

Thanks, Randy

FYI

https://github.com/floragunncom/search-guard-helm (This is repo is considered experimental and not officially supported. Use at your own risk.)

···

On Thursday, 25 October 2018 01:08:30 UTC+2, … wrote:

I’m struggling to understand why my single-pod in an ElasticSearch kubernetes deployment works as expected; and my scaled multi-pod ElasticSearch in kubernetes does not. Kibana exists in a separate pod in both cases.

Using the same SG/ES/K configuration and docker image version for both env’s:

Multi-Pods: 2 es-data (with persistent volumes); 2 masters; 1 ingest (“client”); 1 kibana

Single-Pod: 1 with all responsibilities and local storage; 1 kibana

Specifically the issues in the multi-node:

  1. The Kibana SearchGuard side bar menu option is not present, I understand the appearance of this option is controlled by the user logged in and the elasticsearch configuration item:

** searchguard.restapi.roles_enabled: [“sg_all_access”, “sg_kibana_server”]

  1. In kibana, under Management attempting to create an Index Pattern fails with message presented in a lower right pop-up:

** Saved object is missing Could not locate that index-pattern (id: d9229440-d7b3-11e8-ae97-a9d1b06dca43),

A few relevant things that do work in both env’s:

  • Login with different users that are defined the Elasticsearch (plugins/search-guard-6/sgconfig/sg_internal_users.yml), I’m presented with the SearchGuard login
  • The multi-pod/node ElasticSearch env works without SearchGuard, we’ve been running this for a a couple months successfully
  • Tenants defined (plugins/search-guard-6/sgconfig/sg_roles.yml) function in the Kibana UI as expected in both single and multi-pod env’s. User with limited access, only the expected tenant is available.
  • sgadmin command works expected in both env’s, I log onto a master pod and executed it, no errors… finishes with “Done with success”

es:9200/_cat/indices?v

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open sg6-auditlog-2018.10.24 czYC2f7SSNyKwRjlQA7jng 5 1 3 0 125kb 62.5kb
green open searchguard OydvEdSITbKFPuCV_m97Kw 1 1 0 6 99.7kb 38.6kb
green open .kibana_-152937574_admintenant VhuHZ6tmRDOTDiFeXiD7Zg 1 1 2 0 18.5kb 9.2kb

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version

SG: search-guard-kibana-plugin-6.4.2-15.zip

ES: 6.4.2 quay.io/pires/docker-elasticsearch:6.4.2

K: 6.4.2 docker.elastic.co/kibana/kibana:6.4.2

  • Installed and used enterprise modules, if any: None
  • JVM version and operating system version:

bash-4.4# java -version

openjdk version “1.8.0_171”

OpenJDK Runtime Environment (IcedTea 3.8.0) (Alpine 8.171.11-r0)

OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)

OS Based on: quay.io/pires/docker-elasticsearch:6.4.2

  • Search Guard configuration files: attached
  • Elasticsearch log messages on debug level: Attached are kibana and es logs at info level; if debug is required let me know
  • Other installed Elasticsearch or Kibana plugins, if any: None

Thanks, Randy

Your help resolved issue #1, the Search Guard menu option appeared. Thanks!!

For Issue #2, creating saved index patterns, I’m able to create index patterns now!

  • The index pattern is created in the expected tenant

To prove it out I deleted all Kubernetes resources, including persistentvolumeclaims

In kibana.yml:

  • searchguard.multitenancy.enabled: false

  • searchguard.multitenancy.tenants.enable_global: true

Bring everything backup, login to kibana, create an Index Pattern “s*” (i.e. sg6-auditlog-2018.10.25)

  • This action created the .kibana index

Then recreate the kibana pod with the above values switched

  • searchguard.multitenancy.enabled: true

  • searchguard.multitenancy.tenants.enable_global: false

Issue #3:

  • After choosing a tenant (i.e. “cust2_tentant”) with an indice limitation of: “util-c2*”, I navigate to Management | Index Patterns I see the “sg6-auditlog-2018.10.25” index

  • When I turn on “Include system indices” I see indices like: .kibana_-152937574_admintenant and .kibana_-152937574_cust1tenant

– This is bad, customer2 can see references to customer1.

– Are these “system indices” in the global tenant now?

Question:

  • Does Search Guard support environment variables in the configuration similar to:

** https://www.elastic.co/guide/en/kibana/current/docker.html#environment-variable-config

** I would like to do something like:

name: “SEARCHGUARD_MULTITENANCY_TENANTS_ENABLE_GLOBAL”

value: “true”

Scratch Issue #3, I was still logged in as a higher privileged user that’s why I was still seeing system indices, etc. Once I logged out and back in as a different user the expected behavior and appropriate indices were presented.
Thanks!