I’m struggling to understand why my single-pod in an ElasticSearch kubernetes deployment works as expected; and my scaled multi-pod ElasticSearch in kubernetes does not. Kibana exists in a separate pod in both cases.
Using the same SG/ES/K configuration and docker image version for both env’s:
Multi-Pods: 2 es-data (with persistent volumes); 2 masters; 1 ingest (“client”); 1 kibana
Single-Pod: 1 with all responsibilities and local storage; 1 kibana
Specifically the issues in the multi-node:
- The Kibana SearchGuard side bar menu option is not present, I understand the appearance of this option is controlled by the user logged in and the elasticsearch configuration item:
** searchguard.restapi.roles_enabled: [“sg_all_access”, “sg_kibana_server”]
- In kibana, under Management attempting to create an Index Pattern fails with message presented in a lower right pop-up:
** Saved object is missing Could not locate that index-pattern (id: d9229440-d7b3-11e8-ae97-a9d1b06dca43),
A few relevant things that do work in both env’s:
-
Login with different users that are defined the Elasticsearch (plugins/search-guard-6/sgconfig/sg_internal_users.yml), I’m presented with the SearchGuard login
-
The multi-pod/node ElasticSearch env works without SearchGuard, we’ve been running this for a a couple months successfully
-
Tenants defined (plugins/search-guard-6/sgconfig/sg_roles.yml) function in the Kibana UI as expected in both single and multi-pod env’s. User with limited access, only the expected tenant is available.
-
sgadmin command works expected in both env’s, I log onto a master pod and executed it, no errors… finishes with “Done with success”
es:9200/_cat/indices?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open sg6-auditlog-2018.10.24 czYC2f7SSNyKwRjlQA7jng 5 1 3 0 125kb 62.5kb
green open searchguard OydvEdSITbKFPuCV_m97Kw 1 1 0 6 99.7kb 38.6kb
green open .kibana_-152937574_admintenant VhuHZ6tmRDOTDiFeXiD7Zg 1 1 2 0 18.5kb 9.2kb
When asking questions, please provide the following information:
- Search Guard and Elasticsearch version
SG: search-guard-kibana-plugin-6.4.2-15.zip
ES: 6.4.2 Quay
K: 6.4.2 docker.elastic.co/kibana/kibana:6.4.2
-
Installed and used enterprise modules, if any: None
-
JVM version and operating system version:
bash-4.4# java -version
openjdk version “1.8.0_171”
OpenJDK Runtime Environment (IcedTea 3.8.0) (Alpine 8.171.11-r0)
OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)
OS Based on: Quay
-
Search Guard configuration files: attached
-
Elasticsearch log messages on debug level: Attached are kibana and es logs at info level; if debug is required let me know
-
Other installed Elasticsearch or Kibana plugins, if any: None
Thanks, Randy
kibana.yml (5.59 KB)
sg_config.yml (9.36 KB)
sg_internal_users.yml (1.43 KB)
sg_roles.yml (6.56 KB)
sg_roles_mapping.yml (759 Bytes)
es-client-7d46689dff-wnbpj-logs.txt (14.6 KB)
es-data-0-logs.txt (12 KB)
es-data-1-logs.txt (10.5 KB)
es-master-5858d4896f-6mdws-logs.txt (25.3 KB)
es-master-5858d4896f-6wxvd-logs.txt (16.5 KB)
kibana-68ccd84494-lcwml-logs.txt (176 KB)