kibana always create index pattern fail after search guard integrate with k8s

When I try to create an “Index Patterns”, always fail as below:

elastic search log :

[2018-11-27T15:46:42,474][INFO ][o.e.c.m.MetaDataMappingService] [voROLHr] [searchguard/-1GwoUFQTeaYFWoxGFWKCQ] update_mapping [sg]

[2018-11-27T15:46:42,631][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Info: No license needed because enterprise modules are not enabled

[2018-11-27T15:47:06,605][INFO ][o.e.c.m.MetaDataCreateIndexService] [voROLHr] [logstash-2018.11.27] creating index, cause [auto(bulk api)], templates , shards [5]/[1], mappings

[2018-11-27T15:47:06,792][INFO ][o.e.c.m.MetaDataMappingService] [voROLHr] [logstash-2018.11.27/AiH27yT9QE-8QATtHGFZqw] create_mapping [fluentd]

[2018-11-27T15:48:11,049][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]

[2018-11-27T15:48:12,802][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]

[2018-11-27T15:48:12,917][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]

[2018-11-27T15:48:13,190][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]

[2018-11-27T15:48:13,312][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]

[2018-11-27T15:48:13,748][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]

kibana UI show error:

  • Not Found

       Error: Not Found
    at Anonymous function (http://a537d1a7eeeb911e889170eff959d76b-2102519552.us-east-1.elb.amazonaws.com:5601/bundles/commons.bundle.js?v=16627:1:312436)
    at processQueue (http://a537d1a7eeeb911e889170eff959d76b-2102519552.us-east-1.elb.amazonaws.com:5601/bundles/vendors.bundle.js?v=16627:58:132433)
    at Anonymous function (http://a537d1a7eeeb911e889170eff959d76b-2102519552.us-east-1.elb.amazonaws.com:5601/bundles/vendors.bundle.js?v=16627:58:133349)
    at Scope.prototype.$digest (http://a537d1a7eeeb911e889170eff959d76b-2102519552.us-east-1.elb.amazonaws.com:5601/bundles/vendors.bundle.js?v=16627:58:144182)
    at Scope.prototype.$apply (http://a537d1a7eeeb911e889170eff959d76b-2102519552.us-east-1.elb.amazonaws.com:5601/bundles/vendors.bundle.js?v=16627:58:147007)
    at done (http://a537d1a7eeeb911e889170eff959d76b-2102519552.us-east-1.elb.amazonaws.com:5601/bundles/vendors.bundle.js?v=16627:58:99799)
    at completeRequest (http://a537d1a7eeeb911e889170eff959d76b-2102519552.us-east-1.elb.amazonaws.com:5601/bundles/vendors.bundle.js?v=16627:58:104624)
    at xhr.onload (http://a537d1a7eeeb911e889170eff959d76b-2102519552.us-east-1.elb.amazonaws.com:5601/bundles/vendors.bundle.js?v=16627:58:105361)
    

elastic search log

Note:

Works well for below command:

kubectl get pods -n
kube-system -l k8s-app=elasticsearch-logging |awk ‘{print $1}’|sed ‘1d’|xargs
-i sh -c ‘kubectl exec -ti {} -n
kube-system – curl http://localhost:9200/_cat/health
-k -u admin:admin’

kubectl get pods -n
kube-system -l k8s-app=elasticsearch-logging |awk ‘{print $1}’|sed ‘1d’|xargs
-i sh -c ‘kubectl exec -ti {} -n
kube-system – curl http://localhost:9200/_cat/indices?v
-k -u admin:admin’

Also works find for kibana log in.

Dockerfile_kibana_image.txt (235 Bytes)

elasticsearch_image.tar (40 KB)

es-statefulset.yaml (2.27 KB)

Pls delete the .kibana index and report if this fixes the problem.

If not pls. upgrade to SG 6.4.3-23.2 and SG Kibana v16 and report if the error is still occuring.

On k8s you also can try GitHub - floragunncom/search-guard-helm: Search Guard Helm Chart for Kubernetes

···

On Tuesday, 27 November 2018 16:50:40 UTC+1, … wrote:

When I try to create an “Index Patterns”, always fail as below:

elastic search log :

[2018-11-27T15:46:42,474][INFO ][o.e.c.m.MetaDataMappingService] [voROLHr] [searchguard/-1GwoUFQTeaYFWoxGFWKCQ] update_mapping [sg]

[2018-11-27T15:46:42,631][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Info: No license needed because enterprise modules are not enabled

[2018-11-27T15:47:06,605][INFO ][o.e.c.m.MetaDataCreateIndexService] [voROLHr] [logstash-2018.11.27] creating index, cause [auto(bulk api)], templates , shards [5]/[1], mappings

[2018-11-27T15:47:06,792][INFO ][o.e.c.m.MetaDataMappingService] [voROLHr] [logstash-2018.11.27/AiH27yT9QE-8QATtHGFZqw] create_mapping [fluentd]

[2018-11-27T15:48:11,049][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]

[2018-11-27T15:48:12,802][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]

[2018-11-27T15:48:12,917][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]

[2018-11-27T15:48:13,190][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]

[2018-11-27T15:48:13,312][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]

[2018-11-27T15:48:13,748][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]

kibana UI show error:

  • Not Found
   Error: Not Found
at Anonymous function ([http://a537d1a7eeeb911e889170eff959d76b-2102519552.us-east-1.elb.amazonaws.com:5601/bundles/commons.bundle.js?v=16627:1:312436](http://a537d1a7eeeb911e889170eff959d76b-2102519552.us-east-1.elb.amazonaws.com:5601/bundles/commons.bundle.js?v=16627:1:312436)   )
at processQueue ([http://a537d1a7eeeb911e889170eff959d76b-2102519552.us-east-1.elb.amazonaws.com:5601/bundles/vendors.bundle.js?v=16627:58:132433](http://a537d1a7eeeb911e889170eff959d76b-2102519552.us-east-1.elb.amazonaws.com:5601/bundles/vendors.bundle.js?v=16627:58:132433)   )
at Anonymous function ([http://a537d1a7eeeb911e889170eff959d76b-2102519552.us-east-1.elb.amazonaws.com:5601/bundles/vendors.bundle.js?v=16627:58:133349](http://a537d1a7eeeb911e889170eff959d76b-2102519552.us-east-1.elb.amazonaws.com:5601/bundles/vendors.bundle.js?v=16627:58:133349)   )
at Scope.prototype.$digest ([http://a537d1a7eeeb911e889170eff959d76b-2102519552.us-east-1.elb.amazonaws.com:5601/bundles/vendors.bundle.js?v=16627:58:144182](http://a537d1a7eeeb911e889170eff959d76b-2102519552.us-east-1.elb.amazonaws.com:5601/bundles/vendors.bundle.js?v=16627:58:144182)   )
at Scope.prototype.$apply ([http://a537d1a7eeeb911e889170eff959d76b-2102519552.us-east-1.elb.amazonaws.com:5601/bundles/vendors.bundle.js?v=16627:58:147007](http://a537d1a7eeeb911e889170eff959d76b-2102519552.us-east-1.elb.amazonaws.com:5601/bundles/vendors.bundle.js?v=16627:58:147007)   )
at done ([http://a537d1a7eeeb911e889170eff959d76b-2102519552.us-east-1.elb.amazonaws.com:5601/bundles/vendors.bundle.js?v=16627:58:99799](http://a537d1a7eeeb911e889170eff959d76b-2102519552.us-east-1.elb.amazonaws.com:5601/bundles/vendors.bundle.js?v=16627:58:99799)   )
at completeRequest ([http://a537d1a7eeeb911e889170eff959d76b-2102519552.us-east-1.elb.amazonaws.com:5601/bundles/vendors.bundle.js?v=16627:58:104624](http://a537d1a7eeeb911e889170eff959d76b-2102519552.us-east-1.elb.amazonaws.com:5601/bundles/vendors.bundle.js?v=16627:58:104624)   )
at xhr.onload ([http://a537d1a7eeeb911e889170eff959d76b-2102519552.us-east-1.elb.amazonaws.com:5601/bundles/vendors.bundle.js?v=16627:58:105361](http://a537d1a7eeeb911e889170eff959d76b-2102519552.us-east-1.elb.amazonaws.com:5601/bundles/vendors.bundle.js?v=16627:58:105361))

elastic search log

Note:

Works well for below command:

kubectl get pods -n
kube-system -l k8s-app=elasticsearch-logging |awk ‘{print $1}’|sed ‘1d’|xargs
-i sh -c ‘kubectl exec -ti {} -n
kube-system – curl http://localhost:9200/_cat/health
-k -u admin:admin’

kubectl get pods -n
kube-system -l k8s-app=elasticsearch-logging |awk ‘{print $1}’|sed ‘1d’|xargs
-i sh -c ‘kubectl exec -ti {} -n
kube-system – curl http://localhost:9200/_cat/indices?v
-k -u admin:admin’

Also works find for kibana log in.

It works for delete .kibana.
(Before delete, I found 2 out of 3 has .kibana index, 1 out of 3 has no .kibana)

After delete .kibana, sometimes I can create index pattern successfully. Thanks.

I tried a lot of times, the fail rate is very high for 6.2.4.

So, I upgraded to SG 6.4.3-23.2 and SG Kibana v16 per your suggestion, the issue still raised!

1 The error alert in kibana:

Saved object is missing

Could not locate that index-pattern (id: c0838140-f2e0-11e8-a5e0-ef9f9f948c34), click here to re-create it

2 There are three es nodes, .kibana only be created on 1 or 2 nodes, not all 3 nodes.


health status index               uuid

yellow open   logstash-2018.11.28 3BnNbv9JTImVV4AnC7hQew

green  open   searchguard         L0lqwN1eR1mNHh_fXX8qYQ

health status index               uuid

yellow open   logstash-2018.11.28 ig90IAk1QNi-VQOoqyuycQ

green  open   searchguard         -frvcPLzQLSjjb6JPOu4fA

yellow open   .kibana             EQqOHoWYSeWNvX1o6xoqQQ

health status index               uuid

green  open   searchguard         Ywpmo63DTfqnCznIG-wyGw

yellow open   logstash-2018.11.28 M0cgljPrQKW27e1trnma-w

does it work without search guard installed?

···

On Wednesday, 28 November 2018 08:43:38 UTC+1, … wrote:

I tried a lot of times, the fail rate is very high for 6.2.4.

So, I upgraded to SG 6.4.3-23.2 and SG Kibana v16 per your suggestion, the issue still raised!

1 The error alert in kibana:

Saved object is missing

Could not locate that index-pattern (id: c0838140-f2e0-11e8-a5e0-ef9f9f948c34), click here to re-create it

2 There are three es nodes, .kibana only be created on 1 or 2 nodes, not all 3 nodes.

health status index uuid

yellow open logstash-2018.11.28 3BnNbv9JTImVV4AnC7hQew

green open searchguard L0lqwN1eR1mNHh_fXX8qYQ

health status index uuid

yellow open logstash-2018.11.28 ig90IAk1QNi-VQOoqyuycQ

green open searchguard -frvcPLzQLSjjb6JPOu4fA

yellow open .kibana EQqOHoWYSeWNvX1o6xoqQQ

health status index uuid

green open searchguard Ywpmo63DTfqnCznIG-wyGw

yellow open logstash-2018.11.28 M0cgljPrQKW27e1trnma-w

Yes, it works without search guard installed.

Could you take a look with

my docker image for elastic search: efk_sg_in_k8s/images/elasticsearch_baseonelk at master · johnzheng1975/efk_sg_in_k8s · GitHub

my docker image for kibana: efk_sg_in_k8s/images/kibana at master · johnzheng1975/efk_sg_in_k8s · GitHub

See whether it is correct?

Or you can recommend any successful docker image (es + kibana + sg integration)?

Thanks!

So if the .kibana index is only present on one node this seems more like a Kubernetes or Docker problem to me. The .kibana index is created by Kibana itself and replicated in ES, like any other index. Search Guard does not interfere here in any way.

Regarding Docker images, there is this repository you can refer to:

We have also published experimental Helm charts here:

The Helm charts are not yet officially supported by us, but you can use them as a reference. This setup is running at a customer site, so it should work basically.

···

On Wednesday, November 28, 2018 at 10:00:16 AM UTC+1, johnzhengaz@gmail.com wrote:

Yes, it works without search guard installed.

Could you take a look with

my docker image for elastic search: https://github.com/johnzheng1975/efk_sg_in_k8s/tree/master/images/elasticsearch_baseonelk

my docker image for kibana: https://github.com/johnzheng1975/efk_sg_in_k8s/tree/master/images/kibana

See whether it is correct?

Or you can recommend any successful docker image (es + kibana + sg integration)?

Thanks!

can I use it as free? I means disable enterprise function. thanks.

For GitHub - floragunncom/search-guard-helm: Search Guard Helm Chart for Kubernetes,

Sure, it’s a public repository, feel free to use anything that helps you.

License is Apache2:

https://github.com/floragunncom/search-guard-helm/blob/master/LICENSE

···

On Wednesday, November 28, 2018 at 2:07:43 PM UTC+1, johnzhengaz@gmail.com wrote:

For https://github.com/floragunncom/search-guard-helm,
can I use it as free? I means disable enterprise function. thanks.

That’s great! I am trying search-guard-helm. :slight_smile:

Oh, I know helm is open source.

However, for below, I guess it includes enterprise version, right?
floragunncom/searchguard-6:6.4.1-23.1

floragunncom/sg-kibana:6.4.1-15

floragunncom/sgadmin:6.4.1-23.1

Can I put disable enterprise function?

Is it done like below:

In kibana.yml:

searchguard.auth.type:
“basicauth”

In elasticsearch.yml

searchguard.enterprise_modules_enabled: false

Is there any others I should change?

Thanks very much for your support!

Sure, that’s what I wanted to say: The code in the searchguard-helm repository is Apache2:

https://github.com/floragunncom/search-guard-helm/blob/master/LICENSE

To switch from the Enterprise to the Community Edition you just need to disable the enterprise module in elasticsearch.yml as you pointed out. Nothing more to do.

···

On Thursday, November 29, 2018 at 3:25:49 AM UTC+1, johnzhengaz@gmail.com wrote:

Oh, I know helm is open source.

However, for below, I guess it includes enterprise version, right?
floragunncom/searchguard-6:6.4.1-23.1

floragunncom/sg-kibana:6.4.1-15

floragunncom/sgadmin:6.4.1-23.1

Can I put disable enterprise function?

Is it done like below:

In kibana.yml:

searchguard.auth.type:
“basicauth”

In elasticsearch.yml

searchguard.enterprise_modules_enabled: false

Is there any others I should change?

Thanks very much for your support!

That’s very cool! Thanks.