latest Searchguard bundle not working on a single node

Search Guard
1

Hi All,
Can anyone help me with the below issue I am facing

I used tarfile which was said to be the easiest way to installtion searchguard which also includes elasticsearch package. Please refer to below link
https://github.com/floragunncom/search-guard/wiki/Search-Guard-Bundle.

After extracting tarfile I have started elasticsearch using the steps as mentioned in link and later ran sgadmin.sh script and evrything was looking great.

Once I have installed Kibana and Logstash as root user , installed beats on client machine . Currently beats are able to push logs to logstash whereas logstash is not able to comminucate to elasticsearch.

When I found reason it says “Search Guard not initialized (SG11)” and searchguard index not healthy (timeout: truewhich is really strange.

Note: I deployed search guard tarfile using non rootuser.I also installed Kibana logastah and Elasticsearch with search guard on same server. beats on client machines.

Can anyone guide me here please as I am going crazy with this searchguard installation?

2

pls do not cross post, this one is already logged here https://github.com/floragunncom/search-guard/issues/182#issuecomment-237533599

Can you please provide error messages and your elasticsearch log files as well as your configuration? Otherwise its hard to help.

Start easy just with the Search Guard bundle and curl. Once you initialized Search Guard by runiing sgadmin you normally should not see a "Search Guard not initialized (SG11)" afterwards.

···

Am 04.08.2016 um 15:58 schrieb rocky <rocky.munich@gmail.com>:

Hi All,
Can anyone help me with the below issue I am facing

I used tarfile which was said to be the easiest way to installtion searchguard which also includes elasticsearch package. Please refer to below link
GitHub - floragunncom/search-guard: Search Guard Plugin - Security for Elasticsearch.

After extracting tarfile I have started elasticsearch using the steps as mentioned in link and later ran sgadmin.sh script and evrything was looking great.

Once I have installed Kibana and Logstash as root user , installed beats on client machine . Currently beats are able to push logs to logstash whereas logstash is not able to comminucate to elasticsearch.

When I found reason it says "Search Guard not initialized (SG11)" and searchguard index not healthy (timeout: truewhich is really strange.

Note: I deployed search guard tarfile using non rootuser.I also installed Kibana logastah and Elasticsearch with search guard on same server. beats on client machines.

Can anyone guide me here please as I am going crazy with this searchguard installation?

--
You received this message because you are subscribed to the Google Groups "Search Guard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/68a3415e-eaee-439d-ad0e-9e2759295549%40googlegroups.com\.
For more options, visit https://groups.google.com/d/optout\.

3

I thought previous post was closed hence I opened a new one . Sorry for this confusion******************************************

When I installed SG+ES bundle for first time I was able to successfully executed sgadmin script without any issues. Once I added Logstash and Kibana and when I tried to start ES I got message as searchguard not initialized SG11.

I tried below steps to fix this issue as per previous blogs but no luck. Can you please help on this?

ES logs when disabled both searchguard SSL and SG*********************************************

./elasticsearch-2.3.4-localhost/bin/elasticsearch

[2016-08-04 18:52:41,343][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel,

CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed

[2016-08-04 18:52:41,437][INFO ][node ] [localhost] version[2.3.4], pid[20117], build[e455fd0/2016-06-30T11:24:31Z]

[2016-08-04 18:52:41,437][INFO ][node ] [localhost] initializing …

[2016-08-04 18:52:41,769][INFO ][com.floragunn.searchguard.ssl.SearchGuardSSLPlugin] Search Guard 2 plugin not available

[2016-08-04 18:52:41,789][INFO ][plugins ] [localhost] modules [reindex, lang-expression, lang-groovy], plugins [search-guard-ssl, kopf, search-

guard-2], sites [kopf]

[2016-08-04 18:52:41,803][INFO ][env ] [localhost] using [1] data paths, mounts [[/home (/dev/mapper/vg_sda-lv_home)]], net usable_space [8.9gb],

net total_space [9.7gb], spins? [possibly], types [ext4]

[2016-08-04 18:52:41,803][INFO ][env ] [localhost] heap size [989.8mb], compressed ordinary object pointers [true]

[2016-08-04 18:52:41,803][WARN ][env ] [localhost] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at

least [65536]

[2016-08-04 18:52:41,830][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK

SSL) because of java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-linux-x86_64, netty-tcnative-linux-x86_64-fedora,

netty-tcnative]

[2016-08-04 18:52:42,011][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Config directory is /home/test/test/elasticsearch-2.3.4-localhost/config/, from

there the key- and truststore files are resolved relatively

Exception in thread “main” ElasticsearchException[searchguard.ssl.transport.keystore_filepath must be set if transport ssl is reqested.]

at com.floragunn.searchguard.ssl.SearchGuardKeyStore.initSSLConfig(SearchGuardKeyStore.java:188)

at com.floragunn.searchguard.ssl.SearchGuardKeyStore.(SearchGuardKeyStore.java:139)

at com.floragunn.searchguard.ssl.SearchGuardSSLModule.(SearchGuardSSLModule.java:29)

at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.nodeModules(SearchGuardSSLPlugin.java:126)

at org.elasticsearch.plugins.PluginsService.nodeModules(PluginsService.java:263)

at org.elasticsearch.node.Node.(Node.java:179)

at org.elasticsearch.node.Node.(Node.java:140)

at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:143)

at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178)

at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)

at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)

Refer to the log for complete error details.

···

ES logs after uncommenting SG SSL only. Restarted ES and below are logs

./elasticsearch-2.3.4-localhost/bin/elasticsearch

[2016-08-04 18:59:00,447][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel,

CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed

[2016-08-04 18:59:00,554][INFO ][node ] [localhost] version[2.3.4], pid[20193], build[e455fd0/2016-06-30T11:24:31Z]

[2016-08-04 18:59:00,554][INFO ][node ] [localhost] initializing …

[2016-08-04 18:59:00,892][INFO ][com.floragunn.searchguard.ssl.SearchGuardSSLPlugin] Search Guard 2 plugin also available

[2016-08-04 18:59:00,913][INFO ][plugins ] [localhost] modules [reindex, lang-expression, lang-groovy], plugins [search-guard-ssl, kopf, search-

guard-2], sites [kopf]

[2016-08-04 18:59:00,927][INFO ][env ] [localhost] using [1] data paths, mounts [[/home (/dev/mapper/vg_sda-lv_home)]], net usable_space [8.9gb],

net total_space [9.7gb], spins? [possibly], types [ext4]

[2016-08-04 18:59:00,927][INFO ][env ] [localhost] heap size [989.8mb], compressed ordinary object pointers [true]

[2016-08-04 18:59:00,927][WARN ][env ] [localhost] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at

least [65536]

[2016-08-04 18:59:00,954][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK

SSL) because of java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-linux-x86_64, netty-tcnative-linux-x86_64-fedora,

netty-tcnative]

[2016-08-04 18:59:00,954][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK

SSL) because of java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-linux-x86_64, netty-tcnative-linux-x86_64-fedora,

netty-tcnative]

[2016-08-04 18:59:01,114][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Config directory is /home/test/test/elasticsearch-2.3.4-localhost/config/, from

there the key- and truststore files are resolved relatively

Exception in thread “main” ElasticsearchException[searchguard.ssl.transport.keystore_filepath must be set if transport ssl is reqested.]

at com.floragunn.searchguard.ssl.SearchGuardKeyStore.initSSLConfig(SearchGuardKeyStore.java:188)

at com.floragunn.searchguard.ssl.SearchGuardKeyStore.(SearchGuardKeyStore.java:139)

at com.floragunn.searchguard.ssl.SearchGuardSSLModule.(SearchGuardSSLModule.java:29)

at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.nodeModules(SearchGuardSSLPlugin.java:126)

at org.elasticsearch.plugins.PluginsService.nodeModules(PluginsService.java:263)

at org.elasticsearch.node.Node.(Node.java:179)

at org.elasticsearch.node.Node.(Node.java:140)

at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:143)

at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178)

at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)

at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)

Refer to the log for complete error details.

ES logs after SG . Restarted ES and below are logs****************************************

./elasticsearch-2.3.4-localhost/bin/elasticsearch

[2016-08-04 19:04:26,014][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel,

CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed

[2016-08-04 19:04:26,112][INFO ][node ] [localhost] version[2.3.4], pid[20249], build[e455fd0/2016-06-30T11:24:31Z]

[2016-08-04 19:04:26,112][INFO ][node ] [localhost] initializing …

[2016-08-04 19:04:26,444][INFO ][com.floragunn.searchguard.ssl.SearchGuardSSLPlugin] Search Guard 2 plugin also available

[2016-08-04 19:04:26,464][INFO ][plugins ] [localhost] modules [reindex, lang-expression, lang-groovy], plugins [search-guard-ssl, kopf, search-

guard-2], sites [kopf]

[2016-08-04 19:04:26,479][INFO ][env ] [localhost] using [1] data paths, mounts [[/home (/dev/mapper/vg_sda-lv_home)]], net usable_space [8.9gb],

net total_space [9.7gb], spins? [possibly], types [ext4]

[2016-08-04 19:04:26,479][INFO ][env ] [localhost] heap size [989.8mb], compressed ordinary object pointers [true]

[2016-08-04 19:04:26,479][WARN ][env ] [localhost] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at

least [65536]

[2016-08-04 19:04:26,506][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK

SSL) because of java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-linux-x86_64, netty-tcnative-linux-x86_64-fedora,

netty-tcnative]

[2016-08-04 19:04:26,506][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK

SSL) because of java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-linux-x86_64, netty-tcnative-linux-x86_64-fedora,

netty-tcnative]

[2016-08-04 19:04:26,661][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Config directory is /home/test/test/elasticsearch-2.3.4-localhost/config/, from

there the key- and truststore files are resolved relatively

[2016-08-04 19:04:26,674][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] HTTPS client auth mode OPTIONAL

[2016-08-04 19:04:26,678][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] AES-256 not supported, max key length for AES is 128 bit… That is not an issue, it

just limits possible encryption strength. To enable AES 256 install ‘Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files’

[2016-08-04 19:04:26,678][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslTransportClientProvider:JDK with ciphers

[TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,

TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,

TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]

[2016-08-04 19:04:26,678][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslTransportServerProvider:JDK with ciphers

[TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,

TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,

TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]

[2016-08-04 19:04:26,678][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslHTTPProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,

TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,

TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,

TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]

[2016-08-04 19:04:26,678][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslTransport protocols [TLSv1.2, TLSv1.1]

[2016-08-04 19:04:26,678][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslHTTP protocols [TLSv1.2, TLSv1.1]

[2016-08-04 19:04:26,785][INFO ][http ] [localhost] Using [org.elasticsearch.http.netty.NettyHttpServerTransport] as http transport, overridden by

[search-guard2]

[2016-08-04 19:04:26,831][INFO ][com.floragunn.searchguard.configuration.ConfigurationModule] FLS/DLS valve bound

Search Guard Audit Log is not free software

for commercial use in production.

You have to obtain a license if you

use it in production.

[2016-08-04 19:04:26,833][INFO ][com.floragunn.searchguard.auditlog.AuditLogModule] Auditlog available (AuditLogImpl)

[2016-08-04 19:04:26,874][INFO ][transport ] [localhost] Using [com.floragunn.searchguard.transport.SearchGuardTransportService] as transport service,

overridden by [search-guard2]

[2016-08-04 19:04:26,874][INFO ][transport ] [localhost] Using [com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] as transport,

overridden by [search-guard-ssl]

[2016-08-04 19:04:27,340][INFO ][com.floragunn.searchguard.auditlog.impl.AuditLogImpl] Audit Log class: ESAuditLog

[2016-08-04 19:04:27,755][INFO ][node ] [localhost] initialized

[2016-08-04 19:04:27,755][INFO ][node ] [localhost] starting …

[2016-08-04 19:04:27,816][INFO ][com.floragunn.searchguard.transport.SearchGuardTransportService] [localhost] publish_address {127.0.0.1:9301}, bound_addresses

{[::1]:9301}, {127.0.0.1:9301}

[2016-08-04 19:04:27,820][INFO ][discovery ] [localhost] elasticsearch/14mtfOfFR-yfWh3ZCijeqw

[2016-08-04 19:04:27,823][DEBUG][action.admin.cluster.health] [localhost] no known master node, scheduling a retry

[2016-08-04 19:04:31,080][INFO ][cluster.service ] [localhost] detected_master {localhost}{ux5kIQD6QZCseHxfpN7vcQ}{127.0.0.1}{127.0.0.1:9300}, added

{{localhost}{ux5kIQD6QZCseHxfpN7vcQ}{127.0.0.1}{127.0.0.1:9300},}, reason: zen-disco-receive(from master [{localhost}{ux5kIQD6QZCseHxfpN7vcQ}{127.0.0.1}

{127.0.0.1:9300}])

[2016-08-04 19:04:31,306][INFO ][http ] [localhost] publish_address {127.0.0.1:9201}, bound_addresses {[::1]:9201}, {127.0.0.1:9201}

[2016-08-04 19:04:31,307][INFO ][node ] [localhost] started

Searchguard DLS/FLS(+) Security is not free software

for commercial use in production.

You have to obtain a license if you

use it in production.

(+) Document-/Fieldlevel

[2016-08-04 19:04:31,437][INFO ][com.floragunn.searchguard.configuration.SearchGuardIndexSearcherWrapperModule] FLS/DLS enabled

[2016-08-04 19:05:01,288][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:05:34,289][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:06:07,291][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:06:40,292][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:07:13,293][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

Finally started sgadmin and it throws timeout error error

./sgadmin.sh

Connect to localhost:9300

Cluster state timeout

Logs in ES after starting sgadmin

[2016-08-04 19:13:08,012][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:10,530][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:13,046][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:14,567][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:13:15,564][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:16,304][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:13:18,075][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:20,593][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:23,104][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:25,620][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:28,133][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:30,649][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:33,164][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:35,697][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:38,212][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:40,728][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:43,244][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:45,760][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:47,568][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:13:48,274][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:49,305][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:13:50,790][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:53,305][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:55,821][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:58,333][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:00,850][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:03,363][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:05,881][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:08,393][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:10,923][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:13,437][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:15,952][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:18,467][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:20,568][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:14:20,983][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:22,306][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:14:23,498][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:26,014][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:28,530][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:31,045][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:33,628][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialize

My Infradetails : Logstash, Kibana, SG+ES bundle installed in one server and installed filebeats on client server

Note: I am going with default setup ad per searchguardbundle for testing phase I didnot create any new admin certificates

ES configuration

======================== Elasticsearch Configuration =========================

NOTE: Elasticsearch comes with reasonable defaults for most settings.

Before you set out to tweak and tune the configuration, make sure you

understand what are you trying to accomplish and the consequences.

The primary way of configuring a node is via this file. This template lists

the most important settings you may want to configure for a production cluster.

Please see the documentation for further information on configuration options:

http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html

---------------------------------- Cluster -----------------------------------

Use a descriptive name for your cluster:

cluster.name: my-application

------------------------------------ Node ------------------------------------

Use a descriptive name for the node:

node.name: node-1

Add custom attributes to the node:

node.rack: r1

----------------------------------- Paths ------------------------------------

Path to directory where to store the data (separate multiple locations by comma):

path.data: /path/to/data

Path to log files:

path.logs: /path/to/logs

----------------------------------- Memory -----------------------------------

Lock the memory on startup:

bootstrap.mlockall: true

Make sure that the ES_HEAP_SIZE environment variable is set to about half the memory

available on the system and that the owner of the process is allowed to use this limit.

Elasticsearch performs poorly when the system is swapping the memory.

---------------------------------- Network -----------------------------------

Set the bind address to a specific IP (IPv4 or IPv6):

#network.host: x.x.x.x

Set a custom port for HTTP:

#http.port: 9200

For more information, see the documentation at:

http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html

--------------------------------- Discovery ----------------------------------

Pass an initial list of hosts to perform discovery when new node is started:

The default list of hosts is [“127.0.0.1”, “[::1]”]

discovery.zen.ping.unicast.hosts: [“host1”, “host2”]

Prevent the “split brain” by configuring the majority of nodes (total number of nodes / 2 + 1):

discovery.zen.minimum_master_nodes: 3

For more information, see the documentation at:

http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html

---------------------------------- Gateway -----------------------------------

Block initial recovery after a full cluster restart until N nodes are started:

gateway.recover_after_nodes: 3

For more information, see the documentation at:

http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-gateway.html

---------------------------------- Various -----------------------------------

Disable starting multiple nodes on a single system:

node.max_local_storage_nodes: 1

Require explicit names when deleting indices:

action.destructive_requires_name: true

node.name: localhost

##################################################

Search Guard 2 configuration

#Host: localhost

#Generated: Sun Jul 31 17:50:58 UTC 2016

#Git Hash: 51fced7dba388267d30ca7826cebcba8fb9edfb8

#ES-Version: 2.3.4

#SG-Version: 2.3.4.4

#SGSSL-Version: 2.3.4.14

#NettyNative-Version: 1.1.33.Fork17

#CA_PASS: f3c45ab4d999bc1f0ad8967ff9b42e39980eec58

#CL_ADM_PASS: 553281dad7802f1d6018

#CL_DEMOUSER_PASS: d88242ef83b1c5e48e0e

##################################################

searchguard.ssl.transport.enabled: true

searchguard.ssl.transport.keystore_filepath: CN=localhost-keystore.jks

searchguard.ssl.transport.keystore_password: xxxx

searchguard.ssl.transport.truststore_filepath: truststore.jks

searchguard.ssl.transport.truststore_password: xxxx

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.http.enabled: true

searchguard.ssl.http.keystore_filepath: CN=localhost-keystore.jks

searchguard.ssl.http.keystore_password: xxxxx

searchguard.ssl.http.truststore_filepath: truststore.jks

searchguard.ssl.http.truststore_password: xxxx

searchguard.kerberos.krb5_filepath: /Users/temp/kerberos_ldap_environment/krb5.conf

searchguard.kerberos.acceptor_keytab_filepath: http_srv.keytab

searchguard.audit.type: internal_elasticsearch

searchguard.authcz.admin_dn:

  • CN=sgadmin

On Friday, August 5, 2016 at 1:20:11 AM UTC+5:30, SG wrote:

pls do not cross post, this one is already logged here https://github.com/floragunncom/search-guard/issues/182#issuecomment-237533599

Can you please provide error messages and your elasticsearch log files as well as your configuration? Otherwise its hard to help.

Start easy just with the Search Guard bundle and curl. Once you initialized Search Guard by runiing sgadmin you normally should not see a “Search Guard not initialized (SG11)” afterwards.

Am 04.08.2016 um 15:58 schrieb rocky rocky....@gmail.com:

Hi All,

Can anyone help me with the below issue I am facing

I used tarfile which was said to be the easiest way to installtion searchguard which also includes elasticsearch package. Please refer to below link

https://github.com/floragunncom/search-guard/wiki/Search-Guard-Bundle.

After extracting tarfile I have started elasticsearch using the steps as mentioned in link and later ran sgadmin.sh script and evrything was looking great.

Once I have installed Kibana and Logstash as root user , installed beats on client machine . Currently beats are able to push logs to logstash whereas logstash is not able to comminucate to elasticsearch.

When I found reason it says “Search Guard not initialized (SG11)” and searchguard index not healthy (timeout: truewhich is really strange.

Note: I deployed search guard tarfile using non rootuser.I also installed Kibana logastah and Elasticsearch with search guard on same server. beats on client machines.

Can anyone guide me here please as I am going crazy with this searchguard installation?

–
You received this message because you are subscribed to the Google Groups “Search Guard” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/68a3415e-eaee-439d-ad0e-9e2759295549%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

4

@SG.

Currently I tried to run sgadmin successfully but still I am not able to push logs from logstash to elasticsearch.

My architecture flow is beats(clientmachine) → Logstash → Elasticsearch

Note: Logstash, Kibana are installed with root user and searchguard bundle +ES with non-root user…

Please see additional details which can help you to fix my issue

************logstash configuration ***************************************************************

input {

beats {

port => “5044”

ssl => true

ssl_certificate => “/etc/pki/tls/certs/xxx”

ssl_key => “/etc/pki/tls/private/xxx”

}

}

filter {

if [type] == “syslog” {

grok {

match => { “message” => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?:

%{GREEDYDATA:syslog_message}" }

add_field => [ “received_at”, “%{@timestamp}” ]

add_field => [ “received_from”, “%{host}” ]

}

syslog_pri { }

date {

match => [ “syslog_timestamp”, “MMM d HH:mm:ss”, “MMM dd HH:mm:ss” ]

}

}

}

output {

elasticsearch {

hosts => [“localhost:9200”]

user => “logstash”

password => “xxxx”

ssl => true

ssl_certificate_verification => true

truststore => “/home/test/test/elasticsearch-2.3.4-localhost/config/truststore.jks”

truststore_password => “xxxx”

manage_template => false

index => “%{[@metadata][beat][filebeat]}-%{+YYYY.MM.dd}”

document_type => “%{[@metadata][type]}”

}

}

Logstash errors

{:timestamp=>“2016-08-05T02:56:20.048000-0500”, :message=>"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid

certification path to requested target", :class=>“Manticore::ClientProtocolException”, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-

java/lib/manticore/response.rb:37:in initialize'", "org/jruby/RubyProc.java:281:in call’", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-

java/lib/manticore/response.rb:79:in call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:256:in call_once’",

“/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:153:in `code’”, "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-

transport-1.0.18/lib/elasticsearch/transport/transport/http/manticore.rb:84:in perform_request'", "org/jruby/RubyProc.java:281:in call’",

“/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/base.rb:257:in `perform_request’”,

“/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request’”,

“/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/client.rb:128:in `perform_request’”,

“/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.18/lib/elasticsearch/api/actions/bulk.rb:90:in `bulk’”,

“/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:53:in `non_threadsafe_bulk’”,

“/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk’”,

“org/jruby/ext/thread/Mutex.java:149:in `synchronize’”, "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-

java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk’", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-

java/lib/logstash/outputs/elasticsearch/common.rb:172:in `safe_bulk’", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-

java/lib/logstash/outputs/elasticsearch/common.rb:101:in `submit’", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-

java/lib/logstash/outputs/elasticsearch/common.rb:86:in `retrying_submit’", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-

java/lib/logstash/outputs/elasticsearch/common.rb:29:in multi_receive'", "org/jruby/RubyArray.java:1653:in each_slice’",

“/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:28:in `multi_receive’”,

“/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/output_delegator.rb:130:in `worker_multi_receive’”,

“/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/output_delegator.rb:114:in `multi_receive’”,

“/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:301:in output_batch'", "org/jruby/RubyHash.java:1342:in each’”,

“/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:301:in `output_batch’”,

“/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:232:in `worker_loop’”,

“/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:201:in `start_workers’”], :level=>:warn}

ES log errors

2016-08-05 03:06:19,735][ERROR][com.floragunn.searchguard.http.SearchGuardHttpServerTransport] [localhost] SSL Problem Received fatal alert: certificate_unknown

javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)

at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)

at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634)

at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800)

at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083)

at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)

at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)

at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)

at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1218)

at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:852)

at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)

at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)

at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)

at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)

at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)

at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)

at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)

at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)

at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)

at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)

at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

···

On Friday, August 5, 2016 at 6:02:05 AM UTC+5:30, rocky wrote:

I thought previous post was closed hence I opened a new one . Sorry for this confusion******************************************

When I installed SG+ES bundle for first time I was able to successfully executed sgadmin script without any issues. Once I added Logstash and Kibana and when I tried to start ES I got message as searchguard not initialized SG11.

I tried below steps to fix this issue as per previous blogs but no luck. Can you please help on this?

ES logs when disabled both searchguard SSL and SG*********************************************

./elasticsearch-2.3.4-localhost/bin/elasticsearch

[2016-08-04 18:52:41,343][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel,

CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed

[2016-08-04 18:52:41,437][INFO ][node ] [localhost] version[2.3.4], pid[20117], build[e455fd0/2016-06-30T11:24:31Z]

[2016-08-04 18:52:41,437][INFO ][node ] [localhost] initializing …

[2016-08-04 18:52:41,769][INFO ][com.floragunn.searchguard.ssl.SearchGuardSSLPlugin] Search Guard 2 plugin not available

[2016-08-04 18:52:41,789][INFO ][plugins ] [localhost] modules [reindex, lang-expression, lang-groovy], plugins [search-guard-ssl, kopf, search-

guard-2], sites [kopf]

[2016-08-04 18:52:41,803][INFO ][env ] [localhost] using [1] data paths, mounts [[/home (/dev/mapper/vg_sda-lv_home)]], net usable_space [8.9gb],

net total_space [9.7gb], spins? [possibly], types [ext4]

[2016-08-04 18:52:41,803][INFO ][env ] [localhost] heap size [989.8mb], compressed ordinary object pointers [true]

[2016-08-04 18:52:41,803][WARN ][env ] [localhost] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at

least [65536]

[2016-08-04 18:52:41,830][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK

SSL) because of java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-linux-x86_64, netty-tcnative-linux-x86_64-fedora,

netty-tcnative]

[2016-08-04 18:52:42,011][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Config directory is /home/test/test/elasticsearch-2.3.4-localhost/config/, from

there the key- and truststore files are resolved relatively

Exception in thread “main” ElasticsearchException[searchguard.ssl.transport.keystore_filepath must be set if transport ssl is reqested.]

at com.floragunn.searchguard.ssl.SearchGuardKeyStore.initSSLConfig(SearchGuardKeyStore.java:188)

at com.floragunn.searchguard.ssl.SearchGuardKeyStore.(SearchGuardKeyStore.java:139)

at com.floragunn.searchguard.ssl.SearchGuardSSLModule.(SearchGuardSSLModule.java:29)

at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.nodeModules(SearchGuardSSLPlugin.java:126)

at org.elasticsearch.plugins.PluginsService.nodeModules(PluginsService.java:263)

at org.elasticsearch.node.Node.(Node.java:179)

at org.elasticsearch.node.Node.(Node.java:140)

at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:143)

at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178)

at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)

at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)

Refer to the log for complete error details.

ES logs after uncommenting SG SSL only. Restarted ES and below are logs

./elasticsearch-2.3.4-localhost/bin/elasticsearch

[2016-08-04 18:59:00,447][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel,

CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed

[2016-08-04 18:59:00,554][INFO ][node ] [localhost] version[2.3.4], pid[20193], build[e455fd0/2016-06-30T11:24:31Z]

[2016-08-04 18:59:00,554][INFO ][node ] [localhost] initializing …

[2016-08-04 18:59:00,892][INFO ][com.floragunn.searchguard.ssl.SearchGuardSSLPlugin] Search Guard 2 plugin also available

[2016-08-04 18:59:00,913][INFO ][plugins ] [localhost] modules [reindex, lang-expression, lang-groovy], plugins [search-guard-ssl, kopf, search-

guard-2], sites [kopf]

[2016-08-04 18:59:00,927][INFO ][env ] [localhost] using [1] data paths, mounts [[/home (/dev/mapper/vg_sda-lv_home)]], net usable_space [8.9gb],

net total_space [9.7gb], spins? [possibly], types [ext4]

[2016-08-04 18:59:00,927][INFO ][env ] [localhost] heap size [989.8mb], compressed ordinary object pointers [true]

[2016-08-04 18:59:00,927][WARN ][env ] [localhost] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at

least [65536]

[2016-08-04 18:59:00,954][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK

SSL) because of java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-linux-x86_64, netty-tcnative-linux-x86_64-fedora,

netty-tcnative]

[2016-08-04 18:59:00,954][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK

SSL) because of java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-linux-x86_64, netty-tcnative-linux-x86_64-fedora,

netty-tcnative]

[2016-08-04 18:59:01,114][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Config directory is /home/test/test/elasticsearch-2.3.4-localhost/config/, from

there the key- and truststore files are resolved relatively

Exception in thread “main” ElasticsearchException[searchguard.ssl.transport.keystore_filepath must be set if transport ssl is reqested.]

at com.floragunn.searchguard.ssl.SearchGuardKeyStore.initSSLConfig(SearchGuardKeyStore.java:188)

at com.floragunn.searchguard.ssl.SearchGuardKeyStore.(SearchGuardKeyStore.java:139)

at com.floragunn.searchguard.ssl.SearchGuardSSLModule.(SearchGuardSSLModule.java:29)

at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.nodeModules(SearchGuardSSLPlugin.java:126)

at org.elasticsearch.plugins.PluginsService.nodeModules(PluginsService.java:263)

at org.elasticsearch.node.Node.(Node.java:179)

at org.elasticsearch.node.Node.(Node.java:140)

at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:143)

at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178)

at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)

at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)

Refer to the log for complete error details.

ES logs after SG . Restarted ES and below are logs****************************************

./elasticsearch-2.3.4-localhost/bin/elasticsearch

[2016-08-04 19:04:26,014][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel,

CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed

[2016-08-04 19:04:26,112][INFO ][node ] [localhost] version[2.3.4], pid[20249], build[e455fd0/2016-06-30T11:24:31Z]

[2016-08-04 19:04:26,112][INFO ][node ] [localhost] initializing …

[2016-08-04 19:04:26,444][INFO ][com.floragunn.searchguard.ssl.SearchGuardSSLPlugin] Search Guard 2 plugin also available

[2016-08-04 19:04:26,464][INFO ][plugins ] [localhost] modules [reindex, lang-expression, lang-groovy], plugins [search-guard-ssl, kopf, search-

guard-2], sites [kopf]

[2016-08-04 19:04:26,479][INFO ][env ] [localhost] using [1] data paths, mounts [[/home (/dev/mapper/vg_sda-lv_home)]], net usable_space [8.9gb],

net total_space [9.7gb], spins? [possibly], types [ext4]

[2016-08-04 19:04:26,479][INFO ][env ] [localhost] heap size [989.8mb], compressed ordinary object pointers [true]

[2016-08-04 19:04:26,479][WARN ][env ] [localhost] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at

least [65536]

[2016-08-04 19:04:26,506][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK

SSL) because of java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-linux-x86_64, netty-tcnative-linux-x86_64-fedora,

netty-tcnative]

[2016-08-04 19:04:26,506][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK

SSL) because of java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-linux-x86_64, netty-tcnative-linux-x86_64-fedora,

netty-tcnative]

[2016-08-04 19:04:26,661][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Config directory is /home/test/test/elasticsearch-2.3.4-localhost/config/, from

there the key- and truststore files are resolved relatively

[2016-08-04 19:04:26,674][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] HTTPS client auth mode OPTIONAL

[2016-08-04 19:04:26,678][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] AES-256 not supported, max key length for AES is 128 bit… That is not an issue, it

just limits possible encryption strength. To enable AES 256 install ‘Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files’

[2016-08-04 19:04:26,678][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslTransportClientProvider:JDK with ciphers

[TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,

TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,

TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]

[2016-08-04 19:04:26,678][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslTransportServerProvider:JDK with ciphers

[TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,

TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,

TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]

[2016-08-04 19:04:26,678][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslHTTPProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,

TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,

TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,

TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]

[2016-08-04 19:04:26,678][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslTransport protocols [TLSv1.2, TLSv1.1]

[2016-08-04 19:04:26,678][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslHTTP protocols [TLSv1.2, TLSv1.1]

[2016-08-04 19:04:26,785][INFO ][http ] [localhost] Using [org.elasticsearch.http.netty.NettyHttpServerTransport] as http transport, overridden by

[search-guard2]

[2016-08-04 19:04:26,831][INFO ][com.floragunn.searchguard.configuration.ConfigurationModule] FLS/DLS valve bound

Search Guard Audit Log is not free software

for commercial use in production.

You have to obtain a license if you

use it in production.

[2016-08-04 19:04:26,833][INFO ][com.floragunn.searchguard.auditlog.AuditLogModule] Auditlog available (AuditLogImpl)

[2016-08-04 19:04:26,874][INFO ][transport ] [localhost] Using [com.floragunn.searchguard.transport.SearchGuardTransportService] as transport service,

overridden by [search-guard2]

[2016-08-04 19:04:26,874][INFO ][transport ] [localhost] Using [com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] as transport,

overridden by [search-guard-ssl]

[2016-08-04 19:04:27,340][INFO ][com.floragunn.searchguard.auditlog.impl.AuditLogImpl] Audit Log class: ESAuditLog

[2016-08-04 19:04:27,755][INFO ][node ] [localhost] initialized

[2016-08-04 19:04:27,755][INFO ][node ] [localhost] starting …

[2016-08-04 19:04:27,816][INFO ][com.floragunn.searchguard.transport.SearchGuardTransportService] [localhost] publish_address {127.0.0.1:9301}, bound_addresses

{[::1]:9301}, {127.0.0.1:9301}

[2016-08-04 19:04:27,820][INFO ][discovery ] [localhost] elasticsearch/14mtfOfFR-yfWh3ZCijeqw

[2016-08-04 19:04:27,823][DEBUG][action.admin.cluster.health] [localhost] no known master node, scheduling a retry

[2016-08-04 19:04:31,080][INFO ][cluster.service ] [localhost] detected_master {localhost}{ux5kIQD6QZCseHxfpN7vcQ}{127.0.0.1}{127.0.0.1:9300}, added

{{localhost}{ux5kIQD6QZCseHxfpN7vcQ}{127.0.0.1}{127.0.0.1:9300},}, reason: zen-disco-receive(from master [{localhost}{ux5kIQD6QZCseHxfpN7vcQ}{127.0.0.1}

{127.0.0.1:9300}])

[2016-08-04 19:04:31,306][INFO ][http ] [localhost] publish_address {127.0.0.1:9201}, bound_addresses {[::1]:9201}, {127.0.0.1:9201}

[2016-08-04 19:04:31,307][INFO ][node ] [localhost] started

Searchguard DLS/FLS(+) Security is not free software

for commercial use in production.

You have to obtain a license if you

use it in production.

(+) Document-/Fieldlevel

[2016-08-04 19:04:31,437][INFO ][com.floragunn.searchguard.configuration.SearchGuardIndexSearcherWrapperModule] FLS/DLS enabled

[2016-08-04 19:05:01,288][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:05:34,289][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:06:07,291][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:06:40,292][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:07:13,293][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

Finally started sgadmin and it throws timeout error error

./sgadmin.sh

Connect to localhost:9300

Cluster state timeout

Logs in ES after starting sgadmin

[2016-08-04 19:13:08,012][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:10,530][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:13,046][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:14,567][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:13:15,564][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:16,304][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:13:18,075][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:20,593][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:23,104][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:25,620][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:28,133][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:30,649][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:33,164][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:35,697][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:38,212][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:40,728][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:43,244][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:45,760][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:47,568][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:13:48,274][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:49,305][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:13:50,790][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:53,305][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:55,821][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:58,333][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:00,850][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:03,363][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:05,881][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:08,393][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:10,923][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:13,437][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:15,952][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:18,467][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:20,568][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:14:20,983][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:22,306][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:14:23,498][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:26,014][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:28,530][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:31,045][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:33,628][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialize

My Infradetails : Logstash, Kibana, SG+ES bundle installed in one server and installed filebeats on client server

Note: I am going with default setup ad per searchguardbundle for testing phase I didnot create any new admin certificates

ES configuration

======================== Elasticsearch Configuration =========================

NOTE: Elasticsearch comes with reasonable defaults for most settings.

Before you set out to tweak and tune the configuration, make sure you

understand what are you trying to accomplish and the consequences.

The primary way of configuring a node is via this file. This template lists

the most important settings you may want to configure for a production cluster.

Please see the documentation for further information on configuration options:

<http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html>

---------------------------------- Cluster -----------------------------------

Use a descriptive name for your cluster:

cluster.name: my-application

------------------------------------ Node ------------------------------------

Use a descriptive name for the node:

node.name: node-1

Add custom attributes to the node:

node.rack: r1

----------------------------------- Paths ------------------------------------

Path to directory where to store the data (separate multiple locations by comma):

path.data: /path/to/data

Path to log files:

path.logs: /path/to/logs

----------------------------------- Memory -----------------------------------

Lock the memory on startup:

bootstrap.mlockall: true

Make sure that the ES_HEAP_SIZE environment variable is set to about half the memory

available on the system and that the owner of the process is allowed to use this limit.

Elasticsearch performs poorly when the system is swapping the memory.

---------------------------------- Network -----------------------------------

Set the bind address to a specific IP (IPv4 or IPv6):

#network.host: x.x.x.x

Set a custom port for HTTP:

#http.port: 9200

For more information, see the documentation at:

<http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html>

--------------------------------- Discovery ----------------------------------

Pass an initial list of hosts to perform discovery when new node is started:

The default list of hosts is [“127.0.0.1”, “[::1]”]

discovery.zen.ping.unicast.hosts: [“host1”, “host2”]

Prevent the “split brain” by configuring the majority of nodes (total number of nodes / 2 + 1):

discovery.zen.minimum_master_nodes: 3

For more information, see the documentation at:

<http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html>

---------------------------------- Gateway -----------------------------------

Block initial recovery after a full cluster restart until N nodes are started:

gateway.recover_after_nodes: 3

For more information, see the documentation at:

<http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-gateway.html>

---------------------------------- Various -----------------------------------

Disable starting multiple nodes on a single system:

node.max_local_storage_nodes: 1

Require explicit names when deleting indices:

action.destructive_requires_name: true

node.name: localhost

##################################################

Search Guard 2 configuration

#Host: localhost

#Generated: Sun Jul 31 17:50:58 UTC 2016

#Git Hash: 51fced7dba388267d30ca7826cebcba8fb9edfb8

#ES-Version: 2.3.4

#SG-Version: 2.3.4.4

#SGSSL-Version: 2.3.4.14

#NettyNative-Version: 1.1.33.Fork17

#CA_PASS: f3c45ab4d999bc1f0ad8967ff9b42e39980eec58

#CL_ADM_PASS: 553281dad7802f1d6018

#CL_DEMOUSER_PASS: d88242ef83b1c5e48e0e

##################################################

searchguard.ssl.transport.enabled: true

searchguard.ssl.transport.keystore_filepath: CN=localhost-keystore.jks

searchguard.ssl.transport.keystore_password: xxxx

searchguard.ssl.transport.truststore_filepath: truststore.jks

searchguard.ssl.transport.truststore_password: xxxx

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.http.enabled: true

searchguard.ssl.http.keystore_filepath: CN=localhost-keystore.jks

searchguard.ssl.http.keystore_password: xxxxx

searchguard.ssl.http.truststore_filepath: truststore.jks

searchguard.ssl.http.truststore_password: xxxx

searchguard.kerberos.krb5_filepath: /Users/temp/kerberos_ldap_environment/krb5.conf

searchguard.kerberos.acceptor_keytab_filepath: http_srv.keytab

searchguard.audit.type: internal_elasticsearch

searchguard.authcz.admin_dn:

  • CN=sgadmin

On Friday, August 5, 2016 at 1:20:11 AM UTC+5:30, SG wrote:

pls do not cross post, this one is already logged here https://github.com/floragunncom/search-guard/issues/182#issuecomment-237533599

Can you please provide error messages and your elasticsearch log files as well as your configuration? Otherwise its hard to help.

Start easy just with the Search Guard bundle and curl. Once you initialized Search Guard by runiing sgadmin you normally should not see a “Search Guard not initialized (SG11)” afterwards.

Am 04.08.2016 um 15:58 schrieb rocky rocky....@gmail.com:

Hi All,

Can anyone help me with the below issue I am facing

I used tarfile which was said to be the easiest way to installtion searchguard which also includes elasticsearch package. Please refer to below link

https://github.com/floragunncom/search-guard/wiki/Search-Guard-Bundle.

After extracting tarfile I have started elasticsearch using the steps as mentioned in link and later ran sgadmin.sh script and evrything was looking great.

Once I have installed Kibana and Logstash as root user , installed beats on client machine . Currently beats are able to push logs to logstash whereas logstash is not able to comminucate to elasticsearch.

When I found reason it says “Search Guard not initialized (SG11)” and searchguard index not healthy (timeout: truewhich is really strange.

Note: I deployed search guard tarfile using non rootuser.I also installed Kibana logastah and Elasticsearch with search guard on same server. beats on client machines.

Can anyone guide me here please as I am going crazy with this searchguard installation?

–
You received this message because you are subscribed to the Google Groups “Search Guard” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/68a3415e-eaee-439d-ad0e-9e2759295549%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

5

Hi all,

Can any on help me in fixing this issue as I have added new users and roles and when I try to run sgadmin.sh script to update users and roles it give below message.

./sgadmin.sh

Connect to localhost:9300

Cluster state timeout

···

On Thursday, August 4, 2016 at 7:28:25 PM UTC+5:30, rocky wrote:

Hi All,
Can anyone help me with the below issue I am facing

I used tarfile which was said to be the easiest way to installtion searchguard which also includes elasticsearch package. Please refer to below link
https://github.com/floragunncom/search-guard/wiki/Search-Guard-Bundle.

After extracting tarfile I have started elasticsearch using the steps as mentioned in link and later ran sgadmin.sh script and evrything was looking great.

Once I have installed Kibana and Logstash as root user , installed beats on client machine . Currently beats are able to push logs to logstash whereas logstash is not able to comminucate to elasticsearch.

When I found reason it says “Search Guard not initialized (SG11)” and searchguard index not healthy (timeout: truewhich is really strange.

Note: I deployed search guard tarfile using non rootuser.I also installed Kibana logastah and Elasticsearch with search guard on same server. beats on client machines.

Can anyone guide me here please as I am going crazy with this searchguard installation?

6

Hi all,

Can any on help me in fixing this issue as I have added new users and roles and when I try to run sgadmin.sh script to update users and roles it give below message.

I am using one node only

./sgadmin.sh

Connect to localhost:9300

Cluster state timeout

7

Can anyone help me as I am not able to update users and roles I am getting below error and please see the elasticsearch logs

./sgadmin.sh

Connect to localhost:9300

Cluster state timeout

tail -10 /opt/elastic/elasticsearch-2.3.4-localhost/logs/elasticsearch.log

UnavailableShardsException[[auditlog][4] primary shard is not active Timeout: [1m], request: [index {[auditlog][auditlog][AVbPiINBVOSyDPZ_5rxd], source[{“audit_remote_address”:“127.0.0.1:48532”,“audit_request_headers”:“”,“audit_request_class”:“class org.elasticsearch.action.admin.cluster.health.ClusterHealthRequest”,“audit_principal”:null,“audit_details”:“null”,“audit_date”:“Sun Aug 28 00:04:05 CDT 2016”,“audit_reason”:“cluster:monitor/health”,“audit_request_context”:“[_sg_ssl_cipher=>TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, _sg_remote_address=>127.0.0.1:48532, _sg_user=>User [name=kibanaserver, roles=], _sg_ssl_protocol=>TLSv1.2]”,“audit_category”:“AUTHENTICATED”,“audit_request_user”:“kibanaserver”}]}]]

at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.retryBecauseUnavailable(TransportReplicationAction.java:596)

at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:465)

at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)

at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onTimeout(TransportReplicationAction.java:558)

at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:236)

at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:804)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

heapsize details

java -XX:+PrintFlagsFinal -version | grep -iE ‘HeapSize|PermSize|ThreadStackSize’

intx CompilerThreadStackSize = 0 {pd product}

uintx ErgoHeapSizeLimit = 0 {product}

uintx HeapSizePerGCThread = 87241520 {product}

uintx InitialHeapSize := 130023424 {product}

uintx LargePageHeapSizeThreshold = 134217728 {product}

uintx MaxHeapSize := 2053111808 {product}

intx ThreadStackSize = 1024 {pd product}

intx VMThreadStackSize = 1024 {pd product}

java version “1.8.0_73”

Java™ SE Runtime Environment (build 1.8.0_73-b02)

Java HotSpot™ 64-Bit Server VM (build 25.73-b02, mixed mode)

8

How to resolve this question? My es cluster has 5 nodes, and met this question, too.

在 2016年8月5日星期五 UTC+8下午4:17:36，rocky写道：

···

@SG.

Currently I tried to run sgadmin successfully but still I am not able to push logs from logstash to elasticsearch.

My architecture flow is beats(clientmachine) → Logstash → Elasticsearch

Note: Logstash, Kibana are installed with root user and searchguard bundle +ES with non-root user…

Please see additional details which can help you to fix my issue

************logstash configuration ***************************************************************

input {

beats {

port => “5044”

ssl => true

ssl_certificate => “/etc/pki/tls/certs/xxx”

ssl_key => “/etc/pki/tls/private/xxx”

}

}

filter {

if [type] == “syslog” {

grok {

match => { “message” => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?:

%{GREEDYDATA:syslog_message}" }

add_field => [ “received_at”, “%{@timestamp}” ]

add_field => [ “received_from”, “%{host}” ]

}

syslog_pri { }

date {

match => [ “syslog_timestamp”, “MMM d HH:mm:ss”, “MMM dd HH:mm:ss” ]

}

}

}

output {

elasticsearch {

hosts => [“localhost:9200”]

user => “logstash”

password => “xxxx”

ssl => true

ssl_certificate_verification => true

truststore => “/home/test/test/elasticsearch-2.3.4-localhost/config/truststore.jks”

truststore_password => “xxxx”

manage_template => false

index => “%{[@metadata][beat][filebeat]}-%{+YYYY.MM.dd}”

document_type => “%{[@metadata][type]}”

}

}

Logstash errors

{:timestamp=>“2016-08-05T02:56:20.048000-0500”, :message=>"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid

certification path to requested target", :class=>“Manticore::ClientProtocolException”, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-

java/lib/manticore/response.rb:37:in initialize'", "org/jruby/RubyProc.java:281:in call’", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-

java/lib/manticore/response.rb:79:in call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:256:in call_once’",

“/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:153:in `code’”, "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-

transport-1.0.18/lib/elasticsearch/transport/transport/http/manticore.rb:84:in perform_request'", "org/jruby/RubyProc.java:281:in call’",

“/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/base.rb:257:in `perform_request’”,

“/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request’”,

“/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/client.rb:128:in `perform_request’”,

“/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.18/lib/elasticsearch/api/actions/bulk.rb:90:in `bulk’”,

“/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:53:in `non_threadsafe_bulk’”,

“/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk’”,

“org/jruby/ext/thread/Mutex.java:149:in `synchronize’”, "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-

java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk’", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-

java/lib/logstash/outputs/elasticsearch/common.rb:172:in `safe_bulk’", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-

java/lib/logstash/outputs/elasticsearch/common.rb:101:in `submit’", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-

java/lib/logstash/outputs/elasticsearch/common.rb:86:in `retrying_submit’", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-

java/lib/logstash/outputs/elasticsearch/common.rb:29:in multi_receive'", "org/jruby/RubyArray.java:1653:in each_slice’",

“/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:28:in `multi_receive’”,

“/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/output_delegator.rb:130:in `worker_multi_receive’”,

“/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/output_delegator.rb:114:in `multi_receive’”,

“/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:301:in output_batch'", "org/jruby/RubyHash.java:1342:in each’”,

“/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:301:in `output_batch’”,

“/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:232:in `worker_loop’”,

“/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:201:in `start_workers’”], :level=>:warn}

ES log errors

2016-08-05 03:06:19,735][ERROR][com.floragunn.searchguard.http.SearchGuardHttpServerTransport] [localhost] SSL Problem Received fatal alert: certificate_unknown

javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)

at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)

at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634)

at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800)

at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083)

at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)

at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)

at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)

at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1218)

at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:852)

at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)

at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)

at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)

at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)

at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)

at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)

at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)

at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)

at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)

at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)

at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

On Friday, August 5, 2016 at 6:02:05 AM UTC+5:30, rocky wrote:

I thought previous post was closed hence I opened a new one . Sorry for this confusion******************************************

When I installed SG+ES bundle for first time I was able to successfully executed sgadmin script without any issues. Once I added Logstash and Kibana and when I tried to start ES I got message as searchguard not initialized SG11.

I tried below steps to fix this issue as per previous blogs but no luck. Can you please help on this?

ES logs when disabled both searchguard SSL and SG*********************************************

./elasticsearch-2.3.4-localhost/bin/elasticsearch

[2016-08-04 18:52:41,343][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel,

CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed

[2016-08-04 18:52:41,437][INFO ][node ] [localhost] version[2.3.4], pid[20117], build[e455fd0/2016-06-30T11:24:31Z]

[2016-08-04 18:52:41,437][INFO ][node ] [localhost] initializing …

[2016-08-04 18:52:41,769][INFO ][com.floragunn.searchguard.ssl.SearchGuardSSLPlugin] Search Guard 2 plugin not available

[2016-08-04 18:52:41,789][INFO ][plugins ] [localhost] modules [reindex, lang-expression, lang-groovy], plugins [search-guard-ssl, kopf, search-

guard-2], sites [kopf]

[2016-08-04 18:52:41,803][INFO ][env ] [localhost] using [1] data paths, mounts [[/home (/dev/mapper/vg_sda-lv_home)]], net usable_space [8.9gb],

net total_space [9.7gb], spins? [possibly], types [ext4]

[2016-08-04 18:52:41,803][INFO ][env ] [localhost] heap size [989.8mb], compressed ordinary object pointers [true]

[2016-08-04 18:52:41,803][WARN ][env ] [localhost] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at

least [65536]

[2016-08-04 18:52:41,830][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK

SSL) because of java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-linux-x86_64, netty-tcnative-linux-x86_64-fedora,

netty-tcnative]

[2016-08-04 18:52:42,011][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Config directory is /home/test/test/elasticsearch-2.3.4-localhost/config/, from

there the key- and truststore files are resolved relatively

Exception in thread “main” ElasticsearchException[searchguard.ssl.transport.keystore_filepath must be set if transport ssl is reqested.]

at com.floragunn.searchguard.ssl.SearchGuardKeyStore.initSSLConfig(SearchGuardKeyStore.java:188)

at com.floragunn.searchguard.ssl.SearchGuardKeyStore.(SearchGuardKeyStore.java:139)

at com.floragunn.searchguard.ssl.SearchGuardSSLModule.(SearchGuardSSLModule.java:29)

at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.nodeModules(SearchGuardSSLPlugin.java:126)

at org.elasticsearch.plugins.PluginsService.nodeModules(PluginsService.java:263)

at org.elasticsearch.node.Node.(Node.java:179)

at org.elasticsearch.node.Node.(Node.java:140)

at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:143)

at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178)

at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)

at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)

Refer to the log for complete error details.

ES logs after uncommenting SG SSL only. Restarted ES and below are logs

./elasticsearch-2.3.4-localhost/bin/elasticsearch

[2016-08-04 18:59:00,447][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel,

CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed

[2016-08-04 18:59:00,554][INFO ][node ] [localhost] version[2.3.4], pid[20193], build[e455fd0/2016-06-30T11:24:31Z]

[2016-08-04 18:59:00,554][INFO ][node ] [localhost] initializing …

[2016-08-04 18:59:00,892][INFO ][com.floragunn.searchguard.ssl.SearchGuardSSLPlugin] Search Guard 2 plugin also available

[2016-08-04 18:59:00,913][INFO ][plugins ] [localhost] modules [reindex, lang-expression, lang-groovy], plugins [search-guard-ssl, kopf, search-

guard-2], sites [kopf]

[2016-08-04 18:59:00,927][INFO ][env ] [localhost] using [1] data paths, mounts [[/home (/dev/mapper/vg_sda-lv_home)]], net usable_space [8.9gb],

net total_space [9.7gb], spins? [possibly], types [ext4]

[2016-08-04 18:59:00,927][INFO ][env ] [localhost] heap size [989.8mb], compressed ordinary object pointers [true]

[2016-08-04 18:59:00,927][WARN ][env ] [localhost] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at

least [65536]

[2016-08-04 18:59:00,954][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK

SSL) because of java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-linux-x86_64, netty-tcnative-linux-x86_64-fedora,

netty-tcnative]

[2016-08-04 18:59:00,954][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK

SSL) because of java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-linux-x86_64, netty-tcnative-linux-x86_64-fedora,

netty-tcnative]

[2016-08-04 18:59:01,114][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Config directory is /home/test/test/elasticsearch-2.3.4-localhost/config/, from

there the key- and truststore files are resolved relatively

Exception in thread “main” ElasticsearchException[searchguard.ssl.transport.keystore_filepath must be set if transport ssl is reqested.]

at com.floragunn.searchguard.ssl.SearchGuardKeyStore.initSSLConfig(SearchGuardKeyStore.java:188)

at com.floragunn.searchguard.ssl.SearchGuardKeyStore.(SearchGuardKeyStore.java:139)

at com.floragunn.searchguard.ssl.SearchGuardSSLModule.(SearchGuardSSLModule.java:29)

at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.nodeModules(SearchGuardSSLPlugin.java:126)

at org.elasticsearch.plugins.PluginsService.nodeModules(PluginsService.java:263)

at org.elasticsearch.node.Node.(Node.java:179)

at org.elasticsearch.node.Node.(Node.java:140)

at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:143)

at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178)

at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)

at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)

Refer to the log for complete error details.

ES logs after SG . Restarted ES and below are logs****************************************

./elasticsearch-2.3.4-localhost/bin/elasticsearch

[2016-08-04 19:04:26,014][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel,

CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed

[2016-08-04 19:04:26,112][INFO ][node ] [localhost] version[2.3.4], pid[20249], build[e455fd0/2016-06-30T11:24:31Z]

[2016-08-04 19:04:26,112][INFO ][node ] [localhost] initializing …

[2016-08-04 19:04:26,444][INFO ][com.floragunn.searchguard.ssl.SearchGuardSSLPlugin] Search Guard 2 plugin also available

[2016-08-04 19:04:26,464][INFO ][plugins ] [localhost] modules [reindex, lang-expression, lang-groovy], plugins [search-guard-ssl, kopf, search-

guard-2], sites [kopf]

[2016-08-04 19:04:26,479][INFO ][env ] [localhost] using [1] data paths, mounts [[/home (/dev/mapper/vg_sda-lv_home)]], net usable_space [8.9gb],

net total_space [9.7gb], spins? [possibly], types [ext4]

[2016-08-04 19:04:26,479][INFO ][env ] [localhost] heap size [989.8mb], compressed ordinary object pointers [true]

[2016-08-04 19:04:26,479][WARN ][env ] [localhost] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at

least [65536]

[2016-08-04 19:04:26,506][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK

SSL) because of java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-linux-x86_64, netty-tcnative-linux-x86_64-fedora,

netty-tcnative]

[2016-08-04 19:04:26,506][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK

SSL) because of java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-linux-x86_64, netty-tcnative-linux-x86_64-fedora,

netty-tcnative]

[2016-08-04 19:04:26,661][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Config directory is /home/test/test/elasticsearch-2.3.4-localhost/config/, from

there the key- and truststore files are resolved relatively

[2016-08-04 19:04:26,674][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] HTTPS client auth mode OPTIONAL

[2016-08-04 19:04:26,678][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] AES-256 not supported, max key length for AES is 128 bit… That is not an issue, it

just limits possible encryption strength. To enable AES 256 install ‘Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files’

[2016-08-04 19:04:26,678][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslTransportClientProvider:JDK with ciphers

[TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,

TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,

TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]

[2016-08-04 19:04:26,678][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslTransportServerProvider:JDK with ciphers

[TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,

TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,

TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]

[2016-08-04 19:04:26,678][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslHTTPProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,

TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,

TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,

TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]

[2016-08-04 19:04:26,678][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslTransport protocols [TLSv1.2, TLSv1.1]

[2016-08-04 19:04:26,678][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslHTTP protocols [TLSv1.2, TLSv1.1]

[2016-08-04 19:04:26,785][INFO ][http ] [localhost] Using [org.elasticsearch.http.netty.NettyHttpServerTransport] as http transport, overridden by

[search-guard2]

[2016-08-04 19:04:26,831][INFO ][com.floragunn.searchguard.configuration.ConfigurationModule] FLS/DLS valve bound

Search Guard Audit Log is not free software

for commercial use in production.

You have to obtain a license if you

use it in production.

[2016-08-04 19:04:26,833][INFO ][com.floragunn.searchguard.auditlog.AuditLogModule] Auditlog available (AuditLogImpl)

[2016-08-04 19:04:26,874][INFO ][transport ] [localhost] Using [com.floragunn.searchguard.transport.SearchGuardTransportService] as transport service,

overridden by [search-guard2]

[2016-08-04 19:04:26,874][INFO ][transport ] [localhost] Using [com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] as transport,

overridden by [search-guard-ssl]

[2016-08-04 19:04:27,340][INFO ][com.floragunn.searchguard.auditlog.impl.AuditLogImpl] Audit Log class: ESAuditLog

[2016-08-04 19:04:27,755][INFO ][node ] [localhost] initialized

[2016-08-04 19:04:27,755][INFO ][node ] [localhost] starting …

[2016-08-04 19:04:27,816][INFO ][com.floragunn.searchguard.transport.SearchGuardTransportService] [localhost] publish_address {127.0.0.1:9301}, bound_addresses

{[::1]:9301}, {127.0.0.1:9301}

[2016-08-04 19:04:27,820][INFO ][discovery ] [localhost] elasticsearch/14mtfOfFR-yfWh3ZCijeqw

[2016-08-04 19:04:27,823][DEBUG][action.admin.cluster.health] [localhost] no known master node, scheduling a retry

[2016-08-04 19:04:31,080][INFO ][cluster.service ] [localhost] detected_master {localhost}{ux5kIQD6QZCseHxfpN7vcQ}{127.0.0.1}{127.0.0.1:9300}, added

{{localhost}{ux5kIQD6QZCseHxfpN7vcQ}{127.0.0.1}{127.0.0.1:9300},}, reason: zen-disco-receive(from master [{localhost}{ux5kIQD6QZCseHxfpN7vcQ}{127.0.0.1}

{127.0.0.1:9300}])

[2016-08-04 19:04:31,306][INFO ][http ] [localhost] publish_address {127.0.0.1:9201}, bound_addresses {[::1]:9201}, {127.0.0.1:9201}

[2016-08-04 19:04:31,307][INFO ][node ] [localhost] started

Searchguard DLS/FLS(+) Security is not free software

for commercial use in production.

You have to obtain a license if you

use it in production.

(+) Document-/Fieldlevel

[2016-08-04 19:04:31,437][INFO ][com.floragunn.searchguard.configuration.SearchGuardIndexSearcherWrapperModule] FLS/DLS enabled

[2016-08-04 19:05:01,288][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:05:34,289][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:06:07,291][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:06:40,292][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:07:13,293][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

Finally started sgadmin and it throws timeout error error

./sgadmin.sh

Connect to localhost:9300

Cluster state timeout

Logs in ES after starting sgadmin

[2016-08-04 19:13:08,012][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:10,530][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:13,046][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:14,567][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:13:15,564][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:16,304][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:13:18,075][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:20,593][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:23,104][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:25,620][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:28,133][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:30,649][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:33,164][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:35,697][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:38,212][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:40,728][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:43,244][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:45,760][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:47,568][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:13:48,274][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:49,305][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:13:50,790][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:53,305][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:55,821][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:13:58,333][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:00,850][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:03,363][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:05,881][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:08,393][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:10,923][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:13,437][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:15,952][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:18,467][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:20,568][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:14:20,983][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:22,306][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)

[2016-08-04 19:14:23,498][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:26,014][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:28,530][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:31,045][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

[2016-08-04 19:14:33,628][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialize

My Infradetails : Logstash, Kibana, SG+ES bundle installed in one server and installed filebeats on client server

Note: I am going with default setup ad per searchguardbundle for testing phase I didnot create any new admin certificates

ES configuration

======================== Elasticsearch Configuration =========================

NOTE: Elasticsearch comes with reasonable defaults for most settings.

Before you set out to tweak and tune the configuration, make sure you

understand what are you trying to accomplish and the consequences.

The primary way of configuring a node is via this file. This template lists

the most important settings you may want to configure for a production cluster.

Please see the documentation for further information on configuration options:

<http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html>

---------------------------------- Cluster -----------------------------------

Use a descriptive name for your cluster:

cluster.name: my-application

------------------------------------ Node ------------------------------------

Use a descriptive name for the node:

node.name: node-1

Add custom attributes to the node:

node.rack: r1

----------------------------------- Paths ------------------------------------

Path to directory where to store the data (separate multiple locations by comma):

path.data: /path/to/data

Path to log files:

path.logs: /path/to/logs

----------------------------------- Memory -----------------------------------

Lock the memory on startup:

bootstrap.mlockall: true

Make sure that the ES_HEAP_SIZE environment variable is set to about half the memory

available on the system and that the owner of the process is allowed to use this limit.

Elasticsearch performs poorly when the system is swapping the memory.

---------------------------------- Network -----------------------------------

Set the bind address to a specific IP (IPv4 or IPv6):

#network.host: x.x.x.x

Set a custom port for HTTP:

#http.port: 9200

For more information, see the documentation at:

<http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html>

--------------------------------- Discovery ----------------------------------

Pass an initial list of hosts to perform discovery when new node is started:

The default list of hosts is [“127.0.0.1”, “[::1]”]

discovery.zen.ping.unicast.hosts: [“host1”, “host2”]

Prevent the “split brain” by configuring the majority of nodes (total number of nodes / 2 + 1):

discovery.zen.minimum_master_nodes: 3

For more information, see the documentation at:

<http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html>

---------------------------------- Gateway -----------------------------------

Block initial recovery after a full cluster restart until N nodes are started:

gateway.recover_after_nodes: 3

For more information, see the documentation at:

<http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-gateway.html>

---------------------------------- Various -----------------------------------

Disable starting multiple nodes on a single system:

node.max_local_storage_nodes: 1

Require explicit names when deleting indices:

action.destructive_requires_name: true

node.name: localhost

##################################################

Search Guard 2 configuration

#Host: localhost

#Generated: Sun Jul 31 17:50:58 UTC 2016

#Git Hash: 51fced7dba388267d30ca7826cebcba8fb9edfb8

#ES-Version: 2.3.4

#SG-Version: 2.3.4.4

#SGSSL-Version: 2.3.4.14

#NettyNative-Version: 1.1.33.Fork17

#CA_PASS: f3c45ab4d999bc1f0ad8967ff9b42e39980eec58

#CL_ADM_PASS: 553281dad7802f1d6018

#CL_DEMOUSER_PASS: d88242ef83b1c5e48e0e

##################################################

searchguard.ssl.transport.enabled: true

searchguard.ssl.transport.keystore_filepath: CN=localhost-keystore.jks

searchguard.ssl.transport.keystore_password: xxxx

searchguard.ssl.transport.truststore_filepath: truststore.jks

searchguard.ssl.transport.truststore_password: xxxx

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.http.enabled: true

searchguard.ssl.http.keystore_filepath: CN=localhost-keystore.jks

searchguard.ssl.http.keystore_password: xxxxx

searchguard.ssl.http.truststore_filepath: truststore.jks

searchguard.ssl.http.truststore_password: xxxx

searchguard.kerberos.krb5_filepath: /Users/temp/kerberos_ldap_environment/krb5.conf

searchguard.kerberos.acceptor_keytab_filepath: http_srv.keytab

searchguard.audit.type: internal_elasticsearch

searchguard.authcz.admin_dn:

  • CN=sgadmin

On Friday, August 5, 2016 at 1:20:11 AM UTC+5:30, SG wrote:

pls do not cross post, this one is already logged here https://github.com/floragunncom/search-guard/issues/182#issuecomment-237533599

Can you please provide error messages and your elasticsearch log files as well as your configuration? Otherwise its hard to help.

Start easy just with the Search Guard bundle and curl. Once you initialized Search Guard by runiing sgadmin you normally should not see a “Search Guard not initialized (SG11)” afterwards.

Am 04.08.2016 um 15:58 schrieb rocky rocky....@gmail.com:

Hi All,

Can anyone help me with the below issue I am facing

I used tarfile which was said to be the easiest way to installtion searchguard which also includes elasticsearch package. Please refer to below link

https://github.com/floragunncom/search-guard/wiki/Search-Guard-Bundle.

After extracting tarfile I have started elasticsearch using the steps as mentioned in link and later ran sgadmin.sh script and evrything was looking great.

Once I have installed Kibana and Logstash as root user , installed beats on client machine . Currently beats are able to push logs to logstash whereas logstash is not able to comminucate to elasticsearch.

When I found reason it says “Search Guard not initialized (SG11)” and searchguard index not healthy (timeout: truewhich is really strange.

Note: I deployed search guard tarfile using non rootuser.I also installed Kibana logastah and Elasticsearch with search guard on same server. beats on client machines.

Can anyone guide me here please as I am going crazy with this searchguard installation?

–
You received this message because you are subscribed to the Google Groups “Search Guard” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/68a3415e-eaee-439d-ad0e-9e2759295549%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.