Installation order of Operations

Max_Furman · June 28, 2016, 10:26pm

I seem to be running into a bit of a chicken and egg problem when trying to install searchguard-2 and searchguard-ssl, restart ES 2.3.3, and then configure searchguard using the sgadmin.sh tool.

Here is my current ordering.

install search-guard-2 and search-guard-ssl, edit elasticsearch.yml adding static configuration options for server to server transport communication over ssl.
restart ES on all servers.
Seeing the following error on non master nodes:

[2016-06-28 19:06:27,164][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

This indicates that I need to run the sgadmin.sh configuration script w/ proper arguments.
However when I run the script I get the following output:

/usr/share/elasticsearch/plugins/search-guard-2/tools/sgadmin.sh -cd /usr/share/elasticsearch/plugins/search-guard-2/sgconfig/ -ks admin-keystore.jks -kspass xxx -ts admin-truststore.jks -tspass xxx -nhnv

Connect to localhost:9300

Exception in thread “main” NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}]]

at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:290)

at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:207)

at org.elasticsearch.client.transport.support.TransportProxyClient.execute(TransportProxyClient.java:55)

at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:288)

at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)

at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:348)

at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:848)

at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.health(AbstractClient.java:868)

at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:201)

The error, I gather, is happening b/c ES is unable to startup with the existing searchguard transport configuration options in the yml. It seems to me that this would be a chicken and egg problem, so I assume that I am missing some step in the installation. Hoping someone can point that out to me.

-max

searchguard_google_group · June 29, 2016, 7:17am

See inline comments:

I seem to be running into a bit of a chicken and egg problem when trying to install searchguard-2 and searchguard-ssl, restart ES 2.3.3, and then configure searchguard using the sgadmin.sh tool.

Here is my current ordering.

1) install search-guard-2 and search-guard-ssl, edit elasticsearch.yml adding static configuration options for server to server transport communication over ssl.
2) restart ES on all servers.
3) Seeing the following error on non master nodes:

[2016-06-28 19:06:27,164][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

Do you get this error while you try to connect your node via HTTP?

4) This indicates that I need to run the sgadmin.sh configuration script w/ proper arguments.
5) However when I run the script I get the following output:

/usr/share/elasticsearch/plugins/search-guard-2/tools/sgadmin.sh -cd /usr/share/elasticsearch/plugins/search-guard-2/sgconfig/ -ks admin-keystore.jks -kspass xxx -ts admin-truststore.jks -tspass xxx -nhnv

Connect to localhost:9300

Exception in thread "main" NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}]]

    at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:290)

    at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:207)

    at org.elasticsearch.client.transport.support.TransportProxyClient.execute(TransportProxyClient.java:55)

    at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:288)

    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)

    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:348)

    at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:848)

    at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.health(AbstractClient.java:868)

    at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:201)

The error, I gather, is happening b/c ES is unable to startup with the existing searchguard transport configuration options in the yml. It seems to me that this would be a chicken and egg problem, so I assume that I am missing some step in the installation. Hoping someone can point that out to me.

Can you pls. post the elasticsearch logs on DEBUG level and your elasticsearch.yml?

···

Am 29.06.2016 um 00:26 schrieb Max Furman <mx.furman@gmail.com>:

-max

--
You received this message because you are subscribed to the Google Groups "Search Guard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/1bc28b75-7254-4956-81c9-0def378ae571%40googlegroups.com\.
For more options, visit https://groups.google.com/d/optout\.

Max_Furman · June 29, 2016, 9:54pm

If I start the cluster w/out searchguard-2 and searchguard-ssl installed, and with the searchguard elasticsearch.yml lines removed, then the cluster starts up just fine and I have no problem accessing the node via http. If, however, I start with all the searchguard stuffs, then the nodes in the cluster are able to find one another, but any communication beyond that seems broken.

Below is a copy of my elasticsearch.yml:

···

======================== Elasticsearch Configuration =========================

NOTE: Elasticsearch comes with reasonable defaults for most settings.

Before you set out to tweak and tune the configuration, make sure you

understand what are you trying to accomplish and the consequences.

The primary way of configuring a node is via this file. This template lists

the most important settings you may want to configure for a production cluster.

Please see the documentation for further information on configuration options:

http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html

---------------------------------- Cluster -----------------------------------

Use a descriptive name for your cluster:

cluster.name: staging-elasticsearch

------------------------------------ Node ------------------------------------

Use a descriptive name for the node:

node.name: nextkibana02-ops

Add custom attributes to the node:

node.rack: r1

----------------------------------- Paths ------------------------------------

Path to directory where to store the data (separate multiple locations by comma):

path.data: /var/lib/elasticsearch

Path to log files:

path.logs: /var/log/elasticsearch

path.work: /var/run/elasticsearch

----------------------------------- Memory -----------------------------------

Lock the memory on startup:

bootstrap.mlockall: true

Make sure that the `ES_HEAP_SIZE` environment variable is set to about half the memory

available on the system and that the owner of the process is allowed to use this limit.

Elasticsearch performs poorly when the system is swapping the memory.

---------------------------------- Network -----------------------------------

Set the bind address to a specific IP (IPv4 or IPv6):

network.host: 1.2.3.4 (private-ip)

Set a custom host and port for HTTP:

http.host: 127.0.0.1

http.port: 9200

For more information, see the documentation at:

http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html

--------------------------------- Discovery ----------------------------------

Pass an initial list of hosts to perform discovery when new node is started:

The default list of hosts is [“127.0.0.1”, “[::1]”]

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping.unicast.hosts: [“nextkibana01-ops.es.betable.com:9300”,“nextkibana03-ops.es.betable.com:9300”]

Prevent the “split brain” by configuring the majority of nodes (total number of nodes / 2 + 1):

discovery.zen.minimum_master_nodes: 2

For more information, see the documentation at:

http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html

---------------------------------- Gateway -----------------------------------

Block initial recovery after a full cluster restart until N nodes are started:

gateway.recover_after_nodes: 2

gateway.expected_nodes: 3

gateway.recover_after_time: 5m

For more information, see the documentation at:

http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-gateway.html

---------------------------------- Various -----------------------------------

Disable starting multiple nodes on a single system:

node.max_local_storage_nodes: 1

Require explicit names when deleting indices:

action.destructive_requires_name: true

action.disable_delete_all_indices: true

metrics.statsd.host: 1.2.3.4

metrics.statsd.every: 10s

threadpool.bulk.type: fixed

threadpool.bulk.size: 4 # availableProcessors

threadpool.bulk.queue_size: 250

index:

mapping:

ignore_malformed: true

analysis:

analyzer:

default:

type: standard

stopwords: none

id_field:

type: custom

tokenizer: keyword

filter: [lowercase]

Searchguard

#############################################################################################

SEARCH GUARD SSL

Configuration

#############################################################################################

searchguard.authcz.admin_dn:

ST=CA,O=Company,L=San Francisco,C=US,CN=staging-elasticsearch-admin

#############################################################################################

Transport layer SSL

#############################################################################################

Enable or disable node-to-node ssl encryption (default: true)

#searchguard.ssl.transport.enabled: true

JKS or PKCS12 (default: JKS)

searchguard.ssl.transport.keystore_type: JKS

Relative path to the keystore file (mandatory, this stores the server certificates), must be placed under the config/ dir

searchguard.ssl.transport.keystore_filepath: “/etc/elasticsearch/server-keystore”

Alias name (default: first alias which could be found)

#searchguard.ssl.transport.keystore_alias: my_alias

Keystore password (default: changeit)

searchguard.ssl.transport.keystore_password: password

JKS or PKCS12 (default: JKS)

searchguard.ssl.transport.truststore_type: JKS

Relative path to the truststore file (mandatory, this stores the client/root certificates), must be placed under the config/ dir

searchguard.ssl.transport.truststore_filepath: “/etc/elasticsearch/server-truststore”

Alias name (default: trust all aliases)

#searchguard.ssl.transport.truststore_alias: my_alias

Truststore password (default: changeit)

searchguard.ssl.transport.truststore_password: password

Enforce hostname verification (default: true)

#searchguard.ssl.transport.enforce_hostname_verification: false

If hostname verification specify if hostname should be resolved (default: true)

#searchguard.ssl.transport.resolve_hostname: true

Use native Open SSL instead of JDK SSL if available (default: true)

#searchguard.ssl.transport.enable_openssl_if_available: false

Enabled SSL cipher suites for transport protocol (only Java format is supported)

WARNING: Expert setting, do only use if you know what you are doing

If you set wrong values here this this could be a security risk

#searchguard.ssl.transport.enabled_ciphers:

- “TLS_DHE_RSA_WITH_AES_256_CBC_SHA”

- “TLS_DHE_DSS_WITH_AES_128_CBC_SHA256”

Enabled SSL protocols for transport protocol (only Java format is supported)

WARNING: Expert setting, do only use if you know what you are doing

If you set wrong values here this this could be a security risk

#searchguard.ssl.transport.enabled_protocols:

- “TLSv1.2”

#############################################################################################

HTTP/REST layer SSL

#############################################################################################

Enable or disable rest layer security - https, (default: false)

#searchguard.ssl.http.enabled: true

JKS or PKCS12 (default: JKS)

#searchguard.ssl.http.keystore_type: PKCS12

Relative path to the keystore file (this stores the server certificates), must be placed under the config/ dir

#searchguard.ssl.http.keystore_filepath: keystore_https_node1.jks

Alias name (default: first alias which could be found)

#searchguard.ssl.http.keystore_alias: my_alias

Keystore password (default: changeit)

#searchguard.ssl.http.keystore_password: changeit

Do the clients (typically the browser or the proxy) have to authenticate themself to the http server, default is OPTIONAL

To enforce authentication use REQUIRE, to completely disable client certificates use NONE

#searchguard.ssl.http.clientauth_mode: REQUIRE

JKS or PKCS12 (default: JKS)

#searchguard.ssl.http.truststore_type: PKCS12

Relative path to the truststore file (this stores the client certificates), must be placed under the config/ dir

#searchguard.ssl.http.truststore_filepath: truststore_https.jks

Alias name (default: first alias which could be found)

#searchguard.ssl.http.truststore_alias: my_alias

Truststore password (default: changeit)

#searchguard.ssl.http.truststore_password: changeit

Use native Open SSL instead of JDK SSL if available (default: true)

#searchguard.ssl.http.enable_openssl_if_available: false

Enabled SSL cipher suites for http protocol (only Java format is supported)

WARNING: Expert setting, do only use if you know what you are doing

If you set wrong values here this this could be a security risk

#searchguard.ssl.http.enabled_ciphers:

- “TLS_DHE_RSA_WITH_AES_256_CBC_SHA”

- “TLS_DHE_DSS_WITH_AES_128_CBC_SHA256”

Enabled SSL protocols for http protocol (only Java format is supported)

WARNING: Expert setting, do only use if you know what you are doing

If you set wrong values here this this could be a security risk

#searchguard.ssl.http.enabled_protocols:

- “TLSv1.2”

Also below is one of many similar log lines from the master node when I try to start ES 2.3.3 w/ search-guard and the above elasticsearch.yml:

[2016-06-28 19:11:04,518][WARN ][gateway ] [nextkibana02-ops] [logstash-2016.06.05][3]: failed to list shard for shard_store on node [jyc0Tx-wRGeS71JoLf4X-Q]

FailedNodeException[total failure in fetching]; nested: ElasticsearchException[unauthenticated request internal:cluster/nodes/indices/shard/store for user User [name=_sg_internal, roles=]];

at org.elasticsearch.gateway.AsyncShardFetch$1.onFailure(AsyncShardFetch.java:277)

at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:95)

at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:135)

at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170)

at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:144)

at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:85)

at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.list(TransportNodesListShardStoreMetaData.java:88)

at org.elasticsearch.gateway.AsyncShardFetch.asyncFetch(AsyncShardFetch.java:267)

at org.elasticsearch.gateway.AsyncShardFetch.fetchData(AsyncShardFetch.java:117)

at org.elasticsearch.gateway.GatewayAllocator$InternalReplicaShardAllocator.fetchData(GatewayAllocator.java:183)

at org.elasticsearch.gateway.ReplicaShardAllocator.allocateUnassigned(ReplicaShardAllocator.java:137)

at org.elasticsearch.gateway.GatewayAllocator.allocateUnassigned(GatewayAllocator.java:123)

at org.elasticsearch.cluster.routing.allocation.allocator.ShardsAllocators.allocateUnassigned(ShardsAllocators.java:70)

at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:258)

at org.elasticsearch.cluster.routing.allocation.AllocationService.applyStartedShards(AllocationService.java:86)

at org.elasticsearch.cluster.action.shard.ShardStateAction$ShardStartedClusterStateHandler.execute(ShardStateAction.java:218)

at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:468)

at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)

at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)

at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused by: ElasticsearchException[unauthenticated request internal:cluster/nodes/indices/shard/store for user User [name=_sg_internal, roles=]]

… 21 more

=================

I can get more detailed “debug” logs from startup if necessary.

On Wednesday, June 29, 2016 at 12:17:18 AM UTC-7, SG wrote:

See inline comments:

Am 29.06.2016 um 00:26 schrieb Max Furman mx.f...@gmail.com:

I seem to be running into a bit of a chicken and egg problem when trying to install searchguard-2 and searchguard-ssl, restart ES 2.3.3, and then configure searchguard using the sgadmin.sh tool.

Here is my current ordering.

install search-guard-2 and search-guard-ssl, edit elasticsearch.yml adding static configuration options for server to server transport communication over ssl.

restart ES on all servers.

Seeing the following error on non master nodes:
   [2016-06-28 19:06:27,164][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
Do you get this error while you try to connect your node via HTTP?

This indicates that I need to run the sgadmin.sh configuration script w/ proper arguments.

However when I run the script I get the following output:

/usr/share/elasticsearch/plugins/search-guard-2/tools/sgadmin.sh -cd /usr/share/elasticsearch/plugins/search-guard-2/sgconfig/ -ks admin-keystore.jks -kspass xxx -ts admin-truststore.jks -tspass xxx -nhnv

Connect to localhost:9300

Exception in thread “main” NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}]]
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:290)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:207)
at org.elasticsearch.client.transport.support.TransportProxyClient.execute(TransportProxyClient.java:55)
at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:288)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:348)
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:848)
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.health(AbstractClient.java:868)
at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:201)
The error, I gather, is happening b/c ES is unable to startup with the existing searchguard transport configuration options in the yml. It seems to me that this would be a chicken and egg problem, so I assume that I am missing some step in the installation. Hoping someone can point that out to me.

Can you pls. post the elasticsearch logs on DEBUG level and your elasticsearch.yml?

-max

–
You received this message because you are subscribed to the Google Groups “Search Guard” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/1bc28b75-7254-4956-81c9-0def378ae571%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Max_Furman · June 30, 2016, 12:14am

Once I’ve got searchguard installed and turned on via the yml, I cannot reach the http port. The error I get each time is:

Search Guard not initialized (SG11)

And when I try to run the sgadmin.sh command to initialize search-guard I continue getting:

sudo -u elasticsearch /usr/share/elasticsearch/plugins/search-guard-2/tools/sgadmin.sh -cd /usr/share/elasticsearch/plugins/search-guard-2/sgconfig/ -ks client-keystore.jks -kspass XXX -ts client-truststore.jks -tspass XXX -nhnv

Connect to localhost:9300

Exception in thread “main” NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}]]

at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:290)

at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:207)

at org.elasticsearch.client.transport.support.TransportProxyClient.execute(TransportProxyClient.java:55)

at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:288)

at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)

at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:348)

at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:848)

at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.health(AbstractClient.java:868)

at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:201)

Here are a whole bunch of debug startup logs from one (of three) servers in my cluster:

···

[2016-06-29 22:15:18,584][DEBUG][bootstrap ] Linux seccomp filter installation successful, threads: [all]
[2016-06-29 22:15:20,058][DEBUG][bootstrap ] java.class.path: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar:/usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar:/usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.6.jar:/usr/share/elasticsearch/lib/jsr166e-1.1.0.jar:/usr/share/elasticsearch/lib/snakeyaml-1.15.jar:/usr/share/elasticsearch/lib/securesm-1.0.jar:/usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar:/usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar:/usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar:/usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.6.jar:/usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.6.jar:/usr/share/elasticsearch/lib/lucene-core-5.5.0.jar:/usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar:/usr/share/elasticsearch/lib/netty-3.10.5.Final.jar:/usr/share/elasticsearch/lib/jna-4.1.0.jar:/usr/share/elasticsearch/lib/joda-convert-1.2.jar:/usr/share/elasticsearch/lib/lucene-join-5.5.0.jar:/usr/share/elasticsearch/lib/hppc-0.7.1.jar:/usr/share/elasticsearch/lib/jackson-core-2.6.6.jar:/usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar:/usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar:/usr/share/elasticsearch/lib/commons-cli-1.3.1.jar:/usr/share/elasticsearch/lib/guava-18.0.jar:/usr/share/elasticsearch/lib/log4j-1.2.17.jar:/usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar:/usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar:/usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar:/usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar:/usr/share/elasticsearch/lib/jts-1.13.jar:/usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar:/usr/share/elasticsearch/lib/joda-time-2.8.2.jar:/usr/share/elasticsearch/lib/t-digest-3.0.jar:/usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar:/usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar:/usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar:/usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar:/usr/share/elasticsearch/lib/compiler-0.8.13.jar:/usr/share/elasticsearch/lib/spatial4j-0.5.jar
[2016-06-29 22:15:20,059][DEBUG][bootstrap ] sun.boot.class.path: /usr/jre/lib/resources.jar:/usr/jre/lib/rt.jar:/usr/jre/lib/sunrsasign.jar:/usr/jre/lib/jsse.jar:/usr/jre/lib/jce.jar:/usr/jre/lib/charsets.jar:/usr/jre/lib/jfr.jar:/usr/jre/classes
[2016-06-29 22:15:20,059][DEBUG][bootstrap ] classloader urls: [file:/usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar, file:/usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar, file:/usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.6.jar, file:/usr/share/elasticsearch/lib/jsr166e-1.1.0.jar, file:/usr/share/elasticsearch/lib/snakeyaml-1.15.jar, file:/usr/share/elasticsearch/lib/securesm-1.0.jar, file:/usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar, file:/usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar, file:/usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar, file:/usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.6.jar, file:/usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.6.jar, file:/usr/share/elasticsearch/lib/lucene-core-5.5.0.jar, file:/usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar, file:/usr/share/elasticsearch/lib/netty-3.10.5.Final.jar, file:/usr/share/elasticsearch/lib/jna-4.1.0.jar, file:/usr/share/elasticsearch/lib/joda-convert-1.2.jar, file:/usr/share/elasticsearch/lib/lucene-join-5.5.0.jar, file:/usr/share/elasticsearch/lib/hppc-0.7.1.jar, file:/usr/share/elasticsearch/lib/jackson-core-2.6.6.jar, file:/usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar, file:/usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar, file:/usr/share/elasticsearch/lib/commons-cli-1.3.1.jar, file:/usr/share/elasticsearch/lib/guava-18.0.jar, file:/usr/share/elasticsearch/lib/log4j-1.2.17.jar, file:/usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar, file:/usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar, file:/usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar, file:/usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar, file:/usr/share/elasticsearch/lib/jts-1.13.jar, file:/usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar, file:/usr/share/elasticsearch/lib/joda-time-2.8.2.jar, file:/usr/share/elasticsearch/lib/t-digest-3.0.jar, file:/usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar, file:/usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar, file:/usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar, file:/usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar, file:/usr/share/elasticsearch/lib/compiler-0.8.13.jar, file:/usr/share/elasticsearch/lib/spatial4j-0.5.jar]
[2016-06-29 22:15:20,066][DEBUG][bootstrap ] java.home: /usr/jre
[2016-06-29 22:15:20,066][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar
[2016-06-29 22:15:20,104][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar
[2016-06-29 22:15:20,105][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.6.jar
[2016-06-29 22:15:20,107][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar
[2016-06-29 22:15:20,108][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/snakeyaml-1.15.jar
[2016-06-29 22:15:20,109][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/securesm-1.0.jar
[2016-06-29 22:15:20,110][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar
[2016-06-29 22:15:20,111][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar
[2016-06-29 22:15:20,111][DEBUG][bootstrap ] excluding duplicate classpath element: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar
[2016-06-29 22:15:20,112][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.6.jar
[2016-06-29 22:15:20,112][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.6.jar
[2016-06-29 22:15:20,119][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-core-5.5.0.jar
[2016-06-29 22:15:20,124][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar
[2016-06-29 22:15:20,125][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/netty-3.10.5.Final.jar
[2016-06-29 22:15:20,132][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jna-4.1.0.jar
[2016-06-29 22:15:20,133][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/joda-convert-1.2.jar
[2016-06-29 22:15:20,134][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-join-5.5.0.jar
[2016-06-29 22:15:20,135][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/hppc-0.7.1.jar
[2016-06-29 22:15:20,138][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jackson-core-2.6.6.jar
[2016-06-29 22:15:20,139][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar
[2016-06-29 22:15:20,140][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar
[2016-06-29 22:15:20,141][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar
[2016-06-29 22:15:20,142][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/guava-18.0.jar
[2016-06-29 22:15:20,147][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/log4j-1.2.17.jar
[2016-06-29 22:15:20,148][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar
[2016-06-29 22:15:20,152][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar
[2016-06-29 22:15:20,158][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar
[2016-06-29 22:15:20,159][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar
[2016-06-29 22:15:20,160][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jts-1.13.jar
[2016-06-29 22:15:20,164][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar
[2016-06-29 22:15:20,165][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/joda-time-2.8.2.jar
[2016-06-29 22:15:20,167][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/t-digest-3.0.jar
[2016-06-29 22:15:20,168][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar
[2016-06-29 22:15:20,169][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar
[2016-06-29 22:15:20,170][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar
[2016-06-29 22:15:20,171][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar
[2016-06-29 22:15:20,172][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/compiler-0.8.13.jar
[2016-06-29 22:15:20,173][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/spatial4j-0.5.jar
[2016-06-29 22:15:20,619][INFO ][node ] [nextkibana01-ops] version[2.3.3], pid[14543], build[218bdf1/2016-05-17T15:40:04Z]
[2016-06-29 22:15:20,619][INFO ][node ] [nextkibana01-ops] initializing …
[2016-06-29 22:15:20,619][DEBUG][node ] [nextkibana01-ops] using config [/etc/elasticsearch], data [[/var/lib/elasticsearch]], logs [/var/log/elasticsearch], plugins [/usr/share/elasticsearch/plugins]
[2016-06-29 22:15:20,649][DEBUG][bootstrap ] java.home: /usr/jre
[2016-06-29 22:15:20,650][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar
[2016-06-29 22:15:20,676][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar
[2016-06-29 22:15:20,677][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.6.jar
[2016-06-29 22:15:20,678][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar
[2016-06-29 22:15:20,679][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/snakeyaml-1.15.jar
[2016-06-29 22:15:20,680][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/securesm-1.0.jar
[2016-06-29 22:15:20,681][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar
[2016-06-29 22:15:20,681][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar
[2016-06-29 22:15:20,682][DEBUG][bootstrap ] excluding duplicate classpath element: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar
[2016-06-29 22:15:20,682][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.6.jar
[2016-06-29 22:15:20,683][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.6.jar
[2016-06-29 22:15:20,684][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-core-5.5.0.jar
[2016-06-29 22:15:20,689][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar
[2016-06-29 22:15:20,690][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/netty-3.10.5.Final.jar
[2016-06-29 22:15:20,701][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jna-4.1.0.jar
[2016-06-29 22:15:20,702][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/joda-convert-1.2.jar
[2016-06-29 22:15:20,702][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-join-5.5.0.jar
[2016-06-29 22:15:20,703][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/hppc-0.7.1.jar
[2016-06-29 22:15:20,706][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jackson-core-2.6.6.jar
[2016-06-29 22:15:20,707][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar
[2016-06-29 22:15:20,708][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar
[2016-06-29 22:15:20,709][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar
[2016-06-29 22:15:20,710][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/guava-18.0.jar
[2016-06-29 22:15:20,715][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/log4j-1.2.17.jar
[2016-06-29 22:15:20,716][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar
[2016-06-29 22:15:20,718][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar
[2016-06-29 22:15:20,721][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar
[2016-06-29 22:15:20,721][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar
[2016-06-29 22:15:20,722][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jts-1.13.jar
[2016-06-29 22:15:20,724][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar
[2016-06-29 22:15:20,726][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/joda-time-2.8.2.jar
[2016-06-29 22:15:20,728][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/t-digest-3.0.jar
[2016-06-29 22:15:20,728][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar
[2016-06-29 22:15:20,729][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar
[2016-06-29 22:15:20,730][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar
[2016-06-29 22:15:20,732][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar
[2016-06-29 22:15:20,733][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/compiler-0.8.13.jar
[2016-06-29 22:15:20,734][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/spatial4j-0.5.jar
[2016-06-29 22:15:20,734][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/modules/reindex/reindex-2.3.3.jar
[2016-06-29 22:15:21,447][DEBUG][bootstrap ] java.home: /usr/jre
[2016-06-29 22:15:21,447][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar
[2016-06-29 22:15:21,464][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar
[2016-06-29 22:15:21,465][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.6.jar
[2016-06-29 22:15:21,466][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar
[2016-06-29 22:15:21,466][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/snakeyaml-1.15.jar
[2016-06-29 22:15:21,467][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/securesm-1.0.jar
[2016-06-29 22:15:21,468][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar
[2016-06-29 22:15:21,468][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar
[2016-06-29 22:15:21,469][DEBUG][bootstrap ] excluding duplicate classpath element: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar
[2016-06-29 22:15:21,469][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.6.jar
[2016-06-29 22:15:21,470][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.6.jar
[2016-06-29 22:15:21,470][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-core-5.5.0.jar
[2016-06-29 22:15:21,474][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar
[2016-06-29 22:15:21,475][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/netty-3.10.5.Final.jar
[2016-06-29 22:15:21,478][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jna-4.1.0.jar
[2016-06-29 22:15:21,479][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/joda-convert-1.2.jar
[2016-06-29 22:15:21,479][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-join-5.5.0.jar
[2016-06-29 22:15:21,480][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/hppc-0.7.1.jar
[2016-06-29 22:15:21,482][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jackson-core-2.6.6.jar
[2016-06-29 22:15:21,483][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar
[2016-06-29 22:15:21,484][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar
[2016-06-29 22:15:21,485][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar
[2016-06-29 22:15:21,485][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/guava-18.0.jar
[2016-06-29 22:15:21,490][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/log4j-1.2.17.jar
[2016-06-29 22:15:21,491][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar
[2016-06-29 22:15:21,492][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar
[2016-06-29 22:15:21,494][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar
[2016-06-29 22:15:21,495][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar
[2016-06-29 22:15:21,495][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jts-1.13.jar
[2016-06-29 22:15:21,497][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar
[2016-06-29 22:15:21,498][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/joda-time-2.8.2.jar
[2016-06-29 22:15:21,500][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/t-digest-3.0.jar
[2016-06-29 22:15:21,500][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar
[2016-06-29 22:15:21,501][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar
[2016-06-29 22:15:21,502][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar
[2016-06-29 22:15:21,503][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar
[2016-06-29 22:15:21,504][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/compiler-0.8.13.jar
[2016-06-29 22:15:21,505][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/spatial4j-0.5.jar
[2016-06-29 22:15:21,505][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/modules/lang-groovy/groovy-2.4.6-indy.jar
[2016-06-29 22:15:21,597][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/modules/lang-groovy/lang-groovy-2.3.3.jar
[2016-06-29 22:15:21,617][DEBUG][bootstrap ] java.home: /usr/jre
[2016-06-29 22:15:21,617][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar
[2016-06-29 22:15:21,633][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar
[2016-06-29 22:15:21,634][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.6.jar
[2016-06-29 22:15:21,636][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar
[2016-06-29 22:15:21,636][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/snakeyaml-1.15.jar
[2016-06-29 22:15:21,637][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/securesm-1.0.jar
[2016-06-29 22:15:21,637][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar
[2016-06-29 22:15:21,638][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar
[2016-06-29 22:15:21,638][DEBUG][bootstrap ] excluding duplicate classpath element: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar
[2016-06-29 22:15:21,639][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.6.jar
[2016-06-29 22:15:21,639][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.6.jar
[2016-06-29 22:15:21,640][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-core-5.5.0.jar
[2016-06-29 22:15:21,644][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar
[2016-06-29 22:15:21,644][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/netty-3.10.5.Final.jar
[2016-06-29 22:15:21,646][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jna-4.1.0.jar
[2016-06-29 22:15:21,647][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/joda-convert-1.2.jar
[2016-06-29 22:15:21,647][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-join-5.5.0.jar
[2016-06-29 22:15:21,648][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/hppc-0.7.1.jar
[2016-06-29 22:15:21,650][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jackson-core-2.6.6.jar
[2016-06-29 22:15:21,651][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar
[2016-06-29 22:15:21,652][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar
[2016-06-29 22:15:21,652][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar
[2016-06-29 22:15:21,653][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/guava-18.0.jar
[2016-06-29 22:15:21,656][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/log4j-1.2.17.jar
[2016-06-29 22:15:21,657][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar
[2016-06-29 22:15:21,658][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar
[2016-06-29 22:15:21,660][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar
[2016-06-29 22:15:21,661][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar
[2016-06-29 22:15:21,661][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jts-1.13.jar
[2016-06-29 22:15:21,662][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar
[2016-06-29 22:15:21,663][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/joda-time-2.8.2.jar
[2016-06-29 22:15:21,664][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/t-digest-3.0.jar
[2016-06-29 22:15:21,665][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar
[2016-06-29 22:15:21,665][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar
[2016-06-29 22:15:21,666][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar
[2016-06-29 22:15:21,667][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar
[2016-06-29 22:15:21,668][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/compiler-0.8.13.jar
[2016-06-29 22:15:21,668][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/spatial4j-0.5.jar
[2016-06-29 22:15:21,669][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/modules/lang-expression/lang-expression-2.3.3.jar
[2016-06-29 22:15:21,681][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/modules/lang-expression/asm-5.0.4.jar
[2016-06-29 22:15:21,692][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/modules/lang-expression/antlr4-runtime-4.5.1-1.jar
[2016-06-29 22:15:21,695][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/modules/lang-expression/asm-commons-5.0.4.jar
[2016-06-29 22:15:21,698][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/modules/lang-expression/lucene-expressions-5.5.0.jar
[2016-06-29 22:15:22,151][DEBUG][bootstrap ] java.home: /usr/jre
[2016-06-29 22:15:22,151][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar
[2016-06-29 22:15:22,164][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar
[2016-06-29 22:15:22,165][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.6.jar
[2016-06-29 22:15:22,166][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar
[2016-06-29 22:15:22,167][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/snakeyaml-1.15.jar
[2016-06-29 22:15:22,167][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/securesm-1.0.jar
[2016-06-29 22:15:22,168][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar
[2016-06-29 22:15:22,168][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar
[2016-06-29 22:15:22,169][DEBUG][bootstrap ] excluding duplicate classpath element: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar
[2016-06-29 22:15:22,169][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.6.jar
[2016-06-29 22:15:22,169][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.6.jar
[2016-06-29 22:15:22,170][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-core-5.5.0.jar
[2016-06-29 22:15:22,173][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar
[2016-06-29 22:15:22,174][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/netty-3.10.5.Final.jar
[2016-06-29 22:15:22,176][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jna-4.1.0.jar
[2016-06-29 22:15:22,177][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/joda-convert-1.2.jar
[2016-06-29 22:15:22,177][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-join-5.5.0.jar
[2016-06-29 22:15:22,178][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/hppc-0.7.1.jar
[2016-06-29 22:15:22,180][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jackson-core-2.6.6.jar
[2016-06-29 22:15:22,181][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar
[2016-06-29 22:15:22,181][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar
[2016-06-29 22:15:22,182][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar
[2016-06-29 22:15:22,183][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/guava-18.0.jar
[2016-06-29 22:15:22,186][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/log4j-1.2.17.jar
[2016-06-29 22:15:22,188][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar
[2016-06-29 22:15:22,189][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar
[2016-06-29 22:15:22,190][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar
[2016-06-29 22:15:22,191][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar
[2016-06-29 22:15:22,191][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jts-1.13.jar
[2016-06-29 22:15:22,193][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar
[2016-06-29 22:15:22,193][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/joda-time-2.8.2.jar
[2016-06-29 22:15:22,195][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/t-digest-3.0.jar
[2016-06-29 22:15:22,195][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar
[2016-06-29 22:15:22,196][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar
[2016-06-29 22:15:22,197][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar
[2016-06-29 22:15:22,197][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar
[2016-06-29 22:15:22,198][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/compiler-0.8.13.jar
[2016-06-29 22:15:22,199][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/spatial4j-0.5.jar
[2016-06-29 22:15:22,200][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/plugins/search-guard-ssl/search-guard-ssl-2.3.3.13.jar
[2016-06-29 22:15:22,200][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/plugins/search-guard-ssl/netty-handler-4.0.37.Final.jar
[2016-06-29 22:15:22,201][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/plugins/search-guard-ssl/netty-common-4.0.37.Final.jar
[2016-06-29 22:15:22,202][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/plugins/search-guard-ssl/netty-transport-4.0.37.Final.jar
[2016-06-29 22:15:22,203][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/plugins/search-guard-ssl/netty-buffer-4.0.37.Final.jar
[2016-06-29 22:15:22,204][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/plugins/search-guard-ssl/netty-codec-4.0.37.Final.jar
[2016-06-29 22:15:22,205][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/plugins/search-guard-2/search-guard-2-2.3.3.1.jar
[2016-06-29 22:15:22,237][DEBUG][io.netty.util.internal.logging.InternalLoggerFactory] Using Log4J as the default logging framework
[2016-06-29 22:15:22,259][DEBUG][io.netty.util.internal.PlatformDependent0] java.nio.Buffer.address: available
[2016-06-29 22:15:22,259][DEBUG][io.netty.util.internal.PlatformDependent0] sun.misc.Unsafe.theUnsafe: available
[2016-06-29 22:15:22,260][DEBUG][io.netty.util.internal.PlatformDependent0] sun.misc.Unsafe.copyMemory: available
[2016-06-29 22:15:22,261][DEBUG][io.netty.util.internal.PlatformDependent0] java.nio.Bits.unaligned: true
[2016-06-29 22:15:22,261][DEBUG][io.netty.util.internal.PlatformDependent0] java.nio.DirectByteBuffer.(long, int): available
[2016-06-29 22:15:22,279][DEBUG][io.netty.util.internal.PlatformDependent] Java version: 8
[2016-06-29 22:15:22,279][DEBUG][io.netty.util.internal.PlatformDependent] -Dio.netty.noUnsafe: false
[2016-06-29 22:15:22,279][DEBUG][io.netty.util.internal.PlatformDependent] sun.misc.Unsafe: available
[2016-06-29 22:15:22,280][DEBUG][io.netty.util.internal.PlatformDependent] -Dio.netty.noJavassist: false
[2016-06-29 22:15:22,281][DEBUG][io.netty.util.internal.PlatformDependent] Javassist: unavailable
[2016-06-29 22:15:22,281][DEBUG][io.netty.util.internal.PlatformDependent] You don’t have Javassist in your class path or you don’t have enough permission to load dynamically generated classes. Please check the configuration for better performance.
[2016-06-29 22:15:22,281][DEBUG][io.netty.util.internal.PlatformDependent] -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
[2016-06-29 22:15:22,282][DEBUG][io.netty.util.internal.PlatformDependent] -Dio.netty.bitMode: 64 (sun.arch.data.model)
[2016-06-29 22:15:22,282][DEBUG][io.netty.util.internal.PlatformDependent] -Dio.netty.noPreferDirect: false
[2016-06-29 22:15:22,282][DEBUG][io.netty.util.internal.PlatformDependent] io.netty.maxDirectMemory: 4260102144 bytes
[2016-06-29 22:15:22,300][DEBUG][io.netty.handler.ssl.OpenSsl] netty-tcnative not in the classpath; OpenSslEngine will be unavailable.
[2016-06-29 22:15:22,311][DEBUG][bootstrap ] java.home: /usr/jre
[2016-06-29 22:15:22,311][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar
[2016-06-29 22:15:22,327][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar
[2016-06-29 22:15:22,327][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.6.jar
[2016-06-29 22:15:22,328][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar
[2016-06-29 22:15:22,329][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/snakeyaml-1.15.jar
[2016-06-29 22:15:22,329][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/securesm-1.0.jar
[2016-06-29 22:15:22,330][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar
[2016-06-29 22:15:22,330][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar
[2016-06-29 22:15:22,331][DEBUG][bootstrap ] excluding duplicate classpath element: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar
[2016-06-29 22:15:22,331][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.6.jar
[2016-06-29 22:15:22,331][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.6.jar
[2016-06-29 22:15:22,332][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-core-5.5.0.jar
[2016-06-29 22:15:22,336][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar
[2016-06-29 22:15:22,338][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/netty-3.10.5.Final.jar
[2016-06-29 22:15:22,341][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jna-4.1.0.jar
[2016-06-29 22:15:22,341][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/joda-convert-1.2.jar
[2016-06-29 22:15:22,342][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-join-5.5.0.jar
[2016-06-29 22:15:22,342][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/hppc-0.7.1.jar
[2016-06-29 22:15:22,345][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jackson-core-2.6.6.jar
[2016-06-29 22:15:22,345][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar
[2016-06-29 22:15:22,346][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar
[2016-06-29 22:15:22,348][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar
[2016-06-29 22:15:22,349][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/guava-18.0.jar
[2016-06-29 22:15:22,352][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/log4j-1.2.17.jar
[2016-06-29 22:15:22,353][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar
[2016-06-29 22:15:22,354][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar
[2016-06-29 22:15:22,356][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar
[2016-06-29 22:15:22,356][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar
[2016-06-29 22:15:22,357][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jts-1.13.jar
[2016-06-29 22:15:22,358][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar
[2016-06-29 22:15:22,359][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/joda-time-2.8.2.jar
[2016-06-29 22:15:22,360][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/t-digest-3.0.jar
[2016-06-29 22:15:22,361][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar
[2016-06-29 22:15:22,361][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar
[2016-06-29 22:15:22,362][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar
[2016-06-29 22:15:22,363][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar
[2016-06-29 22:15:22,364][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/compiler-0.8.13.jar
[2016-06-29 22:15:22,364][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/spatial4j-0.5.jar
[2016-06-29 22:15:22,366][DEBUG][bootstrap ] java.home: /usr/jre
[2016-06-29 22:15:22,366][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar
[2016-06-29 22:15:22,378][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar
[2016-06-29 22:15:22,378][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.6.jar
[2016-06-29 22:15:22,379][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar
[2016-06-29 22:15:22,380][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/snakeyaml-1.15.jar
[2016-06-29 22:15:22,381][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/securesm-1.0.jar
[2016-06-29 22:15:22,381][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar
[2016-06-29 22:15:22,381][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar
[2016-06-29 22:15:22,382][DEBUG][bootstrap ] excluding duplicate classpath element: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar
[2016-06-29 22:15:22,382][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.6.jar
[2016-06-29 22:15:22,382][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.6.jar
[2016-06-29 22:15:22,383][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-core-5.5.0.jar
[2016-06-29 22:15:22,386][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar
[2016-06-29 22:15:22,386][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/netty-3.10.5.Final.jar
[2016-06-29 22:15:22,389][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jna-4.1.0.jar
[2016-06-29 22:15:22,389][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/joda-convert-1.2.jar
[2016-06-29 22:15:22,390][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-join-5.5.0.jar
[2016-06-29 22:15:22,390][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/hppc-0.7.1.jar
[2016-06-29 22:15:22,392][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jackson-core-2.6.6.jar
[2016-06-29 22:15:22,393][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar
[2016-06-29 22:15:22,394][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar
[2016-06-29 22:15:22,394][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar
[2016-06-29 22:15:22,395][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/guava-18.0.jar
[2016-06-29 22:15:22,398][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/log4j-1.2.17.jar
[2016-06-29 22:15:22,399][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar
[2016-06-29 22:15:22,400][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar
[2016-06-29 22:15:22,401][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar
[2016-06-29 22:15:22,402][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar
[2016-06-29 22:15:22,402][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/jts-1.13.jar
[2016-06-29 22:15:22,404][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar
[2016-06-29 22:15:22,404][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/joda-time-2.8.2.jar
[2016-06-29 22:15:22,406][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/t-digest-3.0.jar
[2016-06-29 22:15:22,406][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar
[2016-06-29 22:15:22,406][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar
[2016-06-29 22:15:22,407][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar
[2016-06-29 22:15:22,408][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar
[2016-06-29 22:15:22,409][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/compiler-0.8.13.jar
[2016-06-29 22:15:22,409][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/lib/spatial4j-0.5.jar
[2016-06-29 22:15:22,410][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/plugins/statsd/elasticsearch-statsd-2.3.3.0.jar
[2016-06-29 22:15:22,433][DEBUG][bootstrap ] examining jar: /usr/share/elasticsearch/plugins/statsd/java-statsd-client-3.0.2.jar
[2016-06-29 22:15:22,461][INFO ][plugins ] [nextkibana01-ops] modules [reindex, lang-expression, lang-groovy], plugins [head, search-guard-ssl, statsd, search-guard-2], sites [head]
[2016-06-29 22:15:22,582][DEBUG][env ] [nextkibana01-ops] using node location [[NodePath{path=/var/lib/elasticsearch/staging-elasticsearch/nodes/0, spins=false}]], local_node_id [0]
[2016-06-29 22:15:22,587][DEBUG][env ] [nextkibana01-ops] node data locations details:
→ /var/lib/elasticsearch/staging-elasticsearch/nodes/0, free_space [152.8gb], usable_space [136.7gb], total_space [314.8gb], spins? [no], mount [/ (/dev/xvda1)], type [ext4]
[2016-06-29 22:15:22,587][INFO ][env ] [nextkibana01-ops] heap size [3.9gb], compressed ordinary object pointers [true]
[2016-06-29 22:15:22,588][WARN ][env ] [nextkibana01-ops] max file descriptors [65535] for elasticsearch process likely too low, consider increasing to at least [65536]
[2016-06-29 22:15:22,771][DEBUG][threadpool ] [nextkibana01-ops] creating thread_pool [force_merge], type [fixed], size [1], queue_size [null]
[2016-06-29 22:15:22,828][DEBUG][threadpool ] [nextkibana01-ops] creating thread_pool [percolate], type [fixed], size [4], queue_size [1k]
[2016-06-29 22:15:22,850][DEBUG][threadpool ] [nextkibana01-ops] creating thread_pool [fetch_shard_started], type [scaling], min [1], size [8], keep_alive [5m]
[2016-06-29 22:15:22,853][DEBUG][threadpool ] [nextkibana01-ops] creating thread_pool [listener], type [fixed], size [2], queue_size [null]
[2016-06-29 22:15:22,853][DEBUG][threadpool ] [nextkibana01-ops] creating thread_pool [index], type [fixed], size [4], queue_size [200]
[2016-06-29 22:15:22,853][DEBUG][threadpool ] [nextkibana01-ops] creating thread_pool [refresh], type [scaling], min [1], size [2], keep_alive [5m]
[2016-06-29 22:15:22,853][DEBUG][threadpool ] [nextkibana01-ops] creating thread_pool [suggest], type [fixed], size [4], queue_size [1k]
[2016-06-29 22:15:22,853][DEBUG][threadpool ] [nextkibana01-ops] creating thread_pool [generic], type [cached], keep_alive [30s]
[2016-06-29 22:15:22,857][DEBUG][threadpool ] [nextkibana01-ops] creating thread_pool [warmer], type [scaling], min [1], size [2], keep_alive [5m]
[2016-06-29 22:15:22,857][DEBUG][threadpool ] [nextkibana01-ops] creating thread_pool [search], type [fixed], size [7], queue_size [1k]
[2016-06-29 22:15:22,857][DEBUG][threadpool ] [nextkibana01-ops] creating thread_pool [flush], type [scaling], min [1], size [2], keep_alive [5m]
[2016-06-29 22:15:22,858][DEBUG][threadpool ] [nextkibana01-ops] creating thread_pool [fetch_shard_store], type [scaling], min [1], size [8], keep_alive [5m]
[2016-06-29 22:15:22,858][DEBUG][threadpool ] [nextkibana01-ops] creating thread_pool [management], type [scaling], min [1], size [5], keep_alive [5m]
[2016-06-29 22:15:22,858][DEBUG][threadpool ] [nextkibana01-ops] creating thread_pool [get], type [fixed], size [4], queue_size [1k]
[2016-06-29 22:15:22,859][DEBUG][threadpool ] [nextkibana01-ops] creating thread_pool [bulk], type [fixed], size [4], queue_size [250]
[2016-06-29 22:15:22,859][DEBUG][threadpool ] [nextkibana01-ops] creating thread_pool [snapshot], type [scaling], min [1], size [2], keep_alive [5m]
[2016-06-29 22:15:23,534][DEBUG][io.netty.handler.ssl.JdkSslContext] Default protocols (JDK): [TLSv1.2, TLSv1.1, TLSv1]
[2016-06-29 22:15:23,534][DEBUG][io.netty.handler.ssl.JdkSslContext] Default cipher suites (JDK): [TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA]
[2016-06-29 22:15:24,854][INFO ][transport ] [nextkibana01-ops] Using [com.floragunn.searchguard.transport.SearchGuardTransportService] as transport service, overridden by [search-guard2]
[2016-06-29 22:15:24,855][INFO ][transport ] [nextkibana01-ops] Using [com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] as transport, overridden by [search-guard-ssl]
[2016-06-29 22:15:27,642][DEBUG][common.network ] configuration:
lo
inet 127.0.0.1 netmask:255.0.0.0 scope:host
inet6 ::1 prefixlen:128 scope:host
UP LOOPBACK mtu:65536 index:1
eth0
inet 162.13.11.152 netmask:255.255.255.0 broadcast:162.13.11.255
inet6 fe80::be76:4eff:fe08:4539 prefixlen:64 scope:link
inet6 2a00:1a48:7806:114:be76:4eff:fe08:4539 prefixlen:64
hardware BC:76:4E:08:45:39
UP MULTICAST mtu:1500 index:2
eth1
inet 10.179.128.71 netmask:255.255.192.0 broadcast:10.179.191.255 scope:site
inet6 fe80::be76:4eff:fe08:4762 prefixlen:64 scope:link
hardware BC:76:4E:08:47:62
UP MULTICAST mtu:1500 index:3
eth2
inet 192.168.5.2 netmask:255.255.255.0 broadcast:192.168.5.255 scope:site
inet6 fe80::be76:4eff:fe08:b6a5 prefixlen:64 scope:link
hardware BC:76:4E:08:B6:A5
UP MULTICAST mtu:1500 index:4
[2016-06-29 22:15:27,658][DEBUG][common.netty ] using gathering [true]
[2016-06-29 22:15:27,752][DEBUG][discovery.zen.elect ] [nextkibana01-ops] using minimum_master_nodes [2]
[2016-06-29 22:15:27,754][DEBUG][discovery.zen.ping.unicast] [nextkibana01-ops] using initial hosts [nextkibana02-ops.es.betable.com:9300, nextkibana03-ops.es.betable.com:9300], with concurrent_connects [10]
[2016-06-29 22:15:27,810][DEBUG][discovery.zen ] [nextkibana01-ops] using ping.timeout [3s], join.timeout [1m], master_election.filter_client [true], master_election.filter_data [false]
[2016-06-29 22:15:27,812][DEBUG][discovery.zen.fd ] [nextkibana01-ops] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2016-06-29 22:15:27,815][DEBUG][discovery.zen.fd ] [nextkibana01-ops] [node ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2016-06-29 22:15:27,900][DEBUG][cluster.routing.allocation.decider] [nextkibana01-ops] using node_concurrent_recoveries [2], node_initial_primaries_recoveries [4]
[2016-06-29 22:15:27,904][DEBUG][cluster.routing.allocation.decider] [nextkibana01-ops] using [cluster.routing.allocation.allow_rebalance] with [indices_all_active]
[2016-06-29 22:15:27,907][DEBUG][monitor.jvm ] [nextkibana01-ops] enabled [true], interval [1s], gc_threshold [{default=GcThreshold{name=‘default’, warnThreshold=10000, infoThreshold=5000, debugThreshold=2000}, young=GcThreshold{name=‘young’, warnThreshold=1000, infoThreshold=700, debugThreshold=400}, old=GcThreshold{name=‘old’, warnThreshold=10000, infoThreshold=5000, debugThreshold=2000}}]
[2016-06-29 22:15:27,915][DEBUG][monitor.os ] [nextkibana01-ops] Using probe [org.elasticsearch.monitor.os.OsProbe@53ce2392] with refresh_interval [1s]
[2016-06-29 22:15:27,922][DEBUG][monitor.process ] [nextkibana01-ops] Using probe [org.elasticsearch.monitor.process.ProcessProbe@1860f3be] with refresh_interval [1s]
[2016-06-29 22:15:27,938][DEBUG][monitor.jvm ] [nextkibana01-ops] Using refresh_interval [1s]
[2016-06-29 22:15:27,943][DEBUG][monitor.fs ] [nextkibana01-ops] Using probe [org.elasticsearch.monitor.fs.FsProbe@20a967fe] with refresh_interval [1s]
[2016-06-29 22:15:28,183][DEBUG][script ] [nextkibana01-ops] using script cache with max_size [100], expire [null]
[2016-06-29 22:15:28,340][DEBUG][cluster.routing.allocation.decider] [nextkibana01-ops] using [cluster_concurrent_rebalance] with [2]
[2016-06-29 22:15:28,345][DEBUG][gateway ] [nextkibana01-ops] using initial_shards [quorum]
[2016-06-29 22:15:28,707][DEBUG][http.netty ] [nextkibana01-ops] using max_chunk_size[8kb], max_header_size[8kb], max_initial_line_length[4kb], max_content_length[100mb], receive_predictor[512kb->512kb], pipelining[true], pipelining_max_events[10000]
[2016-06-29 22:15:28,732][DEBUG][indices.recovery ] [nextkibana01-ops] using max_bytes_per_sec[40mb], concurrent_streams [3], file_chunk_size [512kb], translog_size [512kb], translog_ops [1000], and compress [true]
[2016-06-29 22:15:28,745][DEBUG][indices.store ] [nextkibana01-ops] using indices.store.throttle.type [NONE], with index.store.throttle.max_bytes_per_sec [10gb]
[2016-06-29 22:15:28,746][DEBUG][indices.memory ] [nextkibana01-ops] using indexing buffer size [406.2mb], with indices.memory.min_shard_index_buffer_size [4mb], indices.memory.max_shard_index_buffer_size [512mb], indices.memory.shard_inactive_time [5m], indices.memory.interval [30s]
[2016-06-29 22:15:28,749][DEBUG][indices.cache.query ] [nextkibana01-ops] using [node] query cache with size [10%], actual_size [406.2mb], max filter count [1000]
[2016-06-29 22:15:28,753][DEBUG][indices.fielddata.cache ] [nextkibana01-ops] using size [-1] [-1b]
[2016-06-29 22:15:28,912][DEBUG][common.compress.lzf ] using decoder[VanillaChunkDecoder]
[2016-06-29 22:15:31,491][DEBUG][gateway ] [nextkibana01-ops] took 614ms to load state
[2016-06-29 22:15:31,508][INFO ][node ] [nextkibana01-ops] initialized
[2016-06-29 22:15:31,508][INFO ][node ] [nextkibana01-ops] starting …
[2016-06-29 22:15:31,613][INFO ][com.automattic.elasticsearch.statsd.StatsdService] [nextkibana01-ops] StatsD reporting triggered every [10s] to host [10.177.193.128:8125] with metric prefix [elasticsearch.staging-elasticsearch]
[2016-06-29 22:15:31,647][DEBUG][netty.channel.socket.nio.SelectorUtil] Using select timeout of 500
[2016-06-29 22:15:31,647][DEBUG][netty.channel.socket.nio.SelectorUtil] Epoll-bug workaround enabled = false
[2016-06-29 22:15:31,806][INFO ][discovery ] [nextkibana01-ops] staging-elasticsearch/WfZPVvTVQCmFybPd0DNbJw
[2016-06-29 22:15:31,818][DEBUG][cluster.service ] [nextkibana01-ops] processing [initial_join]: execute
[2016-06-29 22:15:31,830][DEBUG][cluster.service ] [nextkibana01-ops] processing [initial_join]: took 11ms no change in cluster_state
[2016-06-29 22:15:31,839][DEBUG][action.admin.cluster.health] [nextkibana01-ops] no known master node, scheduling a retry
[2016-06-29 22:15:31,966][DEBUG][io.netty.buffer.PooledByteBufAllocator] -Dio.netty.allocator.numHeapArenas: 8
[2016-06-29 22:15:31,966][DEBUG][io.netty.buffer.PooledByteBufAllocator] -Dio.netty.allocator.numDirectArenas: 8
[2016-06-29 22:15:31,966][DEBUG][io.netty.buffer.PooledByteBufAllocator] -Dio.netty.allocator.pageSize: 8192
[2016-06-29 22:15:31,966][DEBUG][io.netty.buffer.PooledByteBufAllocator] -Dio.netty.allocator.maxOrder: 11
[2016-06-29 22:15:31,966][DEBUG][io.netty.buffer.PooledByteBufAllocator] -Dio.netty.allocator.chunkSize: 16777216
[2016-06-29 22:15:31,966][DEBUG][io.netty.buffer.PooledByteBufAllocator] -Dio.netty.allocator.tinyCacheSize: 512
[2016-06-29 22:15:31,966][DEBUG][io.netty.buffer.PooledByteBufAllocator] -Dio.netty.allocator.smallCacheSize: 256
[2016-06-29 22:15:31,966][DEBUG][io.netty.buffer.PooledByteBufAllocator] -Dio.netty.allocator.normalCacheSize: 64
[2016-06-29 22:15:31,966][DEBUG][io.netty.buffer.PooledByteBufAllocator] -Dio.netty.allocator.maxCachedBufferCapacity: 32768
[2016-06-29 22:15:31,966][DEBUG][io.netty.buffer.PooledByteBufAllocator] -Dio.netty.allocator.cacheTrimInterval: 8192
[2016-06-29 22:15:32,341][DEBUG][netty.handler.ssl.SslHandler] [id: 0xb444956c, /192.168.5.2:22096 => nextkibana02-ops.es.betable.com/192.168.5.1:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:32,366][DEBUG][netty.handler.ssl.SslHandler] [id: 0x5d618790, /192.168.5.2:20121 => nextkibana03-ops.es.betable.com/192.168.5.3:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:34,969][DEBUG][discovery.zen ] [nextkibana01-ops] filtered ping responses: (filter_client[true], filter_data[false])
→ ping_response{node [{nextkibana03-ops}{xmQZ0VNFRQqyhXepUNnqjg}{192.168.5.3}{192.168.5.3:9300}], id[20], master [{nextkibana02-ops}{LkwXBR-YSQGnID1qluh5ug}{192.168.5.1}{192.168.5.1:9300}], hasJoinedOnce [true], cluster_name[staging-elasticsearch]}
→ ping_response{node [{nextkibana02-ops}{LkwXBR-YSQGnID1qluh5ug}{192.168.5.1}{192.168.5.1:9300}], id[11], master [{nextkibana02-ops}{LkwXBR-YSQGnID1qluh5ug}{192.168.5.1}{192.168.5.1:9300}], hasJoinedOnce [true], cluster_name[staging-elasticsearch]}
[2016-06-29 22:15:35,227][DEBUG][netty.handler.ssl.SslHandler] [id: 0xbe0498a6, /192.168.5.2:22100 => 192.168.5.1/192.168.5.1:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,236][DEBUG][netty.handler.ssl.SslHandler] [id: 0x9de8bf67, /192.168.5.2:22108 => 192.168.5.1/192.168.5.1:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,283][DEBUG][netty.handler.ssl.SslHandler] [id: 0x2a94fedb, /192.168.5.2:22102 => 192.168.5.1/192.168.5.1:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,290][DEBUG][netty.handler.ssl.SslHandler] [id: 0x06a7ba5e, /192.168.5.2:22106 => 192.168.5.1/192.168.5.1:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,293][DEBUG][netty.handler.ssl.SslHandler] [id: 0x08e2f270, /192.168.5.2:22107 => 192.168.5.1/192.168.5.1:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,297][DEBUG][netty.handler.ssl.SslHandler] [id: 0x6defea1b, /192.168.5.2:22101 => 192.168.5.1/192.168.5.1:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,302][DEBUG][netty.handler.ssl.SslHandler] [id: 0xa7fbcb7f, /192.168.5.2:22103 => 192.168.5.1/192.168.5.1:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,305][DEBUG][netty.handler.ssl.SslHandler] [id: 0x112dc3e9, /192.168.5.2:22109 => 192.168.5.1/192.168.5.1:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,320][DEBUG][netty.handler.ssl.SslHandler] [id: 0x58971d19, /192.168.5.2:22105 => 192.168.5.1/192.168.5.1:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,322][DEBUG][netty.handler.ssl.SslHandler] [id: 0x0d4ae96f, /192.168.5.2:22110 => 192.168.5.1/192.168.5.1:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,330][DEBUG][netty.handler.ssl.SslHandler] [id: 0x64d88caa, /192.168.5.2:22111 => 192.168.5.1/192.168.5.1:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,365][DEBUG][netty.handler.ssl.SslHandler] [id: 0x458d2a7f, /192.168.5.2:22104 => 192.168.5.1/192.168.5.1:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,404][DEBUG][netty.handler.ssl.SslHandler] [id: 0xb3229c81, /192.168.5.2:22112 => 192.168.5.1/192.168.5.1:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,508][DEBUG][netty.handler.ssl.SslHandler] [id: 0x21791476, /192.168.5.1:21599 => /192.168.5.2:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,568][DEBUG][netty.handler.ssl.SslHandler] [id: 0xeda4df8f, /192.168.5.1:21600 => /192.168.5.2:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,569][DEBUG][netty.handler.ssl.SslHandler] [id: 0xcf9010a4, /192.168.5.1:21595 => /192.168.5.2:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,577][DEBUG][netty.handler.ssl.SslHandler] [id: 0x76e700d6, /192.168.5.1:21593 => /192.168.5.2:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,577][DEBUG][netty.handler.ssl.SslHandler] [id: 0x79ecd814, /192.168.5.1:21596 => /192.168.5.2:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,578][DEBUG][netty.handler.ssl.SslHandler] [id: 0x3192c05e, /192.168.5.1:21594 => /192.168.5.2:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,582][DEBUG][netty.handler.ssl.SslHandler] [id: 0x74ce941a, /192.168.5.1:21597 => /192.168.5.2:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,595][DEBUG][netty.handler.ssl.SslHandler] [id: 0x61c2a5da, /192.168.5.1:21598 => /192.168.5.2:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,611][DEBUG][netty.handler.ssl.SslHandler] [id: 0x3dcd8390, /192.168.5.1:21601 => /192.168.5.2:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,659][DEBUG][netty.handler.ssl.SslHandler] [id: 0x810bd9cc, /192.168.5.1:21604 => /192.168.5.2:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,667][DEBUG][netty.handler.ssl.SslHandler] [id: 0xf5e45292, /192.168.5.1:21602 => /192.168.5.2:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,669][DEBUG][netty.handler.ssl.SslHandler] [id: 0x822509ad, /192.168.5.1:21605 => /192.168.5.2:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,688][DEBUG][discovery.zen.publish ] [nextkibana01-ops] received full cluster state version 2 with size 283
[2016-06-29 22:15:35,690][DEBUG][cluster.service ] [nextkibana01-ops] processing [zen-disco-receive(from master [{nextkibana02-ops}{LkwXBR-YSQGnID1qluh5ug}{192.168.5.1}{192.168.5.1:9300}])]: execute
[2016-06-29 22:15:35,700][DEBUG][netty.handler.ssl.SslHandler] [id: 0x37b51eaf, /192.168.5.1:21603 => /192.168.5.2:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,690][DEBUG][discovery.zen.fd ] [nextkibana01-ops] [master] restarting fault detection against master [{nextkibana02-ops}{LkwXBR-YSQGnID1qluh5ug}{192.168.5.1}{192.168.5.1:9300}], reason [new cluster state received and we are monitoring the wrong master [null]]
[2016-06-29 22:15:35,748][DEBUG][discovery.zen ] [nextkibana01-ops] got first state from fresh master [LkwXBR-YSQGnID1qluh5ug]
[2016-06-29 22:15:35,749][DEBUG][cluster.service ] [nextkibana01-ops] cluster state updated, version [2], source [zen-disco-receive(from master [{nextkibana02-ops}{LkwXBR-YSQGnID1qluh5ug}{192.168.5.1}{192.168.5.1:9300}])]
[2016-06-29 22:15:35,750][INFO ][cluster.service ] [nextkibana01-ops] detected_master {nextkibana02-ops}{LkwXBR-YSQGnID1qluh5ug}{192.168.5.1}{192.168.5.1:9300}, added {{nextkibana02-ops}{LkwXBR-YSQGnID1qluh5ug}{192.168.5.1}{192.168.5.1:9300},{nextkibana03-ops}{xmQZ0VNFRQqyhXepUNnqjg}{192.168.5.3}{192.168.5.3:9300},}, reason: zen-disco-receive(from master [{nextkibana02-ops}{LkwXBR-YSQGnID1qluh5ug}{192.168.5.1}{192.168.5.1:9300}])
[2016-06-29 22:15:35,824][DEBUG][cluster.service ] [nextkibana01-ops] set local cluster state to version 2
[2016-06-29 22:15:35,841][DEBUG][netty.handler.ssl.SslHandler] [id: 0xc8bec71f, /192.168.5.3:40982 => /192.168.5.2:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,853][DEBUG][cluster.service ] [nextkibana01-ops] processing [zen-disco-receive(from master [{nextkibana02-ops}{LkwXBR-YSQGnID1qluh5ug}{192.168.5.1}{192.168.5.1:9300}])]: took 163ms done applying updated cluster_state (version: 2, uuid: hLhh6c-XTvSR8lNJdAD0Yg)
[2016-06-29 22:15:35,859][DEBUG][cluster.service ] [nextkibana01-ops] processing [finalize_join ({nextkibana02-ops}{LkwXBR-YSQGnID1qluh5ug}{192.168.5.1}{192.168.5.1/192.168.5.1:9300})]: execute
[2016-06-29 22:15:35,860][DEBUG][cluster.service ] [nextkibana01-ops] processing [finalize_join ({nextkibana02-ops}{LkwXBR-YSQGnID1qluh5ug}{192.168.5.1}{192.168.5.1/192.168.5.1:9300})]: took 0s no change in cluster_state
[2016-06-29 22:15:35,902][DEBUG][netty.handler.ssl.SslHandler] [id: 0xad7b9473, /192.168.5.3:40988 => /192.168.5.2:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,906][DEBUG][netty.handler.ssl.SslHandler] [id: 0x0a82fad2, /192.168.5.3:40994 => /192.168.5.2:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,918][DEBUG][netty.handler.ssl.SslHandler] [id: 0x56023719, /192.168.5.3:40989 => /192.168.5.2:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,922][DEBUG][netty.handler.ssl.SslHandler] [id: 0x8c755c8c, /192.168.5.3:40987 => /192.168.5.2:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,957][DEBUG][netty.handler.ssl.SslHandler] [id: 0x86ac6998, /192.168.5.3:40984 => /192.168.5.2:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,959][DEBUG][netty.handler.ssl.SslHandler] [id: 0x1934ffb1, /192.168.5.3:40985 => /192.168.5.2:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,985][DEBUG][netty.handler.ssl.SslHandler] [id: 0xc38231d5, /192.168.5.2:20147 => 192.168.5.3/192.168.5.3:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,994][DEBUG][netty.handler.ssl.SslHandler] [id: 0xc2dad193, /192.168.5.3:40992 => /192.168.5.2:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:35,997][DEBUG][netty.handler.ssl.SslHandler] [id: 0xe8146bf9, /192.168.5.2:20146 => 192.168.5.3/192.168.5.3:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:36,009][DEBUG][netty.handler.ssl.SslHandler] [id: 0xdc593065, /192.168.5.3:40983 => /192.168.5.2:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:36,020][DEBUG][netty.handler.ssl.SslHandler] [id: 0x3547ef97, /192.168.5.2:20148 => 192.168.5.3/192.168.5.3:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:36,032][DEBUG][netty.handler.ssl.SslHandler] [id: 0xc4ac3c20, /192.168.5.3:40986 => /192.168.5.2:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:36,032][DEBUG][netty.handler.ssl.SslHandler] [id: 0x7e2e008b, /192.168.5.3:40991 => /192.168.5.2:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:36,034][DEBUG][netty.handler.ssl.SslHandler] [id: 0xcb447657, /192.168.5.3:40993 => /192.168.5.2:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:36,050][DEBUG][http.netty ] [nextkibana01-ops] Bound http to address {127.0.0.1:9200}
[2016-06-29 22:15:36,050][INFO ][http ] [nextkibana01-ops] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[2016-06-29 22:15:36,050][INFO ][node ] [nextkibana01-ops] started
[2016-06-29 22:15:36,076][DEBUG][netty.handler.ssl.SslHandler] [id: 0xa41c70a6, /192.168.5.3:40990 => /192.168.5.2:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:36,120][DEBUG][netty.handler.ssl.SslHandler] [id: 0xc8fd852f, /192.168.5.2:20142 => 192.168.5.3/192.168.5.3:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:36,127][DEBUG][netty.handler.ssl.SslHandler] [id: 0x65c5baeb, /192.168.5.2:20150 => 192.168.5.3/192.168.5.3:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:36,184][DEBUG][netty.handler.ssl.SslHandler] [id: 0xa052a38d, /192.168.5.2:20151 => 192.168.5.3/192.168.5.3:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:36,186][DEBUG][netty.handler.ssl.SslHandler] [id: 0x96be3613, /192.168.5.2:20145 => 192.168.5.3/192.168.5.3:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:36,193][DEBUG][netty.handler.ssl.SslHandler] [id: 0xe3db3f12, /192.168.5.2:20152 => 192.168.5.3/192.168.5.3:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:36,195][DEBUG][netty.handler.ssl.SslHandler] [id: 0xe9d7720c, /192.168.5.2:20141 => 192.168.5.3/192.168.5.3:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:36,206][DEBUG][netty.handler.ssl.SslHandler] [id: 0x9c0b0cc4, /192.168.5.2:20143 => 192.168.5.3/192.168.5.3:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:36,207][DEBUG][netty.handler.ssl.SslHandler] [id: 0x44531552, /192.168.5.2:20149 => 192.168.5.3/192.168.5.3:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:36,208][DEBUG][netty.handler.ssl.SslHandler] [id: 0xb4477ebd, /192.168.5.2:20144 => 192.168.5.3/192.168.5.3:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:36,210][DEBUG][netty.handler.ssl.SslHandler] [id: 0x8c13a1ba, /192.168.5.2:20153 => 192.168.5.3/192.168.5.3:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2016-06-29 22:15:37,219][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.29][0] shard state info found: [version [3], primary [false]]
[2016-06-29 22:15:37,217][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.29][3] shard state info found: [version [3], primary [false]]
[2016-06-29 22:15:37,237][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.29][2] shard state info found: [version [3], primary [true]]
[2016-06-29 22:15:37,258][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.28][1] shard state info found: [version [43], primary [false]]
[2016-06-29 22:15:37,258][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.27][0] shard state info found: [version [51], primary [true]]
[2016-06-29 22:15:37,264][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.28][3] shard state info found: [version [31], primary [true]]
[2016-06-29 22:15:37,268][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.27][1] shard state info found: [version [44], primary [false]]
[2016-06-29 22:15:37,270][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.28][2] shard state info found: [version [29], primary [true]]
[2016-06-29 22:15:37,284][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.27][2] shard state info found: [version [45], primary [true]]
[2016-06-29 22:15:37,288][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.26][2] shard state info found: [version [45], primary [false]]
[2016-06-29 22:15:37,288][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.26][3] shard state info found: [version [42], primary [false]]
[2016-06-29 22:15:37,317][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.26][0] shard state info found: [version [48], primary [true]]
[2016-06-29 22:15:37,320][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.25][2] shard state info found: [version [45], primary [true]]
[2016-06-29 22:15:37,337][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.24][0] shard state info found: [version [55], primary [true]]
[2016-06-29 22:15:37,337][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.24][3] shard state info found: [version [50], primary [true]]
[2016-06-29 22:15:37,356][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.25][1] shard state info found: [version [44], primary [false]]
[2016-06-29 22:15:37,357][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.25][0] shard state info found: [version [43], primary [true]]
[2016-06-29 22:15:37,364][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.24][1] shard state info found: [version [54], primary [true]]
[2016-06-29 22:15:37,365][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.23][1] shard state info found: [version [59], primary [false]]
[2016-06-29 22:15:37,372][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.23][4] shard state info found: [version [58], primary [true]]
[2016-06-29 22:15:37,373][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.23][3] shard state info found: [version [55], primary [false]]
[2016-06-29 22:15:37,375][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.22][3] shard state info found: [version [57], primary [true]]
[2016-06-29 22:15:37,378][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.22][4] shard state info found: [version [56], primary [true]]
[2016-06-29 22:15:37,379][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.22][2] shard state info found: [version [57], primary [true]]
[2016-06-29 22:15:37,388][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.21][3] shard state info found: [version [57], primary [true]]
[2016-06-29 22:15:37,408][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.23][2] shard state info found: [version [57], primary [false]]
[2016-06-29 22:15:37,411][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.21][4] shard state info found: [version [58], primary [true]]
[2016-06-29 22:15:37,428][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.19][2] shard state info found: [version [57], primary [true]]
[2016-06-29 22:15:37,441][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.21][2] shard state info found: [version [58], primary [false]]
[2016-06-29 22:15:37,442][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.19][0] shard state info found: [version [61], primary [true]]
[2016-06-29 22:15:37,452][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.19][4] shard state info found: [version [58], primary [true]]
[2016-06-29 22:15:37,458][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.18][2] shard state info found: [version [56], primary [true]]
[2016-06-29 22:15:37,463][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.18][0] shard state info found: [version [64], primary [true]]
[2016-06-29 22:15:37,473][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.18][4] shard state info found: [version [55], primary [true]]
[2016-06-29 22:15:37,474][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.17][1] shard state info found: [version [56], primary [false]]
[2016-06-29 22:15:37,474][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.17][4] shard state info found: [version [58], primary [true]]
[2016-06-29 22:15:37,484][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.17][3] shard state info found: [version [55], primary [true]]
[2016-06-29 22:15:37,490][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.15][1] shard state info found: [version [60], primary [true]]
[2016-06-29 22:15:37,492][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.15][4] shard state info found: [version [55], primary [false]]
[2016-06-29 22:15:37,508][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.14][3] shard state info found: [version [65], primary [true]]
[2016-06-29 22:15:37,510][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.14][4] shard state info found: [version [64], primary [true]]
[2016-06-29 22:15:37,521][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.15][2] shard state info found: [version [56], primary [true]]
[2016-06-29 22:15:37,541][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.14][1] shard state info found: [version [68], primary [false]]
[2016-06-29 22:15:37,542][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.13][2] shard state info found: [version [70], primary [false]]
[2016-06-29 22:15:37,581][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.13][3] shard state info found: [version [72], primary [true]]
[2016-06-29 22:15:37,581][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.12][2] shard state info found: [version [67], primary [true]]
[2016-06-29 22:15:37,586][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.12][3] shard state info found: [version [69], primary [false]]
[2016-06-29 22:15:37,587][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.11][1] shard state info found: [version [68], primary [false]]
[2016-06-29 22:15:37,588][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.13][4] shard state info found: [version [69], primary [false]]
[2016-06-29 22:15:37,592][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.12][4] shard state info found: [version [67], primary [true]]
[2016-06-29 22:15:37,596][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.11][3] shard state info found: [version [69], primary [true]]
[2016-06-29 22:15:37,597][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.11][2] shard state info found: [version [65], primary [false]]
[2016-06-29 22:15:37,608][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.10][2] shard state info found: [version [76], primary [false]]
[2016-06-29 22:15:37,614][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.10][4] shard state info found: [version [79], primary [true]]
[2016-06-29 22:15:37,622][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.10][3] shard state info found: [version [75], primary [false]]
[2016-06-29 22:15:37,622][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.09][2] shard state info found: [version [76], primary [false]]
[2016-06-29 22:15:37,627][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.11][4] shard state info found: [version [69], primary [true]]
[2016-06-29 22:15:37,627][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.09][3] shard state info found: [version [74], primary [true]]
[2016-06-29 22:15:37,630][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.09][4] shard state info found: [version [75], primary [false]]
[2016-06-29 22:15:37,630][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.09][1] shard state info found: [version [81], primary [false]]
[2016-06-29 22:15:37,632][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.08][4] shard state info found: [version [75], primary [true]]
[2016-06-29 22:15:37,638][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.08][0] shard state info found: [version [85], primary [false]]
[2016-06-29 22:15:37,640][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.08][1] shard state info found: [version [86], primary [true]]
[2016-06-29 22:15:37,642][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.08][2] shard state info found: [version [72], primary [false]]
[2016-06-29 22:15:37,652][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.07][3] shard state info found: [version [78], primary [false]]
[2016-06-29 22:15:37,658][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.06][2] shard state info found: [version [78], primary [false]]
[2016-06-29 22:15:37,661][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.07][1] shard state info found: [version [80], primary [true]]
[2016-06-29 22:15:37,664][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.07][4] shard state info found: [version [74], primary [true]]
[2016-06-29 22:15:37,673][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.06][0] shard state info found: [version [80], primary [true]]
[2016-06-29 22:15:37,676][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.06][3] shard state info found: [version [77], primary [true]]
[2016-06-29 22:15:37,676][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.06][4] shard state info found: [version [71], primary [false]]
[2016-06-29 22:15:37,683][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.05][4] shard state info found: [version [74], primary [true]]
[2016-06-29 22:15:37,688][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.05][2] shard state info found: [version [75], primary [false]]
[2016-06-29 22:15:37,692][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.05][3] shard state info found: [version [68], primary [false]]
[2016-06-29 22:15:37,702][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.05][1] shard state info found: [version [80], primary [false]]
[2016-06-29 22:15:37,705][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.04][0] shard state info found: [version [81], primary [false]]
[2016-06-29 22:15:37,708][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.04][3] shard state info found: [version [75], primary [true]]
[2016-06-29 22:15:37,709][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.04][2] shard state info found: [version [74], primary [true]]
[2016-06-29 22:15:37,710][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.04][1] shard state info found: [version [69], primary [false]]
[2016-06-29 22:15:37,733][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.03][1] shard state info found: [version [77], primary [true]]
[2016-06-29 22:15:37,742][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.03][3] shard state info found: [version [73], primary [true]]
[2016-06-29 22:15:37,745][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.02][2] shard state info found: [version [77], primary [false]]
[2016-06-29 22:15:37,747][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.03][2] shard state info found: [version [70], primary [false]]
[2016-06-29 22:15:37,748][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.02][4] shard state info found: [version [72], primary [false]]
[2016-06-29 22:15:37,749][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.01][4] shard state info found: [version [70], primary [false]]
[2016-06-29 22:15:37,750][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.01][2] shard state info found: [version [73], primary [true]]
[2016-06-29 22:15:37,763][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.01][3] shard state info found: [version [72], primary [true]]
[2016-06-29 22:15:37,763][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.02][0] shard state info found: [version [79], primary [false]]
[2016-06-29 22:15:37,764][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.06.03][4] shard state info found: [version [74], primary [false]]
[2016-06-29 22:15:37,775][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.31][4] shard state info found: [version [70], primary [false]]
[2016-06-29 22:15:37,776][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.31][1] shard state info found: [version [77], primary [true]]
[2016-06-29 22:15:37,782][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.31][3] shard state info found: [version [71], primary [false]]
[2016-06-29 22:15:37,787][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.30][2] shard state info found: [version [71], primary [false]]
[2016-06-29 22:15:37,794][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.30][0] shard state info found: [version [78], primary [false]]
[2016-06-29 22:15:37,814][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.30][1] shard state info found: [version [75], primary [false]]
[2016-06-29 22:15:37,814][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.29][4] shard state info found: [version [70], primary [false]]
[2016-06-29 22:15:37,816][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.28][4] shard state info found: [version [71], primary [true]]
[2016-06-29 22:15:37,816][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.28][1] shard state info found: [version [71], primary [true]]
[2016-06-29 22:15:37,822][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.29][3] shard state info found: [version [72], primary [false]]
[2016-06-29 22:15:37,822][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.29][2] shard state info found: [version [71], primary [true]]
[2016-06-29 22:15:37,828][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.27][3] shard state info found: [version [72], primary [false]]
[2016-06-29 22:15:37,829][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.27][1] shard state info found: [version [73], primary [true]]
[2016-06-29 22:15:37,831][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.25][0] shard state info found: [version [71], primary [false]]
[2016-06-29 22:15:37,833][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.24][2] shard state info found: [version [70], primary [true]]
[2016-06-29 22:15:37,836][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.27][0] shard state info found: [version [76], primary [false]]
[2016-06-29 22:15:37,838][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.25][3] shard state info found: [version [68], primary [true]]
[2016-06-29 22:15:37,840][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.28][3] shard state info found: [version [70], primary [false]]
[2016-06-29 22:15:37,843][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.25][4] shard state info found: [version [71], primary [true]]
[2016-06-29 22:15:37,854][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.24][3] shard state info found: [version [74], primary [false]]
[2016-06-29 22:15:37,879][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.25][1] shard state info found: [version [77], primary [false]]
[2016-06-29 22:15:37,879][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.24][0] shard state info found: [version [79], primary [false]]
[2016-06-29 22:15:37,881][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.09][1] shard state info found: [version [82], primary [false]]
[2016-06-29 22:15:37,888][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.08][2] shard state info found: [version [80], primary [true]]
[2016-06-29 22:15:37,888][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.07][2] shard state info found: [version [82], primary [true]]
[2016-06-29 22:15:37,895][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.09][3] shard state info found: [version [81], primary [true]]
[2016-06-29 22:15:37,910][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.08][3] shard state info found: [version [81], primary [false]]
[2016-06-29 22:15:37,910][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.07][4] shard state info found: [version [77], primary [false]]
[2016-06-29 22:15:37,910][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.09][2] shard state info found: [version [82], primary [false]]
[2016-06-29 22:15:37,912][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.07][3] shard state info found: [version [83], primary [false]]
[2016-06-29 22:15:37,912][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.01][1] shard state info found: [version [83], primary [false]]
[2016-06-29 22:15:37,916][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.01][3] shard state info found: [version [83], primary [true]]
[2016-06-29 22:15:37,917][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.04.17][1] shard state info found: [version [119], primary [false]]
[2016-06-29 22:15:37,920][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.04.17][0] shard state info found: [version [132], primary [false]]
[2016-06-29 22:15:37,921][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.08][0] shard state info found: [version [83], primary [true]]
[2016-06-29 22:15:37,926][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.04.17][2] shard state info found: [version [122], primary [true]]
[2016-06-29 22:15:37,927][DEBUG][gateway ] [nextkibana01-ops] [logstash-2016.05.01][0] shard state info found: [version [80], primary [false]]
[2016-06-29 22:15:38,261][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-06-29 22:15:38,265][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-06-29 22:15:38,481][DEBUG][discovery.zen.publish ] [nextkibana01-ops] received diff cluster state version 3 with uuid 0nLoZSpeRqKhCRVncdU6EA, diff size 974958
[2016-06-29 22:15:38,481][DEBUG][cluster.service ] [nextkibana01-ops] processing [zen-disco-receive(from master [{nextkibana02-ops}{LkwXBR-YSQGnID1qluh5ug}{192.168.5.1}{192.168.5.1:9300}])]: execute
[2016-06-29 22:15:38,482][DEBUG][cluster.service ] [nextkibana01-ops] cluster state updated, version [3], source [zen-disco-receive(from master [{nextkibana02-ops}{LkwXBR-YSQGnID1qluh5ug}{192.168.5.1}{192.168.5.1:9300}])]
[2016-06-29 22:15:38,482][DEBUG][cluster.service ] [nextkibana01-ops] set local cluster state to version 3
[2016-06-29 22:15:39,359][DEBUG][cluster.service ] [nextkibana01-ops] processing [zen-disco-receive(from master [{nextkibana02-ops}{LkwXBR-YSQGnID1qluh5ug}{192.168.5.1}{192.168.5.1:9300}])]: took 877ms done applying updated cluster_state (version: 3, uuid: 0nLoZSpeRqKhCRVncdU6EA)
[2016-06-29 22:15:40,338][DEBUG][discovery.zen.publish ] [nextkibana01-ops] received diff cluster state version 4 with uuid -ZDTRVbeSU6LkwNroz7uqw, diff size 653
[2016-06-29 22:15:40,343][DEBUG][cluster.service ] [nextkibana01-ops] processing [zen-disco-receive(from master [{nextkibana02-ops}{LkwXBR-YSQGnID1qluh5ug}{192.168.5.1}{192.168.5.1:9300}])]: execute
[2016-06-29 22:15:40,343][DEBUG][cluster.service ] [nextkibana01-ops] cluster state updated, version [4], source [zen-disco-receive(from master [{nextkibana02-ops}{LkwXBR-YSQGnID1qluh5ug}{192.168.5.1}{192.168.5.1:9300}])]
[2016-06-29 22:15:40,344][DEBUG][cluster.service ] [nextkibana01-ops] set local cluster state to version 4
[2016-06-29 22:15:40,345][DEBUG][indices.cluster ] [nextkibana01-ops] [logstash-2016.06.28] creating index
[2016-06-29 22:15:40,348][DEBUG][indices ] [nextkibana01-ops] creating Index [logstash-2016.06.28], shards [5]/[1]
[2016-06-29 22:15:40,432][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-06-29 22:15:40,642][DEBUG][index.store ] [nextkibana01-ops] [logstash-2016.06.28] using index.store.throttle.type [none], with index.store.throttle.max_bytes_per_sec [0b]
[2016-06-29 22:15:40,676][DEBUG][index.mapper ] [nextkibana01-ops] [logstash-2016.06.28] using dynamic[true]
[2016-06-29 22:15:40,690][DEBUG][indices.cluster ] [nextkibana01-ops] [logstash-2016.06.29] creating index
[2016-06-29 22:15:40,690][DEBUG][indices ] [nextkibana01-ops] creating Index [logstash-2016.06.29], shards [5]/[1]
[2016-06-29 22:15:40,706][DEBUG][index.store ] [nextkibana01-ops] [logstash-2016.06.29] using index.store.throttle.type [none], with index.store.throttle.max_bytes_per_sec [0b]
[2016-06-29 22:15:40,712][DEBUG][index.mapper ] [nextkibana01-ops] [logstash-2016.06.29] using dynamic[true]

===========================================

you can see the backendregistry errors starting to crop up.

On Wednesday, June 29, 2016 at 2:54:24 PM UTC-7, Max Furman wrote:

If I start the cluster w/out searchguard-2 and searchguard-ssl installed, and with the searchguard elasticsearch.yml lines removed, then the cluster starts up just fine and I have no problem accessing the node via http. If, however, I start with all the searchguard stuffs, then the nodes in the cluster are able to find one another, but any communication beyond that seems broken.

Below is a copy of my elasticsearch.yml:

======================== Elasticsearch Configuration =========================

NOTE: Elasticsearch comes with reasonable defaults for most settings.

Before you set out to tweak and tune the configuration, make sure you

understand what are you trying to accomplish and the consequences.

The primary way of configuring a node is via this file. This template lists

the most important settings you may want to configure for a production cluster.

Please see the documentation for further information on configuration options:

<http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html>

---------------------------------- Cluster -----------------------------------

Use a descriptive name for your cluster:

cluster.name: staging-elasticsearch

------------------------------------ Node ------------------------------------

Use a descriptive name for the node:

node.name: nextkibana02-ops

Add custom attributes to the node:

node.rack: r1

----------------------------------- Paths ------------------------------------

Path to directory where to store the data (separate multiple locations by comma):

path.data: /var/lib/elasticsearch

Path to log files:

path.logs: /var/log/elasticsearch

path.work: /var/run/elasticsearch

----------------------------------- Memory -----------------------------------

Lock the memory on startup:

bootstrap.mlockall: true

Make sure that the ES_HEAP_SIZE environment variable is set to about half the memory

available on the system and that the owner of the process is allowed to use this limit.

Elasticsearch performs poorly when the system is swapping the memory.

---------------------------------- Network -----------------------------------

Set the bind address to a specific IP (IPv4 or IPv6):

network.host: 1.2.3.4 (private-ip)

Set a custom host and port for HTTP:

http.host: 127.0.0.1

http.port: 9200

For more information, see the documentation at:

<http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html>

--------------------------------- Discovery ----------------------------------

Pass an initial list of hosts to perform discovery when new node is started:

The default list of hosts is [“127.0.0.1”, “[::1]”]

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping.unicast.hosts: [“nextkibana01-ops.es.betable.com:9300”,“nextkibana03-ops.es.betable.com:9300”]

Prevent the “split brain” by configuring the majority of nodes (total number of nodes / 2 + 1):

discovery.zen.minimum_master_nodes: 2

For more information, see the documentation at:

<http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html>

---------------------------------- Gateway -----------------------------------

Block initial recovery after a full cluster restart until N nodes are started:

gateway.recover_after_nodes: 2

gateway.expected_nodes: 3

gateway.recover_after_time: 5m

For more information, see the documentation at:

<http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-gateway.html>

---------------------------------- Various -----------------------------------

Disable starting multiple nodes on a single system:

node.max_local_storage_nodes: 1

Require explicit names when deleting indices:

action.destructive_requires_name: true

action.disable_delete_all_indices: true

metrics.statsd.host: 1.2.3.4

metrics.statsd.every: 10s

threadpool.bulk.type: fixed

threadpool.bulk.size: 4 # availableProcessors

threadpool.bulk.queue_size: 250

index:

mapping:

ignore_malformed: true

analysis:

analyzer:

default:

type: standard

stopwords: none

id_field:

type: custom

tokenizer: keyword

filter: [lowercase]

Searchguard

#############################################################################################

SEARCH GUARD SSL

Configuration

#############################################################################################

searchguard.authcz.admin_dn:

ST=CA,O=Company,L=San Francisco,C=US,CN=staging-elasticsearch-admin

#############################################################################################

Transport layer SSL

#############################################################################################

Enable or disable node-to-node ssl encryption (default: true)

#searchguard.ssl.transport.enabled: true

JKS or PKCS12 (default: JKS)

searchguard.ssl.transport.keystore_type: JKS

Relative path to the keystore file (mandatory, this stores the server certificates), must be placed under the config/ dir

searchguard.ssl.transport.keystore_filepath: “/etc/elasticsearch/server-keystore”

Alias name (default: first alias which could be found)

#searchguard.ssl.transport.keystore_alias: my_alias

Keystore password (default: changeit)

searchguard.ssl.transport.keystore_password: password

JKS or PKCS12 (default: JKS)

searchguard.ssl.transport.truststore_type: JKS

Relative path to the truststore file (mandatory, this stores the client/root certificates), must be placed under the config/ dir

searchguard.ssl.transport.truststore_filepath: “/etc/elasticsearch/server-truststore”

Alias name (default: trust all aliases)

#searchguard.ssl.transport.truststore_alias: my_alias

Truststore password (default: changeit)

searchguard.ssl.transport.truststore_password: password

Enforce hostname verification (default: true)

#searchguard.ssl.transport.enforce_hostname_verification: false

If hostname verification specify if hostname should be resolved (default: true)

#searchguard.ssl.transport.resolve_hostname: true

Use native Open SSL instead of JDK SSL if available (default: true)

#searchguard.ssl.transport.enable_openssl_if_available: false

Enabled SSL cipher suites for transport protocol (only Java format is supported)

WARNING: Expert setting, do only use if you know what you are doing

If you set wrong values here this this could be a security risk

#searchguard.ssl.transport.enabled_ciphers:

- “TLS_DHE_RSA_WITH_AES_256_CBC_SHA”

- “TLS_DHE_DSS_WITH_AES_128_CBC_SHA256”

Enabled SSL protocols for transport protocol (only Java format is supported)

WARNING: Expert setting, do only use if you know what you are doing

If you set wrong values here this this could be a security risk

#searchguard.ssl.transport.enabled_protocols:

- “TLSv1.2”

#############################################################################################

HTTP/REST layer SSL

#############################################################################################

Enable or disable rest layer security - https, (default: false)

#searchguard.ssl.http.enabled: true

JKS or PKCS12 (default: JKS)

#searchguard.ssl.http.keystore_type: PKCS12

Relative path to the keystore file (this stores the server certificates), must be placed under the config/ dir

#searchguard.ssl.http.keystore_filepath: keystore_https_node1.jks

Alias name (default: first alias which could be found)

#searchguard.ssl.http.keystore_alias: my_alias

Keystore password (default: changeit)

#searchguard.ssl.http.keystore_password: changeit

Do the clients (typically the browser or the proxy) have to authenticate themself to the http server, default is OPTIONAL

To enforce authentication use REQUIRE, to completely disable client certificates use NONE

#searchguard.ssl.http.clientauth_mode: REQUIRE

JKS or PKCS12 (default: JKS)

#searchguard.ssl.http.truststore_type: PKCS12

Relative path to the truststore file (this stores the client certificates), must be placed under the config/ dir

#searchguard.ssl.http.truststore_filepath: truststore_https.jks

Alias name (default: first alias which could be found)

#searchguard.ssl.http.truststore_alias: my_alias

Truststore password (default: changeit)

#searchguard.ssl.http.truststore_password: changeit

Use native Open SSL instead of JDK SSL if available (default: true)

#searchguard.ssl.http.enable_openssl_if_available: false

Enabled SSL cipher suites for http protocol (only Java format is supported)

WARNING: Expert setting, do only use if you know what you are doing

If you set wrong values here this this could be a security risk

#searchguard.ssl.http.enabled_ciphers:

- “TLS_DHE_RSA_WITH_AES_256_CBC_SHA”

- “TLS_DHE_DSS_WITH_AES_128_CBC_SHA256”

Enabled SSL protocols for http protocol (only Java format is supported)

WARNING: Expert setting, do only use if you know what you are doing

If you set wrong values here this this could be a security risk

#searchguard.ssl.http.enabled_protocols:

- “TLSv1.2”

Also below is one of many similar log lines from the master node when I try to start ES 2.3.3 w/ search-guard and the above elasticsearch.yml:

[2016-06-28 19:11:04,518][WARN ][gateway ] [nextkibana02-ops] [logstash-2016.06.05][3]: failed to list shard for shard_store on node [jyc0Tx-wRGeS71JoLf4X-Q]

FailedNodeException[total failure in fetching]; nested: ElasticsearchException[unauthenticated request internal:cluster/nodes/indices/shard/store for user User [name=_sg_internal, roles=]];

at org.elasticsearch.gateway.AsyncShardFetch$1.onFailure(AsyncShardFetch.java:277)

at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:95)

at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:135)

at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170)

at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:144)

at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:85)

at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.list(TransportNodesListShardStoreMetaData.java:88)

at org.elasticsearch.gateway.AsyncShardFetch.asyncFetch(AsyncShardFetch.java:267)

at org.elasticsearch.gateway.AsyncShardFetch.fetchData(AsyncShardFetch.java:117)

at org.elasticsearch.gateway.GatewayAllocator$InternalReplicaShardAllocator.fetchData(GatewayAllocator.java:183)

at org.elasticsearch.gateway.ReplicaShardAllocator.allocateUnassigned(ReplicaShardAllocator.java:137)

at org.elasticsearch.gateway.GatewayAllocator.allocateUnassigned(GatewayAllocator.java:123)

at org.elasticsearch.cluster.routing.allocation.allocator.ShardsAllocators.allocateUnassigned(ShardsAllocators.java:70)

at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:258)

at org.elasticsearch.cluster.routing.allocation.AllocationService.applyStartedShards(AllocationService.java:86)

at org.elasticsearch.cluster.action.shard.ShardStateAction$ShardStartedClusterStateHandler.execute(ShardStateAction.java:218)

at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:468)

at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)

at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)

at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused by: ElasticsearchException[unauthenticated request internal:cluster/nodes/indices/shard/store for user User [name=_sg_internal, roles=]]

… 21 more

=================

I can get more detailed “debug” logs from startup if necessary.

On Wednesday, June 29, 2016 at 12:17:18 AM UTC-7, SG wrote:
See inline comments:

Am 29.06.2016 um 00:26 schrieb Max Furman mx.f...@gmail.com:

I seem to be running into a bit of a chicken and egg problem when trying to install searchguard-2 and searchguard-ssl, restart ES 2.3.3, and then configure searchguard using the sgadmin.sh tool.

Here is my current ordering.

install search-guard-2 and search-guard-ssl, edit elasticsearch.yml adding static configuration options for server to server transport communication over ssl.

restart ES on all servers.

Seeing the following error on non master nodes:
   [2016-06-28 19:06:27,164][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
Do you get this error while you try to connect your node via HTTP?

This indicates that I need to run the sgadmin.sh configuration script w/ proper arguments.

However when I run the script I get the following output:

/usr/share/elasticsearch/plugins/search-guard-2/tools/sgadmin.sh -cd /usr/share/elasticsearch/plugins/search-guard-2/sgconfig/ -ks admin-keystore.jks -kspass xxx -ts admin-truststore.jks -tspass xxx -nhnv

Connect to localhost:9300

Exception in thread “main” NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}]]
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:290)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:207)
at org.elasticsearch.client.transport.support.TransportProxyClient.execute(TransportProxyClient.java:55)
at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:288)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:348)
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:848)
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.health(AbstractClient.java:868)
at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:201)
The error, I gather, is happening b/c ES is unable to startup with the existing searchguard transport configuration options in the yml. It seems to me that this would be a chicken and egg problem, so I assume that I am missing some step in the installation. Hoping someone can point that out to me.

Can you pls. post the elasticsearch logs on DEBUG level and your elasticsearch.yml?

-max

–
You received this message because you are subscribed to the Google Groups “Search Guard” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/1bc28b75-7254-4956-81c9-0def378ae571%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Max_Furman · July 6, 2016, 10:22pm

Turns out I needed to run sgadmin.sh with -icl option to ignore the cluster name. Also needed to set the host (-h option) to the private network IP instead of the default 127.0.0.1.

···

On Tuesday, June 28, 2016 at 3:26:22 PM UTC-7, Max Furman wrote:

I seem to be running into a bit of a chicken and egg problem when trying to install searchguard-2 and searchguard-ssl, restart ES 2.3.3, and then configure searchguard using the sgadmin.sh tool.

Here is my current ordering.

install search-guard-2 and search-guard-ssl, edit elasticsearch.yml adding static configuration options for server to server transport communication over ssl.

restart ES on all servers.

Seeing the following error on non master nodes:

[2016-06-28 19:06:27,164][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

This indicates that I need to run the sgadmin.sh configuration script w/ proper arguments.

However when I run the script I get the following output:

/usr/share/elasticsearch/plugins/search-guard-2/tools/sgadmin.sh -cd /usr/share/elasticsearch/plugins/search-guard-2/sgconfig/ -ks admin-keystore.jks -kspass xxx -ts admin-truststore.jks -tspass xxx -nhnv

Connect to localhost:9300

Exception in thread “main” NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}]]

at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:290)

at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:207)

at org.elasticsearch.client.transport.support.TransportProxyClient.execute(TransportProxyClient.java:55)

at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:288)

at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)

at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:348)

at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:848)

at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.health(AbstractClient.java:868)

at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:201)

The error, I gather, is happening b/c ES is unable to startup with the existing searchguard transport configuration options in the yml. It seems to me that this would be a chicken and egg problem, so I assume that I am missing some step in the installation. Hoping someone can point that out to me.

-max

searchguard_google_group · July 7, 2016, 6:35am

You can also run sgadmin with the -cn option and set the clustername

···

Am 07.07.2016 um 00:22 schrieb Max Furman <mx.furman@gmail.com>:

Turns out I needed to run sgadmin.sh with `-icl` option to ignore the cluster name. Also needed to set the host (-h option) to the private network IP instead of the default 127.0.0.1.

On Tuesday, June 28, 2016 at 3:26:22 PM UTC-7, Max Furman wrote:
I seem to be running into a bit of a chicken and egg problem when trying to install searchguard-2 and searchguard-ssl, restart ES 2.3.3, and then configure searchguard using the sgadmin.sh tool.

Here is my current ordering.

1) install search-guard-2 and search-guard-ssl, edit elasticsearch.yml adding static configuration options for server to server transport communication over ssl.
2) restart ES on all servers.
3) Seeing the following error on non master nodes:

       [2016-06-28 19:06:27,164][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized

4) This indicates that I need to run the sgadmin.sh configuration script w/ proper arguments.
5) However when I run the script I get the following output:

/usr/share/elasticsearch/plugins/search-guard-2/tools/sgadmin.sh -cd /usr/share/elasticsearch/plugins/search-guard-2/sgconfig/ -ks admin-keystore.jks -kspass xxx -ts admin-truststore.jks -tspass xxx -nhnv

Connect to localhost:9300

Exception in thread "main" NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}]]

    at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:290)

    at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:207)

    at org.elasticsearch.client.transport.support.TransportProxyClient.execute(TransportProxyClient.java:55)

    at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:288)

    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)

    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:348)

    at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:848)

    at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.health(AbstractClient.java:868)

    at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:201)

The error, I gather, is happening b/c ES is unable to startup with the existing searchguard transport configuration options in the yml. It seems to me that this would be a chicken and egg problem, so I assume that I am missing some step in the installation. Hoping someone can point that out to me.

-max

--
You received this message because you are subscribed to the Google Groups "Search Guard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/7abdd3e3-dbd9-4425-941c-daa5ac8a66f4%40googlegroups.com\.
For more options, visit https://groups.google.com/d/optout\.

Topic		Replies	Views
We cant initialize SeaarchGuard Search Guard	6	414	May 19, 2016
Running sgadmin.sh for first time Search Guard	3	1103	July 21, 2016
Issue Running SGAdmin Search Guard	18	1354	August 16, 2016
Errors setting up with example data Search Guard	2	1140	October 9, 2019
Can't configure search guard Search Guard	5	587	May 13, 2016

Installation order of Operations

======================== Elasticsearch Configuration =========================

NOTE: Elasticsearch comes with reasonable defaults for most settings.

Before you set out to tweak and tune the configuration, make sure you

understand what are you trying to accomplish and the consequences.

The primary way of configuring a node is via this file. This template lists

the most important settings you may want to configure for a production cluster.

Please see the documentation for further information on configuration options:

http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html

---------------------------------- Cluster -----------------------------------

Use a descriptive name for your cluster:

------------------------------------ Node ------------------------------------

Use a descriptive name for the node:

Add custom attributes to the node:

node.rack: r1

----------------------------------- Paths ------------------------------------

Path to directory where to store the data (separate multiple locations by comma):

Path to log files:

----------------------------------- Memory -----------------------------------

Lock the memory on startup:

Make sure that the ES_HEAP_SIZE environment variable is set to about half the memory

available on the system and that the owner of the process is allowed to use this limit.

Elasticsearch performs poorly when the system is swapping the memory.

---------------------------------- Network -----------------------------------

Set the bind address to a specific IP (IPv4 or IPv6):

Set a custom host and port for HTTP:

For more information, see the documentation at:

http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html

--------------------------------- Discovery ----------------------------------

Pass an initial list of hosts to perform discovery when new node is started:

The default list of hosts is [“127.0.0.1”, “[::1]”]

Prevent the “split brain” by configuring the majority of nodes (total number of nodes / 2 + 1):

For more information, see the documentation at:

http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html

---------------------------------- Gateway -----------------------------------

Block initial recovery after a full cluster restart until N nodes are started:

For more information, see the documentation at:

http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-gateway.html

---------------------------------- Various -----------------------------------

Disable starting multiple nodes on a single system:

node.max_local_storage_nodes: 1

Require explicit names when deleting indices:

action.destructive_requires_name: true

Searchguard

SEARCH GUARD SSL

Configuration

Transport layer SSL

Enable or disable node-to-node ssl encryption (default: true)

JKS or PKCS12 (default: JKS)

Relative path to the keystore file (mandatory, this stores the server certificates), must be placed under the config/ dir

Alias name (default: first alias which could be found)

Keystore password (default: changeit)

JKS or PKCS12 (default: JKS)

Relative path to the truststore file (mandatory, this stores the client/root certificates), must be placed under the config/ dir

Alias name (default: trust all aliases)

Truststore password (default: changeit)

Enforce hostname verification (default: true)

If hostname verification specify if hostname should be resolved (default: true)

Use native Open SSL instead of JDK SSL if available (default: true)

Enabled SSL cipher suites for transport protocol (only Java format is supported)

WARNING: Expert setting, do only use if you know what you are doing

If you set wrong values here this this could be a security risk

- “TLS_DHE_RSA_WITH_AES_256_CBC_SHA”

- “TLS_DHE_DSS_WITH_AES_128_CBC_SHA256”

Enabled SSL protocols for transport protocol (only Java format is supported)

WARNING: Expert setting, do only use if you know what you are doing

If you set wrong values here this this could be a security risk

- “TLSv1.2”

HTTP/REST layer SSL

Enable or disable rest layer security - https, (default: false)

JKS or PKCS12 (default: JKS)

Relative path to the keystore file (this stores the server certificates), must be placed under the config/ dir

Alias name (default: first alias which could be found)

Keystore password (default: changeit)

Do the clients (typically the browser or the proxy) have to authenticate themself to the http server, default is OPTIONAL

To enforce authentication use REQUIRE, to completely disable client certificates use NONE

JKS or PKCS12 (default: JKS)

Relative path to the truststore file (this stores the client certificates), must be placed under the config/ dir

Alias name (default: first alias which could be found)

Truststore password (default: changeit)

Make sure that the `ES_HEAP_SIZE` environment variable is set to about half the memory