I have one master node and data node, the configuration as below
ElasticSearch Version: 5.0.2
Search Guard: 5
Keystore and Trustore are generated based on Internal Company Certificates
ElasticSearch - Master Node - elasticsearch.yml
cluster.name: elk-cluster
node.name: Bharath-master-01
node.master: true
node.data: false
path.data: /apps/masterES/data
path.logs: /apps/masterES/logs
bootstrap.memory_lock: true
network.host: 01.02.03.04
discovery.zen.ping.unicast.hosts: [“01.02.03.04”]
discovery.zen.minimum_master_nodes: 1
http.port: 9200
searchguard.audit.type: internal_elasticsearch
logger.com.floragunn.searchguard.ssl: DEBUG
searchguard.authcz.admin_dn:
- “CN=elk.bharath.com, OU=Servers, OU=Applications, O=Bharth Corp., C=us”
searchguard.ssl.transport.keystore_filepath: elk.server.keystore.jks
searchguard.ssl.transport.keystore_password: password
searchguard.ssl.transport.truststore_filepath: elk.server.truststore.jks
searchguard.ssl.transport.truststore_password: password
searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: false
searchguard.ssl.transport.enable_openssl_if_available: false
searchguard.ssl.transport.keystore_alias: elkcluster
Logs of Master ES: [2016-12-27T08:25:19,194][WARN ][o.e.b.JNANatives ] unable to install syscall filter:
java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
at org.elasticsearch.bootstrap.Seccomp.linuxImpl(Seccomp.java:361) ~[elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.bootstrap.Seccomp.init(Seccomp.java:630) ~[elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.bootstrap.JNANatives.trySeccomp(JNANatives.java:215) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.bootstrap.Natives.trySeccomp(Natives.java:99) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:104) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:158) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:291) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:96) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.cli.Command.main(Command.java:62) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) [elasticsearch-5.0.2.jar:5.0.2]
[2016-12-27T08:25:19,607][INFO ][o.e.n.Node ] [Bharath-master-01] initializing …
[2016-12-27T08:25:19,680][INFO ][o.e.e.NodeEnvironment ] [Bharath-master-01] using [1] data paths, mounts [[/apps (/dev/xvdb1)]], net usable_space [44.3gb], net total_space [46.8gb], spins? [no], types [ext4]
[2016-12-27T08:25:19,680][INFO ][o.e.e.NodeEnvironment ] [Bharath-master-01] heap size [1.9gb], compressed ordinary object pointers [true]
[2016-12-27T08:25:19,683][INFO ][o.e.n.Node ] [Bharath-master-01] version[5.0.2], pid[23327], build[f6b4951/2016-11-24T10:07:18.101Z], OS[Linux/2.6.32-573.7.1.el6.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot™ 64-Bit Server VM/1.8.0_60/25.60-b23]
[2016-12-27T08:25:20,445][INFO ][c.f.s.SearchGuardPlugin ] Node [Bharath-master-01] is a transportClient: false/tribeNode: false/tribeNodeClient: false
[2016-12-27T08:25:20,446][INFO ][c.f.s.SearchGuardPlugin ] FLS/DLS module not available
[2016-12-27T08:25:20,449][INFO ][o.e.p.PluginsService ] [Bharath-master-01] loaded module [aggs-matrix-stats]
[2016-12-27T08:25:20,449][INFO ][o.e.p.PluginsService ] [Bharath-master-01] loaded module [ingest-common]
[2016-12-27T08:25:20,449][INFO ][o.e.p.PluginsService ] [Bharath-master-01] loaded module [lang-expression]
[2016-12-27T08:25:20,449][INFO ][o.e.p.PluginsService ] [Bharath-master-01] loaded module [lang-groovy]
[2016-12-27T08:25:20,449][INFO ][o.e.p.PluginsService ] [Bharath-master-01] loaded module [lang-mustache]
[2016-12-27T08:25:20,449][INFO ][o.e.p.PluginsService ] [Bharath-master-01] loaded module [lang-painless]
[2016-12-27T08:25:20,449][INFO ][o.e.p.PluginsService ] [Bharath-master-01] loaded module [percolator]
[2016-12-27T08:25:20,449][INFO ][o.e.p.PluginsService ] [Bharath-master-01] loaded module [reindex]
[2016-12-27T08:25:20,450][INFO ][o.e.p.PluginsService ] [Bharath-master-01] loaded module [transport-netty3]
[2016-12-27T08:25:20,450][INFO ][o.e.p.PluginsService ] [Bharath-master-01] loaded module [transport-netty4]
[2016-12-27T08:25:20,450][INFO ][o.e.p.PluginsService ] [Bharath-master-01] loaded plugin [search-guard-5]
[2016-12-27T08:25:21,345][INFO ][c.f.s.s.SearchGuardKeyStore] Config directory is /apps/elasticsearch-5.0.2/config/, from there the key- and truststore files are resolved relatively
[2016-12-27T08:25:21,349][DEBUG][c.f.s.s.u.SSLCertificateHelper] Keystore has 2 entries/aliases
[2016-12-27T08:25:21,349][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias caroot: is a certificate entry?true/is a key entry?false
[2016-12-27T08:25:21,349][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias elkcluster: is a certificate entry?false/is a key entry?true
[2016-12-27T08:25:21,351][DEBUG][c.f.s.s.u.SSLCertificateHelper] Keystore has 1 entries/aliases
[2016-12-27T08:25:21,351][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias caroot: is a certificate entry?true/is a key entry?false
[2016-12-27T08:25:21,351][DEBUG][c.f.s.s.u.SSLCertificateHelper] No alias given, will trust all of the certificates in the store
[2016-12-27T08:25:21,376][INFO ][c.f.s.s.SearchGuardKeyStore] AES-256 not supported, max key length for AES is 128 bit… That is not an issue, it just limits possible encryption strength. To enable AES 256 install ‘Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files’
[2016-12-27T08:25:21,377][INFO ][c.f.s.s.SearchGuardKeyStore] sslTransportClientProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]
[2016-12-27T08:25:21,377][INFO ][c.f.s.s.SearchGuardKeyStore] sslTransportServerProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]
[2016-12-27T08:25:21,377][INFO ][c.f.s.s.SearchGuardKeyStore] sslHTTPProvider:null with ciphers
[2016-12-27T08:25:21,377][INFO ][c.f.s.s.SearchGuardKeyStore] sslTransport protocols [TLSv1.2, TLSv1.1]
[2016-12-27T08:25:21,377][INFO ][c.f.s.s.SearchGuardKeyStore] sslHTTP protocols [TLSv1.2, TLSv1.1]
[2016-12-27T08:25:22,015][INFO ][c.f.s.c.ConfigurationModule] FLS/DLS valve not bound (noop)
[2016-12-27T08:25:22,017][INFO ][c.f.s.a.AuditLogModule ] Auditlog not available
[2016-12-27T08:25:22,551][INFO ][o.e.n.Node ] [Bharath-master-01] initialized
[2016-12-27T08:25:22,551][INFO ][o.e.n.Node ] [Bharath-master-01] starting …
[2016-12-27T08:25:22,599][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport] [Bharath-master-01] using profile[default], worker_count[12], port[9300-9400], bind_host[null], publish_host[null], compress[false], connect_timeout[30s], connections_per_node[2/3/6/1/1], receive_predictor[512kb->512kb]
[2016-12-27T08:25:22,605][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport] [Bharath-master-01] binding server bootstrap to: [xx.xxx.xxx.xx]
[2016-12-27T08:25:22,678][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport] [Bharath-master-01] Bound profile [default] to address {xx.xxx.xxx.xx:9300}
[2016-12-27T08:25:22,679][INFO ][o.e.t.TransportService ] [Bharath-master-01] publish_address {xx.xxx.xxx.xx:9300}, bound_addresses {xx.xxx.xxx.xx:9300}
[2016-12-27T08:25:22,684][INFO ][o.e.b.BootstrapCheck ] [Bharath-master-01] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2016-12-27T08:25:22,688][INFO ][c.f.s.a.c.TransportConfigUpdateAction] [Bharath-master-01] Check if searchguard index exists …
[2016-12-27T08:25:22,695][DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [Bharath-master-01] no known master node, scheduling a retry
[2016-12-27T08:25:25,782][INFO ][o.e.c.s.ClusterService ] [Bharath-master-01] new_master {Bharath-master-01}{BuFd_aDXTqSMrvcxWBhpmg}{03I9PfK0RiuF6ermc-toCw}{xx.xxx.xxx.xx}{xx.xxx.xxx.xx:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2016-12-27T08:25:25,822][INFO ][c.f.s.a.c.TransportConfigUpdateAction] [Bharath-master-01] searchguard index does not exist yet, so no need to load config on node startup. Use sgadmin to initialize cluster
[2016-12-27T08:25:25,822][INFO ][o.e.g.GatewayService ] [Bharath-master-01] recovered [0] indices into cluster_state
[2016-12-27T08:25:25,907][INFO ][o.e.h.HttpServer ] [Bharath-master-01] publish_address {xx.xxx.xxx.xx:9200}, bound_addresses {xx.xxx.xxx.xx:9200}
[2016-12-27T08:25:25,907][INFO ][o.e.n.Node ] [Bharath-master-01] started
[2016-12-27T08:30:15,029][ERROR][c.f.s.a.BackendRegistry ] Not yet initialized
[2016-12-27T08:30:15,029][ERROR][c.f.s.a.BackendRegistry ] Not yet initialized
[2016-12-27T08:35:15,001][ERROR][c.f.s.a.BackendRegistry ] Not yet initialized
[2016-12-27T08:35:15,017][ERROR][c.f.s.a.BackendRegistry ] Not yet initialized
[2016-12-27T08:40:14,984][ERROR][c.f.s.a.BackendRegistry ] Not yet initialized
[2016-12-27T08:40:15,000][ERROR][c.f.s.a.BackendRegistry ] Not yet initialized
[2016-12-27T08:45:14,978][ERROR][c.f.s.a.BackendRegistry ] Not yet initialized
[2016-12-27T08:45:14,995][ERROR][c.f.s.a.BackendRegistry ] Not yet initialized
[2016-12-27T08:47:07,552][ERROR][c.f.s.a.BackendRegistry ] Not yet initialized
[2016-12-27T08:47:25,891][ERROR][c.f.s.a.BackendRegistry ] Not yet initialized
[2016-12-27T08:50:14,997][ERROR][c.f.s.a.BackendRegistry ] Not yet initialized
[2016-12-27T08:50:15,023][ERROR][c.f.s.a.BackendRegistry ] Not yet initialized**
Once I ran sgadmin.sh I got this:**
plugins/search-guard-5/tools/sgadmin.sh -cd plugins/search-guard-5/sgconfig/ -cn elk-cluster -h bharath.fmr.com -p 9300 -ksalias elkcluster -ks /apps/elasticsearch-5.0.2/config/elk.server.keystore.jks -kspass elkadmin17 -ts /apps/elasticsearch-5.0.2/config/elk.server.truststore.jks -tspass elkadmin17 -nhnv
Search Guard Admin v5
Will connect to bharath.fmr.com:9300 … done
Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …
Cannot retrieve cluster state due to None of the configured nodes are available: [{#transport#-1}{tFfsyLnDQ2a0aPzCNIQ3BQ}{xx.xxx.xxx.xx}{xx.xxx.xxx.xx:9300}]. This is not an error, will keep on trying …
ElasticSearch - Data Node - elasticsearch.yml
cluster.name: elk-cluster
node.name: Bharath-data-01
node.master: false
node.data: true
path.data: /apps/dataES1/data
path.logs: /apps/dataES1/logs
discovery.zen.ping.unicast.hosts: [“01.02.03.04”]
network.host: 05.06.07.08
http.port: 9200
searchguard.audit.type: internal_elasticsearch
logger.com.floragunn.searchguard.ssl: DEBUG
searchguard.authcz.admin_dn: - “CN=elk.bharath.com, OU=Servers, OU=Applications, O=Bharth Corp., C=us”
searchguard.ssl.transport.keystore_filepath: elk.server.keystore.jks
searchguard.ssl.transport.keystore_password: password
searchguard.ssl.transport.truststore_filepath: elk.server.truststore.jks
searchguard.ssl.transport.truststore_password: password
searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.enforce_hostname_verification: true
searchguard.ssl.transport.resolve_hostname: true
searchguard.ssl.transport.enable_openssl_if_available: false
searchguard.ssl.transport.keystore_alias: elkcluster
Logs of Data Node ES if am trying to start after master node: [2016-12-27T09:02:23,196][WARN ][o.e.b.JNANatives ] unable to install syscall filter:
java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
at org.elasticsearch.bootstrap.Seccomp.linuxImpl(Seccomp.java:361) ~[elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.bootstrap.Seccomp.init(Seccomp.java:630) ~[elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.bootstrap.JNANatives.trySeccomp(JNANatives.java:215) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.bootstrap.Natives.trySeccomp(Natives.java:99) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:104) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:158) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:291) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:96) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.cli.Command.main(Command.java:62) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) [elasticsearch-5.0.2.jar:5.0.2]
[2016-12-27T09:02:23,301][INFO ][o.e.n.Node ] [Bharath-data-01] initializing …
[2016-12-27T09:02:23,528][INFO ][o.e.e.NodeEnvironment ] [Bharath-data-01] using [1] data paths, mounts [[/apps (/dev/xvdb1)]], net usable_space [1.7tb], net total_space [1.8tb], spins? [no], types [ext4]
[2016-12-27T09:02:23,528][INFO ][o.e.e.NodeEnvironment ] [Bharath-data-01] heap size [1.9gb], compressed ordinary object pointers [true]
[2016-12-27T09:02:23,531][INFO ][o.e.n.Node ] [Bharath-data-01] version[5.0.2], pid[9346], build[f6b4951/2016-11-24T10:07:18.101Z], OS[Linux/2.6.32-573.7.1.el6.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot™ 64-Bit Server VM/1.8.0_60/25.60-b23]
[2016-12-27T09:02:24,301][INFO ][c.f.s.SearchGuardPlugin ] Node [Bharath-data-01] is a transportClient: false/tribeNode: false/tribeNodeClient: false
[2016-12-27T09:02:24,301][INFO ][c.f.s.SearchGuardPlugin ] FLS/DLS module not available
[2016-12-27T09:02:24,304][INFO ][o.e.p.PluginsService ] [Bharath-data-01] loaded module [aggs-matrix-stats]
[2016-12-27T09:02:24,304][INFO ][o.e.p.PluginsService ] [Bharath-data-01] loaded module [ingest-common]
[2016-12-27T09:02:24,304][INFO ][o.e.p.PluginsService ] [Bharath-data-01] loaded module [lang-expression]
[2016-12-27T09:02:24,305][INFO ][o.e.p.PluginsService ] [Bharath-data-01] loaded module [lang-groovy]
[2016-12-27T09:02:24,305][INFO ][o.e.p.PluginsService ] [Bharath-data-01] loaded module [lang-mustache]
[2016-12-27T09:02:24,305][INFO ][o.e.p.PluginsService ] [Bharath-data-01] loaded module [lang-painless]
[2016-12-27T09:02:24,305][INFO ][o.e.p.PluginsService ] [Bharath-data-01] loaded module [percolator]
[2016-12-27T09:02:24,305][INFO ][o.e.p.PluginsService ] [Bharath-data-01] loaded module [reindex]
[2016-12-27T09:02:24,305][INFO ][o.e.p.PluginsService ] [Bharath-data-01] loaded module [transport-netty3]
[2016-12-27T09:02:24,305][INFO ][o.e.p.PluginsService ] [Bharath-data-01] loaded module [transport-netty4]
[2016-12-27T09:02:24,306][INFO ][o.e.p.PluginsService ] [Bharath-data-01] loaded plugin [search-guard-5]
[2016-12-27T09:02:25,544][INFO ][c.f.s.s.SearchGuardKeyStore] Config directory is /apps/elasticsearch-5.0.2/config/, from there the key- and truststore files are resolved relatively
[2016-12-27T09:02:25,548][DEBUG][c.f.s.s.u.SSLCertificateHelper] Keystore has 2 entries/aliases
[2016-12-27T09:02:25,549][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias caroot: is a certificate entry?true/is a key entry?false
[2016-12-27T09:02:25,549][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias elkcluster: is a certificate entry?false/is a key entry?true
[2016-12-27T09:02:25,551][DEBUG][c.f.s.s.u.SSLCertificateHelper] Keystore has 1 entries/aliases
[2016-12-27T09:02:25,551][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias caroot: is a certificate entry?true/is a key entry?false
[2016-12-27T09:02:25,551][DEBUG][c.f.s.s.u.SSLCertificateHelper] No alias given, will trust all of the certificates in the store
[2016-12-27T09:02:25,575][INFO ][c.f.s.s.SearchGuardKeyStore] AES-256 not supported, max key length for AES is 128 bit… That is not an issue, it just limits possible encryption strength. To enable AES 256 install ‘Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files’
[2016-12-27T09:02:25,576][INFO ][c.f.s.s.SearchGuardKeyStore] sslTransportClientProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]
[2016-12-27T09:02:25,576][INFO ][c.f.s.s.SearchGuardKeyStore] sslTransportServerProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]
[2016-12-27T09:02:25,576][INFO ][c.f.s.s.SearchGuardKeyStore] sslHTTPProvider:null with ciphers
[2016-12-27T09:02:25,576][INFO ][c.f.s.s.SearchGuardKeyStore] sslTransport protocols [TLSv1.2, TLSv1.1]
[2016-12-27T09:02:25,576][INFO ][c.f.s.s.SearchGuardKeyStore] sslHTTP protocols [TLSv1.2, TLSv1.1]
[2016-12-27T09:02:26,333][INFO ][c.f.s.c.ConfigurationModule] FLS/DLS valve not bound (noop)
[2016-12-27T09:02:26,335][INFO ][c.f.s.a.AuditLogModule ] Auditlog not available
[2016-12-27T09:02:26,965][INFO ][o.e.n.Node ] [Bharath-data-01] initialized
[2016-12-27T09:02:26,965][INFO ][o.e.n.Node ] [Bharath-data-01] starting …
[2016-12-27T09:02:27,020][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport] [Bharath-data-01] using profile[default], worker_count[12], port[9300-9400], bind_host[null], publish_host[null], compress[false], connect_timeout[30s], connections_per_node[2/3/6/1/1], receive_predictor[512kb->512kb]
[2016-12-27T09:02:27,025][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport] [Bharath-data-01] binding server bootstrap to: [xx.xxx.xxx.xxx]
[2016-12-27T09:02:27,103][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport] [Bharath-data-01] Bound profile [default] to address {xx.xxx.xxx.xxx:9300}
[2016-12-27T09:02:27,104][INFO ][o.e.t.TransportService ] [Bharath-data-01] publish_address {xx.xxx.xxx.xxx:9300}, bound_addresses {xx.xxx.xxx.xxx:9300}
[2016-12-27T09:02:27,109][INFO ][o.e.b.BootstrapCheck ] [Bharath-data-01] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2016-12-27T09:02:27,113][INFO ][c.f.s.a.c.TransportConfigUpdateAction] [Bharath-data-01] Check if searchguard index exists …
[2016-12-27T09:02:27,121][DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [Bharath-data-01] no known master node, scheduling a retry
[2016-12-27T09:02:27,168][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport$ClientSSLHandler] Hostname of peer is vc2crtp1158510n.fmr.com (vc2crtp1158510n.fmr.com/vc2crtp1158510n.fmr.com) with hostnameVerificationResovleHostName: true
[2016-12-27T09:02:27,196][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport] [Bharath-data-01] connected to node [{#zen_unicast_1#}{HwcDHEz8Tn6BAbO2FPI8Kg}{xx.xxx.xxx.xx}{xx.xxx.xxx.xx:9300}]
[2016-12-27T09:02:27,346][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport] [Bharath-data-01] disconnecting from [{#zen_unicast_1#}{HwcDHEz8Tn6BAbO2FPI8Kg}{xx.xxx.xxx.xx}{xx.xxx.xxx.xx:9300}], channel closed event
[2016-12-27T09:02:27,352][WARN ][o.e.d.z.p.u.UnicastZenPing] [Bharath-data-01] [1] failed send ping to {#zen_unicast_1#}{HwcDHEz8Tn6BAbO2FPI8Kg}{xx.xxx.xxx.xx}{xx.xxx.xxx.xx:9300}
java.lang.IllegalStateException: handshake failed with {#zen_unicast_1#}{HwcDHEz8Tn6BAbO2FPI8Kg}{xx.xxx.xxx.xx}{xx.xxx.xxx.xx:9300}
at org.elasticsearch.transport.TransportService.handshake(TransportService.java:364) ~[elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.transport.TransportService.connectToNodeLightAndHandshake(TransportService.java:339) ~[elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.transport.TransportService.connectToNodeLightAndHandshake(TransportService.java:313) ~[elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$2.run(UnicastZenPing.java:419) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458) [elasticsearch-5.0.2.jar:5.0.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_60]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
Caused by: org.elasticsearch.transport.NodeDisconnectedException: [xx.xxx.xxx.xx:9300][internal:transport/handshake] disconnected
[2016-12-27T09:02:27,365][WARN ][c.f.s.s.t.SearchGuardSSLNettyTransport] [Bharath-data-01] exception caught on transport layer [[id: 0x4aee8402, L:0.0.0.0/0.0.0.0:8534 ! R:vc2crtp1158510n.fmr.com/xx.xxx.xxx.xx:9300]], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442) ~[netty-codec-4.1.5.Final.jar:4.1.5.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248) ~[netty-codec-4.1.5.Final.jar:4.1.5.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:350) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:610) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:513) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:467) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:437) [netty-transport-4.1.5.Final.jar:4.1.5.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873) [netty-common-4.1.5.Final.jar:4.1.5.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1431) ~[?:?]
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_60]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1094) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:966) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:900) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411) ~[?:?]
… 15 more
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) ~[?:?]
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1506) ~[?:?]
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:919) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:916) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_60]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1369) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1120) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1005) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:900) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411) ~[?:?]
… 15 more
Caused by: java.security.cert.CertificateException: No name matching vc2crtp1158510n.fmr.com found
at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:221) ~[?:?]
at sun.security.util.HostnameChecker.match(HostnameChecker.java:95) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:436) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:252) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:?]
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1493) ~[?:?]
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:919) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:916) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_60]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1369) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1120) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1005) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:900) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411) ~[?:?]
… 15 more
^C[2016-12-27T09:02:28,418][INFO ][o.e.n.Node ] [Bharath-data-01] stopping …
[2016-12-27T09:02:28,422][ERROR][c.f.s.a.c.TransportConfigUpdateAction] [Bharath-data-01] Failure while checking NodeClosedException[node closed {Bharath-data-01}{3f3uWihtSfSuOEXnF1BzEg}{RK1BQ5fdTKCgkoADQRlEgQ}{xx.xxx.xxx.xxx}{xx.xxx.xxx.xxx:9300}] index searchguard
org.elasticsearch.node.NodeClosedException: node closed {Bharath-data-01}{3f3uWihtSfSuOEXnF1BzEg}{RK1BQ5fdTKCgkoADQRlEgQ}{xx.xxx.xxx.xxx}{xx.xxx.xxx.xxx:9300}
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$5.onClusterServiceClose(TransportMasterNodeAction.java:208) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onClusterServiceClose(ClusterStateObserver.java:344) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onClose(ClusterStateObserver.java:226) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.cluster.service.ClusterService.doStop(ClusterService.java:216) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:87) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.node.Node.stop(Node.java:613) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.node.Node.close(Node.java:637) [elasticsearch-5.0.2.jar:5.0.2]
at org.apache.lucene.util.IOUtils.close(IOUtils.java:89) [lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20]
at org.apache.lucene.util.IOUtils.close(IOUtils.java:76) [lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20]
at org.elasticsearch.bootstrap.Bootstrap$4.run(Bootstrap.java:172) [elasticsearch-5.0.2.jar:5.0.2]
[2016-12-27T09:02:28,425][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [Bharath-data-01] no known master node, scheduling a retry
[2016-12-27T09:02:28,427][WARN ][c.f.s.a.c.TransportConfigUpdateAction] [Bharath-data-01] index ‘searchguard’ not healthy yet, we try again … (Reason: no response)
Please help me on this , i have tried many ways to execute from several days. Let me know if you need any information
I hope i will get the reply soon