Here are the logs, right after I restart Elasticsearch
[2016-10-17 13:49:21,403][INFO ][node ] [ip-10-22-9-4] version[2.4.0], pid[22291], build[ce9f0c7/2016-08-29T09:14:17Z]
[2016-10-17 13:49:21,403][INFO ][node ] [ip-10-22-9-4] initializing …
[2016-10-17 13:49:22,079][INFO ][com.floragunn.searchguard.ssl.SearchGuardSSLPlugin] Search Guard 2 plugin also available
[2016-10-17 13:49:22,086][INFO ][com.floragunn.searchguard.SearchGuardPlugin] Node [ip-10-22-9-4] is a transportClient: false/tribeNode: false/tribeNodeClient: false
[2016-10-17 13:49:22,160][INFO ][plugins ] [ip-10-22-9-4] modules [reindex, lang-expression, lang-groovy], plugins [head, search-guard-ssl, kopf, search-guard-2], sites [head, kopf]
[2016-10-17 13:49:22,192][INFO ][env ] [ip-10-22-9-4] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [26.8gb], net total_space [29.9gb], spins? [unknown], types [rootfs]
[2016-10-17 13:49:22,192][INFO ][env ] [ip-10-22-9-4] heap size [1007.3mb], compressed ordinary object pointers [true]
[2016-10-17 13:49:22,257][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Open SSL OpenSSL 1.0.1e-fips 11 Feb 2013 available
[2016-10-17 13:49:22,257][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Open SSL OpenSSL 1.0.1e-fips 11 Feb 2013 available
[2016-10-17 13:49:22,762][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Config directory is /etc/elasticsearch/, from there the key- and truststore files are resolved relatively
[2016-10-17 13:49:22,878][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] HTTPS client auth mode OPTIONAL
[2016-10-17 13:49:22,901][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] AES-256 not supported, max key length for AES is 128 bit… That is not an issue, it just limits possible encryption strength. To enable AES 256 install ‘Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files’
[2016-10-17 13:49:22,901][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslTransportClientProvider:OPENSSL with ciphers [TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA]
[2016-10-17 13:49:22,901][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslTransportServerProvider:OPENSSL with ciphers [TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA]
[2016-10-17 13:49:22,901][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslHTTPProvider:OPENSSL with ciphers [TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA]
[2016-10-17 13:49:22,901][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslTransport protocols [TLSv1.2, TLSv1.1]
[2016-10-17 13:49:22,901][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslHTTP protocols [TLSv1.2, TLSv1.1]
[2016-10-17 13:49:23,117][INFO ][http ] [ip-10-22-9-4] Using [org.elasticsearch.http.netty.NettyHttpServerTransport] as http transport, overridden by [search-guard2]
[2016-10-17 13:49:23,220][INFO ][com.floragunn.searchguard.configuration.ConfigurationModule] FLS/DLS valve not bound (noop)
[2016-10-17 13:49:23,222][INFO ][com.floragunn.searchguard.auditlog.AuditLogModule] Auditlog not available
[2016-10-17 13:49:23,308][INFO ][transport ] [ip-10-22-9-4] Using [com.floragunn.searchguard.transport.SearchGuardTransportService] as transport service, overridden by [search-guard2]
[2016-10-17 13:49:23,308][INFO ][transport ] [ip-10-22-9-4] Using [com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] as transport, overridden by [search-guard-ssl]
[2016-10-17 13:49:25,430][INFO ][node ] [ip-10-22-9-4] initialized
[2016-10-17 13:49:25,430][INFO ][node ] [ip-10-22-9-4] starting …
[2016-10-17 13:49:25,512][INFO ][com.floragunn.searchguard.transport.SearchGuardTransportService] [ip-10-22-9-4] publish_address {10.22.9.4:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}, {10.22.9.4:9300}
[2016-10-17 13:49:25,516][INFO ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [ip-10-22-9-4] Check if searchguard index exists …
[2016-10-17 13:49:25,523][DEBUG][action.admin.indices.exists.indices] [ip-10-22-9-4] no known master node, scheduling a retry
[2016-10-17 13:49:25,533][INFO ][discovery ] [ip-10-22-9-4] elk-nova-devops/HE1yxxjSTy-wct_4srljUw
[2016-10-17 13:49:55,534][WARN ][discovery ] [ip-10-22-9-4] waited for 30s and no initial state was set by the discovery
[2016-10-17 13:49:55,549][INFO ][http ] [ip-10-22-9-4] publish_address {10.22.9.4:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}, {10.22.9.4:9200}
[2016-10-17 13:49:55,549][INFO ][node ] [ip-10-22-9-4] started
[2016-10-17 13:49:57,669][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-10-17 13:50:02,266][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-10-17 13:50:25,535][DEBUG][action.admin.indices.exists.indices] [ip-10-22-9-4] timed out while retrying [indices:admin/exists] after failure (timeout [1m])
[2016-10-17 13:50:25,538][ERROR][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [ip-10-22-9-4] Failure while checking searchguard index MasterNotDiscoveredException[null]
MasterNotDiscoveredException[null]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$5.onTimeout(TransportMasterNodeAction.java:234)
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:236)
at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:804)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-10-17 13:50:25,543][DEBUG][action.admin.cluster.health] [ip-10-22-9-4] no known master node, scheduling a retry
[2016-10-17 13:50:25,587][INFO ][cluster.service ] [ip-10-22-9-4] new_master {ip-10-22-9-4}{HE1yxxjSTy-wct_4srljUw}{10.22.9.4}{10.22.9.4:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2016-10-17 13:50:25,664][INFO ][gateway ] [ip-10-22-9-4] recovered [1] indices into cluster_state
[2016-10-17 13:50:34,423][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-10-17 13:50:55,600][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [ip-10-22-9-4] index ‘searchguard’ not healthy yet, we try again … (Reason: timeout)
[2016-10-17 13:51:28,601][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [ip-10-22-9-4] index ‘searchguard’ not healthy yet, we try again … (Reason: timeout)
``
The last line repeats indefinitely.
···
On Saturday, October 15, 2016 at 4:25:57 PM UTC-4, in...@search-guard.com wrote:
try
searchguard.ssl.transport.resolve_hostname: true
searchguard.ssl.transport.enable_openssl_if_available: false
on all nodes in elasticsearch.yml
If this does not help please send the complete logfile.
On Thursday, 6 October 2016 17:38:32 UTC+2, ZillaYT wrote:
This is different from Elastic search will not start after I change data path. I resolved that by updating to v2.4.x
ES v2.4.1
SG-SSL v 2.4.1.16
SG v2.4.1.6
CentOS 7.2
Im able to run ES with SSL. I generated the certs/keys via the example script from Search Guard. Here is the info on the client cert that I generated
openssl x509 -noout -subject -in kirk-signed.pem -text
subject= /C=US/L=Raleigh/O=client/OU=client/CN=kirk
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 3 (0x3)
Signature Algorithm: sha256WithRSAEncryption
Issuer: DC=com, DC=example, O=Example Com Inc., OU=Example Com Inc. Signing CA, CN=Example Com Inc. Signing CA
Validity
Not Before: Oct 6 14:47:01 2016 GMT
Not After : Oct 6 14:47:01 2018 GMT
Subject: C=US, L=Raleigh, O=client, OU=client, CN=kirk
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
``
If I understand correctly, I need to have the following line in my elasticsearch.yml file, correct?
Enable SSL via Search Guard SSL plugin
Enable HTTPS
searchguard.ssl.http.enabled: true
searchguard.ssl.http.keystore_filepath: node-0-keystore.jks
searchguard.ssl.http.keystore_password: pw
searchguard.ssl.http.truststore_filepath: truststore.jks
searchguard.ssl.http.truststore_password: pw
Enable SSL between ES nodes
searchguard.ssl.transport.keystore_filepath: node-0-keystore.jks
searchguard.ssl.transport.keystore_password: pw
searchguard.ssl.transport.truststore_filepath: truststore.jks
searchguard.ssl.transport.truststore_password: pw
searchguard.ssl.transport.enforce_hostname_verification: false
for Search Guard
searchguard.authcz.admin_dn:
- “cn=kirk, ou=client, o=client, l=Raleigh, c=US”
searchguard.cert.oid: ‘1.2.3.4.5.5’
``
But when I run sgadmin.sh, it just times out
/usr/share/elasticsearch/plugins/search-guard-2/tools/sgadmin.sh -cd /etc/elasticsearch/ -ks kirk-keystore.jks -ts truststore.jks -nhnv -kspass pw -tspass pw
Will connect to localhost:9300 … done
Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …
ERR: Timed out while waiting for a green or yellow cluster state.
``
And I see these in elasticsearch.log
[2016-10-06 15:17:41,354][DEBUG][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [Arize] Node started, try to initialize it. Wait for at least yellow cluster state…
[2016-10-06 15:17:41,523][DEBUG][com.floragunn.searchguard.configuration.SearchGuardIndexSearcherWrapperModule] FLS/DLS not enabled
[2016-10-06 15:17:41,667][DEBUG][com.floragunn.searchguard.configuration.SearchGuardIndexSearcherWrapperModule] FLS/DLS not enabled
[2016-10-06 15:17:41,698][DEBUG][com.floragunn.searchguard.configuration.SearchGuardIndexSearcherWrapperModule] FLS/DLS not enabled
[2016-10-06 15:17:41,728][DEBUG][com.floragunn.searchguard.configuration.SearchGuardIndexSearcherWrapperModule] FLS/DLS not enabled
[2016-10-06 15:17:42,099][DEBUG][com.floragunn.searchguard.configuration.SearchGuardIndexSearcherWrapperModule] FLS/DLS not enabled
[2016-10-06 15:18:11,746][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [Arize] index ‘searchguard’ not healthy yet, we try again … (Reason: timeout)
[2016-10-06 15:18:44,747][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [Arize] index ‘searchguard’ not healthy yet, we try again … (Reason: timeout)
[2016-10-06 15:19:17,749][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [Arize] index ‘searchguard’ not healthy yet, we try again … (Reason: timeout)
``
What am I missing?