When asking questions, please provide the following information:
I am upgrading from 6.2.2 to 6.4.3 and have a strange exception during the startup process , i was wondering what that is ( the stack trace is repeated roughly 20 times i cut them out ) and how can i avoid it ?
sending incremental file list
searchguard/
searchguard/sg_action_groups.yml
searchguard/sg_config.yml
searchguard/sg_internal_users.yml
searchguard/sg_roles.yml
searchguard/sg_roles_mapping.yml
searchguard/ssl/
searchguard/ssl/beats-keystore.jks
searchguard/ssl/beats-keystore.p12
searchguard/ssl/beats-signed.pem
searchguard/ssl/beats.all.pem
searchguard/ssl/beats.crt.pem
searchguard/ssl/beats.crtfull.pem
searchguard/ssl/beats.csr
searchguard/ssl/beats.key.pem
searchguard/ssl/elastic-keystore.jks
searchguard/ssl/elastic-keystore.p12
searchguard/ssl/elastic-signed.pem
searchguard/ssl/elastic.all.pem
searchguard/ssl/elastic.crt.pem
searchguard/ssl/elastic.crtfull.pem
searchguard/ssl/elastic.csr
searchguard/ssl/elastic.key.pem
searchguard/ssl/kibana-keystore.jks
searchguard/ssl/kibana-keystore.p12
searchguard/ssl/kibana-signed.pem
searchguard/ssl/kibana.all.pem
searchguard/ssl/kibana.crt.pem
searchguard/ssl/kibana.crtfull.pem
searchguard/ssl/kibana.csr
searchguard/ssl/kibana.key.pem
searchguard/ssl/logstash-keystore.jks
searchguard/ssl/logstash-keystore.p12
searchguard/ssl/logstash-signed.pem
searchguard/ssl/logstash.all.pem
searchguard/ssl/logstash.crt.pem
searchguard/ssl/logstash.crtfull.pem
searchguard/ssl/logstash.csr
searchguard/ssl/logstash.key.pem
searchguard/ssl/monitoring-keystore.jks
searchguard/ssl/monitoring-keystore.p12
searchguard/ssl/monitoring-signed.pem
searchguard/ssl/monitoring.all.pem
searchguard/ssl/monitoring.crt.pem
searchguard/ssl/monitoring.crtfull.pem
searchguard/ssl/monitoring.csr
searchguard/ssl/monitoring.key.pem
searchguard/ssl/truststore.jks
searchguard/ssl/ca/
searchguard/ssl/ca/chain-ca.pem
searchguard/ssl/ca/root-ca.crt
searchguard/ssl/ca/root-ca.csr
searchguard/ssl/ca/root-ca.pem
searchguard/ssl/ca/signing-ca.crt
searchguard/ssl/ca/signing-ca.csr
searchguard/ssl/ca/signing-ca.pem
searchguard/ssl/ca/root-ca/
searchguard/ssl/ca/root-ca/01.pem
searchguard/ssl/ca/root-ca/02.pem
searchguard/ssl/ca/root-ca/db/
searchguard/ssl/ca/root-ca/db/root-ca.crl.srl
searchguard/ssl/ca/root-ca/db/root-ca.crt.srl
searchguard/ssl/ca/root-ca/db/root-ca.crt.srl.old
searchguard/ssl/ca/root-ca/db/root-ca.db
searchguard/ssl/ca/root-ca/db/root-ca.db.attr
searchguard/ssl/ca/root-ca/db/root-ca.db.attr.old
searchguard/ssl/ca/root-ca/db/root-ca.db.old
searchguard/ssl/ca/root-ca/private/
searchguard/ssl/ca/root-ca/private/root-ca.key
searchguard/ssl/ca/signing-ca/
searchguard/ssl/ca/signing-ca/01.pem
searchguard/ssl/ca/signing-ca/02.pem
searchguard/ssl/ca/signing-ca/03.pem
searchguard/ssl/ca/signing-ca/04.pem
searchguard/ssl/ca/signing-ca/05.pem
searchguard/ssl/ca/signing-ca/db/
searchguard/ssl/ca/signing-ca/db/signing-ca.crl.srl
searchguard/ssl/ca/signing-ca/db/signing-ca.crt.srl
searchguard/ssl/ca/signing-ca/db/signing-ca.crt.srl.old
searchguard/ssl/ca/signing-ca/db/signing-ca.db
searchguard/ssl/ca/signing-ca/db/signing-ca.db.attr
searchguard/ssl/ca/signing-ca/db/signing-ca.db.attr.old
searchguard/ssl/ca/signing-ca/db/signing-ca.db.old
searchguard/ssl/ca/signing-ca/private/
searchguard/ssl/ca/signing-ca/private/signing-ca.key
searchguard/ssl/certs/
searchguard/ssl/crl/
searchguard/ssl/etc/
searchguard/ssl/etc/root-ca.conf
searchguard/ssl/etc/signing-ca.conf
sent 181,886 bytes received 1,558 bytes 366,888.00 bytes/sec
total size is 176,413 speedup is 0.96
cluster.name: ${CLUSTER_NAME}
node:
name: ${HOSTNAME}
master: ${NODE_MASTER}
data: ${NODE_DATA}
ingest: ${NODE_INGEST}
comment out when running locally
network:
host:
- 0.0.0.0
cloud:
gce:
project_id: ${PROJECT_ID}
zone: ${ZONE}
comment out when running locally
discovery:
zen:
hosts_provider: gce
minimum_master_nodes: ${MINIMUM_MASTER_NODES}
gce:
tags: ${TAGS}
http:
enabled: ${HTTP_ENABLE}
compression: true
cors:
enabled: ${HTTP_CORS_ENABLE}
allow-origin: ${HTTP_CORS_ALLOW_ORIGIN}
xpack.graph.enabled: false
xpack.ml.enabled: false
xpack.security.enabled: false
xpack.monitoring.enabled: true
xpack.watcher.enabled: true
searchguard:
ssl.transport:
enabled: true
enable_openssl_if_available: true
keystore_type: JKS
keystore_filepath: searchguard/ssl/${NODE_NAME}-keystore.jks
keystore_password: ${KS_PWD}
truststore_type: JKS
truststore_filepath: searchguard/ssl/truststore.jks
truststore_password: ${TS_PWD}
enforce_hostname_verification: false
ssl.http:
enabled: ${HTTP_SSL}
clientauth_mode: OPTIONAL
enable_openssl_if_available: true
keystore_type: JKS
keystore_filepath: searchguard/ssl/${NODE_NAME}-keystore.jks
keystore_password: ${KS_PWD}
truststore_type: JKS
truststore_filepath: searchguard/ssl/truststore.jks
truststore_password: ${TS_PWD}
authcz.admin_dn:
- "CN=elastic ,OU=devops, C=COM"
enable_snapshot_restore_privilege: true
enterprise_modules_enabled: ${ENTERPRISE_ENABLED}
Generating keystore and certificate for node NODE-es-elk-prod-6-4-q65q
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using “keytool -importkeystore -srckeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -destkeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -deststoretype pkcs12”.
Generating certificate signing request for node NODE-es-elk-prod-6-4-q65q
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using “keytool -importkeystore -srckeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -destkeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -deststoretype pkcs12”.
Sign certificate request with CA
Using configuration from etc/signing-ca.conf
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 6 (0x6)
Validity
Not Before: Nov 27 19:21:03 2018 GMT
Not After : Nov 26 19:21:03 2020 GMT
Subject:
countryName = COM
organizationalUnitName = SSL
commonName = NODE-es-elk-prod-6-4-q65q
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Basic Constraints:
CA:FALSE
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Subject Key Identifier:
BF:B9:92:F6:62:2B:D5:FF:4E:44:F0:DB:83:56:40:59:14:78:15:A2
X509v3 Authority Key Identifier:
keyid:57:C2:B6:3C:34:3F:36:76:6A:65:5A:8C:DF:F0:04:2E:96:48:3B:DB
X509v3 Subject Alternative Name:
DNS:NODE-es-elk-prod-6-4-q65q, DNS:localhost, IP Address:127.0.0.1, Registered ID:1.2.3.4.5.5
Certificate is to be certified until Nov 26 19:21:03 2020 GMT (730 days)
Write out database with 1 new entries
Data Base Updated
Import back to keystore (including CA chain)
Certificate reply was installed in keystore
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using “keytool -importkeystore -srckeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -destkeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -deststoretype pkcs12”.
Importing keystore NODE-es-elk-prod-6-4-q65q-keystore.jks to NODE-es-elk-prod-6-4-q65q-keystore.p12…
Entry for alias node-es-elk-prod-6-4-q65q successfully imported.
Import command completed: 1 entries successfully imported, 0 entries failed or cancelled
All done for NODE-es-elk-prod-6-4-q65q
Created elasticsearch keystore in /elasticsearch/config
Stalling for Elasticsearch…
Stalling for Elasticsearch…
Stalling for Elasticsearch…
Stalling for Elasticsearch…
[2018-11-27T19:21:24,748][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] initializing …
[2018-11-27T19:21:24,846][INFO ][o.e.e.NodeEnvironment ] [es-elk-prod-6-4-q65q] using [1] data paths, mounts [[/elasticsearch/data (/dev/sda1)]], net usable_space [1.9tb], net total_space [1.9tb], types [ext4]
[2018-11-27T19:21:24,847][INFO ][o.e.e.NodeEnvironment ] [es-elk-prod-6-4-q65q] heap size [14.9gb], compressed ordinary object pointers [true]
[2018-11-27T19:21:24,848][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] node name [es-elk-prod-6-4-q65q], node ID [Oq9l3kZMQuKhh7u3i71kpQ]
[2018-11-27T19:21:24,849][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] version[6.4.3], pid[227], build[default/tar/fe40335/2018-10-30T23:17:19.084789Z], OS[Linux/4.14.67+/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_191/25.191-b12]
[2018-11-27T19:21:24,850][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -Xms15g, -Xmx15g, -Des.path.home=/elasticsearch, -Des.path.conf=/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
[2018-11-27T19:21:26,919][INFO ][c.f.s.SearchGuardPlugin ] ES Config path is /elasticsearch/config
[2018-11-27T19:21:26,981][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] OpenSSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSL
[2018-11-27T19:21:26,991][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Config directory is /elasticsearch/config/, from there the key- and truststore files are resolved relatively
[2018-11-27T19:21:26,998][INFO ][c.f.s.s.u.SSLCertificateHelper] No alias given, use the first one: node-es-elk-prod-6-4-q65q
[2018-11-27T19:21:27,004][WARN ][c.f.s.s.u.SSLCertificateHelper] Certificate chain for alias node-es-elk-prod-6-4-q65q contains a root certificate
[2018-11-27T19:21:27,027][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Client Provider : JDK
[2018-11-27T19:21:27,027][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Server Provider : JDK
[2018-11-27T19:21:27,028][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS HTTP Provider : null
[2018-11-27T19:21:27,028][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for transport layer : [TLSv1.1, TLSv1.2]
[2018-11-27T19:21:27,028][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for HTTP layer :
[2018-11-27T19:21:27,264][INFO ][c.f.s.SearchGuardPlugin ] Clustername: elk
[2018-11-27T19:21:27,292][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config has insecure file permissions (should be 0700)
[2018-11-27T19:21:27,292][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/scripts has insecure file permissions (should be 0700)
[2018-11-27T19:21:27,293][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/log4j2.properties has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,293][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/jvm.options has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,293][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard has insecure file permissions (should be 0700)
[2018-11-27T19:21:27,293][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/sg_config.yml has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,293][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl has insecure file permissions (should be 0700)
[2018-11-27T19:21:27,294][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash.crtfull.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,294][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic-signed.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,294][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash-keystore.p12 has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,294][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic.all.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,294][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana.crtfull.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,295][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana.csr has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,295][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/truststore.jks has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,295][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana.all.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,295][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring.crt.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,295][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats-keystore.jks has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats-signed.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring.all.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic.csr has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats.csr has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana-keystore.jks has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats.crt.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,297][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana.crt.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,297][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/crl has insecure file permissions (should be 0700)
[2018-11-27T19:21:27,297][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring.key.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,297][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash.all.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,297][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic.key.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/certs has insecure file permissions (should be 0700)
[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana.key.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/etc has insecure file permissions (should be 0700)
[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/etc/signing-ca.conf has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/etc/root-ca.conf has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash.csr has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring.crtfull.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic-keystore.p12 has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/NODE-es-elk-prod-6-4-q65q-signed.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring.csr has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic.crtfull.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash.key.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,300][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring-keystore.jks has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,300][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring-keystore.p12 has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,300][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats.all.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,300][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic.crt.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,300][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash-keystore.jks has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash.crt.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats-keystore.p12 has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana-keystore.p12 has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana-signed.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash-signed.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic-keystore.jks has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats.crtfull.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats.key.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/NODE-es-elk-prod-6-4-q65q-keystore.p12 has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/ca has insecure file permissions (should be 0700)
[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/ca/signing-ca has insecure file permissions (should be 0700)
[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/private/signing-ca.key has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/05.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/06.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/03.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/04.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/01.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/ca/signing-ca/db has insecure file permissions (should be 0700)
[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.crt.srl has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.db has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.crl.srl has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.db.attr has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.crt.srl.old has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.db.attr.old has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.db.old has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/02.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/chain-ca.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca.csr has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca.crt has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca.csr has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca.crt has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/ca/root-ca has insecure file permissions (should be 0700)
[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/private/root-ca.key has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/01.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/ca/root-ca/db has insecure file permissions (should be 0700)
[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.crt.srl.old has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.crl.srl has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.crt.srl has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.db.attr has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.db.old has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.db has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.db.attr.old has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/02.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/NODE-es-elk-prod-6-4-q65q-keystore.jks has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring-signed.pem has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/NODE-es-elk-prod-6-4-q65q.csr has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/sg_roles.yml has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/sg_internal_users.yml has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/sg_action_groups.yml has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/sg_roles_mapping.yml has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/elasticsearch.yml has insecure file permissions (should be 0600)
[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [aggs-matrix-stats]
[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [analysis-common]
[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [ingest-common]
[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [lang-expression]
[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [lang-mustache]
[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [lang-painless]
[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [mapper-extras]
[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [parent-join]
[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [percolator]
[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [rank-eval]
[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [reindex]
[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [repository-url]
[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [transport-netty4]
[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [tribe]
[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-core]
[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-deprecation]
[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-graph]
[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-logstash]
[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-ml]
[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-monitoring]
[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-rollup]
[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-security]
[2018-11-27T19:21:27,538][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-sql]
[2018-11-27T19:21:27,538][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-upgrade]
[2018-11-27T19:21:27,538][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-watcher]
[2018-11-27T19:21:27,538][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded plugin [discovery-gce]
[2018-11-27T19:21:27,538][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded plugin [repository-gcs]
[2018-11-27T19:21:27,539][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded plugin [search-guard-6]
Stalling for Elasticsearch…
[2018-11-27T19:21:31,675][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin com.floragunn.searchguard.SearchGuardPlugin
[2018-11-27T19:21:31,953][INFO ][o.e.d.DiscoveryModule ] [es-elk-prod-6-4-q65q] using discovery type [zen]
[2018-11-27T19:21:32,775][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] initialized
[2018-11-27T19:21:32,775][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] starting …
[2018-11-27T19:21:33,190][INFO ][o.e.t.TransportService ] [es-elk-prod-6-4-q65q] publish_address {10.79.1.5:9300}, bound_addresses {[::]:9300}
[2018-11-27T19:21:33,219][INFO ][o.e.b.BootstrapChecks ] [es-elk-prod-6-4-q65q] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-11-27T19:21:33,239][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Check if searchguard index exists …
[2018-11-27T19:21:33,249][DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [es-elk-prod-6-4-q65q] no known master node, scheduling a retry
[2018-11-27T19:21:33,440][INFO ][o.e.c.g.GceInstancesServiceImpl] [es-elk-prod-6-4-q65q] starting GCE discovery service
Stalling for Elasticsearch…
[2018-11-27T19:21:34,668][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [es-elk-prod-6-4-q65q] SSL Problem General SSLEngine problem
javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1521) ~[?:?]
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:528) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:802) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_191]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1709) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:318) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) ~[?:?]
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) ~[?:?]
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
… 19 more
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:362) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:262) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:?]
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1626) ~[?:?]
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
… 19 more
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:154) ~[?:?]
at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:80) ~[?:?]
at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292) ~[?:1.8.0_191]
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:357) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:262) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:?]
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1626) ~[?:?]
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
… 19 more
{
STACK TRACE REPEATED 20 MORE TIMES
}
[2018-11-27T19:21:37,272][INFO ][o.e.c.s.MasterService ] [es-elk-prod-6-4-q65q] zen-disco-elected-as-master ([0] nodes joined)[, ], reason: new_master {es-elk-prod-6-4-q65q}{Oq9l3kZMQuKhh7u3i71kpQ}{3jwPBjL7QcGx1FY9KBWFCQ}{10.79.1.5}{10.79.1.5:9300}{xpack.installed=true}
[2018-11-27T19:21:37,279][INFO ][o.e.c.s.ClusterApplierService] [es-elk-prod-6-4-q65q] new_master {es-elk-prod-6-4-q65q}{Oq9l3kZMQuKhh7u3i71kpQ}{3jwPBjL7QcGx1FY9KBWFCQ}{10.79.1.5}{10.79.1.5:9300}{xpack.installed=true}, reason: apply cluster state (from master [master {es-elk-prod-6-4-q65q}{Oq9l3kZMQuKhh7u3i71kpQ}{3jwPBjL7QcGx1FY9KBWFCQ}{10.79.1.5}{10.79.1.5:9300}{xpack.installed=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)[, ]]])
[2018-11-27T19:21:37,304][INFO ][c.f.s.h.SearchGuardNonSslHttpServerTransport] [es-elk-prod-6-4-q65q] publish_address {10.79.1.5:9200}, bound_addresses {[::]:9200}
[2018-11-27T19:21:37,304][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] started
[2018-11-27T19:21:37,305][INFO ][c.f.s.SearchGuardPlugin ] 0 Search Guard modules loaded so far:
[2018-11-27T19:21:37,361][INFO ][c.f.s.c.IndexBaseConfigurationRepository] searchguard index does not exist yet, so no need to load config on node startup. Use sgadmin to initialize cluster
[2018-11-27T19:21:37,361][INFO ][o.e.g.GatewayService ] [es-elk-prod-6-4-q65q] recovered [0] indices into cluster_state
[2018-11-27T19:21:37,534][INFO ][o.e.c.m.MetaDataIndexTemplateService] [es-elk-prod-6-4-q65q] adding template [.watch-history-9] for index patterns [.watcher-history-9*]
[2018-11-27T19:21:37,562][INFO ][o.e.c.m.MetaDataIndexTemplateService] [es-elk-prod-6-4-q65q] adding template [.watches] for index patterns [.watches*]
[2018-11-27T19:21:37,587][INFO ][o.e.c.m.MetaDataIndexTemplateService] [es-elk-prod-6-4-q65q] adding template [.triggered_watches] for index patterns [.triggered_watches*]
[2018-11-27T19:21:37,614][INFO ][o.e.c.m.MetaDataIndexTemplateService] [es-elk-prod-6-4-q65q] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-6-*]
[2018-11-27T19:21:37,652][INFO ][o.e.c.m.MetaDataIndexTemplateService] [es-elk-prod-6-4-q65q] adding template [.monitoring-es] for index patterns [.monitoring-es-6-*]
[2018-11-27T19:21:37,683][INFO ][o.e.c.m.MetaDataIndexTemplateService] [es-elk-prod-6-4-q65q] adding template [.monitoring-beats] for index patterns [.monitoring-beats-6-*]
[2018-11-27T19:21:37,707][INFO ][o.e.c.m.MetaDataIndexTemplateService] [es-elk-prod-6-4-q65q] adding template [.monitoring-alerts] for index patterns [.monitoring-alerts-6]
[2018-11-27T19:21:37,735][INFO ][o.e.c.m.MetaDataIndexTemplateService] [es-elk-prod-6-4-q65q] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-6-*]
[2018-11-27T19:21:37,845][INFO ][o.e.l.LicenseService ] [es-elk-prod-6-4-q65q] license [6c1856a6-ec28-4720-844d-7a17d353d9ba] mode [basic] - valid
Stalling for Elasticsearch…
[2018-11-27T19:21:39,392][ERROR][c.f.s.a.BackendRegistry ] Not yet initialized (you may need to run sgadmin)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0
curl: (35) SSL received a record that exceeded the maximum permissible length.
#dont remove the #hashtages next to the changeme , its will break the password generator
elastic:
hash: ‘$2y$12$uDNiJ8zYZfzA6MevrjHj4ewWLZw1GeIl6whKPzs7pazZYCgzQHtmK’ #elastic
roles:
- admin
kibana:
hash: ‘$2y$12$puFPwpG1SdcjnWK3Ms7UQ.IwxXdeoc8EHSPgHtYcTnmuyNLWB0gvu’ #kibana
roles:
- kibana_user
logstash:
hash: ‘$2y$12$mO9arl.4vhb.Ky633G2YQO9MWq4SpFqbKXq4dorQklpV7Y5RtRSr2’ #logstash
roles:
- logstash_user
beats:
hash: ‘$2y$12$YI3yy9ln4URydnqpdASuRusMzKzR7NtQuLsg4vHJ1KqAJGyYDZcM.’ #beats
roles:
- beats_user
monitoring:
hash: ‘$2y$12$v9wTSOL7kddl6dUbYRHZ4uksfuVG6DUBeRHVGBTwptR5ES/v3dXbu’ #monitoring
roles:
- monitoring_user
reader:
hash: ‘$2y$12$U9TDLpw3eVoRGFoMMMvPyOcF9VM4orwyYWP2yVAEDdD2MGw1Io89S’ #reader
roles:
- reader_user
app:
hash: ‘$2y$12$4O0EGlcddjIKiNwiE41lWOoZda1oHRi.ZyRDxyoOJhZcWe3X5VxJG’ #app
roles:
- app_user
Search Guard Admin v6
Will connect to es-elk-prod-6-4-q65q:9300 … done
Elasticsearch Version: 6.4.3
Search Guard Version: 6.4.3-23.2
Connected as CN=elastic,OU=devops,C=COM
Contacting elasticsearch cluster ‘elk’ and wait for YELLOW clusterstate …
Clustername: elk
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
searchguard index does not exists, attempt to create it … [2018-11-27T19:21:53,536][INFO ][o.e.c.m.MetaDataCreateIndexService] [es-elk-prod-6-4-q65q] [searchguard] creating index, cause [api], templates , shards [1]/[1], mappings
[2018-11-27T19:21:53,549][INFO ][o.e.c.r.a.AllocationService] [es-elk-prod-6-4-q65q] updating number_of_replicas to [0] for indices [searchguard]
[2018-11-27T19:21:53,810][INFO ][o.e.c.r.a.AllocationService] [es-elk-prod-6-4-q65q] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[searchguard][0]] …]).
done (0-all replicas)
Populate config from /elasticsearch/config/searchguard/
Will update ‘sg/config’ with /elasticsearch/config/searchguard/sg_config.yml
[2018-11-27T19:21:53,955][INFO ][o.e.c.m.MetaDataMappingService] [es-elk-prod-6-4-q65q] [searchguard/zXRAcS0qSniwrQ8r0Jp6QA] create_mapping [sg]
SUCC: Configuration for ‘config’ created or updated
Will update ‘sg/roles’ with /elasticsearch/config/searchguard/sg_roles.yml
[2018-11-27T19:21:54,198][INFO ][o.e.c.m.MetaDataMappingService] [es-elk-prod-6-4-q65q] [searchguard/zXRAcS0qSniwrQ8r0Jp6QA] update_mapping [sg]
SUCC: Configuration for ‘roles’ created or updated
Will update ‘sg/rolesmapping’ with /elasticsearch/config/searchguard/sg_roles_mapping.yml
[2018-11-27T19:21:54,279][INFO ][o.e.c.m.MetaDataMappingService] [es-elk-prod-6-4-q65q] [searchguard/zXRAcS0qSniwrQ8r0Jp6QA] update_mapping [sg]
SUCC: Configuration for ‘rolesmapping’ created or updated
Will update ‘sg/internalusers’ with /elasticsearch/config/searchguard/sg_internal_users.yml
[2018-11-27T19:21:54,339][INFO ][o.e.c.m.MetaDataMappingService] [es-elk-prod-6-4-q65q] [searchguard/zXRAcS0qSniwrQ8r0Jp6QA] update_mapping [sg]
SUCC: Configuration for ‘internalusers’ created or updated
Will update ‘sg/actiongroups’ with /elasticsearch/config/searchguard/sg_action_groups.yml
[2018-11-27T19:21:54,395][INFO ][o.e.c.m.MetaDataMappingService] [es-elk-prod-6-4-q65q] [searchguard/zXRAcS0qSniwrQ8r0Jp6QA] update_mapping [sg]
SUCC: Configuration for ‘actiongroups’ created or updated
[2018-11-27T19:21:54,471][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Info: No license needed because enterprise modules are not enabled
Done with success
exec chroot --userspec=1000 / “$@”
``
ES version 6.4.3
SG version 23.2