SSL Problem General SSLEngine problem during startup

When asking questions, please provide the following information:

I am upgrading from 6.2.2 to 6.4.3 and have a strange exception during the startup process , i was wondering what that is ( the stack trace is repeated roughly 20 times i cut them out ) and how can i avoid it ?

sending incremental file list

searchguard/

searchguard/sg_action_groups.yml

searchguard/sg_config.yml

searchguard/sg_internal_users.yml

searchguard/sg_roles.yml

searchguard/sg_roles_mapping.yml

searchguard/ssl/

searchguard/ssl/beats-keystore.jks

searchguard/ssl/beats-keystore.p12

searchguard/ssl/beats-signed.pem

searchguard/ssl/beats.all.pem

searchguard/ssl/beats.crt.pem

searchguard/ssl/beats.crtfull.pem

searchguard/ssl/beats.csr

searchguard/ssl/beats.key.pem

searchguard/ssl/elastic-keystore.jks

searchguard/ssl/elastic-keystore.p12

searchguard/ssl/elastic-signed.pem

searchguard/ssl/elastic.all.pem

searchguard/ssl/elastic.crt.pem

searchguard/ssl/elastic.crtfull.pem

searchguard/ssl/elastic.csr

searchguard/ssl/elastic.key.pem

searchguard/ssl/kibana-keystore.jks

searchguard/ssl/kibana-keystore.p12

searchguard/ssl/kibana-signed.pem

searchguard/ssl/kibana.all.pem

searchguard/ssl/kibana.crt.pem

searchguard/ssl/kibana.crtfull.pem

searchguard/ssl/kibana.csr

searchguard/ssl/kibana.key.pem

searchguard/ssl/logstash-keystore.jks

searchguard/ssl/logstash-keystore.p12

searchguard/ssl/logstash-signed.pem

searchguard/ssl/logstash.all.pem

searchguard/ssl/logstash.crt.pem

searchguard/ssl/logstash.crtfull.pem

searchguard/ssl/logstash.csr

searchguard/ssl/logstash.key.pem

searchguard/ssl/monitoring-keystore.jks

searchguard/ssl/monitoring-keystore.p12

searchguard/ssl/monitoring-signed.pem

searchguard/ssl/monitoring.all.pem

searchguard/ssl/monitoring.crt.pem

searchguard/ssl/monitoring.crtfull.pem

searchguard/ssl/monitoring.csr

searchguard/ssl/monitoring.key.pem

searchguard/ssl/truststore.jks

searchguard/ssl/ca/

searchguard/ssl/ca/chain-ca.pem

searchguard/ssl/ca/root-ca.crt

searchguard/ssl/ca/root-ca.csr

searchguard/ssl/ca/root-ca.pem

searchguard/ssl/ca/signing-ca.crt

searchguard/ssl/ca/signing-ca.csr

searchguard/ssl/ca/signing-ca.pem

searchguard/ssl/ca/root-ca/

searchguard/ssl/ca/root-ca/01.pem

searchguard/ssl/ca/root-ca/02.pem

searchguard/ssl/ca/root-ca/db/

searchguard/ssl/ca/root-ca/db/root-ca.crl.srl

searchguard/ssl/ca/root-ca/db/root-ca.crt.srl

searchguard/ssl/ca/root-ca/db/root-ca.crt.srl.old

searchguard/ssl/ca/root-ca/db/root-ca.db

searchguard/ssl/ca/root-ca/db/root-ca.db.attr

searchguard/ssl/ca/root-ca/db/root-ca.db.attr.old

searchguard/ssl/ca/root-ca/db/root-ca.db.old

searchguard/ssl/ca/root-ca/private/

searchguard/ssl/ca/root-ca/private/root-ca.key

searchguard/ssl/ca/signing-ca/

searchguard/ssl/ca/signing-ca/01.pem

searchguard/ssl/ca/signing-ca/02.pem

searchguard/ssl/ca/signing-ca/03.pem

searchguard/ssl/ca/signing-ca/04.pem

searchguard/ssl/ca/signing-ca/05.pem

searchguard/ssl/ca/signing-ca/db/

searchguard/ssl/ca/signing-ca/db/signing-ca.crl.srl

searchguard/ssl/ca/signing-ca/db/signing-ca.crt.srl

searchguard/ssl/ca/signing-ca/db/signing-ca.crt.srl.old

searchguard/ssl/ca/signing-ca/db/signing-ca.db

searchguard/ssl/ca/signing-ca/db/signing-ca.db.attr

searchguard/ssl/ca/signing-ca/db/signing-ca.db.attr.old

searchguard/ssl/ca/signing-ca/db/signing-ca.db.old

searchguard/ssl/ca/signing-ca/private/

searchguard/ssl/ca/signing-ca/private/signing-ca.key

searchguard/ssl/certs/

searchguard/ssl/crl/

searchguard/ssl/etc/

searchguard/ssl/etc/root-ca.conf

searchguard/ssl/etc/signing-ca.conf

sent 181,886 bytes received 1,558 bytes 366,888.00 bytes/sec

total size is 176,413 speedup is 0.96

cluster.name: ${CLUSTER_NAME}

node:

name: ${HOSTNAME}

master: ${NODE_MASTER}

data: ${NODE_DATA}

ingest: ${NODE_INGEST}

comment out when running locally

network:

host:

  • 0.0.0.0

cloud:

gce:

project_id: ${PROJECT_ID}
zone: ${ZONE}

comment out when running locally

discovery:

zen:

hosts_provider: gce
minimum_master_nodes: ${MINIMUM_MASTER_NODES}

gce:

tags: ${TAGS}

http:

enabled: ${HTTP_ENABLE}

compression: true

cors:

enabled: ${HTTP_CORS_ENABLE}
allow-origin: ${HTTP_CORS_ALLOW_ORIGIN}

xpack.graph.enabled: false

xpack.ml.enabled: false

xpack.security.enabled: false

xpack.monitoring.enabled: true

xpack.watcher.enabled: true

searchguard:

ssl.transport:
    enabled: true
    enable_openssl_if_available: true
    keystore_type: JKS
    keystore_filepath: searchguard/ssl/${NODE_NAME}-keystore.jks
    keystore_password: ${KS_PWD}
    truststore_type: JKS
    truststore_filepath: searchguard/ssl/truststore.jks
    truststore_password: ${TS_PWD}
    enforce_hostname_verification: false
ssl.http:
    enabled: ${HTTP_SSL}
    clientauth_mode: OPTIONAL
    enable_openssl_if_available: true
    keystore_type: JKS
    keystore_filepath: searchguard/ssl/${NODE_NAME}-keystore.jks
    keystore_password: ${KS_PWD}
    truststore_type: JKS
    truststore_filepath: searchguard/ssl/truststore.jks
    truststore_password: ${TS_PWD}
authcz.admin_dn:
  - "CN=elastic ,OU=devops, C=COM"
enable_snapshot_restore_privilege: true
enterprise_modules_enabled: ${ENTERPRISE_ENABLED}

Generating keystore and certificate for node NODE-es-elk-prod-6-4-q65q

Warning:

The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using “keytool -importkeystore -srckeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -destkeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -deststoretype pkcs12”.

Generating certificate signing request for node NODE-es-elk-prod-6-4-q65q

Warning:

The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using “keytool -importkeystore -srckeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -destkeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -deststoretype pkcs12”.

Sign certificate request with CA

Using configuration from etc/signing-ca.conf

Check that the request matches the signature

Signature ok

Certificate Details:

    Serial Number: 6 (0x6)
    Validity
        Not Before: Nov 27 19:21:03 2018 GMT
        Not After : Nov 26 19:21:03 2020 GMT
    Subject:
        countryName               = COM
        organizationalUnitName    = SSL
        commonName                = NODE-es-elk-prod-6-4-q65q
    X509v3 extensions:
        X509v3 Key Usage: critical
            Digital Signature, Key Encipherment
        X509v3 Basic Constraints:
            CA:FALSE
        X509v3 Extended Key Usage:
            TLS Web Server Authentication, TLS Web Client Authentication
        X509v3 Subject Key Identifier:
            BF:B9:92:F6:62:2B:D5:FF:4E:44:F0:DB:83:56:40:59:14:78:15:A2
        X509v3 Authority Key Identifier:
            keyid:57:C2:B6:3C:34:3F:36:76:6A:65:5A:8C:DF:F0:04:2E:96:48:3B:DB
        X509v3 Subject Alternative Name:
            DNS:NODE-es-elk-prod-6-4-q65q, DNS:localhost, IP Address:127.0.0.1, Registered ID:1.2.3.4.5.5

Certificate is to be certified until Nov 26 19:21:03 2020 GMT (730 days)

Write out database with 1 new entries

Data Base Updated

Import back to keystore (including CA chain)

Certificate reply was installed in keystore

Warning:

The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using “keytool -importkeystore -srckeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -destkeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -deststoretype pkcs12”.

Importing keystore NODE-es-elk-prod-6-4-q65q-keystore.jks to NODE-es-elk-prod-6-4-q65q-keystore.p12…

Entry for alias node-es-elk-prod-6-4-q65q successfully imported.

Import command completed: 1 entries successfully imported, 0 entries failed or cancelled

All done for NODE-es-elk-prod-6-4-q65q

Created elasticsearch keystore in /elasticsearch/config

Stalling for Elasticsearch…

Stalling for Elasticsearch…

Stalling for Elasticsearch…

Stalling for Elasticsearch…

[2018-11-27T19:21:24,748][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] initializing …

[2018-11-27T19:21:24,846][INFO ][o.e.e.NodeEnvironment ] [es-elk-prod-6-4-q65q] using [1] data paths, mounts [[/elasticsearch/data (/dev/sda1)]], net usable_space [1.9tb], net total_space [1.9tb], types [ext4]

[2018-11-27T19:21:24,847][INFO ][o.e.e.NodeEnvironment ] [es-elk-prod-6-4-q65q] heap size [14.9gb], compressed ordinary object pointers [true]

[2018-11-27T19:21:24,848][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] node name [es-elk-prod-6-4-q65q], node ID [Oq9l3kZMQuKhh7u3i71kpQ]

[2018-11-27T19:21:24,849][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] version[6.4.3], pid[227], build[default/tar/fe40335/2018-10-30T23:17:19.084789Z], OS[Linux/4.14.67+/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_191/25.191-b12]

[2018-11-27T19:21:24,850][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -Xms15g, -Xmx15g, -Des.path.home=/elasticsearch, -Des.path.conf=/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]

[2018-11-27T19:21:26,919][INFO ][c.f.s.SearchGuardPlugin ] ES Config path is /elasticsearch/config

[2018-11-27T19:21:26,981][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] OpenSSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSL

[2018-11-27T19:21:26,991][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Config directory is /elasticsearch/config/, from there the key- and truststore files are resolved relatively

[2018-11-27T19:21:26,998][INFO ][c.f.s.s.u.SSLCertificateHelper] No alias given, use the first one: node-es-elk-prod-6-4-q65q

[2018-11-27T19:21:27,004][WARN ][c.f.s.s.u.SSLCertificateHelper] Certificate chain for alias node-es-elk-prod-6-4-q65q contains a root certificate

[2018-11-27T19:21:27,027][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Client Provider : JDK

[2018-11-27T19:21:27,027][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Server Provider : JDK

[2018-11-27T19:21:27,028][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS HTTP Provider : null

[2018-11-27T19:21:27,028][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for transport layer : [TLSv1.1, TLSv1.2]

[2018-11-27T19:21:27,028][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for HTTP layer :

[2018-11-27T19:21:27,264][INFO ][c.f.s.SearchGuardPlugin ] Clustername: elk

[2018-11-27T19:21:27,292][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,292][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/scripts has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,293][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/log4j2.properties has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,293][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/jvm.options has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,293][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,293][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/sg_config.yml has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,293][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,294][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash.crtfull.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,294][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic-signed.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,294][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash-keystore.p12 has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,294][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic.all.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,294][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana.crtfull.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,295][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,295][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/truststore.jks has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,295][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana.all.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,295][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring.crt.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,295][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats-keystore.jks has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats-signed.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring.all.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana-keystore.jks has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats.crt.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,297][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana.crt.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,297][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/crl has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,297][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring.key.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,297][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash.all.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,297][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic.key.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/certs has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana.key.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/etc has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/etc/signing-ca.conf has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/etc/root-ca.conf has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring.crtfull.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic-keystore.p12 has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/NODE-es-elk-prod-6-4-q65q-signed.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic.crtfull.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash.key.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,300][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring-keystore.jks has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,300][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring-keystore.p12 has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,300][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats.all.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,300][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic.crt.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,300][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash-keystore.jks has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash.crt.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats-keystore.p12 has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana-keystore.p12 has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana-signed.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash-signed.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic-keystore.jks has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats.crtfull.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats.key.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/NODE-es-elk-prod-6-4-q65q-keystore.p12 has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/ca has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/ca/signing-ca has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/private/signing-ca.key has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/05.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/06.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/03.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/04.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/01.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/ca/signing-ca/db has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.crt.srl has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.db has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.crl.srl has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.db.attr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.crt.srl.old has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.db.attr.old has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.db.old has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/02.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/chain-ca.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca.crt has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca.crt has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/ca/root-ca has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/private/root-ca.key has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/01.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/ca/root-ca/db has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.crt.srl.old has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.crl.srl has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.crt.srl has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.db.attr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.db.old has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.db has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.db.attr.old has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/02.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/NODE-es-elk-prod-6-4-q65q-keystore.jks has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring-signed.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/NODE-es-elk-prod-6-4-q65q.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/sg_roles.yml has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/sg_internal_users.yml has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/sg_action_groups.yml has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/sg_roles_mapping.yml has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/elasticsearch.yml has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [aggs-matrix-stats]

[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [analysis-common]

[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [ingest-common]

[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [lang-expression]

[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [lang-mustache]

[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [lang-painless]

[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [mapper-extras]

[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [parent-join]

[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [percolator]

[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [rank-eval]

[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [reindex]

[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [repository-url]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [transport-netty4]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [tribe]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-core]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-deprecation]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-graph]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-logstash]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-ml]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-monitoring]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-rollup]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-security]

[2018-11-27T19:21:27,538][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-sql]

[2018-11-27T19:21:27,538][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-upgrade]

[2018-11-27T19:21:27,538][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-watcher]

[2018-11-27T19:21:27,538][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded plugin [discovery-gce]

[2018-11-27T19:21:27,538][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded plugin [repository-gcs]

[2018-11-27T19:21:27,539][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded plugin [search-guard-6]

Stalling for Elasticsearch…

[2018-11-27T19:21:31,675][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin com.floragunn.searchguard.SearchGuardPlugin

[2018-11-27T19:21:31,953][INFO ][o.e.d.DiscoveryModule ] [es-elk-prod-6-4-q65q] using discovery type [zen]

[2018-11-27T19:21:32,775][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] initialized

[2018-11-27T19:21:32,775][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] starting …

[2018-11-27T19:21:33,190][INFO ][o.e.t.TransportService ] [es-elk-prod-6-4-q65q] publish_address {10.79.1.5:9300}, bound_addresses {[::]:9300}

[2018-11-27T19:21:33,219][INFO ][o.e.b.BootstrapChecks ] [es-elk-prod-6-4-q65q] bound or publishing to a non-loopback address, enforcing bootstrap checks

[2018-11-27T19:21:33,239][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Check if searchguard index exists …

[2018-11-27T19:21:33,249][DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [es-elk-prod-6-4-q65q] no known master node, scheduling a retry

[2018-11-27T19:21:33,440][INFO ][o.e.c.g.GceInstancesServiceImpl] [es-elk-prod-6-4-q65q] starting GCE discovery service

Stalling for Elasticsearch…

[2018-11-27T19:21:34,668][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [es-elk-prod-6-4-q65q] SSL Problem General SSLEngine problem

javax.net.ssl.SSLHandshakeException: General SSLEngine problem

at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1521) ~[?:?]

at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:528) ~[?:?]

at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:802) ~[?:?]

at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766) ~[?:?]

at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_191]

at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]

Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem

at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]

at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1709) ~[?:?]

at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:318) ~[?:?]

at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) ~[?:?]

at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) ~[?:?]

at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) ~[?:?]

at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:?]

at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) ~[?:?]

at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) ~[?:?]

at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]

at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) ~[?:?]

at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

… 19 more

Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:362) ~[?:?]

at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270) ~[?:?]

at sun.security.validator.Validator.validate(Validator.java:262) ~[?:?]

at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]

at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[?:?]

at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:?]

at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1626) ~[?:?]

at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) ~[?:?]

at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:?]

at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) ~[?:?]

at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) ~[?:?]

at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]

at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) ~[?:?]

at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

… 19 more

Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:154) ~[?:?]

at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:80) ~[?:?]

at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292) ~[?:1.8.0_191]

at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:357) ~[?:?]

at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270) ~[?:?]

at sun.security.validator.Validator.validate(Validator.java:262) ~[?:?]

at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]

at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[?:?]

at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:?]

at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1626) ~[?:?]

at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) ~[?:?]

at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:?]

at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) ~[?:?]

at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) ~[?:?]

at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]

at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) ~[?:?]

at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

… 19 more

{

STACK TRACE REPEATED 20 MORE TIMES

}

[2018-11-27T19:21:37,272][INFO ][o.e.c.s.MasterService ] [es-elk-prod-6-4-q65q] zen-disco-elected-as-master ([0] nodes joined)[, ], reason: new_master {es-elk-prod-6-4-q65q}{Oq9l3kZMQuKhh7u3i71kpQ}{3jwPBjL7QcGx1FY9KBWFCQ}{10.79.1.5}{10.79.1.5:9300}{xpack.installed=true}

[2018-11-27T19:21:37,279][INFO ][o.e.c.s.ClusterApplierService] [es-elk-prod-6-4-q65q] new_master {es-elk-prod-6-4-q65q}{Oq9l3kZMQuKhh7u3i71kpQ}{3jwPBjL7QcGx1FY9KBWFCQ}{10.79.1.5}{10.79.1.5:9300}{xpack.installed=true}, reason: apply cluster state (from master [master {es-elk-prod-6-4-q65q}{Oq9l3kZMQuKhh7u3i71kpQ}{3jwPBjL7QcGx1FY9KBWFCQ}{10.79.1.5}{10.79.1.5:9300}{xpack.installed=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)[, ]]])

[2018-11-27T19:21:37,304][INFO ][c.f.s.h.SearchGuardNonSslHttpServerTransport] [es-elk-prod-6-4-q65q] publish_address {10.79.1.5:9200}, bound_addresses {[::]:9200}

[2018-11-27T19:21:37,304][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] started

[2018-11-27T19:21:37,305][INFO ][c.f.s.SearchGuardPlugin ] 0 Search Guard modules loaded so far:

[2018-11-27T19:21:37,361][INFO ][c.f.s.c.IndexBaseConfigurationRepository] searchguard index does not exist yet, so no need to load config on node startup. Use sgadmin to initialize cluster

[2018-11-27T19:21:37,361][INFO ][o.e.g.GatewayService ] [es-elk-prod-6-4-q65q] recovered [0] indices into cluster_state

[2018-11-27T19:21:37,534][INFO ][o.e.c.m.MetaDataIndexTemplateService] [es-elk-prod-6-4-q65q] adding template [.watch-history-9] for index patterns [.watcher-history-9*]

[2018-11-27T19:21:37,562][INFO ][o.e.c.m.MetaDataIndexTemplateService] [es-elk-prod-6-4-q65q] adding template [.watches] for index patterns [.watches*]

[2018-11-27T19:21:37,587][INFO ][o.e.c.m.MetaDataIndexTemplateService] [es-elk-prod-6-4-q65q] adding template [.triggered_watches] for index patterns [.triggered_watches*]

[2018-11-27T19:21:37,614][INFO ][o.e.c.m.MetaDataIndexTemplateService] [es-elk-prod-6-4-q65q] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-6-*]

[2018-11-27T19:21:37,652][INFO ][o.e.c.m.MetaDataIndexTemplateService] [es-elk-prod-6-4-q65q] adding template [.monitoring-es] for index patterns [.monitoring-es-6-*]

[2018-11-27T19:21:37,683][INFO ][o.e.c.m.MetaDataIndexTemplateService] [es-elk-prod-6-4-q65q] adding template [.monitoring-beats] for index patterns [.monitoring-beats-6-*]

[2018-11-27T19:21:37,707][INFO ][o.e.c.m.MetaDataIndexTemplateService] [es-elk-prod-6-4-q65q] adding template [.monitoring-alerts] for index patterns [.monitoring-alerts-6]

[2018-11-27T19:21:37,735][INFO ][o.e.c.m.MetaDataIndexTemplateService] [es-elk-prod-6-4-q65q] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-6-*]

[2018-11-27T19:21:37,845][INFO ][o.e.l.LicenseService ] [es-elk-prod-6-4-q65q] license [6c1856a6-ec28-4720-844d-7a17d353d9ba] mode [basic] - valid

Stalling for Elasticsearch…

[2018-11-27T19:21:39,392][ERROR][c.f.s.a.BackendRegistry ] Not yet initialized (you may need to run sgadmin)

% Total % Received % Xferd Average Speed Time Time Time Current

                             Dload  Upload   Total   Spent    Left  Speed

0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0

curl: (35) SSL received a record that exceeded the maximum permissible length.

#dont remove the #hashtages next to the changeme , its will break the password generator

elastic:

hash: ‘$2y$12$uDNiJ8zYZfzA6MevrjHj4ewWLZw1GeIl6whKPzs7pazZYCgzQHtmK’ #elastic

roles:

- admin

kibana:

hash: ‘$2y$12$puFPwpG1SdcjnWK3Ms7UQ.IwxXdeoc8EHSPgHtYcTnmuyNLWB0gvu’ #kibana

roles:

- kibana_user

logstash:

hash: ‘$2y$12$mO9arl.4vhb.Ky633G2YQO9MWq4SpFqbKXq4dorQklpV7Y5RtRSr2’ #logstash

roles:

- logstash_user

beats:

hash: ‘$2y$12$YI3yy9ln4URydnqpdASuRusMzKzR7NtQuLsg4vHJ1KqAJGyYDZcM.’ #beats

roles:

- beats_user

monitoring:

hash: ‘$2y$12$v9wTSOL7kddl6dUbYRHZ4uksfuVG6DUBeRHVGBTwptR5ES/v3dXbu’ #monitoring

roles:

- monitoring_user

reader:

hash: ‘$2y$12$U9TDLpw3eVoRGFoMMMvPyOcF9VM4orwyYWP2yVAEDdD2MGw1Io89S’ #reader

roles:

- reader_user

app:

hash: ‘$2y$12$4O0EGlcddjIKiNwiE41lWOoZda1oHRi.ZyRDxyoOJhZcWe3X5VxJG’ #app

roles:

- app_user

Search Guard Admin v6

Will connect to es-elk-prod-6-4-q65q:9300 … done

Elasticsearch Version: 6.4.3

Search Guard Version: 6.4.3-23.2

Connected as CN=elastic,OU=devops,C=COM

Contacting elasticsearch cluster ‘elk’ and wait for YELLOW clusterstate …

Clustername: elk

Clusterstate: GREEN

Number of nodes: 1

Number of data nodes: 1

searchguard index does not exists, attempt to create it … [2018-11-27T19:21:53,536][INFO ][o.e.c.m.MetaDataCreateIndexService] [es-elk-prod-6-4-q65q] [searchguard] creating index, cause [api], templates , shards [1]/[1], mappings

[2018-11-27T19:21:53,549][INFO ][o.e.c.r.a.AllocationService] [es-elk-prod-6-4-q65q] updating number_of_replicas to [0] for indices [searchguard]

[2018-11-27T19:21:53,810][INFO ][o.e.c.r.a.AllocationService] [es-elk-prod-6-4-q65q] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[searchguard][0]] …]).

done (0-all replicas)

Populate config from /elasticsearch/config/searchguard/

Will update ‘sg/config’ with /elasticsearch/config/searchguard/sg_config.yml

[2018-11-27T19:21:53,955][INFO ][o.e.c.m.MetaDataMappingService] [es-elk-prod-6-4-q65q] [searchguard/zXRAcS0qSniwrQ8r0Jp6QA] create_mapping [sg]

SUCC: Configuration for ‘config’ created or updated

Will update ‘sg/roles’ with /elasticsearch/config/searchguard/sg_roles.yml

[2018-11-27T19:21:54,198][INFO ][o.e.c.m.MetaDataMappingService] [es-elk-prod-6-4-q65q] [searchguard/zXRAcS0qSniwrQ8r0Jp6QA] update_mapping [sg]

SUCC: Configuration for ‘roles’ created or updated

Will update ‘sg/rolesmapping’ with /elasticsearch/config/searchguard/sg_roles_mapping.yml

[2018-11-27T19:21:54,279][INFO ][o.e.c.m.MetaDataMappingService] [es-elk-prod-6-4-q65q] [searchguard/zXRAcS0qSniwrQ8r0Jp6QA] update_mapping [sg]

SUCC: Configuration for ‘rolesmapping’ created or updated

Will update ‘sg/internalusers’ with /elasticsearch/config/searchguard/sg_internal_users.yml

[2018-11-27T19:21:54,339][INFO ][o.e.c.m.MetaDataMappingService] [es-elk-prod-6-4-q65q] [searchguard/zXRAcS0qSniwrQ8r0Jp6QA] update_mapping [sg]

SUCC: Configuration for ‘internalusers’ created or updated

Will update ‘sg/actiongroups’ with /elasticsearch/config/searchguard/sg_action_groups.yml

[2018-11-27T19:21:54,395][INFO ][o.e.c.m.MetaDataMappingService] [es-elk-prod-6-4-q65q] [searchguard/zXRAcS0qSniwrQ8r0Jp6QA] update_mapping [sg]

SUCC: Configuration for ‘actiongroups’ created or updated

[2018-11-27T19:21:54,471][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Info: No license needed because enterprise modules are not enabled

Done with success

exec chroot --userspec=1000 / “$@”

``

ES version 6.4.3

SG version 23.2

Hi!

so just to make sure I understood the question correctly … when you say the stacktrace is repeating like 20 time you are refering to the stacktrace originating from

Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

Correct? And am I right in the assumption that if you see it like 20 times you also have roughly 20 nodes?

The stacktrace simply means that the certificate that was sent by one node in the cluster could not be verified against the root CA in the truststore on the receiving node. Are you sure you have updated the respective truststore file on all nodes?

Can you follow the instructions on the TLS troubleshooting guide and check whether the certificate chains are correct?

Have a look at:

" Viewing the contents of your Key- and Truststore"

and

“Validating the certificate chain”

in particular.

···

On Tuesday, November 27, 2018 at 8:54:23 PM UTC+1, Roman Kournjaev wrote:

When asking questions, please provide the following information:

I am upgrading from 6.2.2 to 6.4.3 and have a strange exception during the startup process , i was wondering what that is ( the stack trace is repeated roughly 20 times i cut them out ) and how can i avoid it ?

sending incremental file list

searchguard/

searchguard/sg_action_groups.yml

searchguard/sg_config.yml

searchguard/sg_internal_users.yml

searchguard/sg_roles.yml

searchguard/sg_roles_mapping.yml

searchguard/ssl/

searchguard/ssl/beats-keystore.jks

searchguard/ssl/beats-keystore.p12

searchguard/ssl/beats-signed.pem

searchguard/ssl/beats.all.pem

searchguard/ssl/beats.crt.pem

searchguard/ssl/beats.crtfull.pem

searchguard/ssl/beats.csr

searchguard/ssl/beats.key.pem

searchguard/ssl/elastic-keystore.jks

searchguard/ssl/elastic-keystore.p12

searchguard/ssl/elastic-signed.pem

searchguard/ssl/elastic.all.pem

searchguard/ssl/elastic.crt.pem

searchguard/ssl/elastic.crtfull.pem

searchguard/ssl/elastic.csr

searchguard/ssl/elastic.key.pem

searchguard/ssl/kibana-keystore.jks

searchguard/ssl/kibana-keystore.p12

searchguard/ssl/kibana-signed.pem

searchguard/ssl/kibana.all.pem

searchguard/ssl/kibana.crt.pem

searchguard/ssl/kibana.crtfull.pem

searchguard/ssl/kibana.csr

searchguard/ssl/kibana.key.pem

searchguard/ssl/logstash-keystore.jks

searchguard/ssl/logstash-keystore.p12

searchguard/ssl/logstash-signed.pem

searchguard/ssl/logstash.all.pem

searchguard/ssl/logstash.crt.pem

searchguard/ssl/logstash.crtfull.pem

searchguard/ssl/logstash.csr

searchguard/ssl/logstash.key.pem

searchguard/ssl/monitoring-keystore.jks

searchguard/ssl/monitoring-keystore.p12

searchguard/ssl/monitoring-signed.pem

searchguard/ssl/monitoring.all.pem

searchguard/ssl/monitoring.crt.pem

searchguard/ssl/monitoring.crtfull.pem

searchguard/ssl/monitoring.csr

searchguard/ssl/monitoring.key.pem

searchguard/ssl/truststore.jks

searchguard/ssl/ca/

searchguard/ssl/ca/chain-ca.pem

searchguard/ssl/ca/root-ca.crt

searchguard/ssl/ca/root-ca.csr

searchguard/ssl/ca/root-ca.pem

searchguard/ssl/ca/signing-ca.crt

searchguard/ssl/ca/signing-ca.csr

searchguard/ssl/ca/signing-ca.pem

searchguard/ssl/ca/root-ca/

searchguard/ssl/ca/root-ca/01.pem

searchguard/ssl/ca/root-ca/02.pem

searchguard/ssl/ca/root-ca/db/

searchguard/ssl/ca/root-ca/db/root-ca.crl.srl

searchguard/ssl/ca/root-ca/db/root-ca.crt.srl

searchguard/ssl/ca/root-ca/db/root-ca.crt.srl.old

searchguard/ssl/ca/root-ca/db/root-ca.db

searchguard/ssl/ca/root-ca/db/root-ca.db.attr

searchguard/ssl/ca/root-ca/db/root-ca.db.attr.old

searchguard/ssl/ca/root-ca/db/root-ca.db.old

searchguard/ssl/ca/root-ca/private/

searchguard/ssl/ca/root-ca/private/root-ca.key

searchguard/ssl/ca/signing-ca/

searchguard/ssl/ca/signing-ca/01.pem

searchguard/ssl/ca/signing-ca/02.pem

searchguard/ssl/ca/signing-ca/03.pem

searchguard/ssl/ca/signing-ca/04.pem

searchguard/ssl/ca/signing-ca/05.pem

searchguard/ssl/ca/signing-ca/db/

searchguard/ssl/ca/signing-ca/db/signing-ca.crl.srl

searchguard/ssl/ca/signing-ca/db/signing-ca.crt.srl

searchguard/ssl/ca/signing-ca/db/signing-ca.crt.srl.old

searchguard/ssl/ca/signing-ca/db/signing-ca.db

searchguard/ssl/ca/signing-ca/db/signing-ca.db.attr

searchguard/ssl/ca/signing-ca/db/signing-ca.db.attr.old

searchguard/ssl/ca/signing-ca/db/signing-ca.db.old

searchguard/ssl/ca/signing-ca/private/

searchguard/ssl/ca/signing-ca/private/signing-ca.key

searchguard/ssl/certs/

searchguard/ssl/crl/

searchguard/ssl/etc/

searchguard/ssl/etc/root-ca.conf

searchguard/ssl/etc/signing-ca.conf

sent 181,886 bytes received 1,558 bytes 366,888.00 bytes/sec

total size is 176,413 speedup is 0.96

cluster.name: ${CLUSTER_NAME}

node:

name: ${HOSTNAME}

master: ${NODE_MASTER}

data: ${NODE_DATA}

ingest: ${NODE_INGEST}

comment out when running locally

network:

host:

  • 0.0.0.0

cloud:

gce:

project_id: ${PROJECT_ID}
zone: ${ZONE}

comment out when running locally

discovery:

zen:

hosts_provider: gce
minimum_master_nodes: ${MINIMUM_MASTER_NODES}

gce:

tags: ${TAGS}

http:

enabled: ${HTTP_ENABLE}

compression: true

cors:

enabled: ${HTTP_CORS_ENABLE}
allow-origin: ${HTTP_CORS_ALLOW_ORIGIN}

xpack.graph.enabled: false

xpack.ml.enabled: false

xpack.security.enabled: false

xpack.monitoring.enabled: true

xpack.watcher.enabled: true

searchguard:

ssl.transport:
    enabled: true
    enable_openssl_if_available: true
    keystore_type: JKS
    keystore_filepath: searchguard/ssl/${NODE_NAME}-keystore.jks
    keystore_password: ${KS_PWD}
    truststore_type: JKS
    truststore_filepath: searchguard/ssl/truststore.jks
    truststore_password: ${TS_PWD}
    enforce_hostname_verification: false
ssl.http:
    enabled: ${HTTP_SSL}
    clientauth_mode: OPTIONAL
    enable_openssl_if_available: true
    keystore_type: JKS
    keystore_filepath: searchguard/ssl/${NODE_NAME}-keystore.jks
    keystore_password: ${KS_PWD}
    truststore_type: JKS
    truststore_filepath: searchguard/ssl/truststore.jks
    truststore_password: ${TS_PWD}
authcz.admin_dn:
  - "CN=elastic ,OU=devops, C=COM"
enable_snapshot_restore_privilege: true
enterprise_modules_enabled: ${ENTERPRISE_ENABLED}

Generating keystore and certificate for node NODE-es-elk-prod-6-4-q65q

Warning:

The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using “keytool -importkeystore -srckeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -destkeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -deststoretype pkcs12”.

Generating certificate signing request for node NODE-es-elk-prod-6-4-q65q

Warning:

The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using “keytool -importkeystore -srckeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -destkeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -deststoretype pkcs12”.

Sign certificate request with CA

Using configuration from etc/signing-ca.conf

Check that the request matches the signature

Signature ok

Certificate Details:

    Serial Number: 6 (0x6)
    Validity
        Not Before: Nov 27 19:21:03 2018 GMT
        Not After : Nov 26 19:21:03 2020 GMT
    Subject:
        countryName               = COM
        organizationalUnitName    = SSL
        commonName                = NODE-es-elk-prod-6-4-q65q
    X509v3 extensions:
        X509v3 Key Usage: critical
            Digital Signature, Key Encipherment
        X509v3 Basic Constraints:
            CA:FALSE
        X509v3 Extended Key Usage:
            TLS Web Server Authentication, TLS Web Client Authentication
        X509v3 Subject Key Identifier:
            BF:B9:92:F6:62:2B:D5:FF:4E:44:F0:DB:83:56:40:59:14:78:15:A2
        X509v3 Authority Key Identifier:
            keyid:57:C2:B6:3C:34:3F:36:76:6A:65:5A:8C:DF:F0:04:2E:96:48:3B:DB
        X509v3 Subject Alternative Name:
            DNS:NODE-es-elk-prod-6-4-q65q, DNS:localhost, IP Address:127.0.0.1, Registered ID:1.2.3.4.5.5

Certificate is to be certified until Nov 26 19:21:03 2020 GMT (730 days)

Write out database with 1 new entries

Data Base Updated

Import back to keystore (including CA chain)

Certificate reply was installed in keystore

Warning:

The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using “keytool -importkeystore -srckeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -destkeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -deststoretype pkcs12”.

Importing keystore NODE-es-elk-prod-6-4-q65q-keystore.jks to NODE-es-elk-prod-6-4-q65q-keystore.p12…

Entry for alias node-es-elk-prod-6-4-q65q successfully imported.

Import command completed: 1 entries successfully imported, 0 entries failed or cancelled

All done for NODE-es-elk-prod-6-4-q65q

Created elasticsearch keystore in /elasticsearch/config

Stalling for Elasticsearch…

Stalling for Elasticsearch…

Stalling for Elasticsearch…

Stalling for Elasticsearch…

[2018-11-27T19:21:24,748][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] initializing …

[2018-11-27T19:21:24,846][INFO ][o.e.e.NodeEnvironment ] [es-elk-prod-6-4-q65q] using [1] data paths, mounts [[/elasticsearch/data (/dev/sda1)]], net usable_space [1.9tb], net total_space [1.9tb], types [ext4]

[2018-11-27T19:21:24,847][INFO ][o.e.e.NodeEnvironment ] [es-elk-prod-6-4-q65q] heap size [14.9gb], compressed ordinary object pointers [true]

[2018-11-27T19:21:24,848][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] node name [es-elk-prod-6-4-q65q], node ID [Oq9l3kZMQuKhh7u3i71kpQ]

[2018-11-27T19:21:24,849][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] version[6.4.3], pid[227], build[default/tar/fe40335/2018-10-30T23:17:19.084789Z], OS[Linux/4.14.67+/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_191/25.191-b12]

[2018-11-27T19:21:24,850][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -Xms15g, -Xmx15g, -Des.path.home=/elasticsearch, -Des.path.conf=/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]

[2018-11-27T19:21:26,919][INFO ][c.f.s.SearchGuardPlugin ] ES Config path is /elasticsearch/config

[2018-11-27T19:21:26,981][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] OpenSSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSL

[2018-11-27T19:21:26,991][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Config directory is /elasticsearch/config/, from there the key- and truststore files are resolved relatively

[2018-11-27T19:21:26,998][INFO ][c.f.s.s.u.SSLCertificateHelper] No alias given, use the first one: node-es-elk-prod-6-4-q65q

[2018-11-27T19:21:27,004][WARN ][c.f.s.s.u.SSLCertificateHelper] Certificate chain for alias node-es-elk-prod-6-4-q65q contains a root certificate

[2018-11-27T19:21:27,027][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Client Provider : JDK

[2018-11-27T19:21:27,027][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Server Provider : JDK

[2018-11-27T19:21:27,028][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS HTTP Provider : null

[2018-11-27T19:21:27,028][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for transport layer : [TLSv1.1, TLSv1.2]

[2018-11-27T19:21:27,028][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for HTTP layer :

[2018-11-27T19:21:27,264][INFO ][c.f.s.SearchGuardPlugin ] Clustername: elk

[2018-11-27T19:21:27,292][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,292][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/scripts has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,293][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/log4j2.properties has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,293][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/jvm.options has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,293][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,293][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/sg_config.yml has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,293][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,294][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash.crtfull.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,294][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic-signed.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,294][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash-keystore.p12 has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,294][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic.all.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,294][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana.crtfull.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,295][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,295][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/truststore.jks has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,295][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana.all.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,295][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring.crt.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,295][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats-keystore.jks has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats-signed.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring.all.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana-keystore.jks has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats.crt.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,297][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana.crt.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,297][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/crl has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,297][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring.key.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,297][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash.all.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,297][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic.key.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/certs has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana.key.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/etc has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/etc/signing-ca.conf has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/etc/root-ca.conf has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring.crtfull.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic-keystore.p12 has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/NODE-es-elk-prod-6-4-q65q-signed.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic.crtfull.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash.key.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,300][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring-keystore.jks has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,300][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring-keystore.p12 has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,300][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats.all.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,300][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic.crt.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,300][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash-keystore.jks has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash.crt.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats-keystore.p12 has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana-keystore.p12 has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana-signed.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash-signed.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic-keystore.jks has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats.crtfull.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats.key.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/NODE-es-elk-prod-6-4-q65q-keystore.p12 has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/ca has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/ca/signing-ca has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/private/signing-ca.key has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/05.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/06.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/03.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/04.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/01.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/ca/signing-ca/db has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.crt.srl has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.db has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.crl.srl has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.db.attr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.crt.srl.old has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.db.attr.old has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.db.old has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/02.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/chain-ca.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca.crt has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca.crt has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/ca/root-ca has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/private/root-ca.key has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/01.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/ca/root-ca/db has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.crt.srl.old has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.crl.srl has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.crt.srl has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.db.attr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.db.old has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.db has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.db.attr.old has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/02.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/NODE-es-elk-prod-6-4-q65q-keystore.jks has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring-signed.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/NODE-es-elk-prod-6-4-q65q.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/sg_roles.yml has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/sg_internal_users.yml has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/sg_action_groups.yml has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/sg_roles_mapping.yml has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/elasticsearch.yml has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [aggs-matrix-stats]

[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [analysis-common]

[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [ingest-common]

[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [lang-expression]

[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [lang-mustache]

[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [lang-painless]

[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [mapper-extras]

[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [parent-join]

[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [percolator]

[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [rank-eval]

[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [reindex]

[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [repository-url]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [transport-netty4]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [tribe]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-core]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-deprecation]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-graph]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-logstash]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-ml]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-monitoring]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-rollup]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-security]

[2018-11-27T19:21:27,538][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-sql]

[2018-11-27T19:21:27,538][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-upgrade]

[2018-11-27T19:21:27,538][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-watcher]

[2018-11-27T19:21:27,538][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded plugin [discovery-gce]

[2018-11-27T19:21:27,538][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded plugin [repository-gcs]

[2018-11-27T19:21:27,539][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded plugin [search-guard-6]

Stalling for Elasticsearch…

[2018-11-27T19:21:31,675][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin com.floragunn.searchguard.SearchGuardPlugin

[2018-11-27T19:21:31,953][INFO ][o.e.d.DiscoveryModule ] [es-elk-prod-6-4-q65q] using discovery type [zen]

[2018-11-27T19:21:32,775][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] initialized

[2018-11-27T19:21:32,775][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] starting …

[2018-11-27T19:21:33,190][INFO ][o.e.t.TransportService ] [es-elk-prod-6-4-q65q] publish_address {10.79.1.5:9300}, bound_addresses {[::]:9300}

[2018-11-27T19:21:33,219][INFO ][o.e.b.BootstrapChecks ] [es-elk-prod-6-4-q65q] bound or publishing to a non-loopback address, enforcing bootstrap checks

[2018-11-27T19:21:33,239][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Check if searchguard index exists …

[2018-11-27T19:21:33,249][DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [es-elk-prod-6-4-q65q] no known master node, scheduling a retry

[2018-11-27T19:21:33,440][INFO ][o.e.c.g.GceInstancesServiceImpl] [es-elk-prod-6-4-q65q] starting GCE discovery service

Stalling for Elasticsearch…

[2018-11-27T19:21:34,668][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [es-elk-prod-6-4-q65q] SSL Problem General SSLEngine problem

javax.net.ssl.SSLHandshakeException: General SSLEngine problem

at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1521) ~[?:?]
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:528) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:802) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_191]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]

Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem

at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1709) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:318) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) ~[?:?]
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) ~[?:?]
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
... 19 more

Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:362) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:262) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:?]
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1626) ~[?:?]
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
... 19 more

Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:154) ~[?:?]
at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:80) ~[?:?]
at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292) ~[?:1.8.0_191]
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:357) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:262) ~[?:?]

``

This is one node setup I am trying to get running with the new version.
You are right the

Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust ancho

``

is repeated 20 times until the node comes up and then it is not repeated any more

···

On Tuesday, November 27, 2018 at 12:23:11 PM UTC-8, Jochen Kressin wrote:

Hi!

so just to make sure I understood the question correctly … when you say the stacktrace is repeating like 20 time you are refering to the stacktrace originating from

Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

Correct? And am I right in the assumption that if you see it like 20 times you also have roughly 20 nodes?

The stacktrace simply means that the certificate that was sent by one node in the cluster could not be verified against the root CA in the truststore on the receiving node. Are you sure you have updated the respective truststore file on all nodes?

Can you follow the instructions on the TLS troubleshooting guide and check whether the certificate chains are correct?

https://docs.search-guard.com/latest/troubleshooting-tls

Have a look at:

" Viewing the contents of your Key- and Truststore"

and

“Validating the certificate chain”

in particular.

On Tuesday, November 27, 2018 at 8:54:23 PM UTC+1, Roman Kournjaev wrote:

When asking questions, please provide the following information:

I am upgrading from 6.2.2 to 6.4.3 and have a strange exception during the startup process , i was wondering what that is ( the stack trace is repeated roughly 20 times i cut them out ) and how can i avoid it ?

sending incremental file list

searchguard/

searchguard/sg_action_groups.yml

searchguard/sg_config.yml

searchguard/sg_internal_users.yml

searchguard/sg_roles.yml

searchguard/sg_roles_mapping.yml

searchguard/ssl/

searchguard/ssl/beats-keystore.jks

searchguard/ssl/beats-keystore.p12

searchguard/ssl/beats-signed.pem

searchguard/ssl/beats.all.pem

searchguard/ssl/beats.crt.pem

searchguard/ssl/beats.crtfull.pem

searchguard/ssl/beats.csr

searchguard/ssl/beats.key.pem

searchguard/ssl/elastic-keystore.jks

searchguard/ssl/elastic-keystore.p12

searchguard/ssl/elastic-signed.pem

searchguard/ssl/elastic.all.pem

searchguard/ssl/elastic.crt.pem

searchguard/ssl/elastic.crtfull.pem

searchguard/ssl/elastic.csr

searchguard/ssl/elastic.key.pem

searchguard/ssl/kibana-keystore.jks

searchguard/ssl/kibana-keystore.p12

searchguard/ssl/kibana-signed.pem

searchguard/ssl/kibana.all.pem

searchguard/ssl/kibana.crt.pem

searchguard/ssl/kibana.crtfull.pem

searchguard/ssl/kibana.csr

searchguard/ssl/kibana.key.pem

searchguard/ssl/logstash-keystore.jks

searchguard/ssl/logstash-keystore.p12

searchguard/ssl/logstash-signed.pem

searchguard/ssl/logstash.all.pem

searchguard/ssl/logstash.crt.pem

searchguard/ssl/logstash.crtfull.pem

searchguard/ssl/logstash.csr

searchguard/ssl/logstash.key.pem

searchguard/ssl/monitoring-keystore.jks

searchguard/ssl/monitoring-keystore.p12

searchguard/ssl/monitoring-signed.pem

searchguard/ssl/monitoring.all.pem

searchguard/ssl/monitoring.crt.pem

searchguard/ssl/monitoring.crtfull.pem

searchguard/ssl/monitoring.csr

searchguard/ssl/monitoring.key.pem

searchguard/ssl/truststore.jks

searchguard/ssl/ca/

searchguard/ssl/ca/chain-ca.pem

searchguard/ssl/ca/root-ca.crt

searchguard/ssl/ca/root-ca.csr

searchguard/ssl/ca/root-ca.pem

searchguard/ssl/ca/signing-ca.crt

searchguard/ssl/ca/signing-ca.csr

searchguard/ssl/ca/signing-ca.pem

searchguard/ssl/ca/root-ca/

searchguard/ssl/ca/root-ca/01.pem

searchguard/ssl/ca/root-ca/02.pem

searchguard/ssl/ca/root-ca/db/

searchguard/ssl/ca/root-ca/db/root-ca.crl.srl

searchguard/ssl/ca/root-ca/db/root-ca.crt.srl

searchguard/ssl/ca/root-ca/db/root-ca.crt.srl.old

searchguard/ssl/ca/root-ca/db/root-ca.db

searchguard/ssl/ca/root-ca/db/root-ca.db.attr

searchguard/ssl/ca/root-ca/db/root-ca.db.attr.old

searchguard/ssl/ca/root-ca/db/root-ca.db.old

searchguard/ssl/ca/root-ca/private/

searchguard/ssl/ca/root-ca/private/root-ca.key

searchguard/ssl/ca/signing-ca/

searchguard/ssl/ca/signing-ca/01.pem

searchguard/ssl/ca/signing-ca/02.pem

searchguard/ssl/ca/signing-ca/03.pem

searchguard/ssl/ca/signing-ca/04.pem

searchguard/ssl/ca/signing-ca/05.pem

searchguard/ssl/ca/signing-ca/db/

searchguard/ssl/ca/signing-ca/db/signing-ca.crl.srl

searchguard/ssl/ca/signing-ca/db/signing-ca.crt.srl

searchguard/ssl/ca/signing-ca/db/signing-ca.crt.srl.old

searchguard/ssl/ca/signing-ca/db/signing-ca.db

searchguard/ssl/ca/signing-ca/db/signing-ca.db.attr

searchguard/ssl/ca/signing-ca/db/signing-ca.db.attr.old

searchguard/ssl/ca/signing-ca/db/signing-ca.db.old

searchguard/ssl/ca/signing-ca/private/

searchguard/ssl/ca/signing-ca/private/signing-ca.key

searchguard/ssl/certs/

searchguard/ssl/crl/

searchguard/ssl/etc/

searchguard/ssl/etc/root-ca.conf

searchguard/ssl/etc/signing-ca.conf

sent 181,886 bytes received 1,558 bytes 366,888.00 bytes/sec

total size is 176,413 speedup is 0.96

cluster.name: ${CLUSTER_NAME}

node:

name: ${HOSTNAME}

master: ${NODE_MASTER}

data: ${NODE_DATA}

ingest: ${NODE_INGEST}

comment out when running locally

network:

host:

  • 0.0.0.0

cloud:

gce:

project_id: ${PROJECT_ID}
zone: ${ZONE}

comment out when running locally

discovery:

zen:

hosts_provider: gce
minimum_master_nodes: ${MINIMUM_MASTER_NODES}

gce:

tags: ${TAGS}

http:

enabled: ${HTTP_ENABLE}

compression: true

cors:

enabled: ${HTTP_CORS_ENABLE}
allow-origin: ${HTTP_CORS_ALLOW_ORIGIN}

xpack.graph.enabled: false

xpack.ml.enabled: false

xpack.security.enabled: false

xpack.monitoring.enabled: true

xpack.watcher.enabled: true

searchguard:

ssl.transport:
    enabled: true
    enable_openssl_if_available: true
    keystore_type: JKS
    keystore_filepath: searchguard/ssl/${NODE_NAME}-keystore.jks
    keystore_password: ${KS_PWD}
    truststore_type: JKS
    truststore_filepath: searchguard/ssl/truststore.jks
    truststore_password: ${TS_PWD}
    enforce_hostname_verification: false
ssl.http:
    enabled: ${HTTP_SSL}
    clientauth_mode: OPTIONAL
    enable_openssl_if_available: true
    keystore_type: JKS
    keystore_filepath: searchguard/ssl/${NODE_NAME}-keystore.jks
    keystore_password: ${KS_PWD}
    truststore_type: JKS
    truststore_filepath: searchguard/ssl/truststore.jks
    truststore_password: ${TS_PWD}
authcz.admin_dn:
  - "CN=elastic ,OU=devops, C=COM"
enable_snapshot_restore_privilege: true
enterprise_modules_enabled: ${ENTERPRISE_ENABLED}

Generating keystore and certificate for node NODE-es-elk-prod-6-4-q65q

Warning:

The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using “keytool -importkeystore -srckeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -destkeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -deststoretype pkcs12”.

Generating certificate signing request for node NODE-es-elk-prod-6-4-q65q

Warning:

The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using “keytool -importkeystore -srckeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -destkeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -deststoretype pkcs12”.

Sign certificate request with CA

Using configuration from etc/signing-ca.conf

Check that the request matches the signature

Signature ok

Certificate Details:

    Serial Number: 6 (0x6)
    Validity
        Not Before: Nov 27 19:21:03 2018 GMT
        Not After : Nov 26 19:21:03 2020 GMT
    Subject:
        countryName               = COM
        organizationalUnitName    = SSL
        commonName                = NODE-es-elk-prod-6-4-q65q
    X509v3 extensions:
        X509v3 Key Usage: critical
            Digital Signature, Key Encipherment
        X509v3 Basic Constraints:
            CA:FALSE
        X509v3 Extended Key Usage:
            TLS Web Server Authentication, TLS Web Client Authentication
        X509v3 Subject Key Identifier:
            BF:B9:92:F6:62:2B:D5:FF:4E:44:F0:DB:83:56:40:59:14:78:15:A2
        X509v3 Authority Key Identifier:
            keyid:57:C2:B6:3C:34:3F:36:76:6A:65:5A:8C:DF:F0:04:2E:96:48:3B:DB
        X509v3 Subject Alternative Name:
            DNS:NODE-es-elk-prod-6-4-q65q, DNS:localhost, IP Address:127.0.0.1, Registered ID:1.2.3.4.5.5

Certificate is to be certified until Nov 26 19:21:03 2020 GMT (730 days)

Write out database with 1 new entries

Data Base Updated

Import back to keystore (including CA chain)

Certificate reply was installed in keystore

Warning:

The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using “keytool -importkeystore -srckeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -destkeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -deststoretype pkcs12”.

Importing keystore NODE-es-elk-prod-6-4-q65q-keystore.jks to NODE-es-elk-prod-6-4-q65q-keystore.p12…

Entry for alias node-es-elk-prod-6-4-q65q successfully imported.

Import command completed: 1 entries successfully imported, 0 entries failed or cancelled

All done for NODE-es-elk-prod-6-4-q65q

Created elasticsearch keystore in /elasticsearch/config

Stalling for Elasticsearch…

Stalling for Elasticsearch…

Stalling for Elasticsearch…

Stalling for Elasticsearch…

[2018-11-27T19:21:24,748][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] initializing …

[2018-11-27T19:21:24,846][INFO ][o.e.e.NodeEnvironment ] [es-elk-prod-6-4-q65q] using [1] data paths, mounts [[/elasticsearch/data (/dev/sda1)]], net usable_space [1.9tb], net total_space [1.9tb], types [ext4]

[2018-11-27T19:21:24,847][INFO ][o.e.e.NodeEnvironment ] [es-elk-prod-6-4-q65q] heap size [14.9gb], compressed ordinary object pointers [true]

[2018-11-27T19:21:24,848][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] node name [es-elk-prod-6-4-q65q], node ID [Oq9l3kZMQuKhh7u3i71kpQ]

[2018-11-27T19:21:24,849][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] version[6.4.3], pid[227], build[default/tar/fe40335/2018-10-30T23:17:19.084789Z], OS[Linux/4.14.67+/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_191/25.191-b12]

[2018-11-27T19:21:24,850][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -Xms15g, -Xmx15g, -Des.path.home=/elasticsearch, -Des.path.conf=/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]

[2018-11-27T19:21:26,919][INFO ][c.f.s.SearchGuardPlugin ] ES Config path is /elasticsearch/config

[2018-11-27T19:21:26,981][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] OpenSSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSL

[2018-11-27T19:21:26,991][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Config directory is /elasticsearch/config/, from there the key- and truststore files are resolved relatively

[2018-11-27T19:21:26,998][INFO ][c.f.s.s.u.SSLCertificateHelper] No alias given, use the first one: node-es-elk-prod-6-4-q65q

[2018-11-27T19:21:27,004][WARN ][c.f.s.s.u.SSLCertificateHelper] Certificate chain for alias node-es-elk-prod-6-4-q65q contains a root certificate

[2018-11-27T19:21:27,027][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Client Provider : JDK

[2018-11-27T19:21:27,027][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Server Provider : JDK

[2018-11-27T19:21:27,028][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS HTTP Provider : null

[2018-11-27T19:21:27,028][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for transport layer : [TLSv1.1, TLSv1.2]

[2018-11-27T19:21:27,028][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for HTTP layer :

[2018-11-27T19:21:27,264][INFO ][c.f.s.SearchGuardPlugin ] Clustername: elk

[2018-11-27T19:21:27,292][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,292][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/scripts has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,293][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/log4j2.properties has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,293][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/jvm.options has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,293][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,293][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/sg_config.yml has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,293][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,294][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash.crtfull.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,294][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic-signed.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,294][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash-keystore.p12 has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,294][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic.all.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,294][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana.crtfull.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,295][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,295][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/truststore.jks has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,295][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana.all.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,295][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring.crt.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,295][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats-keystore.jks has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats-signed.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring.all.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana-keystore.jks has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats.crt.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,297][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana.crt.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,297][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/crl has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,297][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring.key.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,297][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash.all.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,297][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic.key.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/certs has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana.key.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/etc has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/etc/signing-ca.conf has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/etc/root-ca.conf has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring.crtfull.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic-keystore.p12 has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/NODE-es-elk-prod-6-4-q65q-signed.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic.crtfull.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash.key.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,300][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring-keystore.jks has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,300][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring-keystore.p12 has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,300][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats.all.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,300][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic.crt.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,300][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash-keystore.jks has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash.crt.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats-keystore.p12 has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana-keystore.p12 has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana-signed.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash-signed.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic-keystore.jks has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats.crtfull.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats.key.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/NODE-es-elk-prod-6-4-q65q-keystore.p12 has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/ca has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/ca/signing-ca has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/private/signing-ca.key has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/05.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/06.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/03.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/04.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/01.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/ca/signing-ca/db has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.crt.srl has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.db has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.crl.srl has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.db.attr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.crt.srl.old has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.db.attr.old has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.db.old has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/02.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/chain-ca.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca.crt has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca.crt has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/ca/root-ca has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/private/root-ca.key has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/01.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/ca/root-ca/db has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.crt.srl.old has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.crl.srl has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.crt.srl has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.db.attr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.db.old has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.db has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.db.attr.old has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/02.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/NODE-es-elk-prod-6-4-q65q-keystore.jks has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring-signed.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/NODE-es-elk-prod-6-4-q65q.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/sg_roles.yml has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/sg_internal_users.yml has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/sg_action_groups.yml has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/sg_roles_mapping.yml has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/elasticsearch.yml has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [aggs-matrix-stats]

[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [analysis-common]

[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [ingest-common]

[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [lang-expression]

[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [lang-mustache]

[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [lang-painless]

[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [mapper-extras]

[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [parent-join]

[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [percolator]

[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [rank-eval]

[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [reindex]

[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [repository-url]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [transport-netty4]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [tribe]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-core]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-deprecation]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-graph]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-logstash]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-ml]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-monitoring]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-rollup]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-security]

[2018-11-27T19:21:27,538][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-sql]

[2018-11-27T19:21:27,538][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-upgrade]

[2018-11-27T19:21:27,538][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-watcher]

[2018-11-27T19:21:27,538][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded plugin [discovery-gce]

[2018-11-27T19:21:27,538][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded plugin [repository-gcs]

[2018-11-27T19:21:27,539][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded plugin [search-guard-6]

Stalling for Elasticsearch…

[2018-11-27T19:21:31,675][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin com.floragunn.searchguard.SearchGuardPlugin

[2018-11-27T19:21:31,953][INFO ][o.e.d.DiscoveryModule ] [es-elk-prod-6-4-q65q] using discovery type [zen]

[2018-11-27T19:21:32,775][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] initialized

[2018-11-27T19:21:32,775][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] starting …

[2018-11-27T19:21:33,190][INFO ][o.e.t.TransportService ] [es-elk-prod-6-4-q65q] publish_address {10.79.1.5:9300}, bound_addresses {[::]:9300}

[2018-11-27T19:21:33,219][INFO ][o.e.b.BootstrapChecks ] [es-elk-prod-6-4-q65q] bound or publishing to a non-loopback address, enforcing bootstrap checks

[2018-11-27T19:21:33,239][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Check if searchguard index exists …

[2018-11-27T19:21:33,249][DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [es-elk-prod-6-4-q65q] no known master node, scheduling a retry

[2018-11-27T19:21:33,440][INFO ][o.e.c.g.GceInstancesServiceImpl] [es-elk-prod-6-4-q65q] starting GCE discovery service

Stalling for Elasticsearch…

[2018-11-27T19:21:34,668][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [es-elk-prod-6-4-q65q] SSL Problem General SSLEngine problem

javax.net.ssl.SSLHandshakeException: General SSLEngine problem

at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1521) ~[?:?]
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:528) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:802) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_191]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]

Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem

at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1709) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:318) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) ~[?:?]
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) ~[?:?]
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
... 19 more

Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:362) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:262) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:?]
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1626) ~[?:?]
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:?]

``

From the logfile it seems that you have disabled TLS on HTTP:

[2018-11-27T19:21:27,027][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Client Provider : JDK

[2018-11-27T19:21:27,027][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Server Provider : JDK

[2018-11-27T19:21:27,028][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS HTTP Provider : null

[2018-11-27T19:21:27,028][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for transport layer : [TLSv1.1, TLSv1.2]

[2018-11-27T19:21:27,028][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for HTTP layer : []

[

In your elasticsearch.yml you have set:

clientauth_mode: OPTIONAL

Is that on purpose? Because this configuration would not make much sense IMHO.

The exception you are seeing is happening on on the REST layer, do you use TLS client certificate authentication? If so, is the certificate for these calls correct?

Also, what is this curl call?

% Total % Received % Xferd Average Speed Time Time Time Current

                             Dload  Upload   Total   Spent    Left  Speed

0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0

curl: (35) SSL received a record that exceeded the maximum permissible length.

···

On Tuesday, November 27, 2018 at 9:42:40 PM UTC+1, Roman Kournjaev wrote:

This is one node setup I am trying to get running with the new version.
You are right the

Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust ancho

``

is repeated 20 times until the node comes up and then it is not repeated any more

On Tuesday, November 27, 2018 at 12:23:11 PM UTC-8, Jochen Kressin wrote:

Hi!

so just to make sure I understood the question correctly … when you say the stacktrace is repeating like 20 time you are refering to the stacktrace originating from

Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

Correct? And am I right in the assumption that if you see it like 20 times you also have roughly 20 nodes?

The stacktrace simply means that the certificate that was sent by one node in the cluster could not be verified against the root CA in the truststore on the receiving node. Are you sure you have updated the respective truststore file on all nodes?

Can you follow the instructions on the TLS troubleshooting guide and check whether the certificate chains are correct?

https://docs.search-guard.com/latest/troubleshooting-tls

Have a look at:

" Viewing the contents of your Key- and Truststore"

and

“Validating the certificate chain”

in particular.

On Tuesday, November 27, 2018 at 8:54:23 PM UTC+1, Roman Kournjaev wrote:

When asking questions, please provide the following information:

I am upgrading from 6.2.2 to 6.4.3 and have a strange exception during the startup process , i was wondering what that is ( the stack trace is repeated roughly 20 times i cut them out ) and how can i avoid it ?

sending incremental file list

searchguard/

searchguard/sg_action_groups.yml

searchguard/sg_config.yml

searchguard/sg_internal_users.yml

searchguard/sg_roles.yml

searchguard/sg_roles_mapping.yml

searchguard/ssl/

searchguard/ssl/beats-keystore.jks

searchguard/ssl/beats-keystore.p12

searchguard/ssl/beats-signed.pem

searchguard/ssl/beats.all.pem

searchguard/ssl/beats.crt.pem

searchguard/ssl/beats.crtfull.pem

searchguard/ssl/beats.csr

searchguard/ssl/beats.key.pem

searchguard/ssl/elastic-keystore.jks

searchguard/ssl/elastic-keystore.p12

searchguard/ssl/elastic-signed.pem

searchguard/ssl/elastic.all.pem

searchguard/ssl/elastic.crt.pem

searchguard/ssl/elastic.crtfull.pem

searchguard/ssl/elastic.csr

searchguard/ssl/elastic.key.pem

searchguard/ssl/kibana-keystore.jks

searchguard/ssl/kibana-keystore.p12

searchguard/ssl/kibana-signed.pem

searchguard/ssl/kibana.all.pem

searchguard/ssl/kibana.crt.pem

searchguard/ssl/kibana.crtfull.pem

searchguard/ssl/kibana.csr

searchguard/ssl/kibana.key.pem

searchguard/ssl/logstash-keystore.jks

searchguard/ssl/logstash-keystore.p12

searchguard/ssl/logstash-signed.pem

searchguard/ssl/logstash.all.pem

searchguard/ssl/logstash.crt.pem

searchguard/ssl/logstash.crtfull.pem

searchguard/ssl/logstash.csr

searchguard/ssl/logstash.key.pem

searchguard/ssl/monitoring-keystore.jks

searchguard/ssl/monitoring-keystore.p12

searchguard/ssl/monitoring-signed.pem

searchguard/ssl/monitoring.all.pem

searchguard/ssl/monitoring.crt.pem

searchguard/ssl/monitoring.crtfull.pem

searchguard/ssl/monitoring.csr

searchguard/ssl/monitoring.key.pem

searchguard/ssl/truststore.jks

searchguard/ssl/ca/

searchguard/ssl/ca/chain-ca.pem

searchguard/ssl/ca/root-ca.crt

searchguard/ssl/ca/root-ca.csr

searchguard/ssl/ca/root-ca.pem

searchguard/ssl/ca/signing-ca.crt

searchguard/ssl/ca/signing-ca.csr

searchguard/ssl/ca/signing-ca.pem

searchguard/ssl/ca/root-ca/

searchguard/ssl/ca/root-ca/01.pem

searchguard/ssl/ca/root-ca/02.pem

searchguard/ssl/ca/root-ca/db/

searchguard/ssl/ca/root-ca/db/root-ca.crl.srl

searchguard/ssl/ca/root-ca/db/root-ca.crt.srl

searchguard/ssl/ca/root-ca/db/root-ca.crt.srl.old

searchguard/ssl/ca/root-ca/db/root-ca.db

searchguard/ssl/ca/root-ca/db/root-ca.db.attr

searchguard/ssl/ca/root-ca/db/root-ca.db.attr.old

searchguard/ssl/ca/root-ca/db/root-ca.db.old

searchguard/ssl/ca/root-ca/private/

searchguard/ssl/ca/root-ca/private/root-ca.key

searchguard/ssl/ca/signing-ca/

searchguard/ssl/ca/signing-ca/01.pem

searchguard/ssl/ca/signing-ca/02.pem

searchguard/ssl/ca/signing-ca/03.pem

searchguard/ssl/ca/signing-ca/04.pem

searchguard/ssl/ca/signing-ca/05.pem

searchguard/ssl/ca/signing-ca/db/

searchguard/ssl/ca/signing-ca/db/signing-ca.crl.srl

searchguard/ssl/ca/signing-ca/db/signing-ca.crt.srl

searchguard/ssl/ca/signing-ca/db/signing-ca.crt.srl.old

searchguard/ssl/ca/signing-ca/db/signing-ca.db

searchguard/ssl/ca/signing-ca/db/signing-ca.db.attr

searchguard/ssl/ca/signing-ca/db/signing-ca.db.attr.old

searchguard/ssl/ca/signing-ca/db/signing-ca.db.old

searchguard/ssl/ca/signing-ca/private/

searchguard/ssl/ca/signing-ca/private/signing-ca.key

searchguard/ssl/certs/

searchguard/ssl/crl/

searchguard/ssl/etc/

searchguard/ssl/etc/root-ca.conf

searchguard/ssl/etc/signing-ca.conf

sent 181,886 bytes received 1,558 bytes 366,888.00 bytes/sec

total size is 176,413 speedup is 0.96

cluster.name: ${CLUSTER_NAME}

node:

name: ${HOSTNAME}

master: ${NODE_MASTER}

data: ${NODE_DATA}

ingest: ${NODE_INGEST}

comment out when running locally

network:

host:

  • 0.0.0.0

cloud:

gce:

project_id: ${PROJECT_ID}
zone: ${ZONE}

comment out when running locally

discovery:

zen:

hosts_provider: gce
minimum_master_nodes: ${MINIMUM_MASTER_NODES}

gce:

tags: ${TAGS}

http:

enabled: ${HTTP_ENABLE}

compression: true

cors:

enabled: ${HTTP_CORS_ENABLE}
allow-origin: ${HTTP_CORS_ALLOW_ORIGIN}

xpack.graph.enabled: false

xpack.ml.enabled: false

xpack.security.enabled: false

xpack.monitoring.enabled: true

xpack.watcher.enabled: true

searchguard:

ssl.transport:
    enabled: true
    enable_openssl_if_available: true
    keystore_type: JKS
    keystore_filepath: searchguard/ssl/${NODE_NAME}-keystore.jks
    keystore_password: ${KS_PWD}
    truststore_type: JKS
    truststore_filepath: searchguard/ssl/truststore.jks
    truststore_password: ${TS_PWD}
    enforce_hostname_verification: false
ssl.http:
    enabled: ${HTTP_SSL}
    clientauth_mode: OPTIONAL
    enable_openssl_if_available: true
    keystore_type: JKS
    keystore_filepath: searchguard/ssl/${NODE_NAME}-keystore.jks
    keystore_password: ${KS_PWD}
    truststore_type: JKS
    truststore_filepath: searchguard/ssl/truststore.jks
    truststore_password: ${TS_PWD}
authcz.admin_dn:
  - "CN=elastic ,OU=devops, C=COM"
enable_snapshot_restore_privilege: true
enterprise_modules_enabled: ${ENTERPRISE_ENABLED}

Generating keystore and certificate for node NODE-es-elk-prod-6-4-q65q

Warning:

The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using “keytool -importkeystore -srckeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -destkeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -deststoretype pkcs12”.

Generating certificate signing request for node NODE-es-elk-prod-6-4-q65q

Warning:

The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using “keytool -importkeystore -srckeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -destkeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -deststoretype pkcs12”.

Sign certificate request with CA

Using configuration from etc/signing-ca.conf

Check that the request matches the signature

Signature ok

Certificate Details:

    Serial Number: 6 (0x6)
    Validity
        Not Before: Nov 27 19:21:03 2018 GMT
        Not After : Nov 26 19:21:03 2020 GMT
    Subject:
        countryName               = COM
        organizationalUnitName    = SSL
        commonName                = NODE-es-elk-prod-6-4-q65q
    X509v3 extensions:
        X509v3 Key Usage: critical
            Digital Signature, Key Encipherment
        X509v3 Basic Constraints:
            CA:FALSE
        X509v3 Extended Key Usage:
            TLS Web Server Authentication, TLS Web Client Authentication
        X509v3 Subject Key Identifier:
            BF:B9:92:F6:62:2B:D5:FF:4E:44:F0:DB:83:56:40:59:14:78:15:A2
        X509v3 Authority Key Identifier:
            keyid:57:C2:B6:3C:34:3F:36:76:6A:65:5A:8C:DF:F0:04:2E:96:48:3B:DB
        X509v3 Subject Alternative Name:
            DNS:NODE-es-elk-prod-6-4-q65q, DNS:localhost, IP Address:127.0.0.1, Registered ID:1.2.3.4.5.5

Certificate is to be certified until Nov 26 19:21:03 2020 GMT (730 days)

Write out database with 1 new entries

Data Base Updated

Import back to keystore (including CA chain)

Certificate reply was installed in keystore

Warning:

The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using “keytool -importkeystore -srckeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -destkeystore NODE-es-elk-prod-6-4-q65q-keystore.jks -deststoretype pkcs12”.

Importing keystore NODE-es-elk-prod-6-4-q65q-keystore.jks to NODE-es-elk-prod-6-4-q65q-keystore.p12…

Entry for alias node-es-elk-prod-6-4-q65q successfully imported.

Import command completed: 1 entries successfully imported, 0 entries failed or cancelled

All done for NODE-es-elk-prod-6-4-q65q

Created elasticsearch keystore in /elasticsearch/config

Stalling for Elasticsearch…

Stalling for Elasticsearch…

Stalling for Elasticsearch…

Stalling for Elasticsearch…

[2018-11-27T19:21:24,748][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] initializing …

[2018-11-27T19:21:24,846][INFO ][o.e.e.NodeEnvironment ] [es-elk-prod-6-4-q65q] using [1] data paths, mounts [[/elasticsearch/data (/dev/sda1)]], net usable_space [1.9tb], net total_space [1.9tb], types [ext4]

[2018-11-27T19:21:24,847][INFO ][o.e.e.NodeEnvironment ] [es-elk-prod-6-4-q65q] heap size [14.9gb], compressed ordinary object pointers [true]

[2018-11-27T19:21:24,848][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] node name [es-elk-prod-6-4-q65q], node ID [Oq9l3kZMQuKhh7u3i71kpQ]

[2018-11-27T19:21:24,849][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] version[6.4.3], pid[227], build[default/tar/fe40335/2018-10-30T23:17:19.084789Z], OS[Linux/4.14.67+/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_191/25.191-b12]

[2018-11-27T19:21:24,850][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -Xms15g, -Xmx15g, -Des.path.home=/elasticsearch, -Des.path.conf=/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]

[2018-11-27T19:21:26,919][INFO ][c.f.s.SearchGuardPlugin ] ES Config path is /elasticsearch/config

[2018-11-27T19:21:26,981][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] OpenSSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSL

[2018-11-27T19:21:26,991][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Config directory is /elasticsearch/config/, from there the key- and truststore files are resolved relatively

[2018-11-27T19:21:26,998][INFO ][c.f.s.s.u.SSLCertificateHelper] No alias given, use the first one: node-es-elk-prod-6-4-q65q

[2018-11-27T19:21:27,004][WARN ][c.f.s.s.u.SSLCertificateHelper] Certificate chain for alias node-es-elk-prod-6-4-q65q contains a root certificate

[2018-11-27T19:21:27,027][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Client Provider : JDK

[2018-11-27T19:21:27,027][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Server Provider : JDK

[2018-11-27T19:21:27,028][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS HTTP Provider : null

[2018-11-27T19:21:27,028][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for transport layer : [TLSv1.1, TLSv1.2]

[2018-11-27T19:21:27,028][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for HTTP layer :

[2018-11-27T19:21:27,264][INFO ][c.f.s.SearchGuardPlugin ] Clustername: elk

[2018-11-27T19:21:27,292][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,292][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/scripts has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,293][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/log4j2.properties has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,293][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/jvm.options has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,293][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,293][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/sg_config.yml has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,293][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,294][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash.crtfull.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,294][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic-signed.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,294][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash-keystore.p12 has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,294][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic.all.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,294][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana.crtfull.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,295][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,295][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/truststore.jks has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,295][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana.all.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,295][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring.crt.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,295][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats-keystore.jks has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats-signed.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring.all.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana-keystore.jks has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,296][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats.crt.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,297][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana.crt.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,297][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/crl has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,297][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring.key.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,297][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash.all.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,297][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic.key.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/certs has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana.key.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/etc has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/etc/signing-ca.conf has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/etc/root-ca.conf has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,298][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring.crtfull.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic-keystore.p12 has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/NODE-es-elk-prod-6-4-q65q-signed.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic.crtfull.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,299][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash.key.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,300][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring-keystore.jks has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,300][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring-keystore.p12 has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,300][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats.all.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,300][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic.crt.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,300][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash-keystore.jks has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash.crt.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats-keystore.p12 has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana-keystore.p12 has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/kibana-signed.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/logstash-signed.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,301][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/elastic-keystore.jks has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats.crtfull.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/beats.key.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/NODE-es-elk-prod-6-4-q65q-keystore.p12 has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/ca has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/ca/signing-ca has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,302][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/private/signing-ca.key has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/05.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/06.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/03.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/04.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/01.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/ca/signing-ca/db has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,303][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.crt.srl has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.db has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.crl.srl has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.db.attr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.crt.srl.old has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.db.attr.old has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/db/signing-ca.db.old has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,304][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca/02.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/chain-ca.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca.crt has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,305][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/signing-ca.crt has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/ca/root-ca has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/private/root-ca.key has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/01.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] Directory /elasticsearch/config/searchguard/ssl/ca/root-ca/db has insecure file permissions (should be 0700)

[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.crt.srl.old has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,306][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.crl.srl has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.crt.srl has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.db.attr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.db.old has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.db has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/db/root-ca.db.attr.old has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/ca/root-ca/02.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,307][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/NODE-es-elk-prod-6-4-q65q-keystore.jks has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/monitoring-signed.pem has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/ssl/NODE-es-elk-prod-6-4-q65q.csr has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/sg_roles.yml has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/sg_internal_users.yml has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/sg_action_groups.yml has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/searchguard/sg_roles_mapping.yml has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,308][WARN ][c.f.s.SearchGuardPlugin ] File /elasticsearch/config/elasticsearch.yml has insecure file permissions (should be 0600)

[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [aggs-matrix-stats]

[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [analysis-common]

[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [ingest-common]

[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [lang-expression]

[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [lang-mustache]

[2018-11-27T19:21:27,535][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [lang-painless]

[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [mapper-extras]

[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [parent-join]

[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [percolator]

[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [rank-eval]

[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [reindex]

[2018-11-27T19:21:27,536][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [repository-url]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [transport-netty4]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [tribe]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-core]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-deprecation]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-graph]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-logstash]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-ml]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-monitoring]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-rollup]

[2018-11-27T19:21:27,537][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-security]

[2018-11-27T19:21:27,538][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-sql]

[2018-11-27T19:21:27,538][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-upgrade]

[2018-11-27T19:21:27,538][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded module [x-pack-watcher]

[2018-11-27T19:21:27,538][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded plugin [discovery-gce]

[2018-11-27T19:21:27,538][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded plugin [repository-gcs]

[2018-11-27T19:21:27,539][INFO ][o.e.p.PluginsService ] [es-elk-prod-6-4-q65q] loaded plugin [search-guard-6]

Stalling for Elasticsearch…

[2018-11-27T19:21:31,675][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin com.floragunn.searchguard.SearchGuardPlugin

[2018-11-27T19:21:31,953][INFO ][o.e.d.DiscoveryModule ] [es-elk-prod-6-4-q65q] using discovery type [zen]

[2018-11-27T19:21:32,775][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] initialized

[2018-11-27T19:21:32,775][INFO ][o.e.n.Node ] [es-elk-prod-6-4-q65q] starting …

[2018-11-27T19:21:33,190][INFO ][o.e.t.TransportService ] [es-elk-prod-6-4-q65q] publish_address {10.79.1.5:9300}, bound_addresses {[::]:9300}

[2018-11-27T19:21:33,219][INFO ][o.e.b.BootstrapChecks ] [es-elk-prod-6-4-q65q] bound or publishing to a non-loopback address, enforcing bootstrap checks

[2018-11-27T19:21:33,239][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Check if searchguard index exists …

[2018-11-27T19:21:33,249][DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [es-elk-prod-6-4-q65q] no known master node, scheduling a retry

[2018-11-27T19:21:33,440][INFO ][o.e.c.g.GceInstancesServiceImpl] [es-elk-prod-6-4-q65q] starting GCE discovery service

Stalling for Elasticsearch…

[2018-11-27T19:21:34,668][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [es-elk-prod-6-4-q65q] SSL Problem General SSLEngine problem

javax.net.ssl.SSLHandshakeException: General SSLEngine problem

at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1521) ~[?:?]
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:528) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:802) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_191]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]

Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem

at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1709) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:318) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) ~[?:?]
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) ~[?:?]
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
... 19 more

Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:362) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:262) ~[?:?]

``