[2018-03-13T15:42:49,182][INFO ][o.e.n.Node ] [] initializing ... [2018-03-13T15:42:49,255][INFO ][o.e.e.NodeEnvironment ] [Qrogsm-] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [182.2gb], net total_space [274gb], types [rootfs] [2018-03-13T15:42:49,255][INFO ][o.e.e.NodeEnvironment ] [Qrogsm-] heap size [989.8mb], compressed ordinary object pointers [true] [2018-03-13T15:42:49,301][INFO ][o.e.n.Node ] node name [Qrogsm-] derived from node ID [Qrogsm-GSiOX4dG3YeCUZg]; set [node.name] to override [2018-03-13T15:42:49,301][INFO ][o.e.n.Node ] version[6.2.1], pid[25015], build[7299dc3/2018-02-07T19:34:26.990113Z], OS[Linux/3.10.0-327.28.3.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_121/25.121-b13] [2018-03-13T15:42:49,301][INFO ][o.e.n.Node ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.lOgAWx8d, -XX:+HeapDumpOnOutOfMemoryError, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/home_local/elasticsearch/elasticsearch-6.2.1, -Des.path.conf=/home_local/elasticsearch/elasticsearch-6.2.1/config] [2018-03-13T15:42:50,144][INFO ][c.f.s.SearchGuardPlugin ] ES Config path is /home_local/elasticsearch/elasticsearch-6.2.1/config [2018-03-13T15:42:50,145][DEBUG][c.f.s.SearchGuardPlugin ] Client side initiated TLS renegotiation forcibly disabled. This can prevent DoS attacks. (jdk.tls.rejectClientInitiatedRenegotiation set to true). [2018-03-13T15:42:50,200][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSL [2018-03-13T15:42:50,346][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] JVM supports the following 57 ciphers for https [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV, TLS_DH_anon_WITH_AES_128_GCM_SHA256, TLS_DH_anon_WITH_AES_128_CBC_SHA256, TLS_ECDH_anon_WITH_AES_128_CBC_SHA, TLS_DH_anon_WITH_AES_128_CBC_SHA, TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA, SSL_DH_anon_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_DH_anon_WITH_DES_CBC_SHA, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA, SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA, TLS_RSA_WITH_NULL_SHA256, TLS_ECDHE_ECDSA_WITH_NULL_SHA, TLS_ECDHE_RSA_WITH_NULL_SHA, SSL_RSA_WITH_NULL_SHA, TLS_ECDH_ECDSA_WITH_NULL_SHA, TLS_ECDH_RSA_WITH_NULL_SHA, TLS_ECDH_anon_WITH_NULL_SHA, SSL_RSA_WITH_NULL_MD5, TLS_KRB5_WITH_3DES_EDE_CBC_SHA, TLS_KRB5_WITH_3DES_EDE_CBC_MD5, TLS_KRB5_WITH_DES_CBC_SHA, TLS_KRB5_WITH_DES_CBC_MD5, TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA, TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5] [2018-03-13T15:42:50,349][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] JVM supports the following 57 ciphers for transport [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV, TLS_DH_anon_WITH_AES_128_GCM_SHA256, TLS_DH_anon_WITH_AES_128_CBC_SHA256, TLS_ECDH_anon_WITH_AES_128_CBC_SHA, TLS_DH_anon_WITH_AES_128_CBC_SHA, TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA, SSL_DH_anon_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_DH_anon_WITH_DES_CBC_SHA, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA, SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA, TLS_RSA_WITH_NULL_SHA256, TLS_ECDHE_ECDSA_WITH_NULL_SHA, TLS_ECDHE_RSA_WITH_NULL_SHA, SSL_RSA_WITH_NULL_SHA, TLS_ECDH_ECDSA_WITH_NULL_SHA, TLS_ECDH_RSA_WITH_NULL_SHA, TLS_ECDH_anon_WITH_NULL_SHA, SSL_RSA_WITH_NULL_MD5, TLS_KRB5_WITH_3DES_EDE_CBC_SHA, TLS_KRB5_WITH_3DES_EDE_CBC_MD5, TLS_KRB5_WITH_DES_CBC_SHA, TLS_KRB5_WITH_DES_CBC_MD5, TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA, TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5] [2018-03-13T15:42:50,350][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Config directory is /home_local/elasticsearch/elasticsearch-6.2.1/config/, from there the key- and truststore files are resolved relatively [2018-03-13T15:42:50,350][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.transport.pemcert_filepath is HadoopCertificates/hadoop.rcc.uchicago.edu-chain.crt [2018-03-13T15:42:50,350][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved HadoopCertificates/hadoop.rcc.uchicago.edu-chain.crt to /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates/hadoop.rcc.uchicago.edu-chain.crt against /home_local/elasticsearch/elasticsearch-6.2.1/config [2018-03-13T15:42:50,350][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.transport.pemkey_filepath is HadoopCertificates/hadoop.rcc.uchicago.edu-key.pem [2018-03-13T15:42:50,351][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved HadoopCertificates/hadoop.rcc.uchicago.edu-key.pem to /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates/hadoop.rcc.uchicago.edu-key.pem against /home_local/elasticsearch/elasticsearch-6.2.1/config [2018-03-13T15:42:50,351][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.transport.pemtrustedcas_filepath is HadoopCertificates/incommon.crt [2018-03-13T15:42:50,351][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved HadoopCertificates/incommon.crt to /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates/incommon.crt against /home_local/elasticsearch/elasticsearch-6.2.1/config [2018-03-13T15:42:50,429][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.http.pemtrustedcas_filepath is HadoopCertificates/incommon.crt [2018-03-13T15:42:50,429][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved HadoopCertificates/incommon.crt to /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates/incommon.crt against /home_local/elasticsearch/elasticsearch-6.2.1/config [2018-03-13T15:42:50,429][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.http.pemcert_filepath is HadoopCertificates/hadoop.rcc.uchicago.edu-chain.crt [2018-03-13T15:42:50,429][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved HadoopCertificates/hadoop.rcc.uchicago.edu-chain.crt to /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates/hadoop.rcc.uchicago.edu-chain.crt against /home_local/elasticsearch/elasticsearch-6.2.1/config [2018-03-13T15:42:50,429][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.http.pemkey_filepath is HadoopCertificates/hadoop.rcc.uchicago.edu-key.pem [2018-03-13T15:42:50,430][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved HadoopCertificates/hadoop.rcc.uchicago.edu-key.pem to /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates/hadoop.rcc.uchicago.edu-key.pem against /home_local/elasticsearch/elasticsearch-6.2.1/config [2018-03-13T15:42:50,438][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] AES-256 not supported, max key length for AES is 128 bit. (That is not an issue, it just limits possible encryption strength. To enable AES 256 install 'Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files') [2018-03-13T15:42:50,438][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Client Provider : JDK [2018-03-13T15:42:50,438][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Server Provider : JDK [2018-03-13T15:42:50,438][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS HTTP Provider : JDK [2018-03-13T15:42:50,438][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] sslTransportClientProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256] [2018-03-13T15:42:50,438][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] sslTransportServerProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256] [2018-03-13T15:42:50,439][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] sslHTTPProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256] [2018-03-13T15:42:50,439][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for transport layer : [TLSv1.2, TLSv1.1] [2018-03-13T15:42:50,439][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for HTTP layer : [TLSv1.2, TLSv1.1] [2018-03-13T15:42:50,627][INFO ][c.f.s.SearchGuardPlugin ] Clustername: ELK1 [2018-03-13T15:42:50,627][DEBUG][c.f.s.SearchGuardPlugin ] This node [Qrogsm-] is a transportClient: false/tribeNode: false/tribeNodeClient: false [2018-03-13T15:42:50,668][DEBUG][c.f.s.s.ReflectionHelper ] Loaded module Module [type=DLSFLS, implementing class=com.floragunn.searchguard.configuration.SearchGuardFlsDlsIndexSearcherWrapper] [2018-03-13T15:42:50,676][WARN ][c.f.s.SearchGuardPlugin ] Directory /home_local/elasticsearch/elasticsearch-6.2.1/config has insecure file permissions (should be 0700) [2018-03-13T15:42:50,676][WARN ][c.f.s.SearchGuardPlugin ] File /home_local/elasticsearch/elasticsearch-6.2.1/config/kirk.pem has insecure file permissions (should be 0600) [2018-03-13T15:42:50,676][WARN ][c.f.s.SearchGuardPlugin ] File /home_local/elasticsearch/elasticsearch-6.2.1/config/esnode.pem has insecure file permissions (should be 0600) [2018-03-13T15:42:50,677][WARN ][c.f.s.SearchGuardPlugin ] File /home_local/elasticsearch/elasticsearch-6.2.1/config/root-ca.pem has insecure file permissions (should be 0600) [2018-03-13T15:42:50,677][WARN ][c.f.s.SearchGuardPlugin ] File /home_local/elasticsearch/elasticsearch-6.2.1/config/esnode-key.pem has insecure file permissions (should be 0600) [2018-03-13T15:42:50,677][WARN ][c.f.s.SearchGuardPlugin ] File /home_local/elasticsearch/elasticsearch-6.2.1/config/kirk-key.pem has insecure file permissions (should be 0600) [2018-03-13T15:42:50,677][WARN ][c.f.s.SearchGuardPlugin ] File /home_local/elasticsearch/elasticsearch-6.2.1/config/log.txt has insecure file permissions (should be 0600) [2018-03-13T15:42:50,677][WARN ][c.f.s.SearchGuardPlugin ] Directory /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates has insecure file permissions (should be 0700) [2018-03-13T15:42:50,677][WARN ][c.f.s.SearchGuardPlugin ] Directory /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates/5 has insecure file permissions (should be 0700) [2018-03-13T15:42:50,677][WARN ][c.f.s.SearchGuardPlugin ] File /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates/5/hadoop_rcc_uchicago_edu.p7b has insecure file permissions (should be 0600) [2018-03-13T15:42:50,678][WARN ][c.f.s.SearchGuardPlugin ] Directory /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates/6 has insecure file permissions (should be 0700) [2018-03-13T15:42:50,678][WARN ][c.f.s.SearchGuardPlugin ] File /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates/6/hadoop_rcc_uchicago_edu.cer has insecure file permissions (should be 0600) [2018-03-13T15:42:50,678][WARN ][c.f.s.SearchGuardPlugin ] Directory /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates/2 has insecure file permissions (should be 0700) [2018-03-13T15:42:50,678][WARN ][c.f.s.SearchGuardPlugin ] File /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates/2/hadoop_rcc_uchicago_edu_interm.cer has insecure file permissions (should be 0600) [2018-03-13T15:42:50,678][WARN ][c.f.s.SearchGuardPlugin ] Directory /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates/3 has insecure file permissions (should be 0700) [2018-03-13T15:42:50,678][WARN ][c.f.s.SearchGuardPlugin ] File /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates/3/hadoop_rcc_uchicago_edu_interm.cer has insecure file permissions (should be 0600) [2018-03-13T15:42:50,678][WARN ][c.f.s.SearchGuardPlugin ] Directory /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates/4 has insecure file permissions (should be 0700) [2018-03-13T15:42:50,678][WARN ][c.f.s.SearchGuardPlugin ] File /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates/4/hadoop_rcc_uchicago_edu.crt has insecure file permissions (should be 0600) [2018-03-13T15:42:50,679][WARN ][c.f.s.SearchGuardPlugin ] File /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates/letter.txt has insecure file permissions (should be 0600) [2018-03-13T15:42:50,679][WARN ][c.f.s.SearchGuardPlugin ] Directory /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates/1 has insecure file permissions (should be 0700) [2018-03-13T15:42:50,679][WARN ][c.f.s.SearchGuardPlugin ] File /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates/1/hadoop_rcc_uchicago_edu_cert.cer has insecure file permissions (should be 0600) [2018-03-13T15:42:50,679][WARN ][c.f.s.SearchGuardPlugin ] File /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates/incommon.crt has insecure file permissions (should be 0600) [2018-03-13T15:42:50,679][WARN ][c.f.s.SearchGuardPlugin ] File /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates/hadoop.rcc.uchicago.edu-chain.crt has insecure file permissions (should be 0600) [2018-03-13T15:42:50,679][WARN ][c.f.s.SearchGuardPlugin ] Directory /home_local/elasticsearch/elasticsearch-6.2.1/config/versions has insecure file permissions (should be 0700) [2018-03-13T15:42:50,683][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [aggs-matrix-stats] [2018-03-13T15:42:50,683][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [analysis-common] [2018-03-13T15:42:50,683][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [ingest-common] [2018-03-13T15:42:50,683][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [lang-expression] [2018-03-13T15:42:50,683][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [lang-mustache] [2018-03-13T15:42:50,683][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [lang-painless] [2018-03-13T15:42:50,684][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [mapper-extras] [2018-03-13T15:42:50,684][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [parent-join] [2018-03-13T15:42:50,684][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [percolator] [2018-03-13T15:42:50,684][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [rank-eval] [2018-03-13T15:42:50,684][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [reindex] [2018-03-13T15:42:50,684][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [repository-url] [2018-03-13T15:42:50,684][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [transport-netty4] [2018-03-13T15:42:50,684][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [tribe] [2018-03-13T15:42:50,685][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded plugin [search-guard-6] [2018-03-13T15:42:50,704][INFO ][c.f.s.SearchGuardPlugin ] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in elasticsearch.yml [2018-03-13T15:42:52,322][INFO ][c.f.s.a.i.AuditLogImpl ] Configured categories on rest layer to ignore: [AUTHENTICATED, GRANTED_PRIVILEGES] [2018-03-13T15:42:52,323][INFO ][c.f.s.a.i.AuditLogImpl ] Configured categories on transport layer to ignore: [AUTHENTICATED, GRANTED_PRIVILEGES] [2018-03-13T15:42:52,323][INFO ][c.f.s.a.i.AuditLogImpl ] Configured Users to ignore: [kibanaserver] [2018-03-13T15:42:52,323][INFO ][c.f.s.a.i.AuditLogImpl ] Configured Requests to ignore: [] [2018-03-13T15:42:52,323][DEBUG][c.f.s.a.i.AuditLogImpl ] Create new executor with threadPoolSize: 10 and maxQueueLen: 100000 [2018-03-13T15:42:52,324][INFO ][c.f.s.a.i.AuditLogImpl ] Audit Log class: ESAuditLog [2018-03-13T15:42:52,324][DEBUG][c.f.s.a.i.AuditLogImpl ] Security Manager present [2018-03-13T15:42:52,325][DEBUG][c.f.s.a.i.AuditLogImpl ] Shutdown Hook registered [2018-03-13T15:42:52,325][DEBUG][c.f.s.s.ReflectionHelper ] Loaded module Module [type=AUDITLOG, implementing class=com.floragunn.searchguard.auditlog.impl.AuditLogImpl] [2018-03-13T15:42:52,326][DEBUG][c.f.s.SearchGuardPlugin ] Using com.floragunn.searchguard.transport.DefaultInterClusterRequestEvaluator as intercluster request evaluator class [2018-03-13T15:42:52,328][DEBUG][c.f.s.s.ReflectionHelper ] Loaded module Module [type=MULTITENANCY, implementing class=com.floragunn.searchguard.configuration.PrivilegesInterceptorImpl] [2018-03-13T15:42:52,332][DEBUG][c.f.s.c.AdminDNs ] CN=hadoop.rcc.uchicago.edu,OU=IT Services - Self Enrollment,O=University of Chicago,STREET=6045 S. Kenwood,L=Chicago,ST=IL,2.5.4.17=#13053630363337,C=US is registered as an admin dn [2018-03-13T15:42:52,333][DEBUG][c.f.s.c.AdminDNs ] Loaded 1 admin DN's [CN=hadoop.rcc.uchicago.edu,OU=IT Services - Self Enrollment,O=University of Chicago,STREET=6045 S. Kenwood,L=Chicago,ST=IL,2.5.4.17=#13053630363337,C=US] [2018-03-13T15:42:52,334][DEBUG][c.f.s.c.AdminDNs ] Loaded 0 impersonation DN's {} [2018-03-13T15:42:52,334][DEBUG][c.f.s.c.AdminDNs ] Loaded 0 impersonation users for REST {} [2018-03-13T15:42:52,336][DEBUG][c.f.s.c.ConfigurationLoader] Index is: searchguard [2018-03-13T15:42:52,337][DEBUG][c.f.s.c.LegacyConfigurationLoader] Index is: searchguard [2018-03-13T15:42:52,338][DEBUG][c.f.s.c.IndexBaseConfigurationRepository] Subscribe on configuration changes by type config with listener com.floragunn.searchguard.http.XFFResolver@6d963d70 [2018-03-13T15:42:52,522][DEBUG][c.f.s.c.IndexBaseConfigurationRepository] Subscribe on configuration changes by type config with listener com.floragunn.searchguard.auth.BackendRegistry@3fdcde7a [2018-03-13T15:42:52,618][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin com.floragunn.searchguard.SearchGuardPlugin [2018-03-13T15:42:52,633][DEBUG][c.f.s.h.SearchGuardHttpServerTransport] [Qrogsm-] using max_chunk_size[8kb], max_header_size[8kb], max_initial_line_length[4kb], max_content_length[100mb], receive_predictor[64kb->64kb], pipelining[true], pipelining_max_events[10000] [2018-03-13T15:42:52,729][INFO ][o.e.d.DiscoveryModule ] [Qrogsm-] using discovery type [zen] [2018-03-13T15:42:53,226][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] Globally disabled endpoints: {} [2018-03-13T15:42:53,226][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] No disabled endpoints/methods for permitted role sg_all_access found, allowing all [2018-03-13T15:42:53,227][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] Globally disabled endpoints: {} [2018-03-13T15:42:53,227][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] No disabled endpoints/methods for permitted role sg_all_access found, allowing all [2018-03-13T15:42:53,228][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] Globally disabled endpoints: {} [2018-03-13T15:42:53,228][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] No disabled endpoints/methods for permitted role sg_all_access found, allowing all [2018-03-13T15:42:53,229][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] Globally disabled endpoints: {} [2018-03-13T15:42:53,230][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] No disabled endpoints/methods for permitted role sg_all_access found, allowing all [2018-03-13T15:42:53,231][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] Globally disabled endpoints: {} [2018-03-13T15:42:53,231][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] No disabled endpoints/methods for permitted role sg_all_access found, allowing all [2018-03-13T15:42:53,231][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] Globally disabled endpoints: {} [2018-03-13T15:42:53,232][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] No disabled endpoints/methods for permitted role sg_all_access found, allowing all [2018-03-13T15:42:53,233][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] Globally disabled endpoints: {} [2018-03-13T15:42:53,233][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] No disabled endpoints/methods for permitted role sg_all_access found, allowing all [2018-03-13T15:42:53,234][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] Globally disabled endpoints: {} [2018-03-13T15:42:53,234][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] No disabled endpoints/methods for permitted role sg_all_access found, allowing all [2018-03-13T15:42:53,235][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] Globally disabled endpoints: {} [2018-03-13T15:42:53,235][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] No disabled endpoints/methods for permitted role sg_all_access found, allowing all [2018-03-13T15:42:53,235][DEBUG][c.f.s.s.ReflectionHelper ] Loaded module Module [type=REST_MANAGEMENT_API, implementing class=com.floragunn.searchguard.dlic.rest.api.SearchGuardRestApiActions] [2018-03-13T15:42:53,235][DEBUG][c.f.s.SearchGuardPlugin ] Added 9 management rest handler(s) [2018-03-13T15:42:53,235][INFO ][c.f.s.SearchGuardPlugin ] 4 Search Guard modules loaded so far: [Module [type=AUDITLOG, implementing class=com.floragunn.searchguard.auditlog.impl.AuditLogImpl], Module [type=DLSFLS, implementing class=com.floragunn.searchguard.configuration.SearchGuardFlsDlsIndexSearcherWrapper], Module [type=REST_MANAGEMENT_API, implementing class=com.floragunn.searchguard.dlic.rest.api.SearchGuardRestApiActions], Module [type=MULTITENANCY, implementing class=com.floragunn.searchguard.configuration.PrivilegesInterceptorImpl]] [2018-03-13T15:42:53,236][INFO ][o.e.n.Node ] initialized [2018-03-13T15:42:53,236][INFO ][o.e.n.Node ] [Qrogsm-] starting ... [2018-03-13T15:42:53,287][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport] [Qrogsm-] using profile[default], worker_count[96], port[9300-9400], bind_host[[127.0.0.1, 172.25.180.171, hadoop.rcc.uchicago.edu]], publish_host[[]], compress[false], connect_timeout[30s], connections_per_node[2/3/6/1/1], receive_predictor[64kb->64kb] [2018-03-13T15:42:53,297][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport] [Qrogsm-] binding server bootstrap to: [127.0.0.1, 172.25.180.171, 128.135.112.122] [2018-03-13T15:42:53,370][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport] [Qrogsm-] Bound profile [default] to address {127.0.0.1:9300} [2018-03-13T15:42:53,373][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport] [Qrogsm-] Bound profile [default] to address {172.25.180.171:9300} [2018-03-13T15:42:53,375][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport] [Qrogsm-] Bound profile [default] to address {128.135.112.122:9300} [2018-03-13T15:42:53,377][INFO ][o.e.t.TransportService ] [Qrogsm-] publish_address {128.135.112.122:9300}, bound_addresses {127.0.0.1:9300}, {172.25.180.171:9300}, {128.135.112.122:9300} [2018-03-13T15:42:53,399][INFO ][o.e.b.BootstrapChecks ] [Qrogsm-] bound or publishing to a non-loopback address, enforcing bootstrap checks [2018-03-13T15:42:53,412][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Check if searchguard index exists ... [2018-03-13T15:42:53,418][DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [Qrogsm-] no known master node, scheduling a retry [2018-03-13T15:42:56,479][INFO ][o.e.c.s.MasterService ] [Qrogsm-] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {Qrogsm-}{Qrogsm-GSiOX4dG3YeCUZg}{1hpk979EQwi-HgsYti0GTA}{hadoop.rcc.uchicago.edu}{128.135.112.122:9300} [2018-03-13T15:42:56,488][INFO ][o.e.c.s.ClusterApplierService] [Qrogsm-] new_master {Qrogsm-}{Qrogsm-GSiOX4dG3YeCUZg}{1hpk979EQwi-HgsYti0GTA}{hadoop.rcc.uchicago.edu}{128.135.112.122:9300}, reason: apply cluster state (from master [master {Qrogsm-}{Qrogsm-GSiOX4dG3YeCUZg}{1hpk979EQwi-HgsYti0GTA}{hadoop.rcc.uchicago.edu}{128.135.112.122:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]]) [2018-03-13T15:42:56,495][DEBUG][c.f.s.c.ClusterInfoHolder] Cluster Info Holder now initialized for 'nodes' [2018-03-13T15:42:56,526][DEBUG][c.f.s.h.SearchGuardHttpServerTransport] [Qrogsm-] Bound http to address {127.0.0.1:9200} [2018-03-13T15:42:56,528][DEBUG][c.f.s.h.SearchGuardHttpServerTransport] [Qrogsm-] Bound http to address {172.25.180.171:9200} [2018-03-13T15:42:56,530][DEBUG][c.f.s.h.SearchGuardHttpServerTransport] [Qrogsm-] Bound http to address {128.135.112.122:9200} [2018-03-13T15:42:56,531][INFO ][c.f.s.h.SearchGuardHttpServerTransport] [Qrogsm-] publish_address {128.135.112.122:9200}, bound_addresses {127.0.0.1:9200}, {172.25.180.171:9200}, {128.135.112.122:9200} [2018-03-13T15:42:56,531][INFO ][o.e.n.Node ] [Qrogsm-] started [2018-03-13T15:42:56,586][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.03.13 [2018-03-13T15:42:56,663][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.02.21 [2018-03-13T15:42:56,672][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.02.19 [2018-03-13T15:42:56,681][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.02.20 [2018-03-13T15:42:56,689][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index searchguard [2018-03-13T15:42:56,693][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.02.22 [2018-03-13T15:42:56,700][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.02.16 [2018-03-13T15:42:56,707][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.02.17 [2018-03-13T15:42:56,715][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.02.18 [2018-03-13T15:42:57,142][INFO ][o.e.g.GatewayService ] [Qrogsm-] recovered [9] indices into cluster_state [2018-03-13T15:42:57,142][DEBUG][c.f.s.c.IndexBaseConfigurationRepository] Node started, try to initialize it. Wait for at least yellow cluster state.... [2018-03-13T15:42:57,173][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.03.13 [2018-03-13T15:42:57,607][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.02.22 [2018-03-13T15:42:57,878][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.02.21 [2018-03-13T15:42:58,415][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.02.20 [2018-03-13T15:42:58,699][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.02.19 [2018-03-13T15:42:59,018][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.02.18 [2018-03-13T15:42:59,542][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.02.17 [2018-03-13T15:42:59,825][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.02.16 [2018-03-13T15:43:00,355][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index searchguard [2018-03-13T15:43:00,499][INFO ][o.e.c.r.a.AllocationService] [Qrogsm-] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[searchguard][0]] ...]). [2018-03-13T15:43:00,532][DEBUG][c.f.s.c.IndexBaseConfigurationRepository] Try to load config ... [2018-03-13T15:43:00,532][DEBUG][c.f.s.c.IndexBaseConfigurationRepository] sg index exists and was created with ES 6 (new layout) [2018-03-13T15:43:00,580][DEBUG][c.f.s.c.ConfigurationLoader] Received config for config (of [config, roles, rolesmapping, internalusers, actiongroups]) with current latch value=4 [2018-03-13T15:43:00,582][DEBUG][c.f.s.c.ConfigurationLoader] Received config for roles (of [config, roles, rolesmapping, internalusers, actiongroups]) with current latch value=3 [2018-03-13T15:43:00,582][DEBUG][c.f.s.c.ConfigurationLoader] Received config for rolesmapping (of [config, roles, rolesmapping, internalusers, actiongroups]) with current latch value=2 [2018-03-13T15:43:00,583][DEBUG][c.f.s.c.ConfigurationLoader] Received config for internalusers (of [config, roles, rolesmapping, internalusers, actiongroups]) with current latch value=1 [2018-03-13T15:43:00,584][DEBUG][c.f.s.c.ConfigurationLoader] Received config for actiongroups (of [config, roles, rolesmapping, internalusers, actiongroups]) with current latch value=0 [2018-03-13T15:43:00,590][DEBUG][c.f.s.c.IndexBaseConfigurationRepository] Notify com.floragunn.searchguard.http.XFFResolver@6d963d70 listener about change configuration with type config [2018-03-13T15:43:00,591][DEBUG][c.f.s.c.IndexBaseConfigurationRepository] Notify com.floragunn.searchguard.auth.BackendRegistry@3fdcde7a listener about change configuration with type config [2018-03-13T15:43:00,600][DEBUG][c.f.s.s.ReflectionHelper ] Loaded module Module [type=LDAP_AUTHORIZATION_BACKEND, implementing class=com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend] [2018-03-13T15:43:00,603][DEBUG][c.f.s.s.ReflectionHelper ] Loaded module Module [type=LDAP_AUTHENTICATION_BACKEND, implementing class=com.floragunn.dlic.auth.ldap.backend.LDAPAuthenticationBackend] [2018-03-13T15:43:00,604][DEBUG][c.f.s.s.ReflectionHelper ] Loaded module Module [type=HTTP_BASIC_AUTHENTICATOR, implementing class=com.floragunn.searchguard.http.HTTPBasicAuthenticator] [2018-03-13T15:43:00,605][DEBUG][c.f.s.s.ReflectionHelper ] Loaded module Module [type=INTERNAL_USERS_AUTHENTICATION_BACKEND, implementing class=com.floragunn.searchguard.auth.internal.InternalAuthenticationBackend] [2018-03-13T15:43:00,606][DEBUG][c.f.s.s.ReflectionHelper ] Loaded module Module [type=HTTP_BASIC_AUTHENTICATOR, implementing class=com.floragunn.searchguard.http.HTTPBasicAuthenticator] [2018-03-13T15:43:00,615][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Info: SearchGuardLicense [uid=00000000-0000-0000-0000-000000000000, type=TRIAL, issueDate=2018-02-16, expiryDate=2018-04-19, issuedTo=The world, issuer=floragunn GmbH, startDate=2018-02-16, majorVersion=6, clusterName=*, allowedNodeCount=2147483647, msgs=[], expiresInDays=36, isExpired=false, valid=true, action=, prodUsage=Yes, one cluster with all commercial features and unlimited nodes per cluster., clusterService=org.elasticsearch.cluster.service.ClusterService@13b22890, getMsgs()=[], getExpiresInDays()=36, isExpired()=false, isValid()=true, getAction()=, getProdUsage()=Yes, one cluster with all commercial features and unlimited nodes per cluster.] [2018-03-13T15:43:00,615][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Type: TRIAL, valid [2018-03-13T15:43:00,616][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Node 'Qrogsm-' initialized [2018-03-13T15:43:03,762][DEBUG][c.f.s.a.BackendRegistry ] Try to extract auth creds from basic http authenticator [2018-03-13T15:43:24,164][DEBUG][c.f.s.a.BackendRegistry ] Try to extract auth creds from basic http authenticator [2018-03-13T15:43:24,193][DEBUG][c.f.s.a.BackendRegistry ] ivy2 not cached, return from internal backend directly [2018-03-13T15:43:24,196][DEBUG][c.f.s.a.BackendRegistry ] Can not authenticate ivy2 due to com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[ivy2 not found] com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[ivy2 not found] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2218) ~[guava-23.0.jar:?] at com.google.common.cache.LocalCache.get(LocalCache.java:4147) ~[guava-23.0.jar:?] at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:5053) ~[guava-23.0.jar:?] at com.floragunn.searchguard.auth.BackendRegistry.authcz(BackendRegistry.java:549) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.auth.BackendRegistry.authenticate(BackendRegistry.java:416) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter.checkAndAuthenticateRequest(SearchGuardRestFilter.java:125) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter.access$000(SearchGuardRestFilter.java:48) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:75) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240) [elasticsearch-6.2.1.jar:6.2.1] at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336) [elasticsearch-6.2.1.jar:6.2.1] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174) [elasticsearch-6.2.1.jar:6.2.1] at com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [search-guard-ssl-6.2.1-25.1.jar:6.2.1-25.1] at org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:500) [transport-netty4-client-6.2.1.jar:6.2.1] at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:80) [transport-netty4-client-6.2.1.jar:6.2.1] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68) [transport-netty4-client-6.2.1.jar:6.2.1] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1336) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121] Caused by: org.elasticsearch.ElasticsearchSecurityException: ivy2 not found at com.floragunn.searchguard.auth.internal.InternalAuthenticationBackend.authenticate(InternalAuthenticationBackend.java:102) ~[?:?] at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:555) ~[?:?] at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:549) ~[?:?] at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:5058) ~[?:?] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3708) ~[?:?] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2416) ~[?:?] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2299) ~[?:?] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2212) ~[?:?] ... 62 more [2018-03-13T15:43:24,213][DEBUG][c.f.s.a.BackendRegistry ] Cannot authenticate user ivy2 (or add roles) with authdomain internal/4, try next [2018-03-13T15:43:24,213][DEBUG][c.f.s.a.BackendRegistry ] Try to extract auth creds from basic http authenticator [2018-03-13T15:43:24,214][DEBUG][c.f.s.a.BackendRegistry ] ivy2 not cached, return from ldap backend directly [2018-03-13T15:43:24,221][DEBUG][c.f.s.s.PemKeyReader ] Value for searchguard.ssl.transport.truststore_filepath is null [2018-03-13T15:43:24,222][WARN ][c.f.d.a.l.b.LDAPAuthorizationBackend] Unable to connect to ldapserver ldap.rcc.uchicago.edu:389 due to ElasticsearchException[Empty file path for searchguard.ssl.transport.truststore_filepath]. Try next. [2018-03-13T15:43:24,223][DEBUG][c.f.d.a.l.b.LDAPAuthorizationBackend] Unable to connect to ldapserver due to org.elasticsearch.ElasticsearchException: Empty file path for searchguard.ssl.transport.truststore_filepath at com.floragunn.searchguard.support.PemKeyReader.checkPath(PemKeyReader.java:249) ~[search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.support.PemKeyReader.resolve(PemKeyReader.java:319) ~[search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.configureSSL(LDAPAuthorizationBackend.java:264) ~[dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.getConnection0(LDAPAuthorizationBackend.java:155) [dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.access$000(LDAPAuthorizationBackend.java:77) [dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend$1.run(LDAPAuthorizationBackend.java:112) [dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend$1.run(LDAPAuthorizationBackend.java:109) [dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0_121] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.getConnection(LDAPAuthorizationBackend.java:109) [dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthenticationBackend.authenticate(LDAPAuthenticationBackend.java:77) [dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:555) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:549) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:5058) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3708) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2416) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2299) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2212) [guava-23.0.jar:?] at com.google.common.cache.LocalCache.get(LocalCache.java:4147) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:5053) [guava-23.0.jar:?] at com.floragunn.searchguard.auth.BackendRegistry.authcz(BackendRegistry.java:549) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.auth.BackendRegistry.authenticate(BackendRegistry.java:416) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter.checkAndAuthenticateRequest(SearchGuardRestFilter.java:125) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter.access$000(SearchGuardRestFilter.java:48) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:75) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240) [elasticsearch-6.2.1.jar:6.2.1] at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336) [elasticsearch-6.2.1.jar:6.2.1] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174) [elasticsearch-6.2.1.jar:6.2.1] at com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [search-guard-ssl-6.2.1-25.1.jar:6.2.1-25.1] at org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:500) [transport-netty4-client-6.2.1.jar:6.2.1] at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:80) [transport-netty4-client-6.2.1.jar:6.2.1] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68) [transport-netty4-client-6.2.1.jar:6.2.1] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1336) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121] [2018-03-13T15:43:24,225][DEBUG][c.f.s.s.PemKeyReader ] Value for searchguard.ssl.transport.truststore_filepath is null [2018-03-13T15:43:24,226][WARN ][c.f.d.a.l.b.LDAPAuthorizationBackend] Unable to connect to ldapserver ldap.rcc.uchicago.edu:636 due to ElasticsearchException[Empty file path for searchguard.ssl.transport.truststore_filepath]. Try next. [2018-03-13T15:43:24,226][DEBUG][c.f.d.a.l.b.LDAPAuthorizationBackend] Unable to connect to ldapserver due to org.elasticsearch.ElasticsearchException: Empty file path for searchguard.ssl.transport.truststore_filepath at com.floragunn.searchguard.support.PemKeyReader.checkPath(PemKeyReader.java:249) ~[search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.support.PemKeyReader.resolve(PemKeyReader.java:319) ~[search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.configureSSL(LDAPAuthorizationBackend.java:264) ~[dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.getConnection0(LDAPAuthorizationBackend.java:155) [dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.access$000(LDAPAuthorizationBackend.java:77) [dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend$1.run(LDAPAuthorizationBackend.java:112) [dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend$1.run(LDAPAuthorizationBackend.java:109) [dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0_121] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.getConnection(LDAPAuthorizationBackend.java:109) [dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthenticationBackend.authenticate(LDAPAuthenticationBackend.java:77) [dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:555) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:549) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:5058) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3708) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2416) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2299) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2212) [guava-23.0.jar:?] at com.google.common.cache.LocalCache.get(LocalCache.java:4147) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:5053) [guava-23.0.jar:?] at com.floragunn.searchguard.auth.BackendRegistry.authcz(BackendRegistry.java:549) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.auth.BackendRegistry.authenticate(BackendRegistry.java:416) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter.checkAndAuthenticateRequest(SearchGuardRestFilter.java:125) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter.access$000(SearchGuardRestFilter.java:48) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:75) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240) [elasticsearch-6.2.1.jar:6.2.1] at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336) [elasticsearch-6.2.1.jar:6.2.1] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174) [elasticsearch-6.2.1.jar:6.2.1] at com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [search-guard-ssl-6.2.1-25.1.jar:6.2.1-25.1] at org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:500) [transport-netty4-client-6.2.1.jar:6.2.1] at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:80) [transport-netty4-client-6.2.1.jar:6.2.1] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68) [transport-netty4-client-6.2.1.jar:6.2.1] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1336) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121] [2018-03-13T15:43:24,229][DEBUG][c.f.s.s.PemKeyReader ] Value for searchguard.ssl.transport.truststore_filepath is null [2018-03-13T15:43:24,229][WARN ][c.f.d.a.l.b.LDAPAuthorizationBackend] Unable to connect to ldapserver ldap.uchicago.edu:389 due to ElasticsearchException[Empty file path for searchguard.ssl.transport.truststore_filepath]. Try next. [2018-03-13T15:43:24,229][DEBUG][c.f.d.a.l.b.LDAPAuthorizationBackend] Unable to connect to ldapserver due to org.elasticsearch.ElasticsearchException: Empty file path for searchguard.ssl.transport.truststore_filepath at com.floragunn.searchguard.support.PemKeyReader.checkPath(PemKeyReader.java:249) ~[search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.support.PemKeyReader.resolve(PemKeyReader.java:319) ~[search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.configureSSL(LDAPAuthorizationBackend.java:264) ~[dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.getConnection0(LDAPAuthorizationBackend.java:155) [dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.access$000(LDAPAuthorizationBackend.java:77) [dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend$1.run(LDAPAuthorizationBackend.java:112) [dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend$1.run(LDAPAuthorizationBackend.java:109) [dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0_121] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.getConnection(LDAPAuthorizationBackend.java:109) [dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthenticationBackend.authenticate(LDAPAuthenticationBackend.java:77) [dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:555) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:549) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:5058) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3708) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2416) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2299) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2212) [guava-23.0.jar:?] at com.google.common.cache.LocalCache.get(LocalCache.java:4147) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:5053) [guava-23.0.jar:?] at com.floragunn.searchguard.auth.BackendRegistry.authcz(BackendRegistry.java:549) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.auth.BackendRegistry.authenticate(BackendRegistry.java:416) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter.checkAndAuthenticateRequest(SearchGuardRestFilter.java:125) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter.access$000(SearchGuardRestFilter.java:48) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:75) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240) [elasticsearch-6.2.1.jar:6.2.1] at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336) [elasticsearch-6.2.1.jar:6.2.1] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174) [elasticsearch-6.2.1.jar:6.2.1] at com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [search-guard-ssl-6.2.1-25.1.jar:6.2.1-25.1] at org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:500) [transport-netty4-client-6.2.1.jar:6.2.1] at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:80) [transport-netty4-client-6.2.1.jar:6.2.1] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68) [transport-netty4-client-6.2.1.jar:6.2.1] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1336) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121] [2018-03-13T15:43:24,232][DEBUG][c.f.s.s.PemKeyReader ] Value for searchguard.ssl.transport.truststore_filepath is null [2018-03-13T15:43:24,232][WARN ][c.f.d.a.l.b.LDAPAuthorizationBackend] Unable to connect to ldapserver ldap.uchicago.edu:636 due to ElasticsearchException[Empty file path for searchguard.ssl.transport.truststore_filepath]. Try next. [2018-03-13T15:43:24,233][DEBUG][c.f.d.a.l.b.LDAPAuthorizationBackend] Unable to connect to ldapserver due to org.elasticsearch.ElasticsearchException: Empty file path for searchguard.ssl.transport.truststore_filepath at com.floragunn.searchguard.support.PemKeyReader.checkPath(PemKeyReader.java:249) ~[search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.support.PemKeyReader.resolve(PemKeyReader.java:319) ~[search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.configureSSL(LDAPAuthorizationBackend.java:264) ~[dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.getConnection0(LDAPAuthorizationBackend.java:155) [dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.access$000(LDAPAuthorizationBackend.java:77) [dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend$1.run(LDAPAuthorizationBackend.java:112) [dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend$1.run(LDAPAuthorizationBackend.java:109) [dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0_121] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.getConnection(LDAPAuthorizationBackend.java:109) [dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthenticationBackend.authenticate(LDAPAuthenticationBackend.java:77) [dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:555) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:549) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:5058) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3708) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2416) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2299) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2212) [guava-23.0.jar:?] at com.google.common.cache.LocalCache.get(LocalCache.java:4147) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:5053) [guava-23.0.jar:?] at com.floragunn.searchguard.auth.BackendRegistry.authcz(BackendRegistry.java:549) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.auth.BackendRegistry.authenticate(BackendRegistry.java:416) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter.checkAndAuthenticateRequest(SearchGuardRestFilter.java:125) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter.access$000(SearchGuardRestFilter.java:48) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:75) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240) [elasticsearch-6.2.1.jar:6.2.1] at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336) [elasticsearch-6.2.1.jar:6.2.1] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174) [elasticsearch-6.2.1.jar:6.2.1] at com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [search-guard-ssl-6.2.1-25.1.jar:6.2.1-25.1] at org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:500) [transport-netty4-client-6.2.1.jar:6.2.1] at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:80) [transport-netty4-client-6.2.1.jar:6.2.1] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68) [transport-netty4-client-6.2.1.jar:6.2.1] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1336) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121] [2018-03-13T15:43:24,236][DEBUG][c.f.d.a.l.b.LDAPAuthenticationBackend] Unable to authenticate user due to org.ldaptive.LdapException: Unable to connect to any of those ldap servers [ldap.rcc.uchicago.edu:389, ldap.rcc.uchicago.edu:636, ldap.uchicago.edu:389, ldap.uchicago.edu:636] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.getConnection0(LDAPAuthorizationBackend.java:208) ~[dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.access$000(LDAPAuthorizationBackend.java:77) ~[dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend$1.run(LDAPAuthorizationBackend.java:112) ~[dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend$1.run(LDAPAuthorizationBackend.java:109) ~[dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_121] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.getConnection(LDAPAuthorizationBackend.java:109) ~[dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthenticationBackend.authenticate(LDAPAuthenticationBackend.java:77) [dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:555) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:549) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:5058) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3708) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2416) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2299) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2212) [guava-23.0.jar:?] at com.google.common.cache.LocalCache.get(LocalCache.java:4147) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:5053) [guava-23.0.jar:?] at com.floragunn.searchguard.auth.BackendRegistry.authcz(BackendRegistry.java:549) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.auth.BackendRegistry.authenticate(BackendRegistry.java:416) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter.checkAndAuthenticateRequest(SearchGuardRestFilter.java:125) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter.access$000(SearchGuardRestFilter.java:48) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:75) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240) [elasticsearch-6.2.1.jar:6.2.1] at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336) [elasticsearch-6.2.1.jar:6.2.1] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174) [elasticsearch-6.2.1.jar:6.2.1] at com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [search-guard-ssl-6.2.1-25.1.jar:6.2.1-25.1] at org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:500) [transport-netty4-client-6.2.1.jar:6.2.1] at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:80) [transport-netty4-client-6.2.1.jar:6.2.1] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68) [transport-netty4-client-6.2.1.jar:6.2.1] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1336) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121] [2018-03-13T15:43:24,240][DEBUG][c.f.s.a.BackendRegistry ] Can not authenticate ivy2 due to com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[[org.ldaptive.LdapException@1309892896::resultCode=null, matchedDn=null, responseControls=null, referralURLs=null, messageId=-1, message=Unable to connect to any of those ldap servers [ldap.rcc.uchicago.edu:389, ldap.rcc.uchicago.edu:636, ldap.uchicago.edu:389, ldap.uchicago.edu:636], providerException=null]]; nested: LdapException[Unable to connect to any of those ldap servers [ldap.rcc.uchicago.edu:389, ldap.rcc.uchicago.edu:636, ldap.uchicago.edu:389, ldap.uchicago.edu:636]]; com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[[org.ldaptive.LdapException@1309892896::resultCode=null, matchedDn=null, responseControls=null, referralURLs=null, messageId=-1, message=Unable to connect to any of those ldap servers [ldap.rcc.uchicago.edu:389, ldap.rcc.uchicago.edu:636, ldap.uchicago.edu:389, ldap.uchicago.edu:636], providerException=null]]; nested: LdapException[Unable to connect to any of those ldap servers [ldap.rcc.uchicago.edu:389, ldap.rcc.uchicago.edu:636, ldap.uchicago.edu:389, ldap.uchicago.edu:636]]; at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2218) ~[guava-23.0.jar:?] at com.google.common.cache.LocalCache.get(LocalCache.java:4147) ~[guava-23.0.jar:?] at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:5053) ~[guava-23.0.jar:?] at com.floragunn.searchguard.auth.BackendRegistry.authcz(BackendRegistry.java:549) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.auth.BackendRegistry.authenticate(BackendRegistry.java:416) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter.checkAndAuthenticateRequest(SearchGuardRestFilter.java:125) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter.access$000(SearchGuardRestFilter.java:48) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:75) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240) [elasticsearch-6.2.1.jar:6.2.1] at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336) [elasticsearch-6.2.1.jar:6.2.1] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174) [elasticsearch-6.2.1.jar:6.2.1] at com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [search-guard-ssl-6.2.1-25.1.jar:6.2.1-25.1] at org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:500) [transport-netty4-client-6.2.1.jar:6.2.1] at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:80) [transport-netty4-client-6.2.1.jar:6.2.1] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68) [transport-netty4-client-6.2.1.jar:6.2.1] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1336) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121] Caused by: org.elasticsearch.ElasticsearchSecurityException: [org.ldaptive.LdapException@1309892896::resultCode=null, matchedDn=null, responseControls=null, referralURLs=null, messageId=-1, message=Unable to connect to any of those ldap servers [ldap.rcc.uchicago.edu:389, ldap.rcc.uchicago.edu:636, ldap.uchicago.edu:389, ldap.uchicago.edu:636], providerException=null] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthenticationBackend.authenticate(LDAPAuthenticationBackend.java:134) ~[?:?] at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:555) ~[?:?] at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:549) ~[?:?] at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:5058) ~[?:?] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3708) ~[?:?] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2416) ~[?:?] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2299) ~[?:?] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2212) ~[?:?] ... 62 more Caused by: org.ldaptive.LdapException: Unable to connect to any of those ldap servers [ldap.rcc.uchicago.edu:389, ldap.rcc.uchicago.edu:636, ldap.uchicago.edu:389, ldap.uchicago.edu:636] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.getConnection0(LDAPAuthorizationBackend.java:208) ~[?:?] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.access$000(LDAPAuthorizationBackend.java:77) ~[?:?] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend$1.run(LDAPAuthorizationBackend.java:112) ~[?:?] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend$1.run(LDAPAuthorizationBackend.java:109) ~[?:?] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_121] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.getConnection(LDAPAuthorizationBackend.java:109) ~[?:?] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthenticationBackend.authenticate(LDAPAuthenticationBackend.java:77) ~[?:?] at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:555) ~[?:?] at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:549) ~[?:?] at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:5058) ~[?:?] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3708) ~[?:?] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2416) ~[?:?] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2299) ~[?:?] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2212) ~[?:?] ... 62 more [2018-03-13T15:43:24,249][DEBUG][c.f.s.a.BackendRegistry ] Cannot authenticate user ivy2 (or add roles) with authdomain ldap/5, try next [2018-03-13T15:43:24,249][DEBUG][c.f.s.a.BackendRegistry ] User still not authenticated after checking 2 auth domains [2018-03-13T15:43:24,249][DEBUG][c.f.s.a.BackendRegistry ] Rerequest with class com.floragunn.searchguard.http.HTTPBasicAuthenticator [2018-03-13T15:43:24,251][DEBUG][c.f.s.a.BackendRegistry ] Rerequest class com.floragunn.searchguard.http.HTTPBasicAuthenticator failed [2018-03-13T15:43:24,251][WARN ][c.f.s.a.BackendRegistry ] Authentication finally failed for ivy2 [2018-03-13T15:43:30,107][INFO ][c.f.s.a.i.AuditLogImpl ] Closing ESAuditLog [2018-03-13T15:43:30,107][INFO ][o.e.n.Node ] [Qrogsm-] stopping ... [2018-03-13T15:43:32,176][INFO ][o.e.n.Node ] [Qrogsm-] stopped [2018-03-13T15:43:32,176][INFO ][o.e.n.Node ] [Qrogsm-] closing ... [2018-03-13T15:43:32,192][INFO ][o.e.n.Node ] [Qrogsm-] closed