[2018-03-14T14:43:07,422][INFO ][o.e.n.Node ] [] initializing ... [2018-03-14T14:43:07,530][INFO ][o.e.e.NodeEnvironment ] [Qrogsm-] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [182.3gb], net total_space [274gb], types [rootfs] [2018-03-14T14:43:07,531][INFO ][o.e.e.NodeEnvironment ] [Qrogsm-] heap size [989.8mb], compressed ordinary object pointers [true] [2018-03-14T14:43:07,623][INFO ][o.e.n.Node ] node name [Qrogsm-] derived from node ID [Qrogsm-GSiOX4dG3YeCUZg]; set [node.name] to override [2018-03-14T14:43:07,624][INFO ][o.e.n.Node ] version[6.2.1], pid[44136], build[7299dc3/2018-02-07T19:34:26.990113Z], OS[Linux/3.10.0-327.28.3.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_121/25.121-b13] [2018-03-14T14:43:07,624][INFO ][o.e.n.Node ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.dxnPDspC, -XX:+HeapDumpOnOutOfMemoryError, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/home_local/elasticsearch/elasticsearch-6.2.1, -Des.path.conf=/home_local/elasticsearch/elasticsearch-6.2.1/config] [2018-03-14T14:43:08,454][INFO ][c.f.s.SearchGuardPlugin ] ES Config path is /home_local/elasticsearch/elasticsearch-6.2.1/config [2018-03-14T14:43:08,456][DEBUG][c.f.s.SearchGuardPlugin ] Client side initiated TLS renegotiation forcibly disabled. This can prevent DoS attacks. (jdk.tls.rejectClientInitiatedRenegotiation set to true). [2018-03-14T14:43:08,509][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSL [2018-03-14T14:43:08,653][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] JVM supports the following 57 ciphers for https [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV, TLS_DH_anon_WITH_AES_128_GCM_SHA256, TLS_DH_anon_WITH_AES_128_CBC_SHA256, TLS_ECDH_anon_WITH_AES_128_CBC_SHA, TLS_DH_anon_WITH_AES_128_CBC_SHA, TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA, SSL_DH_anon_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_DH_anon_WITH_DES_CBC_SHA, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA, SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA, TLS_RSA_WITH_NULL_SHA256, TLS_ECDHE_ECDSA_WITH_NULL_SHA, TLS_ECDHE_RSA_WITH_NULL_SHA, SSL_RSA_WITH_NULL_SHA, TLS_ECDH_ECDSA_WITH_NULL_SHA, TLS_ECDH_RSA_WITH_NULL_SHA, TLS_ECDH_anon_WITH_NULL_SHA, SSL_RSA_WITH_NULL_MD5, TLS_KRB5_WITH_3DES_EDE_CBC_SHA, TLS_KRB5_WITH_3DES_EDE_CBC_MD5, TLS_KRB5_WITH_DES_CBC_SHA, TLS_KRB5_WITH_DES_CBC_MD5, TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA, TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5] [2018-03-14T14:43:08,655][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] JVM supports the following 57 ciphers for transport [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV, TLS_DH_anon_WITH_AES_128_GCM_SHA256, TLS_DH_anon_WITH_AES_128_CBC_SHA256, TLS_ECDH_anon_WITH_AES_128_CBC_SHA, TLS_DH_anon_WITH_AES_128_CBC_SHA, TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA, SSL_DH_anon_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_DH_anon_WITH_DES_CBC_SHA, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA, SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA, TLS_RSA_WITH_NULL_SHA256, TLS_ECDHE_ECDSA_WITH_NULL_SHA, TLS_ECDHE_RSA_WITH_NULL_SHA, SSL_RSA_WITH_NULL_SHA, TLS_ECDH_ECDSA_WITH_NULL_SHA, TLS_ECDH_RSA_WITH_NULL_SHA, TLS_ECDH_anon_WITH_NULL_SHA, SSL_RSA_WITH_NULL_MD5, TLS_KRB5_WITH_3DES_EDE_CBC_SHA, TLS_KRB5_WITH_3DES_EDE_CBC_MD5, TLS_KRB5_WITH_DES_CBC_SHA, TLS_KRB5_WITH_DES_CBC_MD5, TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA, TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5] [2018-03-14T14:43:08,656][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Config directory is /home_local/elasticsearch/elasticsearch-6.2.1/config/, from there the key- and truststore files are resolved relatively [2018-03-14T14:43:08,656][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.transport.pemcert_filepath is HadoopCertificates/hadoop.rcc.uchicago.edu-chain.crt [2018-03-14T14:43:08,656][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved HadoopCertificates/hadoop.rcc.uchicago.edu-chain.crt to /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates/hadoop.rcc.uchicago.edu-chain.crt against /home_local/elasticsearch/elasticsearch-6.2.1/config [2018-03-14T14:43:08,657][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.transport.pemkey_filepath is HadoopCertificates/hadoop.rcc.uchicago.edu-key.pem [2018-03-14T14:43:08,657][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved HadoopCertificates/hadoop.rcc.uchicago.edu-key.pem to /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates/hadoop.rcc.uchicago.edu-key.pem against /home_local/elasticsearch/elasticsearch-6.2.1/config [2018-03-14T14:43:08,657][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.transport.pemtrustedcas_filepath is HadoopCertificates/incommon.crt [2018-03-14T14:43:08,657][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved HadoopCertificates/incommon.crt to /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates/incommon.crt against /home_local/elasticsearch/elasticsearch-6.2.1/config [2018-03-14T14:43:08,731][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.http.pemtrustedcas_filepath is HadoopCertificates/incommon.crt [2018-03-14T14:43:08,731][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved HadoopCertificates/incommon.crt to /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates/incommon.crt against /home_local/elasticsearch/elasticsearch-6.2.1/config [2018-03-14T14:43:08,731][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.http.pemcert_filepath is HadoopCertificates/hadoop.rcc.uchicago.edu-chain.crt [2018-03-14T14:43:08,731][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved HadoopCertificates/hadoop.rcc.uchicago.edu-chain.crt to /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates/hadoop.rcc.uchicago.edu-chain.crt against /home_local/elasticsearch/elasticsearch-6.2.1/config [2018-03-14T14:43:08,732][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Value for searchguard.ssl.http.pemkey_filepath is HadoopCertificates/hadoop.rcc.uchicago.edu-key.pem [2018-03-14T14:43:08,732][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] Resolved HadoopCertificates/hadoop.rcc.uchicago.edu-key.pem to /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates/hadoop.rcc.uchicago.edu-key.pem against /home_local/elasticsearch/elasticsearch-6.2.1/config [2018-03-14T14:43:08,740][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] AES-256 not supported, max key length for AES is 128 bit. (That is not an issue, it just limits possible encryption strength. To enable AES 256 install 'Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files') [2018-03-14T14:43:08,740][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Client Provider : JDK [2018-03-14T14:43:08,740][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Server Provider : JDK [2018-03-14T14:43:08,740][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS HTTP Provider : JDK [2018-03-14T14:43:08,740][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] sslTransportClientProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256] [2018-03-14T14:43:08,740][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] sslTransportServerProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256] [2018-03-14T14:43:08,740][DEBUG][c.f.s.s.DefaultSearchGuardKeyStore] sslHTTPProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256] [2018-03-14T14:43:08,741][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for transport layer : [TLSv1.2, TLSv1.1] [2018-03-14T14:43:08,741][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for HTTP layer : [TLSv1.2, TLSv1.1] [2018-03-14T14:43:08,928][INFO ][c.f.s.SearchGuardPlugin ] Clustername: ELK1 [2018-03-14T14:43:08,929][DEBUG][c.f.s.SearchGuardPlugin ] This node [Qrogsm-] is a transportClient: false/tribeNode: false/tribeNodeClient: false [2018-03-14T14:43:08,961][DEBUG][c.f.s.s.ReflectionHelper ] Loaded module Module [type=DLSFLS, implementing class=com.floragunn.searchguard.configuration.SearchGuardFlsDlsIndexSearcherWrapper] [2018-03-14T14:43:08,970][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [aggs-matrix-stats] [2018-03-14T14:43:08,970][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [analysis-common] [2018-03-14T14:43:08,970][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [ingest-common] [2018-03-14T14:43:08,971][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [lang-expression] [2018-03-14T14:43:08,971][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [lang-mustache] [2018-03-14T14:43:08,971][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [lang-painless] [2018-03-14T14:43:08,971][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [mapper-extras] [2018-03-14T14:43:08,971][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [parent-join] [2018-03-14T14:43:08,971][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [percolator] [2018-03-14T14:43:08,971][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [rank-eval] [2018-03-14T14:43:08,971][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [reindex] [2018-03-14T14:43:08,971][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [repository-url] [2018-03-14T14:43:08,971][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [transport-netty4] [2018-03-14T14:43:08,971][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [tribe] [2018-03-14T14:43:08,972][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded plugin [search-guard-6] [2018-03-14T14:43:08,991][INFO ][c.f.s.SearchGuardPlugin ] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in elasticsearch.yml [2018-03-14T14:43:10,763][INFO ][c.f.s.a.i.AuditLogImpl ] Configured categories on rest layer to ignore: [AUTHENTICATED, GRANTED_PRIVILEGES] [2018-03-14T14:43:10,764][INFO ][c.f.s.a.i.AuditLogImpl ] Configured categories on transport layer to ignore: [AUTHENTICATED, GRANTED_PRIVILEGES] [2018-03-14T14:43:10,764][INFO ][c.f.s.a.i.AuditLogImpl ] Configured Users to ignore: [kibanaserver] [2018-03-14T14:43:10,764][INFO ][c.f.s.a.i.AuditLogImpl ] Configured Requests to ignore: [] [2018-03-14T14:43:10,764][DEBUG][c.f.s.a.i.AuditLogImpl ] Create new executor with threadPoolSize: 10 and maxQueueLen: 100000 [2018-03-14T14:43:10,765][INFO ][c.f.s.a.i.AuditLogImpl ] Audit Log class: ESAuditLog [2018-03-14T14:43:10,765][DEBUG][c.f.s.a.i.AuditLogImpl ] Security Manager present [2018-03-14T14:43:10,766][DEBUG][c.f.s.a.i.AuditLogImpl ] Shutdown Hook registered [2018-03-14T14:43:10,766][DEBUG][c.f.s.s.ReflectionHelper ] Loaded module Module [type=AUDITLOG, implementing class=com.floragunn.searchguard.auditlog.impl.AuditLogImpl] [2018-03-14T14:43:10,767][DEBUG][c.f.s.SearchGuardPlugin ] Using com.floragunn.searchguard.transport.DefaultInterClusterRequestEvaluator as intercluster request evaluator class [2018-03-14T14:43:10,770][DEBUG][c.f.s.s.ReflectionHelper ] Loaded module Module [type=MULTITENANCY, implementing class=com.floragunn.searchguard.configuration.PrivilegesInterceptorImpl] [2018-03-14T14:43:10,774][DEBUG][c.f.s.c.AdminDNs ] CN=hadoop.rcc.uchicago.edu,OU=IT Services - Self Enrollment,O=University of Chicago,STREET=6045 S. Kenwood,L=Chicago,ST=IL,2.5.4.17=#13053630363337,C=US is registered as an admin dn [2018-03-14T14:43:10,775][DEBUG][c.f.s.c.AdminDNs ] Loaded 1 admin DN's [CN=hadoop.rcc.uchicago.edu,OU=IT Services - Self Enrollment,O=University of Chicago,STREET=6045 S. Kenwood,L=Chicago,ST=IL,2.5.4.17=#13053630363337,C=US] [2018-03-14T14:43:10,776][DEBUG][c.f.s.c.AdminDNs ] Loaded 0 impersonation DN's {} [2018-03-14T14:43:10,776][DEBUG][c.f.s.c.AdminDNs ] Loaded 0 impersonation users for REST {} [2018-03-14T14:43:10,779][DEBUG][c.f.s.c.ConfigurationLoader] Index is: searchguard [2018-03-14T14:43:10,780][DEBUG][c.f.s.c.LegacyConfigurationLoader] Index is: searchguard [2018-03-14T14:43:10,781][DEBUG][c.f.s.c.IndexBaseConfigurationRepository] Subscribe on configuration changes by type config with listener com.floragunn.searchguard.http.XFFResolver@43da0955 [2018-03-14T14:43:11,005][DEBUG][c.f.s.c.IndexBaseConfigurationRepository] Subscribe on configuration changes by type config with listener com.floragunn.searchguard.auth.BackendRegistry@592ca48c [2018-03-14T14:43:11,101][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin com.floragunn.searchguard.SearchGuardPlugin [2018-03-14T14:43:11,118][DEBUG][c.f.s.h.SearchGuardHttpServerTransport] [Qrogsm-] using max_chunk_size[8kb], max_header_size[8kb], max_initial_line_length[4kb], max_content_length[100mb], receive_predictor[64kb->64kb], pipelining[true], pipelining_max_events[10000] [2018-03-14T14:43:11,211][INFO ][o.e.d.DiscoveryModule ] [Qrogsm-] using discovery type [zen] [2018-03-14T14:43:11,678][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] Globally disabled endpoints: {} [2018-03-14T14:43:11,679][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] No disabled endpoints/methods for permitted role sg_all_access found, allowing all [2018-03-14T14:43:11,680][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] Globally disabled endpoints: {} [2018-03-14T14:43:11,680][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] No disabled endpoints/methods for permitted role sg_all_access found, allowing all [2018-03-14T14:43:11,681][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] Globally disabled endpoints: {} [2018-03-14T14:43:11,681][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] No disabled endpoints/methods for permitted role sg_all_access found, allowing all [2018-03-14T14:43:11,682][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] Globally disabled endpoints: {} [2018-03-14T14:43:11,682][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] No disabled endpoints/methods for permitted role sg_all_access found, allowing all [2018-03-14T14:43:11,683][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] Globally disabled endpoints: {} [2018-03-14T14:43:11,683][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] No disabled endpoints/methods for permitted role sg_all_access found, allowing all [2018-03-14T14:43:11,684][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] Globally disabled endpoints: {} [2018-03-14T14:43:11,684][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] No disabled endpoints/methods for permitted role sg_all_access found, allowing all [2018-03-14T14:43:11,685][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] Globally disabled endpoints: {} [2018-03-14T14:43:11,685][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] No disabled endpoints/methods for permitted role sg_all_access found, allowing all [2018-03-14T14:43:11,686][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] Globally disabled endpoints: {} [2018-03-14T14:43:11,686][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] No disabled endpoints/methods for permitted role sg_all_access found, allowing all [2018-03-14T14:43:11,687][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] Globally disabled endpoints: {} [2018-03-14T14:43:11,687][DEBUG][c.f.s.d.r.a.RestApiPrivilegesEvaluator] No disabled endpoints/methods for permitted role sg_all_access found, allowing all [2018-03-14T14:43:11,687][DEBUG][c.f.s.s.ReflectionHelper ] Loaded module Module [type=REST_MANAGEMENT_API, implementing class=com.floragunn.searchguard.dlic.rest.api.SearchGuardRestApiActions] [2018-03-14T14:43:11,688][DEBUG][c.f.s.SearchGuardPlugin ] Added 9 management rest handler(s) [2018-03-14T14:43:11,688][INFO ][c.f.s.SearchGuardPlugin ] 4 Search Guard modules loaded so far: [Module [type=DLSFLS, implementing class=com.floragunn.searchguard.configuration.SearchGuardFlsDlsIndexSearcherWrapper], Module [type=AUDITLOG, implementing class=com.floragunn.searchguard.auditlog.impl.AuditLogImpl], Module [type=MULTITENANCY, implementing class=com.floragunn.searchguard.configuration.PrivilegesInterceptorImpl], Module [type=REST_MANAGEMENT_API, implementing class=com.floragunn.searchguard.dlic.rest.api.SearchGuardRestApiActions]] [2018-03-14T14:43:11,688][INFO ][o.e.n.Node ] initialized [2018-03-14T14:43:11,688][INFO ][o.e.n.Node ] [Qrogsm-] starting ... [2018-03-14T14:43:11,741][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport] [Qrogsm-] using profile[default], worker_count[96], port[9300-9400], bind_host[[127.0.0.1, 172.25.180.171, hadoop.rcc.uchicago.edu]], publish_host[[]], compress[false], connect_timeout[30s], connections_per_node[2/3/6/1/1], receive_predictor[64kb->64kb] [2018-03-14T14:43:11,752][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport] [Qrogsm-] binding server bootstrap to: [127.0.0.1, 172.25.180.171, 128.135.112.122] [2018-03-14T14:43:11,836][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport] [Qrogsm-] Bound profile [default] to address {127.0.0.1:9300} [2018-03-14T14:43:11,838][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport] [Qrogsm-] Bound profile [default] to address {172.25.180.171:9300} [2018-03-14T14:43:11,840][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport] [Qrogsm-] Bound profile [default] to address {128.135.112.122:9300} [2018-03-14T14:43:11,841][INFO ][o.e.t.TransportService ] [Qrogsm-] publish_address {128.135.112.122:9300}, bound_addresses {127.0.0.1:9300}, {172.25.180.171:9300}, {128.135.112.122:9300} [2018-03-14T14:43:11,866][INFO ][o.e.b.BootstrapChecks ] [Qrogsm-] bound or publishing to a non-loopback address, enforcing bootstrap checks [2018-03-14T14:43:11,879][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Check if searchguard index exists ... [2018-03-14T14:43:11,885][DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [Qrogsm-] no known master node, scheduling a retry [2018-03-14T14:43:14,943][INFO ][o.e.c.s.MasterService ] [Qrogsm-] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {Qrogsm-}{Qrogsm-GSiOX4dG3YeCUZg}{Q2d8RoaySVaC1eOA3YBICg}{hadoop.rcc.uchicago.edu}{128.135.112.122:9300} [2018-03-14T14:43:14,951][INFO ][o.e.c.s.ClusterApplierService] [Qrogsm-] new_master {Qrogsm-}{Qrogsm-GSiOX4dG3YeCUZg}{Q2d8RoaySVaC1eOA3YBICg}{hadoop.rcc.uchicago.edu}{128.135.112.122:9300}, reason: apply cluster state (from master [master {Qrogsm-}{Qrogsm-GSiOX4dG3YeCUZg}{Q2d8RoaySVaC1eOA3YBICg}{hadoop.rcc.uchicago.edu}{128.135.112.122:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]]) [2018-03-14T14:43:14,956][DEBUG][c.f.s.c.ClusterInfoHolder] Cluster Info Holder now initialized for 'nodes' [2018-03-14T14:43:14,985][DEBUG][c.f.s.h.SearchGuardHttpServerTransport] [Qrogsm-] Bound http to address {127.0.0.1:9200} [2018-03-14T14:43:14,988][DEBUG][c.f.s.h.SearchGuardHttpServerTransport] [Qrogsm-] Bound http to address {172.25.180.171:9200} [2018-03-14T14:43:14,990][DEBUG][c.f.s.h.SearchGuardHttpServerTransport] [Qrogsm-] Bound http to address {128.135.112.122:9200} [2018-03-14T14:43:14,990][INFO ][c.f.s.h.SearchGuardHttpServerTransport] [Qrogsm-] publish_address {128.135.112.122:9200}, bound_addresses {127.0.0.1:9200}, {172.25.180.171:9200}, {128.135.112.122:9200} [2018-03-14T14:43:14,990][INFO ][o.e.n.Node ] [Qrogsm-] started [2018-03-14T14:43:15,049][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.02.22 [2018-03-14T14:43:15,118][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.02.19 [2018-03-14T14:43:15,126][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.02.16 [2018-03-14T14:43:15,136][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.02.18 [2018-03-14T14:43:15,143][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.02.21 [2018-03-14T14:43:15,150][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.03.13 [2018-03-14T14:43:15,157][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.02.20 [2018-03-14T14:43:15,164][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.02.17 [2018-03-14T14:43:15,172][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index searchguard [2018-03-14T14:43:15,176][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.03.14 [2018-03-14T14:43:15,520][INFO ][o.e.g.GatewayService ] [Qrogsm-] recovered [10] indices into cluster_state [2018-03-14T14:43:15,521][DEBUG][c.f.s.c.IndexBaseConfigurationRepository] Node started, try to initialize it. Wait for at least yellow cluster state.... [2018-03-14T14:43:15,556][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.03.14 [2018-03-14T14:43:15,984][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.03.13 [2018-03-14T14:43:16,346][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.02.22 [2018-03-14T14:43:16,796][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.02.21 [2018-03-14T14:43:17,083][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.02.20 [2018-03-14T14:43:17,371][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.02.19 [2018-03-14T14:43:17,965][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.02.18 [2018-03-14T14:43:18,194][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.02.17 [2018-03-14T14:43:18,741][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index sg6-auditlog-2018.02.16 [2018-03-14T14:43:18,935][DEBUG][c.f.s.SearchGuardPlugin ] FLS/DLS enabled for index searchguard [2018-03-14T14:43:19,147][INFO ][o.e.c.r.a.AllocationService] [Qrogsm-] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[sg6-auditlog-2018.02.16][2], [sg6-auditlog-2018.02.16][4], [searchguard][0]] ...]). [2018-03-14T14:43:19,190][DEBUG][c.f.s.c.IndexBaseConfigurationRepository] Try to load config ... [2018-03-14T14:43:19,190][DEBUG][c.f.s.c.IndexBaseConfigurationRepository] sg index exists and was created with ES 6 (new layout) [2018-03-14T14:43:19,229][DEBUG][c.f.s.c.ConfigurationLoader] Received config for config (of [config, roles, rolesmapping, internalusers, actiongroups]) with current latch value=4 [2018-03-14T14:43:19,232][DEBUG][c.f.s.c.ConfigurationLoader] Received config for roles (of [config, roles, rolesmapping, internalusers, actiongroups]) with current latch value=3 [2018-03-14T14:43:19,232][DEBUG][c.f.s.c.ConfigurationLoader] Received config for rolesmapping (of [config, roles, rolesmapping, internalusers, actiongroups]) with current latch value=2 [2018-03-14T14:43:19,233][DEBUG][c.f.s.c.ConfigurationLoader] Received config for internalusers (of [config, roles, rolesmapping, internalusers, actiongroups]) with current latch value=1 [2018-03-14T14:43:19,234][DEBUG][c.f.s.c.ConfigurationLoader] Received config for actiongroups (of [config, roles, rolesmapping, internalusers, actiongroups]) with current latch value=0 [2018-03-14T14:43:19,240][DEBUG][c.f.s.c.IndexBaseConfigurationRepository] Notify com.floragunn.searchguard.http.XFFResolver@43da0955 listener about change configuration with type config [2018-03-14T14:43:19,240][DEBUG][c.f.s.c.IndexBaseConfigurationRepository] Notify com.floragunn.searchguard.auth.BackendRegistry@592ca48c listener about change configuration with type config [2018-03-14T14:43:19,251][DEBUG][c.f.s.s.ReflectionHelper ] Loaded module Module [type=LDAP_AUTHORIZATION_BACKEND, implementing class=com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend] [2018-03-14T14:43:19,254][DEBUG][c.f.s.s.ReflectionHelper ] Loaded module Module [type=LDAP_AUTHENTICATION_BACKEND, implementing class=com.floragunn.dlic.auth.ldap.backend.LDAPAuthenticationBackend] [2018-03-14T14:43:19,255][DEBUG][c.f.s.s.ReflectionHelper ] Loaded module Module [type=HTTP_BASIC_AUTHENTICATOR, implementing class=com.floragunn.searchguard.http.HTTPBasicAuthenticator] [2018-03-14T14:43:19,256][DEBUG][c.f.s.s.ReflectionHelper ] Loaded module Module [type=INTERNAL_USERS_AUTHENTICATION_BACKEND, implementing class=com.floragunn.searchguard.auth.internal.InternalAuthenticationBackend] [2018-03-14T14:43:19,257][DEBUG][c.f.s.s.ReflectionHelper ] Loaded module Module [type=HTTP_BASIC_AUTHENTICATOR, implementing class=com.floragunn.searchguard.http.HTTPBasicAuthenticator] [2018-03-14T14:43:19,266][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Info: SearchGuardLicense [uid=00000000-0000-0000-0000-000000000000, type=TRIAL, issueDate=2018-02-16, expiryDate=2018-04-19, issuedTo=The world, issuer=floragunn GmbH, startDate=2018-02-16, majorVersion=6, clusterName=*, allowedNodeCount=2147483647, msgs=[], expiresInDays=35, isExpired=false, valid=true, action=, prodUsage=Yes, one cluster with all commercial features and unlimited nodes per cluster., clusterService=org.elasticsearch.cluster.service.ClusterService@29697116, getMsgs()=[], getExpiresInDays()=35, isExpired()=false, isValid()=true, getAction()=, getProdUsage()=Yes, one cluster with all commercial features and unlimited nodes per cluster.] [2018-03-14T14:43:19,266][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Type: TRIAL, valid [2018-03-14T14:43:19,266][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Node 'Qrogsm-' initialized [2018-03-14T14:43:21,045][DEBUG][c.f.s.c.IndexBaseConfigurationRepository] sg index exists and was created with ES 6 (new layout) [2018-03-14T14:43:21,056][DEBUG][c.f.s.c.ConfigurationLoader] Received config for config (of [config, roles, rolesmapping, internalusers, actiongroups]) with current latch value=4 [2018-03-14T14:43:21,058][DEBUG][c.f.s.c.ConfigurationLoader] Received config for roles (of [config, roles, rolesmapping, internalusers, actiongroups]) with current latch value=3 [2018-03-14T14:43:21,058][DEBUG][c.f.s.c.ConfigurationLoader] Received config for rolesmapping (of [config, roles, rolesmapping, internalusers, actiongroups]) with current latch value=2 [2018-03-14T14:43:21,059][DEBUG][c.f.s.c.ConfigurationLoader] Received config for internalusers (of [config, roles, rolesmapping, internalusers, actiongroups]) with current latch value=1 [2018-03-14T14:43:21,060][DEBUG][c.f.s.c.ConfigurationLoader] Received config for actiongroups (of [config, roles, rolesmapping, internalusers, actiongroups]) with current latch value=0 [2018-03-14T14:43:21,060][DEBUG][c.f.s.c.IndexBaseConfigurationRepository] Notify com.floragunn.searchguard.http.XFFResolver@43da0955 listener about change configuration with type config [2018-03-14T14:43:21,060][DEBUG][c.f.s.c.IndexBaseConfigurationRepository] Notify com.floragunn.searchguard.auth.BackendRegistry@592ca48c listener about change configuration with type config [2018-03-14T14:43:21,061][DEBUG][c.f.s.s.ReflectionHelper ] Loaded module Module [type=LDAP_AUTHORIZATION_BACKEND, implementing class=com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend] [2018-03-14T14:43:21,062][DEBUG][c.f.s.s.ReflectionHelper ] Loaded module Module [type=LDAP_AUTHENTICATION_BACKEND, implementing class=com.floragunn.dlic.auth.ldap.backend.LDAPAuthenticationBackend] [2018-03-14T14:43:21,064][DEBUG][c.f.s.s.ReflectionHelper ] Loaded module Module [type=HTTP_BASIC_AUTHENTICATOR, implementing class=com.floragunn.searchguard.http.HTTPBasicAuthenticator] [2018-03-14T14:43:21,065][DEBUG][c.f.s.s.ReflectionHelper ] Loaded module Module [type=INTERNAL_USERS_AUTHENTICATION_BACKEND, implementing class=com.floragunn.searchguard.auth.internal.InternalAuthenticationBackend] [2018-03-14T14:43:21,065][DEBUG][c.f.s.s.ReflectionHelper ] Loaded module Module [type=HTTP_BASIC_AUTHENTICATOR, implementing class=com.floragunn.searchguard.http.HTTPBasicAuthenticator] [2018-03-14T14:43:21,069][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Info: SearchGuardLicense [uid=00000000-0000-0000-0000-000000000000, type=TRIAL, issueDate=2018-02-16, expiryDate=2018-04-19, issuedTo=The world, issuer=floragunn GmbH, startDate=2018-02-16, majorVersion=6, clusterName=*, allowedNodeCount=2147483647, msgs=[], expiresInDays=35, isExpired=false, valid=true, action=, prodUsage=Yes, one cluster with all commercial features and unlimited nodes per cluster., clusterService=org.elasticsearch.cluster.service.ClusterService@29697116, getMsgs()=[], getExpiresInDays()=35, isExpired()=false, isValid()=true, getAction()=, getProdUsage()=Yes, one cluster with all commercial features and unlimited nodes per cluster.] [2018-03-14T14:43:21,069][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Type: TRIAL, valid [2018-03-14T14:43:50,896][DEBUG][c.f.s.a.BackendRegistry ] Try to extract auth creds from basic http authenticator [2018-03-14T14:43:50,897][DEBUG][c.f.s.a.BackendRegistry ] Try to extract auth creds from basic http authenticator [2018-03-14T14:44:06,414][DEBUG][c.f.s.a.BackendRegistry ] Try to extract auth creds from basic http authenticator [2018-03-14T14:44:06,447][DEBUG][c.f.s.a.BackendRegistry ] ivy2 not cached, return from ldap backend directly [2018-03-14T14:44:06,457][DEBUG][c.f.s.s.PemKeyReader ] Value for pemtrustedcas_filepath is HadoopCertificates/incommon.crt [2018-03-14T14:44:06,458][DEBUG][c.f.s.s.PemKeyReader ] Resolved HadoopCertificates/incommon.crt to /home_local/elasticsearch/elasticsearch-6.2.1/config/HadoopCertificates/incommon.crt against /home_local/elasticsearch/elasticsearch-6.2.1/config [2018-03-14T14:44:06,469][DEBUG][c.f.s.s.PemKeyReader ] Value for pemcert_filepath is null [2018-03-14T14:44:06,470][DEBUG][c.f.s.s.PemKeyReader ] Value for pemkey_filepath is null [2018-03-14T14:44:06,474][DEBUG][c.f.d.a.l.b.LDAPAuthorizationBackend] Use PEM to secure communication with LDAP server (client auth is false) [2018-03-14T14:44:06,476][DEBUG][c.f.d.a.l.b.LDAPAuthorizationBackend] enabled ssl/tls protocols for ldaps [TLSv1.2, TLSv1.1] [2018-03-14T14:44:06,476][DEBUG][c.f.d.a.l.b.LDAPAuthorizationBackend] Connect timeout: 5000/ResponseTimeout: -1 [2018-03-14T14:44:06,510][DEBUG][c.f.d.a.l.b.LDAPAuthorizationBackend] bindDn ou=People,dc=rcc,dc=uchicago,dc=edu, password [2018-03-14T14:44:06,510][ERROR][c.f.d.a.l.b.LDAPAuthorizationBackend] No password given for bind_dn ou=People,dc=rcc,dc=uchicago,dc=edu. Will try to authenticate anonymously to ldap [2018-03-14T14:44:06,510][DEBUG][c.f.d.a.l.b.LDAPAuthorizationBackend] Will perform simple bind with bind dn [2018-03-14T14:44:06,634][DEBUG][c.f.d.a.l.b.LDAPAuthenticationBackend] No user ivy2 found [2018-03-14T14:44:06,634][DEBUG][c.f.d.a.l.b.LDAPAuthenticationBackend] Unable to authenticate user due to org.elasticsearch.ElasticsearchSecurityException: No user ivy2 found at com.floragunn.dlic.auth.ldap.backend.LDAPAuthenticationBackend.authenticate(LDAPAuthenticationBackend.java:88) [dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:555) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:549) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:5058) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3708) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2416) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2299) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2212) [guava-23.0.jar:?] at com.google.common.cache.LocalCache.get(LocalCache.java:4147) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:5053) [guava-23.0.jar:?] at com.floragunn.searchguard.auth.BackendRegistry.authcz(BackendRegistry.java:549) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.auth.BackendRegistry.authenticate(BackendRegistry.java:416) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter.checkAndAuthenticateRequest(SearchGuardRestFilter.java:125) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter.access$000(SearchGuardRestFilter.java:48) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:75) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240) [elasticsearch-6.2.1.jar:6.2.1] at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336) [elasticsearch-6.2.1.jar:6.2.1] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174) [elasticsearch-6.2.1.jar:6.2.1] at com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [search-guard-ssl-6.2.1-25.1.jar:6.2.1-25.1] at org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:500) [transport-netty4-client-6.2.1.jar:6.2.1] at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:80) [transport-netty4-client-6.2.1.jar:6.2.1] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68) [transport-netty4-client-6.2.1.jar:6.2.1] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1336) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121] [2018-03-14T14:44:06,653][DEBUG][c.f.s.a.BackendRegistry ] Can not authenticate ivy2 due to com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[ElasticsearchSecurityException[No user ivy2 found]]; nested: ElasticsearchSecurityException[No user ivy2 found]; com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[ElasticsearchSecurityException[No user ivy2 found]]; nested: ElasticsearchSecurityException[No user ivy2 found]; at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2218) ~[guava-23.0.jar:?] at com.google.common.cache.LocalCache.get(LocalCache.java:4147) ~[guava-23.0.jar:?] at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:5053) ~[guava-23.0.jar:?] at com.floragunn.searchguard.auth.BackendRegistry.authcz(BackendRegistry.java:549) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.auth.BackendRegistry.authenticate(BackendRegistry.java:416) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter.checkAndAuthenticateRequest(SearchGuardRestFilter.java:125) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter.access$000(SearchGuardRestFilter.java:48) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:75) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240) [elasticsearch-6.2.1.jar:6.2.1] at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336) [elasticsearch-6.2.1.jar:6.2.1] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174) [elasticsearch-6.2.1.jar:6.2.1] at com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [search-guard-ssl-6.2.1-25.1.jar:6.2.1-25.1] at org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:500) [transport-netty4-client-6.2.1.jar:6.2.1] at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:80) [transport-netty4-client-6.2.1.jar:6.2.1] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68) [transport-netty4-client-6.2.1.jar:6.2.1] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1336) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121] Caused by: org.elasticsearch.ElasticsearchSecurityException: ElasticsearchSecurityException[No user ivy2 found] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthenticationBackend.authenticate(LDAPAuthenticationBackend.java:134) ~[?:?] at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:555) ~[?:?] at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:549) ~[?:?] at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:5058) ~[?:?] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3708) ~[?:?] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2416) ~[?:?] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2299) ~[?:?] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2212) ~[?:?] ... 62 more Caused by: org.elasticsearch.ElasticsearchSecurityException: No user ivy2 found at com.floragunn.dlic.auth.ldap.backend.LDAPAuthenticationBackend.authenticate(LDAPAuthenticationBackend.java:88) ~[?:?] at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:555) ~[?:?] at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:549) ~[?:?] at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:5058) ~[?:?] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3708) ~[?:?] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2416) ~[?:?] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2299) ~[?:?] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2212) ~[?:?] ... 62 more [2018-03-14T14:44:06,664][DEBUG][c.f.s.a.BackendRegistry ] Cannot authenticate user ivy2 (or add roles) with authdomain ldap/1, try next [2018-03-14T14:44:06,664][DEBUG][c.f.s.a.BackendRegistry ] Try to extract auth creds from basic http authenticator [2018-03-14T14:44:06,665][DEBUG][c.f.s.a.BackendRegistry ] ivy2 not cached, return from internal backend directly [2018-03-14T14:44:06,665][DEBUG][c.f.s.a.BackendRegistry ] Can not authenticate ivy2 due to com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[ivy2 not found] com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[ivy2 not found] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2218) ~[guava-23.0.jar:?] at com.google.common.cache.LocalCache.get(LocalCache.java:4147) ~[guava-23.0.jar:?] at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:5053) ~[guava-23.0.jar:?] at com.floragunn.searchguard.auth.BackendRegistry.authcz(BackendRegistry.java:549) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.auth.BackendRegistry.authenticate(BackendRegistry.java:416) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter.checkAndAuthenticateRequest(SearchGuardRestFilter.java:125) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter.access$000(SearchGuardRestFilter.java:48) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:75) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240) [elasticsearch-6.2.1.jar:6.2.1] at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336) [elasticsearch-6.2.1.jar:6.2.1] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174) [elasticsearch-6.2.1.jar:6.2.1] at com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [search-guard-ssl-6.2.1-25.1.jar:6.2.1-25.1] at org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:500) [transport-netty4-client-6.2.1.jar:6.2.1] at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:80) [transport-netty4-client-6.2.1.jar:6.2.1] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68) [transport-netty4-client-6.2.1.jar:6.2.1] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1336) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121] Caused by: org.elasticsearch.ElasticsearchSecurityException: ivy2 not found at com.floragunn.searchguard.auth.internal.InternalAuthenticationBackend.authenticate(InternalAuthenticationBackend.java:102) ~[?:?] at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:555) ~[?:?] at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:549) ~[?:?] at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:5058) ~[?:?] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3708) ~[?:?] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2416) ~[?:?] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2299) ~[?:?] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2212) ~[?:?] ... 62 more [2018-03-14T14:44:06,672][DEBUG][c.f.s.a.BackendRegistry ] Cannot authenticate user ivy2 (or add roles) with authdomain internal/4, try next [2018-03-14T14:44:06,672][DEBUG][c.f.s.a.BackendRegistry ] User still not authenticated after checking 2 auth domains [2018-03-14T14:44:06,672][DEBUG][c.f.s.a.BackendRegistry ] Rerequest with class com.floragunn.searchguard.http.HTTPBasicAuthenticator [2018-03-14T14:44:06,674][DEBUG][c.f.s.a.BackendRegistry ] Rerequest class com.floragunn.searchguard.http.HTTPBasicAuthenticator failed [2018-03-14T14:44:06,674][WARN ][c.f.s.a.BackendRegistry ] Authentication finally failed for ivy2 [2018-03-14T14:44:12,860][INFO ][o.e.n.Node ] [Qrogsm-] stopping ... [2018-03-14T14:44:12,861][INFO ][c.f.s.a.i.AuditLogImpl ] Closing ESAuditLog [2018-03-14T14:44:14,768][INFO ][o.e.n.Node ] [Qrogsm-] stopped [2018-03-14T14:44:14,768][INFO ][o.e.n.Node ] [Qrogsm-] closing ... [2018-03-14T14:44:14,788][INFO ][o.e.n.Node ] [Qrogsm-] closed