Fixed curl, still getting Search Guard not initialized (SG11)

  • [Elasticsearch logfiles on debug level]
[2019-12-24T20:52:49,167][WARN ][o.e.h.AbstractHttpServerTransport] [kibana-centos-7] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:41534}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:473) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:600) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:554) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) [netty-common-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.43.Final.jar:4.1.43.Final]
        at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:324) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:267) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:262) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkClientCerts(CertificateMessage.java:687) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:398) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:366) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1247) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:691) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1192) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1502) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1516) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1400) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:503) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
        ... 16 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:384) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:289) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:138) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkClientCerts(CertificateMessage.java:669) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:398) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:366) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1247) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:691) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1192) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1502) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1516) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1400) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:503) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
        ... 16 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[?:?]
        at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[?:?]
        at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297) ~[?:?]
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:379) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:289) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:138) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkClientCerts(CertificateMessage.java:669) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:398) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:366) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1247) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:691) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1192) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1502) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1516) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1400) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:503) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
        ... 16 more
  • Your Search Guard configuration files
[root@kibana-centos-7 ~]# cat /usr/share/elasticsearch/plugins/search-guard-7/sgconfig/sg_config.yml
# Generated by chef
#
---
_sg_meta:
  type: config
  config_version: '2'
sg_config:
  dynamic:
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: true
        internalProxies: ".*"
        remoteIpHeader: x-forwarded-for
    authc:
      proxy_auth_domain:
        http_enabled: true
        order: 0
        http_authenticator:
          challenge: false
          config:
            user_header: x-proxy-user
            roles_header: x-proxy-roles
        authentication_backend:
          type: noop
      basic_internal_auth_domain:
        order: 1
        http_authenticator:
          challenge: true
  • Your elasticsearch.yml configuration file
[root@kibana-centos-7 ~]# cat /etc/elasticsearch/elasticsearch.yml
# Generated by chef
#
---
cluster:
  name: es_test
  routing:
    allocation:
      allow_rebalance: always
      disk:
        watermark:
          low: 90%
          high: 95%
  initial_master_nodes: 127.0.0.1
path:
  data: "/data"
  logs: "/var/log/elasticsearch/"
network:
  host: 127.0.0.1
http:
  port: '9200'
xpack:
  ml:
    enabled: false
  monitoring:
    enabled: true
    collection:
      enabled: true
  graph:
    enabled: true
  watcher:
    enabled: true
  security:
    enabled: false
  http:
    ssl:
      verification_mode: none
discovery:
  seed_hosts: localhost
  zen:
    minimum_master_nodes: 1
node:
  name: kibana-centos-7
  master: true
  data: true
  ingest: true
searchguard:
  restapi:
    roles_enabled:
    - SGS_ALL_ACCESS
  ssl:
    transport:
      pemcert_filepath: x-pack/elasticsearch.crt
      pemkey_filepath: x-pack/elasticsearch.key
      pemtrustedcas_filepath: x-pack/ca.crt
      enforce_hostname_verification: false
      resolve_hostname: false
    http:
      enabled: true
      pemcert_filepath: x-pack/elasticsearch.crt
      pemkey_filepath: x-pack/elasticsearch.key
      pemtrustedcas_filepath: x-pack/ca.crt
  enterprise_modules_enabled: false
  enable_snapshot_restore_privilege: true
  authcz:
    admin_dn:
    - CN=admin.elastictest.com,OU=Ops,O=elastictest Com\, Inc.,DC=elastictest,DC=com
gateway:
  expected_master_nodes: 1
  • Your kibana.yml configuration file
[root@kibana-centos-7 ~]# cat /etc/kibana/kibana.yml
# Generated by chef
#
---
server:
  host: 0.0.0.0
  ssl:
    enabled: true
    key: "/etc/kibana/kibana.key"
    certificate: "/etc/kibana/kibana.crt"
elasticsearch:
  username: kibanaserver
  password: kibanaserver
  ssl:
    certificateAuthorities: "/etc/kibana/ca.crt"
    key: "/etc/kibana/kibana.key"
    certificate: "/etc/kibana/kibana.crt"
    verificationMode: none
  hosts: https://localhost:9200
  requestHeadersWhitelist:
  - authorization
  - x-forwarded-for
  - x-forwarded-by
  - x-proxy-user
  - x-proxy-roles
xpack:
  security:
    enabled: false
  monitoring:
    enabled: true
  graph:
    enabled: true
  spaces:
    enabled: false
  infra:
    enabled: false
  reporting:
    enabled: true
    encryptionKey: bbySecureCloud
    csv:
      maxSizeBytes: '20971520'
    kibanaServer:
      port: '5601'
      protocol: https
logging:
  dest: "/var/log/kibana/kibana.log"
  verbose: false
pid:
  file: "/var/run/kibana/kibana.pid"
searchguard:
  allow_client_certificates: true
  auth:
    type: basicauth
  cookie:
    secure: true
    password: somerandolongpasswordwithmorethanthirtytwocharactersinit

What I observe below is that there is a successful sgadmin executed, however curl only works when I pass it the certs/keys. There is a ‘troubleshooting tls’ page in SG that talks about Fixing Curl and that seems to suggest that NSS compiled curl (which ships with CentOS7) needs to be replaced with an updated OpenSSL compiled curl.

  • Initialize SG:
[root@kibana-centos-7 ~]# /usr/share/elasticsearch/plugins/search-guard-7/tools/sgadmin.sh -h '127.0.0.1' -cd '/usr/share/elasticsearch/plugins/search-guard-7/sgconfig' -icl -nhnv -key /etc/elasticsearch/kirk.key -cert /etc/elasticsearch/kirk.crt -cacert /etc/elasticsearch/x-pack/ca.crt
WARNING: JAVA_HOME not set, will use /bin/java
Search Guard Admin v7
Will connect to 127.0.0.1:9300 ... done
Connected as CN=admin.elastictest.com,OU=Ops,O=elastictest Com\, Inc.,DC=elastictest,DC=com
Elasticsearch Version: 7.5.0
Search Guard Version: 7.5.0-37.1.0
Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
Clustername: es_test
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
searchguard index already exists, so we do not need to create one.
Populate config from /usr/share/elasticsearch/plugins/search-guard-7/sgconfig/
/usr/share/elasticsearch/plugins/search-guard-7/sgconfig/sg_action_groups.yml OK
/usr/share/elasticsearch/plugins/search-guard-7/sgconfig/sg_internal_users.yml OK
/usr/share/elasticsearch/plugins/search-guard-7/sgconfig/sg_roles.yml OK
/usr/share/elasticsearch/plugins/search-guard-7/sgconfig/sg_roles_mapping.yml OK
/usr/share/elasticsearch/plugins/search-guard-7/sgconfig/sg_config.yml OK
/usr/share/elasticsearch/plugins/search-guard-7/sgconfig/sg_tenants.yml OK
Will update '_doc/config' with /usr/share/elasticsearch/plugins/search-guard-7/sgconfig/sg_config.yml
   SUCC: Configuration for 'config' created or updated
Will update '_doc/roles' with /usr/share/elasticsearch/plugins/search-guard-7/sgconfig/sg_roles.yml
   SUCC: Configuration for 'roles' created or updated
Will update '_doc/rolesmapping' with /usr/share/elasticsearch/plugins/search-guard-7/sgconfig/sg_roles_mapping.yml
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '_doc/internalusers' with /usr/share/elasticsearch/plugins/search-guard-7/sgconfig/sg_internal_users.yml
   SUCC: Configuration for 'internalusers' created or updated
Will update '_doc/actiongroups' with /usr/share/elasticsearch/plugins/search-guard-7/sgconfig/sg_action_groups.yml
   SUCC: Configuration for 'actiongroups' created or updated
Will update '_doc/tenants' with /usr/share/elasticsearch/plugins/search-guard-7/sgconfig/sg_tenants.yml
   SUCC: Configuration for 'tenants' created or updated
Done with success
  • Default curl:
[root@kibana-centos-7 ~]# curl -V
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.36 zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets
  • Default curl does not work:
[root@kibana-centos-7 ~]# curl -k https://admin:<password>@localhost:9200/
Search Guard not initialized (SG11). See https://docs.search-guard.com/latest/sgadmin
  • Default curl works when I pass the --cert and --key values:
[root@kibana-centos-7 ~]# curl -k --cert /etc/elasticsearch/kirk.crt --key /etc/elasticsearch/kirk.key https://localhost:9200
{
  "name" : "kibana-centos-7",
  "cluster_name" : "es_test",
  "cluster_uuid" : "aiP-f8lxQsiZBtKrkbW4Ow",
  "version" : {
    "number" : "7.5.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "e9ccaed468e2fac2275a3761849cbee64b39519f",
    "build_date" : "2019-11-26T01:06:52.518245Z",
    "build_snapshot" : false,
    "lucene_version" : "8.3.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
  • Output of yum info libcurl shows that the CentOS7 package only has release updates available, and not the versions as suggested in the wiki
[root@kibana-centos-7 ~]# yum info libcurl
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * centos-sclo-rh: mirrors.usinternet.com
 * centos-sclo-sclo: mirrors.usinternet.com
Installed Packages
Name        : libcurl
Arch        : x86_64
Version     : 7.29.0
Release     : 51.el7_6.3
Size        : 425 k
Repo        : installed
From repo   : centos-7-update-rpms
Summary     : A library for getting files from web servers
URL         : http://curl.haxx.se/
License     : MIT
Description : libcurl is a free and easy-to-use client-side URL transfer library, supporting
            : FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, LDAP, LDAPS, FILE, IMAP,
            : SMTP, POP3 and RTSP. libcurl supports SSL certificates, HTTP POST, HTTP PUT,
            : FTP uploading, HTTP form based upload, proxies, cookies, user+password
            : authentication (Basic, Digest, NTLM, Negotiate, Kerberos4), file transfer
            : resume, http proxy tunneling and more.

Available Packages
Name        : libcurl
Arch        : i686
Version     : 7.29.0
Release     : 54.el7
Size        : 225 k
Repo        : centos-7-server-rpms
Summary     : A library for getting files from web servers
URL         : http://curl.haxx.se/
License     : MIT
Description : libcurl is a free and easy-to-use client-side URL transfer library, supporting
            : FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, LDAP, LDAPS, FILE, IMAP,
            : SMTP, POP3 and RTSP. libcurl supports SSL certificates, HTTP POST, HTTP PUT,
#!/bin/sh
            : FTP uploading, HTTP form based upload, proxies, cookies, user+password
            : authentication (Basic, Digest, NTLM, Negotiate, Kerberos4), file transfer
            : resume, http proxy tunneling and more.

Name        : libcurl
Arch        : x86_64
Version     : 7.29.0
Release     : 54.el7
Size        : 222 k
Repo        : centos-7-server-rpms
Summary     : A library for getting files from web servers
URL         : http://curl.haxx.se/
License     : MIT
Description : libcurl is a free and easy-to-use client-side URL transfer library, supporting
            : FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, LDAP, LDAPS, FILE, IMAP,
            : SMTP, POP3 and RTSP. libcurl supports SSL certificates, HTTP POST, HTTP PUT,
            : FTP uploading, HTTP form based upload, proxies, cookies, user+password
            : authentication (Basic, Digest, NTLM, Negotiate, Kerberos4), file transfer
            : resume, http proxy tunneling and more.
  • Compile and install libcurl to satisfy the Openssl and 7.50 requirements:
   curl -O https://curl.haxx.se/download/curl-7.58.0.tar.bz2
   tar xvf curl-7.58.0.tar.bz2
   cd curl-7.58.0
   ./configure --with-ssl
   make
   make install
[root@kibana-centos-7 curl-7.58.0]# /usr/local/bin/curl -V
curl 7.58.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.0.2k zlib/1.2.7
Release-Date: 2018-01-24
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile NTLM NTLM_WB SSL libz UnixSockets HTTPS-proxy
  • Test curl against SG:
[root@kibana-centos-7 curl-7.58.0]# /usr/local/bin/curl -k https://localhost:9200
Search Guard not initialized (SG11). See https://docs.search-guard.com/latest/sgadmin
  • Test curl against SG with --cert and --key:
[root@kibana-centos-7 curl-7.58.0]# curl -k --cert /etc/elasticsearch/kirk.crt --key /etc/elasticsearch/kirk.key https://localhost:9200/
{
  "name" : "kibana-centos-7",
  "cluster_name" : "es_test",
  "cluster_uuid" : "oT8hBHRwQF271HphJvpkzQ",
  "version" : {
    "number" : "7.5.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "e9ccaed468e2fac2275a3761849cbee64b39519f",
    "build_date" : "2019-11-26T01:06:52.518245Z",
    "build_snapshot" : false,
    "lucene_version" : "8.3.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
  • Output of kibana.log:
{"type":"log","@timestamp":"2019-12-24T21:07:30Z","tags":["warning","elasticsearch","admin"],"pid":14166,"message":"Unable to revive connection: https://localhost:9200/"}
{"type":"log","@timestamp":"2019-12-24T21:07:30Z","tags":["warning","elasticsearch","admin"],"pid":14166,"message":"No living connections"}
{"type":"log","@timestamp":"2019-12-24T21:07:32Z","tags":["warning","elasticsearch","admin"],"pid":14166,"message":"Unable to revive connection: https://localhost:9200/"}
{"type":"log","@timestamp":"2019-12-24T21:07:32Z","tags":["warning","elasticsearch","admin"],"pid":14166,"message":"No living connections"}

Question is “how do I get Search Guard to initialize successfully”? The sgadmin.sh script appears to exit successfully, and it also appears that the updated curl package has the correct Openssl after compiling. Why do the curl statements not agree with the sgadmin output?

That is something which should not happen. Can you provide full logs from the ES node (on debug level like explained here Search Guard logging | Security for Elasticsearch | Search Guard). With “full logs” I mean the complete logfile from the point where the node is started until you tried to access via curl command without admin cert.

Please also have a look here smoketest.sh · master · search-guard / legacy / Search Guard · GitLab. This is how we test that initialization is working.

Thanks for the response. Providing requested information and logs.

Initial log after build and startup:

[root@test-single-centos-7 ~]# less /var/log/elasticsearch/elk_test_cluster.log

[2020-01-06T14:52:49,316][INFO ][o.e.e.NodeEnvironment    ] [test-single-centos-7] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [33.3gb], net total_space [36.9gb], types [rootfs]
[2020-01-06T14:52:49,318][INFO ][o.e.e.NodeEnvironment    ] [test-single-centos-7] heap size [903.3mb], compressed ordinary object pointers [true]
[2020-01-06T14:52:49,321][INFO ][o.e.n.Node               ] [test-single-centos-7] node name [test-single-centos-7], node ID [gZDCT6YvSnabCzJ-st3dQg], cluster name [elk_test_cluster]
[2020-01-06T14:52:49,321][INFO ][o.e.n.Node               ] [test-single-centos-7] version[7.5.0], pid[29539], build[default/rpm/e9ccaed468e2fac2275a3761849cbee64b39519f/2019-11-26T01:06:52.518245Z], OS[Linux/3.10.0-957.27.2.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13.0.1/13.0.1+9]
[2020-01-06T14:52:49,321][INFO ][o.e.n.Node               ] [test-single-centos-7] JVM home [/usr/share/elasticsearch/jdk]
[2020-01-06T14:52:49,321][INFO ][o.e.n.Node               ] [test-single-centos-7] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms919m, -Xmx919m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -XX:MaxDirectMemorySize=482344960, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2020-01-06T14:52:51,794][INFO ][c.f.s.SearchGuardPlugin  ] [test-single-centos-7] ES Config path is /etc/elasticsearch
[2020-01-06T14:52:51,845][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] [test-single-centos-7] JVM supports TLSv1.3
[2020-01-06T14:52:51,845][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] [test-single-centos-7] Config directory is /etc/elasticsearch/, from there the key- and truststore files are resolved relatively
[2020-01-06T14:52:52,562][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] [test-single-centos-7] TLS Transport Client Provider : JDK
[2020-01-06T14:52:52,562][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] [test-single-centos-7] TLS Transport Server Provider : JDK
[2020-01-06T14:52:52,563][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] [test-single-centos-7] TLS HTTP Provider             : JDK
[2020-01-06T14:52:52,563][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] [test-single-centos-7] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2, TLSv1.1]
[2020-01-06T14:52:52,563][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] [test-single-centos-7] Enabled TLS protocols for HTTP layer      : [TLSv1.3, TLSv1.2, TLSv1.1]
[2020-01-06T14:52:52,948][INFO ][c.f.s.SearchGuardPlugin  ] [test-single-centos-7] Clustername: elk_test_cluster
[2020-01-06T14:52:52,958][WARN ][c.f.s.SearchGuardPlugin  ] [test-single-centos-7] File /etc/elasticsearch/elasticsearch.yml has insecure file permissions (should be 0600)
[2020-01-06T14:52:52,958][WARN ][c.f.s.SearchGuardPlugin  ] [test-single-centos-7] File /etc/elasticsearch/.elasticsearch.keystore.initial_md5sum has insecure file permissions (should be 0600)
[2020-01-06T14:52:52,996][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [aggs-matrix-stats]
[2020-01-06T14:52:52,996][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [analysis-common]
[2020-01-06T14:52:52,996][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [flattened]
[2020-01-06T14:52:52,996][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [frozen-indices]
[2020-01-06T14:52:52,996][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [ingest-common]
[2020-01-06T14:52:52,996][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [ingest-geoip]
[2020-01-06T14:52:52,996][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [ingest-user-agent]
[2020-01-06T14:52:52,997][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [lang-expression]
[2020-01-06T14:52:52,997][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [lang-mustache]
[2020-01-06T14:52:52,997][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [lang-painless]
[2020-01-06T14:52:52,997][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [mapper-extras]
[2020-01-06T14:52:52,997][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [parent-join]
[2020-01-06T14:52:52,997][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [percolator]
[2020-01-06T14:52:52,997][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [rank-eval]
[2020-01-06T14:52:52,997][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [reindex]
[2020-01-06T14:52:52,997][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [repository-url]
[2020-01-06T14:52:52,997][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [search-business-rules]
[2020-01-06T14:52:52,997][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [spatial]
[2020-01-06T14:52:52,997][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [systemd]
...skipping...
[2020-01-06T14:53:00,173][INFO ][c.f.s.SearchGuardPlugin  ] [test-single-centos-7] Node started
[2020-01-06T14:53:00,174][INFO ][c.f.s.c.ConfigurationRepository] [test-single-centos-7] Check if searchguard index exists ...
[2020-01-06T14:53:00,174][INFO ][c.f.s.c.ConfigurationRepository] [test-single-centos-7] searchguard index does not exist yet, so no need to load config on node startup. Use sgadmin to initialize cluster
[2020-01-06T14:53:00,191][INFO ][c.f.s.SearchGuardPlugin  ] [test-single-centos-7] 0 Search Guard modules loaded so far: []
[2020-01-06T14:53:00,193][INFO ][c.f.s.c.ConfigurationRepository] [test-single-centos-7] Background init thread started. Install default config?: false
[2020-01-06T14:53:00,259][INFO ][o.e.g.GatewayService     ] [test-single-centos-7] recovered [0] indices into cluster_state
[2020-01-06T14:53:00,375][INFO ][o.e.c.m.MetaDataIndexTemplateService] [test-single-centos-7] adding template [.triggered_watches] for index patterns [.triggered_watches*]
[2020-01-06T14:53:00,533][INFO ][o.e.c.m.MetaDataIndexTemplateService] [test-single-centos-7] adding template [.watch-history-10] for index patterns [.watcher-history-10*]
[2020-01-06T14:53:00,596][INFO ][o.e.c.m.MetaDataIndexTemplateService] [test-single-centos-7] adding template [.watches] for index patterns [.watches*]
[2020-01-06T14:53:00,665][INFO ][o.e.c.m.MetaDataIndexTemplateService] [test-single-centos-7] adding template [.slm-history] for index patterns [.slm-history-1*]
[2020-01-06T14:53:00,723][INFO ][o.e.c.m.MetaDataIndexTemplateService] [test-single-centos-7] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-7-*]
[2020-01-06T14:53:00,788][INFO ][o.e.c.m.MetaDataIndexTemplateService] [test-single-centos-7] adding template [.monitoring-es] for index patterns [.monitoring-es-7-*]
[2020-01-06T14:53:00,833][INFO ][o.e.c.m.MetaDataIndexTemplateService] [test-single-centos-7] adding template [.monitoring-beats] for index patterns [.monitoring-beats-7-*]
[2020-01-06T14:53:00,895][INFO ][o.e.c.m.MetaDataIndexTemplateService] [test-single-centos-7] adding template [.monitoring-alerts-7] for index patterns [.monitoring-alerts-7]
[2020-01-06T14:53:00,968][INFO ][o.e.c.m.MetaDataIndexTemplateService] [test-single-centos-7] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-7-*]
[2020-01-06T14:53:01,016][INFO ][o.e.a.s.m.TransportMasterNodeAction] [test-single-centos-7] adding index lifecycle policy [watch-history-ilm-policy]
[2020-01-06T14:53:01,061][INFO ][o.e.a.s.m.TransportMasterNodeAction] [test-single-centos-7] adding index lifecycle policy [slm-history-ilm-policy]
[2020-01-06T14:53:01,643][INFO ][o.e.l.LicenseService     ] [test-single-centos-7] license [1d137213-7e00-4cc9-a78d-0930cdfed5ac] mode [basic] - valid
[2020-01-06T14:53:09,868][INFO ][o.e.c.m.MetaDataCreateIndexService] [test-single-centos-7] [.monitoring-es-7-2020.01.06] creating index, cause [auto(bulk api)], templates [.monitoring-es], shards [1]/[0], mappings [_doc]
[2020-01-06T14:53:10,220][INFO ][o.e.c.r.a.AllocationService] [test-single-centos-7] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.monitoring-es-7-2020.01.06][0]]]).
[2020-01-06T14:57:23,236][INFO ][o.e.c.m.MetaDataCreateIndexService] [test-single-centos-7] [searchguard] creating index, cause [api], templates [], shards [1]/[1], mappings []
[2020-01-06T14:57:23,247][INFO ][o.e.c.r.a.AllocationService] [test-single-centos-7] updating number_of_replicas to [0] for indices [searchguard]
[2020-01-06T14:57:23,406][INFO ][o.e.c.r.a.AllocationService] [test-single-centos-7] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[searchguard][0]]]).
[2020-01-06T14:57:23,404][WARN ][c.f.s.c.ConfigurationLoaderSG7] [test-single-centos-7] No data for internalusers while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS]  (index=searchguard and type=null)
[2020-01-06T14:57:23,407][WARN ][c.f.s.c.ConfigurationLoaderSG7] [test-single-centos-7] No data for actiongroups while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS]  (index=searchguard and type=null)
[2020-01-06T14:57:23,407][WARN ][c.f.s.c.ConfigurationLoaderSG7] [test-single-centos-7] No data for config while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS]  (index=searchguard and type=null)
[2020-01-06T14:57:23,407][WARN ][c.f.s.c.ConfigurationLoaderSG7] [test-single-centos-7] No data for roles while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS]  (index=searchguard and type=null)
[2020-01-06T14:57:23,407][WARN ][c.f.s.c.ConfigurationLoaderSG7] [test-single-centos-7] No data for rolesmapping while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS]  (index=searchguard and type=null)
[2020-01-06T14:57:23,407][WARN ][c.f.s.c.ConfigurationLoaderSG7] [test-single-centos-7] No data for tenants while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS]  (index=searchguard and type=null)
[2020-01-06T14:57:23,931][INFO ][o.e.c.m.MetaDataMappingService] [test-single-centos-7] [searchguard/EJRIeHZAT-SoOqM--vWxgw] create_mapping [_doc]
[2020-01-06T14:57:24,089][INFO ][o.e.c.m.MetaDataMappingService] [test-single-centos-7] [searchguard/EJRIeHZAT-SoOqM--vWxgw] update_mapping [_doc]
[2020-01-06T14:57:24,210][INFO ][o.e.c.m.MetaDataMappingService] [test-single-centos-7] [searchguard/EJRIeHZAT-SoOqM--vWxgw] update_mapping [_doc]
[2020-01-06T14:57:24,307][INFO ][o.e.c.m.MetaDataMappingService] [test-single-centos-7] [searchguard/EJRIeHZAT-SoOqM--vWxgw] update_mapping [_doc]
[2020-01-06T14:57:24,387][INFO ][o.e.c.m.MetaDataMappingService] [test-single-centos-7] [searchguard/EJRIeHZAT-SoOqM--vWxgw] update_mapping [_doc]
[2020-01-06T14:57:24,481][INFO ][o.e.c.m.MetaDataMappingService] [test-single-centos-7] [searchguard/EJRIeHZAT-SoOqM--vWxgw] update_mapping [_doc]
[2020-01-06T14:57:28,594][INFO ][c.f.s.c.ConfigurationRepository] [test-single-centos-7] Search Guard License Info: No license needed because enterprise modules are not enabled
[2020-01-06T14:57:31,400][INFO ][c.f.s.c.ConfigurationRepository] [test-single-centos-7] Node 'test-single-centos-7' initialized

Try to curl with user/pass:

[root@test-single-centos-7 ~]# curl -k https://localhost:9200/ -u admin -p
Enter host password for user 'admin':
Search Guard not initialized (SG11). See https://docs.search-guard.com/latest/sgadmin

Try to curl by passing admin certificates:

[root@test-single-centos-7 ~]# curl -k --key /etc/elasticsearch/kirk.key --cert /etc/elasticsearch/kirk.crt https://localhost:9200
{
  "name" : "test-single-centos-7",
  "cluster_name" : "elk_test_cluster",
  "cluster_uuid" : "qUg54x66REGg4GRn8bhYVg",
  "version" : {
    "number" : "7.5.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "e9ccaed468e2fac2275a3761849cbee64b39519f",
    "build_date" : "2019-11-26T01:06:52.518245Z",
    "build_snapshot" : false,
    "lucene_version" : "8.3.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Set the logging to DEBUG level:

[root@test-single-centos-7 ~]# curl -k --cert /etc/elasticsearch/kirk.crt --key /etc/elasticsearch/kirk.key -X PUT "https://localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d '{ "transient": { "logger.com.floragunn": "debug" } }'
{"acknowledged":true,"persistent":{},"transient":{"logger":{"com":{"floragunn":"debug"}}}}[root@test-single-centos-7 ~]#

Ran sgadmin.sh:

[root@test-single-centos-7 ~]# /usr/share/elasticsearch/plugins/search-guard-7/tools/sgadmin.sh -h '127.0.0.1' -cd '/usr/share/elasticsearch/plugins/search-guard-7/sgconfig' -icl -nhnv -key /etc/elasticsearch/kirk.key -cert /etc/elasticsearch/kirk.crt -cacert /etc/elasticsearch/x-pack/ca.crt

WARNING: JAVA_HOME not set, will use /usr/bin/java
Search Guard Admin v7
Will connect to 127.0.0.1:9300 ... done
Connected as CN=admin.elastictest.com,OU=Ops,O=elastictest Com\, Inc.,DC=elastictest,DC=com
Elasticsearch Version: 7.5.0
Search Guard Version: 7.5.0-37.1.0
Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
Clustername: elk_test_cluster
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
searchguard index already exists, so we do not need to create one.
Populate config from /usr/share/elasticsearch/plugins/search-guard-7/sgconfig/
/usr/share/elasticsearch/plugins/search-guard-7/sgconfig/sg_action_groups.yml OK
/usr/share/elasticsearch/plugins/search-guard-7/sgconfig/sg_internal_users.yml OK
/usr/share/elasticsearch/plugins/search-guard-7/sgconfig/sg_roles.yml OK
/usr/share/elasticsearch/plugins/search-guard-7/sgconfig/sg_roles_mapping.yml OK
/usr/share/elasticsearch/plugins/search-guard-7/sgconfig/sg_config.yml OK
/usr/share/elasticsearch/plugins/search-guard-7/sgconfig/sg_tenants.yml OK
Will update '_doc/config' with /usr/share/elasticsearch/plugins/search-guard-7/sgconfig/sg_config.yml
   SUCC: Configuration for 'config' created or updated
Will update '_doc/roles' with /usr/share/elasticsearch/plugins/search-guard-7/sgconfig/sg_roles.yml
   SUCC: Configuration for 'roles' created or updated
Will update '_doc/rolesmapping' with /usr/share/elasticsearch/plugins/search-guard-7/sgconfig/sg_roles_mapping.yml
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '_doc/internalusers' with /usr/share/elasticsearch/plugins/search-guard-7/sgconfig/sg_internal_users.yml
   SUCC: Configuration for 'internalusers' created or updated
Will update '_doc/actiongroups' with /usr/share/elasticsearch/plugins/search-guard-7/sgconfig/sg_action_groups.yml
   SUCC: Configuration for 'actiongroups' created or updated
Will update '_doc/tenants' with /usr/share/elasticsearch/plugins/search-guard-7/sgconfig/sg_tenants.yml
   SUCC: Configuration for 'tenants' created or updated
Done with success

Executed curl with credentials:

[root@test-single-centos-7 ~]# curl -k https://localhost:9200/ -u admin -p
Enter host password for user 'admin':
Search Guard not initialized (SG11). See https://docs.search-guard.com/latest/sgadmin[root@test-single-centos-7 ~]#

Executed curl with keys:

[root@test-single-centos-7 ~]# curl -k --key /etc/elasticsearch/kirk.key --cert /etc/elasticsearch/kirk.crt https://localhost:9200
{
  "name" : "test-single-centos-7",
  "cluster_name" : "elk_test_cluster",
  "cluster_uuid" : "qUg54x66REGg4GRn8bhYVg",
  "version" : {
    "number" : "7.5.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "e9ccaed468e2fac2275a3761849cbee64b39519f",
    "build_date" : "2019-11-26T01:06:52.518245Z",
    "build_snapshot" : false,
    "lucene_version" : "8.3.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Output of elasticsearch log after DEBUG and sgadmin.sh:

[2020-01-06T15:10:38,520][INFO ][o.e.n.Node               ] [test-single-centos-7] stopping ...
[2020-01-06T15:10:38,528][INFO ][o.e.x.w.WatcherService   ] [test-single-centos-7] stopping watch service, reason [shutdown initiated]
[2020-01-06T15:10:38,529][INFO ][o.e.x.w.WatcherLifeCycleService] [test-single-centos-7] watcher has stopped and shutdown
[2020-01-06T15:10:39,276][INFO ][o.e.n.Node               ] [test-single-centos-7] stopped
[2020-01-06T15:10:39,276][INFO ][o.e.n.Node               ] [test-single-centos-7] closing ...
[2020-01-06T15:10:39,308][INFO ][o.e.n.Node               ] [test-single-centos-7] closed
[2020-01-06T15:10:42,600][INFO ][o.e.e.NodeEnvironment    ] [test-single-centos-7] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [32.3gb], net total_space [36.9gb], types [rootfs]
[2020-01-06T15:10:42,603][INFO ][o.e.e.NodeEnvironment    ] [test-single-centos-7] heap size [903.3mb], compressed ordinary object pointers [true]
[2020-01-06T15:10:42,613][INFO ][o.e.n.Node               ] [test-single-centos-7] node name [test-single-centos-7], node ID [gZDCT6YvSnabCzJ-st3dQg], cluster name [elk_test_cluster]
[2020-01-06T15:10:42,613][INFO ][o.e.n.Node               ] [test-single-centos-7] version[7.5.0], pid[4166], build[default/rpm/e9ccaed468e2fac2275a3761849cbee64b39519f/2019-11-26T01:06:52.518245Z], OS[Linux/3.10.0-957.27.2.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13.0.1/13.0.1+9]
[2020-01-06T15:10:42,614][INFO ][o.e.n.Node               ] [test-single-centos-7] JVM home [/usr/share/elasticsearch/jdk]
[2020-01-06T15:10:42,614][INFO ][o.e.n.Node               ] [test-single-centos-7] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms919m, -Xmx919m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -XX:MaxDirectMemorySize=482344960, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2020-01-06T15:10:45,269][INFO ][c.f.s.SearchGuardPlugin  ] [test-single-centos-7] ES Config path is /etc/elasticsearch
[2020-01-06T15:10:45,330][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] [test-single-centos-7] JVM supports TLSv1.3
[2020-01-06T15:10:45,331][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] [test-single-centos-7] Config directory is /etc/elasticsearch/, from there the key- and truststore files are resolved relatively
[2020-01-06T15:10:45,966][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] [test-single-centos-7] TLS Transport Client Provider : JDK
[2020-01-06T15:10:45,967][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] [test-single-centos-7] TLS Transport Server Provider : JDK
[2020-01-06T15:10:45,967][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] [test-single-centos-7] TLS HTTP Provider             : JDK
[2020-01-06T15:10:45,967][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] [test-single-centos-7] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2, TLSv1.1]
[2020-01-06T15:10:45,967][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] [test-single-centos-7] Enabled TLS protocols for HTTP layer      : [TLSv1.3, TLSv1.2, TLSv1.1]
[2020-01-06T15:10:46,329][INFO ][c.f.s.SearchGuardPlugin  ] [test-single-centos-7] Clustername: elk_test_cluster
[2020-01-06T15:10:46,336][WARN ][c.f.s.SearchGuardPlugin  ] [test-single-centos-7] File /etc/elasticsearch/elasticsearch.yml has insecure file permissions (should be 0600)
[2020-01-06T15:10:46,336][WARN ][c.f.s.SearchGuardPlugin  ] [test-single-centos-7] File /etc/elasticsearch/.elasticsearch.keystore.initial_md5sum has insecure file permissions (should be 0600)
[2020-01-06T15:10:46,384][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [aggs-matrix-stats]
[2020-01-06T15:10:46,384][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [analysis-common]
[2020-01-06T15:10:46,384][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [flattened]
[2020-01-06T15:10:46,384][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [frozen-indices]
[2020-01-06T15:10:46,384][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [ingest-common]
[2020-01-06T15:10:46,384][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [ingest-geoip]
[2020-01-06T15:10:46,384][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [ingest-user-agent]
[2020-01-06T15:10:46,384][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [lang-expression]
[2020-01-06T15:10:46,384][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [lang-mustache]
[2020-01-06T15:10:46,384][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [lang-painless]
[2020-01-06T15:10:46,384][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [mapper-extras]
[2020-01-06T15:10:46,384][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [parent-join]
[2020-01-06T15:10:46,384][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [percolator]
[2020-01-06T15:10:46,384][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [rank-eval]
[2020-01-06T15:10:46,385][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [reindex]
[2020-01-06T15:10:46,385][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [repository-url]
[2020-01-06T15:10:46,385][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [search-business-rules]
[2020-01-06T15:10:46,385][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [spatial]
[2020-01-06T15:10:46,385][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [systemd]
[2020-01-06T15:10:46,385][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [transform]
[2020-01-06T15:10:46,385][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [transport-netty4]
[2020-01-06T15:10:46,385][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [vectors]
[2020-01-06T15:10:46,385][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [x-pack-analytics]
[2020-01-06T15:10:46,385][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [x-pack-ccr]
[2020-01-06T15:10:46,385][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [x-pack-core]
[2020-01-06T15:10:46,385][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [x-pack-deprecation]
[2020-01-06T15:10:46,385][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [x-pack-enrich]
[2020-01-06T15:10:46,385][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [x-pack-graph]
[2020-01-06T15:10:46,385][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [x-pack-ilm]
[2020-01-06T15:10:46,385][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [x-pack-logstash]
[2020-01-06T15:10:46,385][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [x-pack-ml]
[2020-01-06T15:10:46,386][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [x-pack-monitoring]
[2020-01-06T15:10:46,386][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [x-pack-rollup]
[2020-01-06T15:10:46,386][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [x-pack-security]
[2020-01-06T15:10:46,386][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [x-pack-sql]
[2020-01-06T15:10:46,386][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [x-pack-voting-only-node]
[2020-01-06T15:10:46,386][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded module [x-pack-watcher]
[2020-01-06T15:10:46,386][INFO ][o.e.p.PluginsService     ] [test-single-centos-7] loaded plugin [search-guard-7]
[2020-01-06T15:10:46,407][INFO ][c.f.s.SearchGuardPlugin  ] [test-single-centos-7] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in elasticsearch.yml
[2020-01-06T15:10:51,802][INFO ][o.e.d.DiscoveryModule    ] [test-single-centos-7] using discovery type [single-node] and seed hosts providers [settings]
[2020-01-06T15:10:52,504][INFO ][o.e.n.Node               ] [test-single-centos-7] initialized
[2020-01-06T15:10:52,505][INFO ][o.e.n.Node               ] [test-single-centos-7] starting ...
[2020-01-06T15:10:52,636][INFO ][o.e.t.TransportService   ] [test-single-centos-7] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2020-01-06T15:10:52,918][INFO ][o.e.c.c.Coordinator      ] [test-single-centos-7] cluster UUID [qUg54x66REGg4GRn8bhYVg]
[2020-01-06T15:10:53,050][INFO ][o.e.c.s.MasterService    ] [test-single-centos-7] elected-as-master ([1] nodes joined)[{test-single-centos-7}{gZDCT6YvSnabCzJ-st3dQg}{iehsnhYMSF6XuofNO2aWOQ}{127.0.0.1}{127.0.0.1:9300}{dim}{xpack.installed=true} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 2, version: 28, delta: master node changed {previous [], current [{test-single-centos-7}{gZDCT6YvSnabCzJ-st3dQg}{iehsnhYMSF6XuofNO2aWOQ}{127.0.0.1}{127.0.0.1:9300}{dim}{xpack.installed=true}]}
[2020-01-06T15:10:53,133][INFO ][o.e.c.s.ClusterApplierService] [test-single-centos-7] master node changed {previous [], current [{test-single-centos-7}{gZDCT6YvSnabCzJ-st3dQg}{iehsnhYMSF6XuofNO2aWOQ}{127.0.0.1}{127.0.0.1:9300}{dim}{xpack.installed=true}]}, term: 2, version: 28, reason: Publication{term=2, version=28}
[2020-01-06T15:10:53,174][INFO ][o.e.h.AbstractHttpServerTransport] [test-single-centos-7] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[2020-01-06T15:10:53,174][INFO ][o.e.n.Node               ] [test-single-centos-7] started
[2020-01-06T15:10:53,203][INFO ][c.f.s.SearchGuardPlugin  ] [test-single-centos-7] Node started
[2020-01-06T15:10:53,203][INFO ][c.f.s.c.ConfigurationRepository] [test-single-centos-7] Check if searchguard index exists ...
[2020-01-06T15:10:53,203][INFO ][c.f.s.c.ConfigurationRepository] [test-single-centos-7] searchguard index does not exist yet, so no need to load config on node startup. Use sgadmin to initialize cluster
[2020-01-06T15:10:53,224][INFO ][c.f.s.SearchGuardPlugin  ] [test-single-centos-7] 0 Search Guard modules loaded so far: []
[2020-01-06T15:10:53,225][INFO ][c.f.s.c.ConfigurationRepository] [test-single-centos-7] Background init thread started. Install default config?: false
[2020-01-06T15:10:53,625][INFO ][o.e.l.LicenseService     ] [test-single-centos-7] license [1d137213-7e00-4cc9-a78d-0930cdfed5ac] mode [basic] - valid
[2020-01-06T15:10:53,637][INFO ][o.e.g.GatewayService     ] [test-single-centos-7] recovered [2] indices into cluster_state
[2020-01-06T15:10:54,146][INFO ][o.e.c.r.a.AllocationService] [test-single-centos-7] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.monitoring-es-7-2020.01.06][0]]]).
[2020-01-06T15:10:54,318][INFO ][c.f.s.c.ConfigurationRepository] [test-single-centos-7] Search Guard License Info: No license needed because enterprise modules are not enabled
[2020-01-06T15:10:54,318][INFO ][c.f.s.c.ConfigurationRepository] [test-single-centos-7] Node 'test-single-centos-7' initialized
[2020-01-06T15:18:15,500][ERROR][c.f.s.a.BackendRegistry  ] [test-single-centos-7] Not yet initialized (you may need to run sgadmin)
[2020-01-06T15:19:53,132][INFO ][c.f.s.c.ConfigurationRepository] [test-single-centos-7] Search Guard License Info: No license needed because enterprise modules are not enabled
[2020-01-06T15:20:18,657][ERROR][c.f.s.a.BackendRegistry  ] [test-single-centos-7] Not yet initialized (you may need to run sgadmin)

The common output in the logs from the curl requests appears to be the ‘Not yet initialized’ messages:

[2020-01-06T15:00:08,440][ERROR][c.f.s.a.BackendRegistry  ] [test-single-centos-7] Not yet initialized (you may need to run sgadmin)
[2020-01-06T15:00:30,268][ERROR][c.f.s.a.BackendRegistry  ] [test-single-centos-7] Not yet initialized (you may need to run sgadmin)
[2020-01-06T15:02:28,135][ERROR][c.f.s.a.BackendRegistry  ] [test-single-centos-7] Not yet initialized (you may need to run sgadmin)
[2020-01-06T15:08:19,310][ERROR][c.f.s.a.BackendRegistry  ] [test-single-centos-7] Not yet initialized (you may need to run sgadmin)
[2020-01-06T15:10:00,411][ERROR][c.f.s.a.BackendRegistry  ] [test-single-centos-7] Not yet initialized (you may need to run sgadmin)

Do you have the same issues if you do not disable the enterprise modules?

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.