Spent many hours configuring this tool... SSL problem

Hello,

I do not know where to begin. Maybe first…

Elasticsearch version:

7.8.1

Server OS version:

Centos 8

Kibana version (if relevant):

7.8.1

Browser version (if relevant):

Firefox

Browser OS version (if relevant):
Firefox

Describe the issue:

I cannot configure connection / run generated certificates using sgtlstool.sh scripts.

When enabling shard allocation it receives an error about no DNS resolved. No subject alternative DNS name matching localhost found. I have the correct DNS configured in the certificate. Please verify. I’ve lost several days on this already and can’t manage, every now and then I get new errors with SSL.

Steps to reproduce:

  1. Elasticsearch starts correctly.
  2. Trying to run the following command to enable shard allocation:

./sgadmin.sh --enable-shard-allocation -cert /etc/elasticsearch/tools/tools/out/kirk.pem -key /etc/elasticsearch/tools/tools/out/kirk.key -keypass changeit -cacert /etc/elasticsearch/tools/tools/out/root-ca.pem

It receives the following error:

ERROR com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport - Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching localhost found.

  1. My certificate is correct, I have specified the private IP address (of my host), and its name.

openssl x509 -noout -subject -in kirk.pem

subject=DC = pl, DC = petty, O = "Petty pl, Inc.", OU = Ops, CN = master-1.elastic.petty.pl

Expected behavior:

Start the cluster properly.

Provide configuration:
elasticsearch/config/elasticsearch.yml

bootstrap.memory_lock: true

#discovery.seed_hosts : []
#cluster.initial_master_nodes : []

cluster.initial_master_nodes:
- master-1
- master-2
cluster.name: elk-1
discovery.seed_hosts:
- master-1:9300
- master-2:9300
- node-1:9300
- node-2:9300
- node-3:9300
http.port: 9200
transport.tcp.port: 9300
network.host: 0.0.0.0
node.data: true
node.master: true
xpack.security.enabled: false
xpack.monitoring.enabled: false
xpack.security.transport.ssl.enabled: false

node.name: master-1


#SSL TRANSPORT

searchguard.enterprise_modules_enabled: false
searchguard.ssl.transport.pemcert_filepath: tools/tools/out/master-1.pem
searchguard.ssl.transport.pemkey_filepath: tools/tools/out/master-1.key
searchguard.ssl.transport.pemkey_password: changeit
searchguard.ssl.transport.pemtrustedcas_filepath: tools/tools/out/root-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: no
searchguard.restapi.roles_enabled: ["SGS_ALL_ACCESS"]

#searchguard.ssl.http.enabled: true
#searchguard.ssl.http.pemcert_filepath: tools/tools/out/master-1_http.pem
#searchguard.ssl.http.pemkey_filepath: tools/tools/out/master-1_http.key
#searchguard.ssl.http.pemtrustedcas_filepath: tools/tools/out/root-ca.pem

#searchguard.nodes_dn:
#  - CN=node-1.elastic.petty.pl,OU=SSL,O=Petty,L=Lodz, C=PL
#  - CN=node-2.elastic.petty.pl,OU=SSL,O=Petty,L=Lodz, C=PL
#  - CN=node-3.elastic.petty.pl,OU=SSL,O=Petty,L=Lodz, C=PL
#searchguard.authcz.admin_dn:
#  - "CN=master-1.elastic.petty.pl,OU=client,O=client,L=Test,C=DE"



#################################### Paths ####################################

# Path to directory containing configuration (this file and logging.yml):

path.data: /data/elasticsearch/data

path.logs: /data/elasticsearch/logs


elasticsearch/plugins/search-guard-7/sgconfig/sg_config.yml (DEFLAUT)
kibana/config/kibana.yml (if relevant)

Provide logs:
Elasticsearch

ERROR com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport - Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching localhost found.

Kibana (if relevant)

Screenshots (if relevant):

Errors in browser console (if relevant):

Additional data:

My configuration to generate SSL:

# Specify the nodes of your ES cluster here
#
nodes:
  - name: master-1
    dn: CN=master-1.elastic.petty.pl,OU=Ops,O=petty pl, Inc.,DC=petty,DC=en
    dns: master-1.elastic.petty.pl
    ip: 10.10.10.27

  - name: node-1
    dn: CN=node-1.elastic.petty.pl,OU=Ops,O=petty pl, Inc.,DC=petty,DC=en
    dns: node-1.elastic.petty.pl
    ip: 10.10.10.28
###
### Clients
###
#
# Specify the clients that shall access your ES cluster with certificate authentication here
#
# At least one client must be an admin user (i.e., a super-user). Admin users can
# be specified with the attribute admin: true
#
clients:
  - name: spock
    dn: CN=spock.example.com,OU=Ops,O=Example Com, Inc.,DC=example,DC=com
  - name: kirk
    dn: CN=master-1.elastic.petty.pl,OU=Ops,O=petty pl, Inc.,DC=petty,DC=pl
    admin: true


A few questions.

  1. Do my nodes need to have https ssl configured to connect to each other?
  2. Do I need to configure https ssl to make the connection work?
  3. If I don’t use -keypass changeit then I get bad SSL information…
  1. does the IP address in the host have to be public?

my /etc/hosts/

10.10.10.27 master-1.elastic.petty.pl master-1
10.10.10.28 node-1.elastic.petty.pl node-1
10.10.10.29 node-2.elastic.petty.pl node-2
10.10.10.30 node-3.elastic.petty.pl node-3
10.10.10.31 kibana.elastic.petty.pl kibana
10.10.10.32 logstash.elastic.petty.pl logstash
10.10.10.33 master-2.elastic.petty.pl master-2

Hello @andrey

In regards to your last questions

  1. Your nodes requires enabled SSL on the transport layer. HTTP can be either run with SSL or not. However, I recommend SSL mode.
  2. As in the previous answer you can either enable SSL on HTTP or not. This is used for API communication on port 9200.
  3. -keypass option is mandatory when your certificates keystore has been created with a password
  4. You can use a private IP range.

In regards to your main question, the error states

`No subject alternative DNS name matching localhost found.’

Every SSL certificate has a field called Subject Alternative Name (SAN). Search Guard plugin is looking for DNS name in that field even when you have CN configured.

You can use openssl (pem certificate) or keytool (jks keystore) to verify the SAN content in the node’s certificate.

Hello @pablo

I have verified the SAN and it appears to be correct.

openssl x509 -text -noout -in kirk.pem -certopt no_subject,no_header,no_version,no_serial,no_signame,no_validity,no_issuer,no_pubkey,no_sigdump,no_aux

Output
  
  >   -certopt no_subject,no_header,no_version,no_serial,no_signame,no_validity,no_issuer,no_pubkey,no_sigdump,no_aux
        X509v3 extensions:
            X509v3 Authority Key Identifier:
                keyid:28:04:69:27:38:6A:88:8B:92:06:09:D2:B0:20:6E:3F:01:B3:72:81
                DirName:/DC=pl/DC=petty/O=Petty/OU=Ops/CN=master-1.elastic.petty.pl
                serial:02

            X509v3 Subject Key Identifier:
                FB:25:A8:E4:C0:A1:79:B2:D8:BE:C0:FB:52:B8:BA:38:A2:4A:6D:29
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Key Usage: critical
                Digital Signature, Non Repudiation, Key Encipherment
            X509v3 Extended Key Usage: critical
                TLS Web Client Authentication

Elasticsearch.yml

searchguard.authcz.admin_dn:
  - CN=master-1.elastic.petty.pl,OU=Ops,O=Petty,DC=master-1.elastic.petty,DC=pl

I did a restart of Elasticsearch and it started up correctly.

but shard-allocation still won’t turn on.

/usr/share/elasticsearch/plugins/search-guard-7/tools/sgadmin.sh --enable-shard-allocation -cert /etc/elasticsearch/tools/tools/out/kirk.pem -key /etc/elasticsearch/tools/tools/out/kirk.key -keypass changeit -cacert /etc/elasticsearch/tools/tools/out/root-ca.pem -cn elk-1

WARNING: JAVA_HOME not set, will use /bin/java
Search Guard Admin v7
Will connect to localhost:9300 ... done
22:17:51.646 [elasticsearch[_client_][transport_worker][T#1]] ERROR com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport - Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching localhost found.
javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching localhost found.
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:1.8.0_302]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:324) ~[?:1.8.0_302]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:267) ~[?:1.8.0_302]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:262) ~[?:1.8.0_302]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654) ~[?:1.8.0_302]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473) ~[?:1.8.0_302]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369) ~[?:1.8.0_302]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377) ~[?:1.8.0_302]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) ~[?:1.8.0_302]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:968) ~[?:1.8.0_302]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:955) ~[?:1.8.0_302]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_302]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:902) ~[?:1.8.0_302]
        at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1542) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1556) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1440) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]
Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching localhost found.
        at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:230) ~[?:1.8.0_302]
        at sun.security.util.HostnameChecker.match(HostnameChecker.java:106) ~[?:1.8.0_302]
        at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:457) ~[?:1.8.0_302]
        at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:431) ~[?:1.8.0_302]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:1.8.0_302]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141) ~[?:1.8.0_302]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:632) ~[?:1.8.0_302]
        ... 31 more
ERR: Cannot connect to Elasticsearch. Please refer to elasticsearch logfile for more information
Trace:
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{KYpDEEmPTKi_aNY0jkwP3g}{localhost}{127.0.0.1:9300}]]
        at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:352)
        at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:248)
        at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:57)
        at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:396)
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:399)
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:388)
        at com.floragunn.searchguard.tools.SearchGuardAdmin.execute(SearchGuardAdmin.java:614)
        at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:156)

@andrey kirk.pem is not the node certificate but admin.
You’ll need to run openssl against the master-1.pem cert file.
Also in output, you should see the following section which I don’t see in your output.

Also in output, you should see the following section which I don’t see in your output. @pablo

Because I do not have this content for the kirk certificate.


I have now changed to master-1.pem

Openssl output:

[root@master-1 out]# openssl x509 -text -noout -in master-1.pem -certopt no_subject,no_header,no_version,no_serial,no_signame,no_validity,no_issuer,no_pubkey,no_sigdump,no_aux
        X509v3 extensions:
            X509v3 Authority Key Identifier:
                keyid:28:04:69:27:38:6A:88:8B:92:06:09:D2:B0:20:6E:3F:01:B3:72:81
                DirName:/DC=pl/DC=petty/O=Petty/OU=Ops/CN=master-1.elastic.petty.pl
                serial:02

            X509v3 Subject Key Identifier:
                F9:B3:36:DB:F7:16:57:AC:1A:72:14:CB:01:14:90:B7:08:77:66:5B
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Key Usage: critical
                Digital Signature, Non Repudiation, Key Encipherment
            X509v3 Extended Key Usage: critical
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Alternative Name:
                DNS:master-1.elastic.petty.pl, IP Address:51.75.34.XX

I am following such a command:

/usr/share/elasticsearch/plugins/search-guard-7/tools/sgadmin.sh --enable-shard-allocation -cert /etc/elasticsearch/tools/tools/out/master-1.pem -key /etc/elasticsearch/tools/tools/out/master-1.key -keypass changeit -cacert /etc/elasticsearch/tools/tools/out/root-ca.pem -cn elk-1

output:

23:28:23.052 [elasticsearch[_client_][transport_worker][T#1]] ERROR com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport - Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching localhost found.
javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching localhost found.
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:1.8.0_302]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:324) ~[?:1.8.0_302]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:267) ~[?:1.8.0_302]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:262) ~[?:1.8.0_302]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654) ~[?:1.8.0_302]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473) ~[?:1.8.0_302]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369) ~[?:1.8.0_302]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377) ~[?:1.8.0_302]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) ~[?:1.8.0_302]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:968) ~[?:1.8.0_302]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:955) ~[?:1.8.0_302]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_302]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:902) ~[?:1.8.0_302]
        at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1542) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1556) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1440) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]
Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching localhost found.
        at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:230) ~[?:1.8.0_302]
        at sun.security.util.HostnameChecker.match(HostnameChecker.java:106) ~[?:1.8.0_302]
        at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:457) ~[?:1.8.0_302]
        at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:431) ~[?:1.8.0_302]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:1.8.0_302]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141) ~[?:1.8.0_302]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:632) ~[?:1.8.0_302]
        ... 31 more
ERR: Cannot connect to Elasticsearch. Please refer to elasticsearch logfile for more information
Trace:
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{op0VjVeXSCCPNvgTzyTftw}{localhost}{127.0.0.1:9300}]]
        at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:352)
        at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:248)
        at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:57)
        at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:396)
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:399)
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:388)
        at com.floragunn.searchguard.tools.SearchGuardAdmin.execute(SearchGuardAdmin.java:614)
        at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:156)

elasticsearch.yml

searchguard.authcz.admin_dn:
  - CN=master-1.elastic.petty.pl,OU=Ops,O=petty,DC=master-1.elastic.petty,DC=pl

Just checked your description again. I don’t see a host option there. When is missing then by default localhost will be used. localhost is not configured in your SAN.

Try the following.

./sgadmin.sh --enable-shard-allocation -cert /etc/elasticsearch/tools/tools/out/kirk.pem -key /etc/elasticsearch/tools/tools/out/kirk.key -keypass changeit -cacert /etc/elasticsearch/tools/tools/out/root-ca.pem -h master-1.elastic.petty.pl

@pablo

Why kirk? You wrote me to change to master-1.pem.
I did openssl validation master-1.pem and there I have the hostname.

openssl x509 -text -noout -in master-1.pem -certopt no_subject,no_header,no_version,no_serial,no_signame,no_validity,no_issuer,no_pubkey,no_sigdump,no_aux
        X509v3 extensions:
            X509v3 Authority Key Identifier:
                keyid:28:04:69:27:38:6A:88:8B:92:06:09:D2:B0:20:6E:3F:01:B3:72:81
                DirName:/DC=pl/DC=petty/O=Petty/OU=Ops/CN=master-1.elastic.petty.pl
                serial:02

            X509v3 Subject Key Identifier:
                F9:B3:36:DB:F7:16:57:AC:1A:72:14:CB:01:14:90:B7:08:77:66:5B
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Key Usage: critical
                Digital Signature, Non Repudiation, Key Encipherment
            X509v3 Extended Key Usage: critical
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Alternative Name:
                **DNS:master-1.elastic.petty.pl, IP Address:51.75.34.XX**

I ran this command:

/usr/share/elasticsearch/plugins/search-guard-7/tools/sgadmin.sh --enable-shard-allocation -cert /etc/elasticsearch/tools/tools/out/master-1.pem -key /etc/elasticsearch/tools/tools/out/master-1.key -keypass changeit -cacert /etc/elasticsearch/tools/tools/out/root-ca.pem -cn elk-1

and i have same problem with DNS:

No subject alternative DNS name matching localhost found.

kirk cert is a client cert that allows executing admin tasks with the sgadmin.sh script

The error doesn’t complain about your client certificate but the node certificate. sgadmin.sh script will compare the host value with the SAN of the node’s certificate at each execution. Since there was no host option in the command, the script took localhost by default.
localhost is not in the SAN of the node’s certificate. That’s why you should use -h option with either IP address or FQDN which are configured in the node certificate’s SAN.

Also, elk-1 is not in the node cert’s SAN.

@andrey

Alternatively, you can use -nhnv option in the command and drop the hostname.
That option disables hostname verification.

image

To see all available options of the sgadmin.sh script run it on its own ‘./sgadmin.sh’.

@pablo

Ok It works, thank you so much.

/usr/share/elasticsearch/plugins/search-guard-7/tools/sgadmin.sh --enable-shard-allocation -cert /etc/elasticsearch/tools/tools/out/kirk.pem -key /etc/elasticsearch/tools/tools/out/kirk.key -keypass changeit -cacert /etc/elasticsearch/tools/tools/out/root-ca.pem -h master-1.elastic.petty.pl -cn elk-1

Search Guard Admin v7
Will connect to master-1.elastic.petty.pl:9300 ... done
Connected as CN=master-1.elastic.petty.pl,OU=Ops,O=Petty,DC=petty,DC=pl
Elasticsearch Version: 7.8.1
Search Guard Version: 7.8.1-43.2.0
Persistent and transient shard allocation enabled

I understand that on all nodes I have to set: searchguard.authcz.admin_dn.

  1. Should I run command “enable shard-allocation” on each node? I have two masters and 3 nodes.?
  2. Should I have additionally configured on each node: searchguard.nodes_dn? I did not find this information in the documentation.?
  3. Is there any way to verify that persistent and transient shard allocation is really enabled?

Thank you

@andrey

  1. enable shard-allocation should be run only once as this is a cluster configuration that is shared across all nodes.

  2. Yes, it’s recommended to use it on all nodes as this option introduce an extra level of security. Once configured, only nodes listed in that option will be allowed to communicate with each other over the transport layer.

  3. Shard-allocation is controlled by Elasticsearch cluster option cluster.routing.allocation.enable.
    By default, this value is set to all (Allow shard allocation for all kinds of shards).
    To verify its status, you’ll need to run curl against https://<es_node>:9200/_cluster/settings
    If the output is missing the cluster.routing.allocation.enable option then it means it’s using the default setting all. Otherwise, it will be present with one of the remaining options: “primaries”, “new_primaries” or “none”.

@pablo

At the moment I get this error:

[master-1] Not yet initialized (you may need to run sgadmin)

I understand that I haven’t initialized the cluster because I don’t have a selected way to create roles and permissions? I have not chosen REST API method or Kibana config GUI? So is that why it is receiving this error?


Additional questions

  1. Should I disable/enable enabled shared every time I generate certificates via tools?
  2. Can I manage permissions / roles via rest api and everything else via Kibana?
  3. Can I connect Kibana to ELK without initialized?

Thank you for your reply.

[2021-11-03T11:12:30,100][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [master-1] Exception during establishing a SSL connection: java.lang.IllegalStateException: transport not ready yet to handle incoming requests
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
        at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:944) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:73) [transport-netty4-client-7.8.1.jar:7.8.1]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-11-03T11:12:30,100][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [master-1] Exception during establishing a SSL connection: java.lang.IllegalStateException: transport not ready yet to handle incoming requests
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
        at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:944) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:73) [transport-netty4-client-7.8.1.jar:7.8.1]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-11-03T11:12:30,111][WARN ][o.e.t.TcpTransport       ] [master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.10.10.27:9300, remoteAddress=/10.10.10.29:39124}], closing connection
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
        at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:944) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:73) [transport-netty4-client-7.8.1.jar:7.8.1]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-11-03T11:12:30,111][WARN ][o.e.t.TcpTransport       ] [master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.10.10.27:9300, remoteAddress=/10.10.10.30:50658}], closing connection
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
        at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:944) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.8.1.jar:7.8.1]r:4.1.49.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-11-03T11:12:30,334][INFO ][o.e.b.BootstrapChecks    ] [master-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2021-11-03T11:12:30,337][INFO ][o.e.c.c.Coordinator      ] [master-1] cluster UUID [rzgRKuFSQ_KHMWgyM70F7w]
[2021-11-03T11:12:30,517][INFO ][o.e.c.s.MasterService    ] [master-1] elected-as-master ([1] nodes joined)[{master-1}{5_tqp9daTVSyeAUWJ2_8cA}{yEEiOIGQT8-R8c4S5_YjVQ}{10.10.10.27}{10.10.10.27:9300}{dilmrt}{ml.machine_memory=14738128896, xpack.installed=true, transform.node=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 67, version: 2344, delta: master node changed {previous [], current [{master-1}{5_tqp9daTVSyeAUWJ2_8cA}{yEEiOIGQT8-R8c4S5_YjVQ}{10.10.10.27}{10.10.10.27:9300}{dilmrt}{ml.machine_memory=14738128896, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}
[2021-11-03T11:12:30,517][WARN ][o.e.d.HandshakingTransportAddressConnector] [master-1] handshake failed for [connectToRemoteMasterNode[10.10.10.29:9300]]
org.elasticsearch.transport.RemoteTransportException: [node-2][10.10.10.29:9300][internal:transport/handshake]
Caused by: org.elasticsearch.ElasticsearchException: Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or searchguard.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See http://docs.search-guard.com/latest/troubleshooting-tls
        at com.floragunn.searchguard.ssl.util.ExceptionUtils.createBadHeaderException(ExceptionUtils.java:57) ~[?:?]
        at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:209) ~[?:?]
        at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:136) ~[?:?]
        at com.floragunn.searchguard.SearchGuardPlugin$7.lambda$interceptHandler$0(SearchGuardPlugin.java:683) ~[?:?]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:73) ~[?:?]]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-11-03T11:12:30,529][WARN ][o.e.d.HandshakingTransportAddressConnector] [master-1] handshake failed for [connectToRemoteMasterNode[10.10.10.30:9300]]
org.elasticsearch.transport.RemoteTransportException: [node-3][10.10.10.30:9300][internal:transport/handshake]
Caused by: org.elasticsearch.ElasticsearchException: Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or searchguard.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See http://docs.search-guard.com/latest/troubleshooting-tls
        at com.floragunn.searchguard.ssl.util.ExceptionUtils.createBadHeaderException(ExceptionUtils.java:57) ~[?:?]
        at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:209) ~[?:?]
        at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:136) ~[?:?]
        at com.floragunn.searchguard.SearchGuardPlugin$7.lambda$interceptHandler$0(SearchGuardPlugin.java:683) ~[?:?]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.8.1.jar:7.8.1]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-11-03T11:12:30,692][INFO ][o.e.c.s.ClusterApplierService] [master-1] master node changed {previous [], current [{master-1}{5_tqp9daTVSyeAUWJ2_8cA}{yEEiOIGQT8-R8c4S5_YjVQ}{10.10.10.27}{10.10.10.27:9300}{dilmrt}{ml.machine_memory=14738128896, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}, term: 67, version: 2344, reason: Publication{term=67, version=2344}
[2021-11-03T11:12:30,766][INFO ][o.e.h.AbstractHttpServerTransport] [master-1] publish_address {10.10.10.27:9200}, bound_addresses {10.10.10.27:9200}
[2021-11-03T11:12:30,767][INFO ][o.e.n.Node               ] [master-1] started
[2021-11-03T11:12:30,779][INFO ][c.f.s.SearchGuardPlugin  ] [master-1] Node started
[2021-11-03T11:12:30,780][INFO ][c.f.s.c.ConfigurationRepository] [master-1] Check if searchguard index exists ...
[2021-11-03T11:12:30,780][INFO ][c.f.s.c.ConfigurationRepository] [master-1] searchguard index does not exist yet, so no need to load config on node startup. Use sgadmin to initialize cluster
[2021-11-03T11:12:30,782][INFO ][c.f.s.SearchGuardPlugin  ] [master-1] 2 Search Guard modules loaded so far: [Module [type=UNKNOWN, implementing class=com.floragunn.signals.Signals], Module [type=UNKNOWN, implementing class=com.floragunn.signals.api.SignalsApiActions]]
[2021-11-03T11:12:30,782][INFO ][c.f.s.c.ConfigurationRepository] [master-1] Background init thread started. Install default config?: false
[2021-11-03T11:12:30,939][INFO ][o.e.c.s.ClusterSettings  ] [master-1] updating [xpack.monitoring.collection.enabled] from [false] to [true]
[2021-11-03T11:12:31,687][INFO ][o.e.l.LicenseService     ] [master-1] license [79370cdc-f2de-4e94-95da-408cc1198505] mode [basic] - valid
[2021-11-03T11:12:31,706][INFO ][o.e.g.GatewayService     ] [master-1] recovered [33] indices into cluster_state

Part II

/usr/share/elasticsearch/plugins/search-guard-7/tools/sgadmin.sh -cert /etc/elasticsearch/tools/tools/out/kirk.pem -key /etc/elasticsearch/tools/tools/out/kirk.key -keypass changeit -cacert /etc/elasticsearch/tools/tools/out/root-ca.pem -h master-1.elastic.petty.pl -cn elk-1

WARNING: JAVA_HOME not set, will use /bin/java
Search Guard Admin v7
Will connect to master-1.elastic.petty.pl:9300 ... done
Connected as CN=master-1.elastic.petty.pl,OU=Ops,O=Petty\, Inc.,DC=petty,DC=pl
Elasticsearch Version: 7.8.1
Search Guard Version: 7.8.1-43.2.0
Contacting elasticsearch cluster 'elk-1' and wait for YELLOW clusterstate ...
Clustername: elk-1
Clusterstate: GREEN
Number of nodes: 4
Number of data nodes: 4
searchguard index does not exists, attempt to create it ... done (0-all replicas)
Populate config from /etc/elasticsearch
ERR: Seems ./sg_action_groups.yml is not in SG 7 format: java.io.FileNotFoundException: ./sg_action_groups.yml (No such file or directory)
ERR: Seems ./sg_internal_users.yml is not in SG 7 format: java.io.FileNotFoundException: ./sg_internal_users.yml (No such file or directory)
ERR: Seems ./sg_roles.yml is not in SG 7 format: java.io.FileNotFoundException: ./sg_roles.yml (No such file or directory)
ERR: Seems ./sg_roles_mapping.yml is not in SG 7 format: java.io.FileNotFoundException: ./sg_roles_mapping.yml (No such file or directory)
ERR: Seems ./sg_config.yml is not in SG 7 format: java.io.FileNotFoundException: ./sg_config.yml (No such file or directory)
ERR: cannot upload configuration because of invalid files, see errors above

try again and:

WARNING: JAVA_HOME not set, will use /bin/java
Search Guard Admin v7
Will connect to master-1.elastic.petty.pl:9300 ... done
Connected as CN=master-1.elastic.petty.pl,OU=Ops,O=Petty\, Inc.,DC=petty,DC=pl
Elasticsearch Version: 7.8.1
Search Guard Version: 7.8.1-43.2.0
Contacting elasticsearch cluster 'elk-1' and wait for YELLOW clusterstate ...

It has this status all the time, pendind status.

I have tried restarting elastisearch on all nodes but same thing…

@andrey in the above command there is no folder name where your configs are.
Try to use -cd option with config folder location.

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.