Hi there,
I’m trying to setup searchguard but sgadmin fails with the following output
vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ sudo ./sgadmin.sh -cert ~/admin001.crt -cacert /etc/elasticsearch/rest.ca -key ~/admin001.key --disable-host-name-verification --diagnose -icl
WARNING: JAVA_HOME not set, will use /usr/bin/java
Search Guard Admin v5
Will connect to localhost:9300 … done
LICENSE NOTICE Search Guard
If you use one or more of the following features in production
make sure you have a valid Search Guard license
(See Licensing | Search Guard Community, Enterprise and Compliance Edition)
- Kibana Multitenancy
- LDAP authentication/authorization
- Active Directory authentication/authorization
- REST Management API
- JSON Web Token (JWT) authentication/authorization
- Kerberos authentication/authorization
- Document- and Fieldlevel Security (DLS/FLS)
- Auditlogging
In case of any doubt mail to sales@floragunn.com
···
###################################
Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-5/tools/sgadmin_diag_trace_2017-Nov-22_17-10-58.txt
Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …
Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying …
Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)
- Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
- Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
- If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
- Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying …
Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException) - Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
- Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
- If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
- Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
Clustername: elasticsearch
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
Trace:
ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1553)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1510)
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1393)
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1273)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1084)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at java.lang.Thread.run(Thread.java:748)
``
Meanwhile, in the elasticsearch logs, we have (traceback redacted for readability);
Nov 22 17:10:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.ConfigurationLoader] Failure no such index retrieving configuration for [roles] (index=searchguard)
Nov 22 17:11:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.IndexBaseConfigurationRepository] Unable to load configuration because of java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for roles
Nov 22 17:11:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.f.SearchGuardFilter] Search Guard not initialized (SG11) for indices:monitor/stats
Nov 22 17:12:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.ConfigurationLoader] Failure no such index retrieving configuration for [roles] (index=searchguard)
Nov 22 17:13:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.IndexBaseConfigurationRepository] Unable to load configuration because of java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for roles
Nov 22 17:13:04 elasticsearch elasticsearch[1180]: java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for roles
Nov 22 17:13:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.f.SearchGuardFilter] Search Guard not initialized (SG11) for indices:admin/exists
``
Here we show that the admin certificate is indeed valid against the specified CA
vagrant@elasticsearch:~$ openssl x509 -noout -text -in ~/admin001.crt | grep Subject:
Subject: CN=admin001
vagrant@elasticsearch:~$ sudo openssl verify -CAfile /etc/elasticsearch/rest.ca ~/admin001.crt
/home/vagrant/admin001.crt: OK
``
Here is the elasticsearch.yml configuration
vagrant@elasticsearch:~$ sudo grep -e ‘^$’ -e ‘^#’ --invert-match /etc/elasticsearch/elasticsearch.yml
cluster.name: “elasticsearch”
node.name: “elasticsearchminion”
node.master: true
node.data: true
network.bind_host: 0.0.0.0
network.publish_host: 0.0.0.0
network.host: 0.0.0.0
http.port: 9201
discovery.zen.ping.unicast.hosts: [
]
cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.low: 15gb
cluster.routing.allocation.disk.watermark.high: 5gb
searchguard.ssl.transport.pemkey_filepath: transport.key
searchguard.ssl.transport.pemcert_filepath: transport.cert
searchguard.ssl.transport.pemtrustedcas_filepath: transport.ca
searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.keystore_type: x509
searchguard.ssl.http.pemkey_filepath: rest.key
searchguard.ssl.http.pemcert_filepath: rest.cert
searchguard.ssl.http.pemtrustedcas_filepath: rest.ca
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.authcz.admin_dn:
- ‘*’
searchguard.nodes_dn: - ‘*’
searchguard.audit.type: com.payplug.auditlog.impl.StdoutAuditLog
searchguard.audit.enable_request_details: true
searchguard.audit.ignore_users: - kibanaserver
``
What am I missing here ? For what it’s worth, all cryptographic material have been generated using openssl/easyrsa (I’m planning on documenting it).
Please let me know if I can provide any extra informations.
Arthur