Caused by: ElasticsearchException[Is a directory Expected file!]

Search Guard
21

Do I need to use these parameters?

https://github.com/floragunncom/search-guard-docs/blob/master/tls_openssl.md

  • searchguard.ssl.transport.enable_openssl_if_available: true
  • searchguard.ssl.http.enable_openssl_if_available: true
···

On Friday, 28 July 2017 10:08:36 UTC+1, Anthony Cleaves wrote:

The reason I asked about what needs to be in the node elasticsearch.ym was because of this:

“All certificate DNs listed here are considered valid node certificates. Wildcards and regular expressions are supported. If you use this approach, please make sure to list only node certificates.”

So is this documentation wrong, or am I not explaining this very well?

At the moment, I have the following on both of my nodes:

searchguard.ssl.transport.pemkey_filepath: globalsign_x-x.com.key
searchguard.ssl.transport.pemcert_filepath: globalsign_x-x.com.crt
searchguard.ssl.transport.pemtrustedcas_filepath: globalsign_x-x.com.crt
searchguard.ssl.http.pemkey_filepath: globalsign_x-x.com.key
searchguard.ssl.http.pemcert_filepath: globalsign_x-x.com.crt
searchguard.ssl.http.pemtrustedcas_filepath: globalsign_x-x.com.crt
searchguard.authcz.admin_dn:
- CN=*.x-x.com,O=x x x,L=x,ST=x,C=GB
searchguard.nodes_dn:

  - CN=*.x-x.com,O=x x x,L=x,ST=x,C=GB

``

On Thursday, 27 July 2017 22:29:59 UTC+1, Anthony Cleaves wrote:

Using the above gives me the error described earlier:

ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md

Heres some openssl for the cert

Subject: C=GB, ST=x, L=x, O=x x PLC, CN=*.x-x.com

Which matches the yml

On 27 July 2017 at 22:10, Anthony Cleaves runtimusprime@gmail.com wrote:

Ref the node numbers, this is purely dev. I am just trying to ansible this whole installation.

So in each configuration for my elected masted and my elected node, I have the following in my elasticsearc.yml file:

searchguard.authcz.admin_dn:

  • CN=*.x-x.com,O=x x PLC,L=x,ST=x,C=GB

searchguard.nodes_dn:

  • CN=*.x-x.com,O=x x PLC,L=x,ST=x,C=GB

Both are identical, as both are using the wildcard certificate.

On 27 July 2017 at 22:05, SG info@search-guard.com wrote:

What you mean with “master” and “node”?

In terms of Search Guard your ES elected master (and all master eligible nodes) are not different from any other node (data node, ingest node, client node …)

That means that the searchguard related configuration for all nodes is typically identical in elasticsearch.yml and you can point sgadmin against any node

(wheter its the elected master, master eligible, data node, ingest node, client node …).

BTW: For a two node cluster you should not specify a dedicated master node, this makes no sense. For a typical production setup you will normally have 3 dedicated master eligible nodes and a minimum of 2 data nodes.

Am 27.07.2017 um 22:42 schrieb Anthony Cleaves runtimusprime@gmail.com:

I’m a little confused, does the master need both a master and a node definition?

Currently in es.yml for master I have a master dn, and for my node a node dn.

On Thursday, 27 July 2017 21:25:18 UTC+1, Search Guard wrote:

Also SSL related, this time its about the client admin certificate:

https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md

https://github.com/floragunncom/search-guard/issues/366

Am 27.07.2017 um 22:20 schrieb Anthony Cleaves runtim...@gmail.com:

Ok, so that is fixed.

You were spot on (as usual) the issue was I was only specifying a master dn, instead of using nodes as it’s a wildcard all must use node.

Now when running sgadmin on the final run, I get this:

Clustername: actual-cluster

Clusterstate: GREEN

Number of nodes: 2

Number of data nodes: 1

ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md

Trace:

ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]

    at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)

    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)

    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)

    at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)

    at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)

    at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:177)

    at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:191)

    at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:139)

    at com.floragunn.searchguard.SearchGuardPlugin$2$1.messageReceived(SearchGuardPlugin.java:336)

    at [org.elasticsearch.transport.Re](http://org.elasticsearch.transport.Re)questHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)

    at [org.elasticsearch.transport.Tc](http://org.elasticsearch.transport.Tc)pTransport$RequestHandler.doRun(TcpTransport.java:1544)

    at org.elasticsearch.common.util.[concurrent.AbstractRunnable.ru](http://concurrent.AbstractRunnable.ru)n(AbstractRunnable.java:37)

    at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)

    at [org.elasticsearch.transport.Tc](http://org.elasticsearch.transport.Tc)pTransport.handleRequest(TcpTransport.java:1501)

    at [org.elasticsearch.transport.Tc](http://org.elasticsearch.transport.Tc)pTransport.messageReceived(TcpTransport.java:1385)

    at [org.elasticsearch.transport.ne](http://org.elasticsearch.transport.ne)tty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)

    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)

    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)

    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)

    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)

    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)

    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)

    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)

    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)

    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)

    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)

    at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)

    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)

    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)

    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)

    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1267)

    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078)

    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)

    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)

    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)

    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)

    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)

    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)

    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)

    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)

    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)

    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)

    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)

    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)

    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)

    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)

    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)

    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

    at java.lang.Thread.run(Thread.java:748)

The command I am running is below:

/bin/bash /usr/share/elasticsearch/plugins/search-guard-5/tools/sgadmin.sh -cd /usr/share/elasticsearch/plugins/search-guard-5/sgconfig -cn actual-cluster -cert /etc/elasticsearch/elastic-master.x-x.com/x-x.com.crt -cacert /etc/elasticsearch/elastic-master.x-x.com/x-x.com.crt -key /etc/elasticsearch/elastic-master.x-x.com/x-x.com.key -nhnv

On Thursday, 27 July 2017 20:33:41 UTC+1, Search Guard wrote:

How did you create your certificates? Make sure you either have the OID in your node certificates or nodes_dn is defined in elasticsearch.yml

More on that you will find here: https://github.com/floragunncom/search-guard-docs/blob/master/tls_node_certificates.md

“Bad Header” means that one node is not trusting your others and that is because of ssl is not configured properly.

On Thursday, 27 July 2017 12:06:55 UTC+2, Anthony Cleaves wrote:

If I remove

searchguard.ssl.transport.enforce_hostname_verification: false

I see

" SSL Problem Received fatal alert: certificate_unknon"

On Thursday, 27 July 2017 09:43:24 UTC+1, Anthony Cleaves wrote:

Sure, I have x’d out some sensitive info, i can always private message you if you prefer with the complete file.

root@ip-172-31-27-116:~# cat /etc/elasticsearch/elastic-master.x-x.com/elasticsearch.yml

cluster.name: actual-cluster

discovery.zen.ping.unicast.hosts:

http.port: 9200

node.data: false

node.master: true

transport.tcp.port: 9300

node.name: 34.248.89.180-elastic-master.x-x.com

network.host: 0.0.0.0

searchguard.ssl.transport.pemkey_filepath: x-x.com.key

searchguard.ssl.transport.pemcert_filepath: x-x.com.crt

searchguard.ssl.transport.pemtrustedcas_filepath: x-x.com.crt

searchguard.ssl.http.pemkey_filepath: x-x.com.key

searchguard.ssl.http.pemcert_filepath: x-x.com.crt

searchguard.ssl.http.pemtrustedcas_filepath: x-x.com.crt

searchguard.authcz.admin_dn:

    - CN=*.[x-x.com](http://x-x.com),O=x x PLC,L=x,ST=x,C=GB

searchguard.ssl.transport.enforce_hostname_verification: false

#################################### Paths ####################################

Path to directory containing configuration (this file and logging.yml):

path.conf: /etc/elasticsearch/elastic-master.x-x.com

path.data: /var/lib/elasticsearch/34.248.89.180-elastic-master.x-x.com

path.logs: /var/log/elasticsearch/34.248.89.180-elastic-x.x-x.com

On Wednesday, 26 July 2017 17:49:59 UTC+1, Search Guard wrote:

pls provide your elasticsearch.yml

Am 26.07.2017 um 18:41 schrieb Anthony Cleaves runtim...@gmail.com:

I seem to be having issues with two clients clustering. The master is currently saying the following:

[2017-07-26T16:39:53,456][ERROR][c.f.s.t.SearchGuardRequestHandler] Error validating header

Where as the node is saying

Caused by: org.elasticsearch.transport.RemoteTransportException: [34.248.89.180-x-x.x-x.com][172.31.27.116:9300][internal:transport/handshake]

Caused by: org.elasticsearch.ElasticsearchException: bad header found

On Wednesday, 26 July 2017 16:49:39 UTC+1, Search Guard wrote:

no problem

Am 26.07.2017 um 17:30 schrieb Anthony Cleaves runtim...@gmail.com:

Scrap that, I think I found the problem. Thanks for everything!

On Wednesday, 26 July 2017 16:19:48 UTC+1, Anthony Cleaves wrote:

Ah yes, I am seeing different errors now. Can you elaborate on this error for me?

[2017-07-26T15:18:44,134][WARN ][c.f.s.a.BackendRegistry ] Transport authentication finally failed for CN=*.x,O=x,L=x,x=x,C=GB

[2017-07-26T15:18:44,134][ERROR][c.f.s.t.SearchGuardRequestHandler] Cannot authenticate null

(I removed sensitive data)

On Wednesday, 26 July 2017 16:14:15 UTC+1, Search Guard wrote:

Works also on a single node (but you need configure SSL/TLS anyway)

Am 26.07.2017 um 17:08 schrieb Anthony Cleaves runtim...@gmail.com:

Now you mention it, that is blindly obvious haha. Thanks.

Can search guard be installed on a single node or does it always expect a cluster? Just of curiosity for my testing (I will run it in a cluster on production)

On Wednesday, 26 July 2017 16:01:21 UTC+1, Search Guard wrote:

you’re missing the -key option.

We know already that the error message is misleading, this will be fixed in the next version

Am 26.07.2017 um 16:41 schrieb Anthony Cleaves runtim...@gmail.com:

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version
  • Used enterprise modules, if any
  • JVM version and operating system version
  • Search Guard configuration files
  • Elasticsearch log messages on debug level

Hello, I am trying to add this code into ansible to make it a bit easier to deploy in future.

I have ran into an issue, when I run the sgadmin I get the following error:

root@ip-172-31-27-116:/usr/share/elasticsearch/plugins/search-guard-5/tools# ./sgadmin.sh -cd /usr/share/elasticsearch/plugins/search-guard-5/sgconfig -cn actual-cluster -cert /etc/elasticsearch/elastic-master.x-x.com/x-x.com.crt -cacert /etc/elasticsearch/elastic-master.x-x.com/x-x.com.crt -nhnv --diagnose

WARNING: JAVA_HOME not set, will use /usr/bin/java

Search Guard Admin v5

Will connect to localhost:9300 … done

LICENSE NOTICE Search Guard

If you use one or more of the following features in production

make sure you have a valid Search Guard license

(See https://floragunn.com/searchguard-validate-license)

  • Kibana Multitenancy
  • LDAP authentication/authorization
  • Active Directory authentication/authorization
  • REST Management API
  • JSON Web Token (JWT) authentication/authorization
  • Kerberos authentication/authorization
  • Document- and Fieldlevel Security (DLS/FLS)
  • Auditlogging

In case of any doubt mail to sa...@floragunn.com

###################################

ERR: An unexpected ElasticsearchException occured: Failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]

Trace:

ElasticsearchException[Failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]]; nested: InvocationTargetException; nested: ElasticsearchException[Is a directory: /usr/share/elasticsearch/plugins/search-guard-5/tools Expected file!];

at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:434)

at org.elasticsearch.plugins.PluginsService.(PluginsService.java:103)

at org.elasticsearch.client.transport.TransportClient.newPluginService(TransportClient.java:101)

at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:126)

at org.elasticsearch.client.transport.TransportClient.(TransportClient.java:254)

at com.floragunn.searchguard.tools.SearchGuardAdmin$TransportClientImpl.(SearchGuardAdmin.java:715)

at com.floragunn.searchguard.tools.SearchGuardAdmin.main0(SearchGuardAdmin.java:370)

at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:109)

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:423)

… 7 more

Caused by: ElasticsearchException[Is a directory: /usr/share/elasticsearch/plugins/search-guard-5/tools Expected file!]

at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.