Using Search Guard with fluentd

Hi folks,

Can we configure search guard with Fluentd ? - I see no topic for that in search guard documentation.

Thanks,

I’m not very familiar with fluentd, but from the perspective of Search Guard, fluentd is an HTTP client like any other (Browser, Kibana etc.).

The requirements for an HTTP client are:

  • must be able to use HTTP instead of HTTPS (if you enable TLS on the ES REST layer)

  • must be able to accept self-signed certificates (if you enable TLS on the ES REST layer AND use self-signed certificate)

  • must be able to add authentication information to the REST requests (e.g. HTTP Basic Authentication)

From a brief look at the configuration options of the fluentd Elasticsearch plugin this seems possible:

http://www.rubydoc.info/gems/fluent-plugin-elasticsearch/1.5.0

Configuration would be very similar to Kibana, you need to set these entries:

host or hosts: Use something like “https://mycluster.com” (note the https)

ssl_verify: set this to false if you use a self signed certificate

user and password: set this to the user fluentd uses to connect to ES/SG

In Search Guard, configure HTTP Basic Authentication, add the fluentd user and give it the permissions to read/write into the respective index.

In theory this should work fine, however, I’ve not really tested it.

···

Am Montag, 30. Januar 2017 11:32:58 UTC+1 schrieb Oucema Bellagha:

Hi folks,

Can we configure search guard with Fluentd ? - I see no topic for that in search guard documentation.

Thanks,

here’s the configurations made but seems to be not working, is there any problems with this config?

td-agent.conf:

<match syslog.**>

@type elasticsearch

host efk1

scheme https

ssl_verify false

user flu

password flu

path /elastic_search/

port 9200

logstash_format true

flush_interval 10s

sg_roles_mapping.yml:

sg_role_fluentd:

users:

  • flu

sg_internal_users.yml:

flu:

hash: $2a$12$rkViH26YsjJsliZLkb/yrO2LFNEY3asz3gEnb64ekPRWgxve85G/C

sg_roles.yml:

sg_role_fluentd:

indices:

‘*’:

‘*’:

  • CRUD

  • CREATE_INDEX

I authenticate to kibana using credentials but so far now data shipped into elasticsearch, fluentd - ES is somehow not working using searchguard config…

up

up up up

···

On Tuesday, January 31, 2017 at 5:04:07 PM UTC+7, Oucema Bellagha wrote:

up

any error messages in fluentd? in efk1’s elasticsearch logfile?

I have the same problem and I use the same conf , the error message is :

<2018-10-22 14:20:16 +0200 [warn]: #0 failed to flush the buffer. retry_time=10 next_retry_seconds=2018-10-22 14:20:15 +0200 chunk=“578d009d721014c4d22a5b76f701c617” error_class=Fluent::Plugin::ElasticsearchOutput::ConnectionFailure error=“Can not reach Elasticsearch cluster ({:host=>"elasticsearch", :port=>9200, :scheme=>"https", :user=>"fluentd", :password=>"obfuscated"})!”

2018-10-22 14:20:16 +0200 [warn]: #0 suppressed same stacktrace>

···

Le lundi 4 juin 2018 09:05:07 UTC+2, Fabien Wernli a écrit :

any error messages in fluentd? in efk1’s elasticsearch logfile?

Please do not crosspost:

https://groups.google.com/forum/#!topic/search-guard/dICTDlMeu5k

···

On Monday, November 5, 2018 at 11:31:32 AM UTC+1, johnzhengaz@gmail.com wrote:

I use user admin, still not work.

@type elasticsearch

host 127.0.0.1

port 9200

logstash_format true

logstash_prefix newf1

logstash_dateformat %Y%m%d

include_tag_key true

type_name access_log

tag_key @log_name

flush_interval 1s

scheme https

ssl_verify false

user admin

password admin

2018-11-05 10:15:09 +0000 [warn]: #0 failed to flush the buffer. retry_time=9 next_retry_seconds=2018-11-05 10:15:08 +0000 chunk=“579e8094725ed8c8ef35e891a1d10f95” error_class=Fluent::Plugin::ElasticsearchOutput::ConnectionFailure error=“Can not reach Elasticsearch cluster ({:host=>"127.0.0.1", :port=>9200, :scheme=>"https", :user=>"admin", :password=>"obfuscated"})!”

I use fluentd with SearchGuard 6.4.0 and it works fine.

I used the logstash user/password defined in the default SG configuration (password hash was updated) and it worked with https. Though admin should also work.

As it did not work with admin then it is likely with your fluentd config and/or connection to ES. You can test connection using curl from the server that is posting the events e.g. curl -sk -u logstash: https://…/_searchguard/health

I’m also fairly new to fluentd and my config is not likely to help you much as my ES traffic goes via a load balancer on https port 443.

Note that we have added a new fluentd chapter to the docs as well:

https://docs.search-guard.com/latest/elasticsearch-fluentd-search-guard

···

On Tuesday, November 13, 2018 at 11:45:35 AM UTC+1, Martin wrote:

I use fluentd with SearchGuard 6.4.0 and it works fine.

I used the logstash user/password defined in the default SG configuration (password hash was updated) and it worked with https. Though admin should also work.

As it did not work with admin then it is likely with your fluentd config and/or connection to ES. You can test connection using curl from the server that is posting the events e.g. curl -sk -u logstash: https://…/_searchguard/health

I’m also fairly new to fluentd and my config is not likely to help you much as my ES traffic goes via a load balancer on https port 443.

I created a github repository for how search guard integrate with fluentd, FYI

Great, thanks, will add this to our docs.

···

On Thursday, November 22, 2018 at 11:44:50 AM UTC+1, johnzhengaz@gmail.com wrote:

I created a github repository for how search guard integrate with fluentd, FYI
https://github.com/johnzheng1975/docker-efk/tree/searchGuard