# Using Search Guard with fluentd **URL:** \ **Category:** Search Guard\ **Created:** [January 30, 2017, 10:32am UTC](https://forum.search-guard.com/t/using-search-guard-with-fluentd/376 "2017-01-30T10:32:58Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Oucema\_Bellagha](https://avatars.discourse-cdn.com/v4/letter/o/eb8c5e/32.png) [@Oucema\_Bellagha](https://forum.search-guard.com/u/Oucema_Bellagha)\ **Post date:** [January 30, 2017, 10:32am UTC](https://forum.search-guard.com/t/using-search-guard-with-fluentd/376/1 "2017-01-30T10:32:58Z")
Hi folks, Can we configure search guard with Fluentd ? - I see no topic for that in search guard documentation. Thanks, ---
**Author:** ![jkressin](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/jkressin/32/9_2.png) [@jkressin](https://forum.search-guard.com/u/jkressin)\ **Post date:** [January 30, 2017, 10:57am UTC](https://forum.search-guard.com/t/using-search-guard-with-fluentd/376/2 "2017-01-30T10:57:26Z")
I’m not very familiar with fluentd, but from the perspective of Search Guard, fluentd is an HTTP client like any other (Browser, Kibana etc.). The requirements for an HTTP client are: - must be able to use HTTP instead of HTTPS (if you enable TLS on the ES REST layer) - must be able to accept self-signed certificates (if you enable TLS on the ES REST layer AND use self-signed certificate) - must be able to add authentication information to the REST requests (e.g. HTTP Basic Authentication) From a brief look at the configuration options of the fluentd Elasticsearch plugin this seems possible: [http://www.rubydoc.info/gems/fluent-plugin-elasticsearch/1.5.0](http://www.rubydoc.info/gems/fluent-plugin-elasticsearch/1.5.0) Configuration would be very similar to Kibana, you need to set these entries: host or hosts: Use something like “[https://mycluster.com](https://mycluster.com)” (note the https) ssl\_verify: set this to false if you use a self signed certificate user and password: set this to the user fluentd uses to connect to ES/SG In Search Guard, configure HTTP Basic Authentication, add the fluentd user and give it the permissions to read/write into the respective index. In theory this should work fine, however, I’ve not really tested it. > **···** > > Am Montag, 30. Januar 2017 11:32:58 UTC+1 schrieb Oucema Bellagha: > > > Hi folks, > > > Can we configure search guard with Fluentd ? - I see no topic for that in search guard documentation. > > > > > > Thanks, ---
**Author:** ![Oucema\_Bellagha](https://avatars.discourse-cdn.com/v4/letter/o/eb8c5e/32.png) [@Oucema\_Bellagha](https://forum.search-guard.com/u/Oucema_Bellagha)\ **Post date:** [January 30, 2017, 3:10pm UTC](https://forum.search-guard.com/t/using-search-guard-with-fluentd/376/3 "2017-01-30T15:10:58Z")
here’s the configurations made but seems to be not working, is there any problems with this config? td-agent.conf: \ @type elasticsearch host efk1 scheme https ssl\_verify false user flu password flu path /elastic\_search/ port 9200 logstash\_format true flush\_interval 10s sg\_roles\_mapping.yml: sg\_role\_fluentd: users: - flu sg\_internal\_users.yml: flu: hash: $2a$12$rkViH26YsjJsliZLkb/yrO2LFNEY3asz3gEnb64ekPRWgxve85G/C sg\_roles.yml: sg\_role\_fluentd: indices: ‘\*’: ‘\*’: - CRUD - CREATE\_INDEX I authenticate to kibana using credentials but so far now data shipped into elasticsearch, fluentd - ES is somehow not working using searchguard config… ---
**Author:** ![Oucema\_Bellagha](https://avatars.discourse-cdn.com/v4/letter/o/eb8c5e/32.png) [@Oucema\_Bellagha](https://forum.search-guard.com/u/Oucema_Bellagha)\ **Post date:** [January 31, 2017, 10:04am UTC](https://forum.search-guard.com/t/using-search-guard-with-fluentd/376/4 "2017-01-31T10:04:07Z")
up ---
**Author:** ![rd85](https://avatars.discourse-cdn.com/v4/letter/r/9f8e36/32.png) [@rd85](https://forum.search-guard.com/u/rd85)\ **Post date:** [May 29, 2018, 1:42am UTC](https://forum.search-guard.com/t/using-search-guard-with-fluentd/376/5 "2018-05-29T01:42:02Z")
up up up > **···** > > On Tuesday, January 31, 2017 at 5:04:07 PM UTC+7, Oucema Bellagha wrote: > > > up ---
**Author:** ![Fabien\_Wernli](https://avatars.discourse-cdn.com/v4/letter/f/48db29/32.png) [@Fabien\_Wernli](https://forum.search-guard.com/u/Fabien_Wernli)\ **Post date:** [June 4, 2018, 7:05am UTC](https://forum.search-guard.com/t/using-search-guard-with-fluentd/376/6 "2018-06-04T07:05:06Z")
any error messages in fluentd? in efk1’s elasticsearch logfile? ---
**Author:** ![sara\_qasmi](https://avatars.discourse-cdn.com/v4/letter/s/7ab992/32.png) [@sara\_qasmi](https://forum.search-guard.com/u/sara_qasmi)\ **Post date:** [October 22, 2018, 12:27pm UTC](https://forum.search-guard.com/t/using-search-guard-with-fluentd/376/7 "2018-10-22T12:27:25Z")
I have the same problem and I use the same conf , the error message is : \<2018-10-22 14:20:16 +0200 [warn]: #0 failed to flush the buffer. retry\_time=10 next\_retry\_seconds=2018-10-22 14:20:15 +0200 chunk=“578d009d721014c4d22a5b76f701c617” error\_class=Fluent::Plugin::ElasticsearchOutput::ConnectionFailure error=“Can not reach Elasticsearch cluster ({:host=\>"elasticsearch", :port=\>9200, :scheme=\>"https", :user=\>"fluentd", :password=\>"obfuscated"})!” 2018-10-22 14:20:16 +0200 [warn]: #0 suppressed same stacktrace\> > **···** > > Le lundi 4 juin 2018 09:05:07 UTC+2, Fabien Wernli a écrit : > > > any error messages in fluentd? in efk1’s elasticsearch logfile? ---
**Author:** ![jkressin](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/jkressin/32/9_2.png) [@jkressin](https://forum.search-guard.com/u/jkressin)\ **Post date:** [November 5, 2018, 10:43am UTC](https://forum.search-guard.com/t/using-search-guard-with-fluentd/376/8 "2018-11-05T10:43:20Z")
Please do not crosspost: [https://groups.google.com/forum/#!topic/search-guard/dICTDlMeu5k](https://groups.google.com/forum/#!topic/search-guard/dICTDlMeu5k) > **···** > > On Monday, November 5, 2018 at 11:31:32 AM UTC+1, [johnzhengaz@gmail.com](mailto:johnzhengaz@gmail.com) wrote: > > > I use user admin, still not work. > > > @type elasticsearch > > > host 127.0.0.1 > > > port 9200 > > > logstash\_format true > > > logstash\_prefix newf1 > > > logstash\_dateformat %Y%m%d > > > include\_tag\_key true > > > type\_name access\_log > > > tag\_key @log\_name > > > flush\_interval 1s > > > **scheme https** > > > **ssl\_verify false** > > > **user admin** > > > **password admin** > > > 2018-11-05 10:15:09 +0000 [warn]: #0 failed to flush the buffer. retry\_time=9 next\_retry\_seconds=2018-11-05 10:15:08 +0000 chunk=“579e8094725ed8c8ef35e891a1d10f95” error\_class=Fluent::Plugin::ElasticsearchOutput::ConnectionFailure error=“Can not reach Elasticsearch cluster ({:host=\>"127.0.0.1", :port=\>9200, :scheme=\>"https", :user=\>"admin", :password=\>"obfuscated"})!” ---
**Author:** ![Martin](https://avatars.discourse-cdn.com/v4/letter/m/73ab20/32.png) [@Martin](https://forum.search-guard.com/u/Martin)\ **Post date:** [November 13, 2018, 10:45am UTC](https://forum.search-guard.com/t/using-search-guard-with-fluentd/376/9 "2018-11-13T10:45:34Z")
I use fluentd with SearchGuard 6.4.0 and it works fine. I used the logstash user/password defined in the default SG configuration (password hash was updated) and it worked with https. Though admin should also work. As it did not work with admin then it is likely with your fluentd config and/or connection to ES. You can test connection using curl from the server that is posting the events e.g. curl -sk -u logstash: https://…/\_searchguard/health I’m also fairly new to fluentd and my config is not likely to help you much as my ES traffic goes via a load balancer on https port 443. ---
**Author:** ![jkressin](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/jkressin/32/9_2.png) [@jkressin](https://forum.search-guard.com/u/jkressin)\ **Post date:** [November 13, 2018, 2:53pm UTC](https://forum.search-guard.com/t/using-search-guard-with-fluentd/376/10 "2018-11-13T14:53:49Z")
Note that we have added a new fluentd chapter to the docs as well: [https://docs.search-guard.com/latest/elasticsearch-fluentd-search-guard](https://docs.search-guard.com/latest/elasticsearch-fluentd-search-guard) > **···** > > On Tuesday, November 13, 2018 at 11:45:35 AM UTC+1, Martin wrote: > > > I use fluentd with SearchGuard 6.4.0 and it works fine. > > > I used the logstash user/password defined in the default SG configuration (password hash was updated) and it worked with https. Though admin should also work. > > > > > > As it did not work with admin then it is likely with your fluentd config and/or connection to ES. You can test connection using curl from the server that is posting the events e.g. curl -sk -u logstash: https://…/\_searchguard/health > > > > > > I’m also fairly new to fluentd and my config is not likely to help you much as my ES traffic goes via a load balancer on https port 443. > > > > > > ---
**Author:** ![johnzhengaz](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/johnzhengaz/32/622_2.png) [@johnzhengaz](https://forum.search-guard.com/u/johnzhengaz)\ **Post date:** [November 22, 2018, 10:44am UTC](https://forum.search-guard.com/t/using-search-guard-with-fluentd/376/11 "2018-11-22T10:44:49Z")
I created a github repository for how search guard integrate with fluentd, FYI > **[GitHub - johnzheng1975/docker-efk at searchGuard](https://github.com/johnzheng1975/docker-efk/tree/searchGuard)** > > searchGuard ---
**Author:** ![jkressin](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/jkressin/32/9_2.png) [@jkressin](https://forum.search-guard.com/u/jkressin)\ **Post date:** [November 26, 2018, 10:28am UTC](https://forum.search-guard.com/t/using-search-guard-with-fluentd/376/12 "2018-11-26T10:28:32Z")
Great, thanks, will add this to our docs. > **···** > > On Thursday, November 22, 2018 at 11:44:50 AM UTC+1, [johnzhengaz@gmail.com](mailto:johnzhengaz@gmail.com) wrote: > > > I created a github repository for how search guard integrate with fluentd, FYI > > [https://github.com/johnzheng1975/docker-efk/tree/searchGuard](https://github.com/johnzheng1975/docker-efk/tree/searchGuard) > > >