Hi folks,
Can we configure search guard with Fluentd ? - I see no topic for that in search guard documentation.
Thanks,
Hi folks,
Can we configure search guard with Fluentd ? - I see no topic for that in search guard documentation.
Thanks,
I’m not very familiar with fluentd, but from the perspective of Search Guard, fluentd is an HTTP client like any other (Browser, Kibana etc.).
The requirements for an HTTP client are:
must be able to use HTTP instead of HTTPS (if you enable TLS on the ES REST layer)
must be able to accept self-signed certificates (if you enable TLS on the ES REST layer AND use self-signed certificate)
must be able to add authentication information to the REST requests (e.g. HTTP Basic Authentication)
From a brief look at the configuration options of the fluentd Elasticsearch plugin this seems possible:
http://www.rubydoc.info/gems/fluent-plugin-elasticsearch/1.5.0
Configuration would be very similar to Kibana, you need to set these entries:
host or hosts: Use something like “https://mycluster.com” (note the https)
ssl_verify: set this to false if you use a self signed certificate
user and password: set this to the user fluentd uses to connect to ES/SG
In Search Guard, configure HTTP Basic Authentication, add the fluentd user and give it the permissions to read/write into the respective index.
In theory this should work fine, however, I’ve not really tested it.
Am Montag, 30. Januar 2017 11:32:58 UTC+1 schrieb Oucema Bellagha:
Hi folks,
Can we configure search guard with Fluentd ? - I see no topic for that in search guard documentation.
Thanks,
here’s the configurations made but seems to be not working, is there any problems with this config?
td-agent.conf:
<match syslog.**>
@type elasticsearch
host efk1
scheme https
ssl_verify false
user flu
password flu
path /elastic_search/
port 9200
logstash_format true
flush_interval 10s
sg_roles_mapping.yml:
sg_role_fluentd:
users:
sg_internal_users.yml:
flu:
hash: $2a$12$rkViH26YsjJsliZLkb/yrO2LFNEY3asz3gEnb64ekPRWgxve85G/C
sg_roles.yml:
sg_role_fluentd:
indices:
‘*’:
‘*’:
CRUD
CREATE_INDEX
I authenticate to kibana using credentials but so far now data shipped into elasticsearch, fluentd - ES is somehow not working using searchguard config…
up
up up up
On Tuesday, January 31, 2017 at 5:04:07 PM UTC+7, Oucema Bellagha wrote:
up
any error messages in fluentd? in efk1’s elasticsearch logfile?
I have the same problem and I use the same conf , the error message is :
<2018-10-22 14:20:16 +0200 [warn]: #0 failed to flush the buffer. retry_time=10 next_retry_seconds=2018-10-22 14:20:15 +0200 chunk=“578d009d721014c4d22a5b76f701c617” error_class=Fluent::Plugin::ElasticsearchOutput::ConnectionFailure error=“Can not reach Elasticsearch cluster ({:host=>"elasticsearch", :port=>9200, :scheme=>"https", :user=>"fluentd", :password=>"obfuscated"})!”
2018-10-22 14:20:16 +0200 [warn]: #0 suppressed same stacktrace>
Le lundi 4 juin 2018 09:05:07 UTC+2, Fabien Wernli a écrit :
any error messages in fluentd? in efk1’s elasticsearch logfile?
Please do not crosspost:
https://groups.google.com/forum/#!topic/search-guard/dICTDlMeu5k
On Monday, November 5, 2018 at 11:31:32 AM UTC+1, johnzhengaz@gmail.com wrote:
I use user admin, still not work.
@type elasticsearch
host 127.0.0.1
port 9200
logstash_format true
logstash_prefix newf1
logstash_dateformat %Y%m%d
include_tag_key true
type_name access_log
tag_key @log_name
flush_interval 1s
scheme https
ssl_verify false
user admin
password admin
2018-11-05 10:15:09 +0000 [warn]: #0 failed to flush the buffer. retry_time=9 next_retry_seconds=2018-11-05 10:15:08 +0000 chunk=“579e8094725ed8c8ef35e891a1d10f95” error_class=Fluent::Plugin::ElasticsearchOutput::ConnectionFailure error=“Can not reach Elasticsearch cluster ({:host=>"127.0.0.1", :port=>9200, :scheme=>"https", :user=>"admin", :password=>"obfuscated"})!”
I use fluentd with SearchGuard 6.4.0 and it works fine.
I used the logstash user/password defined in the default SG configuration (password hash was updated) and it worked with https. Though admin should also work.
As it did not work with admin then it is likely with your fluentd config and/or connection to ES. You can test connection using curl from the server that is posting the events e.g. curl -sk -u logstash: https://…/_searchguard/health
I’m also fairly new to fluentd and my config is not likely to help you much as my ES traffic goes via a load balancer on https port 443.
Note that we have added a new fluentd chapter to the docs as well:
https://docs.search-guard.com/latest/elasticsearch-fluentd-search-guard
On Tuesday, November 13, 2018 at 11:45:35 AM UTC+1, Martin wrote:
I use fluentd with SearchGuard 6.4.0 and it works fine.
I used the logstash user/password defined in the default SG configuration (password hash was updated) and it worked with https. Though admin should also work.
As it did not work with admin then it is likely with your fluentd config and/or connection to ES. You can test connection using curl from the server that is posting the events e.g. curl -sk -u logstash: https://…/_searchguard/health
I’m also fairly new to fluentd and my config is not likely to help you much as my ES traffic goes via a load balancer on https port 443.
I created a github repository for how search guard integrate with fluentd, FYI
Great, thanks, will add this to our docs.
On Thursday, November 22, 2018 at 11:44:50 AM UTC+1, johnzhengaz@gmail.com wrote:
I created a github repository for how search guard integrate with fluentd, FYI
https://github.com/johnzheng1975/docker-efk/tree/searchGuard