Trouble with elasticsearch-filter-plugin

Search Guard
1

Hi,

A bit new into SSL/TLS. I do have managed to get Logstash to work with Searchguard/Elasticsearch (elasticsearch output) on a test server but when trying to use the elasticsearch filter plugin I get a ConnectionFailed warning in Logstash but nothing from Elasticsearch.

logstash.conf

filter {
…
elasticsearch {
hosts => [“https://127.0.0.1:9200”]
index => “logstash-index-ref”
user => “logstash”
password => “*****”
ssl => true
query => “BusinessEmail:%{user}”
fields => { “FirstName” => “FirstName” }
}
…
}

``

output {
elasticsearch {
hosts => [“https://127.0.0.1:9200”]
index => “logstash-%{+YYYY.MM.dd}”
user => “logstash”
password => “*****”
ssl => true
ssl_certificate_verification => false
truststore => “/etc/elasticsearch/truststore.jks”
truststore_password => “changeit”
}
stdout { codec => rubydebug }
}

``

sg_roles.yml

sg_logstash:
cluster:
- indices:admin/template/get
- indices:admin/template/put
- indices:data/write/bulk*
indices:
‘logstash-':
'’:
- CRUD
- CREATE_INDEX
‘beat’:
‘*’:
- CRUD
- CREATE_INDEX

``

I’m using the search-guard demo. Security works across Logstash->ES->Kibana, except I really couldn’t get the elasticsearch filter plugin to work.

logstash-plain.log

[2017-09-10T13:31:19,688][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>“logstash-index-ref”, :query=>“BusinessEmail:-”, :event=>2017-09-10T12:30:54.824Z ubuntu 2017-09-07T04:37:47.805788Z ... -
2017-09-07T04:37:47.808586Z 3.798ms
HTTP/1.1 401 Unauthorized
Content-Type: application/json;charset=UTF-8

{
“responseCode”: 401
}
2017-09-07T04:37:47.809702Z 4.914ms

, :error=>#Faraday::ConnectionFailed}
[2017-09-10T13:31:19,736][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>“logstash-index-ref”, :query=>“BusinessEmail:test@test.com”, :event=>2017-09-10T12:30:54.824Z ubuntu 2017-09-07T04:37:48.216151Z ... test@test.com
2017-09-07T04:37:48.852590Z 637.439ms
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8

{
“responseCode”: 200
}
2017-09-07T04:37:48.853740Z 638.589ms

, :error=>#Faraday::ConnectionFailed}

``

Anyone have an idea? The elasticsearch-filter-plugin doc doesn’t have the other security options in their elasticsearch output API.

Cinto

2

this seems like a logstash issue so please file an issue here Issues · elastic/logstash · GitHub or Logstash - Discuss the Elastic Stack

···

On Sunday, 10 September 2017 15:17:55 UTC+2, c…s@gm…il.com wrote:

Hi,

A bit new into SSL/TLS. I do have managed to get Logstash to work with Searchguard/Elasticsearch (elasticsearch output) on a test server but when trying to use the elasticsearch filter plugin I get a ConnectionFailed warning in Logstash but nothing from Elasticsearch.

logstash.conf

filter {
…
elasticsearch {
hosts => [“https://127.0.0.1:9200”]
index => “logstash-index-ref”
user => “logstash”
password => “*****”
ssl => true
query => “BusinessEmail:%{user}”
fields => { “FirstName” => “FirstName” }
}
…
}

``

output {
elasticsearch {
hosts => [“https://127.0.0.1:9200”]
index => “logstash-%{+YYYY.MM.dd}”
user => “logstash”
password => “*****”
ssl => true
ssl_certificate_verification => false
truststore => “/etc/elasticsearch/truststore.jks”
truststore_password => “changeit”
}
stdout { codec => rubydebug }
}

``

sg_roles.yml

sg_logstash:
cluster:
- indices:admin/template/get
- indices:admin/template/put
- indices:data/write/bulk*
indices:
‘logstash-':
'’:
- CRUD
- CREATE_INDEX
‘beat’:
‘*’:
- CRUD
- CREATE_INDEX

``

I’m using the search-guard demo. Security works across Logstash->ES->Kibana, except I really couldn’t get the elasticsearch filter plugin to work.

logstash-plain.log

[2017-09-10T13:31:19,688][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>“logstash-index-ref”, :query=>“BusinessEmail:-”, :event=>2017-09-10T12:30:54.824Z ubuntu 2017-09-07T04:37:47.805788Z ... -
2017-09-07T04:37:47.808586Z 3.798ms
HTTP/1.1 401 Unauthorized
Content-Type: application/json;charset=UTF-8

{
“responseCode”: 401
}
2017-09-07T04:37:47.809702Z 4.914ms

, :error=>#Faraday::ConnectionFailed}
[2017-09-10T13:31:19,736][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>“logstash-index-ref”, :query=>“BusinessEmail:test@test.com”, :event=>2017-09-10T12:30:54.824Z ubuntu 2017-09-07T04:37:48.216151Z ... test@test.com
2017-09-07T04:37:48.852590Z 637.439ms
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8

{
“responseCode”: 200
}
2017-09-07T04:37:48.853740Z 638.589ms

, :error=>#Faraday::ConnectionFailed}

``

Anyone have an idea? The elasticsearch-filter-plugin doc doesn’t have the other security options in their elasticsearch output API.

Cinto

3

Thanks.

Looked more into this, probably related to:

···

On Monday, September 11, 2017 at 1:11:05 AM UTC+8, Search Guard wrote:

this seems like a logstash issue so please file an issue here https://github.com/elastic/logstash/issues or https://discuss.elastic.co/c/logstash

On Sunday, 10 September 2017 15:17:55 UTC+2, c…s@gm…il.com wrote:

Hi,

A bit new into SSL/TLS. I do have managed to get Logstash to work with Searchguard/Elasticsearch (elasticsearch output) on a test server but when trying to use the elasticsearch filter plugin I get a ConnectionFailed warning in Logstash but nothing from Elasticsearch.

logstash.conf

filter {
…
elasticsearch {
hosts => [“https://127.0.0.1:9200”]
index => “logstash-index-ref”
user => “logstash”
password => “*****”
ssl => true
query => “BusinessEmail:%{user}”
fields => { “FirstName” => “FirstName” }
}
…
}

``

output {
elasticsearch {
hosts => [“https://127.0.0.1:9200”]
index => “logstash-%{+YYYY.MM.dd}”
user => “logstash”
password => “*****”
ssl => true
ssl_certificate_verification => false
truststore => “/etc/elasticsearch/truststore.jks”
truststore_password => “changeit”
}
stdout { codec => rubydebug }
}

``

sg_roles.yml

sg_logstash:
cluster:
- indices:admin/template/get
- indices:admin/template/put
- indices:data/write/bulk*
indices:
‘logstash-':
'’:
- CRUD
- CREATE_INDEX
‘beat’:
‘*’:
- CRUD
- CREATE_INDEX

``

I’m using the search-guard demo. Security works across Logstash->ES->Kibana, except I really couldn’t get the elasticsearch filter plugin to work.

logstash-plain.log

[2017-09-10T13:31:19,688][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>“logstash-index-ref”, :query=>“BusinessEmail:-”, :event=>2017-09-10T12:30:54.824Z ubuntu 2017-09-07T04:37:47.805788Z ... -
2017-09-07T04:37:47.808586Z 3.798ms
HTTP/1.1 401 Unauthorized
Content-Type: application/json;charset=UTF-8

{
“responseCode”: 401
}
2017-09-07T04:37:47.809702Z 4.914ms

, :error=>#Faraday::ConnectionFailed}
[2017-09-10T13:31:19,736][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>“logstash-index-ref”, :query=>“BusinessE...@test.com”, :event=>2017-09-10T12:30:54.824Z ubuntu 2017-09-07T04:37:48.216151Z ... test@test.com
2017-09-07T04:37:48.852590Z 637.439ms
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8

{
“responseCode”: 200
}
2017-09-07T04:37:48.853740Z 638.589ms

, :error=>#Faraday::ConnectionFailed}

``

Anyone have an idea? The elasticsearch-filter-plugin doc doesn’t have the other security options in their elasticsearch output API.

Cinto