pipelines.yml not load with module

Sergey_Murashov · April 12, 2018, 10:52am

ELK version - 6.2.2
SG version - 6.2.2-22.0
JVM version - build 1.8.0_161-b12

I installed the NetFlow module and registered its parameters in logstash.yml:

modules:

name: netflow
var.elasticsearch.hosts: “logstash.taxnet.ru:9200”
var.elasticsearch.username: “admin”
var.elasticsearch.password: “admin”
var.elasticsearch.ssl.enabled: true
var.elasticsearch.ssl.verification)mode: disable
var.elasticsearch.ssl.certificate_authority: “/etc/logstash/”
var.kibana.scheme: “http”
var.kibana.host: “logstash.taxnet.ru”
var.kibana.username: “admin”
var.kibana.password: “admin”
var.input.udp.port: 2055

Also commented path.config.

In pipelines.yml i wrote:

pipeline.id: module_netflow
path.conf: “/etc/logstash/conf.d/logstash.conf”
In logstash.conf:

input {
udp {
port => 2055
codec => netflow
}
}
When I restart logstash.service it’s wrote:

[logstash.config.source.multilocal] Ignoring the ‘pipelines.yml’ file because modules or command line options are specified
I try to use solution from here, but deleting --path.config from ExecStart=/usr/share/logstash/bin/logstash “–path.settings” “/etc/logstash” not work.

In this case, in the logs I see

[2018-04-12T13:29:32,919][DEBUG][logstash.inputs.udp ] config LogStash::Inputs::Udp/@type = “netflow”
[2018-04-12T13:29:32,919][DEBUG][logstash.inputs.udp ] config LogStash::Inputs::Udp/@port = 2055
[2018-04-12T13:29:32,920][DEBUG][logstash.inputs.udp ] config LogStash::Inputs::Udp/@codec = <LogStash::Codecs::Netflow versions=>[5, 9], id=>“5143121d-3863-41a6-9c1f-7c3d80398787”, enable_metric=>true, cache_ttl=>4000,
[2018-04-12T13:29:32,921][DEBUG][logstash.inputs.udp ] config LogStash::Inputs::Udp/@workers = 2
[2018-04-12T13:29:32,921][DEBUG][logstash.inputs.udp ] config LogStash::Inputs::Udp/@receive_buffer_bytes = 212992
[2018-04-12T13:29:32,921][DEBUG][logstash.inputs.udp ] config LogStash::Inputs::Udp/@queue_size = 2000
[2018-04-12T13:29:32,922][DEBUG][logstash.inputs.udp ] config LogStash::Inputs::Udp/@id = “05ed24e130a77512c0667b6ef118aaa26d82f0fb4de6e20bbf451ebfdc5dbfbd”
[2018-04-12T13:29:32,922][DEBUG][logstash.inputs.udp ] config LogStash::Inputs::Udp/@enable_metric = true
[2018-04-12T13:29:32,922][DEBUG][logstash.inputs.udp ] config LogStash::Inputs::Udp/@add_field = {}
[2018-04-12T13:29:32,922][DEBUG][logstash.inputs.udp ] config LogStash::Inputs::Udp/@host = “0.0.0.0”
[2018-04-12T13:29:32,922][DEBUG][logstash.inputs.udp ] config LogStash::Inputs::Udp/@buffer_size = 65536

If I understand correctly, it should start to port 2055 and create a netflow index.

When checking netstat on the port, there is nothing and no indexes are created.

Please help

jkressin · April 15, 2018, 11:27pm

The question is if this is related to Search Guard or not. If your configuration also does not work with Search Guard disabled, the you would need to ask on the Elasticsearch forum I’m afraid.

For a check you can quickly disable SG in elasticsearch.yml like:

···

On Thursday, April 12, 2018 at 2:35:58 AM UTC-7, Sergey Murashov wrote:

ELK version - 6.2.2

SG version - 6.2.2-22.0

JVM version - build 1.8.0_161-b12