Beat cannot send the data to the cluster.

Hi,

I tried to configure Beat with Elasticsearch. My cluster installed Search Guard plugin.

Here is the error message.

2019-03-18T09:10:19.204Z INFO template/load.go:130 Template already exists and will not be overwritten.

2019-03-18T09:10:19.204Z INFO instance/beat.go:894 Template successfully loaded.

2019-03-18T09:10:24.367Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {“monitoring”: {“metrics”: {“beat”:{“cpu”:{“system”:{“ticks”:300,“time”:{“ms”:4}},“total”:{“ticks”:1120,“time”:{“ms”:13},“value”:1120},“user”:{“ticks”:820,“time”:{“ms”:9}}},“handles”:{“limit”:{“hard”:4096,“soft”:1024},“open”:12},“info”:{“ephemeral_id”:“ed18843f-ed37-45b2-a197-518ea9f6efa9”,“uptime”:{“ms”:1650015}},“memstats”:{“gc_next”:18523728,“memory_alloc”:13163256,“memory_total”:85844408}},“filebeat”:{“harvester”:{“open_files”:7,“running”:7}},“libbeat”:{“config”:{“module”:{“running”:0}},“output”:{“read”:{“bytes”:3223},“write”:{“bytes”:5142}},“pipeline”:{“clients”:9,“events”:{“active”:4119,“retry”:100}}},“registrar”:{“states”:{“current”:23}},“system”:{“load”:{“1”:0.02,“15”:0,“5”:0.03,“norm”:{“1”:0.0025,“15”:0,“5”:0.0038}}}}}}

2019-03-18T09:10:35.969Z ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(https://10.49.113.81:9200)): Connection marked as failed because the onConnect callback failed: Error loading pipeline for fileset elasticsearch/audit: couldn’t load pipeline: couldn’t load json. Error: 403 Forbidden: {“error”:{“root_cause”:[{“type”:“security_exception”,“reason”:“no permissions for [cluster:admin/ingest/pipeline/put] and User [name=logstash, roles=[logstash], requestedTenant=null]”}],“type”:“security_exception”,“reason”:“no permissions for [cluster:admin/ingest/pipeline/put] and User [name=logstash, roles=[logstash], requestedTenant=null]”},“status”:403}. Response body: {“error”:{“root_cause”:[{“type”:“security_exception”,“reason”:“no permissions for [cluster:admin/ingest/pipeline/put] and User [name=logstash, roles=[logstash], requestedTenant=null]”}],“type”:“security_exception”,“reason”:“no permissions for [cluster:admin/ingest/pipeline/put] and User [name=logstash, roles=[logstash], requestedTenant=null]”},“status”:403}

2019-03-18T09:10:35.970Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(https://10.49.113.81:9200)) with 42 reconnect attempt(s)

2019-03-18T09:10:35.970Z INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer

2019-03-18T09:10:35.970Z INFO [publish] pipeline/retry.go:191 done

2019-03-18T09:10:35.970Z INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer

2019-03-18T09:10:35.970Z INFO [publish] pipeline/retry.go:168 done

2019-03-18T09:10:35.971Z INFO elasticsearch/client.go:721 Connected to Elasticsearch version 6.6.0

``

Why it need permission to write cluster:admin/ingest/pipeline/put? If it needs this permission, where should I put this grant? sg_action_groups or sg_role_mapping?

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version

24.1 and 6.6.0

  • Installed and used enterprise modules, if any

No

  • JVM version and operating system version

1.8

  • Search Guard configuration files

Attached

  • Elasticsearch log messages on debug level

No, It is Beat logs.

  • Other installed Elasticsearch or Kibana plugins, if any

No

sg_action_groups.yml (2.27 KB)

sg_config.yml (9.4 KB)

sg_internal_users.yml (1.05 KB)

sg_roles_mapping.yml (548 Bytes)

sg_roles.yml (6.88 KB)

filebeat.yml (1.27 KB)

in sg_roles.yml try

sg_logstash:
  cluster:
    - CLUSTER_MONITOR
    - CLUSTER_COMPOSITE_OPS
    - indices:admin/template/get
    - indices:admin/template/put
    - cluster:admin/ingest/pipeline/put
    - cluster:admin/ingest/pipeline/get
  indices:
    'logstash-*':
      '*':
        - CRUD
        - CREATE_INDEX
    '*beat*':
      '*':
        - CRUD
        - CREATE_INDEX

···

Am 18.03.2019 um 09:55 schrieb Worapoj Chokeanankun <worapojc@gmail.com>:

Hi,

I tried to configure Beat with Elasticsearch. My cluster installed Search Guard plugin.

Here is the error message.
2019-03-18T08:43:31.023Z ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(https://10.49.112.126:9200)): Connection marked as failed because the onConnect callback failed: Error loading pipeline for fileset system/syslog: couldn't load pipeline: couldn't load json. Error: 403 Forbidden: {"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [cluster:admin/ingest/pipeline/put] and User [name=logstash, roles=[logstash], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [cluster:admin/ingest/pipeline/put] and User [name=logstash, roles=[logstash], requestedTenant=null]"},"status":403}. Response body: {"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [cluster:admin/ingest/pipeline/put] and User [name=logstash, roles=[logstash], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [cluster:admin/ingest/pipeline/put] and User [name=logstash, roles=[logstash], requestedTenant=null]"},"status":403}

Why it need permission to write cluster:admin/ingest/pipeline/put? If it needs this permission, where should I put this grant? sg_action_groups or sg_role_mapping?

When asking questions, please provide the following information:

* Search Guard and Elasticsearch version
24.1 and 6.6.0
* Installed and used enterprise modules, if any
No
* JVM version and operating system version
1.8
* Search Guard configuration files
Attached
* Elasticsearch log messages on debug level
No, It is Beat logs.
* Other installed Elasticsearch or Kibana plugins, if any
No

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/a08fb070-7545-493a-95ff-d83d470be0da%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
<sg_action_groups.yml><sg_config.yml><sg_internal_users.yml><sg_roles_mapping.yml><sg_roles.yml><filebeat.yml>

It works. Thanks