Logstash won't write to elasticsearch with 403

Search Guard
1

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version
    6.5.1

  • Search Guard configuration files

sg_logstash:
{
“cluster”: [
“CLUSTER_MONITOR”,
“CLUSTER_COMPOSITE_OPS”,
“indices:admin/template/get”,
“indices:admin/template/put”
],
“indices”: {
“logstash-": {
"”: [
“CRUD”,
“CREATE_INDEX”
]
},
“beat”: {
“": [
“CRUD”,
“CREATE_INDEX”
]
},
"robot_montoring_”: {
“": [
“CRUD”,
“CREATE_INDEX”
]
},
"server_monitoring_”: {
“*”: [
“CREATE_INDEX”,
“CRUD”
]
}
},
“tenants”: {}
}

``

I have the demo config as a basis.

My indices are not called ‘logstash-*’ so I added two more indices with the right expression and the same authorization.

  • Logstash log messages

[2018-12-05T09:37:51,153][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({“type”=>“security_exception”, “reason”=>“no permissions for [indices:data/write/index, indices:data/write/bulk[s]] and User [name=logstash, roles=[logstash], requestedTenant=null]”})
[2018-12-05T09:37:51,153][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>3}
[2018-12-05T09:37:55,767][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({“type”=>“security_exception”, “reason”=>“no permissions for [indices:admin/create] and User [name=logstash, roles=[logstash], requestedTenant=null]”})

What’s the solution to logstash refusing to write data here?
Is it because it has no tenants? (I think not, or I wouldn’t know what tenant to give it)
Do I just have to add the rights it asks for? But if yes, why aren’t they set correctly in the demo config?

Regards.

2

Maybe this sounds silly, but have you updated your role definition by using sgadmin? Or did you use the REST API to perform the update?

When logstash prints the error, you should also see corresponding error messages in the Elastic logs, can you post them as well?

···

On Wednesday, December 5, 2018 at 9:52:10 AM UTC+1, marc.zominy@hoomano.com wrote:

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version
    6.5.1
  • Search Guard configuration files

sg_logstash:
{
“cluster”: [
“CLUSTER_MONITOR”,
“CLUSTER_COMPOSITE_OPS”,
“indices:admin/template/get”,
“indices:admin/template/put”
],
“indices”: {
“logstash-": {
"”: [
“CRUD”,
“CREATE_INDEX”
]
},
“beat”: {
“": [
“CRUD”,
“CREATE_INDEX”
]
},
"robot_montoring_”: {
“": [
“CRUD”,
“CREATE_INDEX”
]
},
"server_monitoring_”: {
“*”: [
“CREATE_INDEX”,
“CRUD”
]
}
},
“tenants”: {}
}

``

I have the demo config as a basis.

My indices are not called ‘logstash-*’ so I added two more indices with the right expression and the same authorization.

  • Logstash log messages

[2018-12-05T09:37:51,153][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({“type”=>“security_exception”, “reason”=>“no permissions for [indices:data/write/index, indices:data/write/bulk[s]] and User [name=logstash, roles=[logstash], requestedTenant=null]”})
[2018-12-05T09:37:51,153][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>3}
[2018-12-05T09:37:55,767][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({“type”=>“security_exception”, “reason”=>“no permissions for [indices:admin/create] and User [name=logstash, roles=[logstash], requestedTenant=null]”})

What’s the solution to logstash refusing to write data here?
Is it because it has no tenants? (I think not, or I wouldn’t know what tenant to give it)
Do I just have to add the rights it asks for? But if yes, why aren’t they set correctly in the demo config?

Regards.

3

Thank you for your reply.

No problem asking silly questions, this could totally be my own fault!

Not this time though, at least I think. I’m using the Kibana GUI to update the roles, the JSON I gave you was obtained by using the ‘show JSON’ button in the GUI.

These are the logs written by elasticsearch:

[2018-12-05T10:12:13,260][INFO ][c.f.s.p.PrivilegesEvaluator] [k2TEshW] No index-level perm match for User [name=logstash, roles=[logstash], requestedTenant=null] Resolved [aliases=, indices=[robot_monitoring_mic_energy_0], allIndices=[robot_monitoring_mic_energy_0], types=[doc], isAll()=false, isEmpty()=false] [Action [indices:data/write/bulk[s]]] [RolesChecked [sg_logstash, sg_own_index]]
[2018-12-05T10:12:13,260][INFO ][c.f.s.p.PrivilegesEvaluator] [k2TEshW] No permissions for [indices:data/write/index, indices:data/write/bulk[s]]
[2018-12-05T10:12:13,848][INFO ][c.f.s.p.PrivilegesEvaluator] [k2TEshW] No index-level perm match for User [name=logstash, roles=[logstash], requestedTenant=null] Resolved [aliases=, indices=[robot_monitoring_health_2], allIndices=[robot_monitoring_health_2], types=[*], isAll()=false, isEmpty()=false] [Action [indices:admin/create]] [RolesChecked [sg_logstash, sg_own_index]]
[2018-12-05T10:12:13,848][INFO ][c.f.s.p.PrivilegesEvaluator] [k2TEshW] No permissions for [indices:admin/create]

``

Do you need them in debug mode?

4

Oh … at closer look, I think the solution is quite easy. Seems you have a typo in your index name:

"robot_montoring_*": {
  "*": [
    "CRUD",
    "CREATE_INDEX"
  ]
},

Should read “monitoring” instead of “montoring”

···

On Wednesday, December 5, 2018 at 12:17:02 PM UTC+1, marc.zominy@hoomano.com wrote:

Thank you for your reply.

No problem asking silly questions, this could totally be my own fault!

Not this time though, at least I think. I’m using the Kibana GUI to update the roles, the JSON I gave you was obtained by using the ‘show JSON’ button in the GUI.

These are the logs written by elasticsearch:

[2018-12-05T10:12:13,260][INFO ][c.f.s.p.PrivilegesEvaluator] [k2TEshW] No index-level perm match for User [name=logstash, roles=[logstash], requestedTenant=null] Resolved [aliases=, indices=[robot_monitoring_mic_energy_0], allIndices=[robot_monitoring_mic_energy_0], types=[doc], isAll()=false, isEmpty()=false] [Action [indices:data/write/bulk[s]]] [RolesChecked [sg_logstash, sg_own_index]]
[2018-12-05T10:12:13,260][INFO ][c.f.s.p.PrivilegesEvaluator] [k2TEshW] No permissions for [indices:data/write/index, indices:data/write/bulk[s]]
[2018-12-05T10:12:13,848][INFO ][c.f.s.p.PrivilegesEvaluator] [k2TEshW] No index-level perm match for User [name=logstash, roles=[logstash], requestedTenant=null] Resolved [aliases=, indices=[robot_monitoring_health_2], allIndices=[robot_monitoring_health_2], types=[*], isAll()=false, isEmpty()=false] [Action [indices:admin/create]] [RolesChecked [sg_logstash, sg_own_index]]
[2018-12-05T10:12:13,848][INFO ][c.f.s.p.PrivilegesEvaluator] [k2TEshW] No permissions for [indices:admin/create]

``

Do you need them in debug mode?

5

No problem asking silly questions, this could totally be my own fault!

Can’t say I didn’t tell you haha.

I’m sorry I used your time on a typo, I swear I’ve read it 100 times without ever noticing…

It totally works now, thanks for your help.

6

No problem - happened to me before as well :wink:

···

On Wednesday, December 5, 2018 at 4:58:23 PM UTC+1, marc.zominy@hoomano.com wrote:

No problem asking silly questions, this could totally be my own fault!

Can’t say I didn’t tell you haha.

I’m sorry I used your time on a typo, I swear I’ve read it 100 times without ever noticing…

It totally works now, thanks for your help.