Yes, I was going to create Index alerting from ES/Kibana alerts and actions. And then use SG’s Alerting app to fire those alerts to Slack/pagerduty etc.
{“type”:“log”,“@timestamp”:“2020-08-13T20:07:05Z”,“tags”:[“error”,“plugins”,“alerting”,“plugins”,“alerting”],“pid”:8,“message”:“enable(): Failed to load API key to invalidate on alert a7c15802-3c04-452d-b836-2efde02c09ae: Saved object [alert/a7c15802-3c04-452d-b836-2efde02c09ae] not found”}
Do, we need to add rules/roles in SG to allow creation of Saved Objects alert/<>
The Elastic Kibana Alerting and the Search Guard Signals Alerting are two completely separate applications. Signals is an Elasticsearch plugin that runs watches inside your ES cluster, similar to Elastic Watcher. The Kibana Alerting from Elasticsearch runs completely in Kibana. So the two apps are not compatible with each other.
@sc75651 I see Kibana alerting supports Webhook and Index actions. You can integrate Signals and Kibana alerting using one of this actions.
For example
Setup a Kibana alert to index docs when triggered.
Setup a Signals watch to periodically search for the documents in the Kibana alert index and do one of the available actions.
“type”:“log”,“@timestamp”:“2020-08-13T20:07:05Z”,“tags”:[“error”,“plugins”,“alerting”,“plugins”,“alerting”],“pid”:8,“message”:“enable(): Failed to load API key to invalidate on alert a7c15802-3c04-452d-b836-2efde02c09ae: Saved object [alert/a7c15802-3c04-452d-b836-2efde02c09ae] not found”}
Do, we need to add rules/roles in SG to allow creation of Saved Objects alert/<>
See if you have any other errors in the Elasticsearch log. Usually, there is a hint for the required permission.
Thanks for quick reply, yes we are trying to do same thing… hard luck finding any errors in ES, but only Kibana throws errors mentioned above, hence seem it is UI based as Kibana alerts are originated and executed by them and not ES
We tested with ELK Stack without SG and above Example - Step 1 works, seems After adding SG to mix some permissions are not allowing the Alert to be fired.
Observation from ELK (Without SG and Security)
1 - Alert Created
2 - Attached Actions
3 - Enabled Alert (and this went active)
So, seems SG might be blocking https://kibana/api/alert/61409c90-dda8-11ea-830e-332c8e77d0a1 from execution…
Kibana alerting requires to set API keys (impersonation). The debug log warns about this.
kibana.example.com | {"type":"log","@timestamp":"2020-09-07T12:45:59Z","tags":["debug","plugins","encryptedSavedObjects"],"pid":7,"message":"The following attributes of saved object \"alert,8abf443b-8953-4b61-a874-fd40ec72b107\" should have been decrypted: apiKey, but found only: "}
Again, I think you can do all things you want with our Signals alert. You just need to know how to use Elasticsearch DSL to search for the data. Or use the graph mode if you need simple alerts. Here, look at some videos and the documentation