Signals - watch with query on data from remote cluster

Signals Alerting
1

Hi!
At the beginning a bit of info:
First cluster:
ES 6.8.3. with SG
Second cluster:
ES 7.6.0 with SG and Signals
There is a connection ‘remote cluster’ from 2nd to 1st and it works.

I would like to create watch which is using data from remote cluster(1st). Unfortunately i have an error

     "error": {
        "message": "ElasticsearchSecurityException[No user found for indices:data/read/search]"

I have created the same user on both clusters, on the first cluster there is a role which is mapped to that user and using “Discover” it is possible to use remote data.
Where should i add a user or what roles should i config to signals work correctly on remote data?

Best Patryk

2

Hi. I need more data:

  1. The watch in JSON
  2. Any related error from Elasticsearch and Kibana log
3

Sorry, i was on a holiday.
here you have an error:


{
  "tenant": "_main",
  "watch_id": "count_cdrBSCS_wrn",
  "status": {
    "code": "EXECUTION_FAILED",
    "detail": "Error while executing SearchInput count_cdrBSCS: No user found for indices:data/read/search"
  },
  "error": {
    "message": "ElasticsearchSecurityException[No user found for indices:data/read/search]",
    "detail": {
      "type": "security_exception",
      "reason": "No user found for indices:data/read/search"
    }
  },
  "execution_start": "2020-04-24T08:39:22.199Z",
  "execution_end": "2020-04-24T08:39:22.204Z",
  "data": null,
  "actions": [],
  "node": "kibana1",
  "_id": "RydXq3EBD8fpvNabX3tc",
  "_index": ".signals_log_2020.04.24"
}

and definition of watch:


[
  {
    "type": "search",
    "name": "count_cdrBSCS",
    "target": "count_cdrBSCS",
    "request": {
      "indices": [
        "REMOTE_CLUSTER:bscs-converter-*"
      ],
      "body": {
        "query": {
          "bool": {
            "must": {
              "range": {
                "@timestamp": {
                  "gte": "now/h-24h",
                  "lt": "now/h"
                }
              }
            },
            "filter": {
              "term": {
                "message_type": "Outputed"
              }
            }
          }
        },
        "size": 0,
        "aggs": {
          "avgs": {
            "avg": {
              "field": "records"
            }
          }
        }
      }
    }
  },
  {
    "type": "condition.script",
    "name": "no_records",
    "source": "data.count_cdrBSCS.aggregations.avgs.value > 0"
  }
]

and here is the log from remote cluster node:


[2020-05-04T13:32:42,668][ERROR][c.f.s.f.SearchGuardFilter] [node_name] No user found for indices:data/read/search from 10.250.9.34:57752 LOCAL via netty {_sg_origin_header=LOCAL, _sg_remotecn=oss-elk, _sg_initial_action_class_header=SearchRequest}

Best Patryk

4

One more log from node on local cluster:


[2020-05-05T09:46:31,054][INFO ][c.f.s.e.WatchRunner      ] [esnode2] Finished orca_count_cdrBSCS_wrn_copy: EXECUTION_FAILED Error while executing SearchInput orca_count_cdrBSCS: No user found for indices:data/read/search
[2020-05-05T09:46:31,054][INFO ][c.f.s.e.WatchRunner      ] [esnode2] Error while executing orca_count_cdrBSCS_wrn_copy
com.floragunn.signals.execution.WatchExecutionException: Error while executing SearchInput orca_count_cdrBSCS
        at com.floragunn.signals.execution.WatchRunner.executeChecks(WatchRunner.java:238) ~[sg-suite-signals-7.6.0-40.0.0.jar:7.6.0-40.0.0]
        at com.floragunn.signals.execution.WatchRunner.execute(WatchRunner.java:151) ~[sg-suite-signals-7.6.0-40.0.0.jar:7.6.0-40.0.0]
        at com.floragunn.signals.execution.WatchRunner.execute(WatchRunner.java:126) [sg-suite-signals-7.6.0-40.0.0.jar:7.6.0-40.0.0]
        at com.floragunn.searchsupport.jobs.execution.AuthorizingJobDecorator.execute(AuthorizingJobDecorator.java:37) [sg-suite-scheduler-7.6.0-40.0.0.jar:7.6.0-40.0.0]
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-2.3.1.jar:?]
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) [quartz-2.3.1.jar:?]
Caused by: org.elasticsearch.ElasticsearchSecurityException: No user found for indices:data/read/search

On remote cluster(6.8.6) i have enterprise edition of SG and Community edition on local cluster(7.6.0), so my watch uses default _main tenant. But where should i connect it and set to use the same user as in remote cluster settings?

Another tip, during creation mode of watch, in Simulate mode when i click “execute” the query works correctly but robot which run watch according to the schedule has a problem and errors as I showed you. It looks that in GUI remote_cluster works corectly and it uses remote_user to search data but signals robot tries to use another user. It is possible to set it somewhere to use remote_user too?

5

Hello Patryk!

There is indeed a bug in Signals which prevents the authorization to be correctly propagated to the remote cluster. We will fix that for the next Search Guard version which is scheduled to be released on June, 2nd.

This will however only fix this for the case where the remote cluster is also running Search Guard 7. A remote cluster running Search Guard 6 unfortunately still won’t be able to pick up the authorization. This is because watches executed by Signals need a special mechanism to store their authorization in order to be re-used during the scheduled execution (this is the authorization by the user at the point in time when the user created the watch). Search Guard 6 cannot handle this special mechanism yet.

2 Likes
6

ok, that’s sad news for me but thanks for your reply. We will try it after upgrade 6 version to 7th.
i’d like to add i found another bug, there isn’t posibility to deactivate an alarm. when i click the button “Activate”, there is no reaction.
2020-05-15_08h52_03

8

@Patryk Can you try the latest release for 7.6.0? It was out a week ago. There were some fixes. Latest Releases | Security for Elasticsearch | Search Guard