# Signals runtime data not accessible

**URL:** <https://forum.search-guard.com/t/signals-runtime-data-not-accessible/2203>\
**Category:** Signals Alerting\
**Created:** [August 19, 2021, 7:19am UTC](https://forum.search-guard.com/t/signals-runtime-data-not-accessible/2203 "2021-08-19T07:19:54Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![marwojt](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/marwojt/32/872_2.png) [@marwojt](https://forum.search-guard.com/u/marwojt)\
**Post date:** [August 19, 2021, 7:19am UTC](https://forum.search-guard.com/t/signals-runtime-data-not-accessible/2203/1 "2021-08-19T07:19:54Z")

</div>

Hi,  
few days ago we have upgraded/patched cluster with [Search Guard 7.x Bugfix Release July 2021](https://docs.search-guard.com/latest/changelog-searchguard-7.x-bugfix-20217-07). for ELK 7.9.3  
I see there is some problem with accessing runtime data in type condition.  
For example simple watch.

**watch**  
[  
{  
“type”: “search”,  
“name”: “mysearch”,  
“target”: “mysearch”,  
“request”: {  
“indices”: [  
“kubernetes-prod-logs”  
],  
“body”: {  
“from”: 0,  
“size”: 0,  
“query”: {  
“bool”: {  
“filter”: {  
“range”: {  
“@timestamp”: {  
“gte”: “now-5m”,  
“lte”: “now”  
}  
}  
},  
“must”: [  
{  
“term”: {  
“log.level”: “ERROR”  
}  
}  
]  
}  
}  
}  
}  
},  
{  
“type”: “condition”,  
“name”: “mycondition”,  
“source”: “data.mysearch.hits.total.value \> 0”  
}  
]

**Response**  
{  
“watch”: {  
“id”: “\_\_inline\_watch”,  
“tenant”: “signals-prod”  
},  
“data”: {  
“mysearch”: {  
“\_shards”: {  
“total”: 198,  
“failed”: 0,  
“successful”: 198,  
“skipped”: 189  
},  
“hits”: {  
“hits”: ,  
“total”: {  
“value”: 692,  
“relation”: “eq”  
},  
“max\_score”: null  
},  
“took”: 42,  
“timed\_out”: false  
}  
},  
“severity”: null,  
“trigger”: {  
“triggered\_time”: null,  
“scheduled\_time”: null,  
“previous\_scheduled\_time”: null,  
“next\_scheduled\_time”: null  
},  
“execution\_time”: “2021-08-19T06:57:30.413567858Z”  
}

**result**  
{  
“tenant”: “signals-prod”,  
“watch\_id”: “test”,  
“status”: {  
“code”: “NO\_ACTION”,  
“detail”: “No action needed due to check mycondition”  
},  
“execution\_start”: “2021-08-19T07:07:57.096Z”,  
“execution\_end”: “2021-08-19T07:07:59.566Z”,  
“actions”: ,  
“node”: “pias-essignals002.srv24246aca-kvm.signals”,  
“\_id”: “pk48XXsBW\_l-8rou7kbO”,  
“\_index”: “.signals\_log\_2021.08.19”  
}

Watches created before patch are working correct but new one don’t.

---

<div class="post-metadata">

**Author:** ![marwojt](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/marwojt/32/872_2.png) [@marwojt](https://forum.search-guard.com/u/marwojt)\
**Post date:** [August 19, 2021, 8:07am UTC](https://forum.search-guard.com/t/signals-runtime-data-not-accessible/2203/2 "2021-08-19T08:07:54Z")

</div>

If I add condition like this  
“source”: “data.mysearch.hits.total.value \>= 0”  
then action is executed  
If i add condition  
“source”: “data.mysearch.hits.total.value \>= 1”  
then action is not executed  
I also try with aggs in search

```
    "aggs": {
      "levelcount": {
        "value_count": {
          "field": "log.level"
        }
      }

```

with condition  
“source”: “data.mysearch.aggregations.levelcount.value \>= 0”  
then i got exception

“error”: {  
“check”: “Condition mycondition”,  
“message”: “Cannot invoke "Object.getClass()" because "callArgs[0]" is null”,  
“detail”: {  
“type”: “script\_exception”,  
“reason”: “runtime error”,  
“script\_stack”: [  
“data.mysearch.aggregations.levelcount.value \>= 0”,  
" ^---- HERE"  
],  
“script”: “data.mysearch.aggregations.levelcount.value \>= 0”,  
“lang”: “painless”,  
“position”: {  
“offset”: 26,  
“start”: 0,  
“end”: 48  
},  
“caused\_by”: {  
“type”: “null\_pointer\_exception”,  
“reason”: “Cannot invoke "Object.getClass()" because "callArgs[0]" is null”  
}  
}  
},

```
    }

```

---

<div class="post-metadata">

**Author:** ![marwojt](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/marwojt/32/872_2.png) [@marwojt](https://forum.search-guard.com/u/marwojt)\
**Post date:** [August 20, 2021, 9:42am UTC](https://forum.search-guard.com/t/signals-runtime-data-not-accessible/2203/3 "2021-08-20T09:42:16Z")

</div>

Any idea what can be wrong?

---

<div class="post-metadata">

**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/pablo/32/1443_2.png) [@pablo](https://forum.search-guard.com/u/pablo)\
**Post date:** [August 20, 2021, 6:36pm UTC](https://forum.search-guard.com/t/signals-runtime-data-not-accessible/2203/4 "2021-08-20T18:36:48Z")

</div>

Hello @marwojt

In regards to your first query, do you get any results when you change the size to more than 0 (i.e. 5)?  
Have you tried to run query against keyword and no text field?

---

<div class="post-metadata">

**Author:** ![marwojt](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/marwojt/32/872_2.png) [@marwojt](https://forum.search-guard.com/u/marwojt)\
**Post date:** [August 22, 2021, 7:56pm UTC](https://forum.search-guard.com/t/signals-runtime-data-not-accessible/2203/5 "2021-08-22T19:56:48Z")

</div>

If i change size i get docs in response. I also try condition hits.hits.lenght but it didn’t work also.  
Log.level field it is a keword field.

---

<div class="post-metadata">

**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/pablo/32/1443_2.png) [@pablo](https://forum.search-guard.com/u/pablo)\
**Post date:** [August 23, 2021, 1:02am UTC](https://forum.search-guard.com/t/signals-runtime-data-not-accessible/2203/6 "2021-08-23T01:02:47Z")

</div>

> [@marwojt](#):
>
> “log.level”: “ERROR”

This will access text value and not keyword.

Could you try below instead?

```auto
“log.level.keyword”: “ERROR”

```

---

<div class="post-metadata">

**Author:** ![marwojt](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/marwojt/32/872_2.png) [@marwojt](https://forum.search-guard.com/u/marwojt)\
**Post date:** [August 23, 2021, 11:44am UTC](https://forum.search-guard.com/t/signals-runtime-data-not-accessible/2203/7 "2021-08-23T11:44:49Z")

</div>

I’ve changed it as You suggested but it doesn’t help - still no action executed

**query**  
[  
{  
“type”: “static”,  
“name”: null,  
“target”: “constants”,  
“value”: {  
“threshold\_test”: 0  
}  
},  
{  
“request”: {  
“indices”: [  
“kubernetes-prod-logs”  
],  
“body”: {  
“size”: 1,  
“query”: {  
“bool”: {  
“must”: {  
“range”: {  
“@timestamp”: {  
“gte”: “now/m-5m”,  
“lt”: “now/m”  
}  
}  
}  
}  
},  
“aggs”: {  
“levelcount”: {  
“value\_count”: {  
“field”: “log.level.keyword”  
}  
}  
}  
}  
},  
“type”: “search”,  
“name”: “mysearch”,  
“target”: “mysearch”  
},  
{  
“name”: “mycondition”,  
“source”: “data.mysearch.hits.total.value \> 0”,  
“type”: “condition”,  
“lang”: “painless”  
}  
]  
**response**  
{  
“watch”: {  
“id”: “\_\_inline\_watch”,  
“tenant”: “signals-prod”  
},  
“data”: {  
“mysearch”: {  
“\_shards”: {  
“total”: 207,  
“failed”: 0,  
“successful”: 207,  
“skipped”: 198  
},  
“hits”: {  
“hits”: ,  
“total”: {  
“value”: 10000,  
“relation”: “gte”  
},  
“max\_score”: 3  
},  
“took”: 1601,  
“timed\_out”: false,  
“aggregations”: {  
“levelcount”: {  
“value”: 0  
}  
}  
},  
“constants”: {  
“threshold\_test”: 0  
}  
},  
“severity”: null,  
“trigger”: {  
“triggered\_time”: null,  
“scheduled\_time”: null,  
“previous\_scheduled\_time”: null,  
“next\_scheduled\_time”: null  
},  
“execution\_time”: “2021-08-23T11:43:18.787588482Z”

---

<div class="post-metadata">

**Author:** ![marwojt](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/marwojt/32/872_2.png) [@marwojt](https://forum.search-guard.com/u/marwojt)\
**Post date:** [August 26, 2021, 11:55am UTC](https://forum.search-guard.com/t/signals-runtime-data-not-accessible/2203/8 "2021-08-26T11:55:25Z")

</div>

I’ve checked signals with other search indexes and it works fine.  
But i have a group of indices with signals don;t work. It looks that there is some problem with parsing json.

[sg\_singnals\_log.txt](https://forum.search-guard.com/uploads/short-url/emA4QUGQtA6zncqh6P0OF5Cg5nC.txt) (20.0 KB)

---

<div class="post-metadata">

**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/pablo/32/1443_2.png) [@pablo](https://forum.search-guard.com/u/pablo)\
**Post date:** [August 27, 2021, 1:56pm UTC](https://forum.search-guard.com/t/signals-runtime-data-not-accessible/2203/9 "2021-08-27T13:56:29Z")

</div>

@marwojt

It seems like indexes from the log file have malformed data. How do you collect the logs and push them to Elasticsearch?  
The Signals watch performs a GET API query which you can test in dev tools or with the curl command.

---

<div class="post-metadata">

**Author:** ![marwojt](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/marwojt/32/872_2.png) [@marwojt](https://forum.search-guard.com/u/marwojt)\
**Post date:** [September 14, 2021, 9:07pm UTC](https://forum.search-guard.com/t/signals-runtime-data-not-accessible/2203/10 "2021-09-14T21:07:36Z")

</div>

Hi,  
query from watch is working fine in dev tools and also in signals UI when is executed.  
I supposed is something with .signals\_watches mapping [signals\_watches\_mapping.json](https://forum.search-guard.com/uploads/short-url/ytUljUfB2CHLad2N1jk49CCZoZW.json) (19.4 KB).  
Could you tell me which field is used for checks.request.indices (text/keyword) when watch is executed.

- “index”:{
  - “type”:“text”,
  - “fields”:{
    - “keyword”:{
      - “type”:“keyword”,
      - “ignore\_above”:256}}},

- “indices”:{
  - “type”:“text”,
  - “fields”:{
    - “keyword”:{
      - “type”:“keyword”,
      - “ignore\_above”:256}}},

---

<div class="post-metadata">

**Author:** ![sirHusky](https://avatars.discourse-cdn.com/v4/letter/s/e9a140/32.png) [@sirHusky](https://forum.search-guard.com/u/sirHusky)\
**Post date:** [September 16, 2021, 4:49pm UTC](https://forum.search-guard.com/t/signals-runtime-data-not-accessible/2203/11 "2021-09-16T16:49:55Z")

</div>

@marwojt The .signals\_watches mapping should really not have any impact on this. Would you be able to run the same exact query with size set to 1, and paste the result here, I would suspect the data received back might provide some insight. Please redact any sensitive details.

---

<div class="post-metadata">

**Author:** ![marwojt](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/marwojt/32/872_2.png) [@marwojt](https://forum.search-guard.com/u/marwojt)\
**Post date:** [September 30, 2021, 10:06am UTC](https://forum.search-guard.com/t/signals-runtime-data-not-accessible/2203/12 "2021-09-30T10:06:29Z")

</div>

Hi,  
I have new insights. It looks like issue it’s not related with mapping as i thought.  
It’s something related with indices with specific aliases and creating SearchInput.  
I have indices with ILM for example  
kopernik-k8s-nonprod-dps-logs-2021.08.31-000011 with aliases (kopernik-k8s-nonprod-dps-logs, kubernetes-nonprod-logs).  
If I run watch on this data, does not matter, i write index or alias i get error like this  
[https://forum.search-guard.com/uploads/short-url/emA4QUGQtA6zncqh6P0OF5Cg5nC.txt](https://forum.search-guard.com/uploads/short-url/emA4QUGQtA6zncqh6P0OF5Cg5nC.txt)

If i create index with same data but change name and no aliases for example  
kop-k8s-nonprod-dps-logs-2021.08.31-000011  
It’ working fine.  
If i add alias any of (kopernik-k8s-nonprod-dps-logs, kubernetes-nonprod-logs) it’s not working anymore.  
Even if i add alias with one word “kopernik” or “kubernetes” it’s stop working.

As a workaround we are indexing same data to short name indices like “kop-dps-logs” without aliases and run watches on them.

---

<div class="post-metadata">

**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/pablo/32/1443_2.png) [@pablo](https://forum.search-guard.com/u/pablo)\
**Post date:** [September 30, 2021, 1:25pm UTC](https://forum.search-guard.com/t/signals-runtime-data-not-accessible/2203/13 "2021-09-30T13:25:31Z")

</div>

Hi @marwojt

This error reports specific indexes which might be attached to your alias.  
Have you checked the shards/replicas health status of those indexes?  
Have you tried to remove those indexes from the queried alias?

---

<div class="post-metadata">

**Author:** ![nils](https://avatars.discourse-cdn.com/v4/letter/n/d6d6ee/32.png) [@nils](https://forum.search-guard.com/u/nils)\
**Post date:** [October 4, 2021, 10:59am UTC](https://forum.search-guard.com/t/signals-runtime-data-not-accessible/2203/14 "2021-10-04T10:59:26Z")

</div>

> [@marwojt](#):
>
> If I run watch on this data, does not matter, i write index or alias i get error like this  
> [https://forum.search-guard.com/uploads/short-url/emA4QUGQtA6zncqh6P0OF5Cg5nC.txt](https://forum.search-guard.com/uploads/short-url/emA4QUGQtA6zncqh6P0OF5Cg5nC.txt)

From the logs:

```auto
  "caused_by": {
              "type": "i_o_exception",
              "reason": "java.util.concurrent.ExecutionException: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('\\' (code 92)): was expecting double-quote to start field name\n at [Source: (StringReader); line: 1, column: 3]",
              "caused_by": {
                "type": "execution_exception",
                "reason": "execution_exception: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('\\' (code 92)): was expecting double-quote to start field name\n at [Source: (StringReader); line: 1, column: 3]",
                "caused_by": {
                  "type": "i_o_exception",
                  "reason": "Unexpected character ('\\' (code 92)): was expecting double-quote to start field name\n at [Source: (StringReader); line: 1, column: 3]"
                }
              }
            }

```

Would it be possible that you check the Elasticsearch logs whether you can find log entries corresponding to the above error? If we would have a stack trace of that error, that might be very helpful.