Unable to save watches

lzukel · February 23, 2021, 3:17pm

Elasticsearch version:
7.10.2

Kibana version (if relevant):
7.10.2

Describe the issue:
When creating a watch, I am unable to save the watch.
Kibana returning:

{“type”:“response”,"@timestamp":“2021-02-23T15:11:02Z”,“tags”:,“pid”:16,“method”:“get”,“statusCode”:404,“req”:{“url”:"/api/searchguard-signals/watch/Kibana%20Settings%20Watch",“method”:“get”,“headers”:{“host”:“edited”,“sec-ch-ua”:"“Chromium”;v=“88”, “Google Chrome”;v=“88”, “;Not A Brand”;v=“99"”,“sec-ch-ua-mobile”:"?0",“user-agent”:“Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36”,“kbn-version”:“7.10.2”,“content-type”:“application/json”,“accept”:"/",“sec-fetch-site”:“same-origin”,“sec-fetch-mode”:“cors”,“sec-fetch-dest”:“empty”,“referer”:“https://edited/kibana/app/searchguard-signals",“accept-encoding”:"gzip, deflate, br”,“accept-language”:“en-US,en;q=0.9,fr;q=0.8”,“x-opaque-id”:“mine”,“x-forwarded-for”:“142.136.146.158, 44.128.168.125”,“x-forwarded-host”:“edited”,“x-forwarded-proto”:“https”,“connection”:“close”},“remoteAddress”:“44.128.168.1”,“userAgent”:“Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36”,“referer”:“https://edited/kibana/app/searchguard-signals"},“res”:{“statusCode”:404,“responseTime”:98,“contentLength”:9},“message”:"GET /api/searchguard-signals/watch/Kibana%20Settings%20Watch 404 98ms - 9.0B”}

srgbnd · February 23, 2021, 3:48pm

Do you see any error in the Elasticsearch log? What do you see in UI when you try to save the watch, any error? What do you see in the browser console log (enable the debug mode)?

lzukel · February 23, 2021, 4:10pm

Browser console

Failed to load resource: the server responded with a status of 404 (Not Found)

Nothing that appears relevant in the Elasticsearch logs
The UI does not change when trying to save the watch

srgbnd · February 23, 2021, 5:20pm

Hm. I can’t reproduce it. Do you use Search Guard v49.0.0?
Maybe it is the watch name encoding issue. Try to save a watch with name Kibana_Settings_Watch.

lzukel · February 23, 2021, 5:34pm

search-guard-7	7.10.2-49.0.0

Updated the title to use underscores instead of spaces… Still get 404 in kibana logs

srgbnd · February 23, 2021, 6:54pm

Get the watch for me via the REST API.

curl -k -u admin:admin -X GET https://localhost:9200/_signals/watch/tenant_name/Kibana%20Settings%20Watch

Use your own credentials instead of admin:admin. Also, you need to specify your tenant name instead of tenant_name. If you have the Multitenancy disabled, use _main.

Also, paste the kibana.yml.

lzukel · February 23, 2021, 6:58pm

How to use own credentials, when users are required to use SSO via keycloak?

srgbnd · February 23, 2021, 7:00pm

I think you still have the admin. Because you need it to update the Search Guard config. Try admin.

srgbnd · February 23, 2021, 7:51pm

Can you tell me on which step the error arise? For example

Click Create Watch
Put values. What values?
Click Save
ERROR

Can you attach the screenshot?

lzukel · February 23, 2021, 8:08pm

curl -k -u admin:password -X GET https://host:9200/_signals/watch/_main/Kibana%20Settings%20Watch
{
“status” : 403,
“error” : “no permissions for [cluster:admin:searchguard:tenant:signals:watch/get] and User [name=admin, backend_roles=, requestedTenant=null]”
}

This error does not show in kibana logs when I try to save in UI

lzukel · February 23, 2021, 8:07pm

No error in kibana UI when trying to save the watch

lzukel · February 23, 2021, 9:33pm

I added the admin user to the role that should be able to manage signals/watches…

sg_signals_manager:
reserved: false
hidden: false
backend_roles:

“kibana-watches-users”
users:

“admin”
description: “Keycloak group that allows users access to watches in kibana”

added the missing permissions to that role…

sg_signals_manager:
reserved: false
hidden: false
description: “Role for kibana users to be able to use watches”
cluster_permissions:

“SGS_SIGNALS_ACCOUNT_MANAGE”

“SGS_CLUSTER_COMPOSITE”

“SGS_SIGNALS_ALL”

“admin:searchguard:tenant:signals:watch/*”
index_permissions:

index_patterns:

“?kibana-*”
dls: null
fls: null
masked_fields: null
allowed_actions:

“READ”

index_patterns:

“*”
dls: null
fls: null
masked_fields: null
allowed_actions:

“indices:data/read/field_caps*”

“indices:data/read/search”
tenant_permissions:

tenant_patterns:

“SGS_GLOBAL_TENANT”
allowed_actions:

“SGS_SIGNALS_ALL”

Now when I run the curl command provided earlier:

curl -k -u admin:pasword -X GET https://host:9200/_signals/watch/_main/Kibana_Settings_Watch
{
“status” : 404,
“error” : “Not found”
}

Still unable to save the watch in Kibana

lzukel · February 24, 2021, 1:35pm

occasionallyI get the following error (even when trying to add sample signal):

{
  "statusCode": 500,
  "error": "Internal Server Error",
  "message": "Response Error",
  "attributes": {
    "body": {
      "status": 500,
      "error": "AuthTokenProvider is not configured"
    }
  }
}

srgbnd · February 24, 2021, 1:35pm

The authtoken error is not related.
By doing -X GET you get the watch, not create it. The API response says there is no watch with name Kibana_Settings_Watch in the _main context.

Put a watch, for example

curl -k -u admin:admin -X PUT https://localhost:9200/_signals/watch/_main/a%20watch -H 'Content-Type: application/json' -d'
{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "checks": [
    {
      "type": "search",
      "request": {
        "indices": ["*"],
        "body": {
          "query": {
            "match_all": {}
          }
        }
      }
    }
  ],
  "actions": []
}'

Now get it

curl -k -u admin:admin -X GET https://localhost:9200/_signals/watch/_main/a%20watch?pretty

Does it work for you?

lzukel · February 24, 2021, 1:44pm

First curl does not return anything. No watch was created.
Second curl does not return anything. See: No watch was created

srgbnd · February 24, 2021, 1:48pm

First curl does not return anything. No watch was created.

Do you see any error in the Elasticsearch or Kibana?

Let’s troubleshoot it in UI too.

Do you see any related error in the dev console log of your browser? Enable the verbose mode and show me all the messages in the console log.

Also show me the network calls.

Copy for me the request headers and responses for each failed call.

lzukel · February 24, 2021, 1:55pm

srgbnd · February 24, 2021, 2:03pm

After what action do you see the Internal Server Error toast? After you clicked an add button on the Watch Examples flyout?

lzukel · February 24, 2021, 2:03pm

Anytime when trying to add/create watch

srgbnd · February 24, 2021, 2:05pm

Weird. I have never seen this. Can you send your sg_config.yml and kibana.yml? Don’t forget to obfuscate the credentials.