Unable to save watches

[lzukel]

Elasticsearch version:
7.10.2

Kibana version (if relevant):
7.10.2

Describe the issue:
When creating a watch, I am unable to save the watch.
Kibana returning:

{“type”:“response”,“@timestamp”:“2021-02-23T15:11:02Z”,“tags”:,“pid”:16,“method”:“get”,“statusCode”:404,“req”:{“url”:“/api/searchguard-signals/watch/Kibana%20Settings%20Watch”,“method”:“get”,“headers”:{“host”:“edited”,“sec-ch-ua”:“"Chromium";v="88", "Google Chrome";v="88", ";Not A Brand";v="99"”,“sec-ch-ua-mobile”:“?0”,“user-agent”:“Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36”,“kbn-version”:“7.10.2”,“content-type”:“application/json”,“accept”:“/”,“sec-fetch-site”:“same-origin”,“sec-fetch-mode”:“cors”,“sec-fetch-dest”:“empty”,“referer”:“https://edited/kibana/app/searchguard-signals",“accept-encoding”:"gzip, deflate, br”,“accept-language”:“en-US,en;q=0.9,fr;q=0.8”,“x-opaque-id”:“mine”,“x-forwarded-for”:“142.136.146.158, 44.128.168.125”,“x-forwarded-host”:“edited”,“x-forwarded-proto”:“https”,“connection”:“close”},“remoteAddress”:“44.128.168.1”,“userAgent”:“Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36”,“referer”:“https://edited/kibana/app/searchguard-signals"},“res”:{“statusCode”:404,“responseTime”:98,“contentLength”:9},“message”:"GET /api/searchguard-signals/watch/Kibana%20Settings%20Watch 404 98ms - 9.0B”}

[srgbnd]

Do you see any error in the Elasticsearch log? What do you see in UI when you try to save the watch, any error? What do you see in the browser console log (enable the debug mode)?

[lzukel]

Browser console

Failed to load resource: the server responded with a status of 404 (Not Found)

Nothing that appears relevant in the Elasticsearch logs
The UI does not change when trying to save the watch

[srgbnd]

Hm. I can’t reproduce it. Do you use Search Guard v49.0.0?
Maybe it is the watch name encoding issue. Try to save a watch with name Kibana_Settings_Watch.

[lzukel]

search-guard-7	7.10.2-49.0.0

Updated the title to use underscores instead of spaces… Still get 404 in kibana logs

[srgbnd]

Get the watch for me via the REST API.

curl -k -u admin:admin -X GET https://localhost:9200/_signals/watch/tenant_name/Kibana%20Settings%20Watch

Use your own credentials instead of admin:admin. Also, you need to specify your tenant name instead of tenant_name. If you have the Multitenancy disabled, use _main.

Also, paste the kibana.yml.

[lzukel]

How to use own credentials, when users are required to use SSO via keycloak?

[srgbnd]

I think you still have the admin. Because you need it to update the Search Guard config. Try admin.

[srgbnd]

Can you tell me on which step the error arise? For example

1. Click Create Watch
2. Put values. What values?
3. Click Save
4. ERROR

Can you attach the screenshot?

[lzukel]

curl -k -u admin:password -X GET https://host:9200/_signals/watch/_main/Kibana%20Settings%20Watch
{
“status” : 403,
“error” : “no permissions for [cluster:admin:searchguard:tenant:signals:watch/get] and User [name=admin, backend_roles=, requestedTenant=null]”
}

This error does not show in kibana logs when I try to save in UI

[lzukel]

No error in kibana UI when trying to save the watch

[lzukel]

I added the admin user to the role that should be able to manage signals/watches…

sg_signals_manager:
reserved: false
hidden: false
backend_roles:

• “kibana-watches-users”
  users:
• “admin”
  description: “Keycloak group that allows users access to watches in kibana”

added the missing permissions to that role…

sg_signals_manager:
reserved: false
hidden: false
description: “Role for kibana users to be able to use watches”
cluster_permissions:

• “SGS_SIGNALS_ACCOUNT_MANAGE”
• “SGS_CLUSTER_COMPOSITE”
• “SGS_SIGNALS_ALL”
• “admin:searchguard:tenant:signals:watch/*”
  index_permissions:
• index_patterns:
  • “?kibana-*”
    dls: null
    fls: null
    masked_fields: null
    allowed_actions:
  • “READ”
• index_patterns:
  • “*”
    dls: null
    fls: null
    masked_fields: null
    allowed_actions:
  • “indices:data/read/field_caps*”
  • “indices:data/read/search”
    tenant_permissions:
• tenant_patterns:
  • “SGS_GLOBAL_TENANT”
    allowed_actions:
  • “SGS_SIGNALS_ALL”

Now when I run the curl command provided earlier:

curl -k -u admin:pasword -X GET https://host:9200/_signals/watch/_main/Kibana_Settings_Watch
{
“status” : 404,
“error” : “Not found”
}

Still unable to save the watch in Kibana

[lzukel]

occasionallyI get the following error (even when trying to add sample signal):

{
  "statusCode": 500,
  "error": "Internal Server Error",
  "message": "Response Error",
  "attributes": {
    "body": {
      "status": 500,
      "error": "AuthTokenProvider is not configured"
    }
  }
}

[srgbnd]

The authtoken error is not related.
By doing -X GET you get the watch, not create it. The API response says there is no watch with name Kibana_Settings_Watch in the _main context.

Put a watch, for example

curl -k -u admin:admin -X PUT https://localhost:9200/_signals/watch/_main/a%20watch -H 'Content-Type: application/json' -d'
{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "checks": [
    {
      "type": "search",
      "request": {
        "indices": ["*"],
        "body": {
          "query": {
            "match_all": {}
          }
        }
      }
    }
  ],
  "actions": []
}'

Now get it

curl -k -u admin:admin -X GET https://localhost:9200/_signals/watch/_main/a%20watch?pretty

Does it work for you?

[lzukel]

First curl does not return anything. No watch was created.
Second curl does not return anything. See: No watch was created

[srgbnd]

First curl does not return anything. No watch was created.

Do you see any error in the Elasticsearch or Kibana?

@lzukel

No error in kibana UI when trying to save the watch

Let’s troubleshoot it in UI too.

Do you see any related error in the dev console log of your browser? Enable the verbose mode and show me all the messages in the console log.

Also show me the network calls.

Copy for me the request headers and responses for each failed call.

[lzukel]

[srgbnd]

After what action do you see the Internal Server Error toast? After you clicked an add button on the Watch Examples flyout?

[lzukel]

Anytime when trying to add/create watch

[srgbnd]

Weird. I have never seen this. Can you send your sg_config.yml and kibana.yml? Don’t forget to obfuscate the credentials.