Unable to create snaphot of .kibana* indices

Search Guard
1

Today I’ve discovered that we are currently unable to create a snaphot of the .kibana* indices.

We are using Elasticsearch 8.7.1 with SG FLX 1.3.0-es-8.7.1.

Following API call confirms that our cluster has multiple .kibana indices. Due to privacy concerns I won’t show the index names.

GET "https://elasticsearch:9200/_cat/indices/.kibana*?h=index&expand_wildcards=all" | wc -l 382

We’re using a NFS share as the snapshot repository:

GET _snapshot/elastic_snapshots
{
  "elastic_snapshots" : {
    "type" : "fs",
    "uuid" : "xEgwX2ErSw6NgEK-kd2bXA",
    "settings" : {
      "location" : "/nfs/elastic_snapshots"
    }
  }
}

With following API call I want to create a snaphot of all .kibana* indices:

PUT "https://elasticsearch:9200/_snapshot/elastic_snapshots/kibana-backup?wait_for_completion=true&pretty" -H 'Content-Type: application/json' -d'
{
  "indices": ".kibana*",
  "ignore_unavailable": true,
  "include_global_state": false
}
'

However, only the ‘.kibana-event-log*’ indices will get saved, no hints of the .kibana config and multi-tenant indices.

{
  "snapshot" : {
    "snapshot" : "kibana-backup",
    "uuid" : "ONJTJFiEQI6JofIKoikaFw",
    "repository" : "elastic_snapshots",
    "version_id" : 8070199,
    "version" : "8.7.1",
    "indices" : [
      ".kibana-event-log-8.7.1-000003",
      ".kibana-event-log-7.10.2-000032",
      ".kibana-event-log-8.7.1-000002"
    ],
    "data_streams" : [ ],
    "include_global_state" : false,
    "state" : "SUCCESS",
    "start_time" : "2024-03-21T08:03:42.242Z",
    "start_time_in_millis" : 1711008222242,
    "end_time" : "2024-03-21T08:03:42.242Z",
    "end_time_in_millis" : 1711008222242,
    "duration_in_millis" : 0,
    "failures" : [ ],
    "shards" : {
      "total" : 3,
      "failed" : 0,
      "successful" : 3
    },
    "feature_states" : [ ]
  }

We were creating snapshots of the kibana indices before and I’m pretty sure it was working with ES7.
Unfortunately I cant’ say since what Elastic or SG version this got broken.

This is quite unfortunate since we have no other possibility to easily backup the Kibana data.

Regards,
Alex

2

@trauta Did you upgrade to 8.7.1-1.3.0 or it’s a fresh installation?
If upgraded, what was the previous version and did the snapshot work for .kibana indices?

3

@trauta I did further testing and I’ve found that .kibana indices are backed up when the “include_global_state” is set to true. This is not related to the SG plugin but to the behaviour of Kibana itself.

4

@pablo We’ve upgraded our installation from ES7 to ES8.

I’ve noticed there seems to be major change in how the kibana configuration indices are stored in the snapshots.

Previously, the .kibana indices were stored as separate indices in the snapshot. With ES8 it seems that the Kibana configuration can only be stored via the feature state parameter.

With following snapshot create API call the kibana config get stored, setting include_global_state does not seem to be necessary.

{
  "include_global_state": false,
  "feature_states": [
    "kibana"
  ],
...
}

Restoring the kibana config from this snapshot also works.

I think this topic can be closed, I wasn’t aware of this behavior change.

5

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.