How to backup and restore .signals_* indices

Signals Alerting
1

Hi,
I am using ELK 7.8.0 and SG plugin 7.8.0-43.0.0.
I see with this version of SG plugin, .signals_settings, .signals_watches and etc indices will get created in Elasticsearch.
How to take backup of all these indices and restore it back using Elasticsearch snapshot API.

2

The Signals indices, as well as the Search Guard security index, may contain sensitive data, that’s why they are protected. The easiest way to use the ES snapshot API and to backup and restore protected indices is to use the TLS admin certificate that you would also use for sgadmin for example. If your Elasticsearch API call contains the admin cert, you have basically full access to your cluster. Think like the admin cert is a root user.

3

Hi,
Below is my backup and restore flow.

  1. Take backup on ES indices including signals indices with TLS admin certificates.
    curl -X PUT --insecure --cert /etc/elasticsearch/certs/admin.crt.pem --key /etc/elasticsearch/certs/admin.key.pem https://<ES-IP>:9200/_snapshot/es_backup/es-snapshot?wait_for_completion=true -H 'Content-Type: application/json' -d' {"ignore_unavailable": true,"include_global_state": false,"indices": "*,-searchguard"}'

  2. Delete all the indices in ES including .signals indices. As soon as I delete .signals indices the index will get created back automatically with 0 document count.

  3. Restore the ES Snapshot taken in Step 1.
    curl -XPOST --insecure --cert /etc/elasticsearch/certs/admin.crt.pem --key /etc/elasticsearch/certs/admin.key.pem https://<ES-IP>:9200/_snapshot/es_backup/<snapshot_name>/_restore?wait_for_completion=true -H 'Content-Type: application/json' -d' { "indices": "*,-searchguard", "ignore_unavailable": true,"include_global_state": false, "rename_pattern": "(.+)", "rename_replacement": "$1-restored_'$restore_date_format'" }'

    So when the indices get restored, the index name will be appended with <index-name>-restored_epochtime. Hence after restore _cat/indices example looks like below in which the actual .signals indices having 0 document count and the .signals-restored indices containing the actual data.

curl --insecure --cert /etc/elasticsearch/certs/admin.crt.pem --key /etc/elasticsearch/certs/admin.key.pem https://<ES-IP>:9200/_cat/indices?v
green open .signals_settings VaTRGdyrTIGoEACjxQYXkw 1 1 0 0 12.4kb 4.4kb
green open .signals_watches Y2bAFy81RdGBvryNwTQPvw 1 1 0 0 80.6kb 40.3kb
green open .signals_watches-restored_1597646642 wfLkBTRLTvClVq23tQKfRQ 1 1 2 0 41kb 20.5kb
green open .signals_settings-restored_1597646642 MtQCLXNaTdOAl1zVUc8Maw 1 1 2 0 16kb 8kb
green open .signals_accounts-restored_1597646642 ZtE25uujSSa3vguhXWSDIg 1 1 0 0 416b 208b
green open .signals_accounts 5mh7_qoHSgW0knFXcwoOoA 1 1 0 0 416b 208b
green open .signals_watches_state-restored_1597646642 Cf0FSR70T_e0EkfzwL91fQ 1 1 2 0 416b 208b

  1. Since the actual .signals indices are empty, I tried merging the restored signals indices with the actual signal index using Elasticsearch reindex API.
    curl -X POST --insecure --cert /etc/elasticsearch/certs/admin.crt.pem --key /etc/elasticsearch/certs/admin.key.pem https://<ES_IP>:9200/_reindex?wait_for_completion=true -H 'Content-Type: application/json' -d'{ "source": {"index": ".signals_watches_1597646642" }, "dest": {"index": ".signals_watches"}}'

But this command is taking around 5-6 minutes to copy the documents to destination index and also wait_for_completion=true in the reindex API is not waiting for copying documents to destination index.
Note: For other ES indices the command completes within seconds and also the copy to destination index happens properly as soon as the reindex API finishes. But in signals index there are only 2 document count in my case. Why it is taking so long time for reindexing?

After the document copy is done(takes 5-6 min), I can see my old watches configured in Kibana UI.
Do you have any solutions to satisfy my usecase of Backup and Restore? Or is there any guidelines to Backup, restore signals indices?