Howto delete signal indexes


during some work on my ELK cluster, .signals_accounts and .signals_watches indexes lost one of the main and backup shard turning cluster into red state. I tried deleting the shards with admin user, but it doesn’t have the correct permissions.

I found this post: Deleting searchguard/signals indexes and recreating

I’m using digicert certs for HTTPS (port 9200) access and selfsigned for transport API (port 9300). I tried generating a sgadmin cert with sgtlstool (using the same config I do for generating transport api certs) and I set the same DN under searchguard.authcz.admin_dn. When running the command from the ticket I linked earlier, I get an error:

http: error: SSLError: HTTPSConnectionPool(host='es1.domain.tld', port=9200): Max retries exceeded with url: /.signals_settings (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:2548)'))) while doing a DELETE request to URL: https://es1.domain.tld:9200/.signals_settings

Command I ran: http --cert=sgadmin.pem --cert-key=sgadmin.key DELETE "https://es1.domain.tld:9200/.signals_settings"

I can see the following error in elasticsearch:

[2022-11-10T22:05:32,431][ERROR][c.f.s.s.h.n.SearchGuardSSLNettyHttpServerTransport] [es1] Exception during establishing a SSL connection: PKIX path building failed: unable to find valid certification path to requested target PKIX path building failed: unable to find valid certification path to requested target

ES: 7.6.2
SG: 7.6.2-41.2.0

What am I doing wrong?

@brokencode Could you share your elasticsearch.yml file?


here is my config, a little obfuscated: elasticsearch "es1"
node.attr.location: "dc1"
node.attr.tier: "warm"
node.master: true true
node.ingest: false /opt/esdata/
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true xxx
discovery.seed_hosts: ["xxx", "yyy"]
node.max_local_storage_nodes: 1
action.destructive_requires_name: true
indices.recovery.max_bytes_per_sec: 50mb
cluster.routing.allocation.node_concurrent_recoveries: 8
cluster.routing.allocation.node_initial_primaries_recoveries: 10

searchguard.disabled: false

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.pemkey_filepath: certs/es1_transfer.key
searchguard.ssl.transport.pemkey_password: <password>
searchguard.ssl.transport.pemcert_filepath: certs/es1_transfer.crt
searchguard.ssl.transport.pemtrustedcas_filepath: certs/transportca.crt
searchguard.ssl.transport.enforce_hostname_verification: true

searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemkey_filepath: certs/es1_https.key
searchguard.ssl.http.pemkey_password: <password>
searchguard.ssl.http.pemcert_filepath: certs/es1_https.crt
searchguard.ssl.http.pemtrustedcas_filepath: certs/digicertca.crt

    - CN=sgadmin,OU=XXX,O=YYY,L=Ljubljana,C=SI

    - sg_all_access
    - sg_rest_access
    - SGS_ALL_ACCESS          false
xpack.monitoring.enabled:  true    false
xpack.watcher.enabled:     false

@brokencode Do you get the same message when you run the below command?

curl --cert=sgadmin.pem --cert-key=sgadmin.key GET "https://es1.domain.tld:9200/

Does the admin_dn match sgadmin.pem’s subject?

openssl x509 -in sgadmin.pem -subject -nameopt RFC2253 -noout

Was the node certificate es1_https.crt signed with the same root CA as sgadmin.pem?

@pablo running curl gives me similar error:

curl -XGET "https://es1.domain.tld:9200/" -v \
   --key "sgadmin.key" \
   --cert "sgadmin.pem"
Enter PEM pass phrase:
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to

I tripple ckecked admin_dn and it’s a match.

Regarding es1_https.crt, it is not signed with the same root CA as agadmin.pem. es1_https.crt is signed by public CA whereas sgadmin.pem is internally signed, like our transport certs (for port 9300).

Do certs needs to be signed by the same root CA as the HTTPS certs?

@brokencode Could you try with --insecure?

curl --insecure -XGET "https://es1.domain.tld:9200/" -v \
   --key "sgadmin.key" \
   --cert "sgadmin.pem"\

Also, please try with internal user i.e. admin.

curl --insecure -XGET "https://es1.domain.tld:9200/" -v -u admin:admin

What is your curl version?

curl --version

Hello @pablo,

when running without --insecure, I get curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to es1.domain.tld:9200, but when using --insecure, error is curl: (56) LibreSSL SSL_read: error:1404C416:SSL routines:ST_OK:sslv3 alert certificate unknown, errno 0.

Logging in with internal user admin works if I access / endpoint, but I can’t delete the index with this user (I get 403 Forbidden).

curl --insecure -XDELETE "https://es1.example.tld:9200/.signals_settings" -v -u "admin:xxxxx"

{"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [] and User [name=admin, backend_roles=[], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [] and User [name=admin, backend_roles=[], requestedTenant=null]"},"status":403}

Curl version:

$ curl --version
curl 7.79.1 (x86_64-apple-darwin21.0) libcurl/7.79.1 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.11 nghttp2/1.45.1
Release-Date: 2021-09-22
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL UnixSockets

@brokencode The error generated with --insecure means that the admin’s and node’s root CAs don’t match. Node can’t verify the certificate.

You have two solutions.

  1. Generate admin certificate and sign with node’s root CA.
  2. Add admin’s root CA to node’s CA. Just copy the content and paste it to the node’s root CA.
    The file should look like the below example.

Please be aware that both solutions require Elasticsearch restart.

Great, thank you!

I was suspecting that this could be the issue, since I’m using Sectigo certs for HTTPS endpoint and self-signed for transport endpoint.

Will create a new admin cert. sign it with Sectico and add DN to the admin_dn config.

Success. With a new cert signed by root CA of https endpoint did the trick.

Funny enough, I still had to run with --insecure flag. Running without it returned curl: (56) LibreSSL SSL_read: error:1404C416:SSL routines:ST_OK:sslv3 alert certificate unknown, errno 0.