# Howto delete signal indexes **URL:** \ **Category:** Search Guard\ **Created:** [November 10, 2022, 9:09pm UTC](https://forum.search-guard.com/t/howto-delete-signal-indexes/2372 "2022-11-10T21:09:14Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![brokencode](https://avatars.discourse-cdn.com/v4/letter/b/9de0a6/32.png) [@brokencode](https://forum.search-guard.com/u/brokencode)\ **Post date:** [November 10, 2022, 9:09pm UTC](https://forum.search-guard.com/t/howto-delete-signal-indexes/2372/1 "2022-11-10T21:09:15Z")
Hello, during some work on my ELK cluster, `.signals_accounts` and `.signals_watches` indexes lost one of the main and backup shard turning cluster into red state. I tried deleting the shards with admin user, but it doesn’t have the correct permissions. I found this post: [Deleting searchguard/signals indexes and recreating](https://forum.search-guard.com/t/deleting-searchguard-signals-indexes-and-recreating/2176) I’m using digicert certs for HTTPS (port 9200) access and selfsigned for transport API (port 9300). I tried generating a `sgadmin` cert with sgtlstool (using the same config I do for generating transport api certs) and I set the same DN under `searchguard.authcz.admin_dn`. When running the command from the ticket I linked earlier, I get an error: `http: error: SSLError: HTTPSConnectionPool(host='es1.domain.tld', port=9200): Max retries exceeded with url: /.signals_settings (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:2548)'))) while doing a DELETE request to URL: https://es1.domain.tld:9200/.signals_settings` Command I ran: `http --cert=sgadmin.pem --cert-key=sgadmin.key DELETE "https://es1.domain.tld:9200/.signals_settings"` I can see the following error in elasticsearch: ```auto [2022-11-10T22:05:32,431][ERROR][c.f.s.s.h.n.SearchGuardSSLNettyHttpServerTransport] [es1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target ``` ES: 7.6.2 SG: 7.6.2-41.2.0 What am I doing wrong? ---
**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/pablo/32/1443_2.png) [@pablo](https://forum.search-guard.com/u/pablo)\ **Post date:** [November 11, 2022, 5:22pm UTC](https://forum.search-guard.com/t/howto-delete-signal-indexes/2372/2 "2022-11-11T17:22:25Z")
@brokencode Could you share your elasticsearch.yml file? ---
**Author:** ![brokencode](https://avatars.discourse-cdn.com/v4/letter/b/9de0a6/32.png) [@brokencode](https://forum.search-guard.com/u/brokencode)\ **Post date:** [November 12, 2022, 7:17am UTC](https://forum.search-guard.com/t/howto-delete-signal-indexes/2372/3 "2022-11-12T07:17:06Z")
Hello, here is my config, a little obfuscated: ```auto cluster.name: elasticsearch node.name: "es1" node.attr.location: "dc1" node.attr.tier: "warm" node.master: true node.data: true node.ingest: false path.data: /opt/esdata/ path.logs: /var/log/elasticsearch bootstrap.memory_lock: true network.host: xxx discovery.seed_hosts: ["xxx", "yyy"] node.max_local_storage_nodes: 1 action.destructive_requires_name: true indices.recovery.max_bytes_per_sec: 50mb cluster.routing.allocation.node_concurrent_recoveries: 8 cluster.routing.allocation.node_initial_primaries_recoveries: 10 searchguard.disabled: false searchguard.ssl.transport.enabled: true searchguard.ssl.transport.pemkey_filepath: certs/es1_transfer.key searchguard.ssl.transport.pemkey_password: searchguard.ssl.transport.pemcert_filepath: certs/es1_transfer.crt searchguard.ssl.transport.pemtrustedcas_filepath: certs/transportca.crt searchguard.ssl.transport.enforce_hostname_verification: true searchguard.ssl.http.enabled: true searchguard.ssl.http.pemkey_filepath: certs/es1_https.key searchguard.ssl.http.pemkey_password: searchguard.ssl.http.pemcert_filepath: certs/es1_https.crt searchguard.ssl.http.pemtrustedcas_filepath: certs/digicertca.crt searchguard.authcz.admin_dn: - CN=sgadmin,OU=XXX,O=YYY,L=Ljubljana,C=SI searchguard.restapi.roles_enabled: - sg_all_access - sg_rest_access - SGS_ALL_ACCESS xpack.ml.enabled: false xpack.monitoring.enabled: true xpack.security.enabled: false xpack.watcher.enabled: false ``` ---
**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/pablo/32/1443_2.png) [@pablo](https://forum.search-guard.com/u/pablo)\ **Post date:** [November 14, 2022, 2:26pm UTC](https://forum.search-guard.com/t/howto-delete-signal-indexes/2372/4 "2022-11-14T14:26:13Z")
@brokencode Do you get the same message when you run the below command? ```auto curl --cert=sgadmin.pem --cert-key=sgadmin.key GET "https://es1.domain.tld:9200/ ``` Does the `admin_dn` match sgadmin.pem’s subject? ```auto openssl x509 -in sgadmin.pem -subject -nameopt RFC2253 -noout ``` Was the node certificate es1\_https.crt signed with the same root CA as sgadmin.pem? ---
**Author:** ![brokencode](https://avatars.discourse-cdn.com/v4/letter/b/9de0a6/32.png) [@brokencode](https://forum.search-guard.com/u/brokencode)\ **Post date:** [November 15, 2022, 10:22am UTC](https://forum.search-guard.com/t/howto-delete-signal-indexes/2372/5 "2022-11-15T10:22:32Z")
@pablo running curl gives me similar error: ```auto curl -XGET "https://es1.domain.tld:9200/" -v \ --key "sgadmin.key" \ --cert "sgadmin.pem" Enter PEM pass phrase: curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to es1.arnes.si:9200 ``` I tripple ckecked `admin_dn` and it’s a match. Regarding es1\_https.crt, it is not signed with the same root CA as agadmin.pem. es1\_https.crt is signed by public CA whereas `sgadmin.pem` is internally signed, like our transport certs (for port 9300). Do certs needs to be signed by the same root CA as the HTTPS certs? ---
**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/pablo/32/1443_2.png) [@pablo](https://forum.search-guard.com/u/pablo)\ **Post date:** [November 16, 2022, 1:31pm UTC](https://forum.search-guard.com/t/howto-delete-signal-indexes/2372/6 "2022-11-16T13:31:06Z")
@brokencode Could you try with --insecure? ```auto curl --insecure -XGET "https://es1.domain.tld:9200/" -v \ --key "sgadmin.key" \ --cert "sgadmin.pem"\ ``` Also, please try with internal user i.e. admin. ```auto curl --insecure -XGET "https://es1.domain.tld:9200/" -v -u admin:admin ``` What is your curl version? ```auto curl --version ``` ---
**Author:** ![brokencode](https://avatars.discourse-cdn.com/v4/letter/b/9de0a6/32.png) [@brokencode](https://forum.search-guard.com/u/brokencode)\ **Post date:** [November 16, 2022, 4:38pm UTC](https://forum.search-guard.com/t/howto-delete-signal-indexes/2372/7 "2022-11-16T16:38:57Z")
Hello @pablo, when running without --insecure, I get `curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to es1.domain.tld:9200`, but when using --insecure, error is `curl: (56) LibreSSL SSL_read: error:1404C416:SSL routines:ST_OK:sslv3 alert certificate unknown, errno 0`. Logging in with internal user `admin` works if I access `/` endpoint, but I can’t delete the index with this user (I get 403 Forbidden). ```auto curl --insecure -XDELETE "https://es1.example.tld:9200/.signals_settings" -v -u "admin:xxxxx" {"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [] and User [name=admin, backend_roles=[], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [] and User [name=admin, backend_roles=[], requestedTenant=null]"},"status":403} ``` Curl version: ```bash $ curl --version curl 7.79.1 (x86_64-apple-darwin21.0) libcurl/7.79.1 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.11 nghttp2/1.45.1 Release-Date: 2021-09-22 Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL UnixSockets ``` ---
**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/pablo/32/1443_2.png) [@pablo](https://forum.search-guard.com/u/pablo)\ **Post date:** [November 16, 2022, 5:59pm UTC](https://forum.search-guard.com/t/howto-delete-signal-indexes/2372/8 "2022-11-16T17:59:12Z")
@brokencode The error generated with --insecure means that the admin’s and node’s root CAs don’t match. Node can’t verify the certificate. You have two solutions. 1. Generate admin certificate and sign with node’s root CA. 2. Add admin’s root CA to node’s CA. Just copy the content and paste it to the node’s root CA. The file should look like the below example. ```auto -----BEGIN CERTIFICATE----- MIIDyDCCArCgAwIBAgIBATANBgkqhkiG9w0BAQsFADB1MRMwEQYKCZImiZPyLGQB GRYDY29tMRcwFQYKCZImiZPyLGQBGRYHZXhhbXBsZTEaMBgGA1UECgwRRXhhbXBs ZSBDb20sIEluYy4xCzAJBgNVBAsMAkNBMRwwGgYDVQQDDBNyb290LmNhLmV4YW1w bGUuY29tMB4XDTIyMTEwNDE4MDMwM1oXDTMyMTEwMTE4MDMwM1owdTETMBEGCgmS JomT8ixkARkWA2NvbTEXMBUGCgmSJomT8ixkARkWB2V4YW1wbGUxGjAYBgNVBAoM EUV4YW1wbGUgQ29tLCBJbmMuMQswCQYDVQQLDAJDQTEcMBoGA1UEAwwTcm9vdC5j YS5leGFtcGxlLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALuP gpL2bZ4VCq4TppoKxw59Sk1Tc60U7+SjrvikZFBa+xb+B0cDvanBEEA6TWGdVF7t .......... -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- MIIDyDCCArCgAwIBAgIBATANBgkqhkiG9w0BAQsFADB1MRMwEQYKCZImiZPyLGQB GRYDY29tMRcwFQYKCZImiZPyLGQBGRYHZXhhbXBsZTEaMBgGA1UECgwRRXhhbXBs ZSBDb20sIEluYy4xCzAJBgNVBAsMAkNBMRwwGgYDVQQDDBNyb290LmNhLmV4YW1w bGUuY29tMB4XDTIyMTExNjE2NTAxNloXDTMyMTExMzE2NTAxNlowdTETMBEGCgmS JomT8ixkARkWA2NvbTEXMBUGCgmSJomT8ixkARkWB2V4YW1wbGUxGjAYBgNVBAoM EUV4YW1wbGUgQ29tLCBJbmMuMQswCQYDVQQLDAJDQTEcMBoGA1UEAwwTcm9vdC5j YS5leGFtcGxlLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKaz ZC6JWI+EYy9o0m6E+nFy+0rx0e7ucVMlVYv4y4Lf7NQle9+h/9FxqvamrrtN9gws ........ -----END CERTIFICATE----- ``` Please be aware that both solutions require Elasticsearch restart. ---
**Author:** ![brokencode](https://avatars.discourse-cdn.com/v4/letter/b/9de0a6/32.png) [@brokencode](https://forum.search-guard.com/u/brokencode)\ **Post date:** [November 16, 2022, 6:03pm UTC](https://forum.search-guard.com/t/howto-delete-signal-indexes/2372/9 "2022-11-16T18:03:22Z")
Great, thank you! I was suspecting that this could be the issue, since I’m using Sectigo certs for HTTPS endpoint and self-signed for transport endpoint. Will create a new admin cert. sign it with Sectico and add DN to the admin\_dn config. ---
**Author:** ![brokencode](https://avatars.discourse-cdn.com/v4/letter/b/9de0a6/32.png) [@brokencode](https://forum.search-guard.com/u/brokencode)\ **Post date:** [November 16, 2022, 6:42pm UTC](https://forum.search-guard.com/t/howto-delete-signal-indexes/2372/10 "2022-11-16T18:42:06Z")
Success. With a new cert signed by root CA of https endpoint did the trick. Funny enough, I still had to run with `--insecure` flag. Running without it returned `curl: (56) LibreSSL SSL_read: error:1404C416:SSL routines:ST_OK:sslv3 alert certificate unknown, errno 0`. ---
**Author:** ![system](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/system/32/1870_2.png) [@system](https://forum.search-guard.com/u/system)\ **Post date:** [December 7, 2022, 6:42pm UTC](https://forum.search-guard.com/t/howto-delete-signal-indexes/2372/11 "2022-12-07T18:42:54Z")
This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.