Howto delete signal indexes

[brokencode]

Hello,

during some work on my ELK cluster, .signals_accounts and .signals_watches indexes lost one of the main and backup shard turning cluster into red state. I tried deleting the shards with admin user, but it doesn’t have the correct permissions.

I found this post: Deleting searchguard/signals indexes and recreating

I’m using digicert certs for HTTPS (port 9200) access and selfsigned for transport API (port 9300). I tried generating a sgadmin cert with sgtlstool (using the same config I do for generating transport api certs) and I set the same DN under searchguard.authcz.admin_dn. When running the command from the ticket I linked earlier, I get an error:

http: error: SSLError: HTTPSConnectionPool(host='es1.domain.tld', port=9200): Max retries exceeded with url: /.signals_settings (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:2548)'))) while doing a DELETE request to URL: https://es1.domain.tld:9200/.signals_settings

Command I ran: http --cert=sgadmin.pem --cert-key=sgadmin.key DELETE "https://es1.domain.tld:9200/.signals_settings"

I can see the following error in elasticsearch:

[2022-11-10T22:05:32,431][ERROR][c.f.s.s.h.n.SearchGuardSSLNettyHttpServerTransport] [es1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

ES: 7.6.2
SG: 7.6.2-41.2.0

What am I doing wrong?

[pablo]

@brokencode Could you share your elasticsearch.yml file?

[brokencode]

Hello,

here is my config, a little obfuscated:

cluster.name: elasticsearch
node.name: "es1"
node.attr.location: "dc1"
node.attr.tier: "warm"
node.master: true
node.data: true
node.ingest: false
path.data: /opt/esdata/
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: xxx
discovery.seed_hosts: ["xxx", "yyy"]
node.max_local_storage_nodes: 1
action.destructive_requires_name: true
indices.recovery.max_bytes_per_sec: 50mb
cluster.routing.allocation.node_concurrent_recoveries: 8
cluster.routing.allocation.node_initial_primaries_recoveries: 10


searchguard.disabled: false

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.pemkey_filepath: certs/es1_transfer.key
searchguard.ssl.transport.pemkey_password: <password>
searchguard.ssl.transport.pemcert_filepath: certs/es1_transfer.crt
searchguard.ssl.transport.pemtrustedcas_filepath: certs/transportca.crt
searchguard.ssl.transport.enforce_hostname_verification: true

searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemkey_filepath: certs/es1_https.key
searchguard.ssl.http.pemkey_password: <password>
searchguard.ssl.http.pemcert_filepath: certs/es1_https.crt
searchguard.ssl.http.pemtrustedcas_filepath: certs/digicertca.crt

searchguard.authcz.admin_dn:
    - CN=sgadmin,OU=XXX,O=YYY,L=Ljubljana,C=SI

searchguard.restapi.roles_enabled:
    - sg_all_access
    - sg_rest_access
    - SGS_ALL_ACCESS


xpack.ml.enabled:          false
xpack.monitoring.enabled:  true
xpack.security.enabled:    false
xpack.watcher.enabled:     false

[pablo]

@brokencode Do you get the same message when you run the below command?

curl --cert=sgadmin.pem --cert-key=sgadmin.key GET "https://es1.domain.tld:9200/

Does the admin_dn match sgadmin.pem’s subject?

openssl x509 -in sgadmin.pem -subject -nameopt RFC2253 -noout

Was the node certificate es1_https.crt signed with the same root CA as sgadmin.pem?

[brokencode]

@pablo running curl gives me similar error:

curl -XGET "https://es1.domain.tld:9200/" -v \
   --key "sgadmin.key" \
   --cert "sgadmin.pem"
Enter PEM pass phrase:
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to es1.arnes.si:9200

I tripple ckecked admin_dn and it’s a match.

Regarding es1_https.crt, it is not signed with the same root CA as agadmin.pem. es1_https.crt is signed by public CA whereas sgadmin.pem is internally signed, like our transport certs (for port 9300).

Do certs needs to be signed by the same root CA as the HTTPS certs?

[pablo]

@brokencode Could you try with --insecure?

curl --insecure -XGET "https://es1.domain.tld:9200/" -v \
   --key "sgadmin.key" \
   --cert "sgadmin.pem"\

Also, please try with internal user i.e. admin.

curl --insecure -XGET "https://es1.domain.tld:9200/" -v -u admin:admin

What is your curl version?

curl --version

[brokencode]

Hello @pablo,

when running without --insecure, I get curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to es1.domain.tld:9200, but when using --insecure, error is curl: (56) LibreSSL SSL_read: error:1404C416:SSL routines:ST_OK:sslv3 alert certificate unknown, errno 0.

Logging in with internal user admin works if I access / endpoint, but I can’t delete the index with this user (I get 403 Forbidden).

curl --insecure -XDELETE "https://es1.example.tld:9200/.signals_settings" -v -u "admin:xxxxx"

{"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [] and User [name=admin, backend_roles=[], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [] and User [name=admin, backend_roles=[], requestedTenant=null]"},"status":403}

Curl version:

$ curl --version
curl 7.79.1 (x86_64-apple-darwin21.0) libcurl/7.79.1 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.11 nghttp2/1.45.1
Release-Date: 2021-09-22
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL UnixSockets

[pablo]

@brokencode The error generated with --insecure means that the admin’s and node’s root CAs don’t match. Node can’t verify the certificate.

You have two solutions.

1. Generate admin certificate and sign with node’s root CA.
2. Add admin’s root CA to node’s CA. Just copy the content and paste it to the node’s root CA.
   The file should look like the below example.

-----BEGIN CERTIFICATE-----
MIIDyDCCArCgAwIBAgIBATANBgkqhkiG9w0BAQsFADB1MRMwEQYKCZImiZPyLGQB
GRYDY29tMRcwFQYKCZImiZPyLGQBGRYHZXhhbXBsZTEaMBgGA1UECgwRRXhhbXBs
ZSBDb20sIEluYy4xCzAJBgNVBAsMAkNBMRwwGgYDVQQDDBNyb290LmNhLmV4YW1w
bGUuY29tMB4XDTIyMTEwNDE4MDMwM1oXDTMyMTEwMTE4MDMwM1owdTETMBEGCgmS
JomT8ixkARkWA2NvbTEXMBUGCgmSJomT8ixkARkWB2V4YW1wbGUxGjAYBgNVBAoM
EUV4YW1wbGUgQ29tLCBJbmMuMQswCQYDVQQLDAJDQTEcMBoGA1UEAwwTcm9vdC5j
YS5leGFtcGxlLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALuP
gpL2bZ4VCq4TppoKxw59Sk1Tc60U7+SjrvikZFBa+xb+B0cDvanBEEA6TWGdVF7t
..........
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDyDCCArCgAwIBAgIBATANBgkqhkiG9w0BAQsFADB1MRMwEQYKCZImiZPyLGQB
GRYDY29tMRcwFQYKCZImiZPyLGQBGRYHZXhhbXBsZTEaMBgGA1UECgwRRXhhbXBs
ZSBDb20sIEluYy4xCzAJBgNVBAsMAkNBMRwwGgYDVQQDDBNyb290LmNhLmV4YW1w
bGUuY29tMB4XDTIyMTExNjE2NTAxNloXDTMyMTExMzE2NTAxNlowdTETMBEGCgmS
JomT8ixkARkWA2NvbTEXMBUGCgmSJomT8ixkARkWB2V4YW1wbGUxGjAYBgNVBAoM
EUV4YW1wbGUgQ29tLCBJbmMuMQswCQYDVQQLDAJDQTEcMBoGA1UEAwwTcm9vdC5j
YS5leGFtcGxlLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKaz
ZC6JWI+EYy9o0m6E+nFy+0rx0e7ucVMlVYv4y4Lf7NQle9+h/9FxqvamrrtN9gws
........
-----END CERTIFICATE-----

Please be aware that both solutions require Elasticsearch restart.

[brokencode]

Great, thank you!

I was suspecting that this could be the issue, since I’m using Sectigo certs for HTTPS endpoint and self-signed for transport endpoint.

Will create a new admin cert. sign it with Sectico and add DN to the admin_dn config.

[brokencode]

Success. With a new cert signed by root CA of https endpoint did the trick.

Funny enough, I still had to run with --insecure flag. Running without it returned curl: (56) LibreSSL SSL_read: error:1404C416:SSL routines:ST_OK:sslv3 alert certificate unknown, errno 0.