SearchGuard + Curator + cert auth = Unauthorized

Hi,

I’m having some issue with curator and cert auth as described here https://docs.search-guard.com/latest/elasticsearch-curator-search-guard
Please note, that those certs work perfectly fine using curl, this is really specific to curator.

Elasticsearch version: 7.8.1
Server OS version: CentOS 7.8
Describe the issue: Curator does not seem to work with SG and cert auth
Provide configuration:
elasticsearch/plugins/search-guard-7/sgconfig/sg_config.yml

_sg_meta:
  type: "config"
  config_version: 2

sg_config:
  dynamic:
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
    authc:
      basic:
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: intern
      clientcert_auth_domain:
        http_enabled: true
        order: 2
        http_authenticator:
          type: clientcert
          config:
            username_attribute: cn
          challenge: false
        authentication_backend:
          type: noop

curator.yml

client:
  hosts:
    - 127.0.0.1
  port: 9200
  use_ssl: true
  certificate: /etc/elasticsearch/certs/root-ca.pem
  client_cert: /etc/elasticsearch/certs/admin.pem
  client_key: /etc/elasticsearch/certs/admin.key
  ssl_no_validate: true
  timeout: 30
  master_only: false
logging:
  loglevel: INFO
  logformat: default

Provide logs:
CLI:

/usr/local/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py:190: UserWarning: Connecting to jarvis.lcsb.uni.lu using SSL with verify_certs=False is insecure.
  % host
/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/curator/utils.py", line 904, in get_client
    check_version(client)
  File "/usr/local/lib/python3.6/site-packages/curator/utils.py", line 690, in check_version
    version_number = get_version(client)
  File "/usr/local/lib/python3.6/site-packages/curator/utils.py", line 663, in get_version
    version = client.info()['version']['number']
  File "/usr/local/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/elasticsearch/client/__init__.py", line 259, in info
    return self.transport.perform_request("GET", "/", params=params)
  File "/usr/local/lib/python3.6/site-packages/elasticsearch/transport.py", line 353, in perform_request
    timeout=timeout,
  File "/usr/local/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 251, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python3.6/site-packages/elasticsearch/connection/base.py", line 178, in _raise_error
    status_code, error_message, additional_info
elasticsearch.exceptions.AuthenticationException: AuthenticationException(401, 'Unauthorized')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/curator", line 11, in <module>
    load_entry_point('elasticsearch-curator==5.7.6', 'console_scripts', 'curator')()
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/curator/cli.py", line 218, in cli
    run(config, action_file, dry_run)
  File "/usr/local/lib/python3.6/site-packages/curator/cli.py", line 165, in run
    client = get_client(**client_args)
  File "/usr/local/lib/python3.6/site-packages/curator/utils.py", line 911, in get_client
    'Error: {0}'.format(e)
elasticsearch.exceptions.ElasticsearchException: Unable to create client connection to Elasticsearch.  Error: AuthenticationException(401, 'Unauthorized')

Hi.
Show your elasticsearch.yml.
Do you see any related error in the Elasticsearch logs? If yes, show the logs.

Hi,

No related logs (not even any auth errors, surprisingly) , but here is the file you asked for:

elasticsearch.yml:

cluster.name: jarvis-cluster
network.host: 0.0.0.0
http.compression: true
bootstrap.memory_lock: false
action.auto_create_index: true
path.repo:
  - /mnt/snapshots/

xpack:
  security.enabled: false
  watcher.enabled: false
  ml.enabled: false
  graph.enabled: false

searchguard:
  enterprise_modules_enabled: true
  compliance:
    immutable_indices:
      - '*beat*'
  ssl:
    transport:
      enforce_hostname_verification: false
      resolve_hostname: false
      pemcert_filepath: /usr/share/elasticsearch/config/certs/${node.name}.pem
      pemkey_filepath: /usr/share/elasticsearch/config/certs/${node.name}.key
      pemtrustedcas_filepath: /usr/share/elasticsearch/config/certs/root-ca.pem
    http:
      enabled: true
      pemcert_filepath: /usr/share/elasticsearch/config/certs/${node.name}_http.pem
      pemkey_filepath: /usr/share/elasticsearch/config/certs/${node.name}_http.key
      pemtrustedcas_filepath: /usr/share/elasticsearch/config/certs/root-ca.pem
      clientauth_mode: OPTIONAL
  nodes_dn:
    - CN=es*,OU=Biocore,O=LCSB,DC=uni.lu
  authcz.admin_dn:
    - CN=admin.jarvis,OU=Biocore,O=LCSB,DC=uni.lu
  restapi.roles_enabled:
    - SGS_ALL_ACCESS

Hi. I tried to use the curator and couldn’t reproduce the authentication issue you have.

Please note, that those certs work perfectly fine using curl, this is really specific to curator.

Are you sure you use the valid TLS certificates? If you used curl -k, this made curl proceed even if the server connection otherwise considered insecure.

Look below at the steps I did to reproduce the issue.

  1. Install curator.
sudo pip3 install elasticsearch-curator
  1. Create TLS certificates.
  wget https://maven.search-guard.com/search-guard-tlstool/1.8/search-guard-tlstool-$TLS_TOOL_VERSION.zip
  unzip search-guard-tlstool-$TLS_TOOL_VERSION.zip -d search-guard-tlstool
  cd search-guard-tlstool
  ./tools/sgtlstool.sh -c config/example.yml -ca -crt
  1. Configure Elasticsearch nodes to use the TLS certificates.
  2. Enable clientcert_auth_domain in sg_config.yml,
    https://git.floragunn.com/search-guard/search-guard-labs/-/commit/9b5176d86a5c17ef710603cc22e624f96c547adf
  3. Run Elasticsearch.
  4. Create the curator config.
    curator_config.yml
client:
  hosts:
    - 127.0.0.1
  port: 9200
  use_ssl: true
  certificate: /Users/user/Development/kibana/search-guard-labs/search-guard-tlstool/out/root-ca.pem
  client_cert: /Users/user/Development/kibana/search-guard-labs/search-guard-tlstool/out/kirk.pem
  client_key: /Users/user/Development/kibana/search-guard-labs/search-guard-tlstool/out/kirk.key
  ssl_no_validate: true
  timeout: 30
  master_only: false
logging:
  loglevel: INFO
  logformat: default
  1. List indices using the curator.
$ curator_cli --config curator_config.yml show_indices
2020-09-01 11:08:37,186 INFO      Instantiating client object
/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_requests.py:133: UserWarning: Connecting to https://127.0.0.1:9200 using SSL with verify_certs=False is insecure.
  % self.host
2020-09-01 11:08:37,187 INFO      Testing client connectivity
Enter PEM pass phrase:

/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
2020-09-01 11:09:13,837 INFO      Successfully created Elasticsearch client object with provided settings
...
.apm-agent-configuration
.apm-custom-link
.kibana_1
.kibana_task_manager_1
.signals_accounts
.signals_settings
.signals_watches
.signals_watches_state
searchguard

You can use this docker repo to try it in my lab environment https://git.floragunn.com/search-guard/search-guard-labs. The branch name is forum_1941_curator. Read README before starting the lab https://git.floragunn.com/search-guard/search-guard-labs/-/blob/master/README.md I set the proper config in the branch already, do ./create_config.sh before starting the docker.

Hi @srgbnd,

Thank you, I’m gonna try again with your instructions.

I however don’t understand your first comment:

Are you sure you use the valid TLS certificates? If you used curl -k , this made curl proceed even if the server connection otherwise considered insecure.

How is this related to authentication? curl -k should be roughly the equivalent of curators ssl_no_validate: true, which makes sense for a localhost hostname. I don’t see how that relates to admin certificates

Maybe rephrasing my question:

Why does this work?

curl -k --key /etc/elasticsearch/certs/admin.key --cert /etc/elasticsearch/certs/admin.pem 'https://localhost:9200/...'

But this does not?

  hosts:
    - 127.0.0.1
  port: 9200
  use_ssl: true
  certificate: /etc/elasticsearch/certs/root-ca.pem
  client_cert: /etc/elasticsearch/certs/admin.pem
  client_key: /etc/elasticsearch/certs/admin.key
  ssl_no_validate: true
  timeout: 30
  master_only: false

Hi.

Why does this work?

curl -k --key /etc/elasticsearch/certs/admin.key --cert /etc/elasticsearch/certs/admin.pem > 'https://localhost:9200/...'

This way, curl (client) doesn’t verify the Elasticsearch node (server) certificate. It is like visiting a WEB site where a self-signed certificate was installed, your browser warns you that the connection is insecure. But you can ignore the warning and navigate to the site by accepting the self-signed certificate. This is what the -k (--insecure) option does for the curl requests.

But this does not?

certificate: /etc/elasticsearch/certs/root-ca.pem
client_cert: /etc/elasticsearch/certs/admin.pem
client_key: /etc/elasticsearch/certs/admin.key

Here you use the root CA (certificate) to verify the Elasticsearch node (server) TLS certificate. Probably it fails because the server certificate was not signed by a common valid root CA or intermediate certificate.

Look, a curl request without -k and root CA certificate, only the client key and certificate. Do you see the error? SSL certificate problem: unable to get local issuer certificate. Because no proper root CA was specified.

$ curl -v \
    --cert /Users/user/Development/kibana/search-guard-labs/search-guard-tlstool/out/kirk.pem \
    --key /Users/user/Development/kibana/search-guard-labs/search-guard-tlstool/out/kirk.key \
    https://node1.example.com:9200/_cat/indices

* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to node1.example.com (127.0.0.1) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
Enter PEM pass phrase:
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Now, let’s add -k. See, curl accepts the server certificate and proceeds with the connection. It doesn’t validate the server certificate SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway..

$ curl -v \
    --cert /Users/user/Development/kibana/search-guard-labs/search-guard-tlstool/out/kirk.pem \
    --key /Users/user/Development/kibana/search-guard-labs/search-guard-tlstool/out/kirk.key \
    https://node1.example.com:9200/_cat/indices

* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to node1.example.com (127.0.0.1) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
Enter PEM pass phrase:
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: DC=com; DC=example; O=Example Com, Inc.; OU=Ops; CN=node1.example.com
*  start date: Sep  1 08:40:29 2020 GMT
*  expire date: Aug 30 08:40:29 2030 GMT
*  issuer: DC=com; DC=example; O=Example Com, Inc.; OU=CA; CN=signing.ca.example.com
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET /_cat/indices HTTP/1.1
> Host: node1.example.com:9200
> User-Agent: curl/7.64.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< content-type: text/plain; charset=UTF-8
< content-length: 830
< 
green open searchguard              Msm3W5d-Qbi7r1l_ll78lA 1 2  8 0  87.8kb 39.6kb
green open .signals_settings        Jvmw75LsThiowN7LCAeIyQ 1 1  2 0  12.3kb  7.9kb
green open .signals_watches         L5W4bsgVSMWy26PkV1W9aw 1 1  0 0    416b   208b
green open .apm-custom-link         uCbzNJDIRAiHpEWL_SZWJA 1 1  0 0    416b   208b
green open sg7-auditlog-2020.09.02  d_tlkwMBROi06DDgG8b86A 1 1 32 0 160.1kb 86.5kb
green open .kibana_task_manager_1   Ha16g34_Ts69EIzFmFWPvQ 1 1  5 0 120.8kb 55.6kb
green open .apm-agent-configuration CCKUzn1GTpyHA8LfRYmBHw 1 1  0 0    416b   208b
green open .signals_accounts        MEe2xVt-R_KlWLnbdu2SVw 1 1  0 0    416b   208b
green open .signals_watches_state   FnC1YPFBQrKTw0ktOf5TtA 1 1  0 0    416b   208b
green open .kibana_1                bFgWve3sSwGe2Y0vrkGjvw 1 1  3 0  46.4kb 23.2kb
* Connection #0 to host node1.example.com left intact
* Closing connection 0

Now, let’s use the proper root CA certificate and remove -k option. See, the connection was established successfully. Notice the message SSL certificate verify ok..

$curl -v \
    --cacert /Users/user/Development/kibana/search-guard-labs/search-guard-tlstool/out/root-ca.pem \
    --cert /Users/user/Development/kibana/search-guard-labs/search-guard-tlstool/out/kirk.pem \
    --key /Users/user/Development/kibana/search-guard-labs/search-guard-tlstool/out/kirk.key \
    https://node1.example.com:9200/_cat/indices

* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to node1.example.com (127.0.0.1) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
Enter PEM pass phrase:
* successfully set certificate verify locations:
*   CAfile: /Users/user/Development/kibana/search-guard-labs/search-guard-tlstool/out/root-ca.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: DC=com; DC=example; O=Example Com, Inc.; OU=Ops; CN=node1.example.com
*  start date: Sep  1 08:40:29 2020 GMT
*  expire date: Aug 30 08:40:29 2030 GMT
*  subjectAltName: host "node1.example.com" matched cert's "node1.example.com"
*  issuer: DC=com; DC=example; O=Example Com, Inc.; OU=CA; CN=signing.ca.example.com
*  SSL certificate verify ok.
> GET /_cat/indices HTTP/1.1
> Host: node1.example.com:9200
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: text/plain; charset=UTF-8
< content-length: 830
<
green open searchguard              Msm3W5d-Qbi7r1l_ll78lA 1 2  8 0  87.8kb 39.6kb
green open .signals_settings        Jvmw75LsThiowN7LCAeIyQ 1 1  2 0  12.3kb  7.9kb
green open .signals_watches         L5W4bsgVSMWy26PkV1W9aw 1 1  0 0    416b   208b
green open .apm-custom-link         uCbzNJDIRAiHpEWL_SZWJA 1 1  0 0    416b   208b
green open sg7-auditlog-2020.09.02  d_tlkwMBROi06DDgG8b86A 1 1 32 0 160.1kb 86.5kb
green open .kibana_task_manager_1   Ha16g34_Ts69EIzFmFWPvQ 1 1  5 0 120.8kb 55.6kb
green open .signals_accounts        MEe2xVt-R_KlWLnbdu2SVw 1 1  0 0    416b   208b
green open .apm-agent-configuration CCKUzn1GTpyHA8LfRYmBHw 1 1  0 0    416b   208b
green open .signals_watches_state   FnC1YPFBQrKTw0ktOf5TtA 1 1  0 0    416b   208b
green open .kibana_1                bFgWve3sSwGe2Y0vrkGjvw 1 1  3 0  46.4kb 23.2kb
* Connection #0 to host node1.example.com left intact
* Closing connection 0

Have you been able to try the curator in the lab environment?

Hi @srgbnd,

I understand what -k does, but if what you say regarding the invalid root-ca.pem is true, then this setting here probably has no effect at all: ssl_no_validate: true, at least not in combination with the certificate setting.

If that is correct, then I think I understand the issue. I replaced the master node http certificate with a real valid certificate, but did not modify the root-ca.pem. That’s did never cause any issues, because the communication on 9300 still uses the one from tlstools in between nodes. But now that curator checks the nodes validity on http against the root-ca.pem, this became an issue.