SSL exception when using the remote reindex API

Search Guard
1

  • Search Guard and Elasticsearch version: searchguard 5.1.1 r11

  • Installed and used enterprise modules, if any: DLSFLS, LDAP

  • JVM version and operating system version: java oracle1.8.0.151, RHEL 7.4

  • Search Guard configuration files: don’t think they are relevant here

  • Elasticsearch log messages on debug level: on request (contains basically the same info as below)

  • Other installed Elasticsearch or Kibana plugins, if any: n/a

Hi,

I tried to run following remote reindex command (in the DEV tools console or using curator) - adding user/password doesn’t change anything:

POST _reindex

{

“source”: {

“remote”: {

“host”: “https://kib-webtst01.ourdomain:9200”

},

“index”: " logstash-test"

},

“dest”: {

“index”: “logstash-radius-udet-test-2017”

}

}

Which returns:

{

“error”: {

“root_cause”: [

{

“type”: “s_s_l_handshake_exception”,

“reason”: “General SSLEngine problem”

}

],

“type”: “s_s_l_handshake_exception”,

“reason”: “General SSLEngine problem”,

“caused_by”: {

“type”: “s_s_l_handshake_exception”,

“reason”: “General SSLEngine problem”,

“caused_by”: {

“type”: “validator_exception”,

“reason”: “PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target”,

“caused_by”: {

“type”: “sun_cert_path_builder_exception”,

“reason”: “unable to find valid certification path to requested target”

}

}

}

},

“status”: 500

}

I have included the CA certificate (and the signed node certificate) in the truststore.jks within /etc/elasticsearch. The remote server is also whitelisted in the ES config.

In the truststore:

keytool -list -keystore /etc/elasticsearch/truststore.jks

…

kib-webtst01.ourdomain, Nov 2, 2017, trustedCertEntry,

Certificate fingerprint (SHA1): CE:34:40:87:D3:23:1B:5A:C1:DE:81:BA:02:2F:6A:22:93:38:F4:CD

root-ca-chain, Jan 17, 2017, trustedCertEntry,

Certificate fingerprint (SHA1): A8:18:97:BA:54:94:2A:B8:75:95:E6:82:A0:27:0C:E7:2C:40:76:49

root-ca-chain-test, Nov 2, 2017, trustedCertEntry,

Certificate fingerprint (SHA1): 74:4D:F6:97:04:F6:E2:EB:BA:0C:A8:D9:7A:16:1D:89:C4:D9:20:7C

When I run SSLPoke (Test of java SSL / keystore / cert setup. Check the comment #1 for howto. · GitHub) I can successfully connect to the remote host using the same truststore.jks.

java -Djavax.net.ssl.trustStore=/etc/elasticsearch/truststore.jks -cp /tmp SS LPoke kib-webtst01.ourdomain 9200

Successfully connected

Thanks for any hints,

Andreas

2

Before digging deeper into this: Does the exception really only occure for remote reindex API?
So local reindexing or other api calls are working?

Pls. post also your elasticsearch.yml, thx

···

Am 03.11.2017 um 01:06 schrieb Andreas Freudenreich <andreas.freudenreich@icloud.com>:

* Search Guard and Elasticsearch version: searchguard 5.1.1 r11
* Installed and used enterprise modules, if any: DLSFLS, LDAP
* JVM version and operating system version: java oracle1.8.0.151, RHEL 7.4
* Search Guard configuration files: don't think they are relevant here
* Elasticsearch log messages on debug level: on request (contains basically the same info as below)
* Other installed Elasticsearch or Kibana plugins, if any: n/a

Hi,
I tried to run following remote reindex command (in the DEV tools console or using curator) - adding user/password doesn't change anything:

POST _reindex
{
  "source": {
    "remote": {
      "host": "https://kib-webtst01.ourdomain:9200"
    },
    "index": " logstash-test"
  },
  "dest": {
    "index": "logstash-radius-udet-test-2017"
  }
}

Which returns:
{
  "error": {
    "root_cause": [
      {
        "type": "s_s_l_handshake_exception",
        "reason": "General SSLEngine problem"
      }
    ],
    "type": "s_s_l_handshake_exception",
    "reason": "General SSLEngine problem",
    "caused_by": {
      "type": "s_s_l_handshake_exception",
      "reason": "General SSLEngine problem",
      "caused_by": {
        "type": "validator_exception",
        "reason": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
        "caused_by": {
          "type": "sun_cert_path_builder_exception",
          "reason": "unable to find valid certification path to requested target"
        }
      }
    }
  },
  "status": 500
}

I have included the CA certificate (and the signed node certificate) in the truststore.jks within /etc/elasticsearch. The remote server is also whitelisted in the ES config.
In the truststore:
# keytool -list -keystore /etc/elasticsearch/truststore.jks
...
kib-webtst01.ourdomain, Nov 2, 2017, trustedCertEntry,
Certificate fingerprint (SHA1): CE:34:40:87:D3:23:1B:5A:C1:DE:81:BA:02:2F:6A:22:93:38:F4:CD
root-ca-chain, Jan 17, 2017, trustedCertEntry,
Certificate fingerprint (SHA1): A8:18:97:BA:54:94:2A:B8:75:95:E6:82:A0:27:0C:E7:2C:40:76:49
root-ca-chain-test, Nov 2, 2017, trustedCertEntry,
Certificate fingerprint (SHA1): 74:4D:F6:97:04:F6:E2:EB:BA:0C:A8:D9:7A:16:1D:89:C4:D9:20:7C

When I run SSLPoke (https://gist.github.com/4ndrej/4547029\) I can successfully connect to the remote host using the same truststore.jks.
# java -Djavax.net.ssl.trustStore=/etc/elasticsearch/truststore.jks -cp /tmp SS LPoke kib-webtst01.ourdomain 9200
Successfully connected

Thanks for any hints,
Andreas

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/f4aee903-2e99-4b46-9114-f6cc33949e40%40googlegroups.com\.
For more options, visit https://groups.google.com/d/optout\.

3

Other API calls and local indexing on both source and target cluster work (both are using searchguard):
POST _reindex

{

“source”: {

“index”: “logstash-test-2017”

},

“dest”: {

“index”: “logstash-test-2017_reindexed”

}

}

Result:

{

“took”: 1239,

“timed_out”: false,

“total”: 1,

“updated”: 0,

“created”: 1,

“deleted”: 0,

“batches”: 1,

“version_conflicts”: 0,

“noops”: 0,

“retries”: {

“bulk”: 0,

“search”: 0

},

“throttled_millis”: 0,

“requests_per_second”: -1,

“throttled_until_millis”: 0,

“failures”:

}

es_config_remote.yml (1.59 KB)

es_config_source.yml (1.46 KB)

···

On Thursday, 2 November 2017 17:06:36 UTC-7, Andreas Freudenreich wrote:

  • Search Guard and Elasticsearch version: searchguard 5.1.1 r11
  • Installed and used enterprise modules, if any: DLSFLS, LDAP
  • JVM version and operating system version: java oracle1.8.0.151, RHEL 7.4
  • Search Guard configuration files: don’t think they are relevant here
  • Elasticsearch log messages on debug level: on request (contains basically the same info as below)
  • Other installed Elasticsearch or Kibana plugins, if any: n/a

Hi,

I tried to run following remote reindex command (in the DEV tools console or using curator) - adding user/password doesn’t change anything:

POST _reindex

{

“source”: {

“remote”: {

“host”: “https://kib-webtst01.ourdomain:9200”

},

“index”: " logstash-test"

},

“dest”: {

“index”: “logstash-radius-udet-test-2017”

}

}

Which returns:

{

“error”: {

“root_cause”: [

{

“type”: “s_s_l_handshake_exception”,

“reason”: “General SSLEngine problem”

}

],

“type”: “s_s_l_handshake_exception”,

“reason”: “General SSLEngine problem”,

“caused_by”: {

“type”: “s_s_l_handshake_exception”,

“reason”: “General SSLEngine problem”,

“caused_by”: {

“type”: “validator_exception”,

“reason”: “PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target”,

“caused_by”: {

“type”: “sun_cert_path_builder_exception”,

“reason”: “unable to find valid certification path to requested target”

}

}

}

},

“status”: 500

}

I have included the CA certificate (and the signed node certificate) in the truststore.jks within /etc/elasticsearch. The remote server is also whitelisted in the ES config.

In the truststore:

keytool -list -keystore /etc/elasticsearch/truststore.jks

…

kib-webtst01.ourdomain, Nov 2, 2017, trustedCertEntry,

Certificate fingerprint (SHA1): CE:34:40:87:D3:23:1B:5A:C1:DE:81:BA:02:2F:6A:22:93:38:F4:CD

root-ca-chain, Jan 17, 2017, trustedCertEntry,

Certificate fingerprint (SHA1): A8:18:97:BA:54:94:2A:B8:75:95:E6:82:A0:27:0C:E7:2C:40:76:49

root-ca-chain-test, Nov 2, 2017, trustedCertEntry,

Certificate fingerprint (SHA1): 74:4D:F6:97:04:F6:E2:EB:BA:0C:A8:D9:7A:16:1D:89:C4:D9:20:7C

When I run SSLPoke (https://gist.github.com/4ndrej/4547029) I can successfully connect to the remote host using the same truststore.jks.

java -Djavax.net.ssl.trustStore=/etc/elasticsearch/truststore.jks -cp /tmp SS LPoke kib-webtst01.ourdomain 9200

Successfully connected

Thanks for any hints,

Andreas

4

tracked here Remote _reindex fail for HTTPS remotes · Issue #27267 · elastic/elasticsearch · GitHub

···

Am 03.11.2017 um 23:27 schrieb Andreas Freudenreich <andreas.freudenreich@icloud.com>:

Other API calls and local indexing on both source and target cluster work (both are using searchguard):
POST _reindex
{
  "source": {
    "index": "logstash-test-2017"
  },
  "dest": {
    "index": "logstash-test-2017_reindexed"
  }
}

Result:
{
  "took": 1239,
  "timed_out": false,
  "total": 1,
  "updated": 0,
  "created": 1,
  "deleted": 0,
  "batches": 1,
  "version_conflicts": 0,
  "noops": 0,
  "retries": {
    "bulk": 0,
    "search": 0
  },
  "throttled_millis": 0,
  "requests_per_second": -1,
  "throttled_until_millis": 0,
  "failures":
}

On Thursday, 2 November 2017 17:06:36 UTC-7, Andreas Freudenreich wrote:
* Search Guard and Elasticsearch version: searchguard 5.1.1 r11
* Installed and used enterprise modules, if any: DLSFLS, LDAP
* JVM version and operating system version: java oracle1.8.0.151, RHEL 7.4
* Search Guard configuration files: don't think they are relevant here
* Elasticsearch log messages on debug level: on request (contains basically the same info as below)
* Other installed Elasticsearch or Kibana plugins, if any: n/a

Hi,
I tried to run following remote reindex command (in the DEV tools console or using curator) - adding user/password doesn't change anything:

POST _reindex
{
  "source": {
    "remote": {
      "host": "https://kib-webtst01.ourdomain:9200"
    },
    "index": " logstash-test"
  },
  "dest": {
    "index": "logstash-radius-udet-test-2017"
  }
}

Which returns:
{
  "error": {
    "root_cause": [
      {
        "type": "s_s_l_handshake_exception",
        "reason": "General SSLEngine problem"
      }
    ],
    "type": "s_s_l_handshake_exception",
    "reason": "General SSLEngine problem",
    "caused_by": {
      "type": "s_s_l_handshake_exception",
      "reason": "General SSLEngine problem",
      "caused_by": {
        "type": "validator_exception",
        "reason": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
        "caused_by": {
          "type": "sun_cert_path_builder_exception",
          "reason": "unable to find valid certification path to requested target"
        }
      }
    }
  },
  "status": 500
}

I have included the CA certificate (and the signed node certificate) in the truststore.jks within /etc/elasticsearch. The remote server is also whitelisted in the ES config.
In the truststore:
# keytool -list -keystore /etc/elasticsearch/truststore.jks
...
kib-webtst01.ourdomain, Nov 2, 2017, trustedCertEntry,
Certificate fingerprint (SHA1): CE:34:40:87:D3:23:1B:5A:C1:DE:81:BA:02:2F:6A:22:93:38:F4:CD
root-ca-chain, Jan 17, 2017, trustedCertEntry,
Certificate fingerprint (SHA1): A8:18:97:BA:54:94:2A:B8:75:95:E6:82:A0:27:0C:E7:2C:40:76:49
root-ca-chain-test, Nov 2, 2017, trustedCertEntry,
Certificate fingerprint (SHA1): 74:4D:F6:97:04:F6:E2:EB:BA:0C:A8:D9:7A:16:1D:89:C4:D9:20:7C

When I run SSLPoke (https://gist.github.com/4ndrej/4547029\) I can successfully connect to the remote host using the same truststore.jks.
# java -Djavax.net.ssl.trustStore=/etc/elasticsearch/truststore.jks -cp /tmp SS LPoke kib-webtst01.ourdomain 9200
Successfully connected

Thanks for any hints,
Andreas

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/491d58da-d490-47ad-8718-49c235463274%40googlegroups.com\.
For more options, visit https://groups.google.com/d/optout\.
<es_config_remote.yml><es_config_source.yml>