SSL exception when using the remote reindex API

  • Search Guard and Elasticsearch version: searchguard 5.1.1 r11

  • Installed and used enterprise modules, if any: DLSFLS, LDAP

  • JVM version and operating system version: java oracle1.8.0.151, RHEL 7.4

  • Search Guard configuration files: don’t think they are relevant here

  • Elasticsearch log messages on debug level: on request (contains basically the same info as below)

  • Other installed Elasticsearch or Kibana plugins, if any: n/a

Hi,

I tried to run following remote reindex command (in the DEV tools console or using curator) - adding user/password doesn’t change anything:

POST _reindex

{

“source”: {

“remote”: {

“host”: “https://kib-webtst01.ourdomain:9200

},

“index”: " logstash-test"

},

“dest”: {

“index”: “logstash-radius-udet-test-2017”

}

}

Which returns:

{

“error”: {

“root_cause”: [

{

“type”: “s_s_l_handshake_exception”,

“reason”: “General SSLEngine problem”

}

],

“type”: “s_s_l_handshake_exception”,

“reason”: “General SSLEngine problem”,

“caused_by”: {

“type”: “s_s_l_handshake_exception”,

“reason”: “General SSLEngine problem”,

“caused_by”: {

“type”: “validator_exception”,

“reason”: “PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target”,

“caused_by”: {

“type”: “sun_cert_path_builder_exception”,

“reason”: “unable to find valid certification path to requested target”

}

}

}

},

“status”: 500

}

I have included the CA certificate (and the signed node certificate) in the truststore.jks within /etc/elasticsearch. The remote server is also whitelisted in the ES config.

In the truststore:

keytool -list -keystore /etc/elasticsearch/truststore.jks

kib-webtst01.ourdomain, Nov 2, 2017, trustedCertEntry,

Certificate fingerprint (SHA1): CE:34:40:87:D3:23:1B:5A:C1:DE:81:BA:02:2F:6A:22:93:38:F4:CD

root-ca-chain, Jan 17, 2017, trustedCertEntry,

Certificate fingerprint (SHA1): A8:18:97:BA:54:94:2A:B8:75:95:E6:82:A0:27:0C:E7:2C:40:76:49

root-ca-chain-test, Nov 2, 2017, trustedCertEntry,

Certificate fingerprint (SHA1): 74:4D:F6:97:04:F6:E2:EB:BA:0C:A8:D9:7A:16:1D:89:C4:D9:20:7C

When I run SSLPoke (https://gist.github.com/4ndrej/4547029) I can successfully connect to the remote host using the same truststore.jks.

java -Djavax.net.ssl.trustStore=/etc/elasticsearch/truststore.jks -cp /tmp SS LPoke kib-webtst01.ourdomain 9200

Successfully connected

Thanks for any hints,

Andreas

Before digging deeper into this: Does the exception really only occure for remote reindex API?
So local reindexing or other api calls are working?

Pls. post also your elasticsearch.yml, thx

···

Am 03.11.2017 um 01:06 schrieb Andreas Freudenreich <andreas.freudenreich@icloud.com>:

* Search Guard and Elasticsearch version: searchguard 5.1.1 r11
* Installed and used enterprise modules, if any: DLSFLS, LDAP
* JVM version and operating system version: java oracle1.8.0.151, RHEL 7.4
* Search Guard configuration files: don't think they are relevant here
* Elasticsearch log messages on debug level: on request (contains basically the same info as below)
* Other installed Elasticsearch or Kibana plugins, if any: n/a

Hi,
I tried to run following remote reindex command (in the DEV tools console or using curator) - adding user/password doesn't change anything:

POST _reindex
{
  "source": {
    "remote": {
      "host": "https://kib-webtst01.ourdomain:9200"
    },
    "index": " logstash-test"
  },
  "dest": {
    "index": "logstash-radius-udet-test-2017"
  }
}

Which returns:
{
  "error": {
    "root_cause": [
      {
        "type": "s_s_l_handshake_exception",
        "reason": "General SSLEngine problem"
      }
    ],
    "type": "s_s_l_handshake_exception",
    "reason": "General SSLEngine problem",
    "caused_by": {
      "type": "s_s_l_handshake_exception",
      "reason": "General SSLEngine problem",
      "caused_by": {
        "type": "validator_exception",
        "reason": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
        "caused_by": {
          "type": "sun_cert_path_builder_exception",
          "reason": "unable to find valid certification path to requested target"
        }
      }
    }
  },
  "status": 500
}

I have included the CA certificate (and the signed node certificate) in the truststore.jks within /etc/elasticsearch. The remote server is also whitelisted in the ES config.
In the truststore:
# keytool -list -keystore /etc/elasticsearch/truststore.jks
...
kib-webtst01.ourdomain, Nov 2, 2017, trustedCertEntry,
Certificate fingerprint (SHA1): CE:34:40:87:D3:23:1B:5A:C1:DE:81:BA:02:2F:6A:22:93:38:F4:CD
root-ca-chain, Jan 17, 2017, trustedCertEntry,
Certificate fingerprint (SHA1): A8:18:97:BA:54:94:2A:B8:75:95:E6:82:A0:27:0C:E7:2C:40:76:49
root-ca-chain-test, Nov 2, 2017, trustedCertEntry,
Certificate fingerprint (SHA1): 74:4D:F6:97:04:F6:E2:EB:BA:0C:A8:D9:7A:16:1D:89:C4:D9:20:7C

When I run SSLPoke (https://gist.github.com/4ndrej/4547029) I can successfully connect to the remote host using the same truststore.jks.
# java -Djavax.net.ssl.trustStore=/etc/elasticsearch/truststore.jks -cp /tmp SS LPoke kib-webtst01.ourdomain 9200
Successfully connected

Thanks for any hints,
Andreas

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/f4aee903-2e99-4b46-9114-f6cc33949e40%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Other API calls and local indexing on both source and target cluster work (both are using searchguard):
POST _reindex

{

“source”: {

“index”: “logstash-test-2017”

},

“dest”: {

“index”: “logstash-test-2017_reindexed”

}

}

Result:

{

“took”: 1239,

“timed_out”: false,

“total”: 1,

“updated”: 0,

“created”: 1,

“deleted”: 0,

“batches”: 1,

“version_conflicts”: 0,

“noops”: 0,

“retries”: {

“bulk”: 0,

“search”: 0

},

“throttled_millis”: 0,

“requests_per_second”: -1,

“throttled_until_millis”: 0,

“failures”:

}

es_config_remote.yml (1.59 KB)

es_config_source.yml (1.46 KB)

···

On Thursday, 2 November 2017 17:06:36 UTC-7, Andreas Freudenreich wrote:

  • Search Guard and Elasticsearch version: searchguard 5.1.1 r11
  • Installed and used enterprise modules, if any: DLSFLS, LDAP
  • JVM version and operating system version: java oracle1.8.0.151, RHEL 7.4
  • Search Guard configuration files: don’t think they are relevant here
  • Elasticsearch log messages on debug level: on request (contains basically the same info as below)
  • Other installed Elasticsearch or Kibana plugins, if any: n/a

Hi,

I tried to run following remote reindex command (in the DEV tools console or using curator) - adding user/password doesn’t change anything:

POST _reindex

{

“source”: {

“remote”: {

“host”: “https://kib-webtst01.ourdomain:9200

},

“index”: " logstash-test"

},

“dest”: {

“index”: “logstash-radius-udet-test-2017”

}

}

Which returns:

{

“error”: {

“root_cause”: [

{

“type”: “s_s_l_handshake_exception”,

“reason”: “General SSLEngine problem”

}

],

“type”: “s_s_l_handshake_exception”,

“reason”: “General SSLEngine problem”,

“caused_by”: {

“type”: “s_s_l_handshake_exception”,

“reason”: “General SSLEngine problem”,

“caused_by”: {

“type”: “validator_exception”,

“reason”: “PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target”,

“caused_by”: {

“type”: “sun_cert_path_builder_exception”,

“reason”: “unable to find valid certification path to requested target”

}

}

}

},

“status”: 500

}

I have included the CA certificate (and the signed node certificate) in the truststore.jks within /etc/elasticsearch. The remote server is also whitelisted in the ES config.

In the truststore:

keytool -list -keystore /etc/elasticsearch/truststore.jks

kib-webtst01.ourdomain, Nov 2, 2017, trustedCertEntry,

Certificate fingerprint (SHA1): CE:34:40:87:D3:23:1B:5A:C1:DE:81:BA:02:2F:6A:22:93:38:F4:CD

root-ca-chain, Jan 17, 2017, trustedCertEntry,

Certificate fingerprint (SHA1): A8:18:97:BA:54:94:2A:B8:75:95:E6:82:A0:27:0C:E7:2C:40:76:49

root-ca-chain-test, Nov 2, 2017, trustedCertEntry,

Certificate fingerprint (SHA1): 74:4D:F6:97:04:F6:E2:EB:BA:0C:A8:D9:7A:16:1D:89:C4:D9:20:7C

When I run SSLPoke (https://gist.github.com/4ndrej/4547029) I can successfully connect to the remote host using the same truststore.jks.

java -Djavax.net.ssl.trustStore=/etc/elasticsearch/truststore.jks -cp /tmp SS LPoke kib-webtst01.ourdomain 9200

Successfully connected

Thanks for any hints,

Andreas

tracked here https://github.com/elastic/elasticsearch/issues/27267

···

Am 03.11.2017 um 23:27 schrieb Andreas Freudenreich <andreas.freudenreich@icloud.com>:

Other API calls and local indexing on both source and target cluster work (both are using searchguard):
POST _reindex
{
  "source": {
    "index": "logstash-test-2017"
  },
  "dest": {
    "index": "logstash-test-2017_reindexed"
  }
}

Result:
{
  "took": 1239,
  "timed_out": false,
  "total": 1,
  "updated": 0,
  "created": 1,
  "deleted": 0,
  "batches": 1,
  "version_conflicts": 0,
  "noops": 0,
  "retries": {
    "bulk": 0,
    "search": 0
  },
  "throttled_millis": 0,
  "requests_per_second": -1,
  "throttled_until_millis": 0,
  "failures":
}

On Thursday, 2 November 2017 17:06:36 UTC-7, Andreas Freudenreich wrote:
* Search Guard and Elasticsearch version: searchguard 5.1.1 r11
* Installed and used enterprise modules, if any: DLSFLS, LDAP
* JVM version and operating system version: java oracle1.8.0.151, RHEL 7.4
* Search Guard configuration files: don't think they are relevant here
* Elasticsearch log messages on debug level: on request (contains basically the same info as below)
* Other installed Elasticsearch or Kibana plugins, if any: n/a

Hi,
I tried to run following remote reindex command (in the DEV tools console or using curator) - adding user/password doesn't change anything:

POST _reindex
{
  "source": {
    "remote": {
      "host": "https://kib-webtst01.ourdomain:9200"
    },
    "index": " logstash-test"
  },
  "dest": {
    "index": "logstash-radius-udet-test-2017"
  }
}

Which returns:
{
  "error": {
    "root_cause": [
      {
        "type": "s_s_l_handshake_exception",
        "reason": "General SSLEngine problem"
      }
    ],
    "type": "s_s_l_handshake_exception",
    "reason": "General SSLEngine problem",
    "caused_by": {
      "type": "s_s_l_handshake_exception",
      "reason": "General SSLEngine problem",
      "caused_by": {
        "type": "validator_exception",
        "reason": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
        "caused_by": {
          "type": "sun_cert_path_builder_exception",
          "reason": "unable to find valid certification path to requested target"
        }
      }
    }
  },
  "status": 500
}

I have included the CA certificate (and the signed node certificate) in the truststore.jks within /etc/elasticsearch. The remote server is also whitelisted in the ES config.
In the truststore:
# keytool -list -keystore /etc/elasticsearch/truststore.jks
...
kib-webtst01.ourdomain, Nov 2, 2017, trustedCertEntry,
Certificate fingerprint (SHA1): CE:34:40:87:D3:23:1B:5A:C1:DE:81:BA:02:2F:6A:22:93:38:F4:CD
root-ca-chain, Jan 17, 2017, trustedCertEntry,
Certificate fingerprint (SHA1): A8:18:97:BA:54:94:2A:B8:75:95:E6:82:A0:27:0C:E7:2C:40:76:49
root-ca-chain-test, Nov 2, 2017, trustedCertEntry,
Certificate fingerprint (SHA1): 74:4D:F6:97:04:F6:E2:EB:BA:0C:A8:D9:7A:16:1D:89:C4:D9:20:7C

When I run SSLPoke (https://gist.github.com/4ndrej/4547029) I can successfully connect to the remote host using the same truststore.jks.
# java -Djavax.net.ssl.trustStore=/etc/elasticsearch/truststore.jks -cp /tmp SS LPoke kib-webtst01.ourdomain 9200
Successfully connected

Thanks for any hints,
Andreas

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/491d58da-d490-47ad-8718-49c235463274%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
<es_config_remote.yml><es_config_source.yml>