Please help me figure out why one SSL configuration works and one does not

Environment:

Elasticsearch 5.4.2

Search Guard 5.4.2 (search-guard-5-5.4.2-12.jar, serach-guard-ssl-5.4.2-22.jar)

Summary: I can get my java application connecting to elasticsearch working under 1 circumstance, but not another. I’m unsure if this is a bug, but if someone could help me figure out what is going on, I would be eternally appreciative.

···

Non-working configuration:

elasticsearch.yml:

searchguard.ssl.transport.enabled: true

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.keystore_filepath: keystore.jks

searchguard.ssl.transport.truststore_filepath: truststore.jks

searchguard.ssl.transport.keystore_password: “[password here]”

searchguard.ssl.transport.truststore_password: “[password here]”

searchguard.ssl.transport.keystore_alias: key_alias

searchguard.ssl.transport.truststore_alias: trust_alias

searchguard.ssl.http.enabled: true

searchguard.ssl.http.keystore_filepath: keystore.jks

searchguard.ssl.http.truststore_filepath: truststore.jks

searchguard.ssl.http.keystore_password: “[password here]”

searchguard.ssl.http.truststore_password: “[password here]”

searchguard.ssl.http.keystore_alias: key_alias

searchguard.ssl.http.truststore_alias: trust_alias

keystore.jks:

key_alias

[chain1]server cert[privatekeyentry] -> signed by intermediate cert[trustedcertentry]

[chain2]intermediate cert[trustedcertentry] -> signed by root CA[trustedcertentry]

truststore.jks:

trust_alias

[chain1]root CA[trustedcertentry] -> self signed

Java app:

Settings.Builder builder = Settings.builder()

                .put("client.transport.sniff", true)

                .put("[cluster.name](http://cluster.name)", ES_CLUSTER_NAME)

                .put("path.home", ".")

                .put("node.data", false)

                .put("searchguard.ssl.transport.enabled", true)

                .put("searchguard.ssl.transport.keystore_filepath", [abs_path_to_keystore_above])

                .put("searchguard.ssl.transport.truststore_filepath", [abs_path_to_truststore_above])

                .put("searchguard.ssl.transport.keystore_alias", [keystore_alias_above])

                .put("searchguard.ssl.transport.truststore_alias", [truststore_alias_above])

                .put("searchguard.ssl.transport.enforce_hostname_verification", false);

if(StringUtils.isNotEmpty([keystore_password_above])){

builder.put("searchguard.ssl.transport.keystore_password", [keystore_password_above]);

}

if(StringUtils.isNotEmpty([truststore_password_above])){

builder.put("searchguard.ssl.transport.truststore_password", [truststore_password_above]);

}

final Settings settings = builder.build();

client = new PreBuiltTransportClient(settings, SearchGuardPlugin.class);

Problem:

My java app cannot access elasticsearch this way. Adding javax.net.debug=ssl to my java application reveals the following 2 errors:

This one happens when my app starts up:

ERROR com.floragunn.searchguard.ssl.util.SSLCertificateHelper - Alias key_alias does not contain hold a certificate entry.

And this one happens later:

elasticsearch[client][transport_client_boss][T#13], fatal error: 80: problem unwrapping net record
java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
%% Invalidated: [Session-157, [cipher used to be here]]
elasticsearch[client][transport_client_boss][T#13]
, SEND TLSv1.2 ALERT:
fatal,
description = internal_error


elasticsearch[client][transport_client_boss][T#13], WRITE: TLSv1.2 Alert, length = 2
elasticsearch[client][transport_client_boss][T#11], fatal error: 80: problem unwrapping net record
java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty


Working configuration:

Curiously, (and here is where I’m going to ask for help to figure out what is going on), if all I do is change the truststore alias to be the same as the keystore alias, everything works perfectly.

elasticsearch.yml (new values):

searchguard.ssl.transport.keystore_alias: key_alias

searchguard.ssl.transport.truststore_alias: key_alias

searchguard.ssl.http.keystore_alias: key_alias

searchguard.ssl.http.truststore_alias: key_alias

truststore.jks (updated alias):

key_alias

[chain1]root CA[trustedcertentry] -> self signed

Could anyone help me figure out what is going on here? Does the same alias name have to exist in both keystore and truststore??

Sorry, but at the moment I have to admit that I can’t explain this behavior. The non-working alias is for the truststore file, right? So what puzzles me is that the same truststore file with the “trust_alias” works when you configure it in elasticsearch.yml. Means, your cluster / node starts up without errors, right?

The strange thing here is that both ways of loading the keystore and truststore (the SG plugin and the TransportClient) use the exact same code. So the behavior should also be consistent.

···

On Tuesday, October 16, 2018 at 3:23:24 PM UTC+2, Steve Haertel wrote:

Environment:

Elasticsearch 5.4.2

Search Guard 5.4.2 (search-guard-5-5.4.2-12.jar, serach-guard-ssl-5.4.2-22.jar)

Summary: I can get my java application connecting to elasticsearch working under 1 circumstance, but not another. I’m unsure if this is a bug, but if someone could help me figure out what is going on, I would be eternally appreciative.


Non-working configuration:

elasticsearch.yml:

searchguard.ssl.transport.enabled: true

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.keystore_filepath: keystore.jks

searchguard.ssl.transport.truststore_filepath: truststore.jks

searchguard.ssl.transport.keystore_password: “[password here]”

searchguard.ssl.transport.truststore_password: “[password here]”

searchguard.ssl.transport.keystore_alias: key_alias

searchguard.ssl.transport.truststore_alias: trust_alias

searchguard.ssl.http.enabled: true

searchguard.ssl.http.keystore_filepath: keystore.jks

searchguard.ssl.http.truststore_filepath: truststore.jks

searchguard.ssl.http.keystore_password: “[password here]”

searchguard.ssl.http.truststore_password: “[password here]”

searchguard.ssl.http.keystore_alias: key_alias

searchguard.ssl.http.truststore_alias: trust_alias

keystore.jks:

key_alias

[chain1]server cert[privatekeyentry] -> signed by intermediate cert[trustedcertentry]

[chain2]intermediate cert[trustedcertentry] -> signed by root CA[trustedcertentry]

truststore.jks:

trust_alias

[chain1]root CA[trustedcertentry] -> self signed

Java app:

Settings.Builder builder = Settings.builder()

                .put("client.transport.sniff", true)
                .put("[cluster.name](http://cluster.name)", ES_CLUSTER_NAME)
                .put("path.home", ".")
                .put("node.data", false)
                .put("searchguard.ssl.transport.enabled", true)
                .put("searchguard.ssl.transport.keystore_filepath", [abs_path_to_keystore_above])
                .put("searchguard.ssl.transport.truststore_filepath", [abs_path_to_truststore_above])
                .put("searchguard.ssl.transport.keystore_alias", [keystore_alias_above])
                .put("searchguard.ssl.transport.truststore_alias", [truststore_alias_above])
                .put("searchguard.ssl.transport.enforce_hostname_verification", false);

if(StringUtils.isNotEmpty([keystore_password_above])){

builder.put("searchguard.ssl.transport.keystore_password", [keystore_password_above]);

}

if(StringUtils.isNotEmpty([truststore_password_above])){

builder.put("searchguard.ssl.transport.truststore_password", [truststore_password_above]);

}

final Settings settings = builder.build();

client = new PreBuiltTransportClient(settings, SearchGuardPlugin.class);

Problem:

My java app cannot access elasticsearch this way. Adding javax.net.debug=ssl to my java application reveals the following 2 errors:

This one happens when my app starts up:

ERROR com.floragunn.searchguard.ssl.util.SSLCertificateHelper - Alias key_alias does not contain hold a certificate entry.

And this one happens later:

elasticsearch[client][transport_client_boss][T#13], fatal error: 80: problem unwrapping net record
java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
%% Invalidated: [Session-157, [cipher used to be here]]
elasticsearch[client][transport_client_boss][T#13]
, SEND TLSv1.2 ALERT:
fatal,
description = internal_error


elasticsearch[client][transport_client_boss][T#13], WRITE: TLSv1.2 Alert, length = 2
elasticsearch[client][transport_client_boss][T#11], fatal error: 80: problem unwrapping net record
java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty


Working configuration:

Curiously, (and here is where I’m going to ask for help to figure out what is going on), if all I do is change the truststore alias to be the same as the keystore alias, everything works perfectly.

elasticsearch.yml (new values):

searchguard.ssl.transport.keystore_alias: key_alias

searchguard.ssl.transport.truststore_alias: key_alias

searchguard.ssl.http.keystore_alias: key_alias

searchguard.ssl.http.truststore_alias: key_alias

truststore.jks (updated alias):

key_alias

[chain1]root CA[trustedcertentry] -> self signed

Could anyone help me figure out what is going on here? Does the same alias name have to exist in both keystore and truststore??

Yes, that’s correct. The cluster looks like it loads without any errors. This is certainly puzzling me too, haha.