Error loading private key in securityadmin.sh in FIPS mode RHEL 8

First, thanks for all the work done here. The tool looks really flexible.

Elasticsearch version:
7.12 (Using opendistroforelasticsearch version 1.13.2)

Server OS version:
RedHat 8.3, FIPS mode enabled

Kibana version (if relevant):
N/A
Browser version (if relevant):
N/A
Browser OS version (if relevant):
N/A
Describe the issue:
I am trying to get my administrator certificate to be read by securityadmin.sh. I am able to get the transport and ssl security working, and then incrementally disabled the ‘allow_unsafe_democertificates’. Elasticsearch started without issues so I next started to tackle the admin certificate for search guard. However, no matter what, I can’t get the key to load during execution (before the tool even tries to reach out to the actual elasticsearch process/server). I tried looking at the last error on the stack, but I really have no idea what ‘version must be 1’ is referring to or how to address that. There is no passphrase on the private key (at least not yet). If I temporarily disable FIPS by issuing update-crypto-policies --set DEFAULT, the command succeeds (or at least gets past the key issue). Setting it back to update-crypto-policies --set FIPS creates the error again. I tried moving formats between various PKCS types, including 8 which it seems to want, but I can’t seem to get a configuration which will load the key. A system requirement is that the machine be in FIPS mode for the operating environment. Any help is much appreciated!

Steps to reproduce:

  1. Generate a key and CSR.
  2. Sign the CSR with the CA. (In our case, we are using an existing CA)
  3. Try to load the default security guard configuration with the following command:
    ./tools/securityadmin.sh -cd sgconfig/ -cacert /root/ca.crt -cert /root/admincert.crt -key /root/admincert.key -cd securityconfig/ -icl -nhnv

Expected behavior:
The key loads and the script continues to run.

Provide configuration:
elasticsearch/config/elasticsearch.yml
cluster.name: my-cluster
path.logs: /var/log/elasticsearch
network.host: localhost
http.port: 9200
opendistro_security.ssl.transport.pemcert_filepath: es.crt
opendistro_security.ssl.transport.pemkey_filepath: es.pem
opendistro_security.ssl.transport.pemtrustedcas_filepath: es-ca.crt
opendistro_security.ssl.transport.enforce_hostname_verification: false
opendistro_security.ssl.http.enabled: true
opendistro_security.ssl.http.pemcert_filepath: es.crt
opendistro_security.ssl.http.pemkey_filepath: es.pem
opendistro_security.ssl.http.pemtrustedcas_filepath: ssl_certs/ca-bundle.crt
opendistro_security.allow_unsafe_democertificates: false
opendistro_security.allow_default_init_securityindex: true
opendistro_security.authcz.admin_dn:

  • CN=myuser,O=DOMAIN.COM
    opendistro_security.audit.type: internal_elasticsearch
    opendistro_security.enable_snapshot_restore_privilege: true
    opendistro_security.check_snapshot_restore_write_privileges: true
    opendistro_security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]
    opendistro_security.system_indices.enabled: true
    opendistro_security.system_indices.indices: [".opendistro-alerting-config", “.opendistro-alerting-alert*”, “.opendistro-anomaly-results*”, “.opendistro-anomaly-detector*”, “.opendistro-anomaly-checkpoints”, “.opendistro-anomaly-detection-state”, “.opendistro-reports-", ".opendistro-notifications-”, “.opendistro-notebooks”, “.opendistro-asynchronous-search-response*”]
    cluster.routing.allocation.disk.threshold_enabled: false
    node.max_local_storage_nodes: 3

elasticsearch/plugins/search-guard-7/sgconfig/sg_config.yml
kibana/config/kibana.yml (if relevant)

Provide logs:
Elasticsearch
root@machine opendistro_security> ./tools/securityadmin.sh -cd sgconfig/ -cacert /root/ca.crt -cert /root/admincert.crt -key /root/admincert.pem -cd securityconfig/ -icl -nhnv
WARNING: JAVA_HOME not set, will use /bin/java
Open Distro Security Admin v7
Will connect to localhost:9300 … done
10:17:20.373 [main] ERROR com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore - Your keystore or PEM does not contain a key. If you specified a key password, try removing it. If you did not specify a key password, perhaps you need to if the key is in fact password-protected. Maybe you just confused keys and certificates.
ERR: An unexpected IllegalStateException occured: failed to load plugin class [com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin]
Trace:
java.lang.IllegalStateException: failed to load plugin class [com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:722)
at org.elasticsearch.plugins.PluginsService.(PluginsService.java:124)
at org.elasticsearch.client.transport.TransportClient.newPluginService(TransportClient.java:114)
at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:139)
at org.elasticsearch.client.transport.TransportClient.(TransportClient.java:282)
at com.amazon.opendistroforelasticsearch.security.tools.OpenDistroSecurityAdmin$TransportClientImpl.(OpenDistroSecurityAdmin.java:1007)
at com.amazon.opendistroforelasticsearch.security.tools.OpenDistroSecurityAdmin.execute(OpenDistroSecurityAdmin.java:520)
at com.amazon.opendistroforelasticsearch.security.tools.OpenDistroSecurityAdmin.main(OpenDistroSecurityAdmin.java:157)
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:713)
… 7 more
Caused by: ElasticsearchSecurityException[Error while initializing transport SSL layer from PEM: java.lang.IllegalArgumentException: File does not contain valid private key: /root/admincert.pem]; nested: IllegalArgumentException[File does not contain valid private key: /root/admincert.pem]; nested: InvalidKeySpecException[Neither RSA, DSA nor EC worked]; nested: InvalidKeySpecException[Could not create EC private key]; nested: InvalidKeySpecException[java.security.InvalidKeyException: Invalid EC private key]; nested: InvalidKeyException[Invalid EC private key]; nested: IOException[Version must be 1];
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.initTransportSSLConfig(DefaultOpenDistroSecurityKeyStore.java:409)
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.initSSLConfig(DefaultOpenDistroSecurityKeyStore.java:248)
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.(DefaultOpenDistroSecurityKeyStore.java:169)
at com.amazon.opendistroforelasticsearch.security.ssl.OpenDistroSecuritySSLPlugin.(OpenDistroSecuritySSLPlugin.java:217)
at com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin.(OpenDistroSecurityPlugin.java:246)
… 12 more
Caused by: java.lang.IllegalArgumentException: File does not contain valid private key: /root/admincert.pem
at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:350)
at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:107)
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.buildSSLServerContext(DefaultOpenDistroSecurityKeyStore.java:865)
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.initTransportSSLConfig(DefaultOpenDistroSecurityKeyStore.java:395)
… 16 more
Caused by: java.security.spec.InvalidKeySpecException: Neither RSA, DSA nor EC worked
at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1144)
at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1113)
at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:348)
… 19 more
Caused by: java.security.spec.InvalidKeySpecException: Could not create EC private key
at jdk.crypto.cryptoki/sun.security.pkcs11.P11ECKeyFactory.engineGeneratePrivate(P11ECKeyFactory.java:203)
at java.base/java.security.KeyFactory.generatePrivate(KeyFactory.java:390)
at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1142)
… 21 more
Caused by: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: Invalid EC private key
at jdk.crypto.ec/sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:169)
at java.base/java.security.KeyFactory.generatePrivate(KeyFactory.java:384)
at java.base/sun.security.util.ECUtil.decodePKCS8ECPrivateKey(ECUtil.java:116)
at jdk.crypto.cryptoki/sun.security.pkcs11.P11ECKeyFactory.engineGeneratePrivate(P11ECKeyFactory.java:200)
… 23 more
Caused by: java.security.InvalidKeyException: Invalid EC private key
at jdk.crypto.ec/sun.security.ec.ECPrivateKeyImpl.parseKeyBits(ECPrivateKeyImpl.java:206)
at java.base/sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:342)
at java.base/sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:355)
at jdk.crypto.ec/sun.security.ec.ECPrivateKeyImpl.(ECPrivateKeyImpl.java:74)
at jdk.crypto.ec/sun.security.ec.ECKeyFactory.implGeneratePrivate(ECKeyFactory.java:237)
at jdk.crypto.ec/sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:165)
… 26 more
Caused by: java.io.IOException: Version must be 1
at jdk.crypto.ec/sun.security.ec.ECPrivateKeyImpl.parseKeyBits(ECPrivateKeyImpl.java:184)
… 31 more
Kibana (if relevant)

Screenshots (if relevant):

Errors in browser console (if relevant):

Additional data:
Since it is just a dummy key, here is the content of the key that won’t load.
-----BEGIN PRIVATE KEY-----
MIIJRAIBADANBgkqhkiG9w0BAQEFAASCCS4wggkqAgEAAoICAQDvJA12bzVfuUSl
SjWFRk6QcKeP+EVAzC78yexqQ4kXOIiWnjxF5Cq2MRqPr7h5pMJHHR6Vv4hgBdBc
VL9g8WQP84PvASHY57qzx3i6vuzcHQfKoYokKzurkvn36t5Qfif41yhGP0AB8Dph
dTfTULje9H4jb+yR6SAdoFOGnFRbERANeloz7KlYp7Nb1YyxDBwMm3RZ3uRLvHI0
DKCktQNHROvkF182uihbGD4OXIO36CHycd2PAFgWIMu23lw7czHQM2p9eMP4AzZN
ybNJxxkwC8z/rb07EbppLJJnwmUCLpiJSvPRKzDfOH44KnuBdAorZpgfkt5Fi/Fl
eVKCzja7A7InxPmi2E3hxl1D9kQW1LnatK3kEs/xV+stCVE1a0jY0NPs/wBQMnvJ
p77W3jywEJ0mKDay04Dqo4h+CH6HvK4tUD0EjMmFtvfcpgyjSJj9frP7Zt3JV5v6
EuOuYJpW0QumKUqb+4x53uDh2RCrVu7d2IMQ/7HYIyx8ztsUSQR9todV7T0vWz/T
eOLYzokB6pkbnoKppVngeOHiX1nxnwAf1B7nIWgBR5AnhjOzitfVbje6DUi9LapK
G1djoXRKk8Ig4zpss5Y3/vnns1XefsHucdiup4zewR50s8I1sbaxO5Xowo8iEIPs
FaOTBwJaw7xkpw1wipiuPiuKAI/SWQIDAQABAoICAFwR9Vzz1AjWu6cnuxNkzjqu
JNU5MPCRrEWHOI5xEgburSfQJESEZC1hGmjXrp6m4GWgwHXHGCpgEqh5uHUKewvp
JZKPTQjFrwwocEL13xuf0bAiIZ1rR733CGZ69rzm1Vzo+YiSMYKNiuyaxLOHEKZG
W4opPVuv9b1yTLY22gN1vkhyIuh65XhMmLFytogzsVjpQRKYziyakR2Y/wBq8Kgv
0OIwXWgkpATugHXnCI91wBwn2rWD3GYomkX9tmTtJPCT2xwTst953R512aE+FmR5
JqQJsCFd4fc8QeHQ5jLA3dl+EgEmL9rBVVcqhAKDOqcbCNafJRSBnJ8E5s+VtVvs
pKQr7WJa9Lb3kBth5FGiuFJosyo9GNwyn2Mu+rt6KSaig1Ey9DPl8cTZy82ZTob+
vK0bTgXQ0zaxcL4XdlTjmzXWM1QImlV7kKo8IQMws3Cg+x7VcmsgG8s6JRZ1v4XY
Dr6SYJDpoCdHqIb0+BdTfUo4uzCXFWqX/FWQhAlKZPK5qZiUWfi0v6pM2PAVcWNe
0wBp3QKRm/b0ZUwr+ptfz2sfJWPf3MXtylizEiioGs0q9MKXe0f5HuYwpUZKWGem
ysJKTJf+hFzVoPMDfaUtIbVTFdu9b39zXfjdqRp0/K6bfypRoB5r5wFj6S/3klBu
ju5IiQ61eZ32vQdcuvkBAoIBAQD+6s/WXfay7h1sHJXPXEIbScDGQBcAKI0bSElB
u/PpTR83uiupGDpP8woGVBp7jp7UK/M7WWoJmmGUYwHbwDnMfYzLZUg8ExzJzxZL
TG558tIf95utCY7SYv34RMPY0BMs6l4QK34A6XvrJ14KAyOj2FAq7z5YIuRxV9qR
l0n9Uz+b5WCbjRKDbgqSAeRdimRpeC90nnoBhVfyLxd2GLUQV0pw868BT3rQPVfy
9BcjKjR+9/oQJH9E8PnFYDYTO3trQ5KrGvm2P47Ksd1UHxjdfZMzArteNPY/mNup
sZyeexJcRAEeiF2oW20EdqxK5fMgAN2Jlk8C0kqYoXKtYYeFAoIBAQDwKBYE1P5d
TrPVET++sMOboTx/f6DAzmts5OaOTOlPnOoEduWieSRkV/BgJve/aWWtY5rma6O1
B7zL3WnpAWKduTdP/ce4TENeFL+7OaOAVkmufBd7UWHHi/wLMiWCiON2d7z3hvop
ik7gkj6EjSltuY89gyuXGg4E4AWekUlBpUJWu78e2gZKqqwKuLouGSYGVvzA4z+Q
CldV38DVYaDvoB5zuL5SaaIeMxcHNPC23Kbh6wlm16A9EU465t/NOHbbXIkitMXU
Thq8gHHlS3Z20/8nWKQIyG/c8RnQapxXVy5Kzq4qqEniqjpzmsMDg7roZ/VDigzt
zFxWiKY95DXFAoIBAQD4k7sWG8ysUNvqB7niWeGwwGu2JtsG8YQf3Y5qBkDj+Kt/
aRCYNKLwQVw+LZchc0LYPP9TmW54bM3Ip/8w7bKvAKLh2CLnSe/hrwnbZW25hxp1
x7tLsi1OZO+pQE0GSGjafLQ66l5Vv1hyR5JBCyye4hnStDdENMo06GmfXTruR1aY
y8U4CgnKlezC0guBV6lw25zTCJKKlx/abydeptsxUx2pJUc7bOii1Z7EsFc7mDVW
trNnOF/Ot3moiZhRishgIaWNKqHkoPe6GlVsaWKS4Zg/vuM9C0IxY/qX8DV8qSCP
TlUVE/7YZEbbemmqI6/APnfJagQkCTJqwcyU260NAoIBAQCom7w50EB1yhe/88hJ
XVk4Hs+WxEhclZAed+bW6XC0kHpMgB1hIKRIPuVo3HL8VEcqoM/UDqZ1JuWKS/1b
tblC+xK2enqKDiEOIIssK597Lxm0AzOJRO3dZ46hXaewKeNo/VASiRWBrLUMPiDe
dy0q+Ch1C27DO2u0SIQpe99u01PsCQx54KCiM4luTZuJWea0dMYntmSeAGSoydtc
9JpFIqm+nWoh+j/z6/KzAJwwBmK0IowLREAgH9JI0Gc4cfggsqslTr3F/MAQsC1G
yuB1NQni2fzwnajL+gohpXf2aiL6KnllQZn+JX28RF++S/Ih0TsFHBtPqTb7Gh19
PofxAoIBAQDS03TYUZaodsm/LY+YA0VvC+QnXpRrosly9j+KR81AkeixWJ3IrIMQ
RcN3SRM/TCgsL6cox3gGbF7efx+jWqssJEWwrMVXkb3FQJ4zTjjsiZLCWf4a55X0
1ZGtfukukXE46LVg/HOwTSlqpR6MAvoHo20B63YrDQ6T11CVusUxw5U+WOKEIRex
kbTCdnG+W33mGES0s4Mr2gRMXVY04yiXQ7kF7Q/8zUosMXWKBAX6UicZfAgdH57p
gVrSxS0l2R3MC/EDuvkX4FPxe0BymqQdTvQPqMpKQ1PTrfaOQT2cN/MeTqcYFd16
ID4bP/CvWcu3XBpKJvbS9kdLQuN7ynqa
-----END PRIVATE KEY-----

As you are using Open Distro for Elasticsearch, you should direct your question to the respective forum:

This forum is only about Search Guard. Sorry for any inconvenience.

My apologies. I thought this was the upstream and where the experts would be on the topic. I am sorry I was mistaken. Have a great day.