First, thanks for all the work done here. The tool looks really flexible.
Elasticsearch version:
7.12 (Using opendistroforelasticsearch version 1.13.2)
Server OS version:
RedHat 8.3, FIPS mode enabled
Kibana version (if relevant):
N/A
Browser version (if relevant):
N/A
Browser OS version (if relevant):
N/A
Describe the issue:
I am trying to get my administrator certificate to be read by securityadmin.sh. I am able to get the transport and ssl security working, and then incrementally disabled the ‘allow_unsafe_democertificates’. Elasticsearch started without issues so I next started to tackle the admin certificate for search guard. However, no matter what, I can’t get the key to load during execution (before the tool even tries to reach out to the actual elasticsearch process/server). I tried looking at the last error on the stack, but I really have no idea what ‘version must be 1’ is referring to or how to address that. There is no passphrase on the private key (at least not yet). If I temporarily disable FIPS by issuing update-crypto-policies --set DEFAULT, the command succeeds (or at least gets past the key issue). Setting it back to update-crypto-policies --set FIPS creates the error again. I tried moving formats between various PKCS types, including 8 which it seems to want, but I can’t seem to get a configuration which will load the key. A system requirement is that the machine be in FIPS mode for the operating environment. Any help is much appreciated!
Steps to reproduce:
- Generate a key and CSR.
- Sign the CSR with the CA. (In our case, we are using an existing CA)
- Try to load the default security guard configuration with the following command:
./tools/securityadmin.sh -cd sgconfig/ -cacert /root/ca.crt -cert /root/admincert.crt -key /root/admincert.key -cd securityconfig/ -icl -nhnv
Expected behavior:
The key loads and the script continues to run.
Provide configuration:
elasticsearch/config/elasticsearch.yml
cluster.name: my-cluster
path.logs: /var/log/elasticsearch
network.host: localhost
http.port: 9200
opendistro_security.ssl.transport.pemcert_filepath: es.crt
opendistro_security.ssl.transport.pemkey_filepath: es.pem
opendistro_security.ssl.transport.pemtrustedcas_filepath: es-ca.crt
opendistro_security.ssl.transport.enforce_hostname_verification: false
opendistro_security.ssl.http.enabled: true
opendistro_security.ssl.http.pemcert_filepath: es.crt
opendistro_security.ssl.http.pemkey_filepath: es.pem
opendistro_security.ssl.http.pemtrustedcas_filepath: ssl_certs/ca-bundle.crt
opendistro_security.allow_unsafe_democertificates: false
opendistro_security.allow_default_init_securityindex: true
opendistro_security.authcz.admin_dn:
- CN=myuser,O=DOMAIN.COM
opendistro_security.audit.type: internal_elasticsearch
opendistro_security.enable_snapshot_restore_privilege: true
opendistro_security.check_snapshot_restore_write_privileges: true
opendistro_security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]
opendistro_security.system_indices.enabled: true
opendistro_security.system_indices.indices: [“.opendistro-alerting-config”, “.opendistro-alerting-alert*”, “.opendistro-anomaly-results*”, “.opendistro-anomaly-detector*”, “.opendistro-anomaly-checkpoints”, “.opendistro-anomaly-detection-state”, “.opendistro-reports-", ".opendistro-notifications-”, “.opendistro-notebooks”, “.opendistro-asynchronous-search-response*”]
cluster.routing.allocation.disk.threshold_enabled: false
node.max_local_storage_nodes: 3
elasticsearch/plugins/search-guard-7/sgconfig/sg_config.yml
kibana/config/kibana.yml (if relevant)
Provide logs:
Elasticsearch
root@machine opendistro_security> ./tools/securityadmin.sh -cd sgconfig/ -cacert /root/ca.crt -cert /root/admincert.crt -key /root/admincert.pem -cd securityconfig/ -icl -nhnv
WARNING: JAVA_HOME not set, will use /bin/java
Open Distro Security Admin v7
Will connect to localhost:9300 … done
10:17:20.373 [main] ERROR com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore - Your keystore or PEM does not contain a key. If you specified a key password, try removing it. If you did not specify a key password, perhaps you need to if the key is in fact password-protected. Maybe you just confused keys and certificates.
ERR: An unexpected IllegalStateException occured: failed to load plugin class [com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin]
Trace:
java.lang.IllegalStateException: failed to load plugin class [com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:722)
at org.elasticsearch.plugins.PluginsService.(PluginsService.java:124)
at org.elasticsearch.client.transport.TransportClient.newPluginService(TransportClient.java:114)
at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:139)
at org.elasticsearch.client.transport.TransportClient.(TransportClient.java:282)
at com.amazon.opendistroforelasticsearch.security.tools.OpenDistroSecurityAdmin$TransportClientImpl.(OpenDistroSecurityAdmin.java:1007)
at com.amazon.opendistroforelasticsearch.security.tools.OpenDistroSecurityAdmin.execute(OpenDistroSecurityAdmin.java:520)
at com.amazon.opendistroforelasticsearch.security.tools.OpenDistroSecurityAdmin.main(OpenDistroSecurityAdmin.java:157)
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:713)
… 7 more
Caused by: ElasticsearchSecurityException[Error while initializing transport SSL layer from PEM: java.lang.IllegalArgumentException: File does not contain valid private key: /root/admincert.pem]; nested: IllegalArgumentException[File does not contain valid private key: /root/admincert.pem]; nested: InvalidKeySpecException[Neither RSA, DSA nor EC worked]; nested: InvalidKeySpecException[Could not create EC private key]; nested: InvalidKeySpecException[java.security.InvalidKeyException: Invalid EC private key]; nested: InvalidKeyException[Invalid EC private key]; nested: IOException[Version must be 1];
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.initTransportSSLConfig(DefaultOpenDistroSecurityKeyStore.java:409)
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.initSSLConfig(DefaultOpenDistroSecurityKeyStore.java:248)
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.(DefaultOpenDistroSecurityKeyStore.java:169)
at com.amazon.opendistroforelasticsearch.security.ssl.OpenDistroSecuritySSLPlugin.(OpenDistroSecuritySSLPlugin.java:217)
at com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin.(OpenDistroSecurityPlugin.java:246)
… 12 more
Caused by: java.lang.IllegalArgumentException: File does not contain valid private key: /root/admincert.pem
at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:350)
at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:107)
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.buildSSLServerContext(DefaultOpenDistroSecurityKeyStore.java:865)
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.initTransportSSLConfig(DefaultOpenDistroSecurityKeyStore.java:395)
… 16 more
Caused by: java.security.spec.InvalidKeySpecException: Neither RSA, DSA nor EC worked
at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1144)
at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1113)
at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:348)
… 19 more
Caused by: java.security.spec.InvalidKeySpecException: Could not create EC private key
at jdk.crypto.cryptoki/sun.security.pkcs11.P11ECKeyFactory.engineGeneratePrivate(P11ECKeyFactory.java:203)
at java.base/java.security.KeyFactory.generatePrivate(KeyFactory.java:390)
at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1142)
… 21 more
Caused by: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: Invalid EC private key
at jdk.crypto.ec/sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:169)
at java.base/java.security.KeyFactory.generatePrivate(KeyFactory.java:384)
at java.base/sun.security.util.ECUtil.decodePKCS8ECPrivateKey(ECUtil.java:116)
at jdk.crypto.cryptoki/sun.security.pkcs11.P11ECKeyFactory.engineGeneratePrivate(P11ECKeyFactory.java:200)
… 23 more
Caused by: java.security.InvalidKeyException: Invalid EC private key
at jdk.crypto.ec/sun.security.ec.ECPrivateKeyImpl.parseKeyBits(ECPrivateKeyImpl.java:206)
at java.base/sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:342)
at java.base/sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:355)
at jdk.crypto.ec/sun.security.ec.ECPrivateKeyImpl.(ECPrivateKeyImpl.java:74)
at jdk.crypto.ec/sun.security.ec.ECKeyFactory.implGeneratePrivate(ECKeyFactory.java:237)
at jdk.crypto.ec/sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:165)
… 26 more
Caused by: java.io.IOException: Version must be 1
at jdk.crypto.ec/sun.security.ec.ECPrivateKeyImpl.parseKeyBits(ECPrivateKeyImpl.java:184)
… 31 more
Kibana (if relevant)
Screenshots (if relevant):
Errors in browser console (if relevant):
Additional data:
Since it is just a dummy key, here is the content of the key that won’t load.
-----BEGIN PRIVATE KEY-----
MIIJRAIBADANBgkqhkiG9w0BAQEFAASCCS4wggkqAgEAAoICAQDvJA12bzVfuUSl
SjWFRk6QcKeP+EVAzC78yexqQ4kXOIiWnjxF5Cq2MRqPr7h5pMJHHR6Vv4hgBdBc
VL9g8WQP84PvASHY57qzx3i6vuzcHQfKoYokKzurkvn36t5Qfif41yhGP0AB8Dph
dTfTULje9H4jb+yR6SAdoFOGnFRbERANeloz7KlYp7Nb1YyxDBwMm3RZ3uRLvHI0
DKCktQNHROvkF182uihbGD4OXIO36CHycd2PAFgWIMu23lw7czHQM2p9eMP4AzZN
ybNJxxkwC8z/rb07EbppLJJnwmUCLpiJSvPRKzDfOH44KnuBdAorZpgfkt5Fi/Fl
eVKCzja7A7InxPmi2E3hxl1D9kQW1LnatK3kEs/xV+stCVE1a0jY0NPs/wBQMnvJ
p77W3jywEJ0mKDay04Dqo4h+CH6HvK4tUD0EjMmFtvfcpgyjSJj9frP7Zt3JV5v6
EuOuYJpW0QumKUqb+4x53uDh2RCrVu7d2IMQ/7HYIyx8ztsUSQR9todV7T0vWz/T
eOLYzokB6pkbnoKppVngeOHiX1nxnwAf1B7nIWgBR5AnhjOzitfVbje6DUi9LapK
G1djoXRKk8Ig4zpss5Y3/vnns1XefsHucdiup4zewR50s8I1sbaxO5Xowo8iEIPs
FaOTBwJaw7xkpw1wipiuPiuKAI/SWQIDAQABAoICAFwR9Vzz1AjWu6cnuxNkzjqu
JNU5MPCRrEWHOI5xEgburSfQJESEZC1hGmjXrp6m4GWgwHXHGCpgEqh5uHUKewvp
JZKPTQjFrwwocEL13xuf0bAiIZ1rR733CGZ69rzm1Vzo+YiSMYKNiuyaxLOHEKZG
W4opPVuv9b1yTLY22gN1vkhyIuh65XhMmLFytogzsVjpQRKYziyakR2Y/wBq8Kgv
0OIwXWgkpATugHXnCI91wBwn2rWD3GYomkX9tmTtJPCT2xwTst953R512aE+FmR5
JqQJsCFd4fc8QeHQ5jLA3dl+EgEmL9rBVVcqhAKDOqcbCNafJRSBnJ8E5s+VtVvs
pKQr7WJa9Lb3kBth5FGiuFJosyo9GNwyn2Mu+rt6KSaig1Ey9DPl8cTZy82ZTob+
vK0bTgXQ0zaxcL4XdlTjmzXWM1QImlV7kKo8IQMws3Cg+x7VcmsgG8s6JRZ1v4XY
Dr6SYJDpoCdHqIb0+BdTfUo4uzCXFWqX/FWQhAlKZPK5qZiUWfi0v6pM2PAVcWNe
0wBp3QKRm/b0ZUwr+ptfz2sfJWPf3MXtylizEiioGs0q9MKXe0f5HuYwpUZKWGem
ysJKTJf+hFzVoPMDfaUtIbVTFdu9b39zXfjdqRp0/K6bfypRoB5r5wFj6S/3klBu
ju5IiQ61eZ32vQdcuvkBAoIBAQD+6s/WXfay7h1sHJXPXEIbScDGQBcAKI0bSElB
u/PpTR83uiupGDpP8woGVBp7jp7UK/M7WWoJmmGUYwHbwDnMfYzLZUg8ExzJzxZL
TG558tIf95utCY7SYv34RMPY0BMs6l4QK34A6XvrJ14KAyOj2FAq7z5YIuRxV9qR
l0n9Uz+b5WCbjRKDbgqSAeRdimRpeC90nnoBhVfyLxd2GLUQV0pw868BT3rQPVfy
9BcjKjR+9/oQJH9E8PnFYDYTO3trQ5KrGvm2P47Ksd1UHxjdfZMzArteNPY/mNup
sZyeexJcRAEeiF2oW20EdqxK5fMgAN2Jlk8C0kqYoXKtYYeFAoIBAQDwKBYE1P5d
TrPVET++sMOboTx/f6DAzmts5OaOTOlPnOoEduWieSRkV/BgJve/aWWtY5rma6O1
B7zL3WnpAWKduTdP/ce4TENeFL+7OaOAVkmufBd7UWHHi/wLMiWCiON2d7z3hvop
ik7gkj6EjSltuY89gyuXGg4E4AWekUlBpUJWu78e2gZKqqwKuLouGSYGVvzA4z+Q
CldV38DVYaDvoB5zuL5SaaIeMxcHNPC23Kbh6wlm16A9EU465t/NOHbbXIkitMXU
Thq8gHHlS3Z20/8nWKQIyG/c8RnQapxXVy5Kzq4qqEniqjpzmsMDg7roZ/VDigzt
zFxWiKY95DXFAoIBAQD4k7sWG8ysUNvqB7niWeGwwGu2JtsG8YQf3Y5qBkDj+Kt/
aRCYNKLwQVw+LZchc0LYPP9TmW54bM3Ip/8w7bKvAKLh2CLnSe/hrwnbZW25hxp1
x7tLsi1OZO+pQE0GSGjafLQ66l5Vv1hyR5JBCyye4hnStDdENMo06GmfXTruR1aY
y8U4CgnKlezC0guBV6lw25zTCJKKlx/abydeptsxUx2pJUc7bOii1Z7EsFc7mDVW
trNnOF/Ot3moiZhRishgIaWNKqHkoPe6GlVsaWKS4Zg/vuM9C0IxY/qX8DV8qSCP
TlUVE/7YZEbbemmqI6/APnfJagQkCTJqwcyU260NAoIBAQCom7w50EB1yhe/88hJ
XVk4Hs+WxEhclZAed+bW6XC0kHpMgB1hIKRIPuVo3HL8VEcqoM/UDqZ1JuWKS/1b
tblC+xK2enqKDiEOIIssK597Lxm0AzOJRO3dZ46hXaewKeNo/VASiRWBrLUMPiDe
dy0q+Ch1C27DO2u0SIQpe99u01PsCQx54KCiM4luTZuJWea0dMYntmSeAGSoydtc
9JpFIqm+nWoh+j/z6/KzAJwwBmK0IowLREAgH9JI0Gc4cfggsqslTr3F/MAQsC1G
yuB1NQni2fzwnajL+gohpXf2aiL6KnllQZn+JX28RF++S/Ih0TsFHBtPqTb7Gh19
PofxAoIBAQDS03TYUZaodsm/LY+YA0VvC+QnXpRrosly9j+KR81AkeixWJ3IrIMQ
RcN3SRM/TCgsL6cox3gGbF7efx+jWqssJEWwrMVXkb3FQJ4zTjjsiZLCWf4a55X0
1ZGtfukukXE46LVg/HOwTSlqpR6MAvoHo20B63YrDQ6T11CVusUxw5U+WOKEIRex
kbTCdnG+W33mGES0s4Mr2gRMXVY04yiXQ7kF7Q/8zUosMXWKBAX6UicZfAgdH57p
gVrSxS0l2R3MC/EDuvkX4FPxe0BymqQdTvQPqMpKQ1PTrfaOQT2cN/MeTqcYFd16
ID4bP/CvWcu3XBpKJvbS9kdLQuN7ynqa
-----END PRIVATE KEY-----