No OID or searchguard.nodes_dn incorrect configured

Search Guard
1

Hi,

Context -
I have used the example scripts at Sample PKI scripts | Security for Elasticsearch | Search Guard to generate TLS certificates for searchguard.

I deploy ELK via helm charts in kubernetes environment as pods. Earlier, I always generated certs with default OID (from the example script) and things worked fine.
Now, my certificates cannot have an OID field and so, i intend to use the property searchguard.nodes_dn to list the DNs of the nodes.

I am trying to add this property as an environment variable to the elasticsearch pod in the helm chart. Because using the docker-entrypoint.sh (https://github.com/elastic/elasticsearch/blob/7.0/distribution/docker/src/docker/bin/docker-entrypoint.sh), all envs (in a format) get added to es_opts passed to the elasticsearch process.
I have been successful in configuring other searchguard parameters like searchguard.ssl.http.enabled_ciphers too as env to pods and it gets reflected properly.

Issue -
My keystore.jks is something like this - (It contains node certificate and the chain of CAs signing it):

Certificate[1]:
Owner: CN=elasticsearch.shiv1, C=ELK
Issuer: CN=Example Com Inc. Signing CA, OU=Example Com Inc. Signing CA, O=Example Com Inc., DC=example, DC=com
..
Certificate[2]:
Owner: CN=Example Com Inc. Signing CA, OU=Example Com Inc. Signing CA, O=Example Com Inc., DC=example, DC=com
Issuer: CN=elasticsearch. Root CA, DC=elasticsearch
..
Certificate[3]:
Owner: CN=elasticsearch. Root CA, DC=elasticsearch
Issuer: CN=elasticsearch. Root CA, DC=elasticsearch

Now, i am trying to configure nodes_dn as an env in the pods like:

  - name: "searchguard.nodes_dn"
    value: "CN=elasticsearch.shiv1,C=ELK"

When i check the running process, this property does appear -

elastic+ 70 13 44 05:12 ? 00:00:35 /etc/alternatives/jre_openjdk//bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC ... -Des.path.conf=/etc/elasticsearch -Esearchguard.nodes_dn=CN=elasticsearch.shiv1,C=ELK

But I still get this error in elasticsearch logs and cluster doesnt come up -

“logger”:“c.f.s.t.SearchGuardRequestHandler”,“timezone”:“UTC”,“log”:“ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or searchguard.nodes_dn incorrect configured) or that someone
> is spoofing requests. Check your TLS certificate setup as described here: See TLS help | Security for Elasticsearch | Search Guard]”}

When i add this parameter to elasticsearch.yml file directly, with the same key value pair, it works fine and cluster formation happens as expected.

Could you help me understand why this particular parameter refuses to get accepted when added as an env? Or if there is a particular way of setting this parameter?

Version info - ELK + searchguard : 7.0.1

Thanks,
Shivani

2

Hi.
Can you share the file where you set the env variables?

3

Also, please provide the following files:

  1. kibana.yml
  2. elasticsearch.yml
  3. elasticsearch/plugins/search-guard-7/sgconfig/sg_config.yml
4

Would the ES command line parser be able to parse this properly with all the = mixed into the string? Maybe this is the problem.

... Esearchguard.nodes_dn=CN=elasticsearch.shiv1,C=ELK

Can you share the full command you use to run Elasticsearch? I wonder if you have options with similar syntax there.

5

Hi,
Attaching required files:

  1. elasticsearch.yml elasticsearch.yml (1.4 KB)
  2. sg_config.yaml sg_config.yml (1.7 KB)
  3. Docker entrypoint which starts elasticsearch process in the pod - docker-entrypoint.sh (4.5 KB)
  4. List of environment variables set in the pod - env-list.txt (3.0 KB)
    ( I have not installed kibana yet)

The file elasticsearch.yml is fixed in my case. The file is packaged in the docker image. As you can see, most of the parameters(keys) in elasticsearch.yml read their values from environment variables. So, depending on what environment variables are changed for the pod at run-time, the elasticsearch.yml properties would get configured accordingly and get used for the ES process.
Acc to docker-entrypoint.sh, other envs in the pod that have at least two dot separated lowercase words, they get passed to the ES process too.

Environment variables are set for the container (pod) in the helm chart template. For ex.

containers:
- name: es-client
image: {{.Values.global.registry}}/{{.Values.searchguard.image.repo}}:{{.Values.searchguard.image.tag}}
imagePullPolicy: {{.Values.elasticsearch_master.ImagePullPolicy}}
env:
- name: “CLUSTER_NAME”
value: {{ printf “%s-%s” .Release.Namespace .Release.Name | quote }}
- name: “NODE_MASTER”
value: “false”
- name: “AUTH_ADMIN_DN”
value: “{{.Values.searchguard.auth_admin_identity}}”
- name: “SG_CONFIG_DIRECTORY”
value: “/usr/share/elasticsearch/plugins/search-guard-7/sgconfig/”
- name: “searchguard.nodes_dn”
value: “CN=elasticsearch.shiv1,C=ELK”

The whole output of the running process -

elastic+    70    13  2 May01 ?        01:08:40 /etc/alternatives/jre_openjdk//bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch-17200160406598074423 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Djava.locale.providers=COMPAT -Dio.netty.allocator.type=unpooled -Des.cgroups.hierarchy.override=/ -Xms1g -Xmx1g -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=oss -Des.distribution.type=rpm -Des.bundled_jdk=true -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -Esearchguard.nodes_dn=CN=elasticsearch.shiv1,C=ELK
elastic+   173   150  0 10:03 pts/0    00:00:00 grep --color=auto 70

I suspect this could be an issue as there are no other properties with “=” here. I tried using escape characters too so that the process would read it as

Esearchguard.nodes_dn="CN=elasticsearch.shiv1,C=ELK"

But that did not work either. Any suggestions?

Thanks,
Shivani

6

But I don’t see other searchguard options among the process options, only searchguard.nodes_dn. This -E option, I can’t find it in the docs, could you please give me a link for the docs where this option is described?

7

This -E option…

I meant the -E prefix here -Esearchguard.nodes_dn=CN=elasticsearch.shiv1,C=ELK

8

I had not configured any more searchguard properties other than nodes_dn in the output i had shared.
But just to show that I have been successful in configuring other properties like searchguard.ssl.http.enabled_ciphers:

elastic+ 70 13 9 02:40 ? 00:00:31 /etc/alternatives/jre_openjdk//bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch-14668466814958942139 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Djava.locale.providers=COMPAT -Dio.netty.allocator.type=unpooled -Des.cgroups.hierarchy.override=/ -Djava.net.preferIPv4Stack=true -Xms2g -Xmx2g -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=oss -Des.distribution.type=rpm -Des.bundled_jdk=true -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -Esearchguard.ssl.http.enabled_ciphers=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

I don’t know about where -E prefix would be documented. If you can look through the docker-entrypoint.sh file shared, (it is taken from https://github.com/elastic/elasticsearch/blob/7.0/distribution/docker/src/docker/bin/docker-entrypoint.sh), it loops through all the environment variables that have at least two dot separated lowercase words, and they get passed to the ES process (/usr/share/elasticsearch/bin/elasticsearch) as es_opts.

9

Today I run some SearchGuard dockers and found that it is definitely possible to have = sign in the Elasticsearch argument value. For example, look at the option -Esearchguard.ssl.transport.keystore_filepath in this running process:


elastic+ 16 3.7 18.2 4811864 1490352 ? Sl May06 22:22 /usr/lib/jvm/java-8-oracle/jre//bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -XX:+HeapDumpOnOutOfMemoryError -Dsun.security.krb5.rcache=none -Dsg.display_lic_none=true -Dcom.sun.jndi.ldap.object.disableEndpointIdentification=true -ea -Xms1g -Xmx1g -Dsun.security.krb5.debug=false -Dsun.security.spnego.debug=false -Djava.security.debug=none -Djava.security.auth.debug=none -Dio.netty.allocator.type=unpooled -XX:MaxDirectMemorySize=536870912 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=deb -Des.bundled_jdk=true -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -Enetwork.host=sgssl-0.example.com -Esearchguard.ssl.transport.keystore_filepath=CN=sgssl-0.example.com,OU=SSL,O=Test,L=Test,C=DE-keystore.jks -Esearchguard.ssl.http.keystore_filepath=CN=sgssl-0.example.com,OU=SSL,O=Test,L=Test,C=DE-keystore.jks -Esearchguard.kerberos.acceptor_principal=HTTP/sgssl-0.example.com -Esearchguard.kerberos.acceptor_keytab_filepath=sgssl-0.example.com.http_srv.keytab -Enode.name=sgssl-0.example.com -Ecluster.initial_master_nodes=172.16.0.1 -Ediscovery.seed_hosts=172.16.0.1 -Esearchguard.ssl.transport.enable_openssl_if_available=false -Esearchguard.ssl.http.enable_openssl_if_available=false

Let’s get back to the initial data provided by you. It seems the searchguard.nodes_dn value is wrong somewhere.

That’s why you get this error:

Are you sure you have searchguard.nodes_dn set in each node of the cluster? Recheck it.

10

@srgbnd, Thanks for the analysis.
But, I have ensured & re-checked that searchguard.nodes_dn parameter is set in each node of the cluster.

I have tried configuring it in following 3 ways, but I get the same error:
-Esearchguard.nodes_dn="CN=elasticsearch.shiv1,C=ELK"
-Esearchguard.nodes_dn=CN=elasticsearch.shiv1,C=ELK
-Esearchguard.nodes_dn=CN=elasticsearch.shiv1, C=ELK

My certificate looks like this -

$ keytool -list -v -keystore keystore.jks -storepass changeit
Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: elasticsearch
Creation date: Apr 29, 2020
Entry type: PrivateKeyEntry
Certificate chain length: 3
Certificate[1]:
Owner: CN=elasticsearch.shiv1, C=ELK
Issuer: CN=Example Com Inc. Signing CA, OU=Example Com Inc. Signing CA, O=Example Com Inc., DC=example, DC=com
Serial number: 1
Valid from: Wed Apr 29 03:29:34 UTC 2020 until: Fri Apr 29 03:29:34 UTC 2022
Certificate fingerprints:
SHA1: 02:FC:35:16:DC:E3:FC:A3:06:97:74:30:7E:90:9B:52:62:53:EA:19
SHA256: 7E:23:80:CD:DE:D3:5D:01:22:8D:D5:49:EC:C5:A9:84:8B:05:3E:FC:DD:D3:05:64:17:14:89:3F:90:90:B9:65
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 5C 60 A2 3D B0 3E 86 BD B6 01 53 93 EF 50 F4 CF `.=.>…S…P…
0010: 0B 1E 50 F9 …P.
]
]

#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
CA:false
PathLen: undefined
]

#3: ObjectId: 2.5.29.31 Criticality=false
CRLDistributionPoints [
[DistributionPoint:
[URIName: https://raw.githubusercontent.com/floragunncom/unittest-assets/master/revoked.crl]
]]

#4: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
serverAuth
clientAuth
]

#5: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
DigitalSignature
Key_Encipherment
]

#6: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
DNSName: elasticsearch.shiv1
DNSName: localhost
IPAddress: 127.0.0.1
]

#7: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: FD 39 E6 75 5D 51 31 FD 44 41 88 84 1E 6B 5E 3D .9.u]Q1.DA…k^=
0010: C0 E6 E9 48 …H
]
]

Am I missing something obvious here? Pls help.

11

Did you always have C=ELK? It looks wrong. The values must be of size 2 and a valid ISO 3166 Country Code.

Look, I try to create a certificate with the country name of size > 2 and have the validation error

$ openssl req -new -sha256 -key clientB-key.pem -out clientB-csr.pem
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) []:ELK
string is too long, it needs to be less than  2 bytes long
12

Yes, i always have C=ELK.
I am generating certificates using the example scripts in Sample PKI scripts | Security for Elasticsearch | Search Guard (in the folder example-pki-scripts).
Here, the script gen_node_cert.sh defines the certificate DN as DN=“CN=$NODE_NAME.logging,C=ELK”.
I have just modified this script & removed the oid part from certificate’s SAN -

-ext san=dns:$NODE_NAME.logging,dns:localhost,ip:127.0.0.1,oid:1.2.3.4.5.5
##changed to
-ext san=dns:$NODE_NAME.logging,dns:localhost,ip:127.0.0.1

And changed the DN to DN=“CN=$NODE_NAME.shiv1,C=ELK”.

13

I will try to reproduce this soon and let you know. Meanwhile, can you try to set values of size 2 for the country, for example C=NL?

14

Apologies for the delay!
I generated certificates with CN=elasticsearch.shiv1,C=IN. This did not work either and threw the same error - no OID or searchguard.nodes_dn incorrect configured.

15

Ok, I can reproduce your issue in my environment.

But I can’t find a way to provide the searchguard.nodes_dn value correctly if it is an environment variable. The option expects a value of type array. Arrays can be provided to Elasticsearch as a string concatenated with a comma “,”. For Elasticsearch CN=elasticsearch.shiv1,C=ELK means ["CN=elasticsearch.shiv1", "C=ELK"]. I tried to escape the comma value: "CN=elasticsearch.shiv1\,C=ELK" but it didn’t work.

Now, my certificates cannot have an OID field and so

Why can’t you use OID? if you want to keep certificate auth values as an environment variable this will work for you.

16

A brief update to this:

This can be solved by using the fact that RFC 2253 allows semicolons as an alternative separator for RDNs inside DNs.

So, just configure your DNs using semicolons instead of commas. Thus, you are unaffected by Elasticsearch splitting lists on commas.