Memory utilisation of elasticsearch-master is growing continuously when searchguard is enabled in ELK-7

[chaitra_hegde]

Hi,
I am using ELK 7.0.1 with SG-7.0.1 plugin. I am using it in k8s(helm) environment.
When i see the memory utilization of elasticsearch-master pod, it is growing continuously causing the pod OOM-killed.
The same setup when SG is disabled works fine.
The issue is same with openjdk 8 and openjdk 11.
I have used the below parameter for setting jvm.
es_java_opts: “-Djava.net.preferIPv4Stack=true -Xms1g -Xmx1g”

I see the “heap.percent” utilization for master is only 10-15% but the “ram.percent” is reaching to 95-99%.

So how can I bring down the memory consumption?

[mikew]

I’ve not yet tried Search Guard with Elasticsearch 7 so this may be totally off, but are you using openSSL?

When I first started using Search Guard I enabled openSSL because the documentation recommends it. The memory usage on nodes kept growing until they crashed. I disabled openSSL and the problem went away. (Irritatingly the problem didn’t occur on our test cluster when Search Guard was implemented on that, presumably due to the vastly smaller amount of data on that.) Last time I looked it seemed like the bug was fixed, so I tried re-enabling openSSL but the problem re-occurred so I disabled openSSL again. I can’t just find links to the bug in question so maybe it is now fixed and openSSL is not the problem. But if you are using openSSL try setting

searchguard.ssl.transport.enable_openssl_if_available: false
searchguard.ssl.http.enable_openssl_if_available: false

in elasticsearch.yml and if you have set it also remove

searchguard.ssl.http.enabled_ciphers:
  - whatever
  - values
  - you
  - have
  - listed

[cstaley]

Can you try with -Xms4g -Xmx4g and -Xms2g -Xmx2g

For further diagnostigs it would also be helpful if you can send the out of Nodes hot threads API | Elasticsearch Guide [8.4] | Elastic (of the node in question).