High Heap usage and GC with SearchGuard

Search Guard
1

After installing SearchGuard v5.6.9, the heap usage is 2X the normal heap usage on an average. Sometimes the heap usage is even 3X. It shows a saw tooth pattern; compared to a flat line observed before SearchGuard installation. Similar behaviour has been observed for GC count and time. This behaviour is seen only on Elasticsearch coordinating/load balancer node, the data node shows the same metrics for heap usage and GC as before. To be specific, we have 32GB of max heap configured for the jvm and before SearchGuard installation the average heap usage was 4GB, post SG installation, the average heap usage is 11GB and max heap usage is 23GB. Is this expected? Can a large number of clients connecting frequently(say 50 clients connecting to a single Elasticsearch coordinating node every 10seconds) cause this issue? I am yet to take a heap dump and analyse, with revert with details.

  • Search Guard and Elasticsearch version - 5.6.9

  • Installed and used enterprise modules, if any - No

  • JVM version and operating system version - 1.8, CentOS

  • Search Guard configuration files

  • Elasticsearch log messages on debug level

  • Other installed Elasticsearch or Kibana plugins, if any - None

2

No, this is not expected. However, you might have run into a serialization issue that would might explain the behavior you are seeing. This was fixed in v19.1 of Search Guard which is available for ES 5.6.10:

So the first thing you might want to do is to upgrade to the latest Search Guard version. If this does not help we need to look into your specific setup.

ยทยทยท

On Friday, August 31, 2018 at 2:59:52 AM UTC-4, Vinay Madyalkar wrote:

After installing SearchGuard v5.6.9, the heap usage is 2X the normal heap usage on an average. Sometimes the heap usage is even 3X. It shows a saw tooth pattern; compared to a flat line observed before SearchGuard installation. Similar behaviour has been observed for GC count and time. This behaviour is seen only on Elasticsearch coordinating/load balancer node, the data node shows the same metrics for heap usage and GC as before. To be specific, we have 32GB of max heap configured for the jvm and before SearchGuard installation the average heap usage was 4GB, post SG installation, the average heap usage is 11GB and max heap usage is 23GB. Is this expected? Can a large number of clients connecting frequently(say 50 clients connecting to a single Elasticsearch coordinating node every 10seconds) cause this issue? I am yet to take a heap dump and analyse, with revert with details.

  • Search Guard and Elasticsearch version - 5.6.9
  • Installed and used enterprise modules, if any - No
  • JVM version and operating system version - 1.8, CentOS
  • Search Guard configuration files
  • Elasticsearch log messages on debug level
  • Other installed Elasticsearch or Kibana plugins, if any - None
3

Thanks. I was thinking if there was a temporary workaround for this issue, as we did not want to upgrade immediately.