Potential memory issue related to zjsonpatch library

Search Guard
1

Elasticsearch version: 6.5.4

Server OS version: CentOS

Describe the issue:

We are getting OOM Error and after analyzing heap dump we believe that this is directly related with problem
that zjsonpatch library (version 0.4.4) has and which was described in Memory issue with 0.4.0+ · Issue #77 · flipkart-incubator/zjsonpatch · GitHub and fixed in version 0.4.5.

Is newer version of Search Guard plugin use different version of zjsonpatch (>=0.4.5) or code was adjusted in a way that this problem no longer exists?

Provide logs:

[2021-03-12T07:16:35,559][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [data-14] fatal error in thread [elasticsearch[data-14][write][T#16]], exiting
java.lang.OutOfMemoryError: Java heap space
        at com.flipkart.zjsonpatch.InternalUtils.longestCommonSubsequence(InternalUtils.java:31) ~[?:?]
        at com.flipkart.zjsonpatch.JsonDiff.getLCS(JsonDiff.java:446) ~[?:?]
        at com.flipkart.zjsonpatch.JsonDiff.compareArray(JsonDiff.java:337) ~[?:?]
        at com.flipkart.zjsonpatch.JsonDiff.generateDiffs(JsonDiff.java:324) ~[?:?]
        at com.flipkart.zjsonpatch.JsonDiff.compareObjects(JsonDiff.java:425) ~[?:?]
        at com.flipkart.zjsonpatch.JsonDiff.generateDiffs(JsonDiff.java:327) ~[?:?]
        at com.flipkart.zjsonpatch.JsonDiff.compareObjects(JsonDiff.java:425) ~[?:?]
        at com.flipkart.zjsonpatch.JsonDiff.generateDiffs(JsonDiff.java:327) ~[?:?]
        at com.flipkart.zjsonpatch.JsonDiff.compareObjects(JsonDiff.java:425) ~[?:?]
        at com.flipkart.zjsonpatch.JsonDiff.generateDiffs(JsonDiff.java:327) ~[?:?]
        at com.flipkart.zjsonpatch.JsonDiff.asJson(JsonDiff.java:47) ~[?:?]
        at com.flipkart.zjsonpatch.JsonDiff.asJson(JsonDiff.java:38) ~[?:?]
        at com.floragunn.searchguard.auditlog.impl.AbstractAuditLog.logDocumentWritten(AbstractAuditLog.java:571) ~[?:?]
        at com.floragunn.searchguard.auditlog.impl.AuditLogImpl.logDocumentWritten(AuditLogImpl.java:197) ~[?:?]
        at com.floragunn.searchguard.compliance.ComplianceIndexingOperationListenerImpl.postIndex(ComplianceIndexingOperationListenerImpl.java:157) ~[?:?]
        at org.elasticsearch.index.shard.IndexingOperationListener$CompositeListener.postIndex(IndexingOperationListener.java:107) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:777) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:741) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:705) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$3(TransportShardBulkAction.java:461) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda$3734/0x0000000800ecec40.get(Unknown Source) ~[?:?]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:483) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:459) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:216) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:159) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:151) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:139) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:79) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1022) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1000) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:102) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:356) ~[elasticsearch-6.5.4.jar:6.5.4]

image
image899×268 47.9 KB

image
image621×547 102 KB

Please advise.

Thanks,
Daniel

2

Hello Daniel!

The problem was not known so far. I filed an issue, you can track it here:

How did you encounter the issue? Is a specific JSON necessary?

3

Hello Nils,

Thank you for creating gitlab issue! We are sending huge number of documents and their size differ. Currently, we do not have environment that we can use to reproduce that issue in order to find out which document may cause that problem.
How likely it is that the fix will be back ported to older plugin?
Is there is any way that we can fix it by ourselves by swapping JAR file or recompiling SG?
Do you think that defining searchguard.compliance.history.read.watched_fields is going to help in this case as we are going to watch less number of fields?

Thanks,
Daniel

4

Hello Daniel,

I have to admit, as all SG versions for ES 6.5.x have reached end of life since quite a while, it is quite unlikely that there will be a patch release.

However, if zjsonpatch 0.4.5 just fixes bugs and does not change APIs and contracts, it should be indeed possible to just replace the JAR file:

Of course, you’d need to do your own testing. Also, you should be conservative regarding the newer version, i.e. really just choose 0.4.5 if it fixes the problem and not any newer versions of zjsonpatch.

Hope that helps.

5

Hello Nils,

Thank you for that instruction, it was tested and it work.
We found out that we have searchguard.compliance.history.write.log_diffs option enabled and based on code analyze we can say that by disabling it we gonna avoid running zjsonpatch.asJson method, so for now we are going to set this option to false.

Thanks,
Daniel

6

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.