Elasticsearch Bulk index rejections

elasticsearch : 7.3.0
filebeat: 7.3.0
Kibana: 7.3.0
search-guard-7-7.3.0-37.0.0

Hi,

We have 4 node cluster, each node have 1 TB Hard disk, 32GB RAM (24GB assigned to elastic)

Issue No: 1

We are getting elastic index bulk rejection issues and it’s keep repeating, when i posted on elastic community one of the developer suggest me to check with Searchguard, someone please help me, Thanks.

[2019-10-01T15:10:40,013][ERROR][o.e.a.b.TransportBulkAction] [elastic1] failed to execute pipeline for a bulk request
org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of 
org.elasticsearch.ingest.IngestService$3@2b6e73db on EsThreadPoolExecutor[name = elastic1/write, queue 
capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6a1f1bd4[Running, pool 
size = 8, active threads = 8, queued tasks = 203, completed tasks = 43945401]]
at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:48) ~ . 
[elasticsearch-7.3.0.jar:7.3.0]
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355) ~[?:?]
at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:84) 
 [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.ingest.IngestService.executeBulkRequest(IngestService.java:333) [elasticsearch- 
7.3.0.jar:7.3.0]
at  org.elasticsearch.action.bulk.TransportBulkAction.processBulkIndexIngestRequest(TransportBulkAction.java:590) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:221) [elasticsearch-7.3.0.jar:7.3.0] at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:93) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:145) [elasticsearch-7.3.0.jar:7.3.0]
at com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:208) [search-guard-7-7.3.0-36.0.0.jar:7.3.0-36.0.0]
at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107) [search-guard-7-7.3.0-36.0.0.jar:7.3.0-36.0.0]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:143) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:121) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:392) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.client.support.AbstractClient.bulk(AbstractClient.java:470) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.xpack.core.ClientHelper.executeAsyncWithOrigin(ClientHelper.java:74) [x-pack-core-7.3.0.jar:7.3.0]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.doFlush(LocalBulk.java:105) [x-pack-monitoring-7.3.0.jar:7.3.0]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.flush(ExportBulk.java:63) [x-pack-monitoring-7.3.0.jar:7.3.0]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.lambda$doFlush$1(ExportBulk.java:107) [x-pack-monitoring-7.3.0.jar:7.3.0]
at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:102) [x-pack-core-7.3.0.jar:7.3.0]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.doFlush(ExportBulk.java:123) [x-pack-monitoring-7.3.0.jar:7.3.0]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.flush(ExportBulk.java:63) [x-pack-monitoring-7.3.0.jar:7.3.0]
at org.elasticsearch.xpack.monitoring.exporter.Exporters.doExport(Exporters.java:236) [x-pack-monitoring-7.3.0.jar:7.3.0]
at org.elasticsearch.xpack.monitoring.exporter.Exporters.lambda$export$3(Exporters.java:211) [x-pack-monitoring-7.3.0.jar:7.3.0]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-7.3.0.jar:7.3.0]atorg.elasticsearch.xpack.monitoring.exporter.Exporters$AccumulatingExportBulkActionListener.delegateIfComplete(Exporters.java:310) [x-pack-monitoring-7.3.0.jar:7.3.0]atorg.elasticsearch.xpack.monitoring.exporter.Exporters$AccumulatingExportBulkActionListener.onResponse(Exporters.java:289) [x-pack-monitoring-7.3.0.jar:7.3.0]atorg.elasticsearch.xpack.monitoring.exporter.Exporters$AccumulatingExportBulkActionListener.onResponse(Exporters.java:260) [x-pack-monitoring-7.3.0.jar:7.3.0]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalExporter.openBulk(LocalExporter.java:150) [x-pack-monitoring-7.3.0.jar:7.3.0]
at org.elasticsearch.xpack.monitoring.exporter.Exporters.wrapExportBulk(Exporters.java:197) [x-pack-monitoring-7.3.0.jar:7.3.0]
at org.elasticsearch.xpack.monitoring.exporter.Exporters.export(Exporters.java:209) [x-pack-monitoring-7.3.0.jar:7.3.0]atorg.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:252) [x-pack-monitoring-7.3.0.jar:7.3.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.0.jar:7.3.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]atorg.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) [elasticsearch-7.3.0.jar:7.3.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]

Issue No: 2:

Kibana UI showing whole DNs name of elastic cluster next to logout button.

Please provide your sg_config.yml, elasticsearch.yml and kibana.yml.