Please help with "failed to create shard" exception

ZillaYT · October 18, 2016, 1:32pm

ES v2.4.0
SG-SSL v2.4.0.16
SG v2.4.0.7

ES runs fine without SG (even with SG-SSL), but I get this when I install the SG plugin?

[2016-10-18 13:22:34,437][WARN ][indices.cluster ] [ip-10-22-9-4] [[searchguard][0]] marking and sending shard failed due to [failed to create shard]
java.lang.NullPointerException
at org.elasticsearch.index.shard.ShardPath.selectNewPathForShard(ShardPath.java:241)
at org.elasticsearch.index.IndexService.createShard(IndexService.java:336)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:620)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewOrUpdatedShards(IndicesClusterStateService.java:520)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:177)
at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:610)
at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-10-18 13:22:34,439][WARN ][cluster.action.shard ] [ip-10-22-9-4] [searchguard][0] received shard failed for target shard [[searchguard][0], node[uQKoE6NvRNaJQonOnP3Rew], [P], v[1], s[INITIALIZING], a[id=vnE3JztFSQC_G3gdkYWiNQ], unassigned_info[[reason=INDEX_CREATED], at[2016-10-18T13:22:34.380Z]]], indexUUID [0VEHCGcVTVKA-cj3l_qRfQ], message [failed to create shard], failure [NullPointerException[null]]

java.lang.NullPointerException

at org.elasticsearch.index.shard.ShardPath.selectNewPathForShard(ShardPath.java:241)

at org.elasticsearch.index.IndexService.createShard(IndexService.java:336)

at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:620)

at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewOrUpdatedShards(IndicesClusterStateService.java:520)

at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:177)

at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:610)

at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)

at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)

at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

``

ZillaYT · October 18, 2016, 2:09pm

This turned out to be an AWS security group issue with port 9300 being locked. Once our AWS admin added a rule to allow port 9300 access to our AWS VPC, the cluster came up.

···

On Tuesday, October 18, 2016 at 9:32:52 AM UTC-4, ZillaYT wrote:

[2016-10-18 13:22:34,439][WARN ][cluster.action.shard ] [ip-10-22-9-4] [searchguard][0] received shard failed for target shard [[searchguard][0], node[uQKoE6NvRNaJQonOnP3Rew], [P], v[1], s[INITIALIZING], a[id=vnE3JztFSQC_G3gdkYWiNQ], unassigned_info[[reason=INDEX_CREATED], at[2016-10-18T13:22:34.380Z]]], indexUUID [0VEHCGcVTVKA-cj3l_qRfQ], message [failed to create shard], failure [NullPointerException[null]]