Odd issue with sg-auditlog entries being created incorrectly

Search Guard
1

I am having a problem with my searchguard node (it is a single node only instance) where my auditlogs are being created with what looks like an attempt to write duplicate data to my node, creating unassigned nodes and dropping my es node to a yellow state. How can I fix this?

Below is the output of the following command

curl -XGET https://127.0.0.1:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED

{

“index” : “sg6-auditlog-2018.06.04”,

“shard” : 3,

“primary” : false,

“current_state” : “unassigned”,

“unassigned_info” : {

“reason” : “INDEX_CREATED”,

“at” : “2018-06-04T13:19:11.824Z”,

“last_allocation_status” : “no_attempt”

},

“can_allocate” : “no”,

“allocate_explanation” : “cannot allocate because allocation is not permitted to any of the nodes”,

“node_allocation_decisions” : [

{

“node_id” : “UokknJlMSYeJcwcJIA8oNQ”,

“node_name” : “UokknJl”,

“transport_address” : “127.0.0.1:9300”,

“node_decision” : “no”,

“weight_ranking” : 1,

“deciders” : [

{

“decider” : “same_shard”,

“decision” : “NO”,

“explanation” : “the shard cannot be allocated to the same node on which a copy of the shard already exists [[sg6-auditlog-2018.06.04][3], node[UokknJlMSYeJcwcJIA8oNQ], [P], s[STARTED], a[id=UZ7zsqsxRoGEEqqKBtLQOg]]”

}

]

}

]

}

sg6-auditlog-2018.06.04 3 r UNASSIGNED INDEX_CREATED

sg6-auditlog-2018.06.04 4 r UNASSIGNED INDEX_CREATED

sg6-auditlog-2018.06.04 1 r UNASSIGNED INDEX_CREATED

sg6-auditlog-2018.06.04 2 r UNASSIGNED INDEX_CREATED

sg6-auditlog-2018.06.04 0 r UNASSIGNED INDEX_CREATED

curl -H ‘Content-Type: application/json’ -XPUT ‘https://127.0.0.1:9200/sg6-auditlog-*/_settings’ -d ‘{“number_of_replicas”:0}’

curl -XDELETE ‘https://127.0.0.1:9200/sg6-auditlog*’

2

That is not related to Search Guard or auditlog.

If you have only one node and one or more indices are configured with one or more replicas then its always yellow because replicas can not be assigned.

In other words: With a one node cluster you must not have replicas. By default all indices in ES (also the auditlog index) are defined to have one replica.

Turn them off with:

curl -Ssk -u admin:admin -X PUT “https://localhost:9200/_settings” -H ‘Content-Type: application/json’ -d’

{

“index” : {

“number_of_replicas” : 0

}

}

’

···

On Monday, 4 June 2018 15:31:02 UTC+2, erik clark wrote:

I am having a problem with my searchguard node (it is a single node only instance) where my auditlogs are being created with what looks like an attempt to write duplicate data to my node, creating unassigned nodes and dropping my es node to a yellow state. How can I fix this?

Below is the output of the following command

curl -XGET https://127.0.0.1:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED

{

“index” : “sg6-auditlog-2018.06.04”,

“shard” : 3,

“primary” : false,

“current_state” : “unassigned”,

“unassigned_info” : {

“reason” : “INDEX_CREATED”,

“at” : “2018-06-04T13:19:11.824Z”,

“last_allocation_status” : “no_attempt”

},

“can_allocate” : “no”,

“allocate_explanation” : “cannot allocate because allocation is not permitted to any of the nodes”,

“node_allocation_decisions” : [

{

“node_id” : “UokknJlMSYeJcwcJIA8oNQ”,

“node_name” : “UokknJl”,

“transport_address” : “127.0.0.1:9300”,

“node_decision” : “no”,

“weight_ranking” : 1,

“deciders” : [

{

“decider” : “same_shard”,

“decision” : “NO”,

“explanation” : “the shard cannot be allocated to the same node on which a copy of the shard already exists [[sg6-auditlog-2018.06.04][3], node[UokknJlMSYeJcwcJIA8oNQ], [P], s[STARTED], a[id=UZ7zsqsxRoGEEqqKBtLQOg]]”

}

]

}

]

}

sg6-auditlog-2018.06.04 3 r UNASSIGNED INDEX_CREATED

sg6-auditlog-2018.06.04 4 r UNASSIGNED INDEX_CREATED

sg6-auditlog-2018.06.04 1 r UNASSIGNED INDEX_CREATED

sg6-auditlog-2018.06.04 2 r UNASSIGNED INDEX_CREATED

sg6-auditlog-2018.06.04 0 r UNASSIGNED INDEX_CREATED

curl -H ‘Content-Type: application/json’ -XPUT ‘https://127.0.0.1:9200/sg6-auditlog-*/_settings’ -d ‘{“number_of_replicas”:0}’

curl -XDELETE ‘https://127.0.0.1:9200/sg6-auditlog*’

3

No need to delete the index afterwards!

···

On Monday, 4 June 2018 15:48:11 UTC+2, Search Guard wrote:

That is not related to Search Guard or auditlog.

If you have only one node and one or more indices are configured with one or more replicas then its always yellow because replicas can not be assigned.

In other words: With a one node cluster you must not have replicas. By default all indices in ES (also the auditlog index) are defined to have one replica.

Turn them off with:

curl -Ssk -u admin:admin -X PUT “https://localhost:9200/_settings” -H ‘Content-Type: application/json’ -d’

{

“index” : {

“number_of_replicas” : 0

}

}

’

https://www.elastic.co/guide/en/elasticsearch/reference/6.2/indices-update-settings.html

On Monday, 4 June 2018 15:31:02 UTC+2, erik clark wrote:

I am having a problem with my searchguard node (it is a single node only instance) where my auditlogs are being created with what looks like an attempt to write duplicate data to my node, creating unassigned nodes and dropping my es node to a yellow state. How can I fix this?

Below is the output of the following command

curl -XGET https://127.0.0.1:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED

{

“index” : “sg6-auditlog-2018.06.04”,

“shard” : 3,

“primary” : false,

“current_state” : “unassigned”,

“unassigned_info” : {

“reason” : “INDEX_CREATED”,

“at” : “2018-06-04T13:19:11.824Z”,

“last_allocation_status” : “no_attempt”

},

“can_allocate” : “no”,

“allocate_explanation” : “cannot allocate because allocation is not permitted to any of the nodes”,

“node_allocation_decisions” : [

{

“node_id” : “UokknJlMSYeJcwcJIA8oNQ”,

“node_name” : “UokknJl”,

“transport_address” : “127.0.0.1:9300”,

“node_decision” : “no”,

“weight_ranking” : 1,

“deciders” : [

{

“decider” : “same_shard”,

“decision” : “NO”,

“explanation” : “the shard cannot be allocated to the same node on which a copy of the shard already exists [[sg6-auditlog-2018.06.04][3], node[UokknJlMSYeJcwcJIA8oNQ], [P], s[STARTED], a[id=UZ7zsqsxRoGEEqqKBtLQOg]]”

}

]

}

]

}

sg6-auditlog-2018.06.04 3 r UNASSIGNED INDEX_CREATED

sg6-auditlog-2018.06.04 4 r UNASSIGNED INDEX_CREATED

sg6-auditlog-2018.06.04 1 r UNASSIGNED INDEX_CREATED

sg6-auditlog-2018.06.04 2 r UNASSIGNED INDEX_CREATED

sg6-auditlog-2018.06.04 0 r UNASSIGNED INDEX_CREATED

curl -H ‘Content-Type: application/json’ -XPUT ‘https://127.0.0.1:9200/sg6-auditlog-*/_settings’ -d ‘{“number_of_replicas”:0}’

curl -XDELETE ‘https://127.0.0.1:9200/sg6-auditlog*’