ElasticsearchSecurityException[password does not match]

Search Guard
1

I am getting an exception on setting up Elasticsearch cluster:
[2017-04-12T11:25:52,374][ERROR][c.f.s.a.BackendRegistry ] Unexpected exception com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]

[2017-04-12T11:25:52,374][INFO ][c.f.s.a.BackendRegistry ] Cannot authenticate user (or add roles) with ad 4 due to ElasticsearchSecurityException[com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]]; nested: UncheckedExecutionException[ElasticsearchSecurityException[password does not match]]; nested: ElasticsearchSecurityException[password does not match];, try next

I have a 6 node setup (2 each on master, client and data).

I am guessing this error is related to the internal users that I have setup on sgconfig.

I am generating the hash (using hash.bat tool) and putting it in the internal_users config.

Each node generates its own hash from the same password and therefore each hash is different. Is this is what is causing this exception?

I am not able to access the nodes either with the configured password.

Any help on this would be awesome.

Regards,

Amar

2

pls. share your sg_internal_users.yml and the console output when you run sgadmin

···

On Wednesday, 12 April 2017 20:42:39 UTC+2, Amar Kumar Dubedy wrote:

I am getting an exception on setting up Elasticsearch cluster:
[2017-04-12T11:25:52,374][ERROR][c.f.s.a.BackendRegistry ] Unexpected exception com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]

[2017-04-12T11:25:52,374][INFO ][c.f.s.a.BackendRegistry ] Cannot authenticate user (or add roles) with ad 4 due to ElasticsearchSecurityException[com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]]; nested: UncheckedExecutionException[ElasticsearchSecurityException[password does not match]]; nested: ElasticsearchSecurityException[password does not match];, try next

I have a 6 node setup (2 each on master, client and data).

I am guessing this error is related to the internal users that I have setup on sgconfig.

I am generating the hash (using hash.bat tool) and putting it in the internal_users config.

Each node generates its own hash from the same password and therefore each hash is different. Is this is what is causing this exception?

I am not able to access the nodes either with the configured password.

Any help on this would be awesome.

Regards,

Amar

3

Here is my sg_internal_users.config on one master

This is the internal user database

The hash value is a bcrypt hash and can be generated with plugin/tools/hash.sh

offerstoreadmin:

hash: $2a$12$1eHRCRUAf5s15DeP.NmDEeTkAVCy/GMryo4REnUKVryN5O9LieM0i

kibanaserver:

hash: $2a$12$1eHRCRUAf5s15DeP.NmDEeTkAVCy/GMryo4REnUKVryN5O9LieM0i

kibanaro:

hash: $2a$12$1eHRCRUAf5s15DeP.NmDEeTkAVCy/GMryo4REnUKVryN5O9LieM0i

Here is the sg_internal_users config on another master:

This is the internal user database

The hash value is a bcrypt hash and can be generated with plugin/tools/hash.sh

offerstoreadmin:

hash: $2a$12$9hP/4.Lzd6K1tROpIPUBJukOQ.1SODbdYF/tHTSy6GxsWRR8Yw3Pi

kibanaserver:

hash: $2a$12$9hP/4.Lzd6K1tROpIPUBJukOQ.1SODbdYF/tHTSy6GxsWRR8Yw3Pi

kibanaro:

hash: $2a$12$9hP/4.Lzd6K1tROpIPUBJukOQ.1SODbdYF/tHTSy6GxsWRR8Yw3Pi

Here is the sgadmin output on both machines.

04-12-2017 11:21:42EAP010043044016sgadmin error: at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:201)

04-12-2017 11:21:42EAP010043044016sgadmin error: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

04-12-2017 11:21:42EAP010043044016sgadmin error: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

04-12-2017 11:21:42EAP010043044016sgadmin error: at java.lang.Thread.run(Thread.java:745)

04-12-2017 11:21:43EAP010043044016sgadmin ran

04-12-2017 11:21:43EAP010043044019sgadmin output: searchguard index does not exists, attempt to create it … ERR: An unexpected ResourceAlreadyExistsException occured: index [searchguard/UJ4NmKegR9ednImF6rXoLw] already exists

04-12-2017 11:21:43EAP010043044019sgadmin output: Trace:

04-12-2017 11:21:43EAP010043044019sgadmin error: [searchguard/UJ4NmKegR9ednImF6rXoLw] ResourceAlreadyExistsException[index [searchguard/UJ4NmKegR9ednImF6rXoLw] already exists]

04-12-2017 11:21:43EAP010043044019sgadmin error: at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.validateIndexName(MetaDataCreateIndexService.java:139)

04-12-2017 11:21:43EAP010043044019sgadmin error: at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.validate(MetaDataCreateIndexService.java:479)

04-12-2017 11:21:43EAP010043044019sgadmin error: at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.access$000(MetaDataCreateIndexService.java:103)

04-12-2017 11:21:43EAP010043044019sgadmin error: at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$1.execute(MetaDataCreateIndexService.java:228)

04-12-2017 11:21:43EAP010043044019sgadmin error: at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45)

04-12-2017 11:21:43EAP010043044019sgadmin error: at org.elasticsearch.cluster.service.ClusterService.runTasksForExecutor(ClusterService.java:581)

04-12-2017 11:21:43EAP010043044019sgadmin error: at org.elasticsearch.cluster.service.ClusterService$UpdateTask.run(ClusterService.java:920)

04-12-2017 11:21:43EAP010043044019sgadmin error: at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458)

04-12-2017 11:21:43EAP010043044019sgadmin error: at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:201)

04-12-2017 11:21:43EAP010043044019sgadmin error: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

04-12-2017 11:21:43EAP010043044019sgadmin error: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

04-12-2017 11:21:43EAP010043044019sgadmin error: at java.lang.Thread.run(Thread.java:745)

04-12-2017 11:21:43EAP010043044019sgadmin ran

04-12-2017 11:49:01EAP010043044019sgadmin output: Search Guard Admin v5

04-12-2017 11:49:01EAP010043044019sgadmin output: WARNING: Seems you want connect to the a HTTP port.

04-12-2017 11:49:01EAP010043044019sgadmin output: sgadmin connect through the transport port which is normally 9300.

04-12-2017 11:49:01EAP010043044019sgadmin output: Will connect to ...:8900 … done

04-12-2017 11:49:05EAP010043044019sgadmin output: Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …

04-12-2017 11:49:31EAP010043044019sgadmin output: Clustername: ClusterName

04-12-2017 11:49:31EAP010043044019sgadmin output: Clusterstate: GREEN

04-12-2017 11:49:31EAP010043044019sgadmin output: Number of nodes: 6

04-12-2017 11:49:31EAP010043044019sgadmin output: Number of data nodes: 2

04-12-2017 11:49:31EAP010043044019sgadmin output: searchguard index already exists, so we do not need to create one.

04-12-2017 11:49:31EAP010043044019sgadmin output: Populate config from C:\elasticsearch512xx\elasticsearch\plugins\search-guard-5\sgconfig\

04-12-2017 11:49:31EAP010043044019sgadmin output: Will update ‘config’ with C:\elasticsearch512xx\elasticsearch\plugins\search-guard-5\sgconfig\sg_config.yml

04-12-2017 11:49:32EAP010043044019sgadmin output: SUCC: Configuration for ‘config’ created or updated

04-12-2017 11:49:32EAP010043044019sgadmin output: Will update ‘roles’ with C:\elasticsearch512xx\elasticsearch\plugins\search-guard-5\sgconfig\sg_roles.yml

04-12-2017 11:49:32EAP010043044019sgadmin output: SUCC: Configuration for ‘roles’ created or updated

04-12-2017 11:49:32EAP010043044019sgadmin output: Will update ‘rolesmapping’ with C:\elasticsearch512xx\elasticsearch\plugins\search-guard-5\sgconfig\sg_roles_mapping.yml

04-12-2017 11:49:32EAP010043044019sgadmin output: SUCC: Configuration for ‘rolesmapping’ created or updated

04-12-2017 11:49:32EAP010043044019sgadmin output: Will update ‘internalusers’ with C:\elasticsearch512xx\elasticsearch\plugins\search-guard-5\sgconfig\sg_internal_users.yml

04-12-2017 11:49:32EAP010043044019sgadmin output: SUCC: Configuration for ‘internalusers’ created or updated

04-12-2017 11:49:32EAP010043044019sgadmin output: Will update ‘actiongroups’ with C:\elasticsearch512xx\elasticsearch\plugins\search-guard-5\sgconfig\sg_action_groups.yml

04-12-2017 11:49:32EAP010043044019sgadmin output: SUCC: Configuration for ‘actiongroups’ created or updated

04-12-2017 11:49:32EAP010043044019sgadmin output: Done with success

04-12-2017 11:49:33EAP010043044019sgadmin ran

Is this helpful in any way?

···

On Wednesday, April 12, 2017 at 11:46:58 AM UTC-7, Search Guard wrote:

pls. share your sg_internal_users.yml and the console output when you run sgadmin

On Wednesday, 12 April 2017 20:42:39 UTC+2, Amar Kumar Dubedy wrote:

I am getting an exception on setting up Elasticsearch cluster:
[2017-04-12T11:25:52,374][ERROR][c.f.s.a.BackendRegistry ] Unexpected exception com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]

[2017-04-12T11:25:52,374][INFO ][c.f.s.a.BackendRegistry ] Cannot authenticate user (or add roles) with ad 4 due to ElasticsearchSecurityException[com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]]; nested: UncheckedExecutionException[ElasticsearchSecurityException[password does not match]]; nested: ElasticsearchSecurityException[password does not match];, try next

I have a 6 node setup (2 each on master, client and data).

I am guessing this error is related to the internal users that I have setup on sgconfig.

I am generating the hash (using hash.bat tool) and putting it in the internal_users config.

Each node generates its own hash from the same password and therefore each hash is different. Is this is what is causing this exception?

I am not able to access the nodes either with the configured password.

Any help on this would be awesome.

Regards,

Amar

4

for clarification: you have only ONE single configuration and push them via sgadmin to one arbitrary node.
Then searchguard will take care to replicate this immediately to all other nodes in the cluster.

Pls. read here about the concept:

https://github.com/floragunncom/search-guard-docs/blob/master/concepts.md

https://github.com/floragunncom/search-guard-docs/blob/master/configuration.md

https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md

···

On Wednesday, 12 April 2017 22:02:13 UTC+2, Amar Kumar Dubedy wrote:

Here is my sg_internal_users.config on one master

This is the internal user database

The hash value is a bcrypt hash and can be generated with plugin/tools/hash.sh

offerstoreadmin:

hash: $2a$12$1eHRCRUAf5s15DeP.NmDEeTkAVCy/GMryo4REnUKVryN5O9LieM0i

kibanaserver:

hash: $2a$12$1eHRCRUAf5s15DeP.NmDEeTkAVCy/GMryo4REnUKVryN5O9LieM0i

kibanaro:

hash: $2a$12$1eHRCRUAf5s15DeP.NmDEeTkAVCy/GMryo4REnUKVryN5O9LieM0i

Here is the sg_internal_users config on another master:

This is the internal user database

The hash value is a bcrypt hash and can be generated with plugin/tools/hash.sh

offerstoreadmin:

hash: $2a$12$9hP/4.Lzd6K1tROpIPUBJukOQ.1SODbdYF/tHTSy6GxsWRR8Yw3Pi

kibanaserver:

hash: $2a$12$9hP/4.Lzd6K1tROpIPUBJukOQ.1SODbdYF/tHTSy6GxsWRR8Yw3Pi

kibanaro:

hash: $2a$12$9hP/4.Lzd6K1tROpIPUBJukOQ.1SODbdYF/tHTSy6GxsWRR8Yw3Pi

Here is the sgadmin output on both machines.

04-12-2017 11:21:42EAP010043044016sgadmin error: at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:201)

04-12-2017 11:21:42EAP010043044016sgadmin error: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

04-12-2017 11:21:42EAP010043044016sgadmin error: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

04-12-2017 11:21:42EAP010043044016sgadmin error: at java.lang.Thread.run(Thread.java:745)

04-12-2017 11:21:43EAP010043044016sgadmin ran

04-12-2017 11:21:43EAP010043044019sgadmin output: searchguard index does not exists, attempt to create it … ERR: An unexpected ResourceAlreadyExistsException occured: index [searchguard/UJ4NmKegR9ednImF6rXoLw] already exists

04-12-2017 11:21:43EAP010043044019sgadmin output: Trace:

04-12-2017 11:21:43EAP010043044019sgadmin error: [searchguard/UJ4NmKegR9ednImF6rXoLw] ResourceAlreadyExistsException[index [searchguard/UJ4NmKegR9ednImF6rXoLw] already exists]

04-12-2017 11:21:43EAP010043044019sgadmin error: at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.validateIndexName(MetaDataCreateIndexService.java:139)

04-12-2017 11:21:43EAP010043044019sgadmin error: at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.validate(MetaDataCreateIndexService.java:479)

04-12-2017 11:21:43EAP010043044019sgadmin error: at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.access$000(MetaDataCreateIndexService.java:103)

04-12-2017 11:21:43EAP010043044019sgadmin error: at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$1.execute(MetaDataCreateIndexService.java:228)

04-12-2017 11:21:43EAP010043044019sgadmin error: at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45)

04-12-2017 11:21:43EAP010043044019sgadmin error: at org.elasticsearch.cluster.service.ClusterService.runTasksForExecutor(ClusterService.java:581)

04-12-2017 11:21:43EAP010043044019sgadmin error: at org.elasticsearch.cluster.service.ClusterService$UpdateTask.run(ClusterService.java:920)

04-12-2017 11:21:43EAP010043044019sgadmin error: at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458)

04-12-2017 11:21:43EAP010043044019sgadmin error: at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:201)

04-12-2017 11:21:43EAP010043044019sgadmin error: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

04-12-2017 11:21:43EAP010043044019sgadmin error: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

04-12-2017 11:21:43EAP010043044019sgadmin error: at java.lang.Thread.run(Thread.java:745)

04-12-2017 11:21:43EAP010043044019sgadmin ran

04-12-2017 11:49:01EAP010043044019sgadmin output: Search Guard Admin v5

04-12-2017 11:49:01EAP010043044019sgadmin output: WARNING: Seems you want connect to the a HTTP port.

04-12-2017 11:49:01EAP010043044019sgadmin output: sgadmin connect through the transport port which is normally 9300.

04-12-2017 11:49:01EAP010043044019sgadmin output: Will connect to ...:8900 … done

04-12-2017 11:49:05EAP010043044019sgadmin output: Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …

04-12-2017 11:49:31EAP010043044019sgadmin output: Clustername: ClusterName

04-12-2017 11:49:31EAP010043044019sgadmin output: Clusterstate: GREEN

04-12-2017 11:49:31EAP010043044019sgadmin output: Number of nodes: 6

04-12-2017 11:49:31EAP010043044019sgadmin output: Number of data nodes: 2

04-12-2017 11:49:31EAP010043044019sgadmin output: searchguard index already exists, so we do not need to create one.

04-12-2017 11:49:31EAP010043044019sgadmin output: Populate config from C:\elasticsearch512xx\elasticsearch\plugins\search-guard-5\sgconfig\

04-12-2017 11:49:31EAP010043044019sgadmin output: Will update ‘config’ with C:\elasticsearch512xx\elasticsearch\plugins\search-guard-5\sgconfig\sg_config.yml

04-12-2017 11:49:32EAP010043044019sgadmin output: SUCC: Configuration for ‘config’ created or updated

04-12-2017 11:49:32EAP010043044019sgadmin output: Will update ‘roles’ with C:\elasticsearch512xx\elasticsearch\plugins\search-guard-5\sgconfig\sg_roles.yml

04-12-2017 11:49:32EAP010043044019sgadmin output: SUCC: Configuration for ‘roles’ created or updated

04-12-2017 11:49:32EAP010043044019sgadmin output: Will update ‘rolesmapping’ with C:\elasticsearch512xx\elasticsearch\plugins\search-guard-5\sgconfig\sg_roles_mapping.yml

04-12-2017 11:49:32EAP010043044019sgadmin output: SUCC: Configuration for ‘rolesmapping’ created or updated

04-12-2017 11:49:32EAP010043044019sgadmin output: Will update ‘internalusers’ with C:\elasticsearch512xx\elasticsearch\plugins\search-guard-5\sgconfig\sg_internal_users.yml

04-12-2017 11:49:32EAP010043044019sgadmin output: SUCC: Configuration for ‘internalusers’ created or updated

04-12-2017 11:49:32EAP010043044019sgadmin output: Will update ‘actiongroups’ with C:\elasticsearch512xx\elasticsearch\plugins\search-guard-5\sgconfig\sg_action_groups.yml

04-12-2017 11:49:32EAP010043044019sgadmin output: SUCC: Configuration for ‘actiongroups’ created or updated

04-12-2017 11:49:32EAP010043044019sgadmin output: Done with success

04-12-2017 11:49:33EAP010043044019sgadmin ran

Is this helpful in any way?

On Wednesday, April 12, 2017 at 11:46:58 AM UTC-7, Search Guard wrote:

pls. share your sg_internal_users.yml and the console output when you run sgadmin

On Wednesday, 12 April 2017 20:42:39 UTC+2, Amar Kumar Dubedy wrote:

I am getting an exception on setting up Elasticsearch cluster:
[2017-04-12T11:25:52,374][ERROR][c.f.s.a.BackendRegistry ] Unexpected exception com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]

[2017-04-12T11:25:52,374][INFO ][c.f.s.a.BackendRegistry ] Cannot authenticate user (or add roles) with ad 4 due to ElasticsearchSecurityException[com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]]; nested: UncheckedExecutionException[ElasticsearchSecurityException[password does not match]]; nested: ElasticsearchSecurityException[password does not match];, try next

I have a 6 node setup (2 each on master, client and data).

I am guessing this error is related to the internal users that I have setup on sgconfig.

I am generating the hash (using hash.bat tool) and putting it in the internal_users config.

Each node generates its own hash from the same password and therefore each hash is different. Is this is what is causing this exception?

I am not able to access the nodes either with the configured password.

Any help on this would be awesome.

Regards,

Amar