problem with configuration initialization/reload

Search Guard
1

Dear SearchGuard community,

I am currently having a issue with the initialization of SearchGuard. After I initialized my SearchGuard, the HTTP Basic Authentication was working as expected, with the preconfigured users.

I am using Elasticsearch 6.3.1 and SearchGuard 6.3.1-22.3. My JVM version is openjdk-8 (8u151-b12-1).

I changed the configuration and tried to reload with the folowing sgadmin.sh call:

/usr/share/elasticsearch/plugins/search-guard-6/tools/sgadmin.sh -nhnv -cacert …/…/…/…/…/…/etc/elasticsearch/root-ca.pem -cert …/…/…/…/…/…/etc/elasticsearch/adm.pem -key …/…/…/…/…/…/etc/elasticsearch/adm.key -cd …/sgconfig -icl --reload

``

which returned the following output:

WARNING: JAVA_HOME not set, will use /usr/bin/java

Search Guard Admin v6

WARN: It makes no sense to specify -cd as well as -r

Will connect to localhost:9300 … done

Elasticsearch Version: 6.3.1

Search Guard Version: 6.3.1-22.3

Connected as CN=adm.macd.com,OU=MAD,O=MACD GmbH,DC=MACD,DC=GmbH

Reload config on all nodes

``

But when I try to login with a new configured user, I am not able to login, because the user is not present.

After this I tried to reinitialize SearchGuard with sgadmin.sh and without the --reload/-rl option, which resulted in the following output:

WARNING: JAVA_HOME not set, will use /usr/bin/java

Search Guard Admin v6

Will connect to localhost:9300 … done

Elasticsearch Version: 6.3.1

Search Guard Version: 6.3.1-22.3

Connected as CN=adm.macd.com,OU=MAD,O=MACD GmbH,DC=MACD,DC=GmbH

Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …

Clustername: searchguard

Clusterstate: YELLOW

Number of nodes: 1

Number of data nodes: 1

searchguard index already exists, so we do not need to create one.

Populate config from /usr/share/elasticsearch/plugins/search-guard-6/sgconfig

Will update ‘sg/config’ with …/sgconfig/sg_config.yml

SUCC: Configuration for ‘config’ created or updated

Will update ‘sg/roles’ with …/sgconfig/sg_roles.yml

SUCC: Configuration for ‘roles’ created or updated

Will update ‘sg/rolesmapping’ with …/sgconfig/sg_roles_mapping.yml

SUCC: Configuration for ‘rolesmapping’ created or updated

Will update ‘sg/internalusers’ with …/sgconfig/sg_internal_users.yml

SUCC: Configuration for ‘internalusers’ created or updated

Will update ‘sg/actiongroups’ with …/sgconfig/sg_action_groups.yml

SUCC: Configuration for ‘actiongroups’ created or updated

Done with success

``

Thus I thought my changes were read and applied, but after this, even the default users (which I did not removed) are no longer able to login and I get the following error every second:

[2018-08-15T15:50:48,010][ERROR][c.f.s.a.BackendRegistry ] Not yet initialized (you may need to run sgadmin)

``

In the searchguard.log the following output is indicating some problems with enterprise modules, but so far I assumed, that ‘basicauth’ is not a enterprise module.

Below is also the SearchGuard configuration in my elasticsearch.yml:

searchguard.enterprise_modules_enabled: false

searchguard.ssl.transport.pemcert_filepath: elk.pem

searchguard.ssl.transport.pemkey_filepath: elk.key

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: elk.pem

searchguard.ssl.http.pemkey_filepath: elk.key

searchguard.ssl.http.pemtrustedcas_filepath: root-ca.pem

searchguard.allow_unsafe_democertificates: true

searchguard.allow_default_init_sgindex: true

searchguard.authcz.admin_dn:

searchguard.audit.type: internal_elasticsearch

searchguard.enable_snapshot_restore_privilege: true

searchguard.check_snapshot_restore_write_privileges: true

searchguard.restapi.roles_enabled: [“sg_all_access”]

cluster.routing.allocation.disk.threshold_enabled: false

cluster.name: searchguard

discovery.zen.minimum_master_nodes: 1

node.max_local_storage_nodes: 3

xpack.security.enabled: false

xpack.monitoring.enabled: false

searchguard.roles_mapping_resolution: MAPPING_ONLY

searchguard.cache.ttl_minutes: 15

``

I am still working on this and will update any progress made on this issue, but I thought that maybe somebody stumbled across the same problem.

Greetings,

Yannick

2

I don’t know for sure, but I suspect that you have introduced an error/syntactic error in one of the logfiles. We stumbled upon this behavior once, but we are still investigating under which circumstances and with which SG versions it appears.

(Note that when you apply configuration changes you do not need to add the reload switch, it has no effect here.)

You are right, since the actual sgadmin.sh call succeeds, the files have been read and applied. So it must be an syntax error in one of the files itself. Can you check what changes you made, and revert them for debugging purposes? If I’m right in my assumption it would also help us to know what the error was, so we can debug this issue further.

Thanks!

···

On Wednesday, August 15, 2018 at 3:57:15 PM UTC+2, Yannick K wrote:

Dear SearchGuard community,

I am currently having a issue with the initialization of SearchGuard. After I initialized my SearchGuard, the HTTP Basic Authentication was working as expected, with the preconfigured users.

I am using Elasticsearch 6.3.1 and SearchGuard 6.3.1-22.3. My JVM version is openjdk-8 (8u151-b12-1).

I changed the configuration and tried to reload with the folowing sgadmin.sh call:

/usr/share/elasticsearch/plugins/search-guard-6/tools/sgadmin.sh -nhnv -cacert …/…/…/…/…/…/etc/elasticsearch/root-ca.pem -cert …/…/…/…/…/…/etc/elasticsearch/adm.pem -key …/…/…/…/…/…/etc/elasticsearch/adm.key -cd …/sgconfig -icl --reload

``

which returned the following output:

WARNING: JAVA_HOME not set, will use /usr/bin/java

Search Guard Admin v6

WARN: It makes no sense to specify -cd as well as -r

Will connect to localhost:9300 … done

Elasticsearch Version: 6.3.1

Search Guard Version: 6.3.1-22.3

Connected as CN=adm.macd.com,OU=MAD,O=MACD GmbH,DC=MACD,DC=GmbH

Reload config on all nodes

``

But when I try to login with a new configured user, I am not able to login, because the user is not present.

After this I tried to reinitialize SearchGuard with sgadmin.sh and without the --reload/-rl option, which resulted in the following output:

WARNING: JAVA_HOME not set, will use /usr/bin/java

Search Guard Admin v6

Will connect to localhost:9300 … done

Elasticsearch Version: 6.3.1

Search Guard Version: 6.3.1-22.3

Connected as CN=adm.macd.com,OU=MAD,O=MACD GmbH,DC=MACD,DC=GmbH

Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …

Clustername: searchguard

Clusterstate: YELLOW

Number of nodes: 1

Number of data nodes: 1

searchguard index already exists, so we do not need to create one.

Populate config from /usr/share/elasticsearch/plugins/search-guard-6/sgconfig

Will update ‘sg/config’ with …/sgconfig/sg_config.yml

SUCC: Configuration for ‘config’ created or updated

Will update ‘sg/roles’ with …/sgconfig/sg_roles.yml

SUCC: Configuration for ‘roles’ created or updated

Will update ‘sg/rolesmapping’ with …/sgconfig/sg_roles_mapping.yml

SUCC: Configuration for ‘rolesmapping’ created or updated

Will update ‘sg/internalusers’ with …/sgconfig/sg_internal_users.yml

SUCC: Configuration for ‘internalusers’ created or updated

Will update ‘sg/actiongroups’ with …/sgconfig/sg_action_groups.yml

SUCC: Configuration for ‘actiongroups’ created or updated

Done with success

``

Thus I thought my changes were read and applied, but after this, even the default users (which I did not removed) are no longer able to login and I get the following error every second:

[2018-08-15T15:50:48,010][ERROR][c.f.s.a.BackendRegistry ] Not yet initialized (you may need to run sgadmin)

``

In the searchguard.log the following output is indicating some problems with enterprise modules, but so far I assumed, that ‘basicauth’ is not a enterprise module.

Below is also the SearchGuard configuration in my elasticsearch.yml:

searchguard.enterprise_modules_enabled: false

searchguard.ssl.transport.pemcert_filepath: elk.pem

searchguard.ssl.transport.pemkey_filepath: elk.key

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: elk.pem

searchguard.ssl.http.pemkey_filepath: elk.key

searchguard.ssl.http.pemtrustedcas_filepath: root-ca.pem

searchguard.allow_unsafe_democertificates: true

searchguard.allow_default_init_sgindex: true

searchguard.authcz.admin_dn:

searchguard.audit.type: internal_elasticsearch

searchguard.enable_snapshot_restore_privilege: true

searchguard.check_snapshot_restore_write_privileges: true

searchguard.restapi.roles_enabled: [“sg_all_access”]

cluster.routing.allocation.disk.threshold_enabled: false

cluster.name: searchguard

discovery.zen.minimum_master_nodes: 1

node.max_local_storage_nodes: 3

xpack.security.enabled: false

xpack.monitoring.enabled: false

searchguard.roles_mapping_resolution: MAPPING_ONLY

searchguard.cache.ttl_minutes: 15

``

I am still working on this and will update any progress made on this issue, but I thought that maybe somebody stumbled across the same problem.

Greetings,

Yannick

3

Hey Jochen,

thanks for your fast reply. I just checked the configuration files and you were right!

Here is the formatting difference I encountered:

Original sg_internal_users.yml:

This is the internal user database

The hash value is a bcrypt hash and can be generated with plugin/tools/hash.sh

#password is: admin

admin:

readonly: true

hash: $2a$12$VcCDgh2NDk07JGN0rjGbM.Ad41qVR/YFJcgHp0UGns5JDymv…TOG

roles:

  • admin

attributes:

#no dots allowed in attribute names

attribute1: value1

attribute2: value2

attribute3: value3

``

Then I read the active SearchGuard configuration and the formatting was as follows:

···

admin:

readonly: true

hash: “$2a$12$VcCDgh2NDk07JGN0rjGbM.Ad41qVR/YFJcgHp0UGns5JDymv…TOG”

roles:

  • “admin”

attributes:

attribute1: “value1”

attribute2: “value2”

attribute3: “value3”

In the next step I applied my changes to the read configuration (with the same formatting syntax) and voila - Its working like a charm :slight_smile:

Should have tested this before raising this issue, but hopefully it will help some other users, while debugging.

Thanks a lot for pushing me in the correct direction!

Kind regards,

Yannick

Am Mittwoch, 15. August 2018 15:57:15 UTC+2 schrieb Yannick K:

Dear SearchGuard community,

I am currently having a issue with the initialization of SearchGuard. After I initialized my SearchGuard, the HTTP Basic Authentication was working as expected, with the preconfigured users.

I am using Elasticsearch 6.3.1 and SearchGuard 6.3.1-22.3. My JVM version is openjdk-8 (8u151-b12-1).

I changed the configuration and tried to reload with the folowing sgadmin.sh call:

/usr/share/elasticsearch/plugins/search-guard-6/tools/sgadmin.sh -nhnv -cacert …/…/…/…/…/…/etc/elasticsearch/root-ca.pem -cert …/…/…/…/…/…/etc/elasticsearch/adm.pem -key …/…/…/…/…/…/etc/elasticsearch/adm.key -cd …/sgconfig -icl --reload

``

which returned the following output:

WARNING: JAVA_HOME not set, will use /usr/bin/java

Search Guard Admin v6

WARN: It makes no sense to specify -cd as well as -r

Will connect to localhost:9300 … done

Elasticsearch Version: 6.3.1

Search Guard Version: 6.3.1-22.3

Connected as CN=adm.macd.com,OU=MAD,O=MACD GmbH,DC=MACD,DC=GmbH

Reload config on all nodes

``

But when I try to login with a new configured user, I am not able to login, because the user is not present.

After this I tried to reinitialize SearchGuard with sgadmin.sh and without the --reload/-rl option, which resulted in the following output:

WARNING: JAVA_HOME not set, will use /usr/bin/java

Search Guard Admin v6

Will connect to localhost:9300 … done

Elasticsearch Version: 6.3.1

Search Guard Version: 6.3.1-22.3

Connected as CN=adm.macd.com,OU=MAD,O=MACD GmbH,DC=MACD,DC=GmbH

Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …

Clustername: searchguard

Clusterstate: YELLOW

Number of nodes: 1

Number of data nodes: 1

searchguard index already exists, so we do not need to create one.

Populate config from /usr/share/elasticsearch/plugins/search-guard-6/sgconfig

Will update ‘sg/config’ with …/sgconfig/sg_config.yml

SUCC: Configuration for ‘config’ created or updated

Will update ‘sg/roles’ with …/sgconfig/sg_roles.yml

SUCC: Configuration for ‘roles’ created or updated

Will update ‘sg/rolesmapping’ with …/sgconfig/sg_roles_mapping.yml

SUCC: Configuration for ‘rolesmapping’ created or updated

Will update ‘sg/internalusers’ with …/sgconfig/sg_internal_users.yml

SUCC: Configuration for ‘internalusers’ created or updated

Will update ‘sg/actiongroups’ with …/sgconfig/sg_action_groups.yml

SUCC: Configuration for ‘actiongroups’ created or updated

Done with success

``

Thus I thought my changes were read and applied, but after this, even the default users (which I did not removed) are no longer able to login and I get the following error every second:

[2018-08-15T15:50:48,010][ERROR][c.f.s.a.BackendRegistry ] Not yet initialized (you may need to run sgadmin)

``

In the searchguard.log the following output is indicating some problems with enterprise modules, but so far I assumed, that ‘basicauth’ is not a enterprise module.

Below is also the SearchGuard configuration in my elasticsearch.yml:

searchguard.enterprise_modules_enabled: false

searchguard.ssl.transport.pemcert_filepath: elk.pem

searchguard.ssl.transport.pemkey_filepath: elk.key

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: elk.pem

searchguard.ssl.http.pemkey_filepath: elk.key

searchguard.ssl.http.pemtrustedcas_filepath: root-ca.pem

searchguard.allow_unsafe_democertificates: true

searchguard.allow_default_init_sgindex: true

searchguard.authcz.admin_dn:

searchguard.audit.type: internal_elasticsearch

searchguard.enable_snapshot_restore_privilege: true

searchguard.check_snapshot_restore_write_privileges: true

searchguard.restapi.roles_enabled: [“sg_all_access”]

cluster.routing.allocation.disk.threshold_enabled: false

cluster.name: searchguard

discovery.zen.minimum_master_nodes: 1

node.max_local_storage_nodes: 3

xpack.security.enabled: false

xpack.monitoring.enabled: false

searchguard.roles_mapping_resolution: MAPPING_ONLY

searchguard.cache.ttl_minutes: 15

``

I am still working on this and will update any progress made on this issue, but I thought that maybe somebody stumbled across the same problem.

Greetings,

Yannick