Can't initialize Search Guard

**Elasticsearch version:6.4.3

**Server OS version:Centos8

**Kibana version (if relevant):6.4.3

Describe the issue:
Hello, i tried to initializate searchguard for elasticsearch using sgadmin. sh.
Index has been created, but files from /sgconfig doesn’t apply.
Steps to reproduce:
Run the command:
./sgadmin -h “IP_ADDRESS” -icl -nhnv -cd …/sgconfig/ -cacert /path/ -cert /path/ -key /path/

WARNING: JAVA_HOME not set, will use /usr/bin/java

Search Guard Admin v6

Will connect to IP:9300 … done

Elasticsearch Version: 6.4.3

Search Guard Version: 6.4.3-25.5

Connected as CN=—.---,OU=----,O=1—, Inc.,DC=—,DC=—

Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …

Clustername: test-app

Clusterstate: YELLOW

Number of nodes: 1

Number of data nodes: 1

searchguard index already exists, so we do not need to create one.

INFO: searchguard index state is YELLOW, it seems you miss some replicas

Populate config from /usr/share/elasticsearch/plugins/search-guard-6/sgconfig

Will update ‘sg/config’ with …/sgconfig/sg_config.yml

FAIL: Configuration for ‘config’ failed because of java.lang.IllegalArgumentException: Rejecting mapping update to [searchguard] as the final mapping would have more than 1 type: [sg, doc]

Will update ‘sg/roles’ with …/sgconfig/sg_roles.yml

FAIL: Configuration for ‘roles’ failed because of java.lang.IllegalArgumentException: Rejecting mapping update to [searchguard] as the final mapping would have more than 1 type: [sg, doc]

Will update ‘sg/rolesmapping’ with …/sgconfig/sg_roles_mapping.yml

FAIL: Configuration for ‘rolesmapping’ failed because of java.lang.IllegalArgumentException: Rejecting mapping update to [searchguard] as the final mapping would have more than 1 type: [sg, doc]

Will update ‘sg/internalusers’ with …/sgconfig/sg_internal_users.yml

FAIL: Configuration for ‘internalusers’ failed because of java.lang.IllegalArgumentException: Rejecting mapping update to [searchguard] as the final mapping would have more than 1 type: [sg, doc]

Will update ‘sg/actiongroups’ with …/sgconfig/sg_action_groups.yml

FAIL: Configuration for ‘actiongroups’ failed because of java.lang.IllegalArgumentException: Rejecting mapping update to [searchguard] as the final mapping would have more than 1 type: [sg, doc]

FAIL: 1 nodes reported failures. First failure is FailedNodeException[Failed node [5sJBCIw6TBmq6OAi4mzcrQ]]; nested: RemoteTransportException[[es.1cloud.ru][192.168.101.101:9300][cluster:admin/searchguard/config/update[n]]]; nested: ElasticsearchException[java.util.concurrent.TimeoutException: Timeout after 5SECONDS while retrieving configuration for config, roles, rolesmapping, internalusers, actiongroups]; nested: NotSerializableExceptionWrapper[timeout_exception: Timeout after 5SECONDS while retrieving configuration for config, roles, rolesmapping, internalusers, actiongroups];

FAIL: Expected 1 nodes to return response, but got 0

Done with failures

Additional data:
Im tried delete index “searchguard” and run it again, but it doesn’t work.

How i can fix this? Thank you.

Is it the command you execute or you have hidden the paths deliberately?

Im tried delete index “searchguard” and run it again, but it doesn’t work.

Did the sgadmin command recreate the searchguard index? If you delete the index and run the command it should recreate the index.

Do you have any error in the Elasticsearch log? Post it here.

Run the sgadmin with diagnose switch to have more data for troubleshooting.

./sgadmin.sh -diagnose ...

Thank you for answer!

I’m hidden the paths deliberately.

Yes. sgadmin command recreate index. Also i tried create index without sgadmin, but it still doesn’t work.

no errors in elasticsearch log.

should i execute this command as:
./sgadmin -h “IP_ADDRESS” -icl -nhnv -cd …/sgconfig/ -cacert /path/ -cert /path/ -key /path/ -diagnose ?

sgadmin_diag_trace_2020-Sep-17_18-23-15.txt (378.1 KB)
this is result of ./sgadmin -diagnose …
I just changed name of indexes and username for a privacy.

Also i’m tried run the install_demo_configuration.sh and get the same error with it.

Maybe i should initialize searchguard without any indexes in the Elasticsearch? Early i created some indexes in the Elasticsearch, and after this i’m tried install the Searchguard. Maybe Elsticsearch must be empty before installing searchguard?

Hm. I see nothing suspicious in the diagnosis output.

Can you start with clear Elasticsearch? I tried to reproduce your issue but saw no problems. Please look at my commands below. I didn’t hide the options to show the full process to you.

Install and setup the SearchGuard. Run Elasticsearch.

$ tar -xzvf elasticsearch-6.4.3.tar.gz
$ cd elasticsearch-6.4.3/
$ ./bin/elasticsearch-plugin install https://maven.search-guard.com/search-guard-release/com/floragunn/search-guard-6/6.4.3-25.5/search-guard-6-6.4.3-25.5.zip
$ cd plugins/search-guard-6/tools/
$ chmod +x install_demo_configuration.sh
$ ./install_demo_configuration.sh

Search Guard 6 Demo Installer
 ** Warning: Do not use on production or public reachable systems **
Install demo certificates? [y/N] y
Initialize Search Guard? [y/N] y
Cluster mode requires maybe additional setup of:   
  - Virtual memory (vm.max_map_count)
    See https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html

Enable cluster mode? [y/N] n
...
### Success
...
$ cd -
$ pwd
/Users/srgbnd/Development/kibana/dist/elasticsearch-6.4.3
$ ./bin/elasticsearch

Create an index.

$ curl -X PUT https://localhost:9200/test_01 \
  --cacert "/Users/srgbnd/Development/kibana/dist/elasticsearch-6.4.3/config/root-ca.pem" \
  --cert "/Users/srgbnd/Development/kibana/dist/elasticsearch-6.4.3/config/kirk.pem" \
  --key "/Users/srgbnd/Development/kibana/dist/elasticsearch-6.4.3/config/kirk-key.pem"

Existing indices at this point.

green  open searchguard C7PIzOC_R6it3kM1BfLVYA 1 0 6 0 38.3kb 38.3kb
yellow open test_01     Zx7CPz3pQCCHbwrP2lK4xw 5 1 0 0  1.1kb  1.1kb

Delete SearchGuard index.

$ curl -X DELETE https://localhost:9200/searchguard \
  --cacert "/Users/srgbnd/Development/kibana/dist/elasticsearch-6.4.3/config/root-ca.pem" \
  --cert "/Users/srgbnd/Development/kibana/dist/elasticsearch-6.4.3/config/kirk.pem" \
  --key "/Users/srgbnd/Development/kibana/dist/elasticsearch-6.4.3/config/kirk-key.pem"

Existing indices at this point.

yellow open test_01     Zx7CPz3pQCCHbwrP2lK4xw 5 1 0 0  1.1kb  1.1kb

Trigger sgadmin.

$ "/Users/srgbnd/Development/kibana/dist/elasticsearch-6.4.3/plugins/search-guard-6/tools/sgadmin.sh" -cd "/Users/srgbnd/Development/kibana/dist/elasticsearch-6.4.3/plugins/search-guard-6/sgconfig" -icl -key "/Users/srgbnd/Development/kibana/dist/elasticsearch-6.4.3/config/kirk-key.pem" -cert "/Users/srgbnd/Development/kibana/dist/elasticsearch-6.4.3/config/kirk.pem" -cacert "/Users/srgbnd/Development/kibana/dist/elasticsearch-6.4.3/config/root-ca.pem" -nhnv

Search Guard Admin v6
Will connect to localhost:9300 ... done
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.bouncycastle.jcajce.provider.drbg.DRBG (file:/Users/srgbnd/Development/kibana/dist/elasticsearch-6.4.3/plugins/search-guard-6/bcprov-jdk15on-1.60.jar) to constructor sun.security.provider.Sun()
WARNING: Please consider reporting this to the maintainers of org.bouncycastle.jcajce.provider.drbg.DRBG
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Elasticsearch Version: 6.4.3
Search Guard Version: 6.4.3-25.5
Connected as CN=kirk,OU=client,O=client,L=test,C=de
Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
Clustername: searchguard_demo
Clusterstate: YELLOW
Number of nodes: 1
Number of data nodes: 1
searchguard index does not exists, attempt to create it ... done (0-all replicas)
Populate config from /Users/srgbnd/Development/kibana/dist/elasticsearch-6.4.3/plugins/search-guard-6/sgconfig/
Will update 'sg/config' with /Users/srgbnd/Development/kibana/dist/elasticsearch-6.4.3/plugins/search-guard-6/sgconfig/sg_config.yml
   SUCC: Configuration for 'config' created or updated
Will update 'sg/roles' with /Users/srgbnd/Development/kibana/dist/elasticsearch-6.4.3/plugins/search-guard-6/sgconfig/sg_roles.yml
   SUCC: Configuration for 'roles' created or updated
Will update 'sg/rolesmapping' with /Users/srgbnd/Development/kibana/dist/elasticsearch-6.4.3/plugins/search-guard-6/sgconfig/sg_roles_mapping.yml
   SUCC: Configuration for 'rolesmapping' created or updated
Will update 'sg/internalusers' with /Users/srgbnd/Development/kibana/dist/elasticsearch-6.4.3/plugins/search-guard-6/sgconfig/sg_internal_users.yml
   SUCC: Configuration for 'internalusers' created or updated
Will update 'sg/actiongroups' with /Users/srgbnd/Development/kibana/dist/elasticsearch-6.4.3/plugins/search-guard-6/sgconfig/sg_action_groups.yml
   SUCC: Configuration for 'actiongroups' created or updated
Done with success

Existing indices at this point.

green  open searchguard vwia9IuQR9izKItdfY2KvQ 1 0 6 0 37.9kb 37.9kb
yellow open test_01     Zx7CPz3pQCCHbwrP2lK4xw 5 1 0 0  1.1kb  1.1kb

Check the authinfo.

$ curl -k -u admin:admin -X GET https://localhost:9200/_searchguard/authinfo?pretty

{
  "user" : "User [name=admin, roles=[admin], requestedTenant=null]",
  "user_name" : "admin",
  "user_requested_tenant" : null,
  "remote_address" : "127.0.0.1:53671",
  "backend_roles" : [
    "admin"
  ],
  "custom_attribute_names" : [
    "attr.internal.attribute1",
    "attr.internal.attribute2",
    "attr.internal.attribute3"
  ],
  "sg_roles" : [
    "sg_all_access",
    "sg_own_index"
  ],
  "sg_tenants" : {
    "admin_tenant" : true,
    "admin" : true
  },
  "principal" : null,
  "peer_certificates" : "0",
  "sso_logout_url" : null
}