Unexpected exception [_xpack] InvalidIndexNameException for Filebeat 6.7.1

#1

Environment:

  • CentOS Linux release 7.6.1810 (Core)
  • Elasticsearch: elasticsearch-oss:6.7.1 (docker image)
  • Filebeat: filebeat-6.7.1-x86_64.rpm
  • Search Guard: 6.7.1-24.3

I am running the OSS flavor of ES as a Docker image (no xpack installed). The Search Guard plugin has been installed successfully and I am using the example sgconfig from the search-guard-6 project. The config has been successfully loaded using the sgamin.sh script. I’ve also installed Filebeat from the RPM and configured it to use SSL and the ‘logstash’ user/password from the sgconfig that is loaded.

Filebeat pings Elasticsearch successfully, however, it fails when checking Elasticsearch license information:

2019-04-09T17:25:30.027Z        DEBUG   [elasticsearch] elasticsearch/client.go:715     ES Ping(url=https://xxxxxxxxx:9200)
2019-04-09T17:25:30.103Z        DEBUG   [elasticsearch] elasticsearch/client.go:738     Ping status code: 200
2019-04-09T17:25:30.103Z        INFO    elasticsearch/client.go:739     Attempting to connect to Elasticsearch version 6.7.1
2019-04-09T17:25:30.103Z        DEBUG   [elasticsearch] elasticsearch/client.go:757     GET https://xxxxxxxxx:9200/_xpack?human=false  <nil>
2019-04-09T17:25:31.830Z        ERROR   pipeline/output.go:100  Failed to connect to backoff(elasticsearch(https://xxxxxxxxx:9200)): Connection marked as failed because the onConnect callback failed: cannot retrieve the elasticsearch license: error from server, response code: 500

This error appears to keep any of the bulk From what I can tell is that Filebeat indeed checks for licensing and falls back to the OSS license if not found. Verified the same behavior using curl as the ‘logstash’ user:

# curl -kv -XGET -u logstash:logstash "https://localhost:9200/_xpack?human=false&include_type_name=true"
* About to connect() to localhost port 9200 (#0)
*   Trying ::1...
* Connected to localhost (::1) port 9200 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* 	subject: CN= <a class="attachment" href="/uploads/search_guard/original/1X/1d8a25a3dd5d5328441cc9c60bd694e4d2c77c06.log">es.log</a> (38.9 KB)
 ,OU= xxxxxxxxx,O= xxxxxxxxx,L= xxxxxxxxx,ST= xxxxxxxxx,C=US
* 	start date: Apr 09 14:52:53 2019 GMT
* 	expire date: Apr 06 14:52:53 2029 GMT
* 	common name: xxxxxxxxx
* 	issuer: CN= xxxxxxxxx,OU=n xxxxxxxxx BA,O= xxxxxxxxx,L= xxxxxxxxx,ST= xxxxxxxxx,C=US
* Server auth using Basic with user 'logstash'
> GET /_xpack?human=false&include_type_name=true HTTP/1.1
> Authorization: Basic bG9nc3Rhc2g6bG9nc3Rhc2g=
> User-Agent: curl/7.29.0
> Host: localhost:9200
> Accept: */*
>
< HTTP/1.1 500 Internal Server Error
< content-type: application/json; charset=UTF-8
< content-length: 197
<
* Connection #0 to host localhost left intact
{"error":{"root_cause":[{"type":"security_exception","reason":"Unexpected exception indices:admin/get"}],"type":"security_exception","reason":"Unexpected exception indices:admin/get"},"status":500}

After seeing the response above, I added ‘indices:admin/get’ for the _xpack index the logstash role, but, I get the same errors with that configuration as well.

ES log shows:

2019-04-09 17:24:30.453Z ERROR [elasticsearch[esnode-xxxxxxxxx][http_server_worker][T#2]] com.floragunn.searchguard.filter.SearchGuardFilter - Unexpected exception [_xpack] InvalidIndexNameException[Invalid index name [xpack], must not start with '’.]
org.elasticsearch.indices.InvalidIndexNameException: Invalid index name [xpack], must not start with '’.

2019-04-09 17:24:30.463Z WARN [elasticsearch[esnode-xxxxxxxxx][http_server_worker][T#2]] rest.suppressed - path: /_xpack, params: {index=_xpack, human=false}
org.elasticsearch.ElasticsearchSecurityException: Unexpected exception indices:admin/get

Log is attached. Any ideas on what may be causing this and how to resolve the issue?

Thanks,
James

es.log (38.9 KB)

0 Likes

assigned jkressin #2
0 Likes

#3

If you are using the pure OSS version of Elasticsearch, the X-Pack HTTP endpoints are not registered.

The call you are seeing/making to

https://localhost:9200/_xpack

IMHO does not refer to an index called “_xpack”, but to an X-Pack HTTP/REST handler. Which is not available since you are not using any X-Pack features. This would be in line with the exception in the ES logs you are seeing:

org.elasticsearch.indices.InvalidIndexNameException: Invalid index name [_xpack], must not start with '_'.

Which version of filebeat are you using? Please check that you are using the OSS version of filebeat:

https://www.elastic.co/downloads/beats/filebeat-oss

0 Likes

#4

Thank you for the prompt reply Jochen. You are right, I am using the wrong Filebeat RPM in my installation. Was using ‘filebeat-6.7.1-x86_64.rpm’ and should be ‘filebeat-oss-6.7.1-x86_64.rpm’. This corrects my issue.

James

0 Likes