Elasticsearch version:
7.17.9
Describe the issue:
I have upgraded my Elasticsearch server, Kibana, Filebeat, Metricbeat, and SearchGuard to version 7.17.9 from 7.9.3. We are not using the FLX version of SearchGuard.
Filebeat and Metricbeat will not connect to Elasticsearch from a remote server. If filebeat and metricbeat are on the local server, they work perfectly. I don’t see any errors in Elasticsearch saying why my filebeat_internal or metricbeat_internal users could not connect. In fact, I don’t see any mentions of filebeat_internal or metricbeat_internal at all. I get errors like these in the filebeat log:
2023-03-21T13:55:51.425-0500 ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://aln-nbadev4.labs.server.com:9200)): 401 Unauthorized: Unauthorized
2023-03-21T13:55:51.425-0500 INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(elasticsearch(https://aln-nbadev4.labs.server.com:9200)) with 1 reconnect attempt(s)
2023-03-21T13:55:51.425-0500 DEBUG [esclientleg] eslegclient/connection.go:261 ES Ping(url=https://aln-nbadev4.labs.server.com:9200)
2023-03-21T13:55:51.425-0500 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2023-03-21T13:55:51.426-0500 INFO [publisher] pipeline/retry.go:223 done
2023-03-21T13:55:51.427-0500 DEBUG [esclientleg] eslegclient/connection.go:265 Ping request failed with: 401 Unauthorized: Unauthorized
Testing filebeat on a server not running Elasticsearch:
[root@aln-nbav173-95 filebeat]# filebeat test output
elasticsearch: https://aln-nbadev4.labs.server.com:9200...
parse url... OK
connection...
parse host... OK
dns lookup... OK
addresses: 172.20.133.11
dial up... OK
TLS...
security: server's certificate chain verification is enabled
handshake... OK
TLS version: TLSv1.3
dial up... OK
talk to server... ERROR 401 Unauthorized: Unauthorized
Testing on the server that is running Elasticsearch:
[root@aln-nbadev4 elasticsearch]# filebeat test output
elasticsearch: https://aln-nbadev4.labs.server.com:9200...
parse url... OK
connection...
parse host... OK
dns lookup... OK
addresses: 172.20.133.11
dial up... OK
TLS...
security: server's certificate chain verification is enabled
handshake... OK
TLS version: TLSv1.3
dial up... OK
talk to server... OK
version: 7.17.9
The issue is not the username or password because I can use curl to get Elasticsearch to respond:
[root@aln-nbav173-95 filebeat]# curl -k -u filebeat_internal -XGET https://aln-nbadev4.labs.server.com:9200
Enter host password for user 'filebeat_internal':
{
"name" : "esnode-aln-nbadev4",
"cluster_name" : "nba_elasticsearch_cluster",
"cluster_uuid" : "mkT5taWDRGOaDPXOMo5_5g",
"version" : {
"number" : "7.17.9",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "ef48222227ee6b9e70e502f0f0daa52435ee634d",
"build_date" : "2023-01-31T05:34:43.305517834Z",
"build_snapshot" : false,
"lucene_version" : "8.11.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
Searchguard user information:
[root@aln-nbav173-95 filebeat]# curl -v --cacert /etc/filebeat/cacert.pem -u filebeat_internal https://aln-nbadev4.labs.server.com:9200/_searchguard/authinfo
Enter host password for user 'filebeat_internal':
* About to connect() to aln-nbadev4.labs.server.com port 9200 (#0)
* Trying 172.20.133.11...
* Connected to aln-nbadev4.labs.server.com (172.20.133.11) port 9200 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/filebeat/cacert.pem
CApath: none
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=esnode-aln-nbadev4.labs.server.com,OU=nBA,O=Server Systems Inc,L=City,ST=MA,C=US
* start date: Mar 22 14:13:37 2023 GMT
* expire date: Mar 20 14:13:37 2033 GMT
* common name: esnode-aln-nbadev4.labs.server.com
* issuer: CN=Default,O=Server,L=City,ST=Massachusetts,C=US
* Server auth using Basic with user 'filebeat_internal'
> GET /_searchguard/authinfo HTTP/1.1
> Authorization: Basic ZmlsZWJlYXRfaW50ZXJuYWw6bmV0c2NvdXQx
> User-Agent: curl/7.29.0
> Host: aln-nbadev4.labs.server.com:9200
> Accept: */*
>
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json; charset=UTF-8
< content-length: 402
<
* Connection #0 to host aln-nbadev4.labs.server.com left intact
{"user":"User filebeat_internal <basic/internal> [backend_roles=[logstash]]","user_name":"filebeat_internal","user_requested_tenant":null,"remote_address":"172.20.173.95:55596","backend_roles":["logstash"],"custom_attribute_names":[],"attribute_names":[],"sg_roles":["SGS_LOGSTASH","SGS_OWN_INDEX"],"sg_tenants":{"filebeat_internal":true},"principal":null,"peer_certificates":"0","sso_logout_url":null}
My filebeat configuration:
[root@aln-nbadev4 ~]# more /etc/filebeat/filebeat.yml
---
# Filebeat Configuration
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/current/configuring-howto-filebeat.html
name: filebeat
filebeat.shutdown_timeout: "5s"
filebeat.registry.path: "/var/lib/filebeat/registry"
filebeat.registry.file_permissions: "0600"
filebeat.registry.flush: "0s"
filebeat.config.inputs:
enabled: true
path: "/etc/filebeat/prospector-*.yml"
reload.enabled: true
reload.period: "30s"
processors:
- add_cloud_metadata: ~
- truncate_fields:
fields:
- log_event
max_characters: 10000
fail_on_error: false
ignore_missing: true
output.elasticsearch:
enabled: true
hosts: [ "https://aln-nbadev4.labs.server.com:9200" ]
username: ${ES_FILEBEAT_USERNAME}
password: ${ES_FILEBEAT_PASSWORD}
protocol: "https"
ssl.certificate_authorities: [ "/etc/filebeat/cacert.pem" ]
logging.level: "debug"
logging.to_files: true
logging.files:
path: "/var/log/filebeat"
name: "filebeat.log"
rotateeverybytes: 104857600
interval: "24h"
keepfiles: 7
permissions: "0644"
logging.metrics:
enalbled: true
period: "60s"
setup.ilm.enabled: false
setup.template.enabled: false
My sg_config.yml:
root@elasticsearch-aln-nbadev4:/usr/share/elasticsearch/plugins/search-guard-7/sgconfig# cat sg_config.yml
---
# This is the main Search Guard configuration file where authentication
# and authorization is defined.
#
# You need to configure at least one authentication domain in the authc of this file.
# An authentication domain is responsible for extracting the user credentials from
# the request and for validating them against an authentication backend like Active Directory for example.
#
# If more than one authentication domain is configured the first one which succeeds wins.
# If all authentication domains fail then the request is unauthenticated.
# In this case an exception is thrown and/or the HTTP status is set to 401.
#
# After authentication authorization (authz) will be applied. There can be zero or more authorizers which collect
# the roles from a given backend for the authenticated user.
#
# Both, authc and auth can be enabled/disabled separately for REST and TRANSPORT layer. Default is true for both.
# http_enabled: true
# transport_enabled: true
#
# For HTTP it is possible to allow anonymous authentication. If that is the case then the HTTP authenticators try to
# find user credentials in the HTTP request. If credentials are found then the user gets regularly authenticated.
# If none can be found the user will be authenticated as an "anonymous" user. This user has always the username "sg_anonymous"
# and one role named "sg_anonymous_backendrole".
# If you enable anonymous authentication all HTTP authenticators will not challenge.
#
#
# Note: If you define more than one HTTP authenticators make sure to put non-challenging authenticators like "proxy" or "clientcert"
# first and the challenging one last.
# Because it's not possible to challenge a client with two different authentication methods (for example
# Kerberos and Basic) only one can have the challenge flag set to true. You can cope with this situation
# by using pre-authentication, e.g. sending a HTTP Basic authentication header in the request.
#
# Default value of the challenge flag is true.
#
#
# HTTP
# basic (challenging)
# proxy (not challenging, needs xff)
# kerberos (challenging) NOT FREE FOR COMMERCIAL
# clientcert (not challenging, needs https)
# jwt (not challenging) NOT FREE FOR COMMERCIAL
# host (not challenging) #DEPRECATED, will be removed in a future version.
# host based authentication is configurable in sg_roles_mapping
# Authc
# internal
# noop
# ldap NOT FREE FOR COMMERCIAL USE
# Authz
# ldap NOT FREE FOR COMMERCIAL USE
# noop
# For more details pls refer to https://docs.search-guard.com/latest/authentication-authorization
_sg_meta:
type: "config"
config_version: 2
sg_config:
dynamic:
# Set filtered_alias_mode to 'disallow' to forbid more than 2 filtered aliases per index
# Set filtered_alias_mode to 'warn' to allow more than 2 filtered aliases per index but warns about it (default)
# Set filtered_alias_mode to 'nowarn' to allow more than 2 filtered aliases per index silently
#filtered_alias_mode: warn
#do_not_fail_on_forbidden: false
#kibana:
# Kibana multitenancy - NOT FREE FOR COMMERCIAL USE
# In addition to the config options below you need to set do_not_fail_on_forbidden to true (see above).
# Kibana needs to be configured for multi tenancy as well.
# See https://docs.search-guard.com/latest/kibana-multi-tenancy for details
#multitenancy_enabled: true
#server_username: kibanaserver
#index: '.kibana'
http:
anonymous_auth_enabled: false
xff:
enabled: false
internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
#internalProxies: '.*' # trust all internal proxies, regex pattern
#remoteIpHeader: 'x-forwarded-for'
###### see https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html for regex help
###### more information about XFF https://en.wikipedia.org/wiki/X-Forwarded-For
###### and here https://tools.ietf.org/html/rfc7239
###### and https://tomcat.apache.org/tomcat-8.0-doc/config/valve.html#Remote_IP_Valve
auth_token_provider: # NOT FREE FOR COMMERCIAL USE
# To enable using Search Guard auth tokens, you also need to enable the sg_issued_jwt_auth_domain below.
enabled: false
# JWTs produced by Search Guard are signed by default with a symmetric HMAC512 hash. For production systems,
# you must replace the value specified here by your own key. You can generate a new key for example with:
# openssl rand -base64 512 | tr '/+' '_-'
# If you want ot use another signature algorithm, you can specify a complete JWK using the attriubute jwt_signing_key.
# Refer to the documentation for details.
jwt_signing_key_hs512: "eTDZjSqRD9Abhod9iqeGX_7o93a-eElTeXWAF6FmzQshmRIrPD-C9ET3pFjJ_IBrzmWIZDk8ig-X_PIyGmKsxNMsrU-0BNWF5gJq5xOp4rYTl8z66Tw9wr8tHLxLxgJqkLSuUCRBZvlZlQ7jNdhBBxgM-hdSSzsN1T33qdIwhrUeJ-KXI5yKUXHjoWFYb9tETbYQ4NvONowkCsXK_flp-E3F_OcKe_z5iVUszAV8QfCod1zhbya540kDejXCL6N_XMmhWJqum7UJ3hgf6DEtroPSnVpHt4iR5w9ArKK-IBgluPght03gNcoNqwz7p77TFbdOmUKF_PWy1bcdbaUoSg"
# JWTs produced by Search Guard are unencrypted by default. Set a key here to activate encryption using AES Key Wrap.
# If you want ot use another signature algorithm, you can specify a complete JWK using the attriubute jwt_encryption_key.
# Refer to the documentation for details.
#jwt_encryption_key_a256kw: "..."
# Specify the maximum time period an auth token may be valid. Omit max_validity to have keys with unlimited lifetime.
# Note that when creating auth tokens, users can specify an even shorter time period.
max_validity: "1y"
# This specifies the maximum number of valid tokens a user can have at the same time.
max_tokens_per_user: 100
authc:
kerberos_auth_domain:
http_enabled: false
transport_enabled: false
order: 6
http_authenticator:
type: kerberos # NOT FREE FOR COMMERCIAL USE
challenge: true
config:
# If true a lot of kerberos/security related debugging output will be logged to standard out
krb_debug: false
# If true then the realm will be stripped from the user name
strip_realm_from_principal: true
authentication_backend:
type: noop
basic_internal_auth_domain:
description: "Authenticate via HTTP Basic against internal users database"
http_enabled: true
transport_enabled: true
order: 4
http_authenticator:
type: basic
challenge: true
authentication_backend:
type: intern
proxy_auth_domain:
description: "Authenticate via proxy"
http_enabled: false
transport_enabled: false
order: 3
http_authenticator:
type: proxy
challenge: false
config:
user_header: "x-proxy-user"
roles_header: "x-proxy-roles"
authentication_backend:
type: noop
jwt_auth_domain:
description: "Authenticate via Json Web Token"
http_enabled: false
transport_enabled: false
order: 0
http_authenticator:
type: jwt
challenge: false
config:
signing_key: "base64 encoded HMAC key or public RSA/ECDSA pem key"
jwt_header: "Authorization"
jwt_url_parameter: null
roles_key: null
subject_key: null
authentication_backend:
type: noop
sg_issued_jwt_auth_domain:
description: "Authenticate via Json Web Tokens issued by Search Guard"
http_enabled: false
# This auth domain is only available for HTTP
order: 1
http_authenticator:
type: sg_auth_token
challenge: false
# This auth domain automatically pulls configuration from the auth_token_provider config above
authentication_backend:
type: sg_auth_token
clientcert_auth_domain:
description: "Authenticate via SSL client certificates"
http_enabled: false
transport_enabled: false
order: 2
http_authenticator:
type: clientcert
config:
username_attribute: cn #optional, if omitted DN becomes username
challenge: false
authentication_backend:
type: noop
ldap:
description: "Authenticate via LDAP or Active Directory"
http_enabled: false
transport_enabled: false
order: 5
http_authenticator:
type: basic
challenge: false
authentication_backend:
# LDAP authentication backend (authenticate users against a LDAP or Active Directory)
type: ldap # NOT FREE FOR COMMERCIAL USE
config:
# enable ldaps
enable_ssl: false
# enable start tls, enable_ssl should be false
enable_start_tls: false
# send client certificate
enable_ssl_client_auth: false
# verify ldap hostname
verify_hostnames: true
hosts:
- localhost:8389
bind_dn: null
password: null
userbase: 'ou=people,dc=example,dc=com'
# Filter to search for users (currently in the whole subtree beneath userbase)
# {0} is substituted with the username
usersearch: '(sAMAccountName={0})'
# Use this attribute from the user as username (if not set then DN is used)
username_attribute: null
authz:
roles_from_myldap:
description: "Authorize via LDAP or Active Directory"
http_enabled: false
transport_enabled: false
authorization_backend:
# LDAP authorization backend (gather roles from a LDAP or Active Directory, you have to configure the above LDAP authentication backend settings too)
type: ldap # NOT FREE FOR COMMERCIAL USE
config:
# enable ldaps
enable_ssl: false
# enable start tls, enable_ssl should be false
enable_start_tls: false
# send client certificate
enable_ssl_client_auth: false
# verify ldap hostname
verify_hostnames: true
hosts:
- localhost:8389
bind_dn: null
password: null
rolebase: 'ou=groups,dc=example,dc=com'
# Filter to search for roles (currently in the whole subtree beneath rolebase)
# {0} is substituted with the DN of the user
# {1} is substituted with the username
# {2} is substituted with an attribute value from user's directory entry, of the authenticated user. Use userroleattribute to specify the name of the attribute
rolesearch: '(member={0})'
# Specify the name of the attribute which value should be substituted with {2} above
userroleattribute: null
# Roles as an attribute of the user entry
userrolename: disabled
#userrolename: memberOf
# The attribute in a role entry containing the name of that role, Default is "name".
# Can also be "dn" to use the full DN as rolename.
rolename: cn
# Resolve nested roles transitive (roles which are members of other roles and so on ...)
resolve_nested_roles: true
userbase: 'ou=people,dc=example,dc=com'
# Filter to search for users (currently in the whole subtree beneath userbase)
# {0} is substituted with the username
usersearch: '(uid={0})'
# Skip users matching a user name, a wildcard or a regex pattern
#skip_users:
# - 'cn=Michael Jackson,ou*people,o=TEST'
# - '/\S*/'
roles_from_another_ldap:
description: "Authorize via another Active Directory"
http_enabled: false
transport_enabled: false
authorization_backend:
type: ldap # NOT FREE FOR COMMERCIAL USE
#config goes here ...
# auth_failure_listeners:
# ip_rate_limiting:
# type: ip
# allowed_tries: 10
# time_window_seconds: 3600
# block_expiry_seconds: 600
# max_blocked_clients: 100000
# max_tracked_clients: 100000
# internal_authentication_backend_limiting:
# type: username
# authentication_backend: intern
# allowed_tries: 10
# time_window_seconds: 3600
# block_expiry_seconds: 600
# max_blocked_clients: 100000
# max_tracked_clients: 100000
The strange thing is that if I replace the username/password in filebeat.yml with elastic or kibanaserver, the messages come through to Elasticsearch. Even the nonstandard username elastalert_internal can get messages through to Elasticsearch. But the usernames filebeat_internal and metricbeat_internal don’t work, so it seems like SearchGuard is blocking them for some reason. Other than the ping unauthorized error, I cannot determine why they are blocked.