Filebeat and Metricbeat on remote servers will not connect

Elasticsearch version:
7.17.9

Describe the issue:
I have upgraded my Elasticsearch server, Kibana, Filebeat, Metricbeat, and SearchGuard to version 7.17.9 from 7.9.3. We are not using the FLX version of SearchGuard.

Filebeat and Metricbeat will not connect to Elasticsearch from a remote server. If filebeat and metricbeat are on the local server, they work perfectly. I don’t see any errors in Elasticsearch saying why my filebeat_internal or metricbeat_internal users could not connect. In fact, I don’t see any mentions of filebeat_internal or metricbeat_internal at all. I get errors like these in the filebeat log:

2023-03-21T13:55:51.425-0500    ERROR   [publisher_pipeline_output]     pipeline/output.go:154  Failed to connect to backoff(elasticsearch(https://aln-nbadev4.labs.server.com:9200)): 401 Unauthorized: Unauthorized
2023-03-21T13:55:51.425-0500    INFO    [publisher_pipeline_output]     pipeline/output.go:145  Attempting to reconnect to backoff(elasticsearch(https://aln-nbadev4.labs.server.com:9200)) with 1 reconnect attempt(s)
2023-03-21T13:55:51.425-0500    DEBUG   [esclientleg]   eslegclient/connection.go:261   ES Ping(url=https://aln-nbadev4.labs.server.com:9200)
2023-03-21T13:55:51.425-0500    INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2023-03-21T13:55:51.426-0500    INFO    [publisher]     pipeline/retry.go:223     done
2023-03-21T13:55:51.427-0500    DEBUG   [esclientleg]   eslegclient/connection.go:265   Ping request failed with: 401 Unauthorized: Unauthorized

Testing filebeat on a server not running Elasticsearch:

[root@aln-nbav173-95 filebeat]# filebeat test output
elasticsearch: https://aln-nbadev4.labs.server.com:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 172.20.133.11
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... ERROR 401 Unauthorized: Unauthorized

Testing on the server that is running Elasticsearch:

[root@aln-nbadev4 elasticsearch]# filebeat test output
elasticsearch: https://aln-nbadev4.labs.server.com:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 172.20.133.11
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... OK
  version: 7.17.9

The issue is not the username or password because I can use curl to get Elasticsearch to respond:

[root@aln-nbav173-95 filebeat]# curl -k -u filebeat_internal -XGET https://aln-nbadev4.labs.server.com:9200
Enter host password for user 'filebeat_internal':
{
  "name" : "esnode-aln-nbadev4",
  "cluster_name" : "nba_elasticsearch_cluster",
  "cluster_uuid" : "mkT5taWDRGOaDPXOMo5_5g",
  "version" : {
    "number" : "7.17.9",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "ef48222227ee6b9e70e502f0f0daa52435ee634d",
    "build_date" : "2023-01-31T05:34:43.305517834Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Searchguard user information:

[root@aln-nbav173-95 filebeat]# curl -v --cacert /etc/filebeat/cacert.pem -u filebeat_internal https://aln-nbadev4.labs.server.com:9200/_searchguard/authinfo
Enter host password for user 'filebeat_internal':
* About to connect() to aln-nbadev4.labs.server.com port 9200 (#0)
*   Trying 172.20.133.11...
* Connected to aln-nbadev4.labs.server.com (172.20.133.11) port 9200 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/filebeat/cacert.pem
  CApath: none
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=esnode-aln-nbadev4.labs.server.com,OU=nBA,O=Server Systems Inc,L=City,ST=MA,C=US
*       start date: Mar 22 14:13:37 2023 GMT
*       expire date: Mar 20 14:13:37 2033 GMT
*       common name: esnode-aln-nbadev4.labs.server.com
*       issuer: CN=Default,O=Server,L=City,ST=Massachusetts,C=US
* Server auth using Basic with user 'filebeat_internal'
> GET /_searchguard/authinfo HTTP/1.1
> Authorization: Basic ZmlsZWJlYXRfaW50ZXJuYWw6bmV0c2NvdXQx
> User-Agent: curl/7.29.0
> Host: aln-nbadev4.labs.server.com:9200
> Accept: */*
>
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json; charset=UTF-8
< content-length: 402
<
* Connection #0 to host aln-nbadev4.labs.server.com left intact
{"user":"User filebeat_internal <basic/internal> [backend_roles=[logstash]]","user_name":"filebeat_internal","user_requested_tenant":null,"remote_address":"172.20.173.95:55596","backend_roles":["logstash"],"custom_attribute_names":[],"attribute_names":[],"sg_roles":["SGS_LOGSTASH","SGS_OWN_INDEX"],"sg_tenants":{"filebeat_internal":true},"principal":null,"peer_certificates":"0","sso_logout_url":null}

My filebeat configuration:

[root@aln-nbadev4 ~]# more /etc/filebeat/filebeat.yml
---
# Filebeat Configuration
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/current/configuring-howto-filebeat.html

name: filebeat

filebeat.shutdown_timeout: "5s"
filebeat.registry.path: "/var/lib/filebeat/registry"
filebeat.registry.file_permissions: "0600"
filebeat.registry.flush: "0s"

filebeat.config.inputs:
  enabled: true
  path: "/etc/filebeat/prospector-*.yml"
  reload.enabled: true
  reload.period: "30s"

processors:
  - add_cloud_metadata: ~
  - truncate_fields:
      fields:
        - log_event
      max_characters: 10000
      fail_on_error: false
      ignore_missing: true

output.elasticsearch:
  enabled: true
  hosts: [ "https://aln-nbadev4.labs.server.com:9200" ]
  username: ${ES_FILEBEAT_USERNAME}
  password: ${ES_FILEBEAT_PASSWORD}
  protocol: "https"
  ssl.certificate_authorities: [ "/etc/filebeat/cacert.pem" ]

logging.level: "debug"
logging.to_files: true
logging.files:
  path: "/var/log/filebeat"
  name: "filebeat.log"
  rotateeverybytes: 104857600
  interval: "24h"
  keepfiles: 7
  permissions: "0644"
logging.metrics:
  enalbled: true
  period: "60s"

setup.ilm.enabled: false
setup.template.enabled: false

My sg_config.yml:

root@elasticsearch-aln-nbadev4:/usr/share/elasticsearch/plugins/search-guard-7/sgconfig# cat sg_config.yml
---

# This is the main Search Guard configuration file where authentication
# and authorization is defined.
#
# You need to configure at least one authentication domain in the authc of this file.
# An authentication domain is responsible for extracting the user credentials from
# the request and for validating them against an authentication backend like Active Directory for example.
#
# If more than one authentication domain is configured the first one which succeeds wins.
# If all authentication domains fail then the request is unauthenticated.
# In this case an exception is thrown and/or the HTTP status is set to 401.
#
# After authentication authorization (authz) will be applied. There can be zero or more authorizers which collect
# the roles from a given backend for the authenticated user.
#
# Both, authc and auth can be enabled/disabled separately for REST and TRANSPORT layer. Default is true for both.
#        http_enabled: true
#        transport_enabled: true
#
# For HTTP it is possible to allow anonymous authentication. If that is the case then the HTTP authenticators try to
# find user credentials in the HTTP request. If credentials are found then the user gets regularly authenticated.
# If none can be found the user will be authenticated as an "anonymous" user. This user has always the username "sg_anonymous"
# and one role named "sg_anonymous_backendrole".
# If you enable anonymous authentication all HTTP authenticators will not challenge.
#
#
# Note: If you define more than one HTTP authenticators make sure to put non-challenging authenticators like "proxy" or "clientcert"
# first and the challenging one last.
# Because it's not possible to challenge a client with two different authentication methods (for example
# Kerberos and Basic) only one can have the challenge flag set to true. You can cope with this situation
# by using pre-authentication, e.g. sending a HTTP Basic authentication header in the request.
#
# Default value of the challenge flag is true.
#
#
# HTTP
#   basic (challenging)
#   proxy (not challenging, needs xff)
#   kerberos (challenging) NOT FREE FOR COMMERCIAL
#   clientcert (not challenging, needs https)
#   jwt (not challenging) NOT FREE FOR COMMERCIAL
#   host (not challenging) #DEPRECATED, will be removed in a future version.
#                          host based authentication is configurable in sg_roles_mapping

# Authc
#   internal
#   noop
#   ldap  NOT FREE FOR COMMERCIAL USE

# Authz
#   ldap  NOT FREE FOR COMMERCIAL USE
#   noop


# For more details pls refer to  https://docs.search-guard.com/latest/authentication-authorization

_sg_meta:
  type: "config"
  config_version: 2

sg_config:
  dynamic:
      # Set filtered_alias_mode to 'disallow' to forbid more than 2 filtered aliases per index
      # Set filtered_alias_mode to 'warn' to allow more than 2 filtered aliases per index but warns about it (default)
      # Set filtered_alias_mode to 'nowarn' to allow more than 2 filtered aliases per index silently
      #filtered_alias_mode: warn
      #do_not_fail_on_forbidden: false
      #kibana:
        # Kibana multitenancy - NOT FREE FOR COMMERCIAL USE
        # In addition to the config options below you need to set do_not_fail_on_forbidden to true (see above).
        # Kibana needs to be configured for multi tenancy as well.
        # See https://docs.search-guard.com/latest/kibana-multi-tenancy for details
        #multitenancy_enabled: true
        #server_username: kibanaserver
        #index: '.kibana'
      http:
        anonymous_auth_enabled: false
        xff:
          enabled: false
          internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
          #internalProxies: '.*' # trust all internal proxies, regex pattern
          #remoteIpHeader:  'x-forwarded-for'
          ###### see https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html for regex help
          ###### more information about XFF https://en.wikipedia.org/wiki/X-Forwarded-For
          ###### and here https://tools.ietf.org/html/rfc7239
          ###### and https://tomcat.apache.org/tomcat-8.0-doc/config/valve.html#Remote_IP_Valve
      auth_token_provider: # NOT FREE FOR COMMERCIAL USE
        # To enable using Search Guard auth tokens, you also need to enable the sg_issued_jwt_auth_domain below.
        enabled: false
        # JWTs produced by Search Guard are signed by default with a symmetric HMAC512 hash. For production systems,
        # you must replace the value specified here by your own key. You can generate a new key for example with:
        #  openssl rand -base64 512 | tr '/+' '_-'
        # If you want ot use another signature algorithm, you can specify a complete JWK using the attriubute jwt_signing_key.
        # Refer to the documentation for details.
        jwt_signing_key_hs512: "eTDZjSqRD9Abhod9iqeGX_7o93a-eElTeXWAF6FmzQshmRIrPD-C9ET3pFjJ_IBrzmWIZDk8ig-X_PIyGmKsxNMsrU-0BNWF5gJq5xOp4rYTl8z66Tw9wr8tHLxLxgJqkLSuUCRBZvlZlQ7jNdhBBxgM-hdSSzsN1T33qdIwhrUeJ-KXI5yKUXHjoWFYb9tETbYQ4NvONowkCsXK_flp-E3F_OcKe_z5iVUszAV8QfCod1zhbya540kDejXCL6N_XMmhWJqum7UJ3hgf6DEtroPSnVpHt4iR5w9ArKK-IBgluPght03gNcoNqwz7p77TFbdOmUKF_PWy1bcdbaUoSg"
        # JWTs produced by Search Guard are unencrypted by default. Set a key here to activate encryption using AES Key Wrap.
        # If you want ot use another signature algorithm, you can specify a complete JWK using the attriubute jwt_encryption_key.
        # Refer to the documentation for details.
        #jwt_encryption_key_a256kw: "..."
        # Specify the maximum time period an auth token may be valid. Omit max_validity to have keys with unlimited lifetime.
        # Note that when creating auth tokens, users can specify an even shorter time period.
        max_validity: "1y"
        # This specifies the maximum number of valid tokens a user can have at the same time.
        max_tokens_per_user: 100
      authc:
        kerberos_auth_domain:
          http_enabled: false
          transport_enabled: false
          order: 6
          http_authenticator:
            type: kerberos # NOT FREE FOR COMMERCIAL USE
            challenge: true
            config:
              # If true a lot of kerberos/security related debugging output will be logged to standard out
              krb_debug: false
              # If true then the realm will be stripped from the user name
              strip_realm_from_principal: true
          authentication_backend:
            type: noop
        basic_internal_auth_domain:
          description: "Authenticate via HTTP Basic against internal users database"
          http_enabled: true
          transport_enabled: true
          order: 4
          http_authenticator:
            type: basic
            challenge: true
          authentication_backend:
            type: intern
        proxy_auth_domain:
          description: "Authenticate via proxy"
          http_enabled: false
          transport_enabled: false
          order: 3
          http_authenticator:
            type: proxy
            challenge: false
            config:
              user_header: "x-proxy-user"
              roles_header: "x-proxy-roles"
          authentication_backend:
            type: noop
        jwt_auth_domain:
          description: "Authenticate via Json Web Token"
          http_enabled: false
          transport_enabled: false
          order: 0
          http_authenticator:
            type: jwt
            challenge: false
            config:
              signing_key: "base64 encoded HMAC key or public RSA/ECDSA pem key"
              jwt_header: "Authorization"
              jwt_url_parameter: null
              roles_key: null
              subject_key: null
          authentication_backend:
            type: noop
        sg_issued_jwt_auth_domain:
          description: "Authenticate via Json Web Tokens issued by Search Guard"
          http_enabled: false
          # This auth domain is only available for HTTP
          order: 1
          http_authenticator:
            type: sg_auth_token
            challenge: false
            # This auth domain automatically pulls configuration from the auth_token_provider config above
          authentication_backend:
            type: sg_auth_token
        clientcert_auth_domain:
          description: "Authenticate via SSL client certificates"
          http_enabled: false
          transport_enabled: false
          order: 2
          http_authenticator:
            type: clientcert
            config:
              username_attribute: cn #optional, if omitted DN becomes username
            challenge: false
          authentication_backend:
            type: noop
        ldap:
          description: "Authenticate via LDAP or Active Directory"
          http_enabled: false
          transport_enabled: false
          order: 5
          http_authenticator:
            type: basic
            challenge: false
          authentication_backend:
            # LDAP authentication backend (authenticate users against a LDAP or Active Directory)
            type: ldap # NOT FREE FOR COMMERCIAL USE
            config:
              # enable ldaps
              enable_ssl: false
              # enable start tls, enable_ssl should be false
              enable_start_tls: false
              # send client certificate
              enable_ssl_client_auth: false
              # verify ldap hostname
              verify_hostnames: true
              hosts:
                - localhost:8389
              bind_dn: null
              password: null
              userbase: 'ou=people,dc=example,dc=com'
              # Filter to search for users (currently in the whole subtree beneath userbase)
              # {0} is substituted with the username
              usersearch: '(sAMAccountName={0})'
              # Use this attribute from the user as username (if not set then DN is used)
              username_attribute: null
      authz:
        roles_from_myldap:
          description: "Authorize via LDAP or Active Directory"
          http_enabled: false
          transport_enabled: false
          authorization_backend:
            # LDAP authorization backend (gather roles from a LDAP or Active Directory, you have to configure the above LDAP authentication backend settings too)
            type: ldap # NOT FREE FOR COMMERCIAL USE
            config:
              # enable ldaps
              enable_ssl: false
              # enable start tls, enable_ssl should be false
              enable_start_tls: false
              # send client certificate
              enable_ssl_client_auth: false
              # verify ldap hostname
              verify_hostnames: true
              hosts:
                - localhost:8389
              bind_dn: null
              password: null
              rolebase: 'ou=groups,dc=example,dc=com'
              # Filter to search for roles (currently in the whole subtree beneath rolebase)
              # {0} is substituted with the DN of the user
              # {1} is substituted with the username
              # {2} is substituted with an attribute value from user's directory entry, of the authenticated user. Use userroleattribute to specify the name of the attribute
              rolesearch: '(member={0})'
              # Specify the name of the attribute which value should be substituted with {2} above
              userroleattribute: null
              # Roles as an attribute of the user entry
              userrolename: disabled
              #userrolename: memberOf
              # The attribute in a role entry containing the name of that role, Default is "name".
              # Can also be "dn" to use the full DN as rolename.
              rolename: cn
              # Resolve nested roles transitive (roles which are members of other roles and so on ...)
              resolve_nested_roles: true
              userbase: 'ou=people,dc=example,dc=com'
              # Filter to search for users (currently in the whole subtree beneath userbase)
              # {0} is substituted with the username
              usersearch: '(uid={0})'
              # Skip users matching a user name, a wildcard or a regex pattern
              #skip_users:
              #  - 'cn=Michael Jackson,ou*people,o=TEST'
              #  - '/\S*/'
        roles_from_another_ldap:
          description: "Authorize via another Active Directory"
          http_enabled: false
          transport_enabled: false
          authorization_backend:
            type: ldap # NOT FREE FOR COMMERCIAL USE
            #config goes here ...
  #    auth_failure_listeners:
  #      ip_rate_limiting:
  #        type: ip
  #        allowed_tries: 10
  #        time_window_seconds: 3600
  #        block_expiry_seconds: 600
  #        max_blocked_clients: 100000
  #        max_tracked_clients: 100000
  #      internal_authentication_backend_limiting:
  #        type: username
  #        authentication_backend: intern
  #        allowed_tries: 10
  #        time_window_seconds: 3600
  #        block_expiry_seconds: 600
  #        max_blocked_clients: 100000
  #        max_tracked_clients: 100000

The strange thing is that if I replace the username/password in filebeat.yml with elastic or kibanaserver, the messages come through to Elasticsearch. Even the nonstandard username elastalert_internal can get messages through to Elasticsearch. But the usernames filebeat_internal and metricbeat_internal don’t work, so it seems like SearchGuard is blocking them for some reason. Other than the ping unauthorized error, I cannot determine why they are blocked.

I am seeing this error in the Elasticsearch logs frequently:

2023-03-23 21:12:46.654Z DEBUG [elasticsearch[esnode-aln-nbadev4][transport_worker][T#20]] com.floragunn.searchguard.auth.RestAuthenticationProcessor - 'java.lang.IllegalArgumentException: password must not be null or empty' extracting credentials from basic http authenticator
java.lang.IllegalArgumentException: password must not be null or empty
        at com.floragunn.searchguard.user.AuthCredentials$Builder.password(AuthCredentials.java:335) ~[search-guard-suite-security-7.17.9-53.6.0.jar:7.17.9-53.6.0]
        at com.floragunn.searchguard.support.HTTPHelper.extractCredentials(HTTPHelper.java:70) ~[search-guard-suite-security-7.17.9-53.6.0.jar:7.17.9-53.6.0]
        at com.floragunn.searchguard.http.HTTPBasicAuthenticator.extractCredentials(HTTPBasicAuthenticator.java:55) ~[search-guard-suite-security-7.17.9-53.6.0.jar:7.17.9-53.6.0]
        at com.floragunn.searchguard.auth.RestAuthenticationProcessor.checkCurrentAuthenticationDomain(RestAuthenticationProcessor.java:181) [search-guard-suite-security-7.17.9-53.6.0.jar:7.17.9-53.6.0]
        at com.floragunn.searchguard.auth.RestAuthenticationProcessor.checkNextAuthenticationDomains(RestAuthenticationProcessor.java:117) [search-guard-suite-security-7.17.9-53.6.0.jar:7.17.9-53.6.0]
        at com.floragunn.searchguard.auth.RestAuthenticationProcessor.authenticate(RestAuthenticationProcessor.java:109) [search-guard-suite-security-7.17.9-53.6.0.jar:7.17.9-53.6.0]
        at com.floragunn.searchguard.auth.BackendRegistry.authenticate(BackendRegistry.java:405) [search-guard-suite-security-7.17.9-53.6.0.jar:7.17.9-53.6.0]
        at com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:90) [search-guard-suite-security-7.17.9-53.6.0.jar:7.17.9-53.6.0]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:337) [elasticsearch-7.17.9.jar:7.17.9]
        at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:403) [elasticsearch-7.17.9.jar:7.17.9]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:255) [elasticsearch-7.17.9.jar:7.17.9]
        at com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [search-guard-suite-security-7.17.9-53.6.0.jar:7.17.9-53.6.0]

It turned out that the Elasticsearch password was saved incorrectly in the filebeat (and metricbeat) keystore, so the issue was definitely just an authentication problem.

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.