Metricbeat and Elasticsearch module 401

[rschirin]

Hi,
I’m using Elasticsearch 6.6.1 and Metricbeat 6.6.1. I have already installed SG with the correct version and everything works properly (i.e. LDAP authentication). currently I have installed metricbeat to collect data using Elasticsearch module and below you can find my section:

• module: elasticsearch
  metricsets:
  • node
  • node_stats
  • index
  • index_recovery
  • index_summary
  • shard
  • ml_job
  • ccr
  • cluster_stats
  • pending_tasks
  • shard
    period: 30s
    enabled: true
    hosts: [“https://elk01ci.env.net:9200”]
    user: admin
    password:xxxxx
    ssl.certificate_authorities: [“/etc/elasticsearch/certs/chain.pem”]
    ssl.certificate: “/etc/elasticsearch/certs/elk01ci.cer”
    ssl.key: “/etc/elasticsearch/certs/elk01ci.key”
    ssl.verification_mode: “none”
    ssl.enabled: true

unfortunately, when I start Metricbeat it is not able to collect data with the following error.

ERROR   [elasticsearch] elastic/elastic.go:117  error determining if connected Elasticsearch node is master: HTTP error 401 in shard: 401 Unauthorized

For sure, that node is master since it is my only one node but I cannot fix the 401 error code. Password is correct since I tested it with cURL and user has the correct permissions.

Furthermore, if I disable SG from elasticsearch.yml, everything works correctly.

[cstaley]

Please post your sg_config.yml, elasticsearch.yml and logs from the ES node (if there are no interesting logs pls enable debug logging as explained here Search Guard logging | Security for Elasticsearch | Search Guard).

And i think you should remove ssl.certificate and ssl.key properties in your metricbeat configuration because you already provided a username and a password.

[rschirin]

here you can find my sg_config.yml. basically, I added only the LDAP method:

searchguard:
dynamic:
kibana:
multitenancy/wiki
http:
anonymous_auth_enabled: false
xff:
enabled: false
internalProxies: ‘192.168.0.10|192.168.0.11’ # regex pattern
remoteIpHeader: ‘x-forwarded-for’
proxiesHeader: ‘x-forwarded-by’
authc:
kerberos_auth_domain:
enabled: false
order: 6
http_authenticator:
type: kerberos # NOT FREE FOR COMMERCIAL USE
challenge: true
config:
# If true a lot of kerberos/security related debugging output will be logged to standard out
krb_debug: false
# If true then the realm will be stripped from the user name
strip_realm_from_principal: true
authentication_backend:
type: noop
basic_internal_auth_domain:
enabled: true
order: 0
http_authenticator:
type: basic
challenge: true
authentication_backend:
type: intern
proxy_auth_domain:
enabled: false
order: 3
http_authenticator:
type: proxy
challenge: false
config:
user_header: “x-proxy-user”
roles_header: “x-proxy-roles”
authentication_backend:
type: noop
host_auth_domain:
enabled: false
order: 5
http_authenticator:
type: host #DEPRECATED, will be removed in a future version
challenge: false
authentication_backend:
type: noop
jwt_auth_domain:
enabled: false
order: 4
http_authenticator:
type: jwt
challenge: false
config:
signing_key: “base64 encoded key”
jwt_header: “Authorization”
jwt_url_parameter: null
roles_key: null
subject_key: null
authentication_backend:
type: noop
clientcert_auth_domain:
enabled: false
order: 2
http_authenticator:
type: clientcert
config:
username_attribute: cn #optional, if omitted DN becomes username
challenge: false
authentication_backend:
type: noop
ldap:
enabled: true
order: 1
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: ldap # NOT FREE FOR COMMERCIAL USE
config:
enable_ssl: false
enable_start_tls: false
enable_ssl_client_auth: false
verify_hostnames: false
hosts:
- dcoll.net:389
bind_dn: CN=elksget,OU=Application Users,DC=dcoll,DC=net
password: ZPJdd2RiuJ_DgzIllivH
userbase: ‘DC=dcoll,DC=net’
usersearch: ‘(sAMAccountName={0})’
username_attribute: null
authz:
roles_from_myldap:
enabled: true
authorization_backend:
type: ldap # NOT FREE FOR COMMERCIAL USE
config:
enable_ssl: false
enable_start_tls: false
enable_ssl_client_auth: false
verify_hostnames: true
hosts:
- dcoll.net:389
bind_dn: CN=elksget,OU=Application Users,DC=dcoll,DC=net
password: ZPJdd2RiuJ_DgzIllivH
rolebase: ‘OU=Groups,DC=dcoll,DC=net’
rolesearch: ‘(member={0})’
userroleattribute: null
userrolename: memberOf
rolename: cn
resolve_nested_roles: true
userbase: ‘DC=dcoll,DC=net’
usersearch: ‘(sAMAccountName={0})’

my elasticsearch.yml

node.name: ${HOSTNAME}
node.master: true
node.data: true
node.ingest: false
path.data: /var/es
path.logs: /var/log/elasticsearch
network.host: 172.21.40.224
http.port: 9200
discovery.zen.ping.unicast.hosts: [“172.21.40.224”]
xpack.security.enabled: false
searchguard.disabled: false
searchguard.ssl.transport.pemcert_filepath: certs/elkcrc01ci.cer
searchguard.ssl.transport.pemkey_filepath: certs/elkcrc01ci.key
searchguard.ssl.transport.pemtrustedcas_filepath: certs/FullChainCA.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: certs/elkcrc01ci.cer
searchguard.ssl.http.pemkey_filepath: certs/elkcrc01ci.key
searchguard.ssl.http.pemtrustedcas_filepath: certs/FullChainCA.pem
searchguard.nodes_dn:

• xxx
  searchguard.authcz.admin_dn:
• xxx

I have enabled the debug level and this is the result:

[2019-07-08T17:25:37,509][DEBUG][c.f.s.a.BackendRegistry  ] [elkcrc01ci] Rest authentication request from 172.21.40.224:60710 [original: /172.21.40.224:60710]
[2019-07-08T17:25:39,197][DEBUG][c.f.s.a.BackendRegistry  ] [elkcrc01ci] Rest authentication request from 172.21.40.224:60712 [original: /172.21.40.224:60712]
[2019-07-08T17:25:39,805][DEBUG][c.f.s.a.BackendRegistry  ] [elkcrc01ci] Rest authentication request from 172.21.40.224:60714 [original: /172.21.40.224:60714]
[2019-07-08T17:25:40,456][DEBUG][c.f.s.a.BackendRegistry  ] [elkcrc01ci] Rest authentication request from 172.21.40.224:60716 [original: /172.21.40.224:60716]
[2019-07-08T17:25:40,848][DEBUG][c.f.s.a.BackendRegistry  ] [elkcrc01ci] Rest authentication request from 172.21.40.224:60718 [original: /172.21.40.224:60718]

my log file is full of these entries.

[rschirin]

solved.
issue was due a wrong key in elasticsearch module.
user → username