Authentication finally failed for null from $ip_address

Search Guard
1

Hi there,

we have following running:

elastic 7.13.2
kibana 7.13.2
respective sg versions

Currently we get spammed by following logs coming from kibana (which is running on openshift):

Authentication finally failed for null from 1.1.1.1

sg_config.yaml:

  basic_internal_auth_domain:
    description: "Authenticate via HTTP Basic against internal users database"
    http_enabled: true
    transport_enabled: true
    order: 1
    http_authenticator:
      type: "basic"
      challenge: false
    authentication_backend:
      type: internal
  openid_auth_domain:
    http_enabled: true
    order: 0
    http_authenticator:
      type: "openid"
      challenge: false
      config:
        openid_connect_url: "https://auth/.well-known/openid-configuration"
        subject_key: "preferred_username"
        roles_key: "group"
        skip_users:
        - "kibanaserver"
    authentication_backend:
      type: "noop"

kibana.yaml:

server.name: kibana-7-13-2
server.host: "0.0.0.0"
csp.strict: true
elasticsearch.hosts:
  - https://host01.com:9200 
  - https://host02.com:9200 
  - https://host03.com:9200 
elasticsearch.username: ${KIBANASERVER_USER}
elasticsearch.password: ${KIBANASEVER_PASS}
xpack.security.enabled: false
searchguard.auth.type: "openid"
searchguard.openid.connect_url: "https://auth/.well-known/openid-configuration" 
searchguard.openid.client_id: ${KEYCLOAK_CLIENT_ID}
searchguard.openid.client_secret: ${KEYCLOAK_CLIENT_SECRET}
searchguard.openid.base_redirect_url: https://kibana.com/ 
elasticsearch.requestHeadersWhitelist: [ "Authorization", "sgtenant" ]
xpack.reporting.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
xpack.security.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
xpack.encryptedSavedObjects.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
xpack.reporting.csv.maxSizeBytes: 1048576000
elasticsearch.ssl:
  certificateAuthorities: /usr/share/kibana/config/certs/elastic-root-ca
  verificationMode: full

Anyone got any hints?

2

Hi @Kosmonafft

Could you try to change the order in sg_config.yml?

3

Hey Pablo,

thanks for the response.

changed the order to following:

  basic_internal_auth_domain:
    description: "Authenticate via HTTP Basic against internal users database"
    http_enabled: true
    transport_enabled: true
    order: 0
    http_authenticator:
      type: "basic"
      challenge: false
    authentication_backend:
      type: internal
  openid_auth_domain:
    http_enabled: true
    order: 1
    http_authenticator:
      type: "openid"
      challenge: false
      config:
        openid_connect_url: "https://auth/.well-known/openid-configuration"
        subject_key: "preferred_username"
        roles_key: "group"
        skip_users:
        - "kibanaserver"
    authentication_backend:
      type: "noop"

shows exact same behaviour:

[2021-07-13T16:36:10,002][WARN ][c.f.s.a.RestAuthenticationProcessor] [host01.com] Authentication finally failed for null from 1.1.1.1

4

@Kosmonafft, How often do you see that warning? What is 1.1.1.1? Is it Kibana?

5

@Kosmonafft, Could you tell me what IdP solution do you use?

6

I see the warning all the time. I think it shows at every interaction of kibana and elasticsearch. 1.1.1.1. is the IP address of the node where kibana is running.

We are using keycloak as idp provider

7

Could you post here or DM Kibana and ES logs? Just the time frame where ES reports the warning.

8

Here are the logs

elastic node:

[2021-07-22T10:16:10,354][WARN ][c.f.s.a.RestAuthenticationProcessor] [elastic01.node.com] Authentication finally failed for null from 1.1.1.1
[2021-07-22T10:16:10,354][WARN ][c.f.s.a.RestAuthenticationProcessor] [elastic01.node.com] Authentication finally failed for null from 1.1.1.1
[2021-07-22T10:16:49,369][WARN ][c.f.s.a.RestAuthenticationProcessor] [elastic01.node.com] Authentication finally failed for null from 1.1.1.1
[2021-07-22T10:16:49,369][WARN ][c.f.s.a.RestAuthenticationProcessor] [elastic01.node.com] Authentication finally failed for null from 1.1.1.1
[2021-07-22T10:17:10,355][WARN ][c.f.s.a.RestAuthenticationProcessor] [elastic01.node.com] Authentication finally failed for null from 1.1.1.1
[2021-07-22T10:17:10,355][WARN ][c.f.s.a.RestAuthenticationProcessor] [elastic01.node.com] Authentication finally failed for null from 1.1.1.1
[2021-07-22T10:17:10,356][WARN ][c.f.s.a.RestAuthenticationProcessor] [elastic01.node.com] Authentication finally failed for null from 1.1.1.1
[2021-07-22T10:17:10,358][WARN ][c.f.s.a.RestAuthenticationProcessor] [elastic01.node.com] Authentication finally failed for null from 1.1.1.1
[2021-07-22T10:17:10,359][WARN ][c.f.s.a.RestAuthenticationProcessor] [elastic01.node.com] Authentication finally failed for null from 1.1.1.1
[2021-07-22T10:17:10,359][WARN ][c.f.s.a.RestAuthenticationProcessor] [elastic01.node.com] Authentication finally failed for null from 1.1.1.1
[2021-07-22T10:17:10,360][WARN ][c.f.s.a.RestAuthenticationProcessor] [elastic01.node.com] Authentication finally failed for null from 1.1.1.1
[2021-07-22T10:17:10,360][WARN ][c.f.s.a.RestAuthenticationProcessor] [elastic01.node.com] Authentication finally failed for null from 1.1.1.1
[2021-07-22T10:17:10,361][WARN ][c.f.s.a.RestAuthenticationProcessor] [elastic01.node.com] Authentication finally failed for null from 1.1.1.1
[2021-07-22T10:17:10,362][WARN ][c.f.s.a.RestAuthenticationProcessor] [elastic01.node.com] Authentication finally failed for null from 1.1.1.1
[2021-07-22T10:17:10,364][WARN ][c.f.s.a.RestAuthenticationProcessor] [elastic01.node.com] Authentication finally failed for null from 1.1.1.1
[2021-07-22T10:17:10,364][WARN ][c.f.s.a.RestAuthenticationProcessor] [elastic01.node.com] Authentication finally failed for null from 1.1.1.1
[2021-07-22T10:17:10,364][WARN ][c.f.s.a.RestAuthenticationProcessor] [elastic01.node.com] Authentication finally failed for null from 1.1.1.1
[2021-07-22T10:17:10,365][WARN ][c.f.s.a.RestAuthenticationProcessor] [elastic01.node.com] Authentication finally failed for null from 1.1.1.1
[2021-07-22T10:17:10,365][WARN ][c.f.s.a.RestAuthenticationProcessor] [elastic01.node.com] Authentication finally failed for null from 1.1.1.1
[2021-07-22T10:17:10,369][WARN ][c.f.s.a.RestAuthenticationProcessor] [elastic01.node.com] Authentication finally failed for null from 1.1.1.1

Around same time there are these logs in kibana visible. Those consists of:

  • lifeness checks of the deployment (kibana is running on openshift)
  • and some weird errors of the alert plugin which I do not know where it comes from yet
{"type":"log","@timestamp":"2021-07-22T08:15:07+00:00","tags":["error","plugins","alerting","plugins","alerting"],"pid":19,"message":"Executing Alert \"cb7f3650-d9b2-11eb-b74d-b5a9d9fe47d1\" has resulted in Error: Response Error"}
{"type":"log","@timestamp":"2021-07-22T08:15:07+00:00","tags":["error","plugins","alerting","plugins","alerting"],"pid":19,"message":"Executing Alert \"cb80e400-d9b2-11eb-b74d-b5a9d9fe47d1\" has resulted in Error: Response Error"}
{"type":"log","@timestamp":"2021-07-22T08:15:07+00:00","tags":["error","plugins","alerting","plugins","alerting"],"pid":19,"message":"Executing Alert \"cb6daa20-d9b2-11eb-b74d-b5a9d9fe47d1\" has resulted in Error: Response Error"}
{"type":"log","@timestamp":"2021-07-22T08:15:07+00:00","tags":["error","plugins","alerting","plugins","alerting"],"pid":19,"message":"Executing Alert \"cb7fd290-d9b2-11eb-b74d-b5a9d9fe47d1\" has resulted in Error: Response Error"}
{"type":"log","@timestamp":"2021-07-22T08:15:07+00:00","tags":["error","plugins","alerting","plugins","alerting"],"pid":19,"message":"Executing Alert \"cb6ce6d0-d9b2-11eb-b74d-b5a9d9fe47d1\" has resulted in Error: Response Error"}
{"type":"log","@timestamp":"2021-07-22T08:15:07+00:00","tags":["error","plugins","alerting","plugins","alerting"],"pid":19,"message":"Executing Alert \"cb813220-d9b2-11eb-b74d-b5a9d9fe47d1\" has resulted in Error: Response Error"}
{"type":"log","@timestamp":"2021-07-22T08:15:07+00:00","tags":["error","plugins","alerting","plugins","alerting"],"pid":19,"message":"Executing Alert \"cb706940-d9b2-11eb-b74d-b5a9d9fe47d1\" has resulted in Error: Response Error"}
{"type":"log","@timestamp":"2021-07-22T08:15:07+00:00","tags":["error","plugins","alerting","plugins","alerting"],"pid":19,"message":"Executing Alert \"cb783170-d9b2-11eb-b74d-b5a9d9fe47d1\" has resulted in Error: Response Error"}
{"type":"response","@timestamp":"2021-07-22T08:15:11+00:00","tags":["api"],"pid":19,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"host":"localhost:5601","user-agent":"curl/7.61.1","accept":"*/*"},"remoteAddress":"127.0.0.1","userAgent":"curl/7.61.1"},"res":{"statusCode":200,"responseTime":3,"contentLength":22284},"message":"GET /api/status 200 3ms - 21.8KB"}
{"type":"response","@timestamp":"2021-07-22T08:15:21+00:00","tags":["api"],"pid":19,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"host":"localhost:5601","user-agent":"curl/7.61.1","accept":"*/*"},"remoteAddress":"127.0.0.1","userAgent":"curl/7.61.1"},"res":{"statusCode":200,"responseTime":2,"contentLength":22284},"message":"GET /api/status 200 2ms - 21.8KB"}
{"type":"response","@timestamp":"2021-07-22T08:15:31+00:00","tags":["api"],"pid":19,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"host":"localhost:5601","user-agent":"curl/7.61.1","accept":"*/*"},"remoteAddress":"127.0.0.1","userAgent":"curl/7.61.1"},"res":{"statusCode":200,"responseTime":3,"contentLength":22284},"message":"GET /api/status 200 3ms - 21.8KB"}
{"type":"response","@timestamp":"2021-07-22T08:15:41+00:00","tags":["api"],"pid":19,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"host":"localhost:5601","user-agent":"curl/7.61.1","accept":"*/*"},"remoteAddress":"127.0.0.1","userAgent":"curl/7.61.1"},"res":{"statusCode":200,"responseTime":3,"contentLength":22284},"message":"GET /api/status 200 3ms - 21.8KB"}
{"type":"log","@timestamp":"2021-07-22T08:15:43+00:00","tags":["warning","plugins","alerting","plugins","alerting"],"pid":19,"message":"Error executing alerting apiKey invalidation task: Response Error"}
{"type":"response","@timestamp":"2021-07-22T08:15:51+00:00","tags":["api"],"pid":19,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"host":"localhost:5601","user-agent":"curl/7.61.1","accept":"*/*"},"remoteAddress":"127.0.0.1","userAgent":"curl/7.61.1"},"res":{"statusCode":200,"responseTime":6,"contentLength":22285},"message":"GET /api/status 200 6ms - 21.8KB"}
{"type":"response","@timestamp":"2021-07-22T08:16:01+00:00","tags":["api"],"pid":19,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"host":"localhost:5601","user-agent":"curl/7.61.1","accept":"*/*"},"remoteAddress":"127.0.0.1","userAgent":"curl/7.61.1"},"res":{"statusCode":200,"responseTime":2,"contentLength":22284},"message":"GET /api/status 200 2ms - 21.8KB"}
{"type":"log","@timestamp":"2021-07-22T08:16:10+00:00","tags":["error","plugins","alerting","plugins","alerting"],"pid":19,"message":"Executing Alert \"cb8095e0-d9b2-11eb-b74d-b5a9d9fe47d1\" has resulted in Error: Response Error"}
{"type":"response","@timestamp":"2021-07-22T08:16:11+00:00","tags":["api"],"pid":19,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"host":"localhost:5601","user-agent":"curl/7.61.1","accept":"*/*"},"remoteAddress":"127.0.0.1","userAgent":"curl/7.61.1"},"res":{"statusCode":200,"responseTime":4,"contentLength":22285},"message":"GET /api/status 200 4ms - 21.8KB"}
{"type":"response","@timestamp":"2021-07-22T08:16:21+00:00","tags":["api"],"pid":19,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"host":"localhost:5601","user-agent":"curl/7.61.1","accept":"*/*"},"remoteAddress":"127.0.0.1","userAgent":"curl/7.61.1"},"res":{"statusCode":200,"responseTime":2,"contentLength":22284},"message":"GET /api/status 200 2ms - 21.8KB"}
{"type":"response","@timestamp":"2021-07-22T08:16:31+00:00","tags":["api"],"pid":19,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"host":"localhost:5601","user-agent":"curl/7.61.1","accept":"*/*"},"remoteAddress":"127.0.0.1","userAgent":"curl/7.61.1"},"res":{"statusCode":200,"responseTime":3,"contentLength":22283},"message":"GET /api/status 200 3ms - 21.8KB"}
{"type":"response","@timestamp":"2021-07-22T08:16:41+00:00","tags":["api"],"pid":19,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"host":"localhost:5601","user-agent":"curl/7.61.1","accept":"*/*"},"remoteAddress":"127.0.0.1","userAgent":"curl/7.61.1"},"res":{"statusCode":200,"responseTime":3,"contentLength":22283},"message":"GET /api/status 200 3ms - 21.8KB"}
{"type":"log","@timestamp":"2021-07-22T08:16:49+00:00","tags":["error","plugins","alerting","plugins","alerting"],"pid":19,"message":"Executing Alert \"cb7f0f40-d9b2-11eb-b74d-b5a9d9fe47d1\" has resulted in Error: Response Error"}
{"type":"response","@timestamp":"2021-07-22T08:16:51+00:00","tags":["api"],"pid":19,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"host":"localhost:5601","user-agent":"curl/7.61.1","accept":"*/*"},"remoteAddress":"127.0.0.1","userAgent":"curl/7.61.1"},"res":{"statusCode":200,"responseTime":4,"contentLength":22284},"message":"GET /api/status 200 4ms - 21.8KB"}
{"type":"response","@timestamp":"2021-07-22T08:17:01+00:00","tags":["api"],"pid":19,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"host":"localhost:5601","user-agent":"curl/7.61.1","accept":"*/*"},"remoteAddress":"127.0.0.1","userAgent":"curl/7.61.1"},"res":{"statusCode":200,"responseTime":2,"contentLength":22280},"message":"GET /api/status 200 2ms - 21.8KB"}
9

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.