Kibana not working after installing FLX beta 2

faxmodem · July 4, 2022, 9:15am

I followed the procedure described here to upgrade searchguard to FLX beta 2 on one of my ES 7.10.2 clusters.
Everything went smoothly, until I tried to restart Kibana.
Now The browser displays an 50X error, and this is what I see in the kibana log:

kibana[13634]: [ResponseError]: Response Error
kibana[13634]: Multitenancy: Could not get authinfo AuthenticationError: Response Error
kibana[13634]: [security_exception]: Insufficient permissions
kibana[13634]: Error while retrieving auth config { ResponseError: security_exception
kibana[13634]:     at IncomingMessage.response.on (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:272:25)
kibana[13634]:     at IncomingMessage.emit (events.js:203:15)
kibana[13634]:     at endReadableNT (_stream_readable.js:1145:12)
kibana[13634]:     at process._tickCallback (internal/process/next_tick.js:63:19)
kibana[13634]:   name: 'ResponseError',
kibana[13634]:   meta:
kibana[13634]:    { body: { error: [Object], status: 403 },
kibana[13634]:      statusCode: 403,
kibana[13634]:      headers:
kibana[13634]:       { 'content-type': 'application/json; charset=UTF-8',
kibana[13634]:         'content-length': '319' },
kibana[13634]:      meta:
kibana[13634]:       { context: null,
kibana[13634]:         request: [Object],
kibana[13634]:         name: 'elasticsearch-js',
kibana[13634]:         connection: [Object],
kibana[13634]:         attempts: 0,
kibana[13634]:         aborted: false } } }
kibana[13634]: POST /api/core/capabilities 200 102ms - 9.0B
kibana[13634]: [ResponseError]: Response Error
kibana[13634]: Multitenancy: Could not get authinfo AuthenticationError: Response Error
kibana[13634]: GET /api/v1/systeminfo 200 32ms - 9.0B
kibana[13634]: [ResponseError]: Response Error
kibana[13634]: Multitenancy: Could not get authinfo AuthenticationError: Response Error
kibana[13634]: GET /api/v1/systeminfo 200 25ms - 9.0B
kibana[13634]: [ResponseError]: Response Error
kibana[13634]: Multitenancy: Could not get authinfo AuthenticationError: Response Error
kibana[13634]: [security_exception]: Insufficient permissions
kibana[13634]: { ResponseError: security_exception
    at IncomingMessage.response.on (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:272:25)
    at IncomingMessage.emit (events.js:203:15)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  name: 'ResponseError',
  meta:
   { body: { error: [Object], status: 403 },
     statusCode: 403,
     headers:
      { 'content-type': 'application/json; charset=UTF-8',
        'content-length': '319' },
     meta:
      { context: null,
        request: [Object],
        name: 'elasticsearch-js',
        connection: [Object],
        attempts: 0,
        aborted: false } } }
kibana[13634]: Internal Server Error
kibana[13634]: GET /api/v1/auth/config 500 27ms - 9.0B

nils · July 4, 2022, 2:35pm

Thank you for the report!

Can you please also add the following information:

Any exceptions from the Elasticsearch log that appear when you are trying to log in
Your kibana.yml file.
If you are using a custom kibanaserver user, the roles that are assigned to that user

faxmodem · July 5, 2022, 7:13am

Here’s the ES log:

Error while mapping auth credentials for trusted_origin[adca255d]
Authentication failed for null from [request=/_searchguard/authinfo, directIpAddress=192.168.245.175, originatingIpAddress=192.168.245.175, clientCertSubject=null]
No cluster-level perm match for User kibana <basic/internal_users_db> UNKNOWN [Action [cluster:admin:searchguard:auth/frontend/config/get]] [RolesChecked [sg_role_kibana4_server]]:
Evaluated Privileges:
_/cluster:admin:searchguard:auth/frontend/config/get: MISSING

Error while mapping auth credentials for trusted_origin[adca255d]
Error while mapping auth credentials for trusted_origin[adca255d]
Error while mapping auth credentials for trusted_origin[adca255d]
Authentication failed for null from [request=/_searchguard/authinfo, directIpAddress=192.168.245.175, originatingIpAddress=192.168.245.175, clientCertSubject=null]
No cluster-level perm match for User kibana <basic/internal_users_db> UNKNOWN [Action [cluster:admin:searchguard:auth/frontend/config/get]] [RolesChecked [sg_role_kibana4_server]]:
Evaluated Privileges:
_/cluster:admin:searchguard:auth/frontend/config/get: MISSING

Authentication failed for null from [request=/_searchguard/authinfo, directIpAddress=192.168.245.175, originatingIpAddress=192.168.245.175, clientCertSubject=null]
Authentication failed for null from [request=/_searchguard/authinfo, directIpAddress=192.168.245.175, originatingIpAddress=192.168.245.175, clientCertSubject=null]
Authentication failed for null from [request=/_searchguard/authinfo, directIpAddress=192.168.245.175, originatingIpAddress=192.168.245.175, clientCertSubject=null]
No cluster-level perm match for User kibana <basic/internal_users_db> UNKNOWN [Action [cluster:admin:searchguard:auth/frontend/config/get]] [RolesChecked [sg_role_kibana4_server]]:
Evaluated Privileges:
_/cluster:admin:searchguard:auth/frontend/config/get: MISSING

kibana.yml:

---
elasticsearch.hosts:
- https://one:9200/
- https://two:9200/
- https://three:9200/
elasticsearch.password: kibana
elasticsearch.requestHeadersWhitelist:
- sgtenant
- Authorization
elasticsearch.requestTimeout: 1200000
elasticsearch.ssl.certificateAuthorities: "/opt/kibana/ssl/ca.pem"
elasticsearch.ssl.verificationMode: full
elasticsearch.username: kibana
kibana.index: ".kibana"
logging.json: true
searchguard.multitenancy.enabled: true
searchguard.multitenancy.tenants.enable_global: true
searchguard.multitenancy.tenants.enable_private: true
server.host: foo.example.com
server.name: foo.example.com
server.port: 443
server.ssl.certificate: "/opt/kibana/ssl/cert.pem"
server.ssl.certificateAuthorities: "/etc/pki/tls/certs/ca-bundle.trust.crt"
server.ssl.enabled: true
server.ssl.key: "/opt/kibana/ssl/key.pem"

I just realised the kibana user doesn’t have any special credentials:

☠ ES _searchguard/api/internalusers/kibana
HTTP/1.1 200 OK
content-length: 151
content-type: application/json; charset=UTF-8

{
    "kibana": {
        "attributes": {},
        "backend_roles": [],
        "description": "Migrated from v6",
        "hidden": false,
        "reserved": false,
        "search_guard_roles": [],
        "static": false
    }
}

faxmodem · July 5, 2022, 7:17am

EDIT: the kibana user does have privileges, they just don’t show in the internalusers api call.
Here they are:

{
    "sg_role_kibana4_server": {
        "cluster_permissions": [
            "CLUSTER_MONITOR",
            "CLUSTER_COMPOSITE_OPS",
            "cluster:admin/xpack/monitoring*",
            "indices:admin/template*",
            "indices:data/read/scroll*"
        ],
        "description": "Migrated from v6 (all types mapped)",
        "exclude_cluster_permissions": [],
        "exclude_index_permissions": [],
        "hidden": false,
        "index_permissions": [
            {
                "allowed_actions": [
                    "INDICES_ALL"
                ],
                "fls": [],
                "index_patterns": [
                    "?kibana"
                ],
                "masked_fields": []
            },
            {
                "allowed_actions": [
                    "INDICES_ALL"
                ],
                "fls": [],
                "index_patterns": [
                    "?kibana-6"
                ],
                "masked_fields": []
            },
            {
                "allowed_actions": [
                    "INDICES_ALL"
                ],
                "fls": [],
                "index_patterns": [
                    "?kibana_*"
                ],
                "masked_fields": []
            },
            {
                "allowed_actions": [
                    "INDICES_ALL"
                ],
                "fls": [],
                "index_patterns": [
                    "?reporting*"
                ],
                "masked_fields": []
            },
            {
                "allowed_actions": [
                    "INDICES_ALL"
                ],
                "fls": [],
                "index_patterns": [
                    "?monitoring*"
                ],
                "masked_fields": []
            },
            {
                "allowed_actions": [
                    "INDICES_ALL"
                ],
                "fls": [],
                "index_patterns": [
                    "?tasks"
                ],
                "masked_fields": []
            },
            {
                "allowed_actions": [
                    "indices:admin/aliases*"
                ],
                "fls": [],
                "index_patterns": [
                    "*"
                ],
                "masked_fields": []
            }
        ],
        "reserved": true,
        "static": false,
        "tenant_permissions": []
    }
}

faxmodem · July 5, 2022, 7:27am

I added the kibana user to the internal SGS_KIBANA_SERVER role, using this command:

☠ ES PUT _searchguard/api/rolesmapping/SGS_KIBANA_SERVER <<< '{"users":["kibana"]}'
HTTP/1.1 201 Created
content-length: 61
content-type: application/json; charset=UTF-8

{
    "message": "'SGS_KIBANA_SERVER' created.",
    "status": "CREATED"
}

☠ ES _searchguard/api/rolesmapping/SGS_KIBANA_SERVER
HTTP/1.1 200 OK
content-length: 42
content-type: application/json; charset=UTF-8

{
    "SGS_KIBANA_SERVER": {
        "users": [
            "kibana"
        ]
    }
}

But I still get the 50x error

nils · July 5, 2022, 8:16am

Thank you. Can you please try to assign the role SGS_KIBANA_SERVER to search_guard_roles of the user kibana? Only then, the user will have sufficient privileges.

faxmodem · July 5, 2022, 8:20am

☠ ES _searchguard/api/internalusers/kibana
HTTP/1.1 200 OK
content-length: 151
content-type: application/json; charset=UTF-8

{
    "kibana": {
        "attributes": {},
        "backend_roles": [],
        "description": "Migrated from v6",
        "hidden": false,
        "reserved": false,
        "search_guard_roles": [],
        "static": false
    }
}

☠ ES PUT _searchguard/api/internalusers/kibana <<< '{"search_guard_roles":["SGS_KIBANA_SERVER"]}'
HTTP/1.1 200 OK
content-length: 45
content-type: application/json; charset=UTF-8

{
    "message": "'kibana' updated.",
    "status": "OK"
}

☠ ES _searchguard/api/internalusers/kibana
HTTP/1.1 200 OK
content-length: 55
content-type: application/json; charset=UTF-8

{
    "kibana": {
        "search_guard_roles": [
            "SGS_KIBANA_SERVER"
        ]
    }
}

faxmodem · July 5, 2022, 8:20am

This didn’t help

nils · July 5, 2022, 8:27am

Can you please try to restart Kibana?

If that does not help, can you please post the content of the ES log again, like you already did above?

faxmodem · July 5, 2022, 8:42am

I restarted kibana, and the ES log just says:

Error while mapping auth credentials for trusted_origin[adca255d]
Authentication failed for null from [request=/_searchguard/authinfo, directIpAddress=192.168.245.175, originatingIpAddress=192.168.245.175, clientCertSubject=null]

nils · July 5, 2022, 8:45am

Can you please also post

The Kibana logs
sg_authc.yml?

Sorry for the inconvenience!

faxmodem · July 5, 2022, 9:25am

[ResponseError]: Response Error
Multitenancy: Could not get authinfo AuthenticationError: Response Error
GET /api/v1/systeminfo 200 21ms - 9.0B
[ResponseError]: Response Error
Multitenancy: Could not get authinfo AuthenticationError: Response Error
[status_exception]: No such frontend config: default
{ ResponseError: status_exception
    at IncomingMessage.response.on (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:272:25)
    at IncomingMessage.emit (events.js:203:15)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  name: 'ResponseError',
  meta:
   { body: { error: [Object], status: 404 },
     statusCode: 404,
     headers:
      { 'content-type': 'application/json; charset=UTF-8',
        'content-length': '181' },
     meta:
      { context: null,
        request: [Object],
        name: 'elasticsearch-js',
        connection: [Object],
        attempts: 0,
        aborted: false } } }
Internal Server Error
GET /api/v1/auth/config 500 19ms - 9.0B

sg_authc.yaml:

---
auth_domains:
- type: "trusted_origin"
  user_mapping:
    user_name:
      from: "$.request.headers[\"x-authenticated-user\"]"
    roles:
      from_comma_separated_string: "$.request.headers[\"x-authenticated-group\"]"
- type: "clientcert"
- type: "basic/internal_users_db"
- type: "kerberos"
network:
  trusted_proxies_regex: "127.0.0.1"
  http:
    remote_ip_header: "X-Forwarded-For"

nils · July 5, 2022, 9:44am

This indicates that the sg_frontend_authc.yml file is missing. Can you please double check that it is present? It should have been created by the sgctl migrate-config process.

You can upload it using `sgctl.sh update-config path/to/sg_frontend_authc.yml

faxmodem · July 5, 2022, 10:53am

I had done this, when following the documentation.
I just pushed it again, to no avail:

$ ../sgctl-0.2.5/sgctl.sh update-config sg_frontend_authc.yml
Successfully connected to elasticsearch as user CN=sgadmin,OU=client,O=client,L=Test,C=FR
Configuration has been updated

faxmodem · July 5, 2022, 10:55am

Here’s its contents:

---
auth_domains:
- type: "oidc"
  idp.openid_configuration_url: "..."
  client_id: "..."
  client_secret: "..."
  user_mapping.subject: "$['preferred_username']"

nils · July 5, 2022, 11:23am

Oh, was this created by sgctl migrate-config? Then, there are apparently few bugs present.

Please try the following template:

default:
  auth_domains:
  - type: "oidc"
    oidc.idp.openid_configuration_url: "..."
    oidc.client_id: "..."
    oidc.client_secret: "..."
    user_mapping.subject: "$['preferred_username']"

faxmodem · July 5, 2022, 11:28am

yes, sgctl created that file. I added the oidc` prefix to the 3 keys, restarted kibana, still same error.

nils · July 5, 2022, 11:36am

Did you also add the top-level default:?

faxmodem · July 5, 2022, 11:36am

I also saw the following message:

[illegal_argument_exception]: request [/_license] contains unrecognized parameter: [accept_enterprise]

faxmodem · July 5, 2022, 11:38am

Invalid config files:

sg_frontend_authc.yml:
  default.auth_domains.0.user_mapping.subject:
        Unsupported attribute

Topic		Replies	Views
Search Guard FLX DLS/FLS breaks Kibana multi-tenancy Search Guard	21	810	January 6, 2023
Kibana multitenancy plugin - unauthenticated access to /api/status results in AuthenticationError Search Guard	4	1029	May 4, 2022
Kibana does not access Search Guard	8	378	December 4, 2017
6.4.0 can't login and kibanaserver user can't access .kibana Search Guard	11	1221	December 12, 2018
Cannot login into Kibana after installing SearchGuard Search Guard	5	3882	May 14, 2022

Kibana not working after installing FLX beta 2

Related Topics