# Kibana not working after installing FLX beta 2

**URL:** <https://forum.search-guard.com/t/kibana-not-working-after-installing-flx-beta-2/2333>\
**Category:** Search Guard\
**Created:** [July 4, 2022, 9:15am UTC](https://forum.search-guard.com/t/kibana-not-working-after-installing-flx-beta-2/2333 "2022-07-04T09:15:36Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![faxmodem](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/faxmodem/32/22_2.png) [@faxmodem](https://forum.search-guard.com/u/faxmodem)\
**Post date:** [July 4, 2022, 9:15am UTC](https://forum.search-guard.com/t/kibana-not-working-after-installing-flx-beta-2/2333/1 "2022-07-04T09:15:37Z")

</div>

I followed the procedure described [here](https://docs.search-guard.com/flx/sg-classic-config-migration-prod) to upgrade searchguard to FLX beta 2 on one of my ES 7.10.2 clusters.  
Everything went smoothly, until I tried to restart Kibana.  
Now The browser displays an 50X error, and this is what I see in the kibana log:

```auto
kibana[13634]: [ResponseError]: Response Error
kibana[13634]: Multitenancy: Could not get authinfo AuthenticationError: Response Error
kibana[13634]: [security_exception]: Insufficient permissions
kibana[13634]: Error while retrieving auth config { ResponseError: security_exception
kibana[13634]: at IncomingMessage.response.on (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:272:25)
kibana[13634]: at IncomingMessage.emit (events.js:203:15)
kibana[13634]: at endReadableNT (_stream_readable.js:1145:12)
kibana[13634]: at process._tickCallback (internal/process/next_tick.js:63:19)
kibana[13634]: name: 'ResponseError',
kibana[13634]: meta:
kibana[13634]: { body: { error: [Object], status: 403 },
kibana[13634]: statusCode: 403,
kibana[13634]: headers:
kibana[13634]: { 'content-type': 'application/json; charset=UTF-8',
kibana[13634]: 'content-length': '319' },
kibana[13634]: meta:
kibana[13634]: { context: null,
kibana[13634]: request: [Object],
kibana[13634]: name: 'elasticsearch-js',
kibana[13634]: connection: [Object],
kibana[13634]: attempts: 0,
kibana[13634]: aborted: false } } }
kibana[13634]: POST /api/core/capabilities 200 102ms - 9.0B
kibana[13634]: [ResponseError]: Response Error
kibana[13634]: Multitenancy: Could not get authinfo AuthenticationError: Response Error
kibana[13634]: GET /api/v1/systeminfo 200 32ms - 9.0B
kibana[13634]: [ResponseError]: Response Error
kibana[13634]: Multitenancy: Could not get authinfo AuthenticationError: Response Error
kibana[13634]: GET /api/v1/systeminfo 200 25ms - 9.0B
kibana[13634]: [ResponseError]: Response Error
kibana[13634]: Multitenancy: Could not get authinfo AuthenticationError: Response Error
kibana[13634]: [security_exception]: Insufficient permissions
kibana[13634]: { ResponseError: security_exception
    at IncomingMessage.response.on (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:272:25)
    at IncomingMessage.emit (events.js:203:15)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  name: 'ResponseError',
  meta:
   { body: { error: [Object], status: 403 },
     statusCode: 403,
     headers:
      { 'content-type': 'application/json; charset=UTF-8',
        'content-length': '319' },
     meta:
      { context: null,
        request: [Object],
        name: 'elasticsearch-js',
        connection: [Object],
        attempts: 0,
        aborted: false } } }
kibana[13634]: Internal Server Error
kibana[13634]: GET /api/v1/auth/config 500 27ms - 9.0B

```

---

<div class="post-metadata">

**Author:** ![nils](https://avatars.discourse-cdn.com/v4/letter/n/d6d6ee/32.png) [@nils](https://forum.search-guard.com/u/nils)\
**Post date:** [July 4, 2022, 2:35pm UTC](https://forum.search-guard.com/t/kibana-not-working-after-installing-flx-beta-2/2333/2 "2022-07-04T14:35:29Z")

</div>

Thank you for the report!

Can you please also add the following information:

- Any exceptions from the Elasticsearch log that appear when you are trying to log in
- Your kibana.yml file.
- If you are using a custom kibanaserver user, the roles that are assigned to that user

---

<div class="post-metadata">

**Author:** ![faxmodem](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/faxmodem/32/22_2.png) [@faxmodem](https://forum.search-guard.com/u/faxmodem)\
**Post date:** [July 5, 2022, 7:13am UTC](https://forum.search-guard.com/t/kibana-not-working-after-installing-flx-beta-2/2333/3 "2022-07-05T07:13:25Z")

</div>

Here’s the ES log:

```auto
Error while mapping auth credentials for trusted_origin[adca255d]
Authentication failed for null from [request=/_searchguard/authinfo, directIpAddress=192.168.245.175, originatingIpAddress=192.168.245.175, clientCertSubject=null]
No cluster-level perm match for User kibana <basic/internal_users_db> UNKNOWN [Action [cluster:admin:searchguard:auth/frontend/config/get]] [RolesChecked [sg_role_kibana4_server]]:
Evaluated Privileges:
_/cluster:admin:searchguard:auth/frontend/config/get: MISSING

Error while mapping auth credentials for trusted_origin[adca255d]
Error while mapping auth credentials for trusted_origin[adca255d]
Error while mapping auth credentials for trusted_origin[adca255d]
Authentication failed for null from [request=/_searchguard/authinfo, directIpAddress=192.168.245.175, originatingIpAddress=192.168.245.175, clientCertSubject=null]
No cluster-level perm match for User kibana <basic/internal_users_db> UNKNOWN [Action [cluster:admin:searchguard:auth/frontend/config/get]] [RolesChecked [sg_role_kibana4_server]]:
Evaluated Privileges:
_/cluster:admin:searchguard:auth/frontend/config/get: MISSING

Authentication failed for null from [request=/_searchguard/authinfo, directIpAddress=192.168.245.175, originatingIpAddress=192.168.245.175, clientCertSubject=null]
Authentication failed for null from [request=/_searchguard/authinfo, directIpAddress=192.168.245.175, originatingIpAddress=192.168.245.175, clientCertSubject=null]
Authentication failed for null from [request=/_searchguard/authinfo, directIpAddress=192.168.245.175, originatingIpAddress=192.168.245.175, clientCertSubject=null]
No cluster-level perm match for User kibana <basic/internal_users_db> UNKNOWN [Action [cluster:admin:searchguard:auth/frontend/config/get]] [RolesChecked [sg_role_kibana4_server]]:
Evaluated Privileges:
_/cluster:admin:searchguard:auth/frontend/config/get: MISSING

```

kibana.yml:

```auto
---
elasticsearch.hosts:
- https://one:9200/
- https://two:9200/
- https://three:9200/
elasticsearch.password: kibana
elasticsearch.requestHeadersWhitelist:
- sgtenant
- Authorization
elasticsearch.requestTimeout: 1200000
elasticsearch.ssl.certificateAuthorities: "/opt/kibana/ssl/ca.pem"
elasticsearch.ssl.verificationMode: full
elasticsearch.username: kibana
kibana.index: ".kibana"
logging.json: true
searchguard.multitenancy.enabled: true
searchguard.multitenancy.tenants.enable_global: true
searchguard.multitenancy.tenants.enable_private: true
server.host: foo.example.com
server.name: foo.example.com
server.port: 443
server.ssl.certificate: "/opt/kibana/ssl/cert.pem"
server.ssl.certificateAuthorities: "/etc/pki/tls/certs/ca-bundle.trust.crt"
server.ssl.enabled: true
server.ssl.key: "/opt/kibana/ssl/key.pem"

```

I just realised the kibana user doesn’t have any special credentials:

```auto
☠ ES _searchguard/api/internalusers/kibana
HTTP/1.1 200 OK
content-length: 151
content-type: application/json; charset=UTF-8

{
    "kibana": {
        "attributes": {},
        "backend_roles": [],
        "description": "Migrated from v6",
        "hidden": false,
        "reserved": false,
        "search_guard_roles": [],
        "static": false
    }
}

```

---

<div class="post-metadata">

**Author:** ![faxmodem](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/faxmodem/32/22_2.png) [@faxmodem](https://forum.search-guard.com/u/faxmodem)\
**Post date:** [July 5, 2022, 7:17am UTC](https://forum.search-guard.com/t/kibana-not-working-after-installing-flx-beta-2/2333/4 "2022-07-05T07:17:26Z")

</div>

EDIT: the `kibana` user _does_ have privileges, they just don’t show in the `internalusers` api call.  
Here they are:

```auto
{
    "sg_role_kibana4_server": {
        "cluster_permissions": [
            "CLUSTER_MONITOR",
            "CLUSTER_COMPOSITE_OPS",
            "cluster:admin/xpack/monitoring*",
            "indices:admin/template*",
            "indices:data/read/scroll*"
        ],
        "description": "Migrated from v6 (all types mapped)",
        "exclude_cluster_permissions": [],
        "exclude_index_permissions": [],
        "hidden": false,
        "index_permissions": [
            {
                "allowed_actions": [
                    "INDICES_ALL"
                ],
                "fls": [],
                "index_patterns": [
                    "?kibana"
                ],
                "masked_fields": []
            },
            {
                "allowed_actions": [
                    "INDICES_ALL"
                ],
                "fls": [],
                "index_patterns": [
                    "?kibana-6"
                ],
                "masked_fields": []
            },
            {
                "allowed_actions": [
                    "INDICES_ALL"
                ],
                "fls": [],
                "index_patterns": [
                    "?kibana_*"
                ],
                "masked_fields": []
            },
            {
                "allowed_actions": [
                    "INDICES_ALL"
                ],
                "fls": [],
                "index_patterns": [
                    "?reporting*"
                ],
                "masked_fields": []
            },
            {
                "allowed_actions": [
                    "INDICES_ALL"
                ],
                "fls": [],
                "index_patterns": [
                    "?monitoring*"
                ],
                "masked_fields": []
            },
            {
                "allowed_actions": [
                    "INDICES_ALL"
                ],
                "fls": [],
                "index_patterns": [
                    "?tasks"
                ],
                "masked_fields": []
            },
            {
                "allowed_actions": [
                    "indices:admin/aliases*"
                ],
                "fls": [],
                "index_patterns": [
                    "*"
                ],
                "masked_fields": []
            }
        ],
        "reserved": true,
        "static": false,
        "tenant_permissions": []
    }
}

```

---

<div class="post-metadata">

**Author:** ![faxmodem](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/faxmodem/32/22_2.png) [@faxmodem](https://forum.search-guard.com/u/faxmodem)\
**Post date:** [July 5, 2022, 7:27am UTC](https://forum.search-guard.com/t/kibana-not-working-after-installing-flx-beta-2/2333/5 "2022-07-05T07:27:40Z")

</div>

I added the `kibana` user to the internal `SGS_KIBANA_SERVER` role, using this command:

```auto
☠ ES PUT _searchguard/api/rolesmapping/SGS_KIBANA_SERVER <<< '{"users":["kibana"]}'
HTTP/1.1 201 Created
content-length: 61
content-type: application/json; charset=UTF-8

{
    "message": "'SGS_KIBANA_SERVER' created.",
    "status": "CREATED"
}

☠ ES _searchguard/api/rolesmapping/SGS_KIBANA_SERVER
HTTP/1.1 200 OK
content-length: 42
content-type: application/json; charset=UTF-8

{
    "SGS_KIBANA_SERVER": {
        "users": [
            "kibana"
        ]
    }
}

```

But I still get the 50x error

---

<div class="post-metadata">

**Author:** ![nils](https://avatars.discourse-cdn.com/v4/letter/n/d6d6ee/32.png) [@nils](https://forum.search-guard.com/u/nils)\
**Post date:** [July 5, 2022, 8:16am UTC](https://forum.search-guard.com/t/kibana-not-working-after-installing-flx-beta-2/2333/6 "2022-07-05T08:16:55Z")

</div>

Thank you. Can you please try to assign the role `SGS_KIBANA_SERVER` to `search_guard_roles` of the user `kibana`? Only then, the user will have sufficient privileges.

---

<div class="post-metadata">

**Author:** ![faxmodem](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/faxmodem/32/22_2.png) [@faxmodem](https://forum.search-guard.com/u/faxmodem)\
**Post date:** [July 5, 2022, 8:20am UTC](https://forum.search-guard.com/t/kibana-not-working-after-installing-flx-beta-2/2333/7 "2022-07-05T08:20:31Z")

</div>

```auto
☠ ES _searchguard/api/internalusers/kibana
HTTP/1.1 200 OK
content-length: 151
content-type: application/json; charset=UTF-8

{
    "kibana": {
        "attributes": {},
        "backend_roles": [],
        "description": "Migrated from v6",
        "hidden": false,
        "reserved": false,
        "search_guard_roles": [],
        "static": false
    }
}

☠ ES PUT _searchguard/api/internalusers/kibana <<< '{"search_guard_roles":["SGS_KIBANA_SERVER"]}'
HTTP/1.1 200 OK
content-length: 45
content-type: application/json; charset=UTF-8

{
    "message": "'kibana' updated.",
    "status": "OK"
}

☠ ES _searchguard/api/internalusers/kibana
HTTP/1.1 200 OK
content-length: 55
content-type: application/json; charset=UTF-8

{
    "kibana": {
        "search_guard_roles": [
            "SGS_KIBANA_SERVER"
        ]
    }
}

```

---

<div class="post-metadata">

**Author:** ![faxmodem](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/faxmodem/32/22_2.png) [@faxmodem](https://forum.search-guard.com/u/faxmodem)\
**Post date:** [July 5, 2022, 8:20am UTC](https://forum.search-guard.com/t/kibana-not-working-after-installing-flx-beta-2/2333/8 "2022-07-05T08:20:59Z")

</div>

This didn’t help

---

<div class="post-metadata">

**Author:** ![nils](https://avatars.discourse-cdn.com/v4/letter/n/d6d6ee/32.png) [@nils](https://forum.search-guard.com/u/nils)\
**Post date:** [July 5, 2022, 8:27am UTC](https://forum.search-guard.com/t/kibana-not-working-after-installing-flx-beta-2/2333/9 "2022-07-05T08:27:10Z")

</div>

Can you please try to restart Kibana?

If that does not help, can you please post the content of the ES log again, like you already did above?

---

<div class="post-metadata">

**Author:** ![faxmodem](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/faxmodem/32/22_2.png) [@faxmodem](https://forum.search-guard.com/u/faxmodem)\
**Post date:** [July 5, 2022, 8:42am UTC](https://forum.search-guard.com/t/kibana-not-working-after-installing-flx-beta-2/2333/10 "2022-07-05T08:42:01Z")

</div>

I restarted kibana, and the ES log just says:

```auto
Error while mapping auth credentials for trusted_origin[adca255d]
Authentication failed for null from [request=/_searchguard/authinfo, directIpAddress=192.168.245.175, originatingIpAddress=192.168.245.175, clientCertSubject=null]

```

---

<div class="post-metadata">

**Author:** ![nils](https://avatars.discourse-cdn.com/v4/letter/n/d6d6ee/32.png) [@nils](https://forum.search-guard.com/u/nils)\
**Post date:** [July 5, 2022, 8:45am UTC](https://forum.search-guard.com/t/kibana-not-working-after-installing-flx-beta-2/2333/11 "2022-07-05T08:45:12Z")

</div>

Can you please also post

- The Kibana logs
- sg\_authc.yml?

Sorry for the inconvenience!

---

<div class="post-metadata">

**Author:** ![faxmodem](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/faxmodem/32/22_2.png) [@faxmodem](https://forum.search-guard.com/u/faxmodem)\
**Post date:** [July 5, 2022, 9:25am UTC](https://forum.search-guard.com/t/kibana-not-working-after-installing-flx-beta-2/2333/12 "2022-07-05T09:25:27Z")

</div>

```auto
[ResponseError]: Response Error
Multitenancy: Could not get authinfo AuthenticationError: Response Error
GET /api/v1/systeminfo 200 21ms - 9.0B
[ResponseError]: Response Error
Multitenancy: Could not get authinfo AuthenticationError: Response Error
[status_exception]: No such frontend config: default
{ ResponseError: status_exception
    at IncomingMessage.response.on (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:272:25)
    at IncomingMessage.emit (events.js:203:15)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  name: 'ResponseError',
  meta:
   { body: { error: [Object], status: 404 },
     statusCode: 404,
     headers:
      { 'content-type': 'application/json; charset=UTF-8',
        'content-length': '181' },
     meta:
      { context: null,
        request: [Object],
        name: 'elasticsearch-js',
        connection: [Object],
        attempts: 0,
        aborted: false } } }
Internal Server Error
GET /api/v1/auth/config 500 19ms - 9.0B

```

sg\_authc.yaml:

```auto
---
auth_domains:
- type: "trusted_origin"
  user_mapping:
    user_name:
      from: "$.request.headers[\"x-authenticated-user\"]"
    roles:
      from_comma_separated_string: "$.request.headers[\"x-authenticated-group\"]"
- type: "clientcert"
- type: "basic/internal_users_db"
- type: "kerberos"
network:
  trusted_proxies_regex: "127.0.0.1"
  http:
    remote_ip_header: "X-Forwarded-For"

```

---

<div class="post-metadata">

**Author:** ![nils](https://avatars.discourse-cdn.com/v4/letter/n/d6d6ee/32.png) [@nils](https://forum.search-guard.com/u/nils)\
**Post date:** [July 5, 2022, 9:44am UTC](https://forum.search-guard.com/t/kibana-not-working-after-installing-flx-beta-2/2333/13 "2022-07-05T09:44:10Z")

</div>

> [@faxmodem](#):
>
> `[status_exception]: No such frontend config: default`

This indicates that the `sg_frontend_authc.yml` file is missing. Can you please double check that it is present? It should have been created by the `sgctl migrate-config` process.

You can upload it using `sgctl.sh update-config path/to/sg\_frontend\_authc.yml

---

<div class="post-metadata">

**Author:** ![faxmodem](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/faxmodem/32/22_2.png) [@faxmodem](https://forum.search-guard.com/u/faxmodem)\
**Post date:** [July 5, 2022, 10:53am UTC](https://forum.search-guard.com/t/kibana-not-working-after-installing-flx-beta-2/2333/14 "2022-07-05T10:53:42Z")

</div>

I had done this, when following the documentation.  
I just pushed it again, to no avail:

```auto
$ ../sgctl-0.2.5/sgctl.sh update-config sg_frontend_authc.yml
Successfully connected to elasticsearch as user CN=sgadmin,OU=client,O=client,L=Test,C=FR
Configuration has been updated

```

---

<div class="post-metadata">

**Author:** ![faxmodem](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/faxmodem/32/22_2.png) [@faxmodem](https://forum.search-guard.com/u/faxmodem)\
**Post date:** [July 5, 2022, 10:55am UTC](https://forum.search-guard.com/t/kibana-not-working-after-installing-flx-beta-2/2333/15 "2022-07-05T10:55:32Z")

</div>

Here’s its contents:

```auto
---
auth_domains:
- type: "oidc"
  idp.openid_configuration_url: "..."
  client_id: "..."
  client_secret: "..."
  user_mapping.subject: "$['preferred_username']"

```

---

<div class="post-metadata">

**Author:** ![nils](https://avatars.discourse-cdn.com/v4/letter/n/d6d6ee/32.png) [@nils](https://forum.search-guard.com/u/nils)\
**Post date:** [July 5, 2022, 11:23am UTC](https://forum.search-guard.com/t/kibana-not-working-after-installing-flx-beta-2/2333/16 "2022-07-05T11:23:47Z")

</div>

Oh, was this created by `sgctl migrate-config`? Then, there are apparently few bugs present.

Please try the following template:

```auto
default:
  auth_domains:
  - type: "oidc"
    oidc.idp.openid_configuration_url: "..."
    oidc.client_id: "..."
    oidc.client_secret: "..."
    user_mapping.subject: "$['preferred_username']"

```

---

<div class="post-metadata">

**Author:** ![faxmodem](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/faxmodem/32/22_2.png) [@faxmodem](https://forum.search-guard.com/u/faxmodem)\
**Post date:** [July 5, 2022, 11:28am UTC](https://forum.search-guard.com/t/kibana-not-working-after-installing-flx-beta-2/2333/17 "2022-07-05T11:28:14Z")

</div>

yes, `sgctl created that file. I added the `oidc` prefix to the 3 keys, restarted kibana, still same error.

---

<div class="post-metadata">

**Author:** ![nils](https://avatars.discourse-cdn.com/v4/letter/n/d6d6ee/32.png) [@nils](https://forum.search-guard.com/u/nils)\
**Post date:** [July 5, 2022, 11:36am UTC](https://forum.search-guard.com/t/kibana-not-working-after-installing-flx-beta-2/2333/18 "2022-07-05T11:36:30Z")

</div>

Did you also add the top-level `default:`?

---

<div class="post-metadata">

**Author:** ![faxmodem](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/faxmodem/32/22_2.png) [@faxmodem](https://forum.search-guard.com/u/faxmodem)\
**Post date:** [July 5, 2022, 11:36am UTC](https://forum.search-guard.com/t/kibana-not-working-after-installing-flx-beta-2/2333/19 "2022-07-05T11:36:35Z")

</div>

I also saw the following message:

```auto
[illegal_argument_exception]: request [/_license] contains unrecognized parameter: [accept_enterprise]

```

---

<div class="post-metadata">

**Author:** ![faxmodem](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/faxmodem/32/22_2.png) [@faxmodem](https://forum.search-guard.com/u/faxmodem)\
**Post date:** [July 5, 2022, 11:38am UTC](https://forum.search-guard.com/t/kibana-not-working-after-installing-flx-beta-2/2333/20 "2022-07-05T11:38:44Z")

</div>

```auto
Invalid config files:

sg_frontend_authc.yml:
  default.auth_domains.0.user_mapping.subject:
        Unsupported attribute

```

[Next page](https://forum.search-guard.com/t/kibana-not-working-after-installing-flx-beta-2/2333.md?page=2)