Kibana not working after upgrading to ELK 8.14.3

trauta · August 30, 2024, 12:27pm

Hi,

Yesterday we’ve upgraded our production Elastic cluster to version 8.14.3.

Shortly after the upgrade, users reported that searches and visualizations in Kibana are resulting in a error when they take longer than a few seconds.

Looking in our Elasticsearch server logs we’ve discovered following missing privileges messages:

### No cluster privileges for cluster:monitor/async_search/status (org.elasticsearch.xpack.core.async.GetAsyncStatusRequest)
User: User <USER>> [backend_roles=[...] requestedTenant=<TENANT>]
Roles: [SGS_KIBANA_USER_NO_GLOBAL_TENANT, ...]
Status: INSUFFICIENT
Evaluated Privileges:
_/cluster:monitor/async_search/status: MISSING

With a quick look into the Kibana release notes for 8.14.0 we’ve discovered following change:

Management

Kibana now uses Elasticsearch’s _async_search/status/{id} endpoint (instead of _async_search/{id}) when polling on search requests to improve performance.(#178921).

The change is documented in the linked GitHub pull request:

github.com/elastic/kibana

[data.search] Poll on async status

elastic:main ← lukasolson:search/async-status

opened 10:52PM - 18 Mar 24 UTC

lukasolson

+142 -44

## Summary Resolves https://github.com/elastic/kibana/issues/174218. This …PR updates the default (`ese`) search strategy to poll on the [get async status](https://www.elastic.co/guide/en/elasticsearch/reference/8.12/async-search.html#get-async-search-status) endpoint rather than the [get async search](https://www.elastic.co/guide/en/elasticsearch/reference/8.12/async-search.html#get-async-search) endpoint. This takes advantage of optimizations in ES to increase performance of async search. ### Release notes Kibana now uses Elasticsearch's `_async_search/status/{id}` endpoint (instead of `_async_search/{id}`) when polling on search requests to improve performance. ### Checklist - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed ### Risk Matrix | Risk | Probability | Severity | Mitigation/Notes | |---------------------------|-------------|----------|-------------------------| | Multiple Spaces—unexpected behavior in non-default Kibana Space. | Low | High | Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces. | | Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks. | High | Low | Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure. | | Code should gracefully handle cases when feature X or plugin Y are disabled. | Medium | High | Unit tests will verify that any feature flag or plugin combination still results in our service operational. | | [See more potential risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |

We’ve manually added the new cluster:monitor/async_search/status privilege to our SG role definitions, however I think this should be added into default SGS_KIBANA_USER… roles.

Regards,
Alex

mantas · August 30, 2024, 12:53pm

Hi @trauta,

Thanks for sharing your findings, I’ll have a look at what can be done and get back to you. Am I correct to assume that you have multitenancy enabled?

thanks,
mj

trauta · August 30, 2024, 1:15pm

Yes, the multitenancy feature is enabled and we’re using an academic (enterprise) license.

mantas · September 3, 2024, 1:22pm

@trauta, Could you please also confirm if you are using Global Tenant, at all, in your cluster, to help me with my investigation?

Thanks,
Mantas

trauta · September 5, 2024, 6:31am

The global tenant is not used by any regular users, it is only available for our ‘admin’ account.

Topic		Replies	Views
Kibana Monitoring page not working after 23.0 to 23.1 upgrade Search Guard	9	736	October 30, 2018
Problems with Kibana 5 and SG 5.0.1-8 Search Guard	4	456	December 25, 2016
Not able to see monitoring in Kibana after installing 7.7.0/SG41 Search Guard	3	1177	July 20, 2020
Kibana is not logging in with no errors Search Guard	2	487	March 4, 2019
Elasticsearch 6.8.6, green cluster, yellow Kibana Search Guard	2	874	September 7, 2020

Kibana not working after upgrading to ELK 8.14.3

Related Topics