Elasticsearch permission problems when upgrading from 8.8.2 to 8.11.4

silentfilm · January 31, 2024, 9:34pm

I am upgrading my system from ES/Kibana 8.8.2 to ES/Kibana 8.11.4 and Search Guard 1.5.0-es-8.11.4. Elasticsearch came up fine, but I am getting the following error in Elasticsearch when Kibana is starting. Due to the error, Kibana cannot successfully start.

2024-01-31 21:24:49.092Z INFO  [elasticsearch[esnode-aln-nbadev4][transport_worker][T#49]] com.floragunn.searchguard.authz.PrivilegesEvaluator - ### No cluster privileges for cluster:admin/ingest/pipeline/put (org.elasticsearch.action.ingest.PutPipelineRequest)
User: User kibanaserver <basic/internal_users_db>
Roles: [SGS_UNLIMITED, SGS_KIBANA_SERVER, sg_index_maintenance, SGS_OWN_INDEX]
Status: INSUFFICIENT
Evaluated Privileges:
_/cluster:admin/ingest/pipeline/put: MISSING

If you have SGS_UNLIMITED permission, shouldn’t you have all cluster permissions?

system · February 21, 2024, 9:35pm

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

mantas · February 22, 2024, 1:10pm

Hi @silentfilm,

Is this issue still applicable, or have you found a solution?
Have you tried adding cluster:admin/ingest/pipeline/* to your kibanaserver user manually?

Thanks,
Mantas

silentfilm · February 26, 2024, 2:50pm

Since I could not get this to work, I’ve moved on to other issues. I might have time to try this in a couple of weeks.

silentfilm · March 11, 2024, 8:12pm

I am trying the upgrade again with Elasticsearch 8.12.1 and Searchguard 1.5.0-es-8.12.1. We actually have never used the SearchGuard Kibana plugin, only the Elasticsearch plugin, so I can’t manually enable that permission. After I restarted my haproxy, Kibana started up.

I tried adding this to sg_roles_mapping.yml:

sg_ingest:
  reserved: false
  hidden: false
  users:
    - "kibanaserver"
  description: "Allows pipeline ingest"

And I created a new role in sg_roles.yml:

sg_ingest:
  cluster_permissions:
    - "cluster:admin/ingest/pipeline/put"
  index_permissions: [ ]
  tenant_permissions: [ ]
  exclude_cluster_permissions: [ ]
  exclude_index_permissions: [ ]

I also added this role in elasticsearch.yml:

searchguard.restapi.roles_enabled:
  - "SGS_ALL_ACCESS"
  - "SGS_LOGSTASH"
  - "SGS_KIBANA_SERVER"
  - "SGS_KIBANA_USER"
  - "SGS_READALL"
  - "SGS_OWN_INDEX"
  - "SGS_MANAGE_SNAPSHOTS"
  - "SGS_CLUSTER_ALL"
  - "sg_index_maintenance"
  - "sg_alerting"
  - "sg_metrics"
  - "sg_data_integrity"
  - "sg_ingest"

But I still get these errors in Elasticsearch when Kibana starts up:

2024-03-11 19:46:29.632Z INFO  [elasticsearch[esnode-aln-nbadev4][transport_worker][T#17]] com.floragunn.searchguard.authz.PrivilegesEvaluator - ### No cluster privileges for cluster:admin/ingest/pipeline/put (org.elasticsearch.action.ingest.PutPipelineRequest)
User: User kibanaserver <basic/internal_users_db>
Roles: [SGS_KIBANA_SERVER, sg_index_maintenance, SGS_CLUSTER_ALL, sg_ingest, SGS_OWN_INDEX]
Status: INSUFFICIENT
Evaluated Privileges:
_/cluster:admin/ingest/pipeline/put: MISSING

2024-03-11 19:46:29.670Z INFO  [elasticsearch[esnode-aln-nbadev4][transport_worker][T#14]] com.floragunn.searchguard.authz.PrivilegesEvaluator - ### No cluster privileges for cluster:admin/ingest/pipeline/put (org.elasticsearch.action.ingest.PutPipelineRequest)
User: User kibanaserver <basic/internal_users_db>
Roles: [SGS_KIBANA_SERVER, sg_index_maintenance, SGS_CLUSTER_ALL, sg_ingest, SGS_OWN_INDEX]
Status: INSUFFICIENT
Evaluated Privileges:
_/cluster:admin/ingest/pipeline/put: MISSING

And Kibana has these errors:

[2024-03-11T19:46:29.630+00:00][INFO ][plugins.observability] Installing SLO ingest pipeline [.slo-observability.sli.pipeline-v3]
[2024-03-11T19:46:29.633+00:00][WARN ][plugins.securitySolution] Unable to verify endpoint policies in line with license change: failed to fetch package policies: Unauthorized: authentication_exception
[2024-03-11T19:46:29.634+00:00][WARN ][plugins.fleet] Unable to verify agent policies in line with license change: failed to fetch agent policies: Unauthorized: authentication_exception
[2024-03-11T19:46:29.648+00:00][ERROR][plugins.observability] Error installing resources shared for SLO: security_exception
        Root causes:
                security_exception: Insufficient permissions
[2024-03-11T19:46:29.648+00:00][ERROR][plugins.observability] Failed to install SLO common resources
[2024-03-11T19:46:29.649+00:00][INFO ][plugins.alerting] Creating concrete write index - .internal.alerts-observability.uptime.alerts-default-000001
[2024-03-11T19:46:29.650+00:00][INFO ][plugins.alerting] Creating concrete write index - .internal.alerts-observability.slo.alerts-default-000001
[2024-03-11T19:46:29.656+00:00][INFO ][plugins.alerting] Creating concrete write index - .internal.alerts-observability.threshold.alerts-default-000001
[2024-03-11T19:46:29.657+00:00][INFO ][plugins.alerting] Creating concrete write index - .internal.alerts-observability.apm.alerts-default-000001
[2024-03-11T19:46:29.658+00:00][INFO ][plugins.alerting] Creating concrete write index - .internal.alerts-ml.anomaly-detection.alerts-default-000001
[2024-03-11T19:46:29.662+00:00][INFO ][plugins.alerting] Creating concrete write index - .internal.alerts-stack.alerts-default-000001
[2024-03-11T19:46:29.686+00:00][ERROR][plugins.observabilityAIAssistant.service] Failed to initialize service: security_exception
        Root causes:
                security_exception: Insufficient permissions
[2024-03-11T19:46:29.686+00:00][ERROR][plugins.observabilityAIAssistant.service] Could not index 7 entries because of an initialisation error
[2024-03-11T19:46:29.686+00:00][ERROR][plugins.observabilityAIAssistant.service] ResponseError: security_exception
        Root causes:
                security_exception: Insufficient permissions
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:479:27)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
    at Ingest.putPipeline (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/ingest.js:139:16)
    at ObservabilityAIAssistantService.<anonymous> (/usr/share/kibana/node_modules/@kbn/observability-ai-assistant-plugin/server/service/index.js:126:9)

It doesn’t look like there is any way to turn off the AIAssistant. I don’t know if this is related to the ingest error.

silentfilm · March 11, 2024, 10:15pm

I tried installing the SearchGuard plugin, but I get

[2024-03-11T21:48:47.561+00:00][ERROR][plugins.searchguard.searchguard] getRestApiInfo: ResponseError: {"error":"no handler found for uri [/_searchguard/api/permissionsinfo] and method [GET]"}
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:479:27)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
    at SearchGuardConfigurationBackend._client (/usr/share/kibana/plugins/searchguard/server/applications/searchguard/configuration/backend/searchguard_configuration_backend.js:28:20)
    at SearchGuardConfigurationBackend.restapiinfo (/usr/share/kibana/plugins/searchguard/server/applications/searchguard/configuration/backend/searchguard_configuration_backend.js:37:14)
    at /usr/share/kibana/plugins/searchguard/server/applications/searchguard/configuration/routes/routes.js:110:20
    at Router.handle (/usr/share/kibana/node_modules/@kbn/core-http-router-server-internal/src/router.js:171:30)
    at handler (/usr/share/kibana/node_modules/@kbn/core-http-router-server-internal/src/router.js:113:50)
    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)
    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)
[2024-03-11T21:48:47.678+00:00][ERROR][plugins.searchguard.authTokens] getAuthTokens: ResponseError: {"error":"no handler found for uri [/_searchguard/authtoken/_search] and method [POST]"}
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:479:27)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
[2024-03-11T21:48:57.405+00:00][WARN ][savedobjects-service] The get saved object API /api/saved_objects/_find is deprecated.
[2024-03-11T21:49:05.808+00:00][ERROR][plugins.searchguard.searchguard] getRestApiInfo: ResponseError: {"error":"no handler found for uri [/_searchguard/api/permissionsinfo] and method [GET]"}
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:479:27)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
    at SearchGuardConfigurationBackend._client (/usr/share/kibana/plugins/searchguard/server/applications/searchguard/configuration/backend/searchguard_configuration_backend.js:28:20)
    at SearchGuardConfigurationBackend.restapiinfo (/usr/share/kibana/plugins/searchguard/server/applications/searchguard/configuration/backend/searchguard_configuration_backend.js:37:14)
    at /usr/share/kibana/plugins/searchguard/server/applications/searchguard/configuration/routes/routes.js:110:20
    at Router.handle (/usr/share/kibana/node_modules/@kbn/core-http-router-server-internal/src/router.js:171:30)
    at handler (/usr/share/kibana/node_modules/@kbn/core-http-router-server-internal/src/router.js:113:50)
    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)
    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)
[2024-03-11T21:49:05.922+00:00][ERROR][plugins.searchguard.authTokens] getAuthTokens: ResponseError: {"error":"no handler found for uri [/_searchguard/authtoken/_search] and method [POST]"}
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:479:27)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
[2024-03-11T21:49:06.284+00:00][WARN ][plugins.dataViews.dataView.storage] Invalid response. [item.attributes.fields.987.name]: expected value of type [string] but got [undefined]
[2024-03-11T21:49:14.534+00:00][WARN ][savedobjects-service] The get saved object API /api/saved_objects/_find is deprecated.

mantas · March 21, 2024, 11:14am

Hi @silentfilm,

Have you uploaded your configuration changes with sgctl?

Please note: “sgctl will replace the current configuration in your Elasticsearch cluster with the one you provide. We recommended to backup the configuration first before applying changes.”

Best,
Mantas

mantas · March 21, 2024, 11:29am

Would you mind elaborating more on this:
Versions of Elasticsearch (8.11.4?) and the SearchGuard(1.5.0?)?
Process or Docs followed to install it?

Thanks,
Mantas

silentfilm · March 21, 2024, 1:53pm

I’m using Elasticsearch 8.12.1 now and Search Guard 1.5.0-es-8.12.1.

My Dockerfile has this (the search-guard file has been renamed so I don’t have to change the code every time the version number changes).

COPY search-guard-flx.zip ${ES_HOME}
RUN yes y | bin/elasticsearch-plugin install -b file://${ES_HOME}/search-guard-flx.zip && \
    bin/elasticsearch-plugin list && \
    rm -f ${ES_HOME}/search-guard*.zip

The configuration is updated in a shell file when the Elasticsearch docker container starts up:

${ES_HOME}/sgctl-1.1.0/sgctl.sh connect -v ${HOSTNAME} --ca-cert ${ES_CERT_PATH}/${ES_ADMIN_CA_FILENAME} \
    --cert ${ES_CERT_PATH}/${ES_ADMIN_CERT_FILENAME} --key ${ES_CERT_PATH}/${ES_ADMIN_KEY_FILENAME} \
    --key-pass ${1} "${@:2}"

${ES_HOME}/sgctl-1.1.0/sgctl.sh update-config -v ${ES_CONFIG}/sg

mantas · April 10, 2024, 3:01pm

Hi @silentfilm,

Is the issue still applicable?
if yes, could you run the below and share the output?
curl --insecure -u admin:<admin-password> -XGET https://<ES-HOST>:9200/_searchguard/api/roles/sg_inges?pretty

Thanks,
Mantas

silentfilm · April 10, 2024, 3:57pm

The issue is not happening anymore. I can’t remember exactly how I fixed it, but I think it is because I created a specific role for ingestion and assigned it to the kibanaserver.

sg_ingest:
  cluster_permissions:
    - "cluster:admin/ingest/pipeline/put"
  index_permissions: [ ]
  tenant_permissions: [ ]
  exclude_cluster_permissions: [ ]
  exclude_index_permissions: [ ]

It seems like assigning all cluster privileges to the kibanaserver would have worked, but it didn’t.

Domsalvator23 · June 21, 2024, 7:14am

Glad to hear the issue is resolved! Creating a specific role for ingestion and assigning it to the kibanaserver likely fixed it. Here is the role I used;

sg_ingest:
cluster_permissions:
- “cluster:admin/ingest/pipeline/put”
index_permissions:
tenant_permissions:
exclude_cluster_permissions:
exclude_index_permissions:

Hope this helps for facing similar issue!

silentfilm · June 21, 2024, 2:20pm

I had to create exactly the same role when I upgraded from Elasticsearch 8.8.2 to 8.12.1 and it fixed the issue.

Topic		Replies	Views
Upgrade 6.7 - 7.4 failure Search Guard	5	1162	November 3, 2019
Search guard kibana not working. Search Guard	4	392	January 25, 2018
Permissions issue for tenant X Search Guard	12	524	April 25, 2022
Permissions problem with tenants in ELK 6.8.6 Search Guard	3	610	June 3, 2020
Permissions error retrieving configuration, cannot upgrade, HELP! Search Guard	6	1041	November 7, 2019

Elasticsearch permission problems when upgrading from 8.8.2 to 8.11.4

Related topics