# Elasticsearch permission problems when upgrading from 8.8.2 to 8.11.4

**URL:** <https://forum.search-guard.com/t/elasticsearch-permission-problems-when-upgrading-from-8-8-2-to-8-11-4/2490>\
**Category:** Search Guard\
**Created:** [January 31, 2024, 9:34pm UTC](https://forum.search-guard.com/t/elasticsearch-permission-problems-when-upgrading-from-8-8-2-to-8-11-4/2490 "2024-01-31T21:34:43Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![silentfilm](https://avatars.discourse-cdn.com/v4/letter/s/dbc845/32.png) [@silentfilm](https://forum.search-guard.com/u/silentfilm)\
**Post date:** [January 31, 2024, 9:34pm UTC](https://forum.search-guard.com/t/elasticsearch-permission-problems-when-upgrading-from-8-8-2-to-8-11-4/2490/1 "2024-01-31T21:34:43Z")

</div>

I am upgrading my system from ES/Kibana 8.8.2 to ES/Kibana 8.11.4 and Search Guard 1.5.0-es-8.11.4. Elasticsearch came up fine, but I am getting the following error in Elasticsearch when Kibana is starting. Due to the error, Kibana cannot successfully start.

```auto
2024-01-31 21:24:49.092Z INFO [elasticsearch[esnode-aln-nbadev4][transport_worker][T#49]] com.floragunn.searchguard.authz.PrivilegesEvaluator - ### No cluster privileges for cluster:admin/ingest/pipeline/put (org.elasticsearch.action.ingest.PutPipelineRequest)
User: User kibanaserver <basic/internal_users_db>
Roles: [SGS_UNLIMITED, SGS_KIBANA_SERVER, sg_index_maintenance, SGS_OWN_INDEX]
Status: INSUFFICIENT
Evaluated Privileges:
_/cluster:admin/ingest/pipeline/put: MISSING

```

If you have SGS\_UNLIMITED permission, shouldn’t you have all cluster permissions?

---

<div class="post-metadata">

**Author:** ![system](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/system/32/1870_2.png) [@system](https://forum.search-guard.com/u/system)\
**Post date:** [February 21, 2024, 9:35pm UTC](https://forum.search-guard.com/t/elasticsearch-permission-problems-when-upgrading-from-8-8-2-to-8-11-4/2490/2 "2024-02-21T21:35:08Z")

</div>

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

---

<div class="post-metadata">

**Author:** ![mantas](https://avatars.discourse-cdn.com/v4/letter/m/7bcc69/32.png) [@mantas](https://forum.search-guard.com/u/mantas)\
**Post date:** [February 22, 2024, 1:06pm UTC](https://forum.search-guard.com/t/elasticsearch-permission-problems-when-upgrading-from-8-8-2-to-8-11-4/2490/3 "2024-02-22T13:06:37Z")

</div>



---

<div class="post-metadata">

**Author:** ![mantas](https://avatars.discourse-cdn.com/v4/letter/m/7bcc69/32.png) [@mantas](https://forum.search-guard.com/u/mantas)\
**Post date:** [February 22, 2024, 1:10pm UTC](https://forum.search-guard.com/t/elasticsearch-permission-problems-when-upgrading-from-8-8-2-to-8-11-4/2490/4 "2024-02-22T13:10:13Z")

</div>

Hi @silentfilm,

Is this issue still applicable, or have you found a solution?  
Have you tried adding `cluster:admin/ingest/pipeline/*` to your `kibanaserver` user manually?

Thanks,  
Mantas

---

<div class="post-metadata">

**Author:** ![silentfilm](https://avatars.discourse-cdn.com/v4/letter/s/dbc845/32.png) [@silentfilm](https://forum.search-guard.com/u/silentfilm)\
**Post date:** [February 26, 2024, 2:50pm UTC](https://forum.search-guard.com/t/elasticsearch-permission-problems-when-upgrading-from-8-8-2-to-8-11-4/2490/5 "2024-02-26T14:50:20Z")

</div>

Since I could not get this to work, I’ve moved on to other issues. I might have time to try this in a couple of weeks.

---

<div class="post-metadata">

**Author:** ![silentfilm](https://avatars.discourse-cdn.com/v4/letter/s/dbc845/32.png) [@silentfilm](https://forum.search-guard.com/u/silentfilm)\
**Post date:** [March 11, 2024, 8:12pm UTC](https://forum.search-guard.com/t/elasticsearch-permission-problems-when-upgrading-from-8-8-2-to-8-11-4/2490/6 "2024-03-11T20:12:20Z")

</div>

I am trying the upgrade again with Elasticsearch 8.12.1 and Searchguard 1.5.0-es-8.12.1. We actually have never used the SearchGuard Kibana plugin, only the Elasticsearch plugin, so I can’t manually enable that permission. After I restarted my haproxy, Kibana started up.

I tried adding this to sg\_roles\_mapping.yml:

```auto
sg_ingest:
  reserved: false
  hidden: false
  users:
    - "kibanaserver"
  description: "Allows pipeline ingest"

```

And I created a new role in sg\_roles.yml:

```auto
sg_ingest:
  cluster_permissions:
    - "cluster:admin/ingest/pipeline/put"
  index_permissions: []
  tenant_permissions: []
  exclude_cluster_permissions: []
  exclude_index_permissions: []

```

I also added this role in elasticsearch.yml:

```auto
searchguard.restapi.roles_enabled:
  - "SGS_ALL_ACCESS"
  - "SGS_LOGSTASH"
  - "SGS_KIBANA_SERVER"
  - "SGS_KIBANA_USER"
  - "SGS_READALL"
  - "SGS_OWN_INDEX"
  - "SGS_MANAGE_SNAPSHOTS"
  - "SGS_CLUSTER_ALL"
  - "sg_index_maintenance"
  - "sg_alerting"
  - "sg_metrics"
  - "sg_data_integrity"
  - "sg_ingest"

```

But I still get these errors in Elasticsearch when Kibana starts up:

```auto
2024-03-11 19:46:29.632Z INFO [elasticsearch[esnode-aln-nbadev4][transport_worker][T#17]] com.floragunn.searchguard.authz.PrivilegesEvaluator - ### No cluster privileges for cluster:admin/ingest/pipeline/put (org.elasticsearch.action.ingest.PutPipelineRequest)
User: User kibanaserver <basic/internal_users_db>
Roles: [SGS_KIBANA_SERVER, sg_index_maintenance, SGS_CLUSTER_ALL, sg_ingest, SGS_OWN_INDEX]
Status: INSUFFICIENT
Evaluated Privileges:
_/cluster:admin/ingest/pipeline/put: MISSING

2024-03-11 19:46:29.670Z INFO [elasticsearch[esnode-aln-nbadev4][transport_worker][T#14]] com.floragunn.searchguard.authz.PrivilegesEvaluator - ### No cluster privileges for cluster:admin/ingest/pipeline/put (org.elasticsearch.action.ingest.PutPipelineRequest)
User: User kibanaserver <basic/internal_users_db>
Roles: [SGS_KIBANA_SERVER, sg_index_maintenance, SGS_CLUSTER_ALL, sg_ingest, SGS_OWN_INDEX]
Status: INSUFFICIENT
Evaluated Privileges:
_/cluster:admin/ingest/pipeline/put: MISSING

```

And Kibana has these errors:

```auto
[2024-03-11T19:46:29.630+00:00][INFO][plugins.observability] Installing SLO ingest pipeline [.slo-observability.sli.pipeline-v3]
[2024-03-11T19:46:29.633+00:00][WARN][plugins.securitySolution] Unable to verify endpoint policies in line with license change: failed to fetch package policies: Unauthorized: authentication_exception
[2024-03-11T19:46:29.634+00:00][WARN][plugins.fleet] Unable to verify agent policies in line with license change: failed to fetch agent policies: Unauthorized: authentication_exception
[2024-03-11T19:46:29.648+00:00][ERROR][plugins.observability] Error installing resources shared for SLO: security_exception
        Root causes:
                security_exception: Insufficient permissions
[2024-03-11T19:46:29.648+00:00][ERROR][plugins.observability] Failed to install SLO common resources
[2024-03-11T19:46:29.649+00:00][INFO][plugins.alerting] Creating concrete write index - .internal.alerts-observability.uptime.alerts-default-000001
[2024-03-11T19:46:29.650+00:00][INFO][plugins.alerting] Creating concrete write index - .internal.alerts-observability.slo.alerts-default-000001
[2024-03-11T19:46:29.656+00:00][INFO][plugins.alerting] Creating concrete write index - .internal.alerts-observability.threshold.alerts-default-000001
[2024-03-11T19:46:29.657+00:00][INFO][plugins.alerting] Creating concrete write index - .internal.alerts-observability.apm.alerts-default-000001
[2024-03-11T19:46:29.658+00:00][INFO][plugins.alerting] Creating concrete write index - .internal.alerts-ml.anomaly-detection.alerts-default-000001
[2024-03-11T19:46:29.662+00:00][INFO][plugins.alerting] Creating concrete write index - .internal.alerts-stack.alerts-default-000001
[2024-03-11T19:46:29.686+00:00][ERROR][plugins.observabilityAIAssistant.service] Failed to initialize service: security_exception
        Root causes:
                security_exception: Insufficient permissions
[2024-03-11T19:46:29.686+00:00][ERROR][plugins.observabilityAIAssistant.service] Could not index 7 entries because of an initialisation error
[2024-03-11T19:46:29.686+00:00][ERROR][plugins.observabilityAIAssistant.service] ResponseError: security_exception
        Root causes:
                security_exception: Insufficient permissions
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:479:27)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
    at Ingest.putPipeline (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/ingest.js:139:16)
    at ObservabilityAIAssistantService.<anonymous> (/usr/share/kibana/node_modules/@kbn/observability-ai-assistant-plugin/server/service/index.js:126:9)

```

It doesn’t look like there is any way to turn off the AIAssistant. I don’t know if this is related to the ingest error.

---

<div class="post-metadata">

**Author:** ![silentfilm](https://avatars.discourse-cdn.com/v4/letter/s/dbc845/32.png) [@silentfilm](https://forum.search-guard.com/u/silentfilm)\
**Post date:** [March 11, 2024, 10:15pm UTC](https://forum.search-guard.com/t/elasticsearch-permission-problems-when-upgrading-from-8-8-2-to-8-11-4/2490/7 "2024-03-11T22:15:32Z")

</div>

I tried installing the SearchGuard plugin, but I get

```auto
[2024-03-11T21:48:47.561+00:00][ERROR][plugins.searchguard.searchguard] getRestApiInfo: ResponseError: {"error":"no handler found for uri [/_searchguard/api/permissionsinfo] and method [GET]"}
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:479:27)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
    at SearchGuardConfigurationBackend._client (/usr/share/kibana/plugins/searchguard/server/applications/searchguard/configuration/backend/searchguard_configuration_backend.js:28:20)
    at SearchGuardConfigurationBackend.restapiinfo (/usr/share/kibana/plugins/searchguard/server/applications/searchguard/configuration/backend/searchguard_configuration_backend.js:37:14)
    at /usr/share/kibana/plugins/searchguard/server/applications/searchguard/configuration/routes/routes.js:110:20
    at Router.handle (/usr/share/kibana/node_modules/@kbn/core-http-router-server-internal/src/router.js:171:30)
    at handler (/usr/share/kibana/node_modules/@kbn/core-http-router-server-internal/src/router.js:113:50)
    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)
    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)
[2024-03-11T21:48:47.678+00:00][ERROR][plugins.searchguard.authTokens] getAuthTokens: ResponseError: {"error":"no handler found for uri [/_searchguard/authtoken/_search] and method [POST]"}
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:479:27)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
[2024-03-11T21:48:57.405+00:00][WARN][savedobjects-service] The get saved object API /api/saved_objects/_find is deprecated.
[2024-03-11T21:49:05.808+00:00][ERROR][plugins.searchguard.searchguard] getRestApiInfo: ResponseError: {"error":"no handler found for uri [/_searchguard/api/permissionsinfo] and method [GET]"}
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:479:27)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
    at SearchGuardConfigurationBackend._client (/usr/share/kibana/plugins/searchguard/server/applications/searchguard/configuration/backend/searchguard_configuration_backend.js:28:20)
    at SearchGuardConfigurationBackend.restapiinfo (/usr/share/kibana/plugins/searchguard/server/applications/searchguard/configuration/backend/searchguard_configuration_backend.js:37:14)
    at /usr/share/kibana/plugins/searchguard/server/applications/searchguard/configuration/routes/routes.js:110:20
    at Router.handle (/usr/share/kibana/node_modules/@kbn/core-http-router-server-internal/src/router.js:171:30)
    at handler (/usr/share/kibana/node_modules/@kbn/core-http-router-server-internal/src/router.js:113:50)
    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)
    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)
[2024-03-11T21:49:05.922+00:00][ERROR][plugins.searchguard.authTokens] getAuthTokens: ResponseError: {"error":"no handler found for uri [/_searchguard/authtoken/_search] and method [POST]"}
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:479:27)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
[2024-03-11T21:49:06.284+00:00][WARN][plugins.dataViews.dataView.storage] Invalid response. [item.attributes.fields.987.name]: expected value of type [string] but got [undefined]
[2024-03-11T21:49:14.534+00:00][WARN][savedobjects-service] The get saved object API /api/saved_objects/_find is deprecated.

```

---

<div class="post-metadata">

**Author:** ![mantas](https://avatars.discourse-cdn.com/v4/letter/m/7bcc69/32.png) [@mantas](https://forum.search-guard.com/u/mantas)\
**Post date:** [March 21, 2024, 11:14am UTC](https://forum.search-guard.com/t/elasticsearch-permission-problems-when-upgrading-from-8-8-2-to-8-11-4/2490/8 "2024-03-21T11:14:58Z")

</div>

> [@silentfilm](#):
>
> But I still get these errors in Elasticsearch when Kibana starts up:

Hi @silentfilm,

Have you uploaded your configuration changes with `sgctl`?

> **[Configuration changes](https://docs.search-guard.com/latest/sgctl-configuration-changes#uploading-multiple-configuration-files)**
>
> How to use sgctl to connect to an Elasticsearch cluster and upload configuration changes

Please note: “sgctl will replace the current configuration in your Elasticsearch cluster with the one you provide. We recommended to [backup](https://docs.search-guard.com/latest/sgctl-configuration-changes#backup-and-restore) the configuration first before applying changes.”

Best,  
Mantas

---

<div class="post-metadata">

**Author:** ![mantas](https://avatars.discourse-cdn.com/v4/letter/m/7bcc69/32.png) [@mantas](https://forum.search-guard.com/u/mantas)\
**Post date:** [March 21, 2024, 11:29am UTC](https://forum.search-guard.com/t/elasticsearch-permission-problems-when-upgrading-from-8-8-2-to-8-11-4/2490/9 "2024-03-21T11:29:42Z")

</div>

> [@silentfilm](#):
>
> I tried installing the SearchGuard plugin, but I get

Would you mind elaborating more on this:  
Versions of Elasticsearch (8.11.4?) and the SearchGuard(1.5.0?)?  
Process or Docs followed to install it?

Thanks,  
Mantas

---

<div class="post-metadata">

**Author:** ![silentfilm](https://avatars.discourse-cdn.com/v4/letter/s/dbc845/32.png) [@silentfilm](https://forum.search-guard.com/u/silentfilm)\
**Post date:** [March 21, 2024, 1:53pm UTC](https://forum.search-guard.com/t/elasticsearch-permission-problems-when-upgrading-from-8-8-2-to-8-11-4/2490/10 "2024-03-21T13:53:33Z")

</div>

I’m using Elasticsearch 8.12.1 now and Search Guard 1.5.0-es-8.12.1.

My Dockerfile has this (the search-guard file has been renamed so I don’t have to change the code every time the version number changes).

```auto
COPY search-guard-flx.zip ${ES_HOME}
RUN yes y | bin/elasticsearch-plugin install -b file://${ES_HOME}/search-guard-flx.zip && \
    bin/elasticsearch-plugin list && \
    rm -f ${ES_HOME}/search-guard*.zip

```

The configuration is updated in a shell file when the Elasticsearch docker container starts up:

```auto
${ES_HOME}/sgctl-1.1.0/sgctl.sh connect -v ${HOSTNAME} --ca-cert ${ES_CERT_PATH}/${ES_ADMIN_CA_FILENAME} \
    --cert ${ES_CERT_PATH}/${ES_ADMIN_CERT_FILENAME} --key ${ES_CERT_PATH}/${ES_ADMIN_KEY_FILENAME} \
    --key-pass ${1} "${@:2}"

${ES_HOME}/sgctl-1.1.0/sgctl.sh update-config -v ${ES_CONFIG}/sg

```

---

<div class="post-metadata">

**Author:** ![mantas](https://avatars.discourse-cdn.com/v4/letter/m/7bcc69/32.png) [@mantas](https://forum.search-guard.com/u/mantas)\
**Post date:** [April 10, 2024, 3:01pm UTC](https://forum.search-guard.com/t/elasticsearch-permission-problems-when-upgrading-from-8-8-2-to-8-11-4/2490/11 "2024-04-10T15:01:28Z")

</div>

Hi @silentfilm,

Is the issue still applicable?  
if yes, could you run the below and share the output?  
`curl --insecure -u admin:<admin-password> -XGET https://<ES-HOST>:9200/_searchguard/api/roles/sg_inges?pretty`

Thanks,  
Mantas

---

<div class="post-metadata">

**Author:** ![silentfilm](https://avatars.discourse-cdn.com/v4/letter/s/dbc845/32.png) [@silentfilm](https://forum.search-guard.com/u/silentfilm)\
**Post date:** [April 10, 2024, 3:57pm UTC](https://forum.search-guard.com/t/elasticsearch-permission-problems-when-upgrading-from-8-8-2-to-8-11-4/2490/12 "2024-04-10T15:57:09Z")

</div>

The issue is not happening anymore. I can’t remember exactly how I fixed it, but I think it is because I created a specific role for ingestion and assigned it to the kibanaserver.

```auto
sg_ingest:
  cluster_permissions:
    - "cluster:admin/ingest/pipeline/put"
  index_permissions: []
  tenant_permissions: []
  exclude_cluster_permissions: []
  exclude_index_permissions: []

```

It seems like assigning all cluster privileges to the kibanaserver would have worked, but it didn’t.

---

<div class="post-metadata">

**Author:** ![Domsalvator23](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.search-guard.com/domsalvator23/32/1576_2.png) [@Domsalvator23](https://forum.search-guard.com/u/Domsalvator23)\
**Post date:** [June 21, 2024, 7:14am UTC](https://forum.search-guard.com/t/elasticsearch-permission-problems-when-upgrading-from-8-8-2-to-8-11-4/2490/13 "2024-06-21T07:14:47Z")

</div>

Glad to hear the issue is resolved! Creating a specific role for ingestion and assigning it to the kibanaserver likely fixed it. Here is the role I used;

sg\_ingest:  
cluster\_permissions:  
- “cluster:admin/ingest/pipeline/put”  
index\_permissions:   
tenant\_permissions:   
exclude\_cluster\_permissions:   
exclude\_index\_permissions:

Hope this helps for facing similar issue!

---

<div class="post-metadata">

**Author:** ![silentfilm](https://avatars.discourse-cdn.com/v4/letter/s/dbc845/32.png) [@silentfilm](https://forum.search-guard.com/u/silentfilm)\
**Post date:** [June 21, 2024, 2:20pm UTC](https://forum.search-guard.com/t/elasticsearch-permission-problems-when-upgrading-from-8-8-2-to-8-11-4/2490/14 "2024-06-21T14:20:12Z")

</div>

I had to create exactly the same role when I upgraded from Elasticsearch 8.8.2 to 8.12.1 and it fixed the issue.