Broken Search Guard Indexes after upgrading to ES 8.19

gerhro · April 17, 2026, 10:04am

Elasticsearch version: 8.19.11
Searchguard Version: flx-4_0_1
Operatin System: RHEL 9.6

Describe the issue:

we can’t create any new watches, since the upgrade to 8.19.11.
we cannot use the “StackManagment/Upgrade Assistant”
- except if we delete the .searchguard_* and .signals* index with the old versions.
we can not reproduce the issue on our test environment.
searchguard and signals indexe are old versions (versions.versions_created_string: 7.9,2, some examples below
we cannot migrate the searchguard index to .searchguard

Questions:

is there a way to trigger the migration from the searchguard and signals Index?

Error, when try to migrate the sg-index:

./sgctl-4.0.1.sh special move-sg-index --verbose
Selected cluster: XXXX
Connecting to XXXX:9201 with certificate CN=sgadmin, OU=.......
Successfully connected to cluster XXXX (XXXX) as user CN=sgadmin,....
Moving searchguard index to new-style location
Cannot invoke "org.elasticsearch.common.bytes.BytesReference.length()" because "this.delegate" is null

1|elasticsearch-MASTER  | [2026-04-17T11:33:44,366][ERROR][c.f.s.c.ConfigurationRepository] [lvgmyloges51] Error while doing builk write for migration. Deleting index again
1|elasticsearch-MASTER  | java.lang.NullPointerException: Cannot invoke "org.elasticsearch.common.bytes.BytesReference.length()" because "this.delegate" is null
1|elasticsearch-MASTER  | 	at org.elasticsearch.common.bytes.ReleasableBytesReference.length(ReleasableBytesReference.java:141) ~[elasticsearch-8.19.11.jar:?]
1|elasticsearch-MASTER  | 	at org.elasticsearch.action.index.IndexRequest.ramBytesUsed(IndexRequest.java:865) ~[elasticsearch-8.19.11.jar:?]
1|elasticsearch-MASTER  | 	at java.util.stream.ReferencePipeline$5$1.accept(ReferencePipeline.java:248) ~[?:?]
1|elasticsearch-MASTER  | 	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1716) ~[?:?]
1|elasticsearch-MASTER  | 	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:570) ~[?:?]
1|elasticsearch-MASTER  | 	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:560) ~[?:?]
1|elasticsearch-MASTER  | 	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
1|elasticsearch-MASTER  | 	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265) ~[?:?]
1|elasticsearch-MASTER  | 	at java.util.stream.LongPipeline.reduce(LongPipeline.java:501) ~[?:?]
1|elasticsearch-MASTER  | 	at java.util.stream.LongPipeline.sum(LongPipeline.java:459) ~[?:?]
1|elasticsearch-MASTER  | 	at org.elasticsearch.action.bulk.BulkRequest.ramBytesUsed(BulkRequest.java:504) ~[elasticsearch-8.19.11.jar:?]
1|elasticsearch-MASTER  | 	at org.elasticsearch.action.bulk.TransportAbstractBulkAction.doExecute(TransportAbstractBulkAction.java:114) ~[elasticsearch-8.19.11.jar:?]
1|elasticsearch-MASTER  | 	at org.elasticsearch.action.bulk.TransportAbstractBulkAction.doExecute(TransportAbstractBulkAction.java:56) ~[elasticsearch-8.19.11.jar:?]
1|elasticsearch-MASTER  | 	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:135) ~[elasticsearch-8.19.11.jar:?]
1|elasticsearch-MASTER  | 	at org.elasticsearch.action.support.MappedActionFilters$MappedFilterChain.proceed(MappedActionFilters.java:71) ~[elasticsearch-8.19.11.jar:?]
1|elasticsearch-MASTER  | 	at org.elasticsearch.action.support.MappedActionFilters.apply(MappedActionFilters.java:49) ~[elasticsearch-8.19.11.jar:?]
1|elasticsearch-MASTER  | 	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:132) ~[elasticsearch-8.19.11.jar:?]
1|elasticsearch-MASTER  | 	at com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:231) ~[?:?]
1|elasticsearch-MASTER  | 	at com.floragunn.searchguard.filter.SearchGuardFilter.lambda$apply$0(SearchGuardFilter.java:139) ~[?:?]
1|elasticsearch-MASTER  | 	at com.floragunn.searchguard.privileges.SpecialPrivilegesEvaluationContextProviderRegistry.provide(SpecialPrivilegesEvaluationContextProviderRegistry.java:50) ~[?:?]
1|elasticsearch-MASTER  | 	at com.floragunn.searchguard.privileges.SpecialPrivilegesEvaluationContextProviderRegistry.lambda$provide$0(SpecialPrivilegesEvaluationContextProviderRegistry.java:42) ~[?:?]
1|elasticsearch-MASTER  | 	at com.floragunn.searchguard.authtoken.AuthTokenService.provide(AuthTokenService.java:804) ~[?:?]
1|elasticsearch-MASTER  | 	at com.floragunn.searchguard.privileges.SpecialPrivilegesEvaluationContextProviderRegistry.provide(SpecialPrivilegesEvaluationContextProviderRegistry.java:38) ~[?:?]
1|elasticsearch-MASTER  | 	at com.floragunn.searchguard.privileges.SpecialPrivilegesEvaluationContextProviderRegistry.lambda$provide$0(SpecialPrivilegesEvaluationContextProviderRegistry.java:42) ~[?:?]
1|elasticsearch-MASTER  | 	at com.floragunn.searchguard.internalauthtoken.InternalAuthTokenProvider.userAuthFromToken(InternalAuthTokenProvider.java:133) ~[?:?]
1|elasticsearch-MASTER  | 	at com.floragunn.searchguard.privileges.SpecialPrivilegesEvaluationContextProviderRegistry.provide(SpecialPrivilegesEvaluationContextProviderRegistry.java:38) ~[?:?]
1|elasticsearch-MASTER  | 	at com.floragunn.searchguard.privileges.SpecialPrivilegesEvaluationContextProviderRegistry.provide(SpecialPrivilegesEvaluationContextProviderRegistry.java:28) ~[?:?]
1|elasticsearch-MASTER  | 	at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:136) ~[?:?]
1|elasticsearch-MASTER  | 	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:132) ~[elasticsearch-8.19.11.jar:?]
1|elasticsearch-MASTER  | 	at org.elasticsearch.action.support.TransportAction.handleExecution(TransportAction.java:96) ~[elasticsearch-8.19.11.jar:?]
1|elasticsearch-MASTER  | 	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:59) ~[elasticsearch-8.19.11.jar:?]
1|elasticsearch-MASTER  | 	at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:197) ~[elasticsearch-8.19.11.jar:?]
1|elasticsearch-MASTER  | 	at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:106) ~[elasticsearch-8.19.11.jar:?]
1|elasticsearch-MASTER  | 	at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:84) ~[elasticsearch-8.19.11.jar:?]
1|elasticsearch-MASTER  | 	at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:140) ~[elasticsearch-8.19.11.jar:?]
1|elasticsearch-MASTER  | 	at org.elasticsearch.client.internal.FilterClient.doExecute(FilterClient.java:56) ~[elasticsearch-8.19.11.jar:?]
1|elasticsearch-MASTER  | 	at com.floragunn.searchguard.support.PrivilegedConfigClient.doExecute(PrivilegedConfigClient.java:79) ~[?:?]
1|elasticsearch-MASTER  | 	at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:140) ~[elasticsearch-8.19.11.jar:?]
1|elasticsearch-MASTER  | 	at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:126) ~[elasticsearch-8.19.11.jar:?]
1|elasticsearch-MASTER  | 	at org.elasticsearch.client.internal.support.AbstractClient.bulk(AbstractClient.java:215) ~[elasticsearch-8.19.11.jar:?]
1|elasticsearch-MASTER  | 	at com.floragunn.searchguard.configuration.ConfigurationRepository.migrateIndex(ConfigurationRepository.java:591) ~[?:?]
1|elasticsearch-MASTER  | 	at com.floragunn.searchguard.configuration.api.MigrateConfigIndexApi$Handler.lambda$doExecute$0(MigrateConfigIndexApi.java:59) ~[?:?]
1|elasticsearch-MASTER  | 	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1789) ~[?:?]
1|elasticsearch-MASTER  | 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:977) ~[elasticsearch-8.19.11.jar:?]
1|elasticsearch-MASTER  | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) ~[?:?]
1|elasticsearch-MASTER  | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) ~[?:?]
1|elasticsearch-MASTER  | 	at java.lang.Thread.run(Thread.java:1474) ~[?:?]

Error when loading the Upgrade Assistant:

/opt/logs/pm2/appladm/kibana/kibanaProd01-out.log last 15 lines:
1|kibana-k | 		index_not_found_exception: no such index [.searchguard_authtokens]
1|kibana-k |     at KibanaTransport._request (/opt/logserver/kibanaProd/kibana-8.19.11/node_modules/@elastic/transport/lib/Transport.js:529:27)
1|kibana-k |     at processTicksAndRejections (node:internal/process/task_queues:105:5)
1|kibana-k |     at KibanaTransport.request (/opt/logserver/kibanaProd/kibana-8.19.11/node_modules/@elastic/transport/lib/Transport.js:642:20)
1|kibana-k |     at KibanaTransport.request (/opt/logserver/kibanaProd/kibana-8.19.11/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:60:16)
1|kibana-k |     at /opt/logserver/kibanaProd/kibana-8.19.11/plugins/searchguard/server/applications/authtokens/routes.js:44:22
1|kibana-k |     at handle (/opt/logserver/kibanaProd/kibana-8.19.11/node_modules/@kbn/core-http-router-server-internal/src/route.js:126:26)
1|kibana-k |     at handler (/opt/logserver/kibanaProd/kibana-8.19.11/node_modules/@kbn/core-http-router-server-internal/src/route.js:47:14)
1|kibana-k |     at Router.handle (/opt/logserver/kibanaProd/kibana-8.19.11/node_modules/@kbn/core-http-router-server-internal/src/router.js:138:30)
1|kibana-k |     at /opt/logserver/kibanaProd/kibana-8.19.11/node_modules/@kbn/core-http-router-server-internal/src/router.js:123:51
1|kibana-k |     at exports.Manager.execute (/opt/logserver/kibanaProd/kibana-8.19.11/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
1|kibana-k |     at Object.internals.handler (/opt/logserver/kibanaProd/kibana-8.19.11/node_modules/@hapi/hapi/lib/handler.js:46:20)
1|kibana-k |     at exports.execute (/opt/logserver/kibanaProd/kibana-8.19.11/node_modules/@hapi/hapi/lib/handler.js:31:20)
1|kibana-k |     at Request._lifecycle (/opt/logserver/kibanaProd/kibana-8.19.11/node_modules/@hapi/hapi/lib/request.js:370:32)
1|kibana-k |     at Request._execute (/opt/logserver/kibanaProd/kibana-8.19.11/node_modules/@hapi/hapi/lib/request.js:280:9)

Examples, different old versions:

./sgctl-4.0.1.sh rest get .searchguard_authtokens/_settings?human
Successfully connected to cluster XXXX (XXXXX) as user CN=sgadmin,OU=....
{
  ".searchguard_authtokens" : {
    "settings" : {
      "index" : {
        "creation_date_string" : "2022-06-25T11:27:51.255Z",
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "hidden" : "true",
        "number_of_shards" : "1",
        "provided_name" : ".searchguard_authtokens",
        "creation_date" : "1656156471255",
        "number_of_replicas" : "1",
        "uuid" : "sagiJ-8TRACgMH2OhSxqMA",
        "version" : {
          "created_string" : "7.9.2",
          "created" : "7090299"
        }
      }
    }
  }

./sgctl-4.0.1.sh rest get searchguard/_settings?human
Successfully connected to cluster XXXX (XXXXX) as user CN=sgadmin,OU=....
{
  "searchguard" : {
    "settings" : {
      "index" : {
        "creation_date_string" : "2021-07-05T06:59:02.519Z",
        "number_of_shards" : "1",
        "auto_expand_replicas" : "false",
        "provided_name" : "searchguard",
        "creation_date" : "1625468342519",
        "number_of_replicas" : "1",
        "uuid" : "HY6z-QcPTBqxRJ1MRShHsw",
        "version" : {
          "created_string" : "7.9.2",
          "created" : "7090299"
        }
      }
    }
  }
}

**Logs,Kibana: **

1|kibana-kibanaProd01  | [2026-04-17T11:21:37.021+02:00][ERROR][plugins.searchguard.signals-watch-routes] stateOfWatch: ResponseError: {"status":503,"error":"Signals is still initializing. Please try again later."}
1|kibana-kibanaProd01  |     at KibanaTransport._request (/opt/logserver/kibanaProd/kibana-8.19.11/node_modules/@elastic/transport/lib/Transport.js:529:27)
1|kibana-kibanaProd01  |     at processTicksAndRejections (node:internal/process/task_queues:105:5)
1|kibana-kibanaProd01  |     at KibanaTransport.request (/opt/logserver/kibanaProd/kibana-8.19.11/node_modules/@elastic/transport/lib/Transport.js:642:20)
1|kibana-kibanaProd01  |     at KibanaTransport.request (/opt/logserver/kibanaProd/kibana-8.19.11/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:60:16)
1|kibana-kibanaProd01  |     at /opt/logserver/kibanaProd/kibana-8.19.11/plugins/searchguard/server/applications/signals/routes/watch/state.js:30:20
1|kibana-kibanaProd01  |     at handle (/opt/logserver/kibanaProd/kibana-8.19.11/node_modules/@kbn/core-http-router-server-internal/src/route.js:126:26)
1|kibana-kibanaProd01  |     at handler (/opt/logserver/kibanaProd/kibana-8.19.11/node_modules/@kbn/core-http-router-server-internal/src/route.js:47:14)
1|kibana-kibanaProd01  |     at Router.handle (/opt/logserver/kibanaProd/kibana-8.19.11/node_modules/@kbn/core-http-router-server-internal/src/router.js:138:30)
1|kibana-kibanaProd01  |     at /opt/logserver/kibanaProd/kibana-8.19.11/node_modules/@kbn/core-http-router-server-internal/src/router.js:123:51
1|kibana-kibanaProd01  |     at exports.Manager.execute (/opt/logserver/kibanaProd/kibana-8.19.11/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
1|kibana-kibanaProd01  |     at Object.internals.handler (/opt/logserver/kibanaProd/kibana-8.19.11/node_modules/@hapi/hapi/lib/handler.js:46:20)
1|kibana-kibanaProd01  |     at exports.execute (/opt/logserver/kibanaProd/kibana-8.19.11/node_modules/@hapi/hapi/lib/handler.js:31:20)
1|kibana-kibanaProd01  |     at Request._lifecycle (/opt/logserver/kibanaProd/kibana-8.19.11/node_modules/@hapi/hapi/lib/request.js:370:32)
1|kibana-kibanaProd01  |     at Request._execute (/opt/logserver/kibanaProd/kibana-8.19.11/node_modules/@hapi/hapi/lib/request.js:280:9)

Error Message when try to create a new Watch:

{
  "statusCode": 500,
  "error": "Internal Server Error",
  "message": "{\"status\":500,\"error\":\"Unexpectd exceptionw while signing watch avg_ticket_price\"}",
  "attributes": {
    "body": {
      "status": 500,
      "error": "Unexpectd exceptionw while signing watch avg_ticket_price"
    }
  }
}

Topic		Replies	Views
Reindex searchguard index? Search Guard	6	1128	January 27, 2020
ES 6.4.0 -> 6.5.2: Kibana cannot migrate index: "No cluster-level perm match for User kibanaserver" Search Guard	2	1669	December 12, 2018
Severe performance problems caused by SearchGuard Search Guard	22	1213	June 28, 2022
In my Elastic Search server, i get this Search guard index error Search Guard	5	61	September 4, 2024
Search Guard 5.3.0-11 released Search Guard	9	450	May 8, 2017

Broken Search Guard Indexes after upgrading to ES 8.19

Related topics