NotSerializableException: io.netty.handler.codec.HeadersUtils$1 on cross-shard queries when using trusted_origin with attrs.from request headers

Signals Alerting
1

Elasticsearch version: 8.19.13

Server OS version: Amazon Linux 2023 (ECS/Docker)

Kibana version (if relevant): 8.19.13

Browser version (if relevant): N/A

Browser OS version (if relevant): N/A

Describe the issue:

All cross-shard queries fail with NotSerializableException: io.netty.handler.codec.HeadersUtils$1 when using the trusted_origin auth domain with attrs.from reading request headers in a 3-node cluster. The error occurs because SearchGuardInterceptor.ensureCorrectHeaders attempts to Java-serialize the user context for inter-node transport, but the context contains a Netty-internal object (HeadersUtils$1, the anonymous AbstractSet returned by HttpHeaders.namesSet()), which is not Java-serializable.

This affects all cross-shard queries for users authenticated via trusted_origin — not just specific indices. Even queries against .kibana_8.19.13 fail when its shard is on a different node than the coordinating node.

Note: transport_enabled: false is set on the trusted_origin domain, yet the serialized user context from that domain leaks into the transport layer.

Steps to reproduce:
Run a 3-node Elasticsearch 8.19.13 cluster with SearchGuard FLX 4.0.1.

  1. Configure trusted_origin auth domain with attrs.from using $.request.headers[...] (see configuration below).
  2. Authenticate a user via a reverse proxy (nginx) that injects x-proxy-user, x-proxy-roles, and x-proxy-attr-* headers.
  3. Issue any query whose shard resides on a node different from the coordinating node (e.g., GET /.kibana_8.19.13/_search).
  4. Observe 500 error with NotSerializableException: io.netty.handler.codec.HeadersUtils$1.

Expected behavior:

Cross-shard queries succeed. The user context stored for transport should contain only plain Java-serializable types (e.g., String, List<String>), with no references to Netty internal objects.

Provide configuration:
sg_authc.yml:

auth_domains:
  - type: trusted_origin
    enabled: true
    transport_enabled: false
    user_mapping:
      user_name:
        from: '$.request.headers["x-proxy-user"]'
      roles:
        from_comma_separated_string: '$.request.headers["x-proxy-roles"]'
      attrs:
        from:
          accesslog: '$.request.headers["x-proxy-attr-accesslog"]'
          aspmanid:  '$.request.headers["x-proxy-attr-aspmanid"]'
          appname:   '$.request.headers["x-proxy-attr-appname"]'
          fqdn:      '$.request.headers["x-proxy-attr-fqdn"]'
          maillog:   '$.request.headers["x-proxy-attr-maillog"]'
  - type: basic/internal_users_db
    enabled: true

elasticsearch.yml (relevant parts):

xpack.security.enabled: false
searchguard.ssl.transport.pemcert_filepath: es-node.pem
searchguard.ssl.transport.pemkey_filepath: es-node.key
searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: es-node.pem
searchguard.ssl.http.pemkey_filepath: es-node.key
searchguard.ssl.http.pemtrustedcas_filepath: root-ca.pem

Provide logs:

/_searchguard/authinfo for the affected user:
{
"user": "User ``user@example.com`` <trusted_origin/noop> [backend_roles=[permit_access_log, admin]]",
"user_name": "``user@example.com``",
"attribute_names": ["accesslog"],
"backend_roles": ["permit_access_log", "admin"],
"sg_roles": ["SGS_KIBANA_USER", "SG_ROLE_ACCESS_LOG", "SGS_ALL_ACCESS", "SGS_OWN_INDEX"]
}

Elasticsearch error log:

[ERROR] path: /.kibana_8.19.13/_search, params: {index=.kibana_8.19.13}, status: 500
Caused by: org.elasticsearch.ElasticsearchException: java.io.NotSerializableException: io.netty.handler.codec.HeadersUtils$1
at com.floragunn.searchguard.support.Base64Helper.serializeObject(Base64Helper.java:57)
at com.floragunn.searchguard.transport.SearchGuardInterceptor.ensureCorrectHeaders(SearchGuardInterceptor.java:240)
at com.floragunn.searchguard.transport.SearchGuardInterceptor.sendRequestDecorate(SearchGuardInterceptor.java:195)
at com.floragunn.searchguard.SearchGuardPlugin$5$1.sendRequest(SearchGuardPlugin.java:657)
at org.elasticsearch.server@8.19.13/org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:846)

Additional data:

  • Reproduced on both FLX 4.0.1-es-8.19.13 and 4.1.0-es-8.19.14.

  • Removing the attrs.from block entirely from sg_authc.yml resolves the error, confirming the issue is specific to attrs.from with request header JSONPath expressions.

  • io.netty.handler.codec.HeadersUtils$1 is the anonymous AbstractSet class returned by Netty’s HttpHeaders.namesSet(). It is not Serializable. The attrs.from evaluation via JSONPath over Netty’s HttpHeaders object appears to leave a reference to this internal class in the stored user context, which is then serialized by Base64Helper.serializeObject during transport request propagation.

    es.log (36.0 KB)