SG - Error in com.floragunn.searchguard.rest.PermissionAction java.lang.NullPointerException

Search Guard
1

Good day

Elasticsearch version: 7.17.9-1

**Kibana version (if relevant):**7.17.9-1

Searchgaurd : search-guard-suite-security-7.17.9-53.6.0.

Describe the issue:
Elastic installed with SG with following settings

searchguard.disabled: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.clientauth_mode: REQUIRE
searchguard.ssl.transport.keystore_filepath: certs/hostname-serverclient.jks
searchguard.ssl.transport.keystore_password: xxxxxxxxxxxxxxx
searchguard.ssl.transport.truststore_filepath: certs/CANAME-ca.jks
searchguard.ssl.transport.truststore_password: xxxxxxxxxxxxxxxx
searchguard.config_index_name: searchguard
searchguard.ssl.http.keystore_filepath: certs/hostname-serverclient.jks
searchguard.ssl.http.keystore_password: xxxxxxxxxxxxxx
searchguard.ssl.http.truststore_filepath: certs/CANAME-ca.jks
searchguard.ssl.http.truststore_password: xxxxxxxxxxxxxxxxx
searchguard.authcz.admin_dn:
  - 'CN=servername'

Kibana setup with

elasticsearch.ssl.verificationMode: certificate
elasticsearch.ssl.keystore.path: "/etc/kibana/certs/servername-serverclient.p12"
elasticsearch.ssl.keystore.password: xxxxxxxxxxxxxxxxxxxx
elasticsearch.ssl.truststore.path: "/etc/kibana/certs/CANAME-ca.p12"
elasticsearch.ssl.truststore.password: xxxxxxxxxxxxxxxxxxxx

elasticsearch.ssl.alwaysPresentCertificate: true
elasticsearch.requestHeadersWhitelist: ["Authorization", "sgtenant"]
searchguard.allow_client_certificates: true
searchguard.basicauth.forbidden_usernames: ["kibanaserver"]
searchguard.auth.anonymous_auth_enabled: false
xpack.security.enabled: false
xpack.encryptedSavedObjects.encryptionKey: abc
xpack.reporting.encryptionKey: def
elasticsearch.ssl.certificateAuthorities: "/etc/kibana/certs/CANAME-ca-cert.pem"

SG works with curl to validate user

curl -k --cert "/data/certs/hosts/hostname/hostname-serverclient-cert.pem" --key "/data/certs/hosts/hostname/hostname-serverclient-key.pem" -XGET "https://localhost:9200/_searchguard/authinfo?pretty"

result with (note it works with kibanaserver user also but I have to change the elastic cert from REQUIRE to OPTIONAL for that )

{
  "user" : "User CN=hostname<tls_cert>",
  "user_name" : "CN=hostname",
  "user_requested_tenant" : null,
  "remote_address" : null,
  "backend_roles" : [ ],
  "custom_attribute_names" : [ ],
  "attribute_names" : [ ],
  "sg_roles" : [
    "SGS_ALL_ACCESS",
    "SGS_KIBANA_SERVER",
    "SGS_OWN_INDEX",
    "SGS_SIGNALS_ALL"
  ],
  "sg_tenants" : {
    "admin_tenant" : true,
    "CN=hostname : true,
    "SGS_GLOBAL_TENANT" : true
  },
  "principal" : "CN=hostname",
  "peer_certificates" : "1",
  "sso_logout_url" : null
}

the curl runs without any error in elastic log file.
But when I try and open Kibana, it does not open (blank page) and elastic has the following error

[2024-07-11T09:54:22,638][ERROR][c.f.s.f.SearchGuardRestFilter] [node-0] Error in com.floragunn.searchguard.rest.PermissionAction@45e3eb23
java.lang.NullPointerException: null
        at java.util.Objects.requireNonNull(Objects.java:233) ~[?:?]
        at com.floragunn.searchguard.rest.PermissionAction$1.accept(PermissionAction.java:72) ~[?:?]
        at com.floragunn.searchguard.rest.PermissionAction$1.accept(PermissionAction.java:66) ~[?:?]
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:109) ~[elasticsearch-7.17.9.jar:7.17.9]
        at com.floragunn.searchguard.filter.SearchGuardRestFilter$1.lambda$handleRequest$0(SearchGuardRestFilter.java:97) [search-guard-suite-security-7.17.9-53.6.0.jar:7.17.9-53.6.0]
        at com.floragunn.searchguard.auth.BackendRegistry.authenticate(BackendRegistry.java:361) [search-guard-suite-security-7.17.9-53.6.0.jar:7.17.9-53.6.0]
        at com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:90) [search-guard-suite-security-7.17.9-53.6.0.jar:7.17.9-53.6.0]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:337) [elasticsearch-7.17.9.jar:7.17.9]
        at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:403) [elasticsearch-7.17.9.jar:7.17.9]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:255) [elasticsearch-7.17.9.jar:7.17.9]
 at com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [search-guard-suite-security-7.17.9-53.6.0.jar:7.17.9-53.6.0]
        at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:382) [elasticsearch-7.17.9.jar:7.17.9]
        at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:461) [elasticsearch-7.17.9.jar:7.17.9]
        at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:357) [elasticsearch-7.17.9.jar:7.17.9]
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:35) [transport-netty4-client-7.17.9.jar:7.17.9]
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:19) [transport-netty4-client-7.17.9.jar:7.17.9]
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:48) [transport-netty4-client-7.17.9.jar:7.17.9]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1374) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1237) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.66.Final.jar:4.1.66.Final]
        at java.lang.Thread.run(Thread.java:1589) [?:?]
[2024-07-11T09:54:22,641][WARN ][r.suppressed             ] [node-0] path: /_searchguard/permission, params: {permissions=cluster:admin:searchguard:tenant:signals:watch/get}
java.lang.NullPointerException: null
        at java.util.Objects.requireNonNull(Objects.java:233) ~[?:?]
        at com.floragunn.searchguard.rest.PermissionAction$1.accept(PermissionAction.java:72) ~[?:?]
        at com.floragunn.searchguard.rest.PermissionAction$1.accept(PermissionAction.java:66) ~[?:?]
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:109) ~[elasticsearch-7.17.9.jar:7.17.9]
        at com.floragunn.searchguard.filter.SearchGuardRestFilter$1.lambda$handleRequest$0(SearchGuardRestFilter.java:97) [search-guard-suite-security-7.17.9-53.6.0.jar:7.17.9-53.6.0]
        at com.floragunn.searchguard.auth.BackendRegistry.authenticate(BackendRegistry.java:361) [search-guard-suite-security-7.17.9-53.6.0.jar:7.17.9-53.6.0]
        at com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:90) [search-guard-suite-security-7.17.9-53.6.0.jar:7.17.9-53.6.0]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:337) [elasticsearch-7.17.9.jar:7.17.9]
        at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:403) [elasticsearch-7.17.9.jar:7.17.9]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:255) [elasticsearch-7.17.9.jar:7.17.9]
        at com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [search-guard-suite-security-7.17.9-53.6.0.jar:7.17.9-53.6.0]
        at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:382) [elasticsearch-7.17.9.jar:7.17.9]
        at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:461) [elasticsearch-7.17.9.jar:7.17.9]
 at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:357) [elasticsearch-7.17.9.jar:7.17.9]
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:35) [transport-netty4-client-7.17.9.jar:7.17.9]
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:19) [transport-netty4-client-7.17.9.jar:7.17.9]
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:48) [transport-netty4-client-7.17.9.jar:7.17.9]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1374) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1237) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
         at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.66.Final.jar:4.1.66.Final]
        at java.lang.Thread.run(Thread.java:1589) [?:?]

The Kibana error is the following

{"type":"log","@timestamp":"2024-07-11T09:54:14+02:00","tags":["info","status"],"pid":1277529,"message":"Kibana is now available (was degraded)"}
{"type":"response","@timestamp":"2024-07-11T09:54:22+02:00","tags":[],"pid":1277529,"method":"get","statusCode":304,"req":{"url":"/translations/en.json","method":"get","headers":{"host":"localhost:5601","connection":"keep-alive","sec-ch-ua":"\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Brave\";v=\"126\"","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-gpc":"1","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"http://localhost:5601/login?nextUrl=%2F","accept-encoding":"gzip, deflate, br, zstd","accept-language":"af,en-US;q=0.9,en;q=0.8","if-none-match":"\"37992637719f97813c3068cfbf877b2d3bb43b97\""},"remoteAddress":"10.140.160.132","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","referer":"http://localhost:5601/login?nextUrl=%2F"},"res":{"statusCode":304,"responseTime":3,"contentLength":29},"message":"GET /translations/en.json 304 3ms - 29.0B"}
{"type":"log","@timestamp":"2024-07-11T09:54:22+02:00","tags":["error","plugins","searchguard","signals-searchguard-routes"],"pid":1277529,"message":"hasPermissions: ResponseError: null_pointer_exception: [null_pointer_exception] Reason: null\n    at onBody (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:367:23)\n    at IncomingMessage.onEnd (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:291:11)\n    at IncomingMessage.emit (node:events:525:35)\n    at endReadableNT (node:internal/streams/readable:1358:12)\n    at processTicksAndRejections (node:internal/process/task_queues:83:21)"}
{"type":"response","@timestamp":"2024-07-11T09:54:22+02:00","tags":[],"pid":1277529,"method":"post","statusCode":307,"req":{"url":"/api/core/capabilities?useDefaultCapabilities=true","method":"post","headers":{"host":"localhost:5601","connection":"keep-alive","content-length":"144","sec-ch-ua":"\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Brave\";v=\"126\"","content-type":"application/json","kbn-version":"7.17.9","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-gpc":"1","origin":"http://localhost:5601","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"http://localhost:5601/login?nextUrl=%2F","accept-encoding":"gzip, deflate, br, zstd","accept-language":"af,en-US;q=0.9,en;q=0.8"},"remoteAddress":"10.140.160.132","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","referer":"http://localhost:5601/login?nextUrl=%2F"},"res":{"statusCode":307,"responseTime":3},"message":"POST /api/core/capabilities?useDefaultCapabilities=true 307 3ms"}
{"type":"error","@timestamp":"2024-07-11T09:54:22+02:00","tags":[],"pid":1277529,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:128:19)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/kibana/src/core/server/http/router/response_adapter.js:82:19)\n    at HapiResponseAdapter.handle (/usr/share/kibana/src/core/server/http/router/response_adapter.js:73:17)\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:164:34)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)\n    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)\n    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)\n    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)\n    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)"},"url":"http://localhost:5601/api/searchguard-signals/searchguard/signals_has_permissions","message":"Internal Server Error"}

Due to the Kibana signal routes permission error I add the signals role to the cert user and kibanaserver, but no luck

Swapping
searchguard.ssl.http.clientauth_mode: REQUIRE
to OPTIONAL does not help
as can see from authinfo result the cert does have access.

Have no idea anymore what else to look for, will appreciate any help

2

Note I look at SG code and it seems this is the issue, just dont know why it is doing it. When I do the following query

curl -k --cert "/data/certs/hosts/hostname/hostname-serverclient-cert.pem" --key "/data/certs/hosts/hostname/hostname-serverclient-key.pem"  -XGET "https://localhost:9200/_searchguard/permission?pretty"

I get the error as seen from the logs

{
  "error" : {
    "root_cause" : [
      {
        "type" : "null_pointer_exception",
        "reason" : null
      }
    ],
    "type" : "null_pointer_exception",
    "reason" : null
  },
  "status" : 500
}
3

Something very odd I set the

searchguard.ssl.http.clientauth_mode: OPTIONAL

and then only used basic authentication

curl --insecure -u "kibanaserver:xxxxxxxxxxxx" -XGET "https://localhost:9200/_searchguard/permission"

and that seems to work, no error in log file and return a blank permission

{"permissions":{}}
4

setting kibana to basicauth only (remove config entries that refer to the certificates, else it sends the certs anyway and fall back to cert verfication)
it seems to work log file wise in elastic, but still results in a blank webpage

elastic log

[2024-07-11T13:19:40,533][DEBUG][c.f.s.p.PrivilegesEvaluator] [node-0] requested [indices:admin/refresh] from 10.140.160.132:34900
[2024-07-11T13:19:40,533][DEBUG][c.f.s.p.PrivilegesEvaluator] [node-0] requested resolved indextypes: Resolved [aliases=[], indices=[.kibana_task_manager_7.17.9_001], allIndices=[.kibana_task_manager_7.17.9_001], types=[*], originalRequested=[.kibana_task_manager_7.17.9_001], remoteIndices=[], localAll=false, indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, expand_wildcards_hidden=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=false, ignore_aliases=false, ignore_throttled=false]]
[2024-07-11T13:19:40,533][DEBUG][c.f.s.p.PrivilegesEvaluator] [node-0] sgr: [SGS_KIBANA_SERVER, SGS_OWN_INDEX]
[2024-07-11T13:19:40,533][DEBUG][c.f.s.p.PrivilegesEvaluator] [node-0] sgr2: [SGS_KIBANA_SERVER, SGS_OWN_INDEX]
[2024-07-11T13:19:40,534][DEBUG][c.f.s.p.PrivilegesEvaluator] [node-0] Allowed because we have all indices permissions for indices:admin/refresh
[2024-07-11T13:19:40,534][DEBUG][c.f.s.f.SearchGuardFilter] [node-0] PrivEvalResponse [allowed=true, missingPrivileges=[indices:admin/refresh], evaluatedDlsFlsConfig=null]
[2024-07-11T13:19:40,534][DEBUG][c.f.s.p.PrivilegesEvaluator] [node-0] ### evaluate permissions for User kibanaserver <basic/internal> on node-0
[2024-07-11T13:19:40,534][DEBUG][c.f.s.p.PrivilegesEvaluator] [node-0] action: indices:admin/refresh[s] (BasicReplicationRequest)
[2024-07-11T13:19:40,534][DEBUG][c.f.s.r.IndexResolverReplacer] [node-0] Resolve aliases, indices and types from BasicReplicationRequest
[2024-07-11T13:19:40,534][DEBUG][c.f.s.r.IndexResolverReplacer] [node-0] Resolved pattern [.kibana_task_manager_7.17.9_001] to [.kibana_task_manager_7.17.9_001]
[2024-07-11T13:19:40,534][DEBUG][c.f.s.p.PrivilegesEvaluator] [node-0] requestedResolved : Resolved [aliases=[], indices=[.kibana_task_manager_7.17.9_001], allIndices=[.kibana_task_manager_7.17.9_001], types=[*], originalRequested=[.kibana_task_manager_7.17.9_001], remoteIndices=[], localAll=false, indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, expand_wildcards_hidden=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=false, ignore_aliases=false, ignore_throttled=false]]
[2024-07-11T13:19:40,534][DEBUG][c.f.s.p.PrivilegesEvaluator] [node-0] requested [indices:admin/refresh[s]] from 10.140.160.132:34900
[2024-07-11T13:19:40,534][DEBUG][c.f.s.p.PrivilegesEvaluator] [node-0] requested resolved indextypes: Resolved [aliases=[], indices=[.kibana_task_manager_7.17.9_001], allIndices=[.kibana_task_manager_7.17.9_001], types=[*], originalRequested=[.kibana_task_manager_7.17.9_001], remoteIndices=[], localAll=false, indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, expand_wildcards_hidden=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=false, ignore_aliases=false, ignore_throttled=false]]
[2024-07-11T13:19:40,534][DEBUG][c.f.s.p.PrivilegesEvaluator] [node-0] sgr: [SGS_KIBANA_SERVER, SGS_OWN_INDEX]
[2024-07-11T13:19:40,534][DEBUG][c.f.s.p.PrivilegesEvaluator] [node-0] sgr2: [SGS_KIBANA_SERVER, SGS_OWN_INDEX]
[2024-07-11T13:19:40,535][DEBUG][c.f.s.p.PrivilegesEvaluator] [node-0] Allowed because we have all indices permissions for indices:admin/refresh[s]
[2024-07-11T13:19:40,535][DEBUG][c.f.s.f.SearchGuardFilter] [node-0] PrivEvalResponse [allowed=true, missingPrivileges=[indices:admin/refresh[s]], evaluatedDlsFlsConfig=null]

kibana log

{"type":"response","@timestamp":"2024-07-11T13:26:07+02:00","tags":[],"pid":1335610,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"host":"localhost:5601","connection":"keep-alive","sec-ch-ua":"\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Brave\";v=\"126\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"Windows\"","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8","sec-gpc":"1","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br, zstd","accept-language":"af,en-US;q=0.9,en;q=0.8"},"remoteAddress":"10.140.160.132","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"},"res":{"statusCode":302,"responseTime":6},"message":"GET / 302 6ms"}
{"type":"response","@timestamp":"2024-07-11T13:26:07+02:00","tags":[],"pid":1335610,"method":"get","statusCode":200,"req":{"url":"/login?nextUrl=%2F","method":"get","headers":{"host":"localhost:5601","connection":"keep-alive","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8","sec-gpc":"1","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","sec-ch-ua":"\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Brave\";v=\"126\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"Windows\"","accept-encoding":"gzip, deflate, br, zstd","accept-language":"af,en-US;q=0.9,en;q=0.8"},"remoteAddress":"10.140.160.132","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"},"res":{"statusCode":200,"responseTime":15,"contentLength":84388},"message":"GET /login?nextUrl=%2F 200 15ms - 82.4KB"}
{"type":"response","@timestamp":"2024-07-11T13:26:08+02:00","tags":[],"pid":1335610,"method":"get","statusCode":304,"req":{"url":"/node_modules/@kbn/ui-framework/dist/kui_light.css","method":"get","headers":{"host":"localhost:5601","connection":"keep-alive","sec-ch-ua":"\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Brave\";v=\"126\"","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"text/css,*/*;q=0.1","sec-gpc":"1","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"style","referer":"http://localhost:5601/login?nextUrl=%2F","accept-encoding":"gzip, deflate, br, zstd","accept-language":"af,en-US;q=0.9,en;q=0.8","if-none-match":"\"47af9b8f6e4640c400d5d0b64bd8c8095c9bbcf5-gzip\"","if-modified-since":"Mon, 30 Jan 2023 12:30:16 GMT"},"remoteAddress":"10.140.160.132","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","referer":"http://localhost:5601/login?nextUrl=%2F"},"res":{"statusCode":304,"responseTime":4,"contentLength":205},"message":"GET /node_modules/@kbn/ui-framework/dist/kui_light.css 304 4ms - 205.0B"}
{"type":"response","@timestamp":"2024-07-11T13:26:08+02:00","tags":[],"pid":1335610,"method":"get","statusCode":304,"req":{"url":"/ui/legacy_light_theme.css","method":"get","headers":{"host":"localhost:5601","connection":"keep-alive","sec-ch-ua":"\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Brave\";v=\"126\"","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"text/css,*/*;q=0.1","sec-gpc":"1","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"style","referer":"http://localhost:5601/login?nextUrl=%2F","accept-encoding":"gzip, deflate, br, zstd","accept-language":"af,en-US;q=0.9,en;q=0.8","if-none-match":"\"4345f9a3b9a243edd0b81b913ba1901bce549a2b-gzip\"","if-modified-since":"Mon, 30 Jan 2023 12:30:17 GMT"},"remoteAddress":"10.140.160.132","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","referer":"http://localhost:5601/login?nextUrl=%2F"},"res":{"statusCode":304,"responseTime":2,"contentLength":206},"message":"GET /ui/legacy_light_theme.css 304 2ms - 206.0B"}
{"type":"response","@timestamp":"2024-07-11T13:26:08+02:00","tags":["api"],"pid":1335610,"method":"get","statusCode":304,"req":{"url":"/bootstrap-anonymous.js","method":"get","headers":{"host":"localhost:5601","connection":"keep-alive","sec-ch-ua":"\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Brave\";v=\"126\"","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-gpc":"1","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"script","referer":"http://localhost:5601/login?nextUrl=%2F","accept-encoding":"gzip, deflate, br, zstd","accept-language":"af,en-US;q=0.9,en;q=0.8","if-none-match":"\"e8d6d16d89a24550043b44419a24e47bf4d74560-gzip\""},"remoteAddress":"10.140.160.132","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","referer":"http://localhost:5601/login?nextUrl=%2F"},"res":{"statusCode":304,"responseTime":4,"contentLength":3772},"message":"GET /bootstrap-anonymous.js 304 4ms - 3.7KB"}
{"type":"response","@timestamp":"2024-07-11T13:26:08+02:00","tags":[],"pid":1335610,"method":"get","statusCode":304,"req":{"url":"/ui/fonts/inter/Inter-Regular.woff2","method":"get","headers":{"host":"localhost:5601","connection":"keep-alive","sec-ch-ua":"\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Brave\";v=\"126\"","origin":"http://localhost:5601","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-gpc":"1","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"font","referer":"http://localhost:5601/login?nextUrl=%2F","accept-encoding":"gzip, deflate, br, zstd","accept-language":"af,en-US;q=0.9,en;q=0.8","if-none-match":"\"d54ba8fa95573fa24442dab96795eaf8db66aa80\"","if-modified-since":"Mon, 30 Jan 2023 12:30:19 GMT"},"remoteAddress":"10.140.160.132","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","referer":"http://localhost:5601/login?nextUrl=%2F"},"res":{"statusCode":304,"responseTime":3,"contentLength":215},"message":"GET /ui/fonts/inter/Inter-Regular.woff2 304 3ms - 215.0B"}
{"type":"response","@timestamp":"2024-07-11T13:26:08+02:00","tags":[],"pid":1335610,"method":"get","statusCode":304,"req":{"url":"/translations/en.json","method":"get","headers":{"host":"localhost:5601","connection":"keep-alive","sec-ch-ua":"\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Brave\";v=\"126\"","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-gpc":"1","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"http://localhost:5601/login?nextUrl=%2F","accept-encoding":"gzip, deflate, br, zstd","accept-language":"af,en-US;q=0.9,en;q=0.8","if-none-match":"\"37992637719f97813c3068cfbf877b2d3bb43b97\""},"remoteAddress":"10.140.160.132","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","referer":"http://localhost:5601/login?nextUrl=%2F"},"res":{"statusCode":304,"responseTime":2,"contentLength":29},"message":"GET /translations/en.json 304 2ms - 29.0B"}
{"type":"log","@timestamp":"2024-07-11T13:26:09+02:00","tags":["error","plugins","searchguard","signals-searchguard-routes"],"pid":1335610,"message":"hasPermissions: AuthenticationError: Unauthorized\n    at SearchGuardBackend.hasPermissions (/usr/share/kibana/plugins/searchguard/server/applications/searchguard/backend/searchguard.js:285:15)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at /usr/share/kibana/plugins/searchguard/server/applications/signals/routes/searchguard/has_permissions.js:23:36\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:163:30)\n    at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)\n    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)\n    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)\n    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)\n    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)"}
{"type":"response","@timestamp":"2024-07-11T13:26:09+02:00","tags":[],"pid":1335610,"method":"post","statusCode":307,"req":{"url":"/api/core/capabilities?useDefaultCapabilities=true","method":"post","headers":{"host":"localhost:5601","connection":"keep-alive","content-length":"144","sec-ch-ua":"\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Brave\";v=\"126\"","content-type":"application/json","kbn-version":"7.17.9","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-gpc":"1","origin":"http://localhost:5601","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"http://localhost:5601/login?nextUrl=%2F","accept-encoding":"gzip, deflate, br, zstd","accept-language":"af,en-US;q=0.9,en;q=0.8"},"remoteAddress":"10.140.160.132","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","referer":"http://localhost:5601/login?nextUrl=%2F"},"res":{"statusCode":307,"responseTime":3},"message":"POST /api/core/capabilities?useDefaultCapabilities=true 307 3ms"}
{"type":"error","@timestamp":"2024-07-11T13:26:09+02:00","tags":[],"pid":1335610,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:128:19)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/kibana/src/core/server/http/router/response_adapter.js:82:19)\n    at HapiResponseAdapter.handle (/usr/share/kibana/src/core/server/http/router/response_adapter.js:73:17)\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:164:34)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)\n    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)\n    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)\n    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)\n    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)"},"url":"http://localhost:5601/api/searchguard-signals/searchguard/signals_has_permissions","message":"Internal Server Error"}
{"type":"response","@timestamp":"2024-07-11T13:26:09+02:00","tags":[],"pid":1335610,"method":"post","statusCode":500,"req":{"url":"/api/searchguard-signals/searchguard/signals_has_permissions","method":"post","headers":{"host":"localhost:5601","connection":"keep-alive","content-length":"2","sec-ch-ua":"\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Brave\";v=\"126\"","content-type":"application/json","kbn-version":"7.17.9","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-gpc":"1","origin":"http://localhost:5601","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"http://localhost:5601/login?nextUrl=%2F","accept-encoding":"gzip, deflate, br, zstd","accept-language":"af,en-US;q=0.9,en;q=0.8"},"remoteAddress":"10.140.160.132","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","referer":"http://localhost:5601/login?nextUrl=%2F"},"res":{"statusCode":500,"responseTime":21,"contentLength":75},"message":"POST /api/searchguard-signals/searchguard/signals_has_permissions 500 21ms - 75.0B"}
{"type":"response","@timestamp":"2024-07-11T13:26:09+02:00","tags":[],"pid":1335610,"method":"post","statusCode":200,"req":{"url":"/api/v1/searchguard/kibana_capabilities","method":"post","headers":{"host":"localhost:5601","connection":"keep-alive","content-length":"144","sec-ch-ua":"\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Brave\";v=\"126\"","content-type":"application/json","kbn-version":"7.17.9","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-gpc":"1","origin":"http://localhost:5601","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"http://localhost:5601/login?nextUrl=%2F","accept-encoding":"gzip, deflate, br, zstd","accept-language":"af,en-US;q=0.9,en;q=0.8"},"remoteAddress":"10.140.160.132","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","referer":"http://localhost:5601/login?nextUrl=%2F"},"res":{"statusCode":200,"responseTime":2,"contentLength":361},"message":"POST /api/v1/searchguard/kibana_capabilities 200 2ms - 361.0B"}
{"type":"response","@timestamp":"2024-07-11T13:26:09+02:00","tags":[],"pid":1335610,"method":"get","statusCode":200,"req":{"url":"/api/v1/systeminfo","method":"get","headers":{"host":"localhost:5601","connection":"keep-alive","sec-ch-ua":"\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Brave\";v=\"126\"","content-type":"application/json","kbn-version":"7.17.9","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-gpc":"1","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"http://localhost:5601/login?nextUrl=%2F","accept-encoding":"gzip, deflate, br, zstd","accept-language":"af,en-US;q=0.9,en;q=0.8"},"remoteAddress":"10.140.160.132","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","referer":"http://localhost:5601/login?nextUrl=%2F"},"res":{"statusCode":200,"responseTime":5,"contentLength":1837},"message":"GET /api/v1/systeminfo 200 5ms - 1.8KB"}
{"type":"response","@timestamp":"2024-07-11T13:26:09+02:00","tags":[],"pid":1335610,"method":"get","statusCode":200,"req":{"url":"/api/v1/systeminfo","method":"get","headers":{"host":"localhost:5601","connection":"keep-alive","sec-ch-ua":"\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Brave\";v=\"126\"","content-type":"application/json","kbn-version":"7.17.9","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-gpc":"1","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"http://localhost:5601/login?nextUrl=%2F","accept-encoding":"gzip, deflate, br, zstd","accept-language":"af,en-US;q=0.9,en;q=0.8"},"remoteAddress":"10.140.160.132","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","referer":"http://localhost:5601/login?nextUrl=%2F"},"res":{"statusCode":200,"responseTime":3,"contentLength":1837},"message":"GET /api/v1/systeminfo 200 3ms - 1.8KB"}
5

first time I look at browser output and see the following

image
image809×443 12.8 KB

6

issue seem related to the C2S forward I am doing on my SSH connection to the linux box.
When I connect directly to the machine from windows machine on network it seems to display login box.

image
image487×728 28.9 KB

will fall back to cert level authentication and see if it still works

7

Works with cert verification also. Thus can close this ticket, issue was SSH port forward of webpage at the end of the day and not the searchgaurd (even if if gives all those nasty errors in the log files)

8

Hi @johannes,

Glad to hear you solved it, and thanks for documenting and sharing your findings.

Best,
Mantas

9

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.