Dear All,
Elasticsearch version:
8.5.3
Server OS version:
Red Hat Enterprise Linux 8.6
Kibana version (if relevant):
8.5.3
Describe the issue:
After configuring SearchGuard FLX 1.1.1 Kibana doesn’t work. It gives the following error:
{“status”:“INTERNAL_SERVER_ERROR”,“error”:“SessionService is not configured”,“headers”:{}}
The authentication via curl is successfull with internal and ldap users too.
Provide configuration:
elasticsearch/config/elasticsearch.yml
cluster.name: elk-cl
node.name: ${HOSTNAME}
node.roles: [ remote_cluster_client ]
path.data: /opt/elasticsearch
path.logs: /opt/elasticsearch/log
network.host: ens192
http.host: local
http.port: 9200
discovery.seed_hosts: [“master1”,“master2”,“master3”]
xpack.security.enabled: false
searchguard.ssl.transport.pemcert_filepath: local.cer
searchguard.ssl.transport.pemkey_filepath: local.key
searchguard.ssl.transport.pemtrustedcas_filepath: ca-chain.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: local.cer
searchguard.ssl.http.pemkey_filepath: local.key
searchguard.ssl.http.pemtrustedcas_filepath: ca-chain.pem
searchguard.ssl.cert_reload_enabled: true
searchguard.authcz.admin_dn:
- CN=sgadmin
- CN=sgadmin
searchguard.restapi.roles_enabled: [“SGS_ALL_ACCESS”]
kibana/config/kibana.yml
server.port: 5601
server.host: “kibana1”
server.publicBaseUrl: “https://kibana1”
server.name: “kibana1”
elasticsearch.hosts: [“https://localhost:9200”]
elasticsearch.username: “kibanaserver”
elasticsearch.password: “pwd”
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/kibana.cer
server.ssl.key: /etc/kibana/kibana.key
elasticsearch.ssl.certificateAuthorities: [ “/etc/kibana/ca-chain.pem” ]
elasticsearch.ssl.verificationMode: none
xpack.reporting.roles.enabled: false
Provide logs:
Elasticsearch
[2023-09-07T01:16:53,674][ERROR][c.f.s.a.b.RequestAuthenticationProcessor] [kibana1] Error while authenticating AuthCredentials [username=ldapuser, subUserName=null, authDomainInfo=AuthDomainInfo [authDomainId=null, authenticatorType=null, authBackendType=null], password=REDACTED, nativeCredentials=null, backendRoles=, searchGuardRoles=, complete=true, authzComplete=false, redirectUri=null, attributes={}, structuredAttributes={}, claims={}, attributesForUserMapping={credentials={user_name=ldapuser}}]
java.lang.Exception: User not authenticated
at com.floragunn.searchguard.authc.base.RequestAuthenticationProcessor.lambda$callAuthcBackends$3(RequestAuthenticationProcessor.java:368) ~[?:?]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:887) ~[?:?]
at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2357) ~[?:?]
at com.floragunn.searchguard.authc.base.RequestAuthenticationProcessor.callAuthcBackends(RequestAuthenticationProcessor.java:360) ~[?:?]
at com.floragunn.searchguard.authc.base.RequestAuthenticationProcessor.proceed(RequestAuthenticationProcessor.java:217) ~[?:?]
at com.floragunn.searchguard.authc.session.ApiAuthenticationProcessor.handleCurrentAuthenticationDomain(ApiAuthenticationProcessor.java:133) ~[?:?]
at com.floragunn.searchguard.authc.base.RequestAuthenticationProcessor.checkCurrentAuthenticationDomain(RequestAuthenticationProcessor.java:179) ~[?:?]
at com.floragunn.searchguard.authc.base.RequestAuthenticationProcessor.checkNextAuthenticationDomains(RequestAuthenticationProcessor.java:143) ~[?:?]
at com.floragunn.searchguard.authc.base.RequestAuthenticationProcessor.authenticate(RequestAuthenticationProcessor.java:98) ~[?:?]
at com.floragunn.searchguard.authc.session.backend.SessionService.authenticate(SessionService.java:416) ~[?:?]
at com.floragunn.searchguard.authc.session.backend.SessionService.authenticateAndCreateSession(SessionService.java:265) ~[?:?]
at com.floragunn.searchguard.authc.session.backend.SessionApi$Rest.lambda$handlePost$6(SessionApi.java:313) ~[?:?]
at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:103) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:119) ~[?:?]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:397) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:531) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:312) ~[elasticsearch-8.5.3.jar:?]
at com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) ~[?:?]
at com.floragunn.searchguard.authc.rest.AuthenticatingRestFilter$AuthenticatingRestHandler.handleRequest(AuthenticatingRestFilter.java:231) ~[?:?]
at com.floragunn.searchguard.authc.rest.AuthenticatingRestFilter$AuthenticatingRestHandler.dispatchRequest(AuthenticatingRestFilter.java:150) ~[?:?]
at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:379) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:460) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:353) ~[elasticsearch-8.5.3.jar:?]
at com.floragunn.searchguard.http.SearchGuardHttpServerTransport.incomingRequest(SearchGuardHttpServerTransport.java:61) ~[?:?]
at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:128) ~[?:?]
at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:118) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372) ~[?:?]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]
[2023-09-07T01:16:53,969][INFO ][c.f.s.a.s.b.SessionService] [kibana1] Creating token failed
com.floragunn.searchguard.authc.session.backend.SessionCreationException: SessionService is not configured
at com.floragunn.searchguard.authc.session.backend.SessionService.createLightweightJwt(SessionService.java:422) ~[?:?]
at com.floragunn.searchguard.authc.session.backend.SessionService.lambda$authenticateAndCreateSession$1(SessionService.java:269) ~[?:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:825) ~[elasticsearch-8.5.3.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]
[2023-09-07T01:19:13,218][WARN ][c.f.s.a.b.RequestAuthenticationProcessor] [kibana1] Authentication failed for null from [request=/.kibana_8.5.3/_search, directIpAddress=127.0.0.1, originatingIpAddress=127.0.0.1, clientCertSubject=null]
Kibana
Sep 7 01:14:10 kibana1 kibana[2090388]: [2023-09-07T01:14:10.219+02:00][ERROR][plugins.security.authentication] License is not available or does not support security features, re-authentication is not possible (available: true, enabled: false).
Sep 7 01:14:10 kibana1 kibana[2090388]: [2023-09-07T01:14:10.219+02:00][WARN ][plugins.alerting] Error executing alerting apiKey invalidation task: Unauthorized: authentication_exception
Sep 7 01:16:53 kibana1 kibana[2090388]: ResponseError: {“status”:“INTERNAL_SERVER_ERROR”,“error”:“SessionService is not configured”,“headers”:{}}
Sep 7 01:16:53 kibana1 kibana[2090388]: at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)
Sep 7 01:16:53 kibana1 kibana[2090388]: at runMicrotasks ()
Sep 7 01:16:53 kibana1 kibana[2090388]: at processTicksAndRejections (node:internal/process/task_queues:96:5)
Sep 7 01:16:53 kibana1 kibana[2090388]: at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/src/create_transport.js:58:16)
Sep 7 01:16:53 kibana1 kibana[2090388]: at SearchGuardBackend._client (/usr/share/kibana/plugins/searchguard/server/applications/searchguard/backend/searchguard.js:32:20)
Sep 7 01:16:53 kibana1 kibana[2090388]: at SearchGuardBackend.authenticateWithSession (/usr/share/kibana/plugins/searchguard/server/applications/searchguard/backend/searchguard.js:70:24)
Sep 7 01:16:53 kibana1 kibana[2090388]: at BasicAuth.authenticate (/usr/share/kibana/plugins/searchguard/server/applications/searchguard/auth/types/AuthType.js:176:31)
Sep 7 01:16:53 kibana1 kibana[2090388]: at BasicAuth.handleAuthenticate (/usr/share/kibana/plugins/searchguard/server/applications/searchguard/auth/types/AuthType.js:496:28)
Sep 7 01:16:53 kibana1 kibana[2090388]: at /usr/share/kibana/plugins/searchguard/server/applications/searchguard/auth/types/basicauth/routes.js:66:24
Sep 7 01:16:53 kibana1 kibana[2090388]: at Router.handle (/usr/share/kibana/node_modules/@kbn/core-http-router-server-internal/target_node/src/router.js:163:30)
Sep 7 01:16:53 kibana1 kibana[2090388]: at handler (/usr/share/kibana/node_modules/@kbn/core-http-router-server-internal/target_node/src/router.js:124:50)
Sep 7 01:16:53 kibana1 kibana[2090388]: at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
Sep 7 01:16:53 kibana1 kibana[2090388]: at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
Sep 7 01:16:53 kibana1 kibana[2090388]: at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
Sep 7 01:16:53 kibana1 kibana[2090388]: at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)
Sep 7 01:16:53 kibana1 kibana[2090388]: at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9) {
Sep 7 01:16:53 kibana1 kibana[2090388]: meta: {
Sep 7 01:16:53 kibana1 kibana[2090388]: body: {
Sep 7 01:16:53 kibana1 kibana[2090388]: status: ‘INTERNAL_SERVER_ERROR’,
Sep 7 01:16:53 kibana1 kibana[2090388]: error: ‘SessionService is not configured’,
Sep 7 01:16:53 kibana1 kibana[2090388]: headers: {}
Sep 7 01:16:53 kibana1 kibana[2090388]: },
Sep 7 01:16:53 kibana1 kibana[2090388]: statusCode: 500,
Sep 7 01:16:53 kibana1 kibana[2090388]: headers: {
Sep 7 01:16:53 kibana1 kibana[2090388]: ‘x-opaque-id’: ‘ec3c1351-48c6-4a4c-bd07-d3e8ed4acdd6;kibana::searchguard-login:’,
Sep 7 01:16:53 kibana1 kibana[2090388]: ‘x-elastic-product’: ‘Elasticsearch’,
Sep 7 01:16:53 kibana1 kibana[2090388]: ‘content-type’: ‘application/json’,
Sep 7 01:16:53 kibana1 kibana[2090388]: ‘content-length’: ‘90’
Sep 7 01:16:53 kibana1 kibana[2090388]: },
Sep 7 01:16:53 kibana1 kibana[2090388]: meta: {
Sep 7 01:16:53 kibana1 kibana[2090388]: context: null,
Sep 7 01:16:53 kibana1 kibana[2090388]: request: [Object],
Sep 7 01:16:53 kibana1 kibana[2090388]: name: ‘elasticsearch-js’,
Sep 7 01:16:53 kibana1 kibana[2090388]: connection: [Object],
Sep 7 01:16:53 kibana1 kibana[2090388]: attempts: 0,
Sep 7 01:16:53 kibana1 kibana[2090388]: aborted: false
Sep 7 01:16:53 kibana1 kibana[2090388]: },
Sep 7 01:16:53 kibana1 kibana[2090388]: warnings: [Getter]
Sep 7 01:16:53 kibana1 kibana[2090388]: }
Sep 7 01:16:53 kibana1 kibana[2090388]: }
Sep 7 01:16:53 kibana1 kibana[2090388]: [2023-09-07T01:16:53.972+02:00][ERROR][plugins.searchguard.searchguard-auth] Basic auth login authorization ResponseError: {“status”:“INTERNAL_SERVER_ERROR”,“error”:“SessionService is not configured”,“headers”:{}}
Sep 7 01:16:53 kibana1 kibana[2090388]: at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)
Sep 7 01:16:53 kibana1 kibana[2090388]: at runMicrotasks ()
Sep 7 01:16:53 kibana1 kibana[2090388]: at processTicksAndRejections (node:internal/process/task_queues:96:5)
Sep 7 01:16:53 kibana1 kibana[2090388]: at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/src/create_transport.js:58:16)
Sep 7 01:16:53 kibana1 kibana[2090388]: at SearchGuardBackend._client (/usr/share/kibana/plugins/searchguard/server/applications/searchguard/backend/searchguard.js:32:20)
Sep 7 01:16:53 kibana1 kibana[2090388]: at SearchGuardBackend.authenticateWithSession (/usr/share/kibana/plugins/searchguard/server/applications/searchguard/backend/searchguard.js:70:24)
Sep 7 01:16:53 kibana1 kibana[2090388]: at BasicAuth.authenticate (/usr/share/kibana/plugins/searchguard/server/applications/searchguard/auth/types/AuthType.js:176:31)
Sep 7 01:16:53 kibana1 kibana[2090388]: at BasicAuth.handleAuthenticate (/usr/share/kibana/plugins/searchguard/server/applications/searchguard/auth/types/AuthType.js:496:28)
Sep 7 01:16:53 kibana1 kibana[2090388]: at /usr/share/kibana/plugins/searchguard/server/applications/searchguard/auth/types/basicauth/routes.js:66:24
Sep 7 01:16:53 kibana1 kibana[2090388]: at Router.handle (/usr/share/kibana/node_modules/@kbn/core-http-router-server-internal/target_node/src/router.js:163:30)
Sep 7 01:16:53 kibana1 kibana[2090388]: at handler (/usr/share/kibana/node_modules/@kbn/core-http-router-server-internal/target_node/src/router.js:124:50)
Sep 7 01:16:53 kibana1 kibana[2090388]: at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
Sep 7 01:16:53 kibana1 kibana[2090388]: at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
Sep 7 01:16:53 kibana1 kibana[2090388]: at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
Sep 7 01:16:53 kibana1 kibana[2090388]: at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)
Sep 7 01:16:53 kibana1 kibana[2090388]: at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)