Cannot retrieve roles for User [name=me@example.com]

Forked thread from Signals Watch getting "Failed to Fetch" - #5 by nils

I would guess that the user names provided by ADFS don’t completely match the user names used by LDAP.

Assuming that the user name from your example is me@example.com is coming from ADFS, it seems that ADFS/SAML provides the user name in the format of an email address. I don’t know so much about ADFS, but generally, the SAML protocol allows a selection of the user name format to use:

According to your sg_config.yml file, the LDAP user search is configured to use the LDAP attribute sAMAccountName whis is not necessarily in email address format.

As the authorization backends are executed independently of the authentication backend which created a user, the LDAP authorization backend is also queried when a user has authenticated by SAML. Thus, you get the information that there is no matching entry in LDAP for me@example.com.

Still, if things would work well, the error would have been without effect, as you would have got the roles via SAML. And thus, you would have got no “Unauthorized” error.

Can you post the whole stacktrace of this log entry plus maybe +/- 10 lines of context from the logs?

Unfortunately, that is correct. We use O365 for our ADFS, which gives them different domains. They would be members of the same backend groups, however, and for everything I’ve used them for, both the LDAP- and SAML-authenticated user ID’s have identical permissions (i.e., I can execute Elasticsearch admin tasks using my account, when authenticating directly to Elasticsearch via LDAP, or through Kibana, when authenticating via SAML).

This also makes sense. I have all the access I expect to have through SAML. But for some reason, with the update from the 7.9.2 stack to 7.10.2, with the corresponding SG update to 49.0.0 that you recommended to address another issue, I’ve started getting this error only for these two roles; I never observed this error prior to upgrading to 7.10.x, but then I also wasn’t starting to turn on alerts in Signals, so maybe I wouldn’t’ve seen it. Ordinarily, the only auth errors I get are if someone attempts to do something that their SAML roles don’t allow, and whenever the ELK stack authenticates with a user ID created using the built-in authentication (i.e., one of the default system accounts).

I may work with my ADFS team to resolve this issue, just to clean up the logs, however. But, since I don’t get this error for any other roles, it looks to me as if SG stops going down the list of authenticators (SAML->LDAP->Internal) as soon as it finds the role it’s looking for assigned to the user; based on what you’re saying, I suspect that even if the SAML/LDAP ID’s matched up perfectly, I’d still be getting this error.

As far as I can tell, these roles aren’t defined in my deployment. I’ve also checked the default config files from 49.0.0, and I can’t even see where these roles are defined. If they were, I’d assign them to the SAML group that we use for admin and see what happened. Could this be the issue - it’s looking for a role that’s not assigned, because it’s not defined anywhere?

Here’s the full error. As far as the context goes, this is all that’s being written to the log at the moment on the node that’s throwing this exception.

{"type": "server", "timestamp": "2021-02-12T14:20:26,168Z", "level": "ERROR", "component": "c.f.s.a.AuthorizationProcessor", "cluster.name": "elk-es", "node.name": "elk-es-coord-1", "message": "Cannot retrieve roles for User [name=me@example.com, backend_roles=[SG_ADMIN, SG_USER], requestedTenant=null] from ldap due to ElasticsearchSecurityException[ElasticsearchSecurityException[No user me@example.com found]]; nested: ElasticsearchSecurityException[No user me@example.com found];", "cluster.uuid": "816ooLAjQmWfG71KY0DSSA", "node.id": "q6a7e-zXRK-xY0zReSL8Tg" , 
"stacktrace": ["org.elasticsearch.ElasticsearchSecurityException: ElasticsearchSecurityException[No user me@example.com found]",
"at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.fillRoles(LDAPAuthorizationBackend.java:945) ~[dlic-search-guard-suite-security-7.10.2-49.0.0.jar:7.10.2-49.0.0]",
"at com.floragunn.searchguard.auth.api.SyncAuthorizationBackend.retrieveRoles(SyncAuthorizationBackend.java:44) [search-guard-suite-security-7.10.2-49.0.0.jar:7.10.2-49.0.0]",
"at com.floragunn.searchguard.auth.AuthorizationProcessor.checkNextAuthzDomain(AuthorizationProcessor.java:95) [search-guard-suite-security-7.10.2-49.0.0.jar:7.10.2-49.0.0]",
"at com.floragunn.searchguard.auth.AuthorizationProcessor.authz(AuthorizationProcessor.java:73) [search-guard-suite-security-7.10.2-49.0.0.jar:7.10.2-49.0.0]",
"at com.floragunn.searchguard.auth.RestAuthenticationProcessor.authz(RestAuthenticationProcessor.java:371) [search-guard-suite-security-7.10.2-49.0.0.jar:7.10.2-49.0.0]",
"at com.floragunn.searchguard.auth.RestAuthenticationProcessor.authzAndCache(RestAuthenticationProcessor.java:356) [search-guard-suite-security-7.10.2-49.0.0.jar:7.10.2-49.0.0]",
"at com.floragunn.searchguard.auth.RestAuthenticationProcessor.lambda$callAuthczBackends$4(RestAuthenticationProcessor.java:330) [search-guard-suite-security-7.10.2-49.0.0.jar:7.10.2-49.0.0]",
"at com.floragunn.searchguard.auth.api.SyncAuthenticationBackend.authenticate(SyncAuthenticationBackend.java:53) [search-guard-suite-security-7.10.2-49.0.0.jar:7.10.2-49.0.0]",
"at com.floragunn.searchguard.auth.RestAuthenticationProcessor.callAuthczBackends(RestAuthenticationProcessor.java:330) [search-guard-suite-security-7.10.2-49.0.0.jar:7.10.2-49.0.0]",
"at com.floragunn.searchguard.auth.RestAuthenticationProcessor.checkCurrentAuthenticationDomain(RestAuthenticationProcessor.java:240) [search-guard-suite-security-7.10.2-49.0.0.jar:7.10.2-49.0.0]",
"at com.floragunn.searchguard.auth.RestAuthenticationProcessor.checkNextAuthenticationDomains(RestAuthenticationProcessor.java:116) [search-guard-suite-security-7.10.2-49.0.0.jar:7.10.2-49.0.0]",
"at com.floragunn.searchguard.auth.RestAuthenticationProcessor.authenticate(RestAuthenticationProcessor.java:108) [search-guard-suite-security-7.10.2-49.0.0.jar:7.10.2-49.0.0]",
"at com.floragunn.searchguard.auth.BackendRegistry.authenticate(BackendRegistry.java:403) [search-guard-suite-security-7.10.2-49.0.0.jar:7.10.2-49.0.0]",
"at com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:86) [search-guard-suite-security-7.10.2-49.0.0.jar:7.10.2-49.0.0]",
"at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:258) [elasticsearch-7.10.2.jar:7.10.2]",
"at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:340) [elasticsearch-7.10.2.jar:7.10.2]",
"at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:191) [elasticsearch-7.10.2.jar:7.10.2]",
"at com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [search-guard-suite-security-7.10.2-49.0.0.jar:7.10.2-49.0.0]",
"at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:319) [elasticsearch-7.10.2.jar:7.10.2]",
"at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:384) [elasticsearch-7.10.2.jar:7.10.2]",
"at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:309) [elasticsearch-7.10.2.jar:7.10.2]",
"at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:42) [transport-netty4-client-7.10.2.jar:7.10.2]",
"at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:28) [transport-netty4-client-7.10.2.jar:7.10.2]",
"at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) [transport-netty4-client-7.10.2.jar:7.10.2]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518) [netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) [netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) [netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) [netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) [netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]",
"at java.lang.Thread.run(Thread.java:832) [?:?]",
"Caused by: org.elasticsearch.ElasticsearchSecurityException: No user me@example.com found",
"at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.fillRoles(LDAPAuthorizationBackend.java:747) ~[dlic-search-guard-suite-security-7.10.2-49.0.0.jar:7.10.2-49.0.0]",
"... 76 more"] }

SG 49 brought some changes to the authcz process which might have also introduced this behavioural change you are observing. We need to investigate this a bit more.

For the time being, it might be possible to use a workaround:

If the user names coming from SAML are structurally easy to tell apart from the user names coming from from LDAP, you could use the skip_users config option. For example, if SAML user names are always email addresses containing a @ and if LDAP user names never contain a @, the you can extend your configuration this way:

  authz:
      adfs_ldap_roles:
        http_enabled: true
        transport_enabled: true
        skip_users:
        - '*@*'
        authorization_backend:
          type: 'ldap'
          config:
            enable_ssl: true
            enable_start_tls: false
            enable_ssl_client_auth: false
            verify_hostnames: false
            pemtrustedcas_filepath: '/usr/share/elasticsearch/config/certs/app/ca_bundle.pem'
            hosts:
              - '${env.SG_LDAP_HOST}'
            bind_dn: '${env.SG_LDAP_BIND_DN}'
            password: '${env.SG_LDAP_BIND_DN_PASSWD}'
            roles:
              athene:
                base: '${env.SG_LDAP_ROLE_BASE}'
                search: '(member={1})'
            userroleattribute: null
            userrolename: 'memberOf'
            rolename: 'cn'
            resolve_nested_roles: true
            users:
              athene:
                base: '${env.SG_LDAP_USER_BASE}'
                search: '(sAMAccountName={0})'
            username_attribute: 'cn'
        description: 'LDAP Roles'

See the docs for details:

I guess that what I’d really like to figure out is why these roles aren’t being assigned via SAML authentication which, as far as I can understand, they should be.

I think, I slightly lost the overview here, sorry!

So, it is about these roles SG_ADMIN and SG_USER you mentioned earlier, right?

Also, if I understand it correctly, the user is authenticated via SAML.

This means that the roles probably come from the assertions in the SAML response. These roles are called backend roles, referring to the IdP as authcz backend.

To give these roles an actual meaning, they need to be specified in the roles mapping configuration. In the roles mapping, which you have provided earlier, I however can’t spot these backend roles.

There are however a couple of env variables which might contain the names of the backend roles:

So, you would need to check:

  • Is the IdP providing SAML supposed to send the roles SG_USER and SG_ADMIN?
  • If no, what roles should the IdP provide instead?
  • If yes, make sure that the backend roles SG_USER and SG_ADMIN are mapped to Search Guard roles in the roles mapping configuration.

Yes. These are the roles that are showing up as missing.

Also correct.

Did a deeper dive. Yes - IDP sends those roles. They are mapped as follows:

  • SG_USER: App-ELK-Users AD group
  • SG_ADMIN: App-ELK-Admin AD group

They’re mapped as follows:

  • SG_USER:
    • SGS_KIBANA_USER - Required for access to Kibana.
    • TLD_BASIC_ACCESS - Gives access to view indices, etc.
  • SG_ADMIN: SGS_ALL_ACCESS - This is the admin role, correct?

Again - with this mapping, I have all the permissions I expect to have. And my user seems to be the only one that is generating these errors.

OK…since I finally located where SG_ADMIN/SG_USER are being used and I understand better what’s going on now, I went ahead and added the skip_users: to the LDAP role config. That has resolved this issue. I was also getting similar errors at startup for the internal users (admin, etc.) and added them as well.

If you can turn on the “Solutions” flag for this forum as well, I’ll mark it as solved.

Thanks!

The “Solution” flag should be already active for this category. Is the button missing for you here?

Yes. Please see below.

image

Oh, this is because the “Solution” button is only available for the topic owner and moderators. As I have forked out this thread, I am the topic owner. Thus, you don’t see the button.

I’ll mark the answer as solution then.

1 Like