SearchGuard 5.x with LDAP authentication issues

I am testing out the LDAP authentication module with Kibana multi tenancy to see if this will fit our needs, and I am having some strange login issues. I believe I have configured it as described in the example documentation, but the FIRST time I attempt to log in I always end up at the 500 internal server error. If I attempt to log in a few more times, eventually it works and seems to keep working for a bit. I have also logged in and then seen this issue when switching tenants. After logging back in a second time everything works including switching.

I don’t really know how to proceed troubleshooting this, but I can’t demo this to anyone and expect them to want me to take this live to production if LDAP is flaky.

I am still using the basic_internal_auth, since I don’t want to add a kibanaserver or logstash user to my LDAP directory, and would prefer those just be in the local database. Authz is being done by mapping users to rolls, which does work once I am logged in.

Here is the relevant section from my sg_config.yml

searchguard:

dynamic:

kibana:

multitenancy_enabled: true

server_username: “kibanaserver”

index: ‘.kibana’

http:

anonymous_auth_enabled: false

xff:

enabled: false

remoteIpHeader: ‘x-forwarded-for’

proxiesHeader: ‘x-forwarded-by’

authc:

basic_internal_auth_domain:

enabled: true

order: 4

http_authenticator:

type: basic

challenge: false

authentication_backend:

type: intern

ldap:

enabled: true

order: 1

http_authenticator:

type: basic

challenge: true

authentication_backend:

config:

enable_ssl: false

enable_start_tls: true

enable_ssl_client_auth: false

verify_hostnames: false

hosts:

bind_dn: ‘uid=admin,cn=users,cn=accounts,dc=example,dc=com’

password: ‘FAKEPASS’

userbase: ‘cn=users,cn=accounts,dc=example,dc=com’

usersearch: ‘(uid={0})’

username_attribute: uid

skip_users:

  • kibanaserver

  • logstash

Which ES/SG/LDAP/MT versions are you using?

Does it work without LDAP (internal users only)?

···

Am 10.07.2017 um 16:56 schrieb Brian Sanders <brian.sanders@gmail.com>:

I am testing out the LDAP authentication module with Kibana multi tenancy to see if this will fit our needs, and I am having some strange login issues. I believe I have configured it as described in the example documentation, but the FIRST time I attempt to log in I always end up at the 500 internal server error. If I attempt to log in a few more times, eventually it works and seems to keep working for a bit. I have also logged in and then seen this issue when switching tenants. After logging back in a second time everything works including switching.

I don't really know how to proceed troubleshooting this, but I can't demo this to anyone and expect them to want me to take this live to production if LDAP is flaky.

I am still using the basic_internal_auth, since I don't want to add a kibanaserver or logstash user to my LDAP directory, and would prefer those just be in the local database. Authz is being done by mapping users to rolls, which does work once I am logged in.

Here is the relevant section from my sg_config.yml

searchguard:
  dynamic:
    kibana:
      multitenancy_enabled: true
      server_username: "kibanaserver"
      index: '.kibana'
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
        remoteIpHeader: 'x-forwarded-for'
        proxiesHeader: 'x-forwarded-by'
    authc:
      basic_internal_auth_domain:
        enabled: true
        order: 4
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: intern
      ldap:
        enabled: true
        order: 1
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          config:
            enable_ssl: false
            enable_start_tls: true
            enable_ssl_client_auth: false
            verify_hostnames: false
            hosts:
              - ipa1.example.com:389
            bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
            password: 'FAKEPASS'
            userbase: 'cn=users,cn=accounts,dc=example,dc=com'
            usersearch: '(uid={0})'
            username_attribute: uid
            skip_users:
              - kibanaserver
              - logstash

--
You received this message because you are subscribed to the Google Groups "Search Guard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/9eba0f76-cbbe-4ea7-85f2-a2a90d750f26%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Sorry I just saw this question, I never had the issue with the local only accounts. I didn’t test that extensively as I was looking to test the LDAP, but it seemed that it worked just fine.

We are running the current versions:

ES: 5.4.2
kibana-multitenancy-5.4-3
authbackend-ldap-5.0-7

And after looking these up it would appear there is now a 5.5 version of elasticsearch. This being in dev, I can always attempt to upgrade, but these were all the latest available versions as of sometime last week when I was starting my build.

···

On Monday, July 10, 2017 at 1:51:24 PM UTC-4, Search Guard wrote:

Which ES/SG/LDAP/MT versions are you using?

Does it work without LDAP (internal users only)?

Am 10.07.2017 um 16:56 schrieb Brian Sanders brian....@gmail.com:

I am testing out the LDAP authentication module with Kibana multi tenancy to see if this will fit our needs, and I am having some strange login issues. I believe I have configured it as described in the example documentation, but the FIRST time I attempt to log in I always end up at the 500 internal server error. If I attempt to log in a few more times, eventually it works and seems to keep working for a bit. I have also logged in and then seen this issue when switching tenants. After logging back in a second time everything works including switching.

I don’t really know how to proceed troubleshooting this, but I can’t demo this to anyone and expect them to want me to take this live to production if LDAP is flaky.

I am still using the basic_internal_auth, since I don’t want to add a kibanaserver or logstash user to my LDAP directory, and would prefer those just be in the local database. Authz is being done by mapping users to rolls, which does work once I am logged in.

Here is the relevant section from my sg_config.yml

searchguard:

dynamic:

kibana:
  multitenancy_enabled: true
  server_username: "kibanaserver"
  index: '.kibana'
http:
  anonymous_auth_enabled: false
  xff:
    enabled: false
    remoteIpHeader:  'x-forwarded-for'
    proxiesHeader:   'x-forwarded-by'
authc:
  basic_internal_auth_domain:
    enabled: true
    order: 4
    http_authenticator:
      type: basic
      challenge: false
    authentication_backend:
      type: intern
  ldap:
    enabled: true
    order: 1
    http_authenticator:
      type: basic
      challenge: true
    authentication_backend:
      config:
        enable_ssl: false
        enable_start_tls: true
        enable_ssl_client_auth: false
        verify_hostnames: false
        hosts:
          - [ipa1.example.com:389](http://ipa1.example.com:389)
        bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
        password: 'FAKEPASS'
        userbase: 'cn=users,cn=accounts,dc=example,dc=com'
        usersearch: '(uid={0})'
        username_attribute: uid
        skip_users:
          - kibanaserver
          - logstash


You received this message because you are subscribed to the Google Groups “Search Guard” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/9eba0f76-cbbe-4ea7-85f2-a2a90d750f26%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Seems I didn’t include the Search Guard version: search-guard-5:5.4.2-12

···

On Monday, July 10, 2017 at 3:55:25 PM UTC-4, Brian Sanders wrote:

Sorry I just saw this question, I never had the issue with the local only accounts. I didn’t test that extensively as I was looking to test the LDAP, but it seemed that it worked just fine.

We are running the current versions:

ES: 5.4.2
kibana-multitenancy-5.4-3
authbackend-ldap-5.0-7

And after looking these up it would appear there is now a 5.5 version of elasticsearch. This being in dev, I can always attempt to upgrade, but these were all the latest available versions as of sometime last week when I was starting my build.

On Monday, July 10, 2017 at 1:51:24 PM UTC-4, Search Guard wrote:

Which ES/SG/LDAP/MT versions are you using?

Does it work without LDAP (internal users only)?

Am 10.07.2017 um 16:56 schrieb Brian Sanders brian....@gmail.com:

I am testing out the LDAP authentication module with Kibana multi tenancy to see if this will fit our needs, and I am having some strange login issues. I believe I have configured it as described in the example documentation, but the FIRST time I attempt to log in I always end up at the 500 internal server error. If I attempt to log in a few more times, eventually it works and seems to keep working for a bit. I have also logged in and then seen this issue when switching tenants. After logging back in a second time everything works including switching.

I don’t really know how to proceed troubleshooting this, but I can’t demo this to anyone and expect them to want me to take this live to production if LDAP is flaky.

I am still using the basic_internal_auth, since I don’t want to add a kibanaserver or logstash user to my LDAP directory, and would prefer those just be in the local database. Authz is being done by mapping users to rolls, which does work once I am logged in.

Here is the relevant section from my sg_config.yml

searchguard:

dynamic:

kibana:
  multitenancy_enabled: true
  server_username: "kibanaserver"
  index: '.kibana'
http:
  anonymous_auth_enabled: false
  xff:
    enabled: false
    remoteIpHeader:  'x-forwarded-for'
    proxiesHeader:   'x-forwarded-by'
authc:
  basic_internal_auth_domain:
    enabled: true
    order: 4
    http_authenticator:
      type: basic
      challenge: false
    authentication_backend:
      type: intern
  ldap:
    enabled: true
    order: 1
    http_authenticator:
      type: basic
      challenge: true
    authentication_backend:
      config:
        enable_ssl: false
        enable_start_tls: true
        enable_ssl_client_auth: false
        verify_hostnames: false
        hosts:
          - [ipa1.example.com:389](http://ipa1.example.com:389)
        bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
        password: 'FAKEPASS'
        userbase: 'cn=users,cn=accounts,dc=example,dc=com'
        usersearch: '(uid={0})'
        username_attribute: uid
        skip_users:
          - kibanaserver
          - logstash


You received this message because you are subscribed to the Google Groups “Search Guard” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/9eba0f76-cbbe-4ea7-85f2-a2a90d750f26%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

can you try this?

searchguard:

dynamic:

kibana:

multitenancy_enabled: true

server_username: “kibanaserver”

index: ‘.kibana’

http:

anonymous_auth_enabled: false

xff:

enabled: false

remoteIpHeader: ‘x-forwarded-for’

proxiesHeader: ‘x-forwarded-by’

authc:

basic_internal_auth_domain:

enabled: true

order: 0

http_authenticator:

type: basic

challenge: false

authentication_backend:

type: intern

ldap:

enabled: true

order: 1

http_authenticator:

type: basic

challenge: false

authentication_backend:

config:

enable_ssl: false

enable_start_tls: true

enable_ssl_client_auth: false

verify_hostnames: false

hosts:

bind_dn: ‘uid=admin,cn=users,cn=accounts,dc=example,dc=com’

password: ‘FAKEPASS’

userbase: ‘cn=users,cn=accounts,dc=example,dc=com’

usersearch: ‘(uid={0})’

username_attribute: uid

#only valid for authz (not authc)

#skip_users:

- kibanaserver

- logstash

(more complete example here https://gist.github.com/floragunncom/fdad5c8f1cab838962842d590c408538)

···

On Monday, 10 July 2017 21:56:25 UTC+2, Brian Sanders wrote:

Sorry I just saw this question, I never had the issue with the local only accounts. I didn’t test that extensively as I was looking to test the LDAP, but it seemed that it worked just fine.

We are running the current versions:

ES: 5.4.2
kibana-multitenancy-5.4-3
authbackend-ldap-5.0-7

And after looking these up it would appear there is now a 5.5 version of elasticsearch. This being in dev, I can always attempt to upgrade, but these were all the latest available versions as of sometime last week when I was starting my build.

On Monday, July 10, 2017 at 1:51:24 PM UTC-4, Search Guard wrote:

Which ES/SG/LDAP/MT versions are you using?

Does it work without LDAP (internal users only)?

Am 10.07.2017 um 16:56 schrieb Brian Sanders brian....@gmail.com:

I am testing out the LDAP authentication module with Kibana multi tenancy to see if this will fit our needs, and I am having some strange login issues. I believe I have configured it as described in the example documentation, but the FIRST time I attempt to log in I always end up at the 500 internal server error. If I attempt to log in a few more times, eventually it works and seems to keep working for a bit. I have also logged in and then seen this issue when switching tenants. After logging back in a second time everything works including switching.

I don’t really know how to proceed troubleshooting this, but I can’t demo this to anyone and expect them to want me to take this live to production if LDAP is flaky.

I am still using the basic_internal_auth, since I don’t want to add a kibanaserver or logstash user to my LDAP directory, and would prefer those just be in the local database. Authz is being done by mapping users to rolls, which does work once I am logged in.

Here is the relevant section from my sg_config.yml

searchguard:

dynamic:

kibana:
  multitenancy_enabled: true
  server_username: "kibanaserver"
  index: '.kibana'
http:
  anonymous_auth_enabled: false
  xff:
    enabled: false
    remoteIpHeader:  'x-forwarded-for'
    proxiesHeader:   'x-forwarded-by'
authc:
  basic_internal_auth_domain:
    enabled: true
    order: 4
    http_authenticator:
      type: basic
      challenge: false
    authentication_backend:
      type: intern
  ldap:
    enabled: true
    order: 1
    http_authenticator:
      type: basic
      challenge: true
    authentication_backend:
      config:
        enable_ssl: false
        enable_start_tls: true
        enable_ssl_client_auth: false
        verify_hostnames: false
        hosts:
          - [ipa1.example.com:389](http://ipa1.example.com:389)
        bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
        password: 'FAKEPASS'
        userbase: 'cn=users,cn=accounts,dc=example,dc=com'
        usersearch: '(uid={0})'
        username_attribute: uid
        skip_users:
          - kibanaserver
          - logstash


You received this message because you are subscribed to the Google Groups “Search Guard” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/9eba0f76-cbbe-4ea7-85f2-a2a90d750f26%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Seems I didn’t include the Search Guard version: search-guard-5:5.4.2-12

On Monday, July 10, 2017 at 3:55:25 PM UTC-4, Brian Sanders wrote:

This is a configuration I have tried previously, but I tried again this morning before logging in to anything to document the exact order of tests and errors seen.

When I first log in, I do see the log message that it can not authenticate the user with ad 0 (the internal database) and it says it says it is going to the next (eg “try next”) However after an extended period the attempted login just shows

{"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"}

``

After that shows up, I simply reselect my URL and press enter again, which this time loads Kibana with out any further prompting for user input. I can perform searches on my data an things appear normal. If I select tenants tab, and change to another tenant, I receive the same:

{"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"}

``

This time pressing enter to reload the page eventually brings me back to the Search Guard login screen. After providing my ldap credentials, I am finally authentiacted and put into Kibana as the tenant. Now I can switch between the 2 tenants just fine… but if I select yet another tenant it takes a long time and eventually puts me at the Search Guard login website, at least it didn’t 500 error this time.

So it would seem that first login does this, as well as having issues with switching to a tenant for the first time as well. I am using FreeIPA as an ldap backend, which is a fairly popular user management group of packages, but as far as this is concerned, it is simply authing against the LDAP portion which clearly CAN work. It just doesn’t seem to work on the first try.

···

On Tuesday, July 11, 2017 at 4:01:37 PM UTC-4, Search Guard wrote:

can you try this?

searchguard:

dynamic:

kibana:

multitenancy_enabled: true

server_username: “kibanaserver”

index: ‘.kibana’

http:

anonymous_auth_enabled: false

xff:

enabled: false

remoteIpHeader: ‘x-forwarded-for’

proxiesHeader: ‘x-forwarded-by’

authc:

basic_internal_auth_domain:

enabled: true

order: 0

http_authenticator:

type: basic

challenge: false

authentication_backend:

type: intern

ldap:

enabled: true

order: 1

http_authenticator:

type: basic

challenge: false

authentication_backend:

config:

enable_ssl: false

enable_start_tls: true

enable_ssl_client_auth: false

verify_hostnames: false

hosts:

bind_dn: ‘uid=admin,cn=users,cn=accounts,dc=example,dc=com’

password: ‘FAKEPASS’

userbase: ‘cn=users,cn=accounts,dc=example,dc=com’

usersearch: ‘(uid={0})’

username_attribute: uid

#only valid for authz (not authc)

#skip_users:

- kibanaserver

- logstash

(more complete example here https://gist.github.com/floragunncom/fdad5c8f1cab838962842d590c408538)

On Monday, 10 July 2017 21:56:25 UTC+2, Brian Sanders wrote:

Sorry I just saw this question, I never had the issue with the local only accounts. I didn’t test that extensively as I was looking to test the LDAP, but it seemed that it worked just fine.

We are running the current versions:

ES: 5.4.2
kibana-multitenancy-5.4-3
authbackend-ldap-5.0-7

And after looking these up it would appear there is now a 5.5 version of elasticsearch. This being in dev, I can always attempt to upgrade, but these were all the latest available versions as of sometime last week when I was starting my build.

On Monday, July 10, 2017 at 1:51:24 PM UTC-4, Search Guard wrote:

Which ES/SG/LDAP/MT versions are you using?

Does it work without LDAP (internal users only)?

Am 10.07.2017 um 16:56 schrieb Brian Sanders brian....@gmail.com:

I am testing out the LDAP authentication module with Kibana multi tenancy to see if this will fit our needs, and I am having some strange login issues. I believe I have configured it as described in the example documentation, but the FIRST time I attempt to log in I always end up at the 500 internal server error. If I attempt to log in a few more times, eventually it works and seems to keep working for a bit. I have also logged in and then seen this issue when switching tenants. After logging back in a second time everything works including switching.

I don’t really know how to proceed troubleshooting this, but I can’t demo this to anyone and expect them to want me to take this live to production if LDAP is flaky.

I am still using the basic_internal_auth, since I don’t want to add a kibanaserver or logstash user to my LDAP directory, and would prefer those just be in the local database. Authz is being done by mapping users to rolls, which does work once I am logged in.

Here is the relevant section from my sg_config.yml

searchguard:

dynamic:

kibana:
  multitenancy_enabled: true
  server_username: "kibanaserver"
  index: '.kibana'
http:
  anonymous_auth_enabled: false
  xff:
    enabled: false
    remoteIpHeader:  'x-forwarded-for'
    proxiesHeader:   'x-forwarded-by'
authc:
  basic_internal_auth_domain:
    enabled: true
    order: 4
    http_authenticator:
      type: basic
      challenge: false
    authentication_backend:
      type: intern
  ldap:
    enabled: true
    order: 1
    http_authenticator:
      type: basic
      challenge: true
    authentication_backend:
      config:
        enable_ssl: false
        enable_start_tls: true
        enable_ssl_client_auth: false
        verify_hostnames: false
        hosts:
          - [ipa1.example.com:389](http://ipa1.example.com:389)
        bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
        password: 'FAKEPASS'
        userbase: 'cn=users,cn=accounts,dc=example,dc=com'
        usersearch: '(uid={0})'
        username_attribute: uid
        skip_users:
          - kibanaserver
          - logstash


You received this message because you are subscribed to the Google Groups “Search Guard” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/9eba0f76-cbbe-4ea7-85f2-a2a90d750f26%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Seems I didn’t include the Search Guard version: search-guard-5:5.4.2-12

On Monday, July 10, 2017 at 3:55:25 PM UTC-4, Brian Sanders wrote:

I verified that the configuration i sent was is working locally for me.
I suggest to reinstall Kibana and the SG Kibana plugin https://github.com/floragunncom/search-guard-kibana-plugin/releases/download/v5.4.2-3/searchguard-kibana-5.4.2-3.zip

The relevant config in kibana.yml (maybe you miss that) looks like:

elasticsearch.requestHeadersWhitelist: [“sg_tenant”, “x-authenticated-user”, “authorization”, “x-forwarded-for”, “x-forwarded-server”, “x-forwarded-by”, “x-proxy-user”, “x-proxy-roles”]

searchguard.multitenancy.enabled: true

searchguard.multitenancy.show_roles: true

searchguard.multitenancy.enable_filter: true

searchguard.multitenancy.tenants.enable_global: true

searchguard.multitenancy.tenants.enable_private: true

searchguard.cookie.password: “”

elasticsearch.username: “kibanaserver”

elasticsearch.password: “kibanaserver”

see https://github.com/floragunncom/search-guard-docs/blob/master/multitenancy.md#kibana-plugin-configuration

···

On Wednesday, 12 July 2017 15:34:52 UTC+2, Brian Sanders wrote:

This is a configuration I have tried previously, but I tried again this morning before logging in to anything to document the exact order of tests and errors seen.

When I first log in, I do see the log message that it can not authenticate the user with ad 0 (the internal database) and it says it says it is going to the next (eg “try next”) However after an extended period the attempted login just shows

{"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"}

``

After that shows up, I simply reselect my URL and press enter again, which this time loads Kibana with out any further prompting for user input. I can perform searches on my data an things appear normal. If I select tenants tab, and change to another tenant, I receive the same:

{"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"}

``

This time pressing enter to reload the page eventually brings me back to the Search Guard login screen. After providing my ldap credentials, I am finally authentiacted and put into Kibana as the tenant. Now I can switch between the 2 tenants just fine… but if I select yet another tenant it takes a long time and eventually puts me at the Search Guard login website, at least it didn’t 500 error this time.

So it would seem that first login does this, as well as having issues with switching to a tenant for the first time as well. I am using FreeIPA as an ldap backend, which is a fairly popular user management group of packages, but as far as this is concerned, it is simply authing against the LDAP portion which clearly CAN work. It just doesn’t seem to work on the first try.

On Tuesday, July 11, 2017 at 4:01:37 PM UTC-4, Search Guard wrote:

can you try this?

searchguard:

dynamic:

kibana:

multitenancy_enabled: true

server_username: “kibanaserver”

index: ‘.kibana’

http:

anonymous_auth_enabled: false

xff:

enabled: false

remoteIpHeader: ‘x-forwarded-for’

proxiesHeader: ‘x-forwarded-by’

authc:

basic_internal_auth_domain:

enabled: true

order: 0

http_authenticator:

type: basic

challenge: false

authentication_backend:

type: intern

ldap:

enabled: true

order: 1

http_authenticator:

type: basic

challenge: false

authentication_backend:

config:

enable_ssl: false

enable_start_tls: true

enable_ssl_client_auth: false

verify_hostnames: false

hosts:

bind_dn: ‘uid=admin,cn=users,cn=accounts,dc=example,dc=com’

password: ‘FAKEPASS’

userbase: ‘cn=users,cn=accounts,dc=example,dc=com’

usersearch: ‘(uid={0})’

username_attribute: uid

#only valid for authz (not authc)

#skip_users:

- kibanaserver

- logstash

(more complete example here https://gist.github.com/floragunncom/fdad5c8f1cab838962842d590c408538)

On Monday, 10 July 2017 21:56:25 UTC+2, Brian Sanders wrote:

Sorry I just saw this question, I never had the issue with the local only accounts. I didn’t test that extensively as I was looking to test the LDAP, but it seemed that it worked just fine.

We are running the current versions:

ES: 5.4.2
kibana-multitenancy-5.4-3
authbackend-ldap-5.0-7

And after looking these up it would appear there is now a 5.5 version of elasticsearch. This being in dev, I can always attempt to upgrade, but these were all the latest available versions as of sometime last week when I was starting my build.

On Monday, July 10, 2017 at 1:51:24 PM UTC-4, Search Guard wrote:

Which ES/SG/LDAP/MT versions are you using?

Does it work without LDAP (internal users only)?

Am 10.07.2017 um 16:56 schrieb Brian Sanders brian....@gmail.com:

I am testing out the LDAP authentication module with Kibana multi tenancy to see if this will fit our needs, and I am having some strange login issues. I believe I have configured it as described in the example documentation, but the FIRST time I attempt to log in I always end up at the 500 internal server error. If I attempt to log in a few more times, eventually it works and seems to keep working for a bit. I have also logged in and then seen this issue when switching tenants. After logging back in a second time everything works including switching.

I don’t really know how to proceed troubleshooting this, but I can’t demo this to anyone and expect them to want me to take this live to production if LDAP is flaky.

I am still using the basic_internal_auth, since I don’t want to add a kibanaserver or logstash user to my LDAP directory, and would prefer those just be in the local database. Authz is being done by mapping users to rolls, which does work once I am logged in.

Here is the relevant section from my sg_config.yml

searchguard:

dynamic:

kibana:
  multitenancy_enabled: true
  server_username: "kibanaserver"
  index: '.kibana'
http:
  anonymous_auth_enabled: false
  xff:
    enabled: false
    remoteIpHeader:  'x-forwarded-for'
    proxiesHeader:   'x-forwarded-by'
authc:
  basic_internal_auth_domain:
    enabled: true
    order: 4
    http_authenticator:
      type: basic
      challenge: false
    authentication_backend:
      type: intern
  ldap:
    enabled: true
    order: 1
    http_authenticator:
      type: basic
      challenge: true
    authentication_backend:
      config:
        enable_ssl: false
        enable_start_tls: true
        enable_ssl_client_auth: false
        verify_hostnames: false
        hosts:
          - [ipa1.example.com:389](http://ipa1.example.com:389)
        bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
        password: 'FAKEPASS'
        userbase: 'cn=users,cn=accounts,dc=example,dc=com'
        usersearch: '(uid={0})'
        username_attribute: uid
        skip_users:
          - kibanaserver
          - logstash


You received this message because you are subscribed to the Google Groups “Search Guard” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/9eba0f76-cbbe-4ea7-85f2-a2a90d750f26%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Seems I didn’t include the Search Guard version: search-guard-5:5.4.2-12

On Monday, July 10, 2017 at 3:55:25 PM UTC-4, Brian Sanders wrote:

I’ll have to try reinstalling, perhaps I’ll upgrade to the most current versions of plugins as well since this isn’t live and there are new versions.

From your suggested settings in kibana.yml

  1. what is searchguard.cookie.password I haven’t seen mention of that before

I have the other settings, and some of those are defaults so I omitted them. I will specificaly add them and see if this makes much difference before trying to re-install.

···

On Wednesday, July 12, 2017 at 10:04:28 AM UTC-4, Search Guard wrote:

I verified that the configuration i sent was is working locally for me.
I suggest to reinstall Kibana and the SG Kibana plugin https://github.com/floragunncom/search-guard-kibana-plugin/releases/download/v5.4.2-3/searchguard-kibana-5.4.2-3.zip

The relevant config in kibana.yml (maybe you miss that) looks like:

elasticsearch.requestHeadersWhitelist: [“sg_tenant”, “x-authenticated-user”, “authorization”, “x-forwarded-for”, “x-forwarded-server”, “x-forwarded-by”, “x-proxy-user”, “x-proxy-roles”]

searchguard.multitenancy.enabled: true

searchguard.multitenancy.show_roles: true

searchguard.multitenancy.enable_filter: true

searchguard.multitenancy.tenants.enable_global: true

searchguard.multitenancy.tenants.enable_private: true

searchguard.cookie.password: “”

elasticsearch.username: “kibanaserver”

elasticsearch.password: “kibanaserver”

see https://github.com/floragunncom/search-guard-docs/blob/master/multitenancy.md#kibana-plugin-configuration

On Wednesday, 12 July 2017 15:34:52 UTC+2, Brian Sanders wrote:

This is a configuration I have tried previously, but I tried again this morning before logging in to anything to document the exact order of tests and errors seen.

When I first log in, I do see the log message that it can not authenticate the user with ad 0 (the internal database) and it says it says it is going to the next (eg “try next”) However after an extended period the attempted login just shows

{"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"}

``

After that shows up, I simply reselect my URL and press enter again, which this time loads Kibana with out any further prompting for user input. I can perform searches on my data an things appear normal. If I select tenants tab, and change to another tenant, I receive the same:

{"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"}

``

This time pressing enter to reload the page eventually brings me back to the Search Guard login screen. After providing my ldap credentials, I am finally authentiacted and put into Kibana as the tenant. Now I can switch between the 2 tenants just fine… but if I select yet another tenant it takes a long time and eventually puts me at the Search Guard login website, at least it didn’t 500 error this time.

So it would seem that first login does this, as well as having issues with switching to a tenant for the first time as well. I am using FreeIPA as an ldap backend, which is a fairly popular user management group of packages, but as far as this is concerned, it is simply authing against the LDAP portion which clearly CAN work. It just doesn’t seem to work on the first try.

On Tuesday, July 11, 2017 at 4:01:37 PM UTC-4, Search Guard wrote:

can you try this?

searchguard:

dynamic:

kibana:

multitenancy_enabled: true

server_username: “kibanaserver”

index: ‘.kibana’

http:

anonymous_auth_enabled: false

xff:

enabled: false

remoteIpHeader: ‘x-forwarded-for’

proxiesHeader: ‘x-forwarded-by’

authc:

basic_internal_auth_domain:

enabled: true

order: 0

http_authenticator:

type: basic

challenge: false

authentication_backend:

type: intern

ldap:

enabled: true

order: 1

http_authenticator:

type: basic

challenge: false

authentication_backend:

config:

enable_ssl: false

enable_start_tls: true

enable_ssl_client_auth: false

verify_hostnames: false

hosts:

bind_dn: ‘uid=admin,cn=users,cn=accounts,dc=example,dc=com’

password: ‘FAKEPASS’

userbase: ‘cn=users,cn=accounts,dc=example,dc=com’

usersearch: ‘(uid={0})’

username_attribute: uid

#only valid for authz (not authc)

#skip_users:

- kibanaserver

- logstash

(more complete example here https://gist.github.com/floragunncom/fdad5c8f1cab838962842d590c408538)

On Monday, 10 July 2017 21:56:25 UTC+2, Brian Sanders wrote:

Sorry I just saw this question, I never had the issue with the local only accounts. I didn’t test that extensively as I was looking to test the LDAP, but it seemed that it worked just fine.

We are running the current versions:

ES: 5.4.2
kibana-multitenancy-5.4-3
authbackend-ldap-5.0-7

And after looking these up it would appear there is now a 5.5 version of elasticsearch. This being in dev, I can always attempt to upgrade, but these were all the latest available versions as of sometime last week when I was starting my build.

On Monday, July 10, 2017 at 1:51:24 PM UTC-4, Search Guard wrote:

Which ES/SG/LDAP/MT versions are you using?

Does it work without LDAP (internal users only)?

Am 10.07.2017 um 16:56 schrieb Brian Sanders brian....@gmail.com:

I am testing out the LDAP authentication module with Kibana multi tenancy to see if this will fit our needs, and I am having some strange login issues. I believe I have configured it as described in the example documentation, but the FIRST time I attempt to log in I always end up at the 500 internal server error. If I attempt to log in a few more times, eventually it works and seems to keep working for a bit. I have also logged in and then seen this issue when switching tenants. After logging back in a second time everything works including switching.

I don’t really know how to proceed troubleshooting this, but I can’t demo this to anyone and expect them to want me to take this live to production if LDAP is flaky.

I am still using the basic_internal_auth, since I don’t want to add a kibanaserver or logstash user to my LDAP directory, and would prefer those just be in the local database. Authz is being done by mapping users to rolls, which does work once I am logged in.

Here is the relevant section from my sg_config.yml

searchguard:

dynamic:

kibana:
  multitenancy_enabled: true
  server_username: "kibanaserver"
  index: '.kibana'
http:
  anonymous_auth_enabled: false
  xff:
    enabled: false
    remoteIpHeader:  'x-forwarded-for'
    proxiesHeader:   'x-forwarded-by'
authc:
  basic_internal_auth_domain:
    enabled: true
    order: 4
    http_authenticator:
      type: basic
      challenge: false
    authentication_backend:
      type: intern
  ldap:
    enabled: true
    order: 1
    http_authenticator:
      type: basic
      challenge: true
    authentication_backend:
      config:
        enable_ssl: false
        enable_start_tls: true
        enable_ssl_client_auth: false
        verify_hostnames: false
        hosts:
          - [ipa1.example.com:389](http://ipa1.example.com:389)
        bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
        password: 'FAKEPASS'
        userbase: 'cn=users,cn=accounts,dc=example,dc=com'
        usersearch: '(uid={0})'
        username_attribute: uid
        skip_users:
          - kibanaserver
          - logstash


You received this message because you are subscribed to the Google Groups “Search Guard” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/9eba0f76-cbbe-4ea7-85f2-a2a90d750f26%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Seems I didn’t include the Search Guard version: search-guard-5:5.4.2-12

On Monday, July 10, 2017 at 3:55:25 PM UTC-4, Brian Sanders wrote:

Regarding "searchguard.cookie.password" pls read https://github.com/floragunncom/search-guard-docs/blob/master/kibana.md#configuring-the-search-guard-kibana-plugin
(at least in production you should not use the default)

If its still not working pls. turn debug logging on and send the logfile from elasticsearch:

PUT /_cluster/settings
{
    "transient" : {
        "logger._root" : "DEBUG"
    }
}

···

Am 12.07.2017 um 16:39 schrieb Brian Sanders <brian.sanders@gmail.com>:

I'll have to try reinstalling, perhaps I'll upgrade to the most current versions of plugins as well since this isn't live and there are new versions.

From your suggested settings in kibana.yml
1) what is searchguard.cookie.password I haven't seen mention of that before

I have the other settings, and some of those are defaults so I omitted them. I will specificaly add them and see if this makes much difference before trying to re-install.

On Wednesday, July 12, 2017 at 10:04:28 AM UTC-4, Search Guard wrote:
I verified that the configuration i sent was is working locally for me.
I suggest to reinstall Kibana and the SG Kibana plugin https://github.com/floragunncom/search-guard-kibana-plugin/releases/download/v5.4.2-3/searchguard-kibana-5.4.2-3.zip

The relevant config in kibana.yml (maybe you miss that) looks like:

elasticsearch.requestHeadersWhitelist: ["sg_tenant", "x-authenticated-user", "authorization", "x-forwarded-for", "x-forwarded-server", "x-forwarded-by", "x-proxy-user", "x-proxy-roles"]
searchguard.multitenancy.enabled: true
searchguard.multitenancy.show_roles: true
searchguard.multitenancy.enable_filter: true
searchguard.multitenancy.tenants.enable_global: true
searchguard.multitenancy.tenants.enable_private: true
searchguard.cookie.password: "<long secure string>"
elasticsearch.username: "kibanaserver"
elasticsearch.password: "kibanaserver"

see https://github.com/floragunncom/search-guard-docs/blob/master/multitenancy.md#kibana-plugin-configuration

On Wednesday, 12 July 2017 15:34:52 UTC+2, Brian Sanders wrote:
This is a configuration I have tried previously, but I tried again this morning before logging in to anything to document the exact order of tests and errors seen.

When I first log in, I do see the log message that it can not authenticate the user with ad 0 (the internal database) and it says it says it is going to the next (eg "try next") However after an extended period the attempted login just shows
{"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"}

After that shows up, I simply reselect my URL and press enter again, which this time loads Kibana with out any further prompting for user input. I can perform searches on my data an things appear normal. If I select tenants tab, and change to another tenant, I receive the same:
{"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"}

This time pressing enter to reload the page eventually brings me back to the Search Guard login screen. After providing my ldap credentials, I am finally authentiacted and put into Kibana as the tenant. Now I can switch between the 2 tenants just fine... but if I select yet another tenant it takes a long time and eventually puts me at the Search Guard login website, at least it didn't 500 error this time.

So it would seem that first login does this, as well as having issues with switching to a tenant for the first time as well. I am using FreeIPA as an ldap backend, which is a fairly popular user management group of packages, but as far as this is concerned, it is simply authing against the LDAP portion which clearly CAN work. It just doesn't seem to work on the first try.

On Tuesday, July 11, 2017 at 4:01:37 PM UTC-4, Search Guard wrote:
can you try this?

searchguard:
  dynamic:
    kibana:
      multitenancy_enabled: true
      server_username: "kibanaserver"
      index: '.kibana'
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
        remoteIpHeader: 'x-forwarded-for'
        proxiesHeader: 'x-forwarded-by'
    authc:
      basic_internal_auth_domain:
        enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: intern
      ldap:
        enabled: true
        order: 1
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          config:
            enable_ssl: false
            enable_start_tls: true
            enable_ssl_client_auth: false
            verify_hostnames: false
            hosts:
              - ipa1.example.com:389
            bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
            password: 'FAKEPASS'
            userbase: 'cn=users,cn=accounts,dc=example,dc=com'
            usersearch: '(uid={0})'
            username_attribute: uid
            
            #only valid for authz (not authc)
            #skip_users:
            # - kibanaserver
            # - logstash

(more complete example here https://gist.github.com/floragunncom/fdad5c8f1cab838962842d590c408538)

On Monday, 10 July 2017 21:56:25 UTC+2, Brian Sanders wrote:
Seems I didn't include the Search Guard version: search-guard-5:5.4.2-12

On Monday, July 10, 2017 at 3:55:25 PM UTC-4, Brian Sanders wrote:
Sorry I just saw this question, I never had the issue with the local only accounts. I didn't test that extensively as I was looking to test the LDAP, but it seemed that it worked just fine.

We are running the current versions:

ES: 5.4.2
kibana-multitenancy-5.4-3
authbackend-ldap-5.0-7

And after looking these up it would appear there is now a 5.5 version of elasticsearch. This being in dev, I can always attempt to upgrade, but these were all the latest available versions as of sometime last week when I was starting my build.

On Monday, July 10, 2017 at 1:51:24 PM UTC-4, Search Guard wrote:
Which ES/SG/LDAP/MT versions are you using?

Does it work without LDAP (internal users only)?

> Am 10.07.2017 um 16:56 schrieb Brian Sanders <brian....@gmail.com>:
>
> I am testing out the LDAP authentication module with Kibana multi tenancy to see if this will fit our needs, and I am having some strange login issues. I believe I have configured it as described in the example documentation, but the FIRST time I attempt to log in I always end up at the 500 internal server error. If I attempt to log in a few more times, eventually it works and seems to keep working for a bit. I have also logged in and then seen this issue when switching tenants. After logging back in a second time everything works including switching.
>
> I don't really know how to proceed troubleshooting this, but I can't demo this to anyone and expect them to want me to take this live to production if LDAP is flaky.
>
> I am still using the basic_internal_auth, since I don't want to add a kibanaserver or logstash user to my LDAP directory, and would prefer those just be in the local database. Authz is being done by mapping users to rolls, which does work once I am logged in.
>
> Here is the relevant section from my sg_config.yml
>
>
> searchguard:
> dynamic:
> kibana:
> multitenancy_enabled: true
> server_username: "kibanaserver"
> index: '.kibana'
> http:
> anonymous_auth_enabled: false
> xff:
> enabled: false
> remoteIpHeader: 'x-forwarded-for'
> proxiesHeader: 'x-forwarded-by'
> authc:
> basic_internal_auth_domain:
> enabled: true
> order: 4
> http_authenticator:
> type: basic
> challenge: false
> authentication_backend:
> type: intern
> ldap:
> enabled: true
> order: 1
> http_authenticator:
> type: basic
> challenge: true
> authentication_backend:
> config:
> enable_ssl: false
> enable_start_tls: true
> enable_ssl_client_auth: false
> verify_hostnames: false
> hosts:
> - ipa1.example.com:389
> bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
> password: 'FAKEPASS'
> userbase: 'cn=users,cn=accounts,dc=example,dc=com'
> usersearch: '(uid={0})'
> username_attribute: uid
> skip_users:
> - kibanaserver
> - logstash
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups "Search Guard" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/9eba0f76-cbbe-4ea7-85f2-a2a90d750f26%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Search Guard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/d487734a-1beb-4a13-93d1-4686a32fc203%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Thank you, I will activate this logging and see what I can get. I tried all the settings above with no difference. I also tried Firefox (was using chrome) with the hope that it was browser related, but it still does the same thing. I’ll see if I c

···

On Wednesday, July 12, 2017 at 11:39:57 AM UTC-4, Search Guard wrote:

Regarding “searchguard.cookie.password” pls read https://github.com/floragunncom/search-guard-docs/blob/master/kibana.md#configuring-the-search-guard-kibana-plugin

(at least in production you should not use the default)

If its still not working pls. turn debug logging on and send the logfile from elasticsearch:

PUT /_cluster/settings

{

"transient" : {

    "logger._root" : "DEBUG"

}

}

Am 12.07.2017 um 16:39 schrieb Brian Sanders brian....@gmail.com:

I’ll have to try reinstalling, perhaps I’ll upgrade to the most current versions of plugins as well since this isn’t live and there are new versions.

From your suggested settings in kibana.yml

  1. what is searchguard.cookie.password I haven’t seen mention of that before

I have the other settings, and some of those are defaults so I omitted them. I will specificaly add them and see if this makes much difference before trying to re-install.

On Wednesday, July 12, 2017 at 10:04:28 AM UTC-4, Search Guard wrote:

I verified that the configuration i sent was is working locally for me.

I suggest to reinstall Kibana and the SG Kibana plugin https://github.com/floragunncom/search-guard-kibana-plugin/releases/download/v5.4.2-3/searchguard-kibana-5.4.2-3.zip

The relevant config in kibana.yml (maybe you miss that) looks like:

elasticsearch.requestHeadersWhitelist: [“sg_tenant”, “x-authenticated-user”, “authorization”, “x-forwarded-for”, “x-forwarded-server”, “x-forwarded-by”, “x-proxy-user”, “x-proxy-roles”]

searchguard.multitenancy.enabled: true

searchguard.multitenancy.show_roles: true

searchguard.multitenancy.enable_filter: true

searchguard.multitenancy.tenants.enable_global: true

searchguard.multitenancy.tenants.enable_private: true

searchguard.cookie.password: “”

elasticsearch.username: “kibanaserver”

elasticsearch.password: “kibanaserver”

see https://github.com/floragunncom/search-guard-docs/blob/master/multitenancy.md#kibana-plugin-configuration

On Wednesday, 12 July 2017 15:34:52 UTC+2, Brian Sanders wrote:

This is a configuration I have tried previously, but I tried again this morning before logging in to anything to document the exact order of tests and errors seen.

When I first log in, I do see the log message that it can not authenticate the user with ad 0 (the internal database) and it says it says it is going to the next (eg “try next”) However after an extended period the attempted login just shows

{“statusCode”:500,“error”:“Internal Server Error”,“message”:“An internal server error occurred”}

After that shows up, I simply reselect my URL and press enter again, which this time loads Kibana with out any further prompting for user input. I can perform searches on my data an things appear normal. If I select tenants tab, and change to another tenant, I receive the same:

{“statusCode”:500,“error”:“Internal Server Error”,“message”:“An internal server error occurred”}

This time pressing enter to reload the page eventually brings me back to the Search Guard login screen. After providing my ldap credentials, I am finally authentiacted and put into Kibana as the tenant. Now I can switch between the 2 tenants just fine… but if I select yet another tenant it takes a long time and eventually puts me at the Search Guard login website, at least it didn’t 500 error this time.

So it would seem that first login does this, as well as having issues with switching to a tenant for the first time as well. I am using FreeIPA as an ldap backend, which is a fairly popular user management group of packages, but as far as this is concerned, it is simply authing against the LDAP portion which clearly CAN work. It just doesn’t seem to work on the first try.

On Tuesday, July 11, 2017 at 4:01:37 PM UTC-4, Search Guard wrote:

can you try this?

searchguard:

dynamic:

kibana:
  multitenancy_enabled: true
  server_username: "kibanaserver"
  index: '.kibana'
http:
  anonymous_auth_enabled: false
  xff:
    enabled: false
    remoteIpHeader:  'x-forwarded-for'
    proxiesHeader:   'x-forwarded-by'
authc:
  basic_internal_auth_domain:
    enabled: true
    order: 0
    http_authenticator:
      type: basic
      challenge: false
    authentication_backend:
      type: intern
  ldap:
    enabled: true
    order: 1
    http_authenticator:
      type: basic
      challenge: false
    authentication_backend:
      config:
        enable_ssl: false
        enable_start_tls: true
        enable_ssl_client_auth: false
        verify_hostnames: false
        hosts:
          - [ipa1.example.com:389](http://ipa1.example.com:389)
        bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
        password: 'FAKEPASS'
        userbase: 'cn=users,cn=accounts,dc=example,dc=com'
        usersearch: '(uid={0})'
        username_attribute: uid
        #only valid for authz (not authc)
        #skip_users:
        #  - kibanaserver
        #  - logstash

(more complete example here https://gist.github.com/floragunncom/fdad5c8f1cab838962842d590c408538)

On Monday, 10 July 2017 21:56:25 UTC+2, Brian Sanders wrote:

Seems I didn’t include the Search Guard version: search-guard-5:5.4.2-12

On Monday, July 10, 2017 at 3:55:25 PM UTC-4, Brian Sanders wrote:

Sorry I just saw this question, I never had the issue with the local only accounts. I didn’t test that extensively as I was looking to test the LDAP, but it seemed that it worked just fine.

We are running the current versions:

ES: 5.4.2

kibana-multitenancy-5.4-3

authbackend-ldap-5.0-7

And after looking these up it would appear there is now a 5.5 version of elasticsearch. This being in dev, I can always attempt to upgrade, but these were all the latest available versions as of sometime last week when I was starting my build.

On Monday, July 10, 2017 at 1:51:24 PM UTC-4, Search Guard wrote:

Which ES/SG/LDAP/MT versions are you using?

Does it work without LDAP (internal users only)?

Am 10.07.2017 um 16:56 schrieb Brian Sanders brian....@gmail.com:

I am testing out the LDAP authentication module with Kibana multi tenancy to see if this will fit our needs, and I am having some strange login issues. I believe I have configured it as described in the example documentation, but the FIRST time I attempt to log in I always end up at the 500 internal server error. If I attempt to log in a few more times, eventually it works and seems to keep working for a bit. I have also logged in and then seen this issue when switching tenants. After logging back in a second time everything works including switching.

I don’t really know how to proceed troubleshooting this, but I can’t demo this to anyone and expect them to want me to take this live to production if LDAP is flaky.

I am still using the basic_internal_auth, since I don’t want to add a kibanaserver or logstash user to my LDAP directory, and would prefer those just be in the local database. Authz is being done by mapping users to rolls, which does work once I am logged in.

Here is the relevant section from my sg_config.yml

searchguard:
dynamic:
kibana:
multitenancy_enabled: true
server_username: “kibanaserver”
index: ‘.kibana’
http:
anonymous_auth_enabled: false
xff:
enabled: false
remoteIpHeader: ‘x-forwarded-for’
proxiesHeader: ‘x-forwarded-by’
authc:
basic_internal_auth_domain:
enabled: true
order: 4
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: intern
ldap:
enabled: true
order: 1
http_authenticator:
type: basic
challenge: true
authentication_backend:
config:
enable_ssl: false
enable_start_tls: true
enable_ssl_client_auth: false
verify_hostnames: false
hosts:
- ipa1.example.com:389

        bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
        password: 'FAKEPASS'
        userbase: 'cn=users,cn=accounts,dc=example,dc=com'
        usersearch: '(uid={0})'
        username_attribute: uid
        skip_users:
          - kibanaserver
          - logstash


You received this message because you are subscribed to the Google Groups “Search Guard” group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/9eba0f76-cbbe-4ea7-85f2-a2a90d750f26%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups “Search Guard” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/d487734a-1beb-4a13-93d1-4686a32fc203%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

I have a log file, is there some way to send it other than attaching it here? I don’t know exactly what all is in it, but I generally don’t like posting debug logs to the internet directly.

···

On Thursday, July 13, 2017 at 9:24:56 AM UTC-4, Brian Sanders wrote:

Thank you, I will activate this logging and see what I can get. I tried all the settings above with no difference. I also tried Firefox (was using chrome) with the hope that it was browser related, but it still does the same thing. I’ll see if I c

On Wednesday, July 12, 2017 at 11:39:57 AM UTC-4, Search Guard wrote:

Regarding “searchguard.cookie.password” pls read https://github.com/floragunncom/search-guard-docs/blob/master/kibana.md#configuring-the-search-guard-kibana-plugin

(at least in production you should not use the default)

If its still not working pls. turn debug logging on and send the logfile from elasticsearch:

PUT /_cluster/settings

{

"transient" : {

    "logger._root" : "DEBUG"

}

}

Am 12.07.2017 um 16:39 schrieb Brian Sanders brian....@gmail.com:

I’ll have to try reinstalling, perhaps I’ll upgrade to the most current versions of plugins as well since this isn’t live and there are new versions.

From your suggested settings in kibana.yml

  1. what is searchguard.cookie.password I haven’t seen mention of that before

I have the other settings, and some of those are defaults so I omitted them. I will specificaly add them and see if this makes much difference before trying to re-install.

On Wednesday, July 12, 2017 at 10:04:28 AM UTC-4, Search Guard wrote:

I verified that the configuration i sent was is working locally for me.

I suggest to reinstall Kibana and the SG Kibana plugin https://github.com/floragunncom/search-guard-kibana-plugin/releases/download/v5.4.2-3/searchguard-kibana-5.4.2-3.zip

The relevant config in kibana.yml (maybe you miss that) looks like:

elasticsearch.requestHeadersWhitelist: [“sg_tenant”, “x-authenticated-user”, “authorization”, “x-forwarded-for”, “x-forwarded-server”, “x-forwarded-by”, “x-proxy-user”, “x-proxy-roles”]

searchguard.multitenancy.enabled: true

searchguard.multitenancy.show_roles: true

searchguard.multitenancy.enable_filter: true

searchguard.multitenancy.tenants.enable_global: true

searchguard.multitenancy.tenants.enable_private: true

searchguard.cookie.password: “”

elasticsearch.username: “kibanaserver”

elasticsearch.password: “kibanaserver”

see https://github.com/floragunncom/search-guard-docs/blob/master/multitenancy.md#kibana-plugin-configuration

On Wednesday, 12 July 2017 15:34:52 UTC+2, Brian Sanders wrote:

This is a configuration I have tried previously, but I tried again this morning before logging in to anything to document the exact order of tests and errors seen.

When I first log in, I do see the log message that it can not authenticate the user with ad 0 (the internal database) and it says it says it is going to the next (eg “try next”) However after an extended period the attempted login just shows

{“statusCode”:500,“error”:“Internal Server Error”,“message”:“An internal server error occurred”}

After that shows up, I simply reselect my URL and press enter again, which this time loads Kibana with out any further prompting for user input. I can perform searches on my data an things appear normal. If I select tenants tab, and change to another tenant, I receive the same:

{“statusCode”:500,“error”:“Internal Server Error”,“message”:“An internal server error occurred”}

This time pressing enter to reload the page eventually brings me back to the Search Guard login screen. After providing my ldap credentials, I am finally authentiacted and put into Kibana as the tenant. Now I can switch between the 2 tenants just fine… but if I select yet another tenant it takes a long time and eventually puts me at the Search Guard login website, at least it didn’t 500 error this time.

So it would seem that first login does this, as well as having issues with switching to a tenant for the first time as well. I am using FreeIPA as an ldap backend, which is a fairly popular user management group of packages, but as far as this is concerned, it is simply authing against the LDAP portion which clearly CAN work. It just doesn’t seem to work on the first try.

On Tuesday, July 11, 2017 at 4:01:37 PM UTC-4, Search Guard wrote:

can you try this?

searchguard:

dynamic:

kibana:
  multitenancy_enabled: true
  server_username: "kibanaserver"
  index: '.kibana'
http:
  anonymous_auth_enabled: false
  xff:
    enabled: false
    remoteIpHeader:  'x-forwarded-for'
    proxiesHeader:   'x-forwarded-by'
authc:
  basic_internal_auth_domain:
    enabled: true
    order: 0
    http_authenticator:
      type: basic
      challenge: false
    authentication_backend:
      type: intern
  ldap:
    enabled: true
    order: 1
    http_authenticator:
      type: basic
      challenge: false
    authentication_backend:
      config:
        enable_ssl: false
        enable_start_tls: true
        enable_ssl_client_auth: false
        verify_hostnames: false
        hosts:
          - [ipa1.example.com:389](http://ipa1.example.com:389)
        bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
        password: 'FAKEPASS'
        userbase: 'cn=users,cn=accounts,dc=example,dc=com'
        usersearch: '(uid={0})'
        username_attribute: uid
        #only valid for authz (not authc)
        #skip_users:
        #  - kibanaserver
        #  - logstash

(more complete example here https://gist.github.com/floragunncom/fdad5c8f1cab838962842d590c408538)

On Monday, 10 July 2017 21:56:25 UTC+2, Brian Sanders wrote:

Seems I didn’t include the Search Guard version: search-guard-5:5.4.2-12

On Monday, July 10, 2017 at 3:55:25 PM UTC-4, Brian Sanders wrote:

Sorry I just saw this question, I never had the issue with the local only accounts. I didn’t test that extensively as I was looking to test the LDAP, but it seemed that it worked just fine.

We are running the current versions:

ES: 5.4.2

kibana-multitenancy-5.4-3

authbackend-ldap-5.0-7

And after looking these up it would appear there is now a 5.5 version of elasticsearch. This being in dev, I can always attempt to upgrade, but these were all the latest available versions as of sometime last week when I was starting my build.

On Monday, July 10, 2017 at 1:51:24 PM UTC-4, Search Guard wrote:

Which ES/SG/LDAP/MT versions are you using?

Does it work without LDAP (internal users only)?

Am 10.07.2017 um 16:56 schrieb Brian Sanders brian....@gmail.com:

I am testing out the LDAP authentication module with Kibana multi tenancy to see if this will fit our needs, and I am having some strange login issues. I believe I have configured it as described in the example documentation, but the FIRST time I attempt to log in I always end up at the 500 internal server error. If I attempt to log in a few more times, eventually it works and seems to keep working for a bit. I have also logged in and then seen this issue when switching tenants. After logging back in a second time everything works including switching.

I don’t really know how to proceed troubleshooting this, but I can’t demo this to anyone and expect them to want me to take this live to production if LDAP is flaky.

I am still using the basic_internal_auth, since I don’t want to add a kibanaserver or logstash user to my LDAP directory, and would prefer those just be in the local database. Authz is being done by mapping users to rolls, which does work once I am logged in.

Here is the relevant section from my sg_config.yml

searchguard:
dynamic:
kibana:
multitenancy_enabled: true
server_username: “kibanaserver”
index: ‘.kibana’
http:
anonymous_auth_enabled: false
xff:
enabled: false
remoteIpHeader: ‘x-forwarded-for’
proxiesHeader: ‘x-forwarded-by’
authc:
basic_internal_auth_domain:
enabled: true
order: 4
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: intern
ldap:
enabled: true
order: 1
http_authenticator:
type: basic
challenge: true
authentication_backend:
config:
enable_ssl: false
enable_start_tls: true
enable_ssl_client_auth: false
verify_hostnames: false
hosts:
- ipa1.example.com:389

        bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
        password: 'FAKEPASS'
        userbase: 'cn=users,cn=accounts,dc=example,dc=com'
        usersearch: '(uid={0})'
        username_attribute: uid
        skip_users:
          - kibanaserver
          - logstash


You received this message because you are subscribed to the Google Groups “Search Guard” group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/9eba0f76-cbbe-4ea7-85f2-a2a90d750f26%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups “Search Guard” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/d487734a-1beb-4a13-93d1-4686a32fc203%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Why not write a PGP encrypted mail? Here is our public key https://floragunn.com/search-guard-public-key/

···

Am 13.07.2017 um 16:10 schrieb Brian Sanders <brian.sanders@gmail.com>:

I have a log file, is there some way to send it other than attaching it here? I don't know exactly what all is in it, but I generally don't like posting debug logs to the internet directly.

On Thursday, July 13, 2017 at 9:24:56 AM UTC-4, Brian Sanders wrote:
Thank you, I will activate this logging and see what I can get. I tried all the settings above with no difference. I also tried Firefox (was using chrome) with the hope that it was browser related, but it still does the same thing. I'll see if I c

On Wednesday, July 12, 2017 at 11:39:57 AM UTC-4, Search Guard wrote:
Regarding "searchguard.cookie.password" pls read https://github.com/floragunncom/search-guard-docs/blob/master/kibana.md#configuring-the-search-guard-kibana-plugin
(at least in production you should not use the default)

If its still not working pls. turn debug logging on and send the logfile from elasticsearch:

PUT /_cluster/settings
{
    "transient" : {
        "logger._root" : "DEBUG"
    }
}

> Am 12.07.2017 um 16:39 schrieb Brian Sanders <brian....@gmail.com>:
>
> I'll have to try reinstalling, perhaps I'll upgrade to the most current versions of plugins as well since this isn't live and there are new versions.
>
> From your suggested settings in kibana.yml
> 1) what is searchguard.cookie.password I haven't seen mention of that before
>
> I have the other settings, and some of those are defaults so I omitted them. I will specificaly add them and see if this makes much difference before trying to re-install.
>
>
> On Wednesday, July 12, 2017 at 10:04:28 AM UTC-4, Search Guard wrote:
> I verified that the configuration i sent was is working locally for me.
> I suggest to reinstall Kibana and the SG Kibana plugin https://github.com/floragunncom/search-guard-kibana-plugin/releases/download/v5.4.2-3/searchguard-kibana-5.4.2-3.zip
>
> The relevant config in kibana.yml (maybe you miss that) looks like:
>
> elasticsearch.requestHeadersWhitelist: ["sg_tenant", "x-authenticated-user", "authorization", "x-forwarded-for", "x-forwarded-server", "x-forwarded-by", "x-proxy-user", "x-proxy-roles"]
> searchguard.multitenancy.enabled: true
> searchguard.multitenancy.show_roles: true
> searchguard.multitenancy.enable_filter: true
> searchguard.multitenancy.tenants.enable_global: true
> searchguard.multitenancy.tenants.enable_private: true
> searchguard.cookie.password: "<long secure string>"
> elasticsearch.username: "kibanaserver"
> elasticsearch.password: "kibanaserver"
>
> see https://github.com/floragunncom/search-guard-docs/blob/master/multitenancy.md#kibana-plugin-configuration
>
> On Wednesday, 12 July 2017 15:34:52 UTC+2, Brian Sanders wrote:
> This is a configuration I have tried previously, but I tried again this morning before logging in to anything to document the exact order of tests and errors seen.
>
> When I first log in, I do see the log message that it can not authenticate the user with ad 0 (the internal database) and it says it says it is going to the next (eg "try next") However after an extended period the attempted login just shows
> {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"}
>
> After that shows up, I simply reselect my URL and press enter again, which this time loads Kibana with out any further prompting for user input. I can perform searches on my data an things appear normal. If I select tenants tab, and change to another tenant, I receive the same:
> {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"}
>
> This time pressing enter to reload the page eventually brings me back to the Search Guard login screen. After providing my ldap credentials, I am finally authentiacted and put into Kibana as the tenant. Now I can switch between the 2 tenants just fine... but if I select yet another tenant it takes a long time and eventually puts me at the Search Guard login website, at least it didn't 500 error this time.
>
> So it would seem that first login does this, as well as having issues with switching to a tenant for the first time as well. I am using FreeIPA as an ldap backend, which is a fairly popular user management group of packages, but as far as this is concerned, it is simply authing against the LDAP portion which clearly CAN work. It just doesn't seem to work on the first try.
>
>
>
> On Tuesday, July 11, 2017 at 4:01:37 PM UTC-4, Search Guard wrote:
> can you try this?
>
> searchguard:
> dynamic:
> kibana:
> multitenancy_enabled: true
> server_username: "kibanaserver"
> index: '.kibana'
> http:
> anonymous_auth_enabled: false
> xff:
> enabled: false
> remoteIpHeader: 'x-forwarded-for'
> proxiesHeader: 'x-forwarded-by'
> authc:
> basic_internal_auth_domain:
> enabled: true
> order: 0
> http_authenticator:
> type: basic
> challenge: false
> authentication_backend:
> type: intern
> ldap:
> enabled: true
> order: 1
> http_authenticator:
> type: basic
> challenge: false
> authentication_backend:
> config:
> enable_ssl: false
> enable_start_tls: true
> enable_ssl_client_auth: false
> verify_hostnames: false
> hosts:
> - ipa1.example.com:389
> bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
> password: 'FAKEPASS'
> userbase: 'cn=users,cn=accounts,dc=example,dc=com'
> usersearch: '(uid={0})'
> username_attribute: uid
>
> #only valid for authz (not authc)
> #skip_users:
> # - kibanaserver
> # - logstash
>
> (more complete example here https://gist.github.com/floragunncom/fdad5c8f1cab838962842d590c408538)
>
> On Monday, 10 July 2017 21:56:25 UTC+2, Brian Sanders wrote:
> Seems I didn't include the Search Guard version: search-guard-5:5.4.2-12
>
> On Monday, July 10, 2017 at 3:55:25 PM UTC-4, Brian Sanders wrote:
> Sorry I just saw this question, I never had the issue with the local only accounts. I didn't test that extensively as I was looking to test the LDAP, but it seemed that it worked just fine.
>
> We are running the current versions:
>
> ES: 5.4.2
> kibana-multitenancy-5.4-3
> authbackend-ldap-5.0-7
>
> And after looking these up it would appear there is now a 5.5 version of elasticsearch. This being in dev, I can always attempt to upgrade, but these were all the latest available versions as of sometime last week when I was starting my build.
>
>
> On Monday, July 10, 2017 at 1:51:24 PM UTC-4, Search Guard wrote:
> Which ES/SG/LDAP/MT versions are you using?
>
> Does it work without LDAP (internal users only)?
>
> > Am 10.07.2017 um 16:56 schrieb Brian Sanders <brian....@gmail.com>:
> >
> > I am testing out the LDAP authentication module with Kibana multi tenancy to see if this will fit our needs, and I am having some strange login issues. I believe I have configured it as described in the example documentation, but the FIRST time I attempt to log in I always end up at the 500 internal server error. If I attempt to log in a few more times, eventually it works and seems to keep working for a bit. I have also logged in and then seen this issue when switching tenants. After logging back in a second time everything works including switching.
> >
> > I don't really know how to proceed troubleshooting this, but I can't demo this to anyone and expect them to want me to take this live to production if LDAP is flaky.
> >
> > I am still using the basic_internal_auth, since I don't want to add a kibanaserver or logstash user to my LDAP directory, and would prefer those just be in the local database. Authz is being done by mapping users to rolls, which does work once I am logged in.
> >
> > Here is the relevant section from my sg_config.yml
> >
> >
> > searchguard:
> > dynamic:
> > kibana:
> > multitenancy_enabled: true
> > server_username: "kibanaserver"
> > index: '.kibana'
> > http:
> > anonymous_auth_enabled: false
> > xff:
> > enabled: false
> > remoteIpHeader: 'x-forwarded-for'
> > proxiesHeader: 'x-forwarded-by'
> > authc:
> > basic_internal_auth_domain:
> > enabled: true
> > order: 4
> > http_authenticator:
> > type: basic
> > challenge: false
> > authentication_backend:
> > type: intern
> > ldap:
> > enabled: true
> > order: 1
> > http_authenticator:
> > type: basic
> > challenge: true
> > authentication_backend:
> > config:
> > enable_ssl: false
> > enable_start_tls: true
> > enable_ssl_client_auth: false
> > verify_hostnames: false
> > hosts:
> > - ipa1.example.com:389
> > bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
> > password: 'FAKEPASS'
> > userbase: 'cn=users,cn=accounts,dc=example,dc=com'
> > usersearch: '(uid={0})'
> > username_attribute: uid
> > skip_users:
> > - kibanaserver
> > - logstash
> >
> >
> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups "Search Guard" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> > To post to this group, send email to search...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/9eba0f76-cbbe-4ea7-85f2-a2a90d750f26%40googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "Search Guard" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/d487734a-1beb-4a13-93d1-4686a32fc203%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Search Guard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/4c3d66b4-bdd4-4616-93a4-7fe9ea926f6c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Yup, that is a good suggestion. Log file attached.

If it helps, my users that fails then succeeds is “brians”, and I have the global tenant turned off but the private tenant turned on. I have also added a few other tenants to his user to test switching.

debuglog.gpg (27.4 KB)

···

On Thursday, July 13, 2017 at 10:40:57 AM UTC-4, Search Guard wrote:

Why not write a PGP encrypted mail? Here is our public key https://floragunn.com/search-guard-public-key/

Am 13.07.2017 um 16:10 schrieb Brian Sanders brian....@gmail.com:

I have a log file, is there some way to send it other than attaching it here? I don’t know exactly what all is in it, but I generally don’t like posting debug logs to the internet directly.

On Thursday, July 13, 2017 at 9:24:56 AM UTC-4, Brian Sanders wrote:

Thank you, I will activate this logging and see what I can get. I tried all the settings above with no difference. I also tried Firefox (was using chrome) with the hope that it was browser related, but it still does the same thing. I’ll see if I c

On Wednesday, July 12, 2017 at 11:39:57 AM UTC-4, Search Guard wrote:

Regarding “searchguard.cookie.password” pls read https://github.com/floragunncom/search-guard-docs/blob/master/kibana.md#configuring-the-search-guard-kibana-plugin

(at least in production you should not use the default)

If its still not working pls. turn debug logging on and send the logfile from elasticsearch:

PUT /_cluster/settings
{
“transient” : {
“logger._root” : “DEBUG”
}
}

Am 12.07.2017 um 16:39 schrieb Brian Sanders brian....@gmail.com:

I’ll have to try reinstalling, perhaps I’ll upgrade to the most current versions of plugins as well since this isn’t live and there are new versions.

From your suggested settings in kibana.yml

  1. what is searchguard.cookie.password I haven’t seen mention of that before

I have the other settings, and some of those are defaults so I omitted them. I will specificaly add them and see if this makes much difference before trying to re-install.

On Wednesday, July 12, 2017 at 10:04:28 AM UTC-4, Search Guard wrote:
I verified that the configuration i sent was is working locally for me.
I suggest to reinstall Kibana and the SG Kibana plugin https://github.com/floragunncom/search-guard-kibana-plugin/releases/download/v5.4.2-3/searchguard-kibana-5.4.2-3.zip

The relevant config in kibana.yml (maybe you miss that) looks like:

elasticsearch.requestHeadersWhitelist: [“sg_tenant”, “x-authenticated-user”, “authorization”, “x-forwarded-for”, “x-forwarded-server”, “x-forwarded-by”, “x-proxy-user”, “x-proxy-roles”]
searchguard.multitenancy.enabled: true
searchguard.multitenancy.show_roles: true
searchguard.multitenancy.enable_filter: true
searchguard.multitenancy.tenants.enable_global: true
searchguard.multitenancy.tenants.enable_private: true
searchguard.cookie.password: “”
elasticsearch.username: “kibanaserver”
elasticsearch.password: “kibanaserver”

see https://github.com/floragunncom/search-guard-docs/blob/master/multitenancy.md#kibana-plugin-configuration

On Wednesday, 12 July 2017 15:34:52 UTC+2, Brian Sanders wrote:
This is a configuration I have tried previously, but I tried again this morning before logging in to anything to document the exact order of tests and errors seen.

When I first log in, I do see the log message that it can not authenticate the user with ad 0 (the internal database) and it says it says it is going to the next (eg “try next”) However after an extended period the attempted login just shows
{“statusCode”:500,“error”:“Internal Server Error”,“message”:“An internal server error occurred”}

After that shows up, I simply reselect my URL and press enter again, which this time loads Kibana with out any further prompting for user input. I can perform searches on my data an things appear normal. If I select tenants tab, and change to another tenant, I receive the same:
{“statusCode”:500,“error”:“Internal Server Error”,“message”:“An internal server error occurred”}

This time pressing enter to reload the page eventually brings me back to the Search Guard login screen. After providing my ldap credentials, I am finally authentiacted and put into Kibana as the tenant. Now I can switch between the 2 tenants just fine… but if I select yet another tenant it takes a long time and eventually puts me at the Search Guard login website, at least it didn’t 500 error this time.

So it would seem that first login does this, as well as having issues with switching to a tenant for the first time as well. I am using FreeIPA as an ldap backend, which is a fairly popular user management group of packages, but as far as this is concerned, it is simply authing against the LDAP portion which clearly CAN work. It just doesn’t seem to work on the first try.

On Tuesday, July 11, 2017 at 4:01:37 PM UTC-4, Search Guard wrote:
can you try this?

searchguard:
dynamic:
kibana:
multitenancy_enabled: true
server_username: “kibanaserver”
index: ‘.kibana’
http:
anonymous_auth_enabled: false
xff:
enabled: false
remoteIpHeader: ‘x-forwarded-for’
proxiesHeader: ‘x-forwarded-by’
authc:
basic_internal_auth_domain:
enabled: true
order: 0
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: intern
ldap:
enabled: true
order: 1
http_authenticator:
type: basic
challenge: false
authentication_backend:
config:
enable_ssl: false
enable_start_tls: true
enable_ssl_client_auth: false
verify_hostnames: false
hosts:
- ipa1.example.com:389

        bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
        password: 'FAKEPASS'
        userbase: 'cn=users,cn=accounts,dc=example,dc=com'
        usersearch: '(uid={0})'
        username_attribute: uid
       
        #only valid for authz (not authc)
        #skip_users:
        #  - kibanaserver
        #  - logstash

(more complete example here https://gist.github.com/floragunncom/fdad5c8f1cab838962842d590c408538)

On Monday, 10 July 2017 21:56:25 UTC+2, Brian Sanders wrote:
Seems I didn’t include the Search Guard version: search-guard-5:5.4.2-12

On Monday, July 10, 2017 at 3:55:25 PM UTC-4, Brian Sanders wrote:
Sorry I just saw this question, I never had the issue with the local only accounts. I didn’t test that extensively as I was looking to test the LDAP, but it seemed that it worked just fine.

We are running the current versions:

ES: 5.4.2
kibana-multitenancy-5.4-3
authbackend-ldap-5.0-7

And after looking these up it would appear there is now a 5.5 version of elasticsearch. This being in dev, I can always attempt to upgrade, but these were all the latest available versions as of sometime last week when I was starting my build.

On Monday, July 10, 2017 at 1:51:24 PM UTC-4, Search Guard wrote:
Which ES/SG/LDAP/MT versions are you using?

Does it work without LDAP (internal users only)?

Am 10.07.2017 um 16:56 schrieb Brian Sanders brian....@gmail.com:

I am testing out the LDAP authentication module with Kibana multi tenancy to see if this will fit our needs, and I am having some strange login issues. I believe I have configured it as described in the example documentation, but the FIRST time I attempt to log in I always end up at the 500 internal server error. If I attempt to log in a few more times, eventually it works and seems to keep working for a bit. I have also logged in and then seen this issue when switching tenants. After logging back in a second time everything works including switching.

I don’t really know how to proceed troubleshooting this, but I can’t demo this to anyone and expect them to want me to take this live to production if LDAP is flaky.

I am still using the basic_internal_auth, since I don’t want to add a kibanaserver or logstash user to my LDAP directory, and would prefer those just be in the local database. Authz is being done by mapping users to rolls, which does work once I am logged in.

Here is the relevant section from my sg_config.yml

searchguard:
dynamic:
kibana:
multitenancy_enabled: true
server_username: “kibanaserver”
index: ‘.kibana’
http:
anonymous_auth_enabled: false
xff:
enabled: false
remoteIpHeader: ‘x-forwarded-for’
proxiesHeader: ‘x-forwarded-by’
authc:
basic_internal_auth_domain:
enabled: true
order: 4
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: intern
ldap:
enabled: true
order: 1
http_authenticator:
type: basic
challenge: true
authentication_backend:
config:
enable_ssl: false
enable_start_tls: true
enable_ssl_client_auth: false
verify_hostnames: false
hosts:
- ipa1.example.com:389

        bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
        password: 'FAKEPASS'
        userbase: 'cn=users,cn=accounts,dc=example,dc=com'
        usersearch: '(uid={0})'
        username_attribute: uid
        skip_users:
          - kibanaserver
          - logstash


You received this message because you are subscribed to the Google Groups “Search Guard” group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/9eba0f76-cbbe-4ea7-85f2-a2a90d750f26%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups “Search Guard” group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/d487734a-1beb-4a13-93d1-4686a32fc203%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups “Search Guard” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/4c3d66b4-bdd4-4616-93a4-7fe9ea926f6c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Any chance someone has been able to find anything in the logs? It very consistently fails the first login, but then after that seems to work. I have been working on getting alerting in place along side this with elastalert, and I think between your plugin and elastalert this could work for us. I just really have problems demoing the setup with login/tenant switching taking 30s due to the timeout. I have avoided upgrading to 5.5, as I don’t want to invalidate the logs I captured if someone finds something telling in them.

···

On Thursday, July 13, 2017 at 10:54:33 AM UTC-4, Brian Sanders wrote:

Yup, that is a good suggestion. Log file attached.

If it helps, my users that fails then succeeds is “brians”, and I have the global tenant turned off but the private tenant turned on. I have also added a few other tenants to his user to test switching.

On Thursday, July 13, 2017 at 10:40:57 AM UTC-4, Search Guard wrote:

Why not write a PGP encrypted mail? Here is our public key https://floragunn.com/search-guard-public-key/

Am 13.07.2017 um 16:10 schrieb Brian Sanders brian....@gmail.com:

I have a log file, is there some way to send it other than attaching it here? I don’t know exactly what all is in it, but I generally don’t like posting debug logs to the internet directly.

On Thursday, July 13, 2017 at 9:24:56 AM UTC-4, Brian Sanders wrote:

Thank you, I will activate this logging and see what I can get. I tried all the settings above with no difference. I also tried Firefox (was using chrome) with the hope that it was browser related, but it still does the same thing. I’ll see if I c

On Wednesday, July 12, 2017 at 11:39:57 AM UTC-4, Search Guard wrote:

Regarding “searchguard.cookie.password” pls read https://github.com/floragunncom/search-guard-docs/blob/master/kibana.md#configuring-the-search-guard-kibana-plugin

(at least in production you should not use the default)

If its still not working pls. turn debug logging on and send the logfile from elasticsearch:

PUT /_cluster/settings
{
“transient” : {
“logger._root” : “DEBUG”
}
}

Am 12.07.2017 um 16:39 schrieb Brian Sanders brian....@gmail.com:

I’ll have to try reinstalling, perhaps I’ll upgrade to the most current versions of plugins as well since this isn’t live and there are new versions.

From your suggested settings in kibana.yml

  1. what is searchguard.cookie.password I haven’t seen mention of that before

I have the other settings, and some of those are defaults so I omitted them. I will specificaly add them and see if this makes much difference before trying to re-install.

On Wednesday, July 12, 2017 at 10:04:28 AM UTC-4, Search Guard wrote:
I verified that the configuration i sent was is working locally for me.
I suggest to reinstall Kibana and the SG Kibana plugin https://github.com/floragunncom/search-guard-kibana-plugin/releases/download/v5.4.2-3/searchguard-kibana-5.4.2-3.zip

The relevant config in kibana.yml (maybe you miss that) looks like:

elasticsearch.requestHeadersWhitelist: [“sg_tenant”, “x-authenticated-user”, “authorization”, “x-forwarded-for”, “x-forwarded-server”, “x-forwarded-by”, “x-proxy-user”, “x-proxy-roles”]
searchguard.multitenancy.enabled: true
searchguard.multitenancy.show_roles: true
searchguard.multitenancy.enable_filter: true
searchguard.multitenancy.tenants.enable_global: true
searchguard.multitenancy.tenants.enable_private: true
searchguard.cookie.password: “”
elasticsearch.username: “kibanaserver”
elasticsearch.password: “kibanaserver”

see https://github.com/floragunncom/search-guard-docs/blob/master/multitenancy.md#kibana-plugin-configuration

On Wednesday, 12 July 2017 15:34:52 UTC+2, Brian Sanders wrote:
This is a configuration I have tried previously, but I tried again this morning before logging in to anything to document the exact order of tests and errors seen.

When I first log in, I do see the log message that it can not authenticate the user with ad 0 (the internal database) and it says it says it is going to the next (eg “try next”) However after an extended period the attempted login just shows
{“statusCode”:500,“error”:“Internal Server Error”,“message”:“An internal server error occurred”}

After that shows up, I simply reselect my URL and press enter again, which this time loads Kibana with out any further prompting for user input. I can perform searches on my data an things appear normal. If I select tenants tab, and change to another tenant, I receive the same:
{“statusCode”:500,“error”:“Internal Server Error”,“message”:“An internal server error occurred”}

This time pressing enter to reload the page eventually brings me back to the Search Guard login screen. After providing my ldap credentials, I am finally authentiacted and put into Kibana as the tenant. Now I can switch between the 2 tenants just fine… but if I select yet another tenant it takes a long time and eventually puts me at the Search Guard login website, at least it didn’t 500 error this time.

So it would seem that first login does this, as well as having issues with switching to a tenant for the first time as well. I am using FreeIPA as an ldap backend, which is a fairly popular user management group of packages, but as far as this is concerned, it is simply authing against the LDAP portion which clearly CAN work. It just doesn’t seem to work on the first try.

On Tuesday, July 11, 2017 at 4:01:37 PM UTC-4, Search Guard wrote:
can you try this?

searchguard:
dynamic:
kibana:
multitenancy_enabled: true
server_username: “kibanaserver”
index: ‘.kibana’
http:
anonymous_auth_enabled: false
xff:
enabled: false
remoteIpHeader: ‘x-forwarded-for’
proxiesHeader: ‘x-forwarded-by’
authc:
basic_internal_auth_domain:
enabled: true
order: 0
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: intern
ldap:
enabled: true
order: 1
http_authenticator:
type: basic
challenge: false
authentication_backend:
config:
enable_ssl: false
enable_start_tls: true
enable_ssl_client_auth: false
verify_hostnames: false
hosts:
- ipa1.example.com:389

        bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
        password: 'FAKEPASS'
        userbase: 'cn=users,cn=accounts,dc=example,dc=com'
        usersearch: '(uid={0})'
        username_attribute: uid
       
        #only valid for authz (not authc)
        #skip_users:
        #  - kibanaserver
        #  - logstash

(more complete example here https://gist.github.com/floragunncom/fdad5c8f1cab838962842d590c408538)

On Monday, 10 July 2017 21:56:25 UTC+2, Brian Sanders wrote:
Seems I didn’t include the Search Guard version: search-guard-5:5.4.2-12

On Monday, July 10, 2017 at 3:55:25 PM UTC-4, Brian Sanders wrote:
Sorry I just saw this question, I never had the issue with the local only accounts. I didn’t test that extensively as I was looking to test the LDAP, but it seemed that it worked just fine.

We are running the current versions:

ES: 5.4.2
kibana-multitenancy-5.4-3
authbackend-ldap-5.0-7

And after looking these up it would appear there is now a 5.5 version of elasticsearch. This being in dev, I can always attempt to upgrade, but these were all the latest available versions as of sometime last week when I was starting my build.

On Monday, July 10, 2017 at 1:51:24 PM UTC-4, Search Guard wrote:
Which ES/SG/LDAP/MT versions are you using?

Does it work without LDAP (internal users only)?

Am 10.07.2017 um 16:56 schrieb Brian Sanders brian....@gmail.com:

I am testing out the LDAP authentication module with Kibana multi tenancy to see if this will fit our needs, and I am having some strange login issues. I believe I have configured it as described in the example documentation, but the FIRST time I attempt to log in I always end up at the 500 internal server error. If I attempt to log in a few more times, eventually it works and seems to keep working for a bit. I have also logged in and then seen this issue when switching tenants. After logging back in a second time everything works including switching.

I don’t really know how to proceed troubleshooting this, but I can’t demo this to anyone and expect them to want me to take this live to production if LDAP is flaky.

I am still using the basic_internal_auth, since I don’t want to add a kibanaserver or logstash user to my LDAP directory, and would prefer those just be in the local database. Authz is being done by mapping users to rolls, which does work once I am logged in.

Here is the relevant section from my sg_config.yml

searchguard:
dynamic:
kibana:
multitenancy_enabled: true
server_username: “kibanaserver”
index: ‘.kibana’
http:
anonymous_auth_enabled: false
xff:
enabled: false
remoteIpHeader: ‘x-forwarded-for’
proxiesHeader: ‘x-forwarded-by’
authc:
basic_internal_auth_domain:
enabled: true
order: 4
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: intern
ldap:
enabled: true
order: 1
http_authenticator:
type: basic
challenge: true
authentication_backend:
config:
enable_ssl: false
enable_start_tls: true
enable_ssl_client_auth: false
verify_hostnames: false
hosts:
- ipa1.example.com:389

        bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
        password: 'FAKEPASS'
        userbase: 'cn=users,cn=accounts,dc=example,dc=com'
        usersearch: '(uid={0})'
        username_attribute: uid
        skip_users:
          - kibanaserver
          - logstash


You received this message because you are subscribed to the Google Groups “Search Guard” group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/9eba0f76-cbbe-4ea7-85f2-a2a90d750f26%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups “Search Guard” group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/d487734a-1beb-4a13-93d1-4686a32fc203%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups “Search Guard” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/4c3d66b4-bdd4-4616-93a4-7fe9ea926f6c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Hi Brian,

we’ve analyzed the log files and tried to replicate the problem here, but no luck so far. The log file also looks fine, I can’t find anything regarding the login or the internal server error.

The first login attempt starts here:

[2017-07-13T13:58:56,298][DEBUG][c.f.s.a.BackendRegistry ] Try to extract auth creds from http basic

[2017-07-13T13:58:56,298][DEBUG][c.f.s.a.BackendRegistry ] User ‘XXX’ is in cache? false (cache size: 1)

… LDAP stuff …

[2017-07-13T13:58:56,466][DEBUG][c.f.d.a.l.b.LDAPAuthenticationBackend] Authenticated username brians

[2017-07-13T13:58:56,468][DEBUG][c.f.s.a.BackendRegistry ] User ‘User [name=XXX, roles=]’ is authenticated

which means that the authentication process takes around 150ms. Which is still a bit high, but depends on network latency and LDAP server performance. So, this seems fine. After that, the user is cached (default is 1 hour), so even the LDAP server should not have any impact anymore:

[2017-07-13T13:58:56,609][DEBUG][c.f.s.a.BackendRegistry ] Try to extract auth creds from http basic

[2017-07-13T13:58:56,609][DEBUG][c.f.s.a.BackendRegistry ] User ‘XXX’ is in cache? true (cache size: 2)

[2017-07-13T13:58:56,610][DEBUG][c.f.s.a.BackendRegistry ] User ‘User [name=XXX, roles=]’ is authenticated

Cache size seems ok, since both your user and the Kibana server user is cached. Also, there are no Exceptions in the log, which I would have expected since Kibana logs an 500 internal server error.

So, at the moment I can’t see anything in the ES log that seems related to the issues you’re facing. Also, LDAP is a major use case for Search Guard, and we did not hear about such a behaviour before. Therefore I guess the problem is not related to the LDAP module at the moment. But just to make sure:

  • If you only use the internal user database and disable LDAP, do you still see these issues?

  • Apart from Search Guard, do you have any other ES plugins installed

  • Apart from the SG Kibana plugin, do you have any other Kibana plugins installed

  • Are there any other services running against ES?

In other words - can you replicate the problem with a minimum installation of ES/SG and KI/SG Kibana plugin?

Next, let’s look at the actual HTTP requests from Kibana. Can you please:

  • Delete all log files

  • Fire up ES and KI

  • In Chrome, open the Developer tools and switch to the Network tab

  • Replicate the login problem

In the network tab, can you spot which request is actually taking so long? If you find anything there, please post (or email) the request incl. headers and payload, and the reponse? And send the ES log file as well?

Sorry for the delayed response, but since we can’t reproduce the problem we’re having a hard time pinpointing the problem.

···

On Tuesday, July 18, 2017 at 3:50:53 PM UTC+2, Brian Sanders wrote:

Any chance someone has been able to find anything in the logs? It very consistently fails the first login, but then after that seems to work. I have been working on getting alerting in place along side this with elastalert, and I think between your plugin and elastalert this could work for us. I just really have problems demoing the setup with login/tenant switching taking 30s due to the timeout. I have avoided upgrading to 5.5, as I don’t want to invalidate the logs I captured if someone finds something telling in them.

On Thursday, July 13, 2017 at 10:54:33 AM UTC-4, Brian Sanders wrote:

Yup, that is a good suggestion. Log file attached.

If it helps, my users that fails then succeeds is “brians”, and I have the global tenant turned off but the private tenant turned on. I have also added a few other tenants to his user to test switching.

On Thursday, July 13, 2017 at 10:40:57 AM UTC-4, Search Guard wrote:

Why not write a PGP encrypted mail? Here is our public key https://floragunn.com/search-guard-public-key/

Am 13.07.2017 um 16:10 schrieb Brian Sanders brian....@gmail.com:

I have a log file, is there some way to send it other than attaching it here? I don’t know exactly what all is in it, but I generally don’t like posting debug logs to the internet directly.

On Thursday, July 13, 2017 at 9:24:56 AM UTC-4, Brian Sanders wrote:

Thank you, I will activate this logging and see what I can get. I tried all the settings above with no difference. I also tried Firefox (was using chrome) with the hope that it was browser related, but it still does the same thing. I’ll see if I c

On Wednesday, July 12, 2017 at 11:39:57 AM UTC-4, Search Guard wrote:

Regarding “searchguard.cookie.password” pls read https://github.com/floragunncom/search-guard-docs/blob/master/kibana.md#configuring-the-search-guard-kibana-plugin

(at least in production you should not use the default)

If its still not working pls. turn debug logging on and send the logfile from elasticsearch:

PUT /_cluster/settings
{
“transient” : {
“logger._root” : “DEBUG”
}
}

Am 12.07.2017 um 16:39 schrieb Brian Sanders brian....@gmail.com:

I’ll have to try reinstalling, perhaps I’ll upgrade to the most current versions of plugins as well since this isn’t live and there are new versions.

From your suggested settings in kibana.yml

  1. what is searchguard.cookie.password I haven’t seen mention of that before

I have the other settings, and some of those are defaults so I omitted them. I will specificaly add them and see if this makes much difference before trying to re-install.

On Wednesday, July 12, 2017 at 10:04:28 AM UTC-4, Search Guard wrote:
I verified that the configuration i sent was is working locally for me.
I suggest to reinstall Kibana and the SG Kibana plugin https://github.com/floragunncom/search-guard-kibana-plugin/releases/download/v5.4.2-3/searchguard-kibana-5.4.2-3.zip

The relevant config in kibana.yml (maybe you miss that) looks like:

elasticsearch.requestHeadersWhitelist: [“sg_tenant”, “x-authenticated-user”, “authorization”, “x-forwarded-for”, “x-forwarded-server”, “x-forwarded-by”, “x-proxy-user”, “x-proxy-roles”]
searchguard.multitenancy.enabled: true
searchguard.multitenancy.show_roles: true
searchguard.multitenancy.enable_filter: true
searchguard.multitenancy.tenants.enable_global: true
searchguard.multitenancy.tenants.enable_private: true
searchguard.cookie.password: “”
elasticsearch.username: “kibanaserver”
elasticsearch.password: “kibanaserver”

see https://github.com/floragunncom/search-guard-docs/blob/master/multitenancy.md#kibana-plugin-configuration

On Wednesday, 12 July 2017 15:34:52 UTC+2, Brian Sanders wrote:
This is a configuration I have tried previously, but I tried again this morning before logging in to anything to document the exact order of tests and errors seen.

When I first log in, I do see the log message that it can not authenticate the user with ad 0 (the internal database) and it says it says it is going to the next (eg “try next”) However after an extended period the attempted login just shows
{“statusCode”:500,“error”:“Internal Server Error”,“message”:“An internal server error occurred”}

After that shows up, I simply reselect my URL and press enter again, which this time loads Kibana with out any further prompting for user input. I can perform searches on my data an things appear normal. If I select tenants tab, and change to another tenant, I receive the same:
{“statusCode”:500,“error”:“Internal Server Error”,“message”:“An internal server error occurred”}

This time pressing enter to reload the page eventually brings me back to the Search Guard login screen. After providing my ldap credentials, I am finally authentiacted and put into Kibana as the tenant. Now I can switch between the 2 tenants just fine… but if I select yet another tenant it takes a long time and eventually puts me at the Search Guard login website, at least it didn’t 500 error this time.

So it would seem that first login does this, as well as having issues with switching to a tenant for the first time as well. I am using FreeIPA as an ldap backend, which is a fairly popular user management group of packages, but as far as this is concerned, it is simply authing against the LDAP portion which clearly CAN work. It just doesn’t seem to work on the first try.

On Tuesday, July 11, 2017 at 4:01:37 PM UTC-4, Search Guard wrote:
can you try this?

searchguard:
dynamic:
kibana:
multitenancy_enabled: true
server_username: “kibanaserver”
index: ‘.kibana’
http:
anonymous_auth_enabled: false
xff:
enabled: false
remoteIpHeader: ‘x-forwarded-for’
proxiesHeader: ‘x-forwarded-by’
authc:
basic_internal_auth_domain:
enabled: true
order: 0
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: intern
ldap:
enabled: true
order: 1
http_authenticator:
type: basic
challenge: false
authentication_backend:
config:
enable_ssl: false
enable_start_tls: true
enable_ssl_client_auth: false
verify_hostnames: false
hosts:
- ipa1.example.com:389

        bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
        password: 'FAKEPASS'
        userbase: 'cn=users,cn=accounts,dc=example,dc=com'
        usersearch: '(uid={0})'
        username_attribute: uid
       
        #only valid for authz (not authc)
        #skip_users:
        #  - kibanaserver
        #  - logstash

(more complete example here https://gist.github.com/floragunncom/fdad5c8f1cab838962842d590c408538)

On Monday, 10 July 2017 21:56:25 UTC+2, Brian Sanders wrote:
Seems I didn’t include the Search Guard version: search-guard-5:5.4.2-12

On Monday, July 10, 2017 at 3:55:25 PM UTC-4, Brian Sanders wrote:
Sorry I just saw this question, I never had the issue with the local only accounts. I didn’t test that extensively as I was looking to test the LDAP, but it seemed that it worked just fine.

We are running the current versions:

ES: 5.4.2
kibana-multitenancy-5.4-3
authbackend-ldap-5.0-7

And after looking these up it would appear there is now a 5.5 version of elasticsearch. This being in dev, I can always attempt to upgrade, but these were all the latest available versions as of sometime last week when I was starting my build.

On Monday, July 10, 2017 at 1:51:24 PM UTC-4, Search Guard wrote:
Which ES/SG/LDAP/MT versions are you using?

Does it work without LDAP (internal users only)?

Am 10.07.2017 um 16:56 schrieb Brian Sanders brian....@gmail.com:

I am testing out the LDAP authentication module with Kibana multi tenancy to see if this will fit our needs, and I am having some strange login issues. I believe I have configured it as described in the example documentation, but the FIRST time I attempt to log in I always end up at the 500 internal server error. If I attempt to log in a few more times, eventually it works and seems to keep working for a bit. I have also logged in and then seen this issue when switching tenants. After logging back in a second time everything works including switching.

I don’t really know how to proceed troubleshooting this, but I can’t demo this to anyone and expect them to want me to take this live to production if LDAP is flaky.

I am still using the basic_internal_auth, since I don’t want to add a kibanaserver or logstash user to my LDAP directory, and would prefer those just be in the local database. Authz is being done by mapping users to rolls, which does work once I am logged in.

Here is the relevant section from my sg_config.yml

searchguard:
dynamic:
kibana:
multitenancy_enabled: true
server_username: “kibanaserver”
index: ‘.kibana’
http:
anonymous_auth_enabled: false
xff:
enabled: false
remoteIpHeader: ‘x-forwarded-for’
proxiesHeader: ‘x-forwarded-by’
authc:
basic_internal_auth_domain:
enabled: true
order: 4
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: intern
ldap:
enabled: true
order: 1
http_authenticator:
type: basic
challenge: true
authentication_backend:
config:
enable_ssl: false
enable_start_tls: true
enable_ssl_client_auth: false
verify_hostnames: false
hosts:
- ipa1.example.com:389

        bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
        password: 'FAKEPASS'
        userbase: 'cn=users,cn=accounts,dc=example,dc=com'
        usersearch: '(uid={0})'
        username_attribute: uid
        skip_users:
          - kibanaserver
          - logstash


You received this message because you are subscribed to the Google Groups “Search Guard” group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/9eba0f76-cbbe-4ea7-85f2-a2a90d750f26%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups “Search Guard” group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/d487734a-1beb-4a13-93d1-4686a32fc203%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups “Search Guard” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/4c3d66b4-bdd4-4616-93a4-7fe9ea926f6c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Thanks for the detailed reply! I’ll quickly answer what I can and get back to you as soon as I can get these tests run.

-I do not have any other ES plugins (not even the X-pack free version)
-Nothing else in SG plugins besides the tenants
-logstash is pushing into elasticsearch. I have Cerebro for checking index status, but that is now a web app and no longer a plugin.

I’ll try local only and also get back to you with the test results you suggested.

So is there some guaranteed way I can clear the Cache? I am assuming that once cached I am good, so if I could do something to clear the cache and force the issue again I could test much quicker.

···

On Thursday, July 20, 2017 at 9:22:52 AM UTC-4, Jochen Kressin wrote:

Hi Brian,

we’ve analyzed the log files and tried to replicate the problem here, but no luck so far. The log file also looks fine, I can’t find anything regarding the login or the internal server error.

The first login attempt starts here:

[2017-07-13T13:58:56,298][DEBUG][c.f.s.a.BackendRegistry ] Try to extract auth creds from http basic

[2017-07-13T13:58:56,298][DEBUG][c.f.s.a.BackendRegistry ] User ‘XXX’ is in cache? false (cache size: 1)

… LDAP stuff …

[2017-07-13T13:58:56,466][DEBUG][c.f.d.a.l.b.LDAPAuthenticationBackend] Authenticated username brians

[2017-07-13T13:58:56,468][DEBUG][c.f.s.a.BackendRegistry ] User ‘User [name=XXX, roles=]’ is authenticated

which means that the authentication process takes around 150ms. Which is still a bit high, but depends on network latency and LDAP server performance. So, this seems fine. After that, the user is cached (default is 1 hour), so even the LDAP server should not have any impact anymore:

[2017-07-13T13:58:56,609][DEBUG][c.f.s.a.BackendRegistry ] Try to extract auth creds from http basic

[2017-07-13T13:58:56,609][DEBUG][c.f.s.a.BackendRegistry ] User ‘XXX’ is in cache? true (cache size: 2)

[2017-07-13T13:58:56,610][DEBUG][c.f.s.a.BackendRegistry ] User ‘User [name=XXX, roles=]’ is authenticated

Cache size seems ok, since both your user and the Kibana server user is cached. Also, there are no Exceptions in the log, which I would have expected since Kibana logs an 500 internal server error.

So, at the moment I can’t see anything in the ES log that seems related to the issues you’re facing. Also, LDAP is a major use case for Search Guard, and we did not hear about such a behaviour before. Therefore I guess the problem is not related to the LDAP module at the moment. But just to make sure:

  • If you only use the internal user database and disable LDAP, do you still see these issues?
  • Apart from Search Guard, do you have any other ES plugins installed
  • Apart from the SG Kibana plugin, do you have any other Kibana plugins installed
  • Are there any other services running against ES?

In other words - can you replicate the problem with a minimum installation of ES/SG and KI/SG Kibana plugin?

Next, let’s look at the actual HTTP requests from Kibana. Can you please:

  • Delete all log files
  • Fire up ES and KI
  • In Chrome, open the Developer tools and switch to the Network tab
  • Replicate the login problem

In the network tab, can you spot which request is actually taking so long? If you find anything there, please post (or email) the request incl. headers and payload, and the reponse? And send the ES log file as well?

Sorry for the delayed response, but since we can’t reproduce the problem we’re having a hard time pinpointing the problem.

On Tuesday, July 18, 2017 at 3:50:53 PM UTC+2, Brian Sanders wrote:

Any chance someone has been able to find anything in the logs? It very consistently fails the first login, but then after that seems to work. I have been working on getting alerting in place along side this with elastalert, and I think between your plugin and elastalert this could work for us. I just really have problems demoing the setup with login/tenant switching taking 30s due to the timeout. I have avoided upgrading to 5.5, as I don’t want to invalidate the logs I captured if someone finds something telling in them.

On Thursday, July 13, 2017 at 10:54:33 AM UTC-4, Brian Sanders wrote:

Yup, that is a good suggestion. Log file attached.

If it helps, my users that fails then succeeds is “brians”, and I have the global tenant turned off but the private tenant turned on. I have also added a few other tenants to his user to test switching.

On Thursday, July 13, 2017 at 10:40:57 AM UTC-4, Search Guard wrote:

Why not write a PGP encrypted mail? Here is our public key https://floragunn.com/search-guard-public-key/

Am 13.07.2017 um 16:10 schrieb Brian Sanders brian....@gmail.com:

I have a log file, is there some way to send it other than attaching it here? I don’t know exactly what all is in it, but I generally don’t like posting debug logs to the internet directly.

On Thursday, July 13, 2017 at 9:24:56 AM UTC-4, Brian Sanders wrote:

Thank you, I will activate this logging and see what I can get. I tried all the settings above with no difference. I also tried Firefox (was using chrome) with the hope that it was browser related, but it still does the same thing. I’ll see if I c

On Wednesday, July 12, 2017 at 11:39:57 AM UTC-4, Search Guard wrote:

Regarding “searchguard.cookie.password” pls read https://github.com/floragunncom/search-guard-docs/blob/master/kibana.md#configuring-the-search-guard-kibana-plugin

(at least in production you should not use the default)

If its still not working pls. turn debug logging on and send the logfile from elasticsearch:

PUT /_cluster/settings
{
“transient” : {
“logger._root” : “DEBUG”
}
}

Am 12.07.2017 um 16:39 schrieb Brian Sanders brian....@gmail.com:

I’ll have to try reinstalling, perhaps I’ll upgrade to the most current versions of plugins as well since this isn’t live and there are new versions.

From your suggested settings in kibana.yml

  1. what is searchguard.cookie.password I haven’t seen mention of that before

I have the other settings, and some of those are defaults so I omitted them. I will specificaly add them and see if this makes much difference before trying to re-install.

On Wednesday, July 12, 2017 at 10:04:28 AM UTC-4, Search Guard wrote:
I verified that the configuration i sent was is working locally for me.
I suggest to reinstall Kibana and the SG Kibana plugin https://github.com/floragunncom/search-guard-kibana-plugin/releases/download/v5.4.2-3/searchguard-kibana-5.4.2-3.zip

The relevant config in kibana.yml (maybe you miss that) looks like:

elasticsearch.requestHeadersWhitelist: [“sg_tenant”, “x-authenticated-user”, “authorization”, “x-forwarded-for”, “x-forwarded-server”, “x-forwarded-by”, “x-proxy-user”, “x-proxy-roles”]
searchguard.multitenancy.enabled: true
searchguard.multitenancy.show_roles: true
searchguard.multitenancy.enable_filter: true
searchguard.multitenancy.tenants.enable_global: true
searchguard.multitenancy.tenants.enable_private: true
searchguard.cookie.password: “”
elasticsearch.username: “kibanaserver”
elasticsearch.password: “kibanaserver”

see https://github.com/floragunncom/search-guard-docs/blob/master/multitenancy.md#kibana-plugin-configuration

On Wednesday, 12 July 2017 15:34:52 UTC+2, Brian Sanders wrote:
This is a configuration I have tried previously, but I tried again this morning before logging in to anything to document the exact order of tests and errors seen.

When I first log in, I do see the log message that it can not authenticate the user with ad 0 (the internal database) and it says it says it is going to the next (eg “try next”) However after an extended period the attempted login just shows
{“statusCode”:500,“error”:“Internal Server Error”,“message”:“An internal server error occurred”}

After that shows up, I simply reselect my URL and press enter again, which this time loads Kibana with out any further prompting for user input. I can perform searches on my data an things appear normal. If I select tenants tab, and change to another tenant, I receive the same:
{“statusCode”:500,“error”:“Internal Server Error”,“message”:“An internal server error occurred”}

This time pressing enter to reload the page eventually brings me back to the Search Guard login screen. After providing my ldap credentials, I am finally authentiacted and put into Kibana as the tenant. Now I can switch between the 2 tenants just fine… but if I select yet another tenant it takes a long time and eventually puts me at the Search Guard login website, at least it didn’t 500 error this time.

So it would seem that first login does this, as well as having issues with switching to a tenant for the first time as well. I am using FreeIPA as an ldap backend, which is a fairly popular user management group of packages, but as far as this is concerned, it is simply authing against the LDAP portion which clearly CAN work. It just doesn’t seem to work on the first try.

On Tuesday, July 11, 2017 at 4:01:37 PM UTC-4, Search Guard wrote:
can you try this?

searchguard:
dynamic:
kibana:
multitenancy_enabled: true
server_username: “kibanaserver”
index: ‘.kibana’
http:
anonymous_auth_enabled: false
xff:
enabled: false
remoteIpHeader: ‘x-forwarded-for’
proxiesHeader: ‘x-forwarded-by’
authc:
basic_internal_auth_domain:
enabled: true
order: 0
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: intern
ldap:
enabled: true
order: 1
http_authenticator:
type: basic
challenge: false
authentication_backend:
config:
enable_ssl: false
enable_start_tls: true
enable_ssl_client_auth: false
verify_hostnames: false
hosts:
- ipa1.example.com:389

        bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
        password: 'FAKEPASS'
        userbase: 'cn=users,cn=accounts,dc=example,dc=com'
        usersearch: '(uid={0})'
        username_attribute: uid
       
        #only valid for authz (not authc)
        #skip_users:
        #  - kibanaserver
        #  - logstash

(more complete example here https://gist.github.com/floragunncom/fdad5c8f1cab838962842d590c408538)

On Monday, 10 July 2017 21:56:25 UTC+2, Brian Sanders wrote:
Seems I didn’t include the Search Guard version: search-guard-5:5.4.2-12

On Monday, July 10, 2017 at 3:55:25 PM UTC-4, Brian Sanders wrote:
Sorry I just saw this question, I never had the issue with the local only accounts. I didn’t test that extensively as I was looking to test the LDAP, but it seemed that it worked just fine.

We are running the current versions:

ES: 5.4.2
kibana-multitenancy-5.4-3
authbackend-ldap-5.0-7

And after looking these up it would appear there is now a 5.5 version of elasticsearch. This being in dev, I can always attempt to upgrade, but these were all the latest available versions as of sometime last week when I was starting my build.

On Monday, July 10, 2017 at 1:51:24 PM UTC-4, Search Guard wrote:
Which ES/SG/LDAP/MT versions are you using?

Does it work without LDAP (internal users only)?

Am 10.07.2017 um 16:56 schrieb Brian Sanders brian....@gmail.com:

I am testing out the LDAP authentication module with Kibana multi tenancy to see if this will fit our needs, and I am having some strange login issues. I believe I have configured it as described in the example documentation, but the FIRST time I attempt to log in I always end up at the 500 internal server error. If I attempt to log in a few more times, eventually it works and seems to keep working for a bit. I have also logged in and then seen this issue when switching tenants. After logging back in a second time everything works including switching.

I don’t really know how to proceed troubleshooting this, but I can’t demo this to anyone and expect them to want me to take this live to production if LDAP is flaky.

I am still using the basic_internal_auth, since I don’t want to add a kibanaserver or logstash user to my LDAP directory, and would prefer those just be in the local database. Authz is being done by mapping users to rolls, which does work once I am logged in.

Here is the relevant section from my sg_config.yml

searchguard:
dynamic:
kibana:
multitenancy_enabled: true
server_username: “kibanaserver”
index: ‘.kibana’
http:
anonymous_auth_enabled: false
xff:
enabled: false
remoteIpHeader: ‘x-forwarded-for’
proxiesHeader: ‘x-forwarded-by’
authc:
basic_internal_auth_domain:
enabled: true
order: 4
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: intern
ldap:
enabled: true
order: 1
http_authenticator:
type: basic
challenge: true
authentication_backend:
config:
enable_ssl: false
enable_start_tls: true
enable_ssl_client_auth: false
verify_hostnames: false
hosts:
- ipa1.example.com:389

        bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
        password: 'FAKEPASS'
        userbase: 'cn=users,cn=accounts,dc=example,dc=com'
        usersearch: '(uid={0})'
        username_attribute: uid
        skip_users:
          - kibanaserver
          - logstash


You received this message because you are subscribed to the Google Groups “Search Guard” group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/9eba0f76-cbbe-4ea7-85f2-a2a90d750f26%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups “Search Guard” group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/d487734a-1beb-4a13-93d1-4686a32fc203%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups “Search Guard” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/4c3d66b4-bdd4-4616-93a4-7fe9ea926f6c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Yes - the cache is cleared each time you either upload new configurations, or reload the existing configuration.

Reloading the existing configuration can be done with the -rl/–reload flag like:

./sgadmin.sh -ts … -tspass … -ks … -kspass … -icl -nhnv -rl

···

On Thursday, July 20, 2017 at 3:58:31 PM UTC+2, Brian Sanders wrote:

Thanks for the detailed reply! I’ll quickly answer what I can and get back to you as soon as I can get these tests run.

-I do not have any other ES plugins (not even the X-pack free version)
-Nothing else in SG plugins besides the tenants
-logstash is pushing into elasticsearch. I have Cerebro for checking index status, but that is now a web app and no longer a plugin.

I’ll try local only and also get back to you with the test results you suggested.

So is there some guaranteed way I can clear the Cache? I am assuming that once cached I am good, so if I could do something to clear the cache and force the issue again I could test much quicker.

On Thursday, July 20, 2017 at 9:22:52 AM UTC-4, Jochen Kressin wrote:

Hi Brian,

we’ve analyzed the log files and tried to replicate the problem here, but no luck so far. The log file also looks fine, I can’t find anything regarding the login or the internal server error.

The first login attempt starts here:

[2017-07-13T13:58:56,298][DEBUG][c.f.s.a.BackendRegistry ] Try to extract auth creds from http basic

[2017-07-13T13:58:56,298][DEBUG][c.f.s.a.BackendRegistry ] User ‘XXX’ is in cache? false (cache size: 1)

… LDAP stuff …

[2017-07-13T13:58:56,466][DEBUG][c.f.d.a.l.b.LDAPAuthenticationBackend] Authenticated username brians

[2017-07-13T13:58:56,468][DEBUG][c.f.s.a.BackendRegistry ] User ‘User [name=XXX, roles=]’ is authenticated

which means that the authentication process takes around 150ms. Which is still a bit high, but depends on network latency and LDAP server performance. So, this seems fine. After that, the user is cached (default is 1 hour), so even the LDAP server should not have any impact anymore:

[2017-07-13T13:58:56,609][DEBUG][c.f.s.a.BackendRegistry ] Try to extract auth creds from http basic

[2017-07-13T13:58:56,609][DEBUG][c.f.s.a.BackendRegistry ] User ‘XXX’ is in cache? true (cache size: 2)

[2017-07-13T13:58:56,610][DEBUG][c.f.s.a.BackendRegistry ] User ‘User [name=XXX, roles=]’ is authenticated

Cache size seems ok, since both your user and the Kibana server user is cached. Also, there are no Exceptions in the log, which I would have expected since Kibana logs an 500 internal server error.

So, at the moment I can’t see anything in the ES log that seems related to the issues you’re facing. Also, LDAP is a major use case for Search Guard, and we did not hear about such a behaviour before. Therefore I guess the problem is not related to the LDAP module at the moment. But just to make sure:

  • If you only use the internal user database and disable LDAP, do you still see these issues?
  • Apart from Search Guard, do you have any other ES plugins installed
  • Apart from the SG Kibana plugin, do you have any other Kibana plugins installed
  • Are there any other services running against ES?

In other words - can you replicate the problem with a minimum installation of ES/SG and KI/SG Kibana plugin?

Next, let’s look at the actual HTTP requests from Kibana. Can you please:

  • Delete all log files
  • Fire up ES and KI
  • In Chrome, open the Developer tools and switch to the Network tab
  • Replicate the login problem

In the network tab, can you spot which request is actually taking so long? If you find anything there, please post (or email) the request incl. headers and payload, and the reponse? And send the ES log file as well?

Sorry for the delayed response, but since we can’t reproduce the problem we’re having a hard time pinpointing the problem.

On Tuesday, July 18, 2017 at 3:50:53 PM UTC+2, Brian Sanders wrote:

Any chance someone has been able to find anything in the logs? It very consistently fails the first login, but then after that seems to work. I have been working on getting alerting in place along side this with elastalert, and I think between your plugin and elastalert this could work for us. I just really have problems demoing the setup with login/tenant switching taking 30s due to the timeout. I have avoided upgrading to 5.5, as I don’t want to invalidate the logs I captured if someone finds something telling in them.

On Thursday, July 13, 2017 at 10:54:33 AM UTC-4, Brian Sanders wrote:

Yup, that is a good suggestion. Log file attached.

If it helps, my users that fails then succeeds is “brians”, and I have the global tenant turned off but the private tenant turned on. I have also added a few other tenants to his user to test switching.

On Thursday, July 13, 2017 at 10:40:57 AM UTC-4, Search Guard wrote:

Why not write a PGP encrypted mail? Here is our public key https://floragunn.com/search-guard-public-key/

Am 13.07.2017 um 16:10 schrieb Brian Sanders brian....@gmail.com:

I have a log file, is there some way to send it other than attaching it here? I don’t know exactly what all is in it, but I generally don’t like posting debug logs to the internet directly.

On Thursday, July 13, 2017 at 9:24:56 AM UTC-4, Brian Sanders wrote:

Thank you, I will activate this logging and see what I can get. I tried all the settings above with no difference. I also tried Firefox (was using chrome) with the hope that it was browser related, but it still does the same thing. I’ll see if I c

On Wednesday, July 12, 2017 at 11:39:57 AM UTC-4, Search Guard wrote:

Regarding “searchguard.cookie.password” pls read https://github.com/floragunncom/search-guard-docs/blob/master/kibana.md#configuring-the-search-guard-kibana-plugin

(at least in production you should not use the default)

If its still not working pls. turn debug logging on and send the logfile from elasticsearch:

PUT /_cluster/settings
{
“transient” : {
“logger._root” : “DEBUG”
}
}

Am 12.07.2017 um 16:39 schrieb Brian Sanders brian....@gmail.com:

I’ll have to try reinstalling, perhaps I’ll upgrade to the most current versions of plugins as well since this isn’t live and there are new versions.

From your suggested settings in kibana.yml

  1. what is searchguard.cookie.password I haven’t seen mention of that before

I have the other settings, and some of those are defaults so I omitted them. I will specificaly add them and see if this makes much difference before trying to re-install.

On Wednesday, July 12, 2017 at 10:04:28 AM UTC-4, Search Guard wrote:
I verified that the configuration i sent was is working locally for me.
I suggest to reinstall Kibana and the SG Kibana plugin https://github.com/floragunncom/search-guard-kibana-plugin/releases/download/v5.4.2-3/searchguard-kibana-5.4.2-3.zip

The relevant config in kibana.yml (maybe you miss that) looks like:

elasticsearch.requestHeadersWhitelist: [“sg_tenant”, “x-authenticated-user”, “authorization”, “x-forwarded-for”, “x-forwarded-server”, “x-forwarded-by”, “x-proxy-user”, “x-proxy-roles”]
searchguard.multitenancy.enabled: true
searchguard.multitenancy.show_roles: true
searchguard.multitenancy.enable_filter: true
searchguard.multitenancy.tenants.enable_global: true
searchguard.multitenancy.tenants.enable_private: true
searchguard.cookie.password: “”
elasticsearch.username: “kibanaserver”
elasticsearch.password: “kibanaserver”

see https://github.com/floragunncom/search-guard-docs/blob/master/multitenancy.md#kibana-plugin-configuration

On Wednesday, 12 July 2017 15:34:52 UTC+2, Brian Sanders wrote:
This is a configuration I have tried previously, but I tried again this morning before logging in to anything to document the exact order of tests and errors seen.

When I first log in, I do see the log message that it can not authenticate the user with ad 0 (the internal database) and it says it says it is going to the next (eg “try next”) However after an extended period the attempted login just shows
{“statusCode”:500,“error”:“Internal Server Error”,“message”:“An internal server error occurred”}

After that shows up, I simply reselect my URL and press enter again, which this time loads Kibana with out any further prompting for user input. I can perform searches on my data an things appear normal. If I select tenants tab, and change to another tenant, I receive the same:
{“statusCode”:500,“error”:“Internal Server Error”,“message”:“An internal server error occurred”}

This time pressing enter to reload the page eventually brings me back to the Search Guard login screen. After providing my ldap credentials, I am finally authentiacted and put into Kibana as the tenant. Now I can switch between the 2 tenants just fine… but if I select yet another tenant it takes a long time and eventually puts me at the Search Guard login website, at least it didn’t 500 error this time.

So it would seem that first login does this, as well as having issues with switching to a tenant for the first time as well. I am using FreeIPA as an ldap backend, which is a fairly popular user management group of packages, but as far as this is concerned, it is simply authing against the LDAP portion which clearly CAN work. It just doesn’t seem to work on the first try.

On Tuesday, July 11, 2017 at 4:01:37 PM UTC-4, Search Guard wrote:
can you try this?

searchguard:
dynamic:
kibana:
multitenancy_enabled: true
server_username: “kibanaserver”
index: ‘.kibana’
http:
anonymous_auth_enabled: false
xff:
enabled: false
remoteIpHeader: ‘x-forwarded-for’
proxiesHeader: ‘x-forwarded-by’
authc:
basic_internal_auth_domain:
enabled: true
order: 0
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: intern
ldap:
enabled: true
order: 1
http_authenticator:
type: basic
challenge: false
authentication_backend:
config:
enable_ssl: false
enable_start_tls: true
enable_ssl_client_auth: false
verify_hostnames: false
hosts:
- ipa1.example.com:389

        bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
        password: 'FAKEPASS'
        userbase: 'cn=users,cn=accounts,dc=example,dc=com'
        usersearch: '(uid={0})'
        username_attribute: uid
       
        #only valid for authz (not authc)
        #skip_users:
        #  - kibanaserver
        #  - logstash

(more complete example here https://gist.github.com/floragunncom/fdad5c8f1cab838962842d590c408538)

On Monday, 10 July 2017 21:56:25 UTC+2, Brian Sanders wrote:
Seems I didn’t include the Search Guard version: search-guard-5:5.4.2-12

On Monday, July 10, 2017 at 3:55:25 PM UTC-4, Brian Sanders wrote:
Sorry I just saw this question, I never had the issue with the local only accounts. I didn’t test that extensively as I was looking to test the LDAP, but it seemed that it worked just fine.

We are running the current versions:

ES: 5.4.2
kibana-multitenancy-5.4-3
authbackend-ldap-5.0-7

And after looking these up it would appear there is now a 5.5 version of elasticsearch. This being in dev, I can always attempt to upgrade, but these were all the latest available versions as of sometime last week when I was starting my build.

On Monday, July 10, 2017 at 1:51:24 PM UTC-4, Search Guard wrote:
Which ES/SG/LDAP/MT versions are you using?

Does it work without LDAP (internal users only)?

Am 10.07.2017 um 16:56 schrieb Brian Sanders brian....@gmail.com:

I am testing out the LDAP authentication module with Kibana multi tenancy to see if this will fit our needs, and I am having some strange login issues. I believe I have configured it as described in the example documentation, but the FIRST time I attempt to log in I always end up at the 500 internal server error. If I attempt to log in a few more times, eventually it works and seems to keep working for a bit. I have also logged in and then seen this issue when switching tenants. After logging back in a second time everything works including switching.

I don’t really know how to proceed troubleshooting this, but I can’t demo this to anyone and expect them to want me to take this live to production if LDAP is flaky.

I am still using the basic_internal_auth, since I don’t want to add a kibanaserver or logstash user to my LDAP directory, and would prefer those just be in the local database. Authz is being done by mapping users to rolls, which does work once I am logged in.

Here is the relevant section from my sg_config.yml

searchguard:
dynamic:
kibana:
multitenancy_enabled: true
server_username: “kibanaserver”
index: ‘.kibana’
http:
anonymous_auth_enabled: false
xff:
enabled: false
remoteIpHeader: ‘x-forwarded-for’
proxiesHeader: ‘x-forwarded-by’
authc:
basic_internal_auth_domain:
enabled: true
order: 4
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: intern
ldap:
enabled: true
order: 1
http_authenticator:
type: basic
challenge: true
authentication_backend:
config:
enable_ssl: false
enable_start_tls: true
enable_ssl_client_auth: false
verify_hostnames: false
hosts:
- ipa1.example.com:389

        bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
        password: 'FAKEPASS'
        userbase: 'cn=users,cn=accounts,dc=example,dc=com'
        usersearch: '(uid={0})'
        username_attribute: uid
        skip_users:
          - kibanaserver
          - logstash


You received this message because you are subscribed to the Google Groups “Search Guard” group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/9eba0f76-cbbe-4ea7-85f2-a2a90d750f26%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups “Search Guard” group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/d487734a-1beb-4a13-93d1-4686a32fc203%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups “Search Guard” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/4c3d66b4-bdd4-4616-93a4-7fe9ea926f6c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

I was about to reply with tests results, when I discovered that doing the reload or even editing and changing the configuration, doesn’t seem to cause the issue to re-surface as I was assuming it would. I will have to let it sit for a while and test again after lunch.

···

On Thursday, July 20, 2017 at 10:01:15 AM UTC-4, Jochen Kressin wrote:

Yes - the cache is cleared each time you either upload new configurations, or reload the existing configuration.

Reloading the existing configuration can be done with the -rl/–reload flag like:

./sgadmin.sh -ts … -tspass … -ks … -kspass … -icl -nhnv -rl

On Thursday, July 20, 2017 at 3:58:31 PM UTC+2, Brian Sanders wrote:

Thanks for the detailed reply! I’ll quickly answer what I can and get back to you as soon as I can get these tests run.

-I do not have any other ES plugins (not even the X-pack free version)
-Nothing else in SG plugins besides the tenants
-logstash is pushing into elasticsearch. I have Cerebro for checking index status, but that is now a web app and no longer a plugin.

I’ll try local only and also get back to you with the test results you suggested.

So is there some guaranteed way I can clear the Cache? I am assuming that once cached I am good, so if I could do something to clear the cache and force the issue again I could test much quicker.

On Thursday, July 20, 2017 at 9:22:52 AM UTC-4, Jochen Kressin wrote:

Hi Brian,

we’ve analyzed the log files and tried to replicate the problem here, but no luck so far. The log file also looks fine, I can’t find anything regarding the login or the internal server error.

The first login attempt starts here:

[2017-07-13T13:58:56,298][DEBUG][c.f.s.a.BackendRegistry ] Try to extract auth creds from http basic

[2017-07-13T13:58:56,298][DEBUG][c.f.s.a.BackendRegistry ] User ‘XXX’ is in cache? false (cache size: 1)

… LDAP stuff …

[2017-07-13T13:58:56,466][DEBUG][c.f.d.a.l.b.LDAPAuthenticationBackend] Authenticated username brians

[2017-07-13T13:58:56,468][DEBUG][c.f.s.a.BackendRegistry ] User ‘User [name=XXX, roles=]’ is authenticated

which means that the authentication process takes around 150ms. Which is still a bit high, but depends on network latency and LDAP server performance. So, this seems fine. After that, the user is cached (default is 1 hour), so even the LDAP server should not have any impact anymore:

[2017-07-13T13:58:56,609][DEBUG][c.f.s.a.BackendRegistry ] Try to extract auth creds from http basic

[2017-07-13T13:58:56,609][DEBUG][c.f.s.a.BackendRegistry ] User ‘XXX’ is in cache? true (cache size: 2)

[2017-07-13T13:58:56,610][DEBUG][c.f.s.a.BackendRegistry ] User ‘User [name=XXX, roles=]’ is authenticated

Cache size seems ok, since both your user and the Kibana server user is cached. Also, there are no Exceptions in the log, which I would have expected since Kibana logs an 500 internal server error.

So, at the moment I can’t see anything in the ES log that seems related to the issues you’re facing. Also, LDAP is a major use case for Search Guard, and we did not hear about such a behaviour before. Therefore I guess the problem is not related to the LDAP module at the moment. But just to make sure:

  • If you only use the internal user database and disable LDAP, do you still see these issues?
  • Apart from Search Guard, do you have any other ES plugins installed
  • Apart from the SG Kibana plugin, do you have any other Kibana plugins installed
  • Are there any other services running against ES?

In other words - can you replicate the problem with a minimum installation of ES/SG and KI/SG Kibana plugin?

Next, let’s look at the actual HTTP requests from Kibana. Can you please:

  • Delete all log files
  • Fire up ES and KI
  • In Chrome, open the Developer tools and switch to the Network tab
  • Replicate the login problem

In the network tab, can you spot which request is actually taking so long? If you find anything there, please post (or email) the request incl. headers and payload, and the reponse? And send the ES log file as well?

Sorry for the delayed response, but since we can’t reproduce the problem we’re having a hard time pinpointing the problem.

On Tuesday, July 18, 2017 at 3:50:53 PM UTC+2, Brian Sanders wrote:

Any chance someone has been able to find anything in the logs? It very consistently fails the first login, but then after that seems to work. I have been working on getting alerting in place along side this with elastalert, and I think between your plugin and elastalert this could work for us. I just really have problems demoing the setup with login/tenant switching taking 30s due to the timeout. I have avoided upgrading to 5.5, as I don’t want to invalidate the logs I captured if someone finds something telling in them.

On Thursday, July 13, 2017 at 10:54:33 AM UTC-4, Brian Sanders wrote:

Yup, that is a good suggestion. Log file attached.

If it helps, my users that fails then succeeds is “brians”, and I have the global tenant turned off but the private tenant turned on. I have also added a few other tenants to his user to test switching.

On Thursday, July 13, 2017 at 10:40:57 AM UTC-4, Search Guard wrote:

Why not write a PGP encrypted mail? Here is our public key https://floragunn.com/search-guard-public-key/

Am 13.07.2017 um 16:10 schrieb Brian Sanders brian....@gmail.com:

I have a log file, is there some way to send it other than attaching it here? I don’t know exactly what all is in it, but I generally don’t like posting debug logs to the internet directly.

On Thursday, July 13, 2017 at 9:24:56 AM UTC-4, Brian Sanders wrote:

Thank you, I will activate this logging and see what I can get. I tried all the settings above with no difference. I also tried Firefox (was using chrome) with the hope that it was browser related, but it still does the same thing. I’ll see if I c

On Wednesday, July 12, 2017 at 11:39:57 AM UTC-4, Search Guard wrote:

Regarding “searchguard.cookie.password” pls read https://github.com/floragunncom/search-guard-docs/blob/master/kibana.md#configuring-the-search-guard-kibana-plugin

(at least in production you should not use the default)

If its still not working pls. turn debug logging on and send the logfile from elasticsearch:

PUT /_cluster/settings
{
“transient” : {
“logger._root” : “DEBUG”
}
}

Am 12.07.2017 um 16:39 schrieb Brian Sanders brian....@gmail.com:

I’ll have to try reinstalling, perhaps I’ll upgrade to the most current versions of plugins as well since this isn’t live and there are new versions.

From your suggested settings in kibana.yml

  1. what is searchguard.cookie.password I haven’t seen mention of that before

I have the other settings, and some of those are defaults so I omitted them. I will specificaly add them and see if this makes much difference before trying to re-install.

On Wednesday, July 12, 2017 at 10:04:28 AM UTC-4, Search Guard wrote:
I verified that the configuration i sent was is working locally for me.
I suggest to reinstall Kibana and the SG Kibana plugin https://github.com/floragunncom/search-guard-kibana-plugin/releases/download/v5.4.2-3/searchguard-kibana-5.4.2-3.zip

The relevant config in kibana.yml (maybe you miss that) looks like:

elasticsearch.requestHeadersWhitelist: [“sg_tenant”, “x-authenticated-user”, “authorization”, “x-forwarded-for”, “x-forwarded-server”, “x-forwarded-by”, “x-proxy-user”, “x-proxy-roles”]
searchguard.multitenancy.enabled: true
searchguard.multitenancy.show_roles: true
searchguard.multitenancy.enable_filter: true
searchguard.multitenancy.tenants.enable_global: true
searchguard.multitenancy.tenants.enable_private: true
searchguard.cookie.password: “”
elasticsearch.username: “kibanaserver”
elasticsearch.password: “kibanaserver”

see https://github.com/floragunncom/search-guard-docs/blob/master/multitenancy.md#kibana-plugin-configuration

On Wednesday, 12 July 2017 15:34:52 UTC+2, Brian Sanders wrote:
This is a configuration I have tried previously, but I tried again this morning before logging in to anything to document the exact order of tests and errors seen.

When I first log in, I do see the log message that it can not authenticate the user with ad 0 (the internal database) and it says it says it is going to the next (eg “try next”) However after an extended period the attempted login just shows
{“statusCode”:500,“error”:“Internal Server Error”,“message”:“An internal server error occurred”}

After that shows up, I simply reselect my URL and press enter again, which this time loads Kibana with out any further prompting for user input. I can perform searches on my data an things appear normal. If I select tenants tab, and change to another tenant, I receive the same:
{“statusCode”:500,“error”:“Internal Server Error”,“message”:“An internal server error occurred”}

This time pressing enter to reload the page eventually brings me back to the Search Guard login screen. After providing my ldap credentials, I am finally authentiacted and put into Kibana as the tenant. Now I can switch between the 2 tenants just fine… but if I select yet another tenant it takes a long time and eventually puts me at the Search Guard login website, at least it didn’t 500 error this time.

So it would seem that first login does this, as well as having issues with switching to a tenant for the first time as well. I am using FreeIPA as an ldap backend, which is a fairly popular user management group of packages, but as far as this is concerned, it is simply authing against the LDAP portion which clearly CAN work. It just doesn’t seem to work on the first try.

On Tuesday, July 11, 2017 at 4:01:37 PM UTC-4, Search Guard wrote:
can you try this?

searchguard:
dynamic:
kibana:
multitenancy_enabled: true
server_username: “kibanaserver”
index: ‘.kibana’
http:
anonymous_auth_enabled: false
xff:
enabled: false
remoteIpHeader: ‘x-forwarded-for’
proxiesHeader: ‘x-forwarded-by’
authc:
basic_internal_auth_domain:
enabled: true
order: 0
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: intern
ldap:
enabled: true
order: 1
http_authenticator:
type: basic
challenge: false
authentication_backend:
config:
enable_ssl: false
enable_start_tls: true
enable_ssl_client_auth: false
verify_hostnames: false
hosts:
- ipa1.example.com:389

        bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
        password: 'FAKEPASS'
        userbase: 'cn=users,cn=accounts,dc=example,dc=com'
        usersearch: '(uid={0})'
        username_attribute: uid
       
        #only valid for authz (not authc)
        #skip_users:
        #  - kibanaserver
        #  - logstash

(more complete example here https://gist.github.com/floragunncom/fdad5c8f1cab838962842d590c408538)

On Monday, 10 July 2017 21:56:25 UTC+2, Brian Sanders wrote:
Seems I didn’t include the Search Guard version: search-guard-5:5.4.2-12

On Monday, July 10, 2017 at 3:55:25 PM UTC-4, Brian Sanders wrote:
Sorry I just saw this question, I never had the issue with the local only accounts. I didn’t test that extensively as I was looking to test the LDAP, but it seemed that it worked just fine.

We are running the current versions:

ES: 5.4.2
kibana-multitenancy-5.4-3
authbackend-ldap-5.0-7

And after looking these up it would appear there is now a 5.5 version of elasticsearch. This being in dev, I can always attempt to upgrade, but these were all the latest available versions as of sometime last week when I was starting my build.

On Monday, July 10, 2017 at 1:51:24 PM UTC-4, Search Guard wrote:
Which ES/SG/LDAP/MT versions are you using?

Does it work without LDAP (internal users only)?

Am 10.07.2017 um 16:56 schrieb Brian Sanders brian....@gmail.com:

I am testing out the LDAP authentication module with Kibana multi tenancy to see if this will fit our needs, and I am having some strange login issues. I believe I have configured it as described in the example documentation, but the FIRST time I attempt to log in I always end up at the 500 internal server error. If I attempt to log in a few more times, eventually it works and seems to keep working for a bit. I have also logged in and then seen this issue when switching tenants. After logging back in a second time everything works including switching.

I don’t really know how to proceed troubleshooting this, but I can’t demo this to anyone and expect them to want me to take this live to production if LDAP is flaky.

I am still using the basic_internal_auth, since I don’t want to add a kibanaserver or logstash user to my LDAP directory, and would prefer those just be in the local database. Authz is being done by mapping users to rolls, which does work once I am logged in.

Here is the relevant section from my sg_config.yml

searchguard:
dynamic:
kibana:
multitenancy_enabled: true
server_username: “kibanaserver”
index: ‘.kibana’
http:
anonymous_auth_enabled: false
xff:
enabled: false
remoteIpHeader: ‘x-forwarded-for’
proxiesHeader: ‘x-forwarded-by’
authc:
basic_internal_auth_domain:
enabled: true
order: 4
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: intern
ldap:
enabled: true
order: 1
http_authenticator:
type: basic
challenge: true
authentication_backend:
config:
enable_ssl: false
enable_start_tls: true
enable_ssl_client_auth: false
verify_hostnames: false
hosts:
- ipa1.example.com:389

        bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
        password: 'FAKEPASS'
        userbase: 'cn=users,cn=accounts,dc=example,dc=com'
        usersearch: '(uid={0})'
        username_attribute: uid
        skip_users:
          - kibanaserver
          - logstash


You received this message because you are subscribed to the Google Groups “Search Guard” group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/9eba0f76-cbbe-4ea7-85f2-a2a90d750f26%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups “Search Guard” group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/d487734a-1beb-4a13-93d1-4686a32fc203%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups “Search Guard” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/4c3d66b4-bdd4-4616-93a4-7fe9ea926f6c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

I was able to see the issue again this morning after clearing logs from kibana and elasticsearch. I captured all the data from the chrome developer tools and have exported them (it seems to be a .har file json formatted). I have attached all the logs to this with the searchguard gpgkey encryption again.

For this test I did get the error, and developer tools shows the timeout of 30s before I get the 500 internal error. I then left the capture running, and logged in a second time. This second attempt works. So this log should contain both the failed 1st and then successful 2nd login attempt. I did not then attempt switching tenants (which generally gives the same issue the first time).

I can try disabeling LDAP module all together, but if clearing the cache doesn’t cause the error on demand, it will be difficult for me to say that disabeling it works, or if it just works because I have already logged in recently.

debuglog2.gpg (628 KB)

···

On Thursday, July 20, 2017 at 11:00:30 AM UTC-4, Brian Sanders wrote:

I was about to reply with tests results, when I discovered that doing the reload or even editing and changing the configuration, doesn’t seem to cause the issue to re-surface as I was assuming it would. I will have to let it sit for a while and test again after lunch.

On Thursday, July 20, 2017 at 10:01:15 AM UTC-4, Jochen Kressin wrote:

Yes - the cache is cleared each time you either upload new configurations, or reload the existing configuration.

Reloading the existing configuration can be done with the -rl/–reload flag like:

./sgadmin.sh -ts … -tspass … -ks … -kspass … -icl -nhnv -rl

On Thursday, July 20, 2017 at 3:58:31 PM UTC+2, Brian Sanders wrote:

Thanks for the detailed reply! I’ll quickly answer what I can and get back to you as soon as I can get these tests run.

-I do not have any other ES plugins (not even the X-pack free version)
-Nothing else in SG plugins besides the tenants
-logstash is pushing into elasticsearch. I have Cerebro for checking index status, but that is now a web app and no longer a plugin.

I’ll try local only and also get back to you with the test results you suggested.

So is there some guaranteed way I can clear the Cache? I am assuming that once cached I am good, so if I could do something to clear the cache and force the issue again I could test much quicker.

On Thursday, July 20, 2017 at 9:22:52 AM UTC-4, Jochen Kressin wrote:

Hi Brian,

we’ve analyzed the log files and tried to replicate the problem here, but no luck so far. The log file also looks fine, I can’t find anything regarding the login or the internal server error.

The first login attempt starts here:

[2017-07-13T13:58:56,298][DEBUG][c.f.s.a.BackendRegistry ] Try to extract auth creds from http basic

[2017-07-13T13:58:56,298][DEBUG][c.f.s.a.BackendRegistry ] User ‘XXX’ is in cache? false (cache size: 1)

… LDAP stuff …

[2017-07-13T13:58:56,466][DEBUG][c.f.d.a.l.b.LDAPAuthenticationBackend] Authenticated username brians

[2017-07-13T13:58:56,468][DEBUG][c.f.s.a.BackendRegistry ] User ‘User [name=XXX, roles=]’ is authenticated

which means that the authentication process takes around 150ms. Which is still a bit high, but depends on network latency and LDAP server performance. So, this seems fine. After that, the user is cached (default is 1 hour), so even the LDAP server should not have any impact anymore:

[2017-07-13T13:58:56,609][DEBUG][c.f.s.a.BackendRegistry ] Try to extract auth creds from http basic

[2017-07-13T13:58:56,609][DEBUG][c.f.s.a.BackendRegistry ] User ‘XXX’ is in cache? true (cache size: 2)

[2017-07-13T13:58:56,610][DEBUG][c.f.s.a.BackendRegistry ] User ‘User [name=XXX, roles=]’ is authenticated

Cache size seems ok, since both your user and the Kibana server user is cached. Also, there are no Exceptions in the log, which I would have expected since Kibana logs an 500 internal server error.

So, at the moment I can’t see anything in the ES log that seems related to the issues you’re facing. Also, LDAP is a major use case for Search Guard, and we did not hear about such a behaviour before. Therefore I guess the problem is not related to the LDAP module at the moment. But just to make sure:

  • If you only use the internal user database and disable LDAP, do you still see these issues?
  • Apart from Search Guard, do you have any other ES plugins installed
  • Apart from the SG Kibana plugin, do you have any other Kibana plugins installed
  • Are there any other services running against ES?

In other words - can you replicate the problem with a minimum installation of ES/SG and KI/SG Kibana plugin?

Next, let’s look at the actual HTTP requests from Kibana. Can you please:

  • Delete all log files
  • Fire up ES and KI
  • In Chrome, open the Developer tools and switch to the Network tab
  • Replicate the login problem

In the network tab, can you spot which request is actually taking so long? If you find anything there, please post (or email) the request incl. headers and payload, and the reponse? And send the ES log file as well?

Sorry for the delayed response, but since we can’t reproduce the problem we’re having a hard time pinpointing the problem.

On Tuesday, July 18, 2017 at 3:50:53 PM UTC+2, Brian Sanders wrote:

Any chance someone has been able to find anything in the logs? It very consistently fails the first login, but then after that seems to work. I have been working on getting alerting in place along side this with elastalert, and I think between your plugin and elastalert this could work for us. I just really have problems demoing the setup with login/tenant switching taking 30s due to the timeout. I have avoided upgrading to 5.5, as I don’t want to invalidate the logs I captured if someone finds something telling in them.

On Thursday, July 13, 2017 at 10:54:33 AM UTC-4, Brian Sanders wrote:

Yup, that is a good suggestion. Log file attached.

If it helps, my users that fails then succeeds is “brians”, and I have the global tenant turned off but the private tenant turned on. I have also added a few other tenants to his user to test switching.

On Thursday, July 13, 2017 at 10:40:57 AM UTC-4, Search Guard wrote:

Why not write a PGP encrypted mail? Here is our public key https://floragunn.com/search-guard-public-key/

Am 13.07.2017 um 16:10 schrieb Brian Sanders brian....@gmail.com:

I have a log file, is there some way to send it other than attaching it here? I don’t know exactly what all is in it, but I generally don’t like posting debug logs to the internet directly.

On Thursday, July 13, 2017 at 9:24:56 AM UTC-4, Brian Sanders wrote:

Thank you, I will activate this logging and see what I can get. I tried all the settings above with no difference. I also tried Firefox (was using chrome) with the hope that it was browser related, but it still does the same thing. I’ll see if I c

On Wednesday, July 12, 2017 at 11:39:57 AM UTC-4, Search Guard wrote:

Regarding “searchguard.cookie.password” pls read https://github.com/floragunncom/search-guard-docs/blob/master/kibana.md#configuring-the-search-guard-kibana-plugin

(at least in production you should not use the default)

If its still not working pls. turn debug logging on and send the logfile from elasticsearch:

PUT /_cluster/settings
{
“transient” : {
“logger._root” : “DEBUG”
}
}

Am 12.07.2017 um 16:39 schrieb Brian Sanders brian....@gmail.com:

I’ll have to try reinstalling, perhaps I’ll upgrade to the most current versions of plugins as well since this isn’t live and there are new versions.

From your suggested settings in kibana.yml

  1. what is searchguard.cookie.password I haven’t seen mention of that before

I have the other settings, and some of those are defaults so I omitted them. I will specificaly add them and see if this makes much difference before trying to re-install.

On Wednesday, July 12, 2017 at 10:04:28 AM UTC-4, Search Guard wrote:
I verified that the configuration i sent was is working locally for me.
I suggest to reinstall Kibana and the SG Kibana plugin https://github.com/floragunncom/search-guard-kibana-plugin/releases/download/v5.4.2-3/searchguard-kibana-5.4.2-3.zip

The relevant config in kibana.yml (maybe you miss that) looks like:

elasticsearch.requestHeadersWhitelist: [“sg_tenant”, “x-authenticated-user”, “authorization”, “x-forwarded-for”, “x-forwarded-server”, “x-forwarded-by”, “x-proxy-user”, “x-proxy-roles”]
searchguard.multitenancy.enabled: true
searchguard.multitenancy.show_roles: true
searchguard.multitenancy.enable_filter: true
searchguard.multitenancy.tenants.enable_global: true
searchguard.multitenancy.tenants.enable_private: true
searchguard.cookie.password: “”
elasticsearch.username: “kibanaserver”
elasticsearch.password: “kibanaserver”

see https://github.com/floragunncom/search-guard-docs/blob/master/multitenancy.md#kibana-plugin-configuration

On Wednesday, 12 July 2017 15:34:52 UTC+2, Brian Sanders wrote:
This is a configuration I have tried previously, but I tried again this morning before logging in to anything to document the exact order of tests and errors seen.

When I first log in, I do see the log message that it can not authenticate the user with ad 0 (the internal database) and it says it says it is going to the next (eg “try next”) However after an extended period the attempted login just shows
{“statusCode”:500,“error”:“Internal Server Error”,“message”:“An internal server error occurred”}

After that shows up, I simply reselect my URL and press enter again, which this time loads Kibana with out any further prompting for user input. I can perform searches on my data an things appear normal. If I select tenants tab, and change to another tenant, I receive the same:
{“statusCode”:500,“error”:“Internal Server Error”,“message”:“An internal server error occurred”}

This time pressing enter to reload the page eventually brings me back to the Search Guard login screen. After providing my ldap credentials, I am finally authentiacted and put into Kibana as the tenant. Now I can switch between the 2 tenants just fine… but if I select yet another tenant it takes a long time and eventually puts me at the Search Guard login website, at least it didn’t 500 error this time.

So it would seem that first login does this, as well as having issues with switching to a tenant for the first time as well. I am using FreeIPA as an ldap backend, which is a fairly popular user management group of packages, but as far as this is concerned, it is simply authing against the LDAP portion which clearly CAN work. It just doesn’t seem to work on the first try.

On Tuesday, July 11, 2017 at 4:01:37 PM UTC-4, Search Guard wrote:
can you try this?

searchguard:
dynamic:
kibana:
multitenancy_enabled: true
server_username: “kibanaserver”
index: ‘.kibana’
http:
anonymous_auth_enabled: false
xff:
enabled: false
remoteIpHeader: ‘x-forwarded-for’
proxiesHeader: ‘x-forwarded-by’
authc:
basic_internal_auth_domain:
enabled: true
order: 0
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: intern
ldap:
enabled: true
order: 1
http_authenticator:
type: basic
challenge: false
authentication_backend:
config:
enable_ssl: false
enable_start_tls: true
enable_ssl_client_auth: false
verify_hostnames: false
hosts:
- ipa1.example.com:389

        bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
        password: 'FAKEPASS'
        userbase: 'cn=users,cn=accounts,dc=example,dc=com'
        usersearch: '(uid={0})'
        username_attribute: uid
       
        #only valid for authz (not authc)
        #skip_users:
        #  - kibanaserver
        #  - logstash

(more complete example here https://gist.github.com/floragunncom/fdad5c8f1cab838962842d590c408538)

On Monday, 10 July 2017 21:56:25 UTC+2, Brian Sanders wrote:
Seems I didn’t include the Search Guard version: search-guard-5:5.4.2-12

On Monday, July 10, 2017 at 3:55:25 PM UTC-4, Brian Sanders wrote:
Sorry I just saw this question, I never had the issue with the local only accounts. I didn’t test that extensively as I was looking to test the LDAP, but it seemed that it worked just fine.

We are running the current versions:

ES: 5.4.2
kibana-multitenancy-5.4-3
authbackend-ldap-5.0-7

And after looking these up it would appear there is now a 5.5 version of elasticsearch. This being in dev, I can always attempt to upgrade, but these were all the latest available versions as of sometime last week when I was starting my build.

On Monday, July 10, 2017 at 1:51:24 PM UTC-4, Search Guard wrote:
Which ES/SG/LDAP/MT versions are you using?

Does it work without LDAP (internal users only)?

Am 10.07.2017 um 16:56 schrieb Brian Sanders brian....@gmail.com:

I am testing out the LDAP authentication module with Kibana multi tenancy to see if this will fit our needs, and I am having some strange login issues. I believe I have configured it as described in the example documentation, but the FIRST time I attempt to log in I always end up at the 500 internal server error. If I attempt to log in a few more times, eventually it works and seems to keep working for a bit. I have also logged in and then seen this issue when switching tenants. After logging back in a second time everything works including switching.

I don’t really know how to proceed troubleshooting this, but I can’t demo this to anyone and expect them to want me to take this live to production if LDAP is flaky.

I am still using the basic_internal_auth, since I don’t want to add a kibanaserver or logstash user to my LDAP directory, and would prefer those just be in the local database. Authz is being done by mapping users to rolls, which does work once I am logged in.

Here is the relevant section from my sg_config.yml

searchguard:
dynamic:
kibana:
multitenancy_enabled: true
server_username: “kibanaserver”
index: ‘.kibana’
http:
anonymous_auth_enabled: false
xff:
enabled: false
remoteIpHeader: ‘x-forwarded-for’
proxiesHeader: ‘x-forwarded-by’
authc:
basic_internal_auth_domain:
enabled: true
order: 4
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: intern
ldap:
enabled: true
order: 1
http_authenticator:
type: basic
challenge: true
authentication_backend:
config:
enable_ssl: false
enable_start_tls: true
enable_ssl_client_auth: false
verify_hostnames: false
hosts:
- ipa1.example.com:389

        bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
        password: 'FAKEPASS'
        userbase: 'cn=users,cn=accounts,dc=example,dc=com'
        usersearch: '(uid={0})'
        username_attribute: uid
        skip_users:
          - kibanaserver
          - logstash


You received this message because you are subscribed to the Google Groups “Search Guard” group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/9eba0f76-cbbe-4ea7-85f2-a2a90d750f26%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups “Search Guard” group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/d487734a-1beb-4a13-93d1-4686a32fc203%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups “Search Guard” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/4c3d66b4-bdd4-4616-93a4-7fe9ea926f6c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Hi Brian,

I have a couple of questions regarding the logs - since I don’t want to post your hostnames and IPs and other internal content here, can we move this over to email? I’d post the end results here for other users once we figured out the problem.

···

On Friday, July 21, 2017 at 3:35:10 PM UTC+2, Brian Sanders wrote:

I was able to see the issue again this morning after clearing logs from kibana and elasticsearch. I captured all the data from the chrome developer tools and have exported them (it seems to be a .har file json formatted). I have attached all the logs to this with the searchguard gpgkey encryption again.

For this test I did get the error, and developer tools shows the timeout of 30s before I get the 500 internal error. I then left the capture running, and logged in a second time. This second attempt works. So this log should contain both the failed 1st and then successful 2nd login attempt. I did not then attempt switching tenants (which generally gives the same issue the first time).

I can try disabeling LDAP module all together, but if clearing the cache doesn’t cause the error on demand, it will be difficult for me to say that disabeling it works, or if it just works because I have already logged in recently.

On Thursday, July 20, 2017 at 11:00:30 AM UTC-4, Brian Sanders wrote:

I was about to reply with tests results, when I discovered that doing the reload or even editing and changing the configuration, doesn’t seem to cause the issue to re-surface as I was assuming it would. I will have to let it sit for a while and test again after lunch.

On Thursday, July 20, 2017 at 10:01:15 AM UTC-4, Jochen Kressin wrote:

Yes - the cache is cleared each time you either upload new configurations, or reload the existing configuration.

Reloading the existing configuration can be done with the -rl/–reload flag like:

./sgadmin.sh -ts … -tspass … -ks … -kspass … -icl -nhnv -rl

On Thursday, July 20, 2017 at 3:58:31 PM UTC+2, Brian Sanders wrote:

Thanks for the detailed reply! I’ll quickly answer what I can and get back to you as soon as I can get these tests run.

-I do not have any other ES plugins (not even the X-pack free version)
-Nothing else in SG plugins besides the tenants
-logstash is pushing into elasticsearch. I have Cerebro for checking index status, but that is now a web app and no longer a plugin.

I’ll try local only and also get back to you with the test results you suggested.

So is there some guaranteed way I can clear the Cache? I am assuming that once cached I am good, so if I could do something to clear the cache and force the issue again I could test much quicker.

On Thursday, July 20, 2017 at 9:22:52 AM UTC-4, Jochen Kressin wrote:

Hi Brian,

we’ve analyzed the log files and tried to replicate the problem here, but no luck so far. The log file also looks fine, I can’t find anything regarding the login or the internal server error.

The first login attempt starts here:

[2017-07-13T13:58:56,298][DEBUG][c.f.s.a.BackendRegistry ] Try to extract auth creds from http basic

[2017-07-13T13:58:56,298][DEBUG][c.f.s.a.BackendRegistry ] User ‘XXX’ is in cache? false (cache size: 1)

… LDAP stuff …

[2017-07-13T13:58:56,466][DEBUG][c.f.d.a.l.b.LDAPAuthenticationBackend] Authenticated username brians

[2017-07-13T13:58:56,468][DEBUG][c.f.s.a.BackendRegistry ] User ‘User [name=XXX, roles=]’ is authenticated

which means that the authentication process takes around 150ms. Which is still a bit high, but depends on network latency and LDAP server performance. So, this seems fine. After that, the user is cached (default is 1 hour), so even the LDAP server should not have any impact anymore:

[2017-07-13T13:58:56,609][DEBUG][c.f.s.a.BackendRegistry ] Try to extract auth creds from http basic

[2017-07-13T13:58:56,609][DEBUG][c.f.s.a.BackendRegistry ] User ‘XXX’ is in cache? true (cache size: 2)

[2017-07-13T13:58:56,610][DEBUG][c.f.s.a.BackendRegistry ] User ‘User [name=XXX, roles=]’ is authenticated

Cache size seems ok, since both your user and the Kibana server user is cached. Also, there are no Exceptions in the log, which I would have expected since Kibana logs an 500 internal server error.

So, at the moment I can’t see anything in the ES log that seems related to the issues you’re facing. Also, LDAP is a major use case for Search Guard, and we did not hear about such a behaviour before. Therefore I guess the problem is not related to the LDAP module at the moment. But just to make sure:

  • If you only use the internal user database and disable LDAP, do you still see these issues?
  • Apart from Search Guard, do you have any other ES plugins installed
  • Apart from the SG Kibana plugin, do you have any other Kibana plugins installed
  • Are there any other services running against ES?

In other words - can you replicate the problem with a minimum installation of ES/SG and KI/SG Kibana plugin?

Next, let’s look at the actual HTTP requests from Kibana. Can you please:

  • Delete all log files
  • Fire up ES and KI
  • In Chrome, open the Developer tools and switch to the Network tab
  • Replicate the login problem

In the network tab, can you spot which request is actually taking so long? If you find anything there, please post (or email) the request incl. headers and payload, and the reponse? And send the ES log file as well?

Sorry for the delayed response, but since we can’t reproduce the problem we’re having a hard time pinpointing the problem.

On Tuesday, July 18, 2017 at 3:50:53 PM UTC+2, Brian Sanders wrote:

Any chance someone has been able to find anything in the logs? It very consistently fails the first login, but then after that seems to work. I have been working on getting alerting in place along side this with elastalert, and I think between your plugin and elastalert this could work for us. I just really have problems demoing the setup with login/tenant switching taking 30s due to the timeout. I have avoided upgrading to 5.5, as I don’t want to invalidate the logs I captured if someone finds something telling in them.

On Thursday, July 13, 2017 at 10:54:33 AM UTC-4, Brian Sanders wrote:

Yup, that is a good suggestion. Log file attached.

If it helps, my users that fails then succeeds is “brians”, and I have the global tenant turned off but the private tenant turned on. I have also added a few other tenants to his user to test switching.

On Thursday, July 13, 2017 at 10:40:57 AM UTC-4, Search Guard wrote:

Why not write a PGP encrypted mail? Here is our public key https://floragunn.com/search-guard-public-key/

Am 13.07.2017 um 16:10 schrieb Brian Sanders brian....@gmail.com:

I have a log file, is there some way to send it other than attaching it here? I don’t know exactly what all is in it, but I generally don’t like posting debug logs to the internet directly.

On Thursday, July 13, 2017 at 9:24:56 AM UTC-4, Brian Sanders wrote:

Thank you, I will activate this logging and see what I can get. I tried all the settings above with no difference. I also tried Firefox (was using chrome) with the hope that it was browser related, but it still does the same thing. I’ll see if I c

On Wednesday, July 12, 2017 at 11:39:57 AM UTC-4, Search Guard wrote:

Regarding “searchguard.cookie.password” pls read https://github.com/floragunncom/search-guard-docs/blob/master/kibana.md#configuring-the-search-guard-kibana-plugin

(at least in production you should not use the default)

If its still not working pls. turn debug logging on and send the logfile from elasticsearch:

PUT /_cluster/settings
{
“transient” : {
“logger._root” : “DEBUG”
}
}

Am 12.07.2017 um 16:39 schrieb Brian Sanders brian....@gmail.com:

I’ll have to try reinstalling, perhaps I’ll upgrade to the most current versions of plugins as well since this isn’t live and there are new versions.

From your suggested settings in kibana.yml

  1. what is searchguard.cookie.password I haven’t seen mention of that before

I have the other settings, and some of those are defaults so I omitted them. I will specificaly add them and see if this makes much difference before trying to re-install.

On Wednesday, July 12, 2017 at 10:04:28 AM UTC-4, Search Guard wrote:
I verified that the configuration i sent was is working locally for me.
I suggest to reinstall Kibana and the SG Kibana plugin https://github.com/floragunncom/search-guard-kibana-plugin/releases/download/v5.4.2-3/searchguard-kibana-5.4.2-3.zip

The relevant config in kibana.yml (maybe you miss that) looks like:

elasticsearch.requestHeadersWhitelist: [“sg_tenant”, “x-authenticated-user”, “authorization”, “x-forwarded-for”, “x-forwarded-server”, “x-forwarded-by”, “x-proxy-user”, “x-proxy-roles”]
searchguard.multitenancy.enabled: true
searchguard.multitenancy.show_roles: true
searchguard.multitenancy.enable_filter: true
searchguard.multitenancy.tenants.enable_global: true
searchguard.multitenancy.tenants.enable_private: true
searchguard.cookie.password: “”
elasticsearch.username: “kibanaserver”
elasticsearch.password: “kibanaserver”

see https://github.com/floragunncom/search-guard-docs/blob/master/multitenancy.md#kibana-plugin-configuration

On Wednesday, 12 July 2017 15:34:52 UTC+2, Brian Sanders wrote:
This is a configuration I have tried previously, but I tried again this morning before logging in to anything to document the exact order of tests and errors seen.

When I first log in, I do see the log message that it can not authenticate the user with ad 0 (the internal database) and it says it says it is going to the next (eg “try next”) However after an extended period the attempted login just shows
{“statusCode”:500,“error”:“Internal Server Error”,“message”:“An internal server error occurred”}

After that shows up, I simply reselect my URL and press enter again, which this time loads Kibana with out any further prompting for user input. I can perform searches on my data an things appear normal. If I select tenants tab, and change to another tenant, I receive the same:
{“statusCode”:500,“error”:“Internal Server Error”,“message”:“An internal server error occurred”}

This time pressing enter to reload the page eventually brings me back to the Search Guard login screen. After providing my ldap credentials, I am finally authentiacted and put into Kibana as the tenant. Now I can switch between the 2 tenants just fine… but if I select yet another tenant it takes a long time and eventually puts me at the Search Guard login website, at least it didn’t 500 error this time.

So it would seem that first login does this, as well as having issues with switching to a tenant for the first time as well. I am using FreeIPA as an ldap backend, which is a fairly popular user management group of packages, but as far as this is concerned, it is simply authing against the LDAP portion which clearly CAN work. It just doesn’t seem to work on the first try.

On Tuesday, July 11, 2017 at 4:01:37 PM UTC-4, Search Guard wrote:
can you try this?

searchguard:
dynamic:
kibana:
multitenancy_enabled: true
server_username: “kibanaserver”
index: ‘.kibana’
http:
anonymous_auth_enabled: false
xff:
enabled: false
remoteIpHeader: ‘x-forwarded-for’
proxiesHeader: ‘x-forwarded-by’
authc:
basic_internal_auth_domain:
enabled: true
order: 0
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: intern
ldap:
enabled: true
order: 1
http_authenticator:
type: basic
challenge: false
authentication_backend:
config:
enable_ssl: false
enable_start_tls: true
enable_ssl_client_auth: false
verify_hostnames: false
hosts:
- ipa1.example.com:389

        bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
        password: 'FAKEPASS'
        userbase: 'cn=users,cn=accounts,dc=example,dc=com'
        usersearch: '(uid={0})'
        username_attribute: uid
       
        #only valid for authz (not authc)
        #skip_users:
        #  - kibanaserver
        #  - logstash

(more complete example here https://gist.github.com/floragunncom/fdad5c8f1cab838962842d590c408538)

On Monday, 10 July 2017 21:56:25 UTC+2, Brian Sanders wrote:
Seems I didn’t include the Search Guard version: search-guard-5:5.4.2-12

On Monday, July 10, 2017 at 3:55:25 PM UTC-4, Brian Sanders wrote:
Sorry I just saw this question, I never had the issue with the local only accounts. I didn’t test that extensively as I was looking to test the LDAP, but it seemed that it worked just fine.

We are running the current versions:

ES: 5.4.2
kibana-multitenancy-5.4-3
authbackend-ldap-5.0-7

And after looking these up it would appear there is now a 5.5 version of elasticsearch. This being in dev, I can always attempt to upgrade, but these were all the latest available versions as of sometime last week when I was starting my build.

On Monday, July 10, 2017 at 1:51:24 PM UTC-4, Search Guard wrote:
Which ES/SG/LDAP/MT versions are you using?

Does it work without LDAP (internal users only)?

Am 10.07.2017 um 16:56 schrieb Brian Sanders brian....@gmail.com:

I am testing out the LDAP authentication module with Kibana multi tenancy to see if this will fit our needs, and I am having some strange login issues. I believe I have configured it as described in the example documentation, but the FIRST time I attempt to log in I always end up at the 500 internal server error. If I attempt to log in a few more times, eventually it works and seems to keep working for a bit. I have also logged in and then seen this issue when switching tenants. After logging back in a second time everything works including switching.

I don’t really know how to proceed troubleshooting this, but I can’t demo this to anyone and expect them to want me to take this live to production if LDAP is flaky.

I am still using the basic_internal_auth, since I don’t want to add a kibanaserver or logstash user to my LDAP directory, and would prefer those just be in the local database. Authz is being done by mapping users to rolls, which does work once I am logged in.

Here is the relevant section from my sg_config.yml

searchguard:
dynamic:
kibana:
multitenancy_enabled: true
server_username: “kibanaserver”
index: ‘.kibana’
http:
anonymous_auth_enabled: false
xff:
enabled: false
remoteIpHeader: ‘x-forwarded-for’
proxiesHeader: ‘x-forwarded-by’
authc:
basic_internal_auth_domain:
enabled: true
order: 4
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: intern
ldap:
enabled: true
order: 1
http_authenticator:
type: basic
challenge: true
authentication_backend:
config:
enable_ssl: false
enable_start_tls: true
enable_ssl_client_auth: false
verify_hostnames: false
hosts:
- ipa1.example.com:389

        bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'
        password: 'FAKEPASS'
        userbase: 'cn=users,cn=accounts,dc=example,dc=com'
        usersearch: '(uid={0})'
        username_attribute: uid
        skip_users:
          - kibanaserver
          - logstash


You received this message because you are subscribed to the Google Groups “Search Guard” group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/9eba0f76-cbbe-4ea7-85f2-a2a90d750f26%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups “Search Guard” group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/d487734a-1beb-4a13-93d1-4686a32fc203%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups “Search Guard” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/4c3d66b4-bdd4-4616-93a4-7fe9ea926f6c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.