Internal user password change

Search Guard
1

Hello all,

I have a problem with change of default admin password.

Please have a look on performed steps:

  1. New password hash generated for admin

[root@logme search-guard-6]# ./tools/hash.sh -p Testpass123
WARNING: JAVA_HOME not set, will use /usr/bin/java
$2y$12$Zu9AUEoMvH4uUf66eT2Kme0gkgG7Q199Qq4CkxeZb/KQmJ8QIF7HW

  1. Hash replaced in sg_internal_users.yml

[root@logme search-guard-6]# vi sgconfig/sg_internal_users.yml

This is the internal user database

The hash value is a bcrypt hash and can be generated with plugin/tools/hash.sh

#password is: admin
admin:
readonly: true
hash: $2y$12$Zu9AUEoMvH4uUf66eT2Kme0gkgG7Q199Qq4CkxeZb/KQmJ8QIF7HW
roles:
- admin
attributes:
#no dots allowed in attribute names
attribute1: value1
attribute2: value2
attribute3: value3

#password is: logstash
logstash:
hash: $2a$12$u1ShR4l4uBS3Uv59Pa2y5.1uQuZBrZtmNfqB3iM/.jL0XoV9sghS2
roles:
- logstash

#password is: kibanaserver
kibanaserver:
readonly: true
hash: $2y$12$Z.gYbD8uEgnSDWsGmj/BPuvw.jHQVCl1/MEdcNpiIZFR.mVQcQPoK

#password is: kibanaro
kibanaro:
hash: $2a$12$JJSXNfTowz7Uu5ttXfeYpeYE0arACvcwlPBStB1F.MI7f0U9Z4DGC
roles:
- kibanauser
- readall

#password is: readall
readall:
hash: $2a$12$ae4ycwzwvLtZxwZ82RmiEunBbIPiAmGZduBAjKN0TXdwQFtCwARz2
#password is: readall
roles:
- readall

#password is: snapshotrestore
snapshotrestore:
hash: $2y$12$DpwmetHKwgYnorbgdvORCenv4NAK8cPUg8AI6pxLCuWf/ALc0.v7W
roles:
- snapshotrestore

  1. Configuretion reloaded:

[root@logme search-guard-6]# ./tools/sgadmin.sh -cd sgconfig -cacert /etc/elasticsearch/root-ca.pem -cert /root/certificates/admin.pem -key /root/certificates/admin.key -keypass ******** -nhnv -icl
WARNING: JAVA_HOME not set, will use /usr/bin/java
Search Guard Admin v6
Will connect to localhost:9300 … done
Elasticsearch Version: 6.5.1
Search Guard Version: 6.5.1-23.2
Connected as CN=admin.xxx.pl,OU=Ops,O=xxx,DC=xxx,DC=pl
Contacting elasticsearch cluster ‘elasticsearch’ and wait for YELLOW clusterstate …
Clustername: tntcluster
Clusterstate: YELLOW
Number of nodes: 1
Number of data nodes: 1
searchguard index already exists, so we do not need to create one.
Populate config from /usr/share/elasticsearch/plugins/search-guard-6/sgconfig
Will update ‘sg/config’ with sgconfig/sg_config.yml
SUCC: Configuration for ‘config’ created or updated
Will update ‘sg/roles’ with sgconfig/sg_roles.yml
SUCC: Configuration for ‘roles’ created or updated
Will update ‘sg/rolesmapping’ with sgconfig/sg_roles_mapping.yml
SUCC: Configuration for ‘rolesmapping’ created or updated
Will update ‘sg/internalusers’ with sgconfig/sg_internal_users.yml
SUCC: Configuration for ‘internalusers’ created or updated
Will update ‘sg/actiongroups’ with sgconfig/sg_action_groups.yml
SUCC: Configuration for ‘actiongroups’ created or updated
Done with success

  1. When i try to log in to kibana with new password im redirected to login page all the time.

  2. When i revert password to default admin everything works ok.

Please find debug log from elasticsearch below.

[2018-11-28T20:13:27,125][DEBUG][c.f.s.a.BackendRegistry ] [node1] Can not authenticate admin due to com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]
com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2050) ~[guava-25.1-jre.jar:?]
at com.google.common.cache.LocalCache.get(LocalCache.java:3951) ~[guava-25.1-jre.jar:?]
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4870) ~[guava-25.1-jre.jar:?]
at com.floragunn.searchguard.auth.BackendRegistry.authcz(BackendRegistry.java:601) [search-guard-6-6.5.1-23.2.jar:6.5.1-23.2]
at com.floragunn.searchguard.auth.BackendRegistry.authenticate(BackendRegistry.java:459) [search-guard-6-6.5.1-23.2.jar:6.5.1-23.2]
at com.floragunn.searchguard.filter.SearchGuardRestFilter.checkAndAuthenticateRequest(SearchGuardRestFilter.java:134) [search-guard-6-6.5.1-23.2.jar:6.5.1-23.2]
at com.floragunn.searchguard.filter.SearchGuardRestFilter.access$000(SearchGuardRestFilter.java:50) [search-guard-6-6.5.1-23.2.jar:6.5.1-23.2]
at com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:80) [search-guard-6-6.5.1-23.2.jar:6.5.1-23.2]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:239) [elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:335) [elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:173) [elasticsearch-6.5.1.jar:6.5.1]
at com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [search-guard-ssl-6.5.1-25.5.jar:6.5.1-25.5]
at org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:545) [transport-netty4-client-6.5.1.jar:6.5.1]
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:137) [transport-netty4-client-6.5.1.jar:6.5.1]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68) [transport-netty4-client-6.5.1.jar:6.5.1]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1429) [netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199) [netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243) [netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) [netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) [netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) [netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-common-4.1.30.Final.jar:4.1.30.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
Caused by: org.elasticsearch.ElasticsearchSecurityException: password does not match
at com.floragunn.searchguard.auth.internal.InternalAuthenticationBackend.authenticate(InternalAuthenticationBackend.java:133) ~[search-guard-6-6.5.1-23.2.jar:6.5.1-23.2]
at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:607) ~[search-guard-6-6.5.1-23.2.jar:6.5.1-23.2]
at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:601) ~[search-guard-6-6.5.1-23.2.jar:6.5.1-23.2]
at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4875) ~[guava-25.1-jre.jar:?]
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-25.1-jre.jar:?]
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2276) ~[guava-25.1-jre.jar:?]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154) ~[guava-25.1-jre.jar:?]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044) ~[guava-25.1-jre.jar:?]
… 62 more
[2018-11-28T20:13:27,128][DEBUG][c.f.s.a.BackendRegistry ] [node1] Cannot authenticate user admin (or add roles) with authdomain internal/4, try next
[2018-11-28T20:13:27,128][DEBUG][c.f.s.a.BackendRegistry ] [node1] User still not authenticated after checking 1 auth domains
[2018-11-28T20:13:27,128][DEBUG][c.f.s.a.BackendRegistry ] [node1] Rerequest with class com.floragunn.searchguard.http.HTTPBasicAuthenticator
[2018-11-28T20:13:27,129][DEBUG][c.f.s.a.BackendRegistry ] [node1] Rerequest class com.floragunn.searchguard.http.HTTPBasicAuthenticator failed
[2018-11-28T20:13:27,129][WARN ][c.f.s.a.BackendRegistry ] [node1] Authentication finally failed for admin

  • Search Guard and Elasticsearch version

ELK 6.5.1 SG 23.2 KI plugin 16

  • Installed and used enterprise modules, if any

Installed not used

  • JVM version and operating system version

1.8

2

Hi again :slight_smile:

Problem isolated to cookie stored locally.

When I clean the cookie or use private mode in the browser I can log in with new password.

I also find out that when I log in for example as an admin and then log off I can use admin cookie by another user.

When I log in again as kibanaro I’m not logged as kibanaro but as previous user, admin…

It looks like security vulnerability for me.

Regards

3

That’s interesting indeed. It looks like the session cookie is not deleted when you log out. What operating system and browser are you using? Can you reproduce the behavior consistently? If so, can you provide the steps to reproduce? We’ve never seen this before, so we need some details so we can try to reproduce it. Thanks!

···

On Wednesday, November 28, 2018 at 10:37:16 PM UTC+1, Mateusz Ilnicki wrote:

Hi again :slight_smile:

Problem isolated to cookie stored locally.

When I clean the cookie or use private mode in the browser I can log in with new password.

I also find out that when I log in for example as an admin and then log off I can use admin cookie by another user.

When I log in again as kibanaro I’m not logged as kibanaro but as previous user, admin…

It looks like security vulnerability for me.

Regards

4

Hi Jochen,

I’m using Windows 10 Pro and Firefox 63.0.3.

Yes I can reproduce this behavior constantly.

Steps to reproduce this:

I have clean Install of ELK and Search Guard on Centos 7.

  1. I’m logging in as demo user admin and I’m logging off.

  2. I’m logging in as demo user kibanaro, but I’m not logged as kibanaro but as previous user admin.

  3. When I open clean private window on my browser and login as kibanaro, everything works correctly.

If you wan’t I can organize tshoot session to show you exact steps live.

Regards

Mateusz

5

I don’t mean to pollute a thread that isn’t mine, but I have a problem that has similarities so it might help:

I have a double authentication (I’ve tried solutions found on old threads with no luck).
When I log in though the searchguard log in page, I am then asked to log in through a log in pop-up from my browser.

That’s where my issue seems to meet this one: if I enter different information on the 2 different login systems, my browser’s log in (the second) always ‘wins’, and after I’ve logged in through the browser pop-up, even if I log out through Kibana, and then log in again through SearchGuard, whatever (correct) login I use my browser doesn’t ask me to log in, and I end up logged in with the account I used in the browser pop-up the first time (so I think I have the same cookie issue), I can never log out of this account, unless I restart my system. I also deleted cookies with no luck.

So I have more or less the same steps:

  1. I log in in SG as admin
  2. I log in ‘in my browser’s pop-up’ as admin
  3. I’m successfully logged in as admin
  4. I log out
  5. I log in in SG as kibanaro
  6. (no browser pop up)
  7. I’m logged in as admin

Ubuntu 14.04

Chrome 70.0.3538.110 (Official Build) (64-bit)

I can reproduce the same behavior consistently.

Do you need more information from my case?

Hope it helps.
I’ll open another thread for my double auth problem if I can’t figure it out alone.

6

Thanks for the information. I think we have an idea now what is happening, and it seems to be a combination of several things.

First, your browser should never pop up the Basic Auth dialogue. This is not expected behavior and also the root of this issues. So I would like to understand why the dialogue is appearing. Which version of Search Guard are you using? There is a known issue with the latest ES 6.5.x releases of Kibana when Spaces are enabled. While we are working on the compatibility issues, you need to disable Spaces for the moment:

Are you using 5.6.x with Spaces? If not, can you attach your sg_config.yml so we can try to reproduce? As I said, the browser dialogue should not be there and is causing the issue.

If you enter the credentials the second time in the browser’s Basic Auth dialogue, the browser will cache the credentials. This is not under our control, means, we cannot prevent that.

Now, when you log off in Kibana, the session cookie and all cached items are indeed cleared. However, if you access Kibana again, the browser will send the cached Basic Auth credentials along with every request. Our Kibana plugin supports those pre-authenticated HTTP calls, mainly so that you can use the Kibana API with curl, like:

curl --insecure -Ss -**u admin:**admin -H ‘Content-Type: application/json’ -H “kbn-xsrf: true” -H “sg_tenant: global” -XPOST "https://kibana.example.com:5601/api/saved_objects/index-pattern/ …

``

Since the browser will now send the cached HTTP Basic credentials with every request, Search Guard will take those and authenticate the user. Which, in your case, is the admin user.

So to recap, this happens due to a combination of:

  1. The browsers HTTP Basic dialogue popping up

  2. The browser caching the HTTP Basic Auth credentials

  3. Search Guard accepting pre-authenticated calls

While we cannot do anything about the browser cache, this is definitely unexpected behavior. So thanks for bringing this to our attention. We are already working on a fix for it.

···

On Thursday, November 29, 2018 at 3:48:36 PM UTC+1, marc.zominy@hoomano.com wrote:

I don’t mean to pollute a thread that isn’t mine, but I have a problem that has similarities so it might help:

I have a double authentication (I’ve tried solutions found on old threads with no luck).
When I log in though the searchguard log in page, I am then asked to log in through a log in pop-up from my browser.

That’s where my issue seems to meet this one: if I enter different information on the 2 different login systems, my browser’s log in (the second) always ‘wins’, and after I’ve logged in through the browser pop-up, even if I log out through Kibana, and then log in again through SearchGuard, whatever (correct) login I use my browser doesn’t ask me to log in, and I end up logged in with the account I used in the browser pop-up the first time (so I think I have the same cookie issue), I can never log out of this account, unless I restart my system. I also deleted cookies with no luck.

So I have more or less the same steps:

  1. I log in in SG as admin
  2. I log in ‘in my browser’s pop-up’ as admin
  3. I’m successfully logged in as admin
  4. I log out
  5. I log in in SG as kibanaro
  6. (no browser pop up)
  7. I’m logged in as admin

Ubuntu 14.04

Chrome 70.0.3538.110 (Official Build) (64-bit)

I can reproduce the same behavior consistently.

Do you need more information from my case?

Hope it helps.
I’ll open another thread for my double auth problem if I can’t figure it out alone.

7

SG 6.5.1

For my part, disabling spaces solved it!

Thanks.

8

I had spaces enabled and disabling it solved the issue :slight_smile:

Thanks!