no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview]

When asking questions, please provide the following information:

  • Search Guard version: 6.1.2x

  • Elasticsearch version: 6.1.2

  • Installed and used enterprise modules, if any

  • JVM version: java version “1.8.0_112”

Java™ SE Runtime Environment (build 1.8.0_112-b15)

Java HotSpot™ 64-Bit Server VM (build 25.112-b15, mixed mode)

  • Operating system version: Linux 2.6.32-642.11.1.el6.x86_64 #1 SMP Wed Oct 26 10:25:23 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux

1st - I am using the elasticsearch-hadoop-6.1.2 plugin in Hive to insert tables into elasticsearch index (test_index)

ADD JAR hdfs:///udf/elasticsearch-hadoop-6.1.2.jar;

ADD JAR /usr/hdp/2.5.5.0-157/hive/lib/commons-httpclient-3.0.1.jar;

CREATE EXTERNAL TABLE dev.test_table (

company STRING,

        sys                         STRING,

	date_time                   STRING

)

STORED BY ‘org.elasticsearch.hadoop.hive.EsStorageHandler’

TBLPROPERTIES(

‘es.nodes’ = “esnode”,

‘es.port’=‘9200’,

‘es.net.ssl’=‘true’,

‘es.net.ssl.keystore.location’=‘file:///etc/pki/ca-trust/extracted/java/cacerts’,

‘es.net.ssl.keystore.pass’=‘pass’,

‘es.net.ssl.keystore.type’=‘JKS’,

‘es.net.ssl.truststore.location’=‘file:///etc/pki/ca-trust/extracted/java/cacerts’,

‘es.net.ssl.truststore.pass’=‘pass’,

‘es.net.http.auth.user’=‘testuser’,

‘es.net.http.auth.pass’=‘dummy_password’,

‘es.net.ssl.cert.allow.self.signed’=‘true’,

‘es.resource.write’ = ‘test_index/account’,

‘es.query’=’?q=*’

);

2nd - Search-guard is enabled and configured on 51 node elasticsearch cluster

3rd - I created a user through the searchguard api to manage the index test_index

curl -XPUT ‘https://esnode:9200/_searchguard/api/user/testuser’ -H “Content-Type: application/json” -d ‘{“password”:“dummy_password”, “roles”:[“testrole”]}’

4th - I created a role with permissions that should allow access to test_index*

curl -XPUT ‘https://esnode:9200/_searchguard/api/roles/testrole’ -H “Content-Type: application/json” -d ‘{“cluster”:[“indices:admin/",“indices:data/read/scroll”,“cluster:monitor/nodes/info”],“indices”:{"test_index”:{"*":[“UNLIMITED”,“MANAGE”]}}}’

curl -XPUT ‘https://esnode:9200/_searchguard/api/rolesmapping/testrole’ -H “Content-Type: application/json” -d ‘{“users”:[“testuser”]}’

5th - I can use elasticsearch.hadoop to create the index and update the index perfectly (as long as there is not a date in the index)

6th - Update the external hive table mapping as follows (date_time is actually just the date):

ADD JAR hdfs:///udf/elasticsearch-hadoop-6.1.2.jar;

ADD JAR /usr/hdp/2.5.5.0-157/hive/lib/commons-httpclient-3.0.1.jar;

CREATE EXTERNAL TABLE dev.test_table (

company STRING,

        sys                         STRING,

	date_time                   STRING

)

STORED BY ‘org.elasticsearch.hadoop.hive.EsStorageHandler’

TBLPROPERTIES(

‘es.nodes’ = “esnode”,

‘es.port’=‘9200’,

‘es.net.ssl’=‘true’,

‘es.net.ssl.keystore.location’=‘file:///etc/pki/ca-trust/extracted/java/cacerts’,

‘es.net.ssl.keystore.pass’=‘pass’,

‘es.net.ssl.keystore.type’=‘JKS’,

‘es.net.ssl.truststore.location’=‘file:///etc/pki/ca-trust/extracted/java/cacerts’,

‘es.net.ssl.truststore.pass’=‘pass’,

‘es.net.http.auth.user’=‘testuser’,

‘es.net.http.auth.pass’=‘dummy_password’,

‘es.net.ssl.cert.allow.self.signed’=‘true’,

‘es.resource.write’ = ‘test_index_{date_time}/account’,

‘es.query’=’?q=*’

);

ISSUE:

as soon as I add the date to the index name I get the following error:

Caused by: org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview], requestedTenant=null]

The data is inserted into the elasticsearch test_index_2018-05-23 - 6 times

I have confirmed this is not an elasticsearch hadoop issue by using the admin user for the whole cluster (which works as expected)

the user "elasticview" does not have the permission for "indices:admin/refresh".
Can you post the roles/roles-mapping for this user?

···

Am 24.05.2018 um 00:22 schrieb Lance Zukel <lzukel@gmail.com>:

When asking questions, please provide the following information:

* Search Guard version: 6.1.2x
* Elasticsearch version: 6.1.2
* Installed and used enterprise modules, if any
* JVM version: java version "1.8.0_112"
                        Java(TM) SE Runtime Environment (build 1.8.0_112-b15)
                        Java HotSpot(TM) 64-Bit Server VM (build 25.112-b15, mixed mode)
* Operating system version: Linux <hostname> 2.6.32-642.11.1.el6.x86_64 #1 SMP Wed Oct 26 10:25:23 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux

1st - I am using the elasticsearch-hadoop-6.1.2 plugin in Hive to insert tables into elasticsearch index (test_index)
ADD JAR hdfs:///udf/elasticsearch-hadoop-6.1.2.jar;
ADD JAR /usr/hdp/2.5.5.0-157/hive/lib/commons-httpclient-3.0.1.jar;

CREATE EXTERNAL TABLE dev.test_table (
                
                company STRING,
            sys STRING,
      date_time STRING
)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES(
            'es.nodes' = "esnode",
            'es.port'='9200',
            'es.net.ssl'='true',
            'es.net.ssl.keystore.location'='file:///etc/pki/ca-trust/extracted/java/cacerts',
            'es.net.ssl.keystore.pass'='pass',
            'es.net.ssl.keystore.type'='JKS',
            'es.net.ssl.truststore.location'='file:///etc/pki/ca-trust/extracted/java/cacerts',
            'es.net.ssl.truststore.pass'='pass',
            'es.net.http.auth.user'='testuser',
            'es.net.http.auth.pass'='dummy_password',
            'es.net.ssl.cert.allow.self.signed'='true',
            'es.resource.write' = 'test_index/account',
            'es.query'='?q=*'
            );

2nd - Search-guard is enabled and configured on 51 node elasticsearch cluster

3rd - I created a user through the searchguard api to manage the index test_index
curl -XPUT 'https://esnode:9200/_searchguard/api/user/testuser' -H "Content-Type: application/json" -d '{"password":"dummy_password", "roles":["testrole"]}'

4th - I created a role with permissions that should allow access to test_index*
curl -XPUT 'https://esnode:9200/_searchguard/api/roles/testrole' -H "Content-Type: application/json" -d '{"cluster":["indices:admin/*","indices:data/read/scroll","cluster:monitor/nodes/info"],"indices":{"test_index*":{"*":["UNLIMITED","MANAGE"]}}}'

curl -XPUT 'https://esnode:9200/_searchguard/api/rolesmapping/testrole' -H "Content-Type: application/json" -d '{"users":["testuser"]}'

5th - I can use elasticsearch.hadoop to create the index and update the index perfectly (as long as there is not a date in the index)

6th - Update the external hive table mapping as follows (date_time is actually just the date):
ADD JAR hdfs:///udf/elasticsearch-hadoop-6.1.2.jar;
ADD JAR /usr/hdp/2.5.5.0-157/hive/lib/commons-httpclient-3.0.1.jar;

CREATE EXTERNAL TABLE dev.test_table (
                
                company STRING,
            sys STRING,
      date_time STRING
)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES(
            'es.nodes' = "esnode",
            'es.port'='9200',
            'es.net.ssl'='true',
            'es.net.ssl.keystore.location'='file:///etc/pki/ca-trust/extracted/java/cacerts',
            'es.net.ssl.keystore.pass'='pass',
            'es.net.ssl.keystore.type'='JKS',
            'es.net.ssl.truststore.location'='file:///etc/pki/ca-trust/extracted/java/cacerts',
            'es.net.ssl.truststore.pass'='pass',
            'es.net.http.auth.user'='testuser',
            'es.net.http.auth.pass'='dummy_password',
            'es.net.ssl.cert.allow.self.signed'='true',
            'es.resource.write' = 'test_index_{date_time}/account',
            'es.query'='?q=*'
            );

ISSUE:
as soon as I add the date to the index name I get the following error:

Caused by: org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview], requestedTenant=null]

The data is inserted into the elasticsearch test_index_2018-05-23 - 6 times
I have confirmed this is not an elasticsearch hadoop issue by using the admin user for the whole cluster (which works as expected)

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/c738739c-4483-4295-93b8-d25ad860a063%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

roles:
{“elasticview”:{“cluster”:[“UNLIMITED”,“indices:","indices:admin/exists”,“indices:admin/refresh”,“indices:admin/refresh*”,“indices:data/read/scroll”,“indices:data/write/bulk*”,“cluster:monitor/nodes/info”,“indices:admin/create”],“indices”:{“test_index_”:{"":[“UNLIMITED”,"",“MANAGE”,“admin/create”]}}}}

rolesmapping:

The error only happens when I use append a date to the index. And the admin user does not the same issue

{“elasticview”:{“users”:[“evtest”,“elasticview”]}}

This appears to be a bug with the way wildcards are handled in the index name of roles.

"indices:admin/refresh" is not a cluster level permissions but an indices one.
So it appears that you execute a refresh on an index not matching "*spectrum_guide_rollout_*"

Try adding a role to elasticview which permitts to execute a "indices:admin/refresh" on every index.

···

Am 24.05.2018 um 21:56 schrieb Lance Zukel <lzukel@gmail.com>:

roles:
{"elasticview":{"cluster":["UNLIMITED","indices:*","indices:admin/exists*","indices:admin/refresh","indices:admin/refresh*","indices:data/read/scroll","indices:data/write/bulk*","cluster:monitor/nodes/info","indices:admin/create"],"indices":{"*spectrum_guide_rollout_*":{"*":["UNLIMITED","*","MANAGE","admin/create"]}}}}

rolesmapping:
{"elasticview":{"users":["evtest","elasticview"]}}

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/c13de74e-818a-402c-863c-1859ba7fed73%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

After giving the following config:
‘{“cluster”:[“cluster:monitor/nodes/info”],“indices”:{“spectrum_guide_rollout_":[“UNLIMITED”,"”,“MANAGE”,“refresh”,“admin/refresh”],"":[“indices:admin/refresh”,“indices:data/write/bulk”,"indices:data/write/bulk"]}}’

I now get:

returned Internal Server Error(500) - Unexpected exception indices:data/write/bulk[s]; Bailing out

···

On Thursday, May 24, 2018 at 2:14:39 PM UTC-6, Search Guard wrote:

“indices:admin/refresh” is not a cluster level permissions but an indices one.

So it appears that you execute a refresh on an index not matching “spectrum_guide_rollout_

Try adding a role to elasticview which permitts to execute a “indices:admin/refresh” on every index.

Am 24.05.2018 um 21:56 schrieb Lance Zukel lzu...@gmail.com:

roles:
{“elasticview”:{“cluster”:[“UNLIMITED”,“indices:","indices:admin/exists”,“indices:admin/refresh”,“indices:admin/refresh*”,“indices:data/read/scroll”,“indices:data/write/bulk*”,“cluster:monitor/nodes/info”,“indices:admin/create”],“indices”:{“spectrum_guide_rollout_”:{"":[“UNLIMITED”,"",“MANAGE”,“admin/create”]}}}}

rolesmapping:

{“elasticview”:{“users”:[“evtest”,“elasticview”]}}


You received this message because you are subscribed to the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/c13de74e-818a-402c-863c-1859ba7fed73%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

indices:data/write/bulk is a cluster permission - please read the docs


···

Am 24.05.2018 um 22:52 schrieb Lance Zukel <lzukel@gmail.com>:

After giving the following config:
'{"cluster":["cluster:monitor/nodes/info"],"indices":{"spectrum_guide_rollout_*":["UNLIMITED","*","MANAGE","refresh","admin/refresh"],"*":["indices:admin/refresh","indices:data/write/bulk","indices:data/write/bulk*"]}}'

I now get:
returned Internal Server Error(500) - Unexpected exception indices:data/write/bulk[s]; Bailing out

On Thursday, May 24, 2018 at 2:14:39 PM UTC-6, Search Guard wrote:
"indices:admin/refresh" is not a cluster level permissions but an indices one.
So it appears that you execute a refresh on an index not matching "*spectrum_guide_rollout_*"

Try adding a role to elasticview which permitts to execute a "indices:admin/refresh" on every index.

> Am 24.05.2018 um 21:56 schrieb Lance Zukel <lzu...@gmail.com>:
>
> roles:
> {"elasticview":{"cluster":["UNLIMITED","indices:*","indices:admin/exists*","indices:admin/refresh","indices:admin/refresh*","indices:data/read/scroll","indices:data/write/bulk*","cluster:monitor/nodes/info","indices:admin/create"],"indices":{"*spectrum_guide_rollout_*":{"*":["UNLIMITED","*","MANAGE","admin/create"]}}}}
>
> rolesmapping:
> {"elasticview":{"users":["evtest","elasticview"]}}
>
>
> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/c13de74e-818a-402c-863c-1859ba7fed73%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/167b15d7-1b9d-4dab-8ff6-f5fced27e817%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

  • The searchguard API documentation is very vague and does not seem to be very helpful.

  • Most of the docs reference yml format, yet the api references json format

  • There seems to not be any explanation for most of the values.

  • I feel that clearer documentation about searchguard is needed

  • Per the documentation “UNLIMITED” on an index should allow that user to perform any action on that index

  • “MANAGE” on an index should allow the user with that role to perform nearly any action on that index

Let me see if I can explain this better…

···

curl -XPUT ‘https://d03nappp0102:9200/_searchguard/api/roles/elasticview’ -d ‘{“cluster”:[“indices:admin/",“indices:data/read/scroll”,“cluster:monitor/nodes/info”],“indices”:{"spectrum_guide_rollout_”:{"*":[“UNLIMITED”,“MANAGE”]}}}’ -> does not work and returns error — > org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview], requestedTenant=null]

curl -XPUT ‘https://es_node:9200/_searchguard/api/roles/elasticview’ -d ‘{“cluster”:[“indices:admin/",“indices:data/read/scroll”,“cluster:monitor/nodes/info”],“indices”:{"”:{"*":[“UNLIMITED”,“MANAGE”]}}}’ -> does not work -> works but allows the user any index


Sending data such as ‘es.resource.write’ = “spectrum_guide_rollout/account”

curl -XPUT ‘https://es_node:9200/_searchguard/api/roles/elasticview’ -d ‘{“cluster”:[“indices:admin/",“indices:data/read/scroll”,“cluster:monitor/nodes/info”],“indices”:{"spectrum_guide_rollout_”:{"*":[“UNLIMITED”,“MANAGE”]}}}’ -> works


Sending data such as ‘es.resource.write’ = “spectrum_guide_rollout_2018_05_24/account”

curl -XPUT ‘https://es_node:9200/_searchguard/api/roles/elasticview’ -d ‘{“cluster”:[“indices:admin/",“indices:data/read/scroll”,“cluster:monitor/nodes/info”],“indices”:{“spectrum_guide_rollout_2018_05_24”:{"”:[“UNLIMITED”,“MANAGE”]}}}’ -> works

curl -XPUT ‘https://es_node:9200/_searchguard/api/roles/elasticview’ -d ‘{“cluster”:[“indices:admin/",“indices:data/read/scroll”,“cluster:monitor/nodes/info”],“indices”:{"spectrum_guide_rollout_”:{"*":[“UNLIMITED”,“MANAGE”]}}}’ -> does not work and returns error — > org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview], requestedTenant=null]


The issue seems to be a bug with the way that wildcard chars are handled when there is a date involved in the index name

If there is a better way to create roles with the appropriate permissions I would appreciate an example, or a detailed explaination. I have found previous responses not very helpful, Documentation is referenced, but the documentation is vague on the points that were made. This is going on 2 days worth of work to implement what should’ve been a simple task. This is not a very intuitive nor user friendly product.

Sending data such as ‘es.resource.write’ = “spectrum_guide_rollout{date_time}/account”

I mean even this fails:
curl -XPUT ‘https://es_node:9200/searchguard/api/roles/elasticview’ -d '{“cluster”:[“CLUSTER_ALL”],“indices”:{"*spectrum_guide_rollout":{"":[“UNLIMITED”,"",“MANAGE”,“INDICES_ALL”]},"":{"*":[“admin/refresh”]}}}’

Error:

no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview], requestedTenant=null]

From the docs:

Cluster-level action groups

Name
Description
CLUSTER_ALL
Grants all cluster permissions. Equates to cluster:*

General

Name
Description
UNLIMITED
Grants complete access, can be used on index- and cluster-level. Equates to "*".

Index-level action groups

Name
Description
INDICES_ALL
Grants all permissions on the index. Equates to indices:*

Let me quickly outline the problem

  • You try to insert data via hadoop into ES

  • As long as you use the admin user all works as expected

  • You are dealing with two users: testuser an elasticview

  • With the later there seems to be a problem if a date pattern is in the indexname (in combination with wildcards)

pls see more comments inline below:

  • The searchguard API documentation is very vague and does not seem to be very helpful.
  • Most of the docs reference yml format, yet the api references json format
  • There seems to not be any explanation for most of the values.
  • I feel that clearer documentation about searchguard is needed
  • Per the documentation “UNLIMITED” on an index should allow that user to perform any action on that index
  • “MANAGE” on an index should allow the user with that role to perform nearly any action on that index

Let me see if I can explain this better…


curl -XPUT ‘https://d03nappp0102:9200/_searchguard/api/roles/elasticview’ -d ‘{“cluster”:[“indices:admin/",“indices:data/read/scroll”,“cluster:monitor/nodes/info”],“indices”:{"spectrum_guide_rollout_”:{"*":[“UNLIMITED”,“MANAGE”]}}}’ -> does not work and returns error — > org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview], requestedTenant=null]

Sending data such as ‘es.resource.write’ = “spectrum_guide_rollout{date_time}/account”

That is because “spectrum_guide_rollout{date_time}” will not match “spectrum_guide_rollout_" but "spectrum_guide_rollout” i guess (mind the underscore)

curl -XPUT ‘https://es_node:9200/_searchguard/api/roles/elasticview’ -d ‘{“cluster”:[“indices:admin/",“indices:data/read/scroll”,“cluster:monitor/nodes/info”],“indices”:{"”:{"*":[“UNLIMITED”,“MANAGE”]}}}’ -> does not work -> works but allows the user any index


Sending data such as ‘es.resource.write’ = “spectrum_guide_rollout/account”

curl -XPUT ‘https://es_node:9200/_searchguard/api/roles/elasticview’ -d ‘{“cluster”:[“indices:admin/",“indices:data/read/scroll”,“cluster:monitor/nodes/info”],“indices”:{"spectrum_guide_rollout_”:{"*":[“UNLIMITED”,“MANAGE”]}}}’ -> works


pattern match unlike above

Sending data such as ‘es.resource.write’ = “spectrum_guide_rollout_2018_05_24/account”

curl -XPUT ‘https://es_node:9200/_searchguard/api/roles/elasticview’ -d ‘{“cluster”:[“indices:admin/",“indices:data/read/scroll”,“cluster:monitor/nodes/info”],“indices”:{“spectrum_guide_rollout_2018_05_24”:{"”:[“UNLIMITED”,“MANAGE”]}}}’ -> works

pattern match

curl -XPUT ‘https://es_node:9200/_searchguard/api/roles/elasticview’ -d ‘{“cluster”:[“indices:admin/",“indices:data/read/scroll”,“cluster:monitor/nodes/info”],“indices”:{"spectrum_guide_rollout_”:{"*":[“UNLIMITED”,“MANAGE”]}}}’ -> does not work and returns error — > org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview], requestedTenant=null]


this one is really strange because this should also match, i need to investigate and try to reproduce

···

On Thursday, 24 May 2018 23:53:47 UTC+2, Lance Zukel wrote:

The issue seems to be a bug with the way that wildcard chars are handled when there is a date involved in the index name

If there is a better way to create roles with the appropriate permissions I would appreciate an example, or a detailed explaination. I have found previous responses not very helpful, Documentation is referenced, but the documentation is vague on the points that were made. This is going on 2 days worth of work to implement what should’ve been a simple task. This is not a very intuitive nor user friendly product.

I mean even this fails:
curl -XPUT ‘https://es_node:9200/searchguard/api/roles/elasticview’ -d '{“cluster”:[“CLUSTER_ALL”],“indices”:{"*spectrum_guide_rollout":{"":[“UNLIMITED”,"",“MANAGE”,“INDICES_ALL”]},"":{"*":[“admin/refresh”]}}}’

should be

curl -XPUT ‘https://es_node:9200/searchguard/api/roles/elasticview’ -d '{“cluster”:[“CLUSTER_ALL”],“indices”:{"*spectrum_guide_rollout":{"":[“UNLIMITED”,"",“MANAGE”,“INDICES_ALL”]},"":{"*":[“indices:admin/refresh”]}}}’

···

On Friday, 25 May 2018 00:17:08 UTC+2, Lance Zukel wrote:

Error:

no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview], requestedTenant=null]

From the docs:

Cluster-level action groups

Name
Description
CLUSTER_ALL
Grants all cluster permissions. Equates to cluster:*

General

Name
Description
UNLIMITED
Grants complete access, can be used on index- and cluster-level. Equates to "*".

Index-level action groups

Name
Description
INDICES_ALL
Grants all permissions on the index. Equates to indices:*

First, I would like to apologize for not getting back to you sooner, I was out on Friday and did not have a chance to test this until this morning.

Secondly, seems the issue is only appears when the date is dynamically generated as shown below:

Here this is sending a hard coded date ->

Sending data such as ‘es.resource.write’ = “spectrum_guide_rollout_2018_05_24/account”

curl -XPUT ‘https://es_node:9200/_searchguard/api/roles/elasticview’ -d ‘{“cluster”:[“indices:admin/",“indices:data/read/scroll”,“cluster:monitor/nodes/info”],“indices”:{“spectrum_guide_rollout_2018_05_24”:{"”:[“UNLIMITED”,“MANAGE”]}}}’ -> works

pattern match

This should not have a pattern match ->

Sending data such as ‘es.resource.write’ = “spectrum_guide_rollout/account”

curl -XPUT ‘https://es_node:9200/_searchguard/api/roles/elasticview’ -d ‘{“cluster”:[“indices:admin/",“indices:data/read/scroll”,“cluster:monitor/nodes/info”],“indices”:{"spectrum_guide_rollout_”:{"*":[“UNLIMITED”,“MANAGE”]}}}’ -> works

I have been trying multiple permutations but still not working as expected ( if I posted all the attempts here it would probably be overwhelming ) ->

curl -XPUT ‘https://d03nappp0102:9200/_searchguard/api/roles/elasticview’ -d ‘{“cluster”:[“indices:admin/",“indices:data/read/scroll”,“cluster:monitor/nodes/info”],“indices”:{"spectrum_guide_rollout_”:{"*":[“UNLIMITED”,“MANAGE”]}}}’ -> does not work and returns error — > org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview], requestedTenant=null]

Sending data such as ‘es.resource.write’ = “spectrum_guide_rollout{date_time}/account”

That is because “spectrum_guide_rollout{date_time}” will not match “spectrum_guide_rollout_" but "spectrum_guide_rollout” i guess (mind the underscore)

(example I have also tried) Sending data such as ‘es.resource.write’ = “spectrum_guide_rollout_{date_time}/account”

curl -XPUT ‘https://d03nappp0102:9200/_searchguard/api/roles/elasticview’ -d ‘{“cluster”:[“indices:admin/",“indices:data/read/scroll”,“cluster:monitor/nodes/info”],“indices”:{"spectrum_guide_rollout_”:{"*":[“UNLIMITED”,“MANAGE”]}}}’ -> does not work and returns error — > org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview], requestedTenant=null]

Third:

This is exactly the point I have been trying to make ->

curl -XPUT ‘https://es_node:9200/_searchguard/api/roles/elasticview’ -d ‘{“cluster”:[“indices:admin/",“indices:data/read/scroll”,“cluster:monitor/nodes/info”],“indices”:{"spectrum_guide_rollout_”:{"*":[“UNLIMITED”,“MANAGE”]}}}’ -> does not work and returns error — > org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview], requestedTenant=null]


this one is really strange because this should also match, i need to investigate and try to reproduce

Fourth:

In response to the below comment:

I have tried both ways with additional options.

I mean even this fails:
curl -XPUT ‘https://es_node:9200/_searchguard/api/roles/elasticview’ -d ‘{“cluster”:[“CLUSTER_ALL”],“indices”:{“spectrum_guide_rollout_”:{"":[“UNLIMITED”,"",“MANAGE”,“INDICES_ALL”]},"":{"":[“admin/refresh”]}}}’

should be

curl -XPUT ‘https://es_node:9200/_searchguard/api/roles/elasticview’ -d ‘{“cluster”:[“CLUSTER_ALL”],“indices”:{“spectrum_guide_rollout_”:{"":[“UNLIMITED”,"",“MANAGE”,“INDICES_ALL”]},"":{"":[“indices:admin/refresh”]}}}’ -> I tried this as well

Role, Rolesmapping, and internal user config

$ curl -XGET ‘https:/es_node:9200/_searchguard/api/roles/elasticview’

{“elasticview”:{“cluster”:[“cluster:monitor/nodes/info”,“indices:data/write/bulk”,“indices:admin/create”],“indices”:{“spectrum_guide_":[“UNLIMITED”,"”,“MANAGE”,“indices:admin/create”]}}}

Based on the documentation - This should work…

    - The elasticview role should be able to perform any action on any index starting with "spectrum_guide_"

    - UNLIMITED and * are both redundant and makes no difference one with or without on or the other. Both were left in for the purposes of this post

    - indices:admin/create is included as well due to error received

$ curl -XGET ‘https://es_node:9200/_searchguard/api/rolesmapping/elasticview

{“elasticview”:{“users”:[“evtest”,“elasticview”]}}

Confirmed the elasticview role is mapped to the elasticview user

$ curl -XGET ‘https://es_node:9200/_searchguard/api/internalusers/elasticview

{“elasticview”:{“roles”:[“elasticview”],“hash”:“somehashedoutvalue”}}

Confirming the elasticview user is assigned the elasticview role as above

Hadoop table config (for reference):

ADD JAR hdfs:///udf/elasticsearch-hadoop-6.1.2.jar;

ADD JAR /usr/hdp/2.5.5.0-157/hive/lib/commons-httpclient-3.0.1.jar;

CREATE EXTERNAL TABLE dev.es_spec_guide_rollout_zuk_test (

legacy_company STRING,

	sys                                    STRING,

	date_time                          STRING

)

STORED BY ‘org.elasticsearch.hadoop.hive.EsStorageHandler’

TBLPROPERTIES(

‘es.nodes’ = “es_node”,

‘es.port’=‘9200’,

‘es.net.ssl’=‘true’,

‘es.net.ssl.keystore.location’=‘file:///etc/pki/ca-trust/extracted/java/cacerts’,

‘es.net.ssl.keystore.pass’=‘changeit’,

‘es.net.ssl.keystore.type’=‘JKS’,

‘es.net.ssl.truststore.location’=‘file:///etc/pki/ca-trust/extracted/java/cacerts’,

‘es.net.ssl.truststore.pass’=‘changeit’,

‘es.net.http.auth.user’=‘elasticview’,

‘es.net.http.auth.pass’=‘password’,

‘es.net.ssl.cert.allow.self.signed’=‘true’,

‘es.resource.write’ = “spectrum_guide_rollout_zuk_{date_time}/account”,

‘es.query’=’?q=*’

);

Hive Command to insert into table:

insert into table es_spec_guide_rollout_zuk_test values (‘zukel’,‘lance’,‘2018-05-29’);

or

insert into table es_spec_guide_rollout_zuk_test values (‘zukel’,‘lance’,‘2018_05_29’);

or

insert into table es_spec_guide_rollout_zuk_test values (‘zukel’,‘lance’,“2018_05_29”);

Error code returned from HIVE for above commands:

Found unrecoverable error [es_node:9202] returned Internal Server Error(500) - Unexpected exception indices:admin/create; Bailing out…

Curl Command attempting to create same index:

curl -XPUT ‘https://es_node:9200/spectrum_guide_rollout_zuk_2018_05_29/’ -u “elasticview:password”

Error returned from curl:

{“error”:{“root_cause”:[{“type”:“security_exception”,“reason”:“Unexpected exception indices:admin/create”}],“type”:“security_exception”,“reason”:“Unexpected exception indices:admin/create”},“status”:500}

Verify admin user working:

curl -XGET ‘https://es_node:9200/_searchguard/api/rolesmapping’

“sg_all_access” : {

“users” : [

“admin”,

“throttle”

]

},

Only relevant results shown above

curl -XGET ‘https://es_node:9200/_searchguard/api/roles/sg_all_access’

{“sg_all_access”:{“cluster”:[“UNLIMITED”],“tenants”:{“test_tenant_ro”:“RW”,“adm_tenant”:“RW”},“indices”:{"":{"":[“UNLIMITED”]}}}}

curl -XPUT ‘https://esnode:9200/spectrum_guide_rollout_zuk_2018_05_29/’ -u “admin:Password”

{“acknowledged”:true,“shards_acknowledged”:true,“index”:“spectrum_guide_rollout_zuk_2018_05_29”}

Works as Expected

curl -XDELETE ‘https://es_node:9200/spectrum_guide_rollout_zuk_2018_05_29/’ -u “admin:Password”

{“acknowledged”:true,“shards_acknowledged”:true,“index”:“spectrum_guide_rollout_zuk_2018_05_29”}

Delete the index to test from the beginning

Adjust Hadoop Table to verify Admin User working:

ADD JAR hdfs:///udf/elasticsearch-hadoop-6.1.2.jar;

ADD JAR /usr/hdp/2.5.5.0-157/hive/lib/commons-httpclient-3.0.1.jar;

CREATE EXTERNAL TABLE dev.es_spec_guide_rollout_zuk_test (

legacy_company STRING,

	sys                                    STRING,

	date_time                          STRING

)

STORED BY ‘org.elasticsearch.hadoop.hive.EsStorageHandler’

TBLPROPERTIES(

‘es.nodes’ = “es_node”,

‘es.port’=‘9200’,

‘es.net.ssl’=‘true’,

‘es.net.ssl.keystore.location’=‘file:///etc/pki/ca-trust/extracted/java/cacerts’,

‘es.net.ssl.keystore.pass’=‘changeit’,

‘es.net.ssl.keystore.type’=‘JKS’,

‘es.net.ssl.truststore.location’=‘file:///etc/pki/ca-trust/extracted/java/cacerts’,

‘es.net.ssl.truststore.pass’=‘changeit’,

‘es.net.http.auth.user’=‘admin’,

‘es.net.http.auth.pass’=‘Password’,

‘es.net.ssl.cert.allow.self.signed’=‘true’,

‘es.resource.write’ = “spectrum_guide_rollout_zuk_{date_time}/account”,

‘es.query’=’?q=*’

);

insert into table es_spec_guide_rollout_zuk_test values (‘zukel’,‘lance’,“2018_05_29”);

Success

Rollback Hadoop table to use elasticview user:

Attempt to solve:

curl -XPUT ‘https://es_node:9200/searchguard/api/roles/elasticview’ -H “Content-Type: application/json” -d '{“cluster”:[“UNLIMITED”],“tenants”:{“adm_tenant”:“RW”,“test_tenant”:“RW”},“indices”:{"spectrum_guide":{"":[“UNLIMITED”]}}}’

Again should above should work - This time mimicked the admin user just narrowed down the index access

curl -XDELETE ‘https://es_node:9200/spectrum_guide_rollout_zuk_2018_05_29/’ -u “admin:Password”

{“acknowledged”:true,“shards_acknowledged”:true,“index”:“spectrum_guide_rollout_zuk_2018_05_29”}

Delete the index to test from the beginning

curl -XPUT ‘https://es_node:9200/spectrum_guide_rollout_zuk_2018_05_29/’ -u “elasticview:password”

{“acknowledged”:true,“shards_acknowledged”:true,“index”:“spectrum_guide_rollout_zuk_2018_05_29”}

This is an improvement

insert into table es_spec_guide_rollout_zuk_test values (‘zukel’,‘lance’,“2018_05_29”);

no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview], requestedTenant=null]

Failing again - when by all logical reasoning this should be working

curl -XDELETE ‘https://es_node:9200/spectrum_guide_rollout_zuk_2018_05_29/’ -u “admin:Password”

{“acknowledged”:true,“shards_acknowledged”:true,“index”:“spectrum_guide_rollout_zuk_2018_05_29”}

Delete the index again to test hive from the beginning

insert into table es_spec_guide_rollout_zuk_test values (‘zukel’,‘lance’,“2018_05_29”);

no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview], requestedTenant=null]

Failing again - when by all logical reasoning this should be working

curl ‘https://es_node:9200/_cat/indices’ --key “/etc/pki/tls/private/esadmin.key” --cert “/etc/pki/tls/private/esadmin.crt:Pl@f0rmPiNiFi” | grep spec

green open spectrum_guide_rollout_zuk_2018_05_29

Out of curiosity check to see if index was created

By all logical reasoning, following the documentation with the above configuration this should be working, I gave the elasticview the same permissions the admin user has, except narrowed down to only any index starting with “spectrum_guide_” The admin user works, The elasticview user does not work when the index is restricted.

I know I’m a bit late to the game, but I think your last observations help to narrow down the issue. I think that this is unrelated the REST API or the permission settings in SG, but more to Hadoop:

I have created the elasticview role, the corresponding user and the roles mapping as described by you:

curl -u … -H “Content-Type: application/json” -XPUT ‘https://es_node:9200/searchguard/api/roles/elasticview’ -d '{“cluster”:[“CLUSTER_ALL”],“indices”:{"spectrum_guide_rollout":{"":[“UNLIMITED”]}}}’

``

This should give the user all access to any index spectrum_guide_rollout_*

Then - similar to you - I tried the access permission in isolation, means without Hadoop. With the elasticview user I am able to create an index like:

curl -u elasticview:password -XPUT ‘https://es_node:9200/spectrum_guide_rollout_zuk_2018_05_29/’

``

And I am also able to perform the indices/refresh command that Hadoop complains about:

curl -u elasticview:password -XPOST ‘https://es_node:9200/spectrum_guide_rollout_zuk_2018_05_29/_refresh’

``

From this observation, my conclusion is that the user and permissions are working. Let’s put Hadoop into the picture: It seems to work when you use the admin role, which has access to all indices. But it fails when you use the elasticview user. I suspect that Hadoop is submitting some ES calls involving other indices than the spectrum_guide_* index, maybe uses a wildcard or something like that.

To pinpoint we need to see what query Hadoop is issuing and which indices are affected. The easiest way would be to:

  1. Set the log level for Search Guard to debug

1.1) This is described here: https://docs.search-guard.com/latest/troubleshooting-tls

  1. Set your Hadoop table to use the elasticview user

  2. Issue the (failing): Hive> insert into table es_spec_guide_rollout_zuk_test values (‘zukel’,‘lance’,“2018_05_29”);

After you execute the failing call you should see an entry in the logfile stating which request has been issued, which indices were affected and what permissions were missing.

Please post this logfile so we can analyze further.

···

On Tuesday, May 29, 2018 at 4:37:01 PM UTC+2, Lance Zukel wrote:

Role, Rolesmapping, and internal user config

$ curl -XGET ‘https:/es_node:9200/_searchguard/api/roles/elasticview’

{“elasticview”:{“cluster”:[“cluster:monitor/nodes/info”,“indices:data/write/bulk”,“indices:admin/create”],“indices”:{“spectrum_guide_":[“UNLIMITED”,"”,“MANAGE”,“indices:admin/create”]}}}

Based on the documentation - This should work…

    - The elasticview role should be able to perform any action on any index starting with "spectrum_guide_"
    - UNLIMITED and * are both redundant and makes no difference one with or without on or the other. Both were left in for the purposes of this post
    - indices:admin/create is included as well due to error received

$ curl -XGET ‘https://es_node:9200/_searchguard/api/rolesmapping/elasticview

{“elasticview”:{“users”:[“evtest”,“elasticview”]}}

Confirmed the elasticview role is mapped to the elasticview user

$ curl -XGET ‘https://es_node:9200/_searchguard/api/internalusers/elasticview

{“elasticview”:{“roles”:[“elasticview”],“hash”:“somehashedoutvalue”}}

Confirming the elasticview user is assigned the elasticview role as above

Hadoop table config (for reference):

ADD JAR hdfs:///udf/elasticsearch-hadoop-6.1.2.jar;

ADD JAR /usr/hdp/2.5.5.0-157/hive/lib/commons-httpclient-3.0.1.jar;

CREATE EXTERNAL TABLE dev.es_spec_guide_rollout_zuk_test (

legacy_company STRING,

  sys                                    STRING,
  date_time                          STRING

)

STORED BY ‘org.elasticsearch.hadoop.hive.EsStorageHandler’

TBLPROPERTIES(

‘es.nodes’ = “es_node”,

‘es.port’=‘9200’,

‘es.net.ssl’=‘true’,

‘es.net.ssl.keystore.location’=‘file:///etc/pki/ca-trust/extracted/java/cacerts’,

‘es.net.ssl.keystore.pass’=‘changeit’,

‘es.net.ssl.keystore.type’=‘JKS’,

‘es.net.ssl.truststore.location’=‘file:///etc/pki/ca-trust/extracted/java/cacerts’,

‘es.net.ssl.truststore.pass’=‘changeit’,

‘es.net.http.auth.user’=‘elasticview’,

‘es.net.http.auth.pass’=‘password’,

‘es.net.ssl.cert.allow.self.signed’=‘true’,

‘es.resource.write’ = “spectrum_guide_rollout_zuk_{date_time}/account”,

‘es.query’=’?q=*’

);

Hive Command to insert into table:

insert into table es_spec_guide_rollout_zuk_test values (‘zukel’,‘lance’,‘2018-05-29’);

or

insert into table es_spec_guide_rollout_zuk_test values (‘zukel’,‘lance’,‘2018_05_29’);

or

insert into table es_spec_guide_rollout_zuk_test values (‘zukel’,‘lance’,“2018_05_29”);

Error code returned from HIVE for above commands:

Found unrecoverable error [es_node:9202] returned Internal Server Error(500) - Unexpected exception indices:admin/create; Bailing out…

Curl Command attempting to create same index:

curl -XPUT ‘https://es_node:9200/spectrum_guide_rollout_zuk_2018_05_29/’ -u “elasticview:password”

Error returned from curl:

{“error”:{“root_cause”:[{“type”:“security_exception”,“reason”:“Unexpected exception indices:admin/create”}],“type”:“security_exception”,“reason”:“Unexpected exception indices:admin/create”},“status”:500}

Verify admin user working:

curl -XGET ‘https://es_node:9200/_searchguard/api/rolesmapping’

“sg_all_access” : {

“users” : [

“admin”,

“throttle”

]

},

Only relevant results shown above

curl -XGET ‘https://es_node:9200/_searchguard/api/roles/sg_all_access’

{“sg_all_access”:{“cluster”:[“UNLIMITED”],“tenants”:{“test_tenant_ro”:“RW”,“adm_tenant”:“RW”},“indices”:{"":{"":[“UNLIMITED”]}}}}

curl -XPUT ‘https://esnode:9200/spectrum_guide_rollout_zuk_2018_05_29/’ -u “admin:Password”

{“acknowledged”:true,“shards_acknowledged”:true,“index”:“spectrum_guide_rollout_zuk_2018_05_29”}

Works as Expected

curl -XDELETE ‘https://es_node:9200/spectrum_guide_rollout_zuk_2018_05_29/’ -u “admin:Password”

{“acknowledged”:true,“shards_acknowledged”:true,“index”:“spectrum_guide_rollout_zuk_2018_05_29”}

Delete the index to test from the beginning

Adjust Hadoop Table to verify Admin User working:

ADD JAR hdfs:///udf/elasticsearch-hadoop-6.1.2.jar;

ADD JAR /usr/hdp/2.5.5.0-157/hive/lib/commons-httpclient-3.0.1.jar;

CREATE EXTERNAL TABLE dev.es_spec_guide_rollout_zuk_test (

legacy_company STRING,

  sys                                    STRING,
  date_time                          STRING

)

STORED BY ‘org.elasticsearch.hadoop.hive.EsStorageHandler’

TBLPROPERTIES(

‘es.nodes’ = “es_node”,

‘es.port’=‘9200’,

‘es.net.ssl’=‘true’,

‘es.net.ssl.keystore.location’=‘file:///etc/pki/ca-trust/extracted/java/cacerts’,

‘es.net.ssl.keystore.pass’=‘changeit’,

‘es.net.ssl.keystore.type’=‘JKS’,

‘es.net.ssl.truststore.location’=‘file:///etc/pki/ca-trust/extracted/java/cacerts’,

‘es.net.ssl.truststore.pass’=‘changeit’,

‘es.net.http.auth.user’=‘admin’,

‘es.net.http.auth.pass’=‘Password’,

‘es.net.ssl.cert.allow.self.signed’=‘true’,

‘es.resource.write’ = “spectrum_guide_rollout_zuk_{date_time}/account”,

‘es.query’=’?q=*’

);

insert into table es_spec_guide_rollout_zuk_test values (‘zukel’,‘lance’,“2018_05_29”);

Success

Rollback Hadoop table to use elasticview user:

Attempt to solve:

curl -XPUT ‘https://es_node:9200/searchguard/api/roles/elasticview’ -H “Content-Type: application/json” -d '{“cluster”:[“UNLIMITED”],“tenants”:{“adm_tenant”:“RW”,“test_tenant”:“RW”},“indices”:{"spectrum_guide":{"":[“UNLIMITED”]}}}’

Again should above should work - This time mimicked the admin user just narrowed down the index access

curl -XDELETE ‘https://es_node:9200/spectrum_guide_rollout_zuk_2018_05_29/’ -u “admin:Password”

{“acknowledged”:true,“shards_acknowledged”:true,“index”:“spectrum_guide_rollout_zuk_2018_05_29”}

Delete the index to test from the beginning

curl -XPUT ‘https://es_node:9200/spectrum_guide_rollout_zuk_2018_05_29/’ -u “elasticview:password”

{“acknowledged”:true,“shards_acknowledged”:true,“index”:“spectrum_guide_rollout_zuk_2018_05_29”}

This is an improvement

insert into table es_spec_guide_rollout_zuk_test values (‘zukel’,‘lance’,“2018_05_29”);

no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview], requestedTenant=null]

Failing again - when by all logical reasoning this should be working

curl -XDELETE ‘https://es_node:9200/spectrum_guide_rollout_zuk_2018_05_29/’ -u “admin:Password”

{“acknowledged”:true,“shards_acknowledged”:true,“index”:“spectrum_guide_rollout_zuk_2018_05_29”}

Delete the index again to test hive from the beginning

insert into table es_spec_guide_rollout_zuk_test values (‘zukel’,‘lance’,“2018_05_29”);

no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview], requestedTenant=null]

Failing again - when by all logical reasoning this should be working

curl ‘https://es_node:9200/_cat/indices’ --key “/etc/pki/tls/private/esadmin.key” --cert “/etc/pki/tls/private/esadmin.crt:Pl@f0rmPiNiFi” | grep spec

green open spectrum_guide_rollout_zuk_2018_05_29

Out of curiosity check to see if index was created

By all logical reasoning, following the documentation with the above configuration this should be working, I gave the elasticview the same permissions the admin user has, except narrowed down to only any index starting with “spectrum_guide_” The admin user works, The elasticview user does not work when the index is restricted.

After thinking about what you indicated in your post, I decided to try the following:

curl -XPUT ‘https://es_node:9200/searchguard/api/roles/elasticview’ -H “Content-Type: application/json” -d '{“cluster”:[“UNLIMITED”],“tenants”:{“adm_tenant”:“RW”,“test_tenant”:“RW”},“indices”:{"spectrum_guide":{"":[“UNLIMITED”]},"":{"":[“indices:admin/*”]}}}’ <—This works

However, I am concerned about the permissions that indices:admin/* has

**Next I tried: **

curl -XPUT ‘https://es_node:9200/searchguard/api/roles/elasticview’ -H “Content-Type: application/json” -d '{“cluster”:[“UNLIMITED”],“tenants”:{“adm_tenant”:“RW”,“test_tenant”:“RW”},“indices”:{"spectrum_guide":{"":[“UNLIMITED”]},"":{"":[“indices:admin/refresh”]}}}’ <----This does not work

This is the direction I need to be going with only the required access, I also tried several other single use permissions none of which worked

This returns: Caused by: org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest:

null

In response to your post:

  1. Set the log level for Search Guard to debug

1.1) This is described here: https://docs.search-guard.com/latest/troubleshooting-tls

  1. Set your Hadoop table to use the elasticview user
  1. Issue the (failing): Hive> insert into table es_spec_guide_rollout_zuk_test values (‘zukel’,‘lance’,“2018_05_29”);
  1. I will configure my test cluster with searchguard installed and enable debug - this is a 51 node production clsuter

  2. I will test after the test cluster is configured I will create a new table using the elasticview user with the assumed appropriate role

Additional concerns:

  1. I cannot find any documentation listing ALL Available - cluster: indices: permissions (seems like this should be included in the searchguard docs, I have seen dozens of issues from a quick google search)

So it’s definitely the right direction and confirms that Hadoop is issuing additional wildcard queries, which cause the security exception. In order to see what queries are issued the ES logs are the right place to go. You should see something like (taken from a Kibana example):

2018-05-31T17:44:36,511][INFO ][c.f.s.c.PrivilegesEvaluator] No index-level perm match for User [name=hr_employee, roles=[kibanauser], requestedTenant=null] [IndexType [index=searchguard, type=], IndexType [index=sg6-auditlog-2018.05.31, type=]] [Action [[indices:data/read/search]]] [RolesChecked [sg_kibana_user, sg_own_index]]

[2018-05-31T17:44:36,511][INFO ][c.f.s.c.PrivilegesEvaluator] No permissions for {sg_own_index=[IndexType [index=sg6-auditlog-2018.05.31, type=], IndexType [index=searchguard, type=]], sg_kibana_user=[IndexType [index=sg6-auditlog-2018.05.31, type=], IndexType [index=searchguard, type=]]}

``

Maybe it’s sufficient to set up a simple one node cluster for testing? The basic goal is to see the exact request that Hadoop issues to debug the permission problems, running it on your 51 node prod cluster is probably not needed.

Regarding your permission settings: Can you try to use:

indices:admin/refresh*

``

Instead of the non-working

indices:admin/refresh

``

If this also does not work there must be some other indices:admin/* query that Hadoop issues.

Unfortunately, the list of available indices: and cluster: permissions changes from ES release to ES release, and there is also no official publication from Elasticsearch regarding this. That’s why we recommend using the pre-defined action groups that ship with SG if possible.

···

On Wednesday, May 30, 2018 at 5:09:26 PM UTC+2, Lance Zukel wrote:

After thinking about what you indicated in your post, I decided to try the following:

curl -XPUT ‘https://es_node:9200/_searchguard/api/roles/elasticview’ -H “Content-Type: application/json” -d ‘{“cluster”:[“UNLIMITED”],“tenants”:{“adm_tenant”:“RW”,“test_tenant”:“RW”},“indices”:{“spectrum_guide_":{"”:[“UNLIMITED”]},"":{"":[“indices:admin/*”]}}}’ <—This works

However, I am concerned about the permissions that indices:admin/* has

**Next I tried: **

curl -XPUT ‘https://es_node:9200/_searchguard/api/roles/elasticview’ -H “Content-Type: application/json” -d ‘{“cluster”:[“UNLIMITED”],“tenants”:{“adm_tenant”:“RW”,“test_tenant”:“RW”},“indices”:{“spectrum_guide_":{"”:[“UNLIMITED”]},"":{"":[“indices:admin/refresh”]}}}’ <----This does not work

This is the direction I need to be going with only the required access, I also tried several other single use permissions none of which worked

This returns: Caused by: org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest:

null

In response to your post:

  1. Set the log level for Search Guard to debug

1.1) This is described here: https://docs.search-guard.com/latest/troubleshooting-tls

  1. Set your Hadoop table to use the elasticview user
  1. Issue the (failing): Hive> insert into table es_spec_guide_rollout_zuk_test values (‘zukel’,‘lance’,“2018_05_29”);
  1. I will configure my test cluster with searchguard installed and enable debug - this is a 51 node production clsuter
  1. I will test after the test cluster is configured I will create a new table using the elasticview user with the assumed appropriate role

Additional concerns:

  1. I cannot find any documentation listing ALL Available - cluster: indices: permissions (seems like this should be included in the searchguard docs, I have seen dozens of issues from a quick google search)

I was able to setup a 1 node cluster with debug enabled.
After making the request and searching through logs. HERE is what worked:

‘{“cluster”:[“UNLIMITED”],“tenants”:{“adm_tenant”:“RW”,“test_tenant”:“RW”},“indices”:{“spectrum_guide_":{"”:[“UNLIMITED”]},"":{"":[“indices:admin/refresh*”,“indices:admin/create*”]}}}’

Thanks for your help

···

On Thursday, May 31, 2018 at 9:49:00 AM UTC-6, Jochen Kressin wrote:

So it’s definitely the right direction and confirms that Hadoop is issuing additional wildcard queries, which cause the security exception. In order to see what queries are issued the ES logs are the right place to go. You should see something like (taken from a Kibana example):

2018-05-31T17:44:36,511][INFO ][c.f.s.c.PrivilegesEvaluator] No index-level perm match for User [name=hr_employee, roles=[kibanauser], requestedTenant=null] [IndexType [index=searchguard, type=], IndexType [index=sg6-auditlog-2018.05.31, type=]] [Action [[indices:data/read/search]]] [RolesChecked [sg_kibana_user, sg_own_index]]

[2018-05-31T17:44:36,511][INFO ][c.f.s.c.PrivilegesEvaluator] No permissions for {sg_own_index=[IndexType [index=sg6-auditlog-2018.05.31, type=], IndexType [index=searchguard, type=]], sg_kibana_user=[IndexType [index=sg6-auditlog-2018.05.31, type=], IndexType [index=searchguard, type=]]}

``

Maybe it’s sufficient to set up a simple one node cluster for testing? The basic goal is to see the exact request that Hadoop issues to debug the permission problems, running it on your 51 node prod cluster is probably not needed.

Regarding your permission settings: Can you try to use:

indices:admin/refresh*

``

Instead of the non-working

indices:admin/refresh

``

If this also does not work there must be some other indices:admin/* query that Hadoop issues.

Unfortunately, the list of available indices: and cluster: permissions changes from ES release to ES release, and there is also no official publication from Elasticsearch regarding this. That’s why we recommend using the pre-defined action groups that ship with SG if possible.

On Wednesday, May 30, 2018 at 5:09:26 PM UTC+2, Lance Zukel wrote:

After thinking about what you indicated in your post, I decided to try the following:

curl -XPUT ‘https://es_node:9200/_searchguard/api/roles/elasticview’ -H “Content-Type: application/json” -d ‘{“cluster”:[“UNLIMITED”],“tenants”:{“adm_tenant”:“RW”,“test_tenant”:“RW”},“indices”:{“spectrum_guide_":{"”:[“UNLIMITED”]},"":{"":[“indices:admin/*”]}}}’ <—This works

However, I am concerned about the permissions that indices:admin/* has

**Next I tried: **

curl -XPUT ‘https://es_node:9200/_searchguard/api/roles/elasticview’ -H “Content-Type: application/json” -d ‘{“cluster”:[“UNLIMITED”],“tenants”:{“adm_tenant”:“RW”,“test_tenant”:“RW”},“indices”:{“spectrum_guide_":{"”:[“UNLIMITED”]},"":{"":[“indices:admin/refresh”]}}}’ <----This does not work

This is the direction I need to be going with only the required access, I also tried several other single use permissions none of which worked

This returns: Caused by: org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest:

null

In response to your post:

  1. Set the log level for Search Guard to debug

1.1) This is described here: https://docs.search-guard.com/latest/troubleshooting-tls

  1. Set your Hadoop table to use the elasticview user
  1. Issue the (failing): Hive> insert into table es_spec_guide_rollout_zuk_test values (‘zukel’,‘lance’,“2018_05_29”);
  1. I will configure my test cluster with searchguard installed and enable debug - this is a 51 node production clsuter
  1. I will test after the test cluster is configured I will create a new table using the elasticview user with the assumed appropriate role

Additional concerns:

  1. I cannot find any documentation listing ALL Available - cluster: indices: permissions (seems like this should be included in the searchguard docs, I have seen dozens of issues from a quick google search)