When asking questions, please provide the following information:
-
Search Guard version: 6.1.2x
-
Elasticsearch version: 6.1.2
-
Installed and used enterprise modules, if any
-
JVM version: java version “1.8.0_112”
Java™ SE Runtime Environment (build 1.8.0_112-b15)
Java HotSpot™ 64-Bit Server VM (build 25.112-b15, mixed mode)
- Operating system version: Linux 2.6.32-642.11.1.el6.x86_64 #1 SMP Wed Oct 26 10:25:23 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
1st - I am using the elasticsearch-hadoop-6.1.2 plugin in Hive to insert tables into elasticsearch index (test_index)
ADD JAR hdfs:///udf/elasticsearch-hadoop-6.1.2.jar;
ADD JAR /usr/hdp/2.5.5.0-157/hive/lib/commons-httpclient-3.0.1.jar;
CREATE EXTERNAL TABLE dev.test_table (
company STRING,
sys STRING,
date_time STRING
)
STORED BY ‘org.elasticsearch.hadoop.hive.EsStorageHandler’
TBLPROPERTIES(
‘es.nodes’ = “esnode”,
‘es.port’=‘9200’,
‘es.net.ssl’=‘true’,
‘es.net.ssl.keystore.location’=‘file:///etc/pki/ca-trust/extracted/java/cacerts’,
‘es.net.ssl.keystore.pass’=‘pass’,
‘es.net.ssl.keystore.type’=‘JKS’,
‘es.net.ssl.truststore.location’=‘file:///etc/pki/ca-trust/extracted/java/cacerts’,
‘es.net.ssl.truststore.pass’=‘pass’,
‘es.net.http.auth.user’=‘testuser’,
‘es.net.http.auth.pass’=‘dummy_password’,
‘es.net.ssl.cert.allow.self.signed’=‘true’,
‘es.resource.write’ = ‘test_index/account’,
‘es.query’=‘?q=*’
);
2nd - Search-guard is enabled and configured on 51 node elasticsearch cluster
3rd - I created a user through the searchguard api to manage the index test_index
curl -XPUT ‘https://esnode:9200/_searchguard/api/user/testuser’ -H “Content-Type: application/json” -d ‘{“password”:“dummy_password”, “roles”:[“testrole”]}’
4th - I created a role with permissions that should allow access to test_index*
curl -XPUT ‘https://esnode:9200/_searchguard/api/roles/testrole’ -H “Content-Type: application/json” -d ‘{“cluster”:[“indices:admin/",“indices:data/read/scroll”,“cluster:monitor/nodes/info”],“indices”:{"test_index”:{“*”:[“UNLIMITED”,“MANAGE”]}}}’
curl -XPUT ‘https://esnode:9200/_searchguard/api/rolesmapping/testrole’ -H “Content-Type: application/json” -d ‘{“users”:[“testuser”]}’
5th - I can use elasticsearch.hadoop to create the index and update the index perfectly (as long as there is not a date in the index)
6th - Update the external hive table mapping as follows (date_time is actually just the date):
ADD JAR hdfs:///udf/elasticsearch-hadoop-6.1.2.jar;
ADD JAR /usr/hdp/2.5.5.0-157/hive/lib/commons-httpclient-3.0.1.jar;
CREATE EXTERNAL TABLE dev.test_table (
company STRING,
sys STRING,
date_time STRING
)
STORED BY ‘org.elasticsearch.hadoop.hive.EsStorageHandler’
TBLPROPERTIES(
‘es.nodes’ = “esnode”,
‘es.port’=‘9200’,
‘es.net.ssl’=‘true’,
‘es.net.ssl.keystore.location’=‘file:///etc/pki/ca-trust/extracted/java/cacerts’,
‘es.net.ssl.keystore.pass’=‘pass’,
‘es.net.ssl.keystore.type’=‘JKS’,
‘es.net.ssl.truststore.location’=‘file:///etc/pki/ca-trust/extracted/java/cacerts’,
‘es.net.ssl.truststore.pass’=‘pass’,
‘es.net.http.auth.user’=‘testuser’,
‘es.net.http.auth.pass’=‘dummy_password’,
‘es.net.ssl.cert.allow.self.signed’=‘true’,
‘es.resource.write’ = ‘test_index_{date_time}/account’,
‘es.query’=‘?q=*’
);
ISSUE:
as soon as I add the date to the index name I get the following error:
Caused by: org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview], requestedTenant=null]
The data is inserted into the elasticsearch test_index_2018-05-23 - 6 times
I have confirmed this is not an elasticsearch hadoop issue by using the admin user for the whole cluster (which works as expected)