Searchguard Transport Client - not able to pass bulk data

Search Guard and Elasticsearch version : elasticsearch 6.3.2, searchguard 6.3.2

Hi,

We are trying to insert bulk data via the searchguard transport client by using Java. The following is the java code snippet for the same:

Settings.Builder settingsBuilder = Settings.builder()

								  .put("searchguard.ssl.transport.pemcert_filepath", CommonConstants.SSL_KEY_LOCATION + "esnode.pem")

								  .put("searchguard.ssl.transport.pemkey_filepath", CommonConstants.SSL_KEY_LOCATION + "esnode-key.pem")

								  .put("searchguard.ssl.transport.pemtrustedcas_filepath", CommonConstants.SSL_KEY_LOCATION + "root-ca.pem")

								  .put("searchguard.ssl.transport.enabled", "true")

								  .put("searchguard.ssl.transport.enforce_hostname_verification", "false")

								  .put("path.home", "/");

Settings settings = settingsBuilder.build();

transportClient = new PreBuiltTransportClient(settings, SearchGuardSSLPlugin.class).addTransportAddress(new TransportAddress(new InetSocketAddress(elasticIpAddress, elasticPort)));

bulkRequest = transportClient.prepareBulk();

The error that we are receiving in the ES console is the following:

[2018-09-05T18:00:25,700][ERROR][c.f.s.f.SearchGuardFilter] No user found for indices:data/write/bulk from 127.0.0.1:55401 TRANSPORT via netty {_sg_initial_action_class_header=BulkRequest}

Any help would be much appreciated.

Thanks,

Pratiksha

Search Guard requires an authenticated user to being able to assign roles and permissions. You have two possibilities when connecting with a transport client:

  1. Add the DN of the certificate you are using to the internal user database, and assign one or more roles. This is described here:

https://search-guard.com/searchguard-elasicsearch-transport-clients/

  1. Add an HTTP Basic Authentication header to the call, and then set up a user and roles for these credentials. This is described here:

https://search-guard.com/transport-client-authentication-authorization/

Note that the articles are for SG2 and SG5, but the basic principles are still correct.

···

On Wednesday, September 5, 2018 at 11:59:51 AM UTC-4, Pratiksha Tewary wrote:

Search Guard and Elasticsearch version : elasticsearch 6.3.2, searchguard 6.3.2

Hi,

We are trying to insert bulk data via the searchguard transport client by using Java. The following is the java code snippet for the same:

Settings.Builder settingsBuilder = Settings.builder()

  							  .put("searchguard.ssl.transport.pemcert_filepath", CommonConstants.SSL_KEY_LOCATION + "esnode.pem")
  							  .put("searchguard.ssl.transport.pemkey_filepath", CommonConstants.SSL_KEY_LOCATION + "esnode-key.pem")
  							  .put("searchguard.ssl.transport.pemtrustedcas_filepath", CommonConstants.SSL_KEY_LOCATION + "root-ca.pem")
  							  .put("searchguard.ssl.transport.enabled", "true")
  							  .put("searchguard.ssl.transport.enforce_hostname_verification", "false")
  							  .put("path.home", "/");

Settings settings = settingsBuilder.build();

transportClient = new PreBuiltTransportClient(settings, SearchGuardSSLPlugin.class).addTransportAddress(new TransportAddress(new InetSocketAddress(elasticIpAddress, elasticPort)));

bulkRequest = transportClient.prepareBulk();

The error that we are receiving in the ES console is the following:

[2018-09-05T18:00:25,700][ERROR][c.f.s.f.SearchGuardFilter] No user found for indices:data/write/bulk from 127.0.0.1:55401 TRANSPORT via netty {_sg_initial_action_class_header=BulkRequest}

Any help would be much appreciated.

Thanks,

Pratiksha

Hi Jochen Kressin,
**I have added Both configurations in my “.yml” file but still am getting the same issue. ****For more details, plz find my attachments **

******Referred the Links which you gave is having only **2.X and 5.x versions. Is it same for 6.x? or we have to some configurations

···

Please guide me on this issue


Thanks,

**VinothKumar.K **


On Thursday, September 6, 2018 at 5:09:03 AM UTC+5:30, Jochen Kressin wrote:

Search Guard requires an authenticated user to being able to assign roles and permissions. You have two possibilities when connecting with a transport client:

  1. Add the DN of the certificate you are using to the internal user database, and assign one or more roles. This is described here:

https://search-guard.com/searchguard-elasicsearch-transport-clients/

  1. Add an HTTP Basic Authentication header to the call, and then set up a user and roles for these credentials. This is described here:

https://search-guard.com/transport-client-authentication-authorization/

Note that the articles are for SG2 and SG5, but the basic principles are still correct.

On Wednesday, September 5, 2018 at 11:59:51 AM UTC-4, Pratiksha Tewary wrote:

Search Guard and Elasticsearch version : elasticsearch 6.3.2, searchguard 6.3.2

Hi,

We are trying to insert bulk data via the searchguard transport client by using Java. The following is the java code snippet for the same:

Settings.Builder settingsBuilder = Settings.builder()

  							  .put("searchguard.ssl.transport.pemcert_filepath", CommonConstants.SSL_KEY_LOCATION + "esnode.pem")
  							  .put("searchguard.ssl.transport.pemkey_filepath", CommonConstants.SSL_KEY_LOCATION + "esnode-key.pem")
  							  .put("searchguard.ssl.transport.pemtrustedcas_filepath", CommonConstants.SSL_KEY_LOCATION + "root-ca.pem")
  							  .put("searchguard.ssl.transport.enabled", "true")
  							  .put("searchguard.ssl.transport.enforce_hostname_verification", "false")
  							  .put("path.home", "/");

Settings settings = settingsBuilder.build();

transportClient = new PreBuiltTransportClient(settings, SearchGuardSSLPlugin.class).addTransportAddress(new TransportAddress(new InetSocketAddress(elasticIpAddress, elasticPort)));

bulkRequest = transportClient.prepareBulk();

The error that we are receiving in the ES console is the following:

[2018-09-05T18:00:25,700][ERROR][c.f.s.f.SearchGuardFilter] No user found for indices:data/write/bulk from 127.0.0.1:55401 TRANSPORT via netty {_sg_initial_action_class_header=BulkRequest}

Any help would be much appreciated.

Thanks,

Pratiksha

Sorry to say, but to me it seems you simply copied the sample entries from the blog post to your configuration as-is without adapting it to your environment.

Of course the entries for internal users and the role mapping must match the DN of the actual certificate you are using. Which, from what I can see from our config, is a node certificate and not the demo “Spock” certificate used in the article.

So in other words:

You connect to ES/SG with a TLS certificate. In your case:

CommonConstants.SSL_KEY_LOCATION + "esnode.pem

You need to add a user for the DN of this certificate and assign at least one SG role to it. Otherwise SG does not know which SG role it should use and what permissions to apply.

···

On Thursday, September 6, 2018 at 7:11:01 AM UTC-4, Vinoth Kumar wrote:

Hi Jochen Kressin,
**I have added Both configurations in my “.yml” file but still am getting the same issue. ****For more details, plz find my attachments **

******Referred the Links which you gave is having only **2.X and 5.x versions. Is it same for 6.x? or we have to some configurations


Please guide me on this issue


Thanks,

**VinothKumar.K **


On Thursday, September 6, 2018 at 5:09:03 AM UTC+5:30, Jochen Kressin wrote:

Search Guard requires an authenticated user to being able to assign roles and permissions. You have two possibilities when connecting with a transport client:

  1. Add the DN of the certificate you are using to the internal user database, and assign one or more roles. This is described here:

https://search-guard.com/searchguard-elasicsearch-transport-clients/

  1. Add an HTTP Basic Authentication header to the call, and then set up a user and roles for these credentials. This is described here:

https://search-guard.com/transport-client-authentication-authorization/

Note that the articles are for SG2 and SG5, but the basic principles are still correct.

On Wednesday, September 5, 2018 at 11:59:51 AM UTC-4, Pratiksha Tewary wrote:

Search Guard and Elasticsearch version : elasticsearch 6.3.2, searchguard 6.3.2

Hi,

We are trying to insert bulk data via the searchguard transport client by using Java. The following is the java code snippet for the same:

Settings.Builder settingsBuilder = Settings.builder()

  							  .put("searchguard.ssl.transport.pemcert_filepath", CommonConstants.SSL_KEY_LOCATION + "esnode.pem")
  							  .put("searchguard.ssl.transport.pemkey_filepath", CommonConstants.SSL_KEY_LOCATION + "esnode-key.pem")
  							  .put("searchguard.ssl.transport.pemtrustedcas_filepath", CommonConstants.SSL_KEY_LOCATION + "root-ca.pem")
  							  .put("searchguard.ssl.transport.enabled", "true")
  							  .put("searchguard.ssl.transport.enforce_hostname_verification", "false")
  							  .put("path.home", "/");

Settings settings = settingsBuilder.build();

transportClient = new PreBuiltTransportClient(settings, SearchGuardSSLPlugin.class).addTransportAddress(new TransportAddress(new InetSocketAddress(elasticIpAddress, elasticPort)));

bulkRequest = transportClient.prepareBulk();

The error that we are receiving in the ES console is the following:

[2018-09-05T18:00:25,700][ERROR][c.f.s.f.SearchGuardFilter] No user found for indices:data/write/bulk from 127.0.0.1:55401 TRANSPORT via netty {_sg_initial_action_class_header=BulkRequest}

Any help would be much appreciated.

Thanks,

Pratiksha

Hi Jochen Kressin,
Thanks for your response (Or) advice. Actually, I have frequently changed Configurations in Java Code as well as an elastic server so that got pretty confusion.

Now its working fine to me.

Thanks,

VinothKumar.K

···

$vinoth kumar$

Hi Jochen

I’ve used SG to define user for my ES. It works fine, and I can send a request using CURL command with my username and password.

Now, I want to insert a few documents into my ES. I use java to connect to ES. Before I install SG, I used these lines of code and the java app worked fine:

TransportClient client;
Settings.Builder settingsBuilder =
Settings.builder();
Settings settings = settingsBuilder.build();
client =
new PreBuiltTransportClient(settings)
.addTransportAddress(new TransportAddress(InetAddress.getByName(“localhost”), 9300));
BulkRequestBuilder bulkRequest = client.prepareBulk();
Map<String, Object> myMap = new HashMap<>();
myMap.put(“varName”, “val”);
bulkRequest.add(client.prepareUpdate(“myindex”, “_doc”, “testID”)
.setDoc(myMap).setUpsert(myMap));

BulkResponse bulkResponse = bulkRequest.get();


The above example insert a document into my ES. Assume all settings are default values. I install SG and as I mentioned, It works with CURL command (Again, assume I didn’t change the default values of SG). But I confused about java class. I’ve changed the class to the blow version:

TransportClient client;
Settings.Builder settingsBuilder =
Settings.builder()
.put(“path.home”, “/”)
.put(SSLConfigConstants.SEARCHGUARD_SSL_TRANSPORT_KEYSTORE_FILEPATH, “node1.pem”)
.put(SSLConfigConstants.SEARCHGUARD_SSL_TRANSPORT_TRUSTSTORE_FILEPATH, “node1.key”)
.put(“searchguard.ssl.transport.pemtrustedcas_filepath”, “root-ca.pem”);
Settings settings = settingsBuilder.build();
client =
new PreBuiltTransportClient(settings,SearchGuardSSLPlugin.class)
.addTransportAddress(new TransportAddress(InetAddress.getByName(“localhost”), 9300));
BulkRequestBuilder bulkRequest = client.prepareBulk();
Map<String, Object> myMap = new HashMap<>();
myMap.put(“varName2”, “val2”);
bulkRequest.add(client.prepareUpdate(“myindex”, “_doc”, “testID2”)
.setDoc(myMap).setUpsert(myMap));

BulkResponse bulkResponse = bulkRequest.get();


I get this error:

java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.ssl.SearchGuardSSLPlugin]

	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607)

	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:112)

	at org.elasticsearch.client.transport.TransportClient.newPluginService(TransportClient.java:108)

	at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:133)

	at org.elasticsearch.client.transport.TransportClient.<init>(TransportClient.java:283)

	at org.elasticsearch.transport.client.PreBuiltTransportClient.<init>(PreBuiltTransportClient.java:128)

	at org.elasticsearch.transport.client.PreBuiltTransportClient.<init>(PreBuiltTransportClient.java:114)

	at org.elasticsearch.transport.client.PreBuiltTransportClient.<init>(PreBuiltTransportClient.java:104)

	at Adverts.elastic.ElasticsearchHbaseAdverts.startup(ElasticsearchHbaseAdverts.java:46)

	at Adverts.elastic.ElasticsearchHbaseAdverts.<init>(ElasticsearchHbaseAdverts.java:28)

	at Adverts.elastic.ElasticsearchHbaseAdverts.main(ElasticsearchHbaseAdverts.java:247)

Caused by: java.lang.reflect.InvocationTargetException

	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:598)

	... 10 more

Caused by: ElasticsearchSecurityException[Error while initializing transport SSL layer: java.io.IOException: Invalid keystore format]; nested: IOException[Invalid keystore format];

	at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:303)

	at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.<init>(DefaultSearchGuardKeyStore.java:145)

	at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.<init>(SearchGuardSSLPlugin.java:193)

	at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.<init>(SearchGuardSSLPlugin.java:96)

	... 15 more

Caused by: java.io.IOException: Invalid keystore format

	at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:658)

	at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)

	at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)

	at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)

	at java.security.KeyStore.load(KeyStore.java:1445)

	at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:250)

	... 18 more

Can you guide me what is wrong in my code?

Thanks

<details class='elided'>
<summary title='Show trimmed content'>&#183;&#183;&#183;</summary>

On Friday, September 7, 2018 at 8:18:35 AM UTC+4:30, Jochen Kressin wrote:
> Sorry to say, but to me it seems you simply copied the sample entries from the blog post to your configuration as-is without adapting it to your environment.
> 

> Of course the entries for internal users and the role mapping must match the DN of the actual certificate you are using. Which, from what I can see from our config, is a node certificate and not the demo "Spock" certificate used in the article.

> 

> So in other words:

> 

> You connect to ES/SG with a TLS certificate. In your case:

> 

> CommonConstants.SSL_KEY_LOCATION + "esnode.pem
> 
> You need to add a user for the DN of this certificate and assign at least one SG role to it. Otherwise SG does not know which SG role it should use and what permissions to apply.

> 
> On Thursday, September 6, 2018 at 7:11:01 AM UTC-4, Vinoth Kumar wrote:
> > Hi Jochen Kressin,
> > **I have added Both configurations in my  ".yml" file but still am getting the same issue. ****For more details, plz find my attachments  **

> > ******Referred the  Links which you gave is having only  ****2.X and 5.x versions. Is it same for  6.x? or we have to some configurations**

> > ****

> > **Please guide me on this issue**

> > ****

> > **Thanks,**

> > **VinothKumar.K **

> > ****
> > On Thursday, September 6, 2018 at 5:09:03 AM UTC+5:30, Jochen Kressin wrote:
> > > Search Guard requires an authenticated user to being able to assign roles and permissions. You have two possibilities when connecting with a transport client:
> > > 

> > > 1. Add the DN of the certificate you are using to the internal user database, and assign one or more roles. This is described here:

> > > 

> > > [https://search-guard.com/searchguard-elasicsearch-transport-clients/](https://search-guard.com/searchguard-elasicsearch-transport-clients/)

> > > 

> > > 2. Add an HTTP Basic Authentication header to the call, and then set up a user and roles for these credentials. This is described here:

> > > 

> > > [https://search-guard.com/transport-client-authentication-authorization/](https://search-guard.com/transport-client-authentication-authorization/)

> > > 

> > > Note that the articles are for SG2 and SG5, but the basic principles are still correct.

> > > 
> > > On Wednesday, September 5, 2018 at 11:59:51 AM UTC-4, Pratiksha Tewary wrote:
> > > > Search Guard and Elasticsearch version : elasticsearch 6.3.2, searchguard 6.3.2
> > > > 

> > > > Hi,

> > > > 

> > > > We are trying to insert bulk data via the searchguard transport client by using Java. The following is the java code snippet for the same:

> > > > 

> > > > Settings.Builder settingsBuilder = Settings.builder()

> > > > 									  .put("searchguard.ssl.transport.pemcert_filepath", CommonConstants.SSL_KEY_LOCATION + "esnode.pem")

> > > > 									  .put("searchguard.ssl.transport.pemkey_filepath", CommonConstants.SSL_KEY_LOCATION + "esnode-key.pem")

> > > > 									  .put("searchguard.ssl.transport.pemtrustedcas_filepath", CommonConstants.SSL_KEY_LOCATION + "root-ca.pem")

> > > > 									  .put("searchguard.ssl.transport.enabled", "true")

> > > > 									  .put("searchguard.ssl.transport.enforce_hostname_verification", "false")

> > > > 									  .put("path.home", "/");

> > > > Settings settings = settingsBuilder.build();

> > > > 									  

> > > > transportClient = new PreBuiltTransportClient(settings, SearchGuardSSLPlugin.class).addTransportAddress(new TransportAddress(new InetSocketAddress(elasticIpAddress, elasticPort)));

> > > > bulkRequest = transportClient.prepareBulk();

> > > > 

> > > > The error that we are receiving in the ES console is the following:

> > > > 

> > > > [2018-09-05T18:00:25,700][ERROR][c.f.s.f.SearchGuardFilter] No user found for indices:data/write/bulk from [127.0.0.1:55401](http://127.0.0.1:55401) TRANSPORT via netty {_sg_initial_action_class_header=BulkRequest}

> > > > 

> > > > Any help would be much appreciated.

> > > > 

> > > > Thanks,

> > > > Pratiksha

</details>