Slow indexing with searchguard

Hi,

We have installed searchguard 2.3.2-beta2 in our elasticsearch cluster. Everything went well with the installation. But what we noticed is that bulk indexing (from hdfs to es particularly) is quite slow after enabling searchguard in the cluster.

For ex, we were indexing 50 million document in our cluster, which would generally take no more than few hours (usually between 2-3), but after installing searchguard the same indexing took more than 9 hours.

Is that an expected behavior while indexing with searchguard? and/or

Is that fixed (or improved) in the RC1 release?

Thank you

Jainin Shah

This is not expected. Can you post your configuration (elasticsearch.yml and sg_config.yml) and explain a little bit more in detail how your ingestion work and what kind of authentication you use.
Forn example: Do you use HTTP or Java Client for indexing? If HTTP: Do you have SSL enabled and if so do you use native OpenSSL support?
Maybe SSL is slowing down the whole thing. Beta1 introduced caching so hitting the authentication backend should normally not cause any pain.

Please provide also informations for the folowing questions:
- How many nodes do you have in your cluster?
- JVM vendor and version
- Operation system vendor and version
- OpenSSL version

Thx

···

Am 10.06.2016 um 18:31 schrieb Jainin Shah <jainin@tresata.com>:

Hi,

We have installed searchguard 2.3.2-beta2 in our elasticsearch cluster. Everything went well with the installation. But what we noticed is that bulk indexing (from hdfs to es particularly) is quite slow after enabling searchguard in the cluster.

For ex, we were indexing 50 million document in our cluster, which would generally take no more than few hours (usually between 2-3), but after installing searchguard the same indexing took more than 9 hours.

Is that an expected behavior while indexing with searchguard? and/or
Is that fixed (or improved) in the RC1 release?

Thank you

Jainin Shah

--
You received this message because you are subscribed to the Google Groups "Search Guard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/aea72d49-2f6e-40c2-8d97-464744316fd6%40googlegroups.com\.
For more options, visit https://groups.google.com/d/optout\.

Thanks for the prompt reply.

To answer your questions:

  • How many nodes do you have in your cluster? : 10

  • JVM vendor and version :

java version “1.7.0_75”

Java™ SE Runtime Environment (build 1.7.0_75-b13)

Java HotSpot™ 64-Bit Server VM (build 24.75-b04, mixed mode)

  • Operation system vendor and version

CentOS release 6.8 (Final)

  • OpenSSL version

OpenSSL 1.0.1e-fips 11 Feb 2013

Also, attached are the elasticsearch.yml file and sg_config.yml file. sg_config.yml is exactly the same as it comes with the package.

And we do not use java client to index, it’s going through http using elasticsearch-spark. And we use basic authentication (username and password) with searchguard.

sg_config.yml (1.19 KB)

elasricsearch.yml (2.05 KB)

···

On Friday, June 10, 2016 at 2:03:14 PM UTC-4, SG wrote:

This is not expected. Can you post your configuration (elasticsearch.yml and sg_config.yml) and explain a little bit more in detail how your ingestion work and what kind of authentication you use.

Forn example: Do you use HTTP or Java Client for indexing? If HTTP: Do you have SSL enabled and if so do you use native OpenSSL support?

Maybe SSL is slowing down the whole thing. Beta1 introduced caching so hitting the authentication backend should normally not cause any pain.

Please provide also informations for the folowing questions:

  • How many nodes do you have in your cluster?

  • JVM vendor and version

  • Operation system vendor and version

  • OpenSSL version

Thx

Am 10.06.2016 um 18:31 schrieb Jainin Shah jai...@tresata.com:

Hi,

We have installed searchguard 2.3.2-beta2 in our elasticsearch cluster. Everything went well with the installation. But what we noticed is that bulk indexing (from hdfs to es particularly) is quite slow after enabling searchguard in the cluster.

For ex, we were indexing 50 million document in our cluster, which would generally take no more than few hours (usually between 2-3), but after installing searchguard the same indexing took more than 9 hours.

Is that an expected behavior while indexing with searchguard? and/or

Is that fixed (or improved) in the RC1 release?

Thank you

Jainin Shah


You received this message because you are subscribed to the Google Groups “Search Guard” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/aea72d49-2f6e-40c2-8d97-464744316fd6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Can you try RC1 because it contains SSL related fixes which should improve speed and fix a possible memory leak with OpenSSL (Beware: config has slightly changed for RC1)

I see from you config file that you NOT using OpenSSL (enable_openssl_if_available: false). We strongly recommend to let OpenSSL handle SSL because Java built-in SSL support in slow
(especially in Java 7, therefore you should also consider upgrading to Oracle Java 8)

See here how to enable OpenSSL support for Search Guard.

Another question: It seems do that you not encrypt your HTTP traffic. Plain HTTP together with basic authentication means that the plaintext password will be sent over the network.
Consider setting searchguard.ssl.http.enabled: true to cope with that.

···

Am 10.06.2016 um 21:54 schrieb Jainin Shah <jainin@tresata.com>:

Thanks for the prompt reply.

To answer your questions:

- How many nodes do you have in your cluster? : 10

- JVM vendor and version :
java version "1.7.0_75"
Java(TM) SE Runtime Environment (build 1.7.0_75-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode)

- Operation system vendor and version
CentOS release 6.8 (Final)

- OpenSSL version
OpenSSL 1.0.1e-fips 11 Feb 2013

Also, attached are the elasticsearch.yml file and sg_config.yml file. sg_config.yml is exactly the same as it comes with the package.

And we do not use java client to index, it's going through http using elasticsearch-spark. And we use basic authentication (username and password) with searchguard.

On Friday, June 10, 2016 at 2:03:14 PM UTC-4, SG wrote:
This is not expected. Can you post your configuration (elasticsearch.yml and sg_config.yml) and explain a little bit more in detail how your ingestion work and what kind of authentication you use.
Forn example: Do you use HTTP or Java Client for indexing? If HTTP: Do you have SSL enabled and if so do you use native OpenSSL support?
Maybe SSL is slowing down the whole thing. Beta1 introduced caching so hitting the authentication backend should normally not cause any pain.

Please provide also informations for the folowing questions:
- How many nodes do you have in your cluster?
- JVM vendor and version
- Operation system vendor and version
- OpenSSL version

Thx

> Am 10.06.2016 um 18:31 schrieb Jainin Shah <jai...@tresata.com>:
>
> Hi,
>
> We have installed searchguard 2.3.2-beta2 in our elasticsearch cluster. Everything went well with the installation. But what we noticed is that bulk indexing (from hdfs to es particularly) is quite slow after enabling searchguard in the cluster.
>
> For ex, we were indexing 50 million document in our cluster, which would generally take no more than few hours (usually between 2-3), but after installing searchguard the same indexing took more than 9 hours.
>
> Is that an expected behavior while indexing with searchguard? and/or
> Is that fixed (or improved) in the RC1 release?
>
> Thank you
>
> Jainin Shah
>
> --
> You received this message because you are subscribed to the Google Groups "Search Guard" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/aea72d49-2f6e-40c2-8d97-464744316fd6%40googlegroups.com\.
> For more options, visit https://groups.google.com/d/optout\.

--
You received this message because you are subscribed to the Google Groups "Search Guard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/c31c23da-503b-4095-9c9a-a326b1beef8c%40googlegroups.com\.
For more options, visit https://groups.google.com/d/optout\.
<sg_config.yml><elasricsearch.yml>

Thanks. Will give RC1 and OpenSSL a try and see if that increases that speed.

···

On Saturday, June 11, 2016 at 4:12:38 AM UTC-4, SG wrote:

Can you try RC1 because it contains SSL related fixes which should improve speed and fix a possible memory leak with OpenSSL (Beware: config has slightly changed for RC1)

I see from you config file that you NOT using OpenSSL (enable_openssl_if_available: false). We strongly recommend to let OpenSSL handle SSL because Java built-in SSL support in slow
(especially in Java 7, therefore you should also consider upgrading to Oracle Java 8)

See here how to enable OpenSSL support for Search Guard.

https://github.com/floragunncom/search-guard-ssl/wiki/Open-SSL-setup/f618d0efa96cc9d598b28439b25464047c5cc98a

Another question: It seems do that you not encrypt your HTTP traffic. Plain HTTP together with basic authentication means that the plaintext password will be sent over the network.

Consider setting searchguard.ssl.http.enabled: true to cope with that.

Am 10.06.2016 um 21:54 schrieb Jainin Shah jai...@tresata.com:

Thanks for the prompt reply.

To answer your questions:

  • How many nodes do you have in your cluster? : 10
  • JVM vendor and version :

java version “1.7.0_75”

Java™ SE Runtime Environment (build 1.7.0_75-b13)

Java HotSpot™ 64-Bit Server VM (build 24.75-b04, mixed mode)

  • Operation system vendor and version
    CentOS release 6.8 (Final)
  • OpenSSL version
    OpenSSL 1.0.1e-fips 11 Feb 2013

Also, attached are the elasticsearch.yml file and sg_config.yml file. sg_config.yml is exactly the same as it comes with the package.

And we do not use java client to index, it’s going through http using elasticsearch-spark. And we use basic authentication (username and password) with searchguard.

On Friday, June 10, 2016 at 2:03:14 PM UTC-4, SG wrote:

This is not expected. Can you post your configuration (elasticsearch.yml and sg_config.yml) and explain a little bit more in detail how your ingestion work and what kind of authentication you use.
Forn example: Do you use HTTP or Java Client for indexing? If HTTP: Do you have SSL enabled and if so do you use native OpenSSL support?
Maybe SSL is slowing down the whole thing. Beta1 introduced caching so hitting the authentication backend should normally not cause any pain.

Please provide also informations for the folowing questions:

  • How many nodes do you have in your cluster?
  • JVM vendor and version
  • Operation system vendor and version
  • OpenSSL version

Thx

Am 10.06.2016 um 18:31 schrieb Jainin Shah jai...@tresata.com:

Hi,

We have installed searchguard 2.3.2-beta2 in our elasticsearch cluster. Everything went well with the installation. But what we noticed is that bulk indexing (from hdfs to es particularly) is quite slow after enabling searchguard in the cluster.

For ex, we were indexing 50 million document in our cluster, which would generally take no more than few hours (usually between 2-3), but after installing searchguard the same indexing took more than 9 hours.

Is that an expected behavior while indexing with searchguard? and/or
Is that fixed (or improved) in the RC1 release?

Thank you

Jainin Shah


You received this message because you are subscribed to the Google Groups “Search Guard” group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/aea72d49-2f6e-40c2-8d97-464744316fd6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups “Search Guard” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/c31c23da-503b-4095-9c9a-a326b1beef8c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

<sg_config.yml><elasricsearch.yml>