Searchguard Configuration with Elastic Azure LoadBalancer DNS(4 node cluster)

Search Guard
1

Elasticsearch version 6.5.4
Kibana Version 6.5.4

Search Guard version 6:6.5.4-24.1

Please suggest me how to setup Searchguard with Load balancer DNS to access Elastic cluster from Client machines to access using client certs.

i have Elasticsearch setup with 4 node cluster and it has Azure Loadbalancer infront of 4 nodes, i have DNS name assigned to Loadbalncer IP so that from client machines will connect through Loadbalancer DNS on port 9200 (http://elastic-nprod-example.net:9200), even if i re-provision Elastic Cluster i just need to update DNS with new Loadbalancer ip, no need to update anything on Client machines.

Right now am trying to install searchguard on all the nodes, i am using Offline TLS tool to generate certs for nodes by following below documentation.

TLS tool: https://search.maven.org/search?q=a:search-guard-tlstool

Once certs generated for all the nodes and initialize sgadmin then am able to access nodes using certs or user/Password from client machines by hitting either node DNS or node ip on port 9200.

i have updated LoadBalancer ip in sg_config.yml to allow access.

i want to access Elastic Cluster all the nodes using certs by hitting Elastic Load balancer DNS not Node DNS, so that tomorrow even one nodes dies others will available to get the logs from Client machines.

i have tried to generate certs in multiple ways with offline TLS tool.

  1. adding Loadbalncer DNS and IP with both nodes section in tls_config.yml

nodes:

  • name: elastic1

dn: CN=elastic1.example.net,OU=Ops,O=example EX, Ltd.,DC=example,DC=net

dns:

ip:

  • 10.10.10.10

  • 10.10.10.9

  • name: elastic2

dn: CN=elastic2.example.net,OU=Ops,O=example EX, Ltd.,DC=example,DC=net

dns:

ip:

  • 10.10.10.11

  • 10.10.10.9

This method seems to be working but not always, sometimes when i hit elastic cluster using Loadbalncer DNS on port 443:9200 using certs it’s complaining SSL(curl: (51) Unable to communicate securely with peer: requested domain name does not match the server’s certificate.)

  1. Adding Loadbalancer DNS in wildcard entry.

nodesDn:

#- “CN=*.example.com,OU=Ops,O=example Com\, Inc.,DC=example,DC=net”

- ‘CN=*.example.com,OU=SSL,O=Test,L=Test,C=DE’

- ‘CN=elk-devcluster*’

- ‘/CN=.*regex/’

i have added Loadbalncer IP into sg_config.yml in both scenarions, this one seems not working, am not able to access cluster on 443 using client certs with Loadbalancer DNS.

Thanks in advance for help!!!

2

i have also added below config in sg_config.yml, to allow traffic through Loadbalancer ip, still not working, Please suggest me whats wrong with it.

http:

anonymous_auth_enabled: false

xff:

  • enabled: true*

internalProxies: ‘10.10.10.9’ # LB IP

#internalProxies: ‘.*’ # trust all internal proxies, regex pattern

remoteIpHeader: ‘x-forwarded-for’

proxiesHeader: ‘x-forwarded-by’

and am trying to hit elastic using load balancer ip with header “f-forwarded-for”

curl --header “x-forwarded-for” https://10.10.10.9:443/_cluster/health?pretty --key /home/elastic/out-new2/admin.key --cert /home/elastic/out-new2/admin.pem --cacert /home/elastic/out-new2/root-ca.pem

Error:

curl: (51) Unable to communicate securely with peer: requested domain name does not match the server’s certificate.

···

On Thursday, March 7, 2019 at 9:19:16 AM UTC-8, Vijay wrote:

Elasticsearch version 6.5.4
Kibana Version 6.5.4

Search Guard version 6:6.5.4-24.1

Please suggest me how to setup Searchguard with Load balancer DNS to access Elastic cluster from Client machines to access using client certs.

i have Elasticsearch setup with 4 node cluster and it has Azure Loadbalancer infront of 4 nodes, i have DNS name assigned to Loadbalncer IP so that from client machines will connect through Loadbalancer DNS on port 9200 (http://elastic-nprod-example.net:9200), even if i re-provision Elastic Cluster i just need to update DNS with new Loadbalancer ip, no need to update anything on Client machines.

Right now am trying to install searchguard on all the nodes, i am using Offline TLS tool to generate certs for nodes by following below documentation.

https://gryzli.info/2018/12/01/elasticsearchelk-stack-security-howto/

TLS tool: https://search.maven.org/search?q=a:search-guard-tlstool

Once certs generated for all the nodes and initialize sgadmin then am able to access nodes using certs or user/Password from client machines by hitting either node DNS or node ip on port 9200.

i have updated LoadBalancer ip in sg_config.yml to allow access.

i want to access Elastic Cluster all the nodes using certs by hitting Elastic Load balancer DNS not Node DNS, so that tomorrow even one nodes dies others will available to get the logs from Client machines.

i have tried to generate certs in multiple ways with offline TLS tool.

  1. adding Loadbalncer DNS and IP with both nodes section in tls_config.yml

nodes:

  • name: elastic1

dn: CN=elastic1.example.net,OU=Ops,O=example EX, Ltd.,DC=example,DC=net

dns:

ip:

  • 10.10.10.10
  • 10.10.10.9
  • name: elastic2

dn: CN=elastic2.example.net,OU=Ops,O=example EX, Ltd.,DC=example,DC=net

dns:

ip:

  • 10.10.10.11
  • 10.10.10.9

This method seems to be working but not always, sometimes when i hit elastic cluster using Loadbalncer DNS on port 443:9200 using certs it’s complaining SSL(curl: (51) Unable to communicate securely with peer: requested domain name does not match the server’s certificate.)

  1. Adding Loadbalancer DNS in wildcard entry.

nodesDn:

#- “CN=*.example.com,OU=Ops,O=example Com\, Inc.,DC=example,DC=net”

- ‘CN=*.example.com,OU=SSL,O=Test,L=Test,C=DE’

- ‘CN=elk-devcluster*’

- ‘/CN=.*regex/’

i have added Loadbalncer IP into sg_config.yml in both scenarions, this one seems not working, am not able to access cluster on 443 using client certs with Loadbalancer DNS.

Thanks in advance for help!!!

3

i have enabled proxy authentication and re-initialized SG, still not able to access elastic cluster with username and password

proxy_auth_domain:

http_enabled: true

transport_enabled: true

order: 3

http_authenticator:

type: proxy

challenge: true

config:

user_header: “x-proxy-user”

roles_header: “x-proxy-roles”

authentication_backend:

type: noop

···

On Thursday, March 7, 2019 at 9:19:16 AM UTC-8, Vijay wrote:

Elasticsearch version 6.5.4
Kibana Version 6.5.4

Search Guard version 6:6.5.4-24.1

Please suggest me how to setup Searchguard with Load balancer DNS to access Elastic cluster from Client machines to access using client certs.

i have Elasticsearch setup with 4 node cluster and it has Azure Loadbalancer infront of 4 nodes, i have DNS name assigned to Loadbalncer IP so that from client machines will connect through Loadbalancer DNS on port 9200 (http://elastic-nprod-example.net:9200), even if i re-provision Elastic Cluster i just need to update DNS with new Loadbalancer ip, no need to update anything on Client machines.

Right now am trying to install searchguard on all the nodes, i am using Offline TLS tool to generate certs for nodes by following below documentation.

https://gryzli.info/2018/12/01/elasticsearchelk-stack-security-howto/

TLS tool: https://search.maven.org/search?q=a:search-guard-tlstool

Once certs generated for all the nodes and initialize sgadmin then am able to access nodes using certs or user/Password from client machines by hitting either node DNS or node ip on port 9200.

i have updated LoadBalancer ip in sg_config.yml to allow access.

i want to access Elastic Cluster all the nodes using certs by hitting Elastic Load balancer DNS not Node DNS, so that tomorrow even one nodes dies others will available to get the logs from Client machines.

i have tried to generate certs in multiple ways with offline TLS tool.

  1. adding Loadbalncer DNS and IP with both nodes section in tls_config.yml

nodes:

  • name: elastic1

dn: CN=elastic1.example.net,OU=Ops,O=example EX, Ltd.,DC=example,DC=net

dns:

ip:

  • 10.10.10.10
  • 10.10.10.9
  • name: elastic2

dn: CN=elastic2.example.net,OU=Ops,O=example EX, Ltd.,DC=example,DC=net

dns:

ip:

  • 10.10.10.11
  • 10.10.10.9

This method seems to be working but not always, sometimes when i hit elastic cluster using Loadbalncer DNS on port 443:9200 using certs it’s complaining SSL(curl: (51) Unable to communicate securely with peer: requested domain name does not match the server’s certificate.)

  1. Adding Loadbalancer DNS in wildcard entry.

nodesDn:

#- “CN=*.example.com,OU=Ops,O=example Com\, Inc.,DC=example,DC=net”

- ‘CN=*.example.com,OU=SSL,O=Test,L=Test,C=DE’

- ‘CN=elk-devcluster*’

- ‘/CN=.*regex/’

i have added Loadbalncer IP into sg_config.yml in both scenarions, this one seems not working, am not able to access cluster on 443 using client certs with Loadbalancer DNS.

Thanks in advance for help!!!

4

@search guard

[BLOCKED]Please someone suggest me whats wrong with my setup, am blocked because of this issue.

I have Azure Loadbalancer rule for port forwarding 443:9200

i red one of the documentation saying that LB forward header to Es cluster, how that can be setup?

i am able to access cluster using node DNS with client cert

curl -u user:password https://10.10.10.10:9200/_cluster/health?pretty --cacert /home/elastic/out-new2/root-ca.pem

{

“cluster_name” : “escluster-elastictest”,

“status” : “green”,

“timed_out” : false,

“number_of_nodes” : 2,

“number_of_data_nodes” : 2,

“active_primary_shards” : 1,

“active_shards” : 2,

“relocating_shards” : 0,

“initializing_shards” : 0,

“unassigned_shards” : 0,

“delayed_unassigned_shards” : 0,

“number_of_pending_tasks” : 0,

“number_of_in_flight_fetch” : 0,

“task_max_waiting_in_queue_millis” : 0,

“active_shards_percent_as_number” : 100.0

sg_config.yml (9.34 KB)

elasticsearch.yml (4.61 KB)

···

On Thursday, March 7, 2019 at 9:19:16 AM UTC-8, Vijay wrote:

Elasticsearch version 6.5.4
Kibana Version 6.5.4

Search Guard version 6:6.5.4-24.1

Please suggest me how to setup Searchguard with Load balancer DNS to access Elastic cluster from Client machines to access using client certs.

i have Elasticsearch setup with 4 node cluster and it has Azure Loadbalancer infront of 4 nodes, i have DNS name assigned to Loadbalncer IP so that from client machines will connect through Loadbalancer DNS on port 9200 (http://elastic-nprod-example.net:9200), even if i re-provision Elastic Cluster i just need to update DNS with new Loadbalancer ip, no need to update anything on Client machines.

Right now am trying to install searchguard on all the nodes, i am using Offline TLS tool to generate certs for nodes by following below documentation.

https://gryzli.info/2018/12/01/elasticsearchelk-stack-security-howto/

TLS tool: https://search.maven.org/search?q=a:search-guard-tlstool

Once certs generated for all the nodes and initialize sgadmin then am able to access nodes using certs or user/Password from client machines by hitting either node DNS or node ip on port 9200.

i have updated LoadBalancer ip in sg_config.yml to allow access.

i want to access Elastic Cluster all the nodes using certs by hitting Elastic Load balancer DNS not Node DNS, so that tomorrow even one nodes dies others will available to get the logs from Client machines.

i have tried to generate certs in multiple ways with offline TLS tool.

  1. adding Loadbalncer DNS and IP with both nodes section in tls_config.yml

nodes:

  • name: elastic1

dn: CN=elastic1.example.net,OU=Ops,O=example EX, Ltd.,DC=example,DC=net

dns:

ip:

  • 10.10.10.10
  • 10.10.10.9
  • name: elastic2

dn: CN=elastic2.example.net,OU=Ops,O=example EX, Ltd.,DC=example,DC=net

dns:

ip:

  • 10.10.10.11
  • 10.10.10.9

This method seems to be working but not always, sometimes when i hit elastic cluster using Loadbalncer DNS on port 443:9200 using certs it’s complaining SSL(curl: (51) Unable to communicate securely with peer: requested domain name does not match the server’s certificate.)

  1. Adding Loadbalancer DNS in wildcard entry.

nodesDn:

#- “CN=*.example.com,OU=Ops,O=example Com\, Inc.,DC=example,DC=net”

- ‘CN=*.example.com,OU=SSL,O=Test,L=Test,C=DE’

- ‘CN=elk-devcluster*’

- ‘/CN=.*regex/’

i have added Loadbalncer IP into sg_config.yml in both scenarions, this one seems not working, am not able to access cluster on 443 using client certs with Loadbalancer DNS.

Thanks in advance for help!!!

5


Why did you try/enabled proxy authentication and xff? Is your LB acting as a proxy?
I have no experience with Azure but a quick look into the documentation makes me believe that they are “Application agnostic and transparent”
(https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-overview)

In that case, you simply need to add the LB DNS name to the https certificates and that should work.
No need to enable proxy auth/xff.

Also, mind the following:

···

On Friday, 8 March 2019 15:47:34 UTC+1, Vijay wrote:

@search guard

[BLOCKED]Please someone suggest me whats wrong with my setup, am blocked because of this issue.

I have Azure Loadbalancer rule for port forwarding 443:9200

i red one of the documentation saying that LB forward header to Es cluster, how that can be setup?

i am able to access cluster using node DNS with client cert

curl -u user:password https://10.10.10.10:9200/_cluster/health?pretty --cacert /home/elastic/out-new2/root-ca.pem

{

“cluster_name” : “escluster-elastictest”,

“status” : “green”,

“timed_out” : false,

“number_of_nodes” : 2,

“number_of_data_nodes” : 2,

“active_primary_shards” : 1,

“active_shards” : 2,

“relocating_shards” : 0,

“initializing_shards” : 0,

“unassigned_shards” : 0,

“delayed_unassigned_shards” : 0,

“number_of_pending_tasks” : 0,

“number_of_in_flight_fetch” : 0,

“task_max_waiting_in_queue_millis” : 0,

“active_shards_percent_as_number” : 100.0

On Thursday, March 7, 2019 at 9:19:16 AM UTC-8, Vijay wrote:

Elasticsearch version 6.5.4
Kibana Version 6.5.4

Search Guard version 6:6.5.4-24.1

Please suggest me how to setup Searchguard with Load balancer DNS to access Elastic cluster from Client machines to access using client certs.

i have Elasticsearch setup with 4 node cluster and it has Azure Loadbalancer infront of 4 nodes, i have DNS name assigned to Loadbalncer IP so that from client machines will connect through Loadbalancer DNS on port 9200 (http://elastic-nprod-example.net:9200), even if i re-provision Elastic Cluster i just need to update DNS with new Loadbalancer ip, no need to update anything on Client machines.

Right now am trying to install searchguard on all the nodes, i am using Offline TLS tool to generate certs for nodes by following below documentation.

https://gryzli.info/2018/12/01/elasticsearchelk-stack-security-howto/

TLS tool: https://search.maven.org/search?q=a:search-guard-tlstool

Once certs generated for all the nodes and initialize sgadmin then am able to access nodes using certs or user/Password from client machines by hitting either node DNS or node ip on port 9200.

i have updated LoadBalancer ip in sg_config.yml to allow access.

i want to access Elastic Cluster all the nodes using certs by hitting Elastic Load balancer DNS not Node DNS, so that tomorrow even one nodes dies others will available to get the logs from Client machines.

i have tried to generate certs in multiple ways with offline TLS tool.

  1. adding Loadbalncer DNS and IP with both nodes section in tls_config.yml

nodes:

  • name: elastic1

dn: CN=elastic1.example.net,OU=Ops,O=example EX, Ltd.,DC=example,DC=net

dns:

ip:

  • 10.10.10.10
  • 10.10.10.9
  • name: elastic2

dn: CN=elastic2.example.net,OU=Ops,O=example EX, Ltd.,DC=example,DC=net

dns:

ip:

  • 10.10.10.11
  • 10.10.10.9

This method seems to be working but not always, sometimes when i hit elastic cluster using Loadbalncer DNS on port 443:9200 using certs it’s complaining SSL(curl: (51) Unable to communicate securely with peer: requested domain name does not match the server’s certificate.)

  1. Adding Loadbalancer DNS in wildcard entry.

nodesDn:

#- “CN=*.example.com,OU=Ops,O=example Com\, Inc.,DC=example,DC=net”

- ‘CN=*.example.com,OU=SSL,O=Test,L=Test,C=DE’

- ‘CN=elk-devcluster*’

- ‘/CN=.*regex/’

i have added Loadbalncer IP into sg_config.yml in both scenarions, this one seems not working, am not able to access cluster on 443 using client certs with Loadbalancer DNS.

Thanks in advance for help!!!