ES Cluster is not formed with searchguard plugin

I have three instances of ES. I have followed following steps to form cluster configuration as follows

  1. I have used same cluster name

  2. I have used different node names

  3. I have marked all nodes and master and data nodes

  4. I have disabled multicaset

  5. Provided unicast ip’s

  6. Disabled searchguard

  7. When I restarted ES and verified the o/p of http://localhost:9201/_cluster/health?pretty=true. I have noticed that ES has formed cluster. It’s all good upto this point

Now,

  1. I have stopped ES.

  2. I have enabled search guard

  3. I have copied searchguard key from one node to other nodes

  4. I have imported SSL certificates of one node to others truststore

  5. restarted ES

  6. The ES didn’t form the cluster. The ES is started with 503 error. If I access any ES urls’s it is giving error as {“error”:“MasterNotDiscoveredException[waited for [30s]]”,“status”:503}

fyi; The above configuration used to work however it is not working now all of sudden.

I have the following searchguard rules stored in ES

{

“acl”: [

{

Comment”: “Default is to execute all filters”,

“filters_bypass”: [

“*”

],

“filters_execute”: [

“*”

]

},

{

Comment”: “Any authenticated user do anything on the ‘_all’, ‘searchguard’, ‘inventory’ index - no filter will be executed”,

“indices”: [

“_all”,

“searchguard”,

“inventory”

],

“filters_bypass”: [

“*”

],

“filters_execute”: [

“*”

]

},

{

Comment”: “For admin role all filters are bypassed (so none will be executed) for all indices. This means unrestricted access at all for this role.”,

“roles”: [

“admin”

],

“filters_bypass”: [

“*”

],

“filters_execute”:

}

]

}

Elasticsearch.yml file

##################### Elasticsearch Configuration Example #####################

This file contains an overview of various configuration settings,

targeted at operations staff. Application developers should

consult the guide at http://elasticsearch.org/guide.

···

The installation procedure is covered at

http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html.

Elasticsearch comes with reasonable defaults for most settings,

so you can try it out without bothering with configuration.

Most of the time, these defaults are just fine for running a production

cluster. If you’re fine-tuning your cluster, or wondering about the

effect of certain configuration option, please do ask on the

mailing list or IRC channel [Free and Open Search: The Creators of Elasticsearch, ELK & Kibana | Elastic].

Any element in the configuration can be replaced with environment variables

by placing them in ${…} notation. For example:

#node.rack: ${RACK_ENV_VAR}

For information on supported formats and syntax for the config file, see

http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html

################################### Cluster ###################################

Cluster name identifies your cluster for auto-discovery. If you’re running

multiple clusters on the same network, make sure you’re using unique names.

#cluster.name: “elasticsearch”

cluster.name: “sample”

#################################### Node #####################################

Node names are generated dynamically on startup, so you’re relieved

from configuring them manually. You can tie this node to a specific name:

node.name: “pm-cdl-win-2”

Every node can be configured to allow or deny being eligible as the master,

and to allow or deny to store the data.

Allow this node to be eligible as a master node (enabled by default):

node.master: true

Allow this node to store data (enabled by default):

node.data: true

You can exploit these settings to design advanced cluster topologies.

1. You want this node to never become a master node, only to hold data.

This will be the “workhorse” of your cluster.

#node.master: true

#node.data: true

2. You want this node to only serve as a master: to not store any data and

to have free resources. This will be the “coordinator” of your cluster.

#node.master: true

#node.data: false

3. You want this node to be neither master nor data node, but

to act as a “search load balancer” (fetching data from nodes,

aggregating results, etc.)

#node.master: false

#node.data: false

Use the Cluster Health API [http://localhost:9200/_cluster/health], the

Node Info API [http://localhost:9200/_nodes] or GUI tools

such as http://www.elasticsearch.org/overview/marvel/,

http://github.com/karmi/elasticsearch-paramedic,

http://github.com/lukas-vlcek/bigdesk and

http://mobz.github.com/elasticsearch-head to inspect the cluster state.

A node can have generic attributes associated with it, which can later be used

for customized shard allocation filtering, or allocation awareness. An attribute

is a simple key value pair, similar to node.key: value, here is an example:

#node.rack: rack314

By default, multiple nodes are allowed to start from the same installation location

to disable it, set the following:

#node.max_local_storage_nodes: 1

node.local: false

#################################### Index ####################################

You can set a number of options (such as shard/replica options, mapping

or analyzer definitions, translog settings, …) for indices globally,

in this file.

Note, that it makes more sense to configure index settings specifically for

a certain index, either when creating it or by using the index templates API.

See http://elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules.html and

http://elasticsearch.org/guide/en/elasticsearch/reference/current/indices-create-index.html

for more information.

Set the number of shards (splits) of an index (5 by default):

#index.number_of_shards: 5

Set the number of replicas (additional copies) of an index (1 by default):

#index.number_of_replicas: 1

Note, that for development on a local machine, with small indices, it usually

makes sense to “disable” the distributed features:

#index.number_of_shards: 1

#index.number_of_replicas: 0

These settings directly affect the performance of index and search operations

in your cluster. Assuming you have enough machines to hold shards and

replicas, the rule of thumb is:

1. Having more shards enhances the indexing performance and allows to

distribute a big index across machines.

2. Having more replicas enhances the search performance and improves the

cluster availability.

The “number_of_shards” is a one-time setting for an index.

The “number_of_replicas” can be increased or decreased anytime,

by using the Index Update Settings API.

Elasticsearch takes care about load balancing, relocating, gathering the

results from nodes, etc. Experiment with different settings to fine-tune

your setup.

Use the Index Status API (http://localhost:9200/A/_status) to inspect

the index status.

#################################### Paths ####################################

Path to directory containing configuration (this file and logging.yml):

#path.conf: /path/to/conf

Path to directory where to store index data allocated for this node.

#path.data: /path/to/data

Can optionally include more than one location, causing data to be striped across

the locations (a la RAID 0) on a file level, favouring locations with most free

space on creation. For example:

#path.data: /path/to/data1,/path/to/data2

Path to temporary files:

#path.work: /path/to/work

Path to log files:

#path.logs: /path/to/logs

Path to where plugins are installed:

#path.plugins: /path/to/plugins

#################################### Plugin ###################################

If a plugin listed here is not installed for current node, the node will not start.

#plugin.mandatory: mapper-attachments,lang-groovy

################################### Memory ####################################

Elasticsearch performs poorly when JVM starts swapping: you should ensure that

it never swaps.

Set this property to true to lock the memory:

#bootstrap.mlockall: true

Make sure that the ES_MIN_MEM and ES_MAX_MEM environment variables are set

to the same value, and that the machine has enough memory to allocate

for Elasticsearch, leaving enough memory for the operating system itself.

You should also make sure that the Elasticsearch process is allowed to lock

the memory, eg. by using ulimit -l unlimited.

############################## Network And HTTP ###############################

Elasticsearch, by default, binds itself to the 0.0.0.0 address, and listens

on port [9200-9300] for HTTP traffic and on port [9300-9400] for node-to-node

communication. (the range means that if the port is busy, it will automatically

try the next port).

Set the bind address specifically (IPv4 or IPv6):

#network.bind_host: localhost

Set the address other nodes will use to communicate with this node. If not

set, it is automatically derived. It must point to an actual IP address.

#network.publish_host: 192.168.0.1

Set both ‘bind_host’ and ‘publish_host’:

#network.host: 192.168.0.1

Set a custom port for the node to node communication (9300 by default):

#transport.tcp.port: 9300

transport.tcp.port: 9300

Enable compression for all communication between nodes (disabled by default):

#transport.tcp.compress: true

Set a custom port to listen for HTTP traffic:

http.port: 9201

Set a custom allowed content length:

#http.max_content_length: 100mb

Disable HTTP completely:

http.enabled: true

################################### Gateway ###################################

The gateway allows for persisting the cluster state between full cluster

restarts. Every change to the state (such as adding an index) will be stored

in the gateway, and when the cluster starts up for the first time,

it will read its state from the gateway.

There are several types of gateway implementations. For more information, see

http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-gateway.html.

The default gateway type is the “local” gateway (recommended):

#gateway.type: local

Settings below control how and when to start the initial recovery process on

a full cluster restart (to reuse as much local data as possible when using shared

gateway).

Allow recovery process after N nodes in a cluster are up:

#gateway.recover_after_nodes: 1

Set the timeout to initiate the recovery process, once the N nodes

from previous setting are up (accepts time value):

#gateway.recover_after_time: 5m

Set how many nodes are expected in this cluster. Once these N nodes

are up (and recover_after_nodes is met), begin recovery process immediately

(without waiting for recover_after_time to expire):

#gateway.expected_nodes: 2

############################# Recovery Throttling #############################

These settings allow to control the process of shards allocation between

nodes during initial recovery, replica allocation, rebalancing,

or when adding and removing nodes.

Set the number of concurrent recoveries happening on a node:

1. During the initial recovery

#cluster.routing.allocation.node_initial_primaries_recoveries: 4

2. During adding/removing nodes, rebalancing, etc

#cluster.routing.allocation.node_concurrent_recoveries: 2

Set to throttle throughput when recovering (eg. 100mb, by default 20mb):

#indices.recovery.max_bytes_per_sec: 20mb

Set to limit the number of open concurrent streams when

recovering a shard from a peer:

#indices.recovery.concurrent_streams: 5

################################## Discovery ##################################

Discovery infrastructure ensures nodes can be found within a cluster

and master node is elected. Multicast discovery is the default.

Set to ensure a node sees N other master eligible nodes to be considered

operational within the cluster. This should be set to a quorum/majority of

the master-eligible nodes in the cluster.

#discovery.zen.minimum_master_nodes: 1

Set the time to wait for ping responses from other nodes when discovering.

Set this option to a higher value on a slow or congested network

to minimize discovery failures:

discovery.zen.ping.timeout: 5s

For more information, see

http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-zen.html

Unicast discovery allows to explicitly control which nodes will be used

to discover the cluster. It can be used when multicast is not present,

or to restrict the cluster communication-wise.

1. Disable multicast discovery (enabled by default):

discovery.zen.ping.multicast.enabled: false

2. Configure an initial list of master nodes in the cluster

to perform discovery when new nodes (master or data) are started:

#discovery.zen.ping.unicast.hosts: [“host”: “host:port”]

discovery.zen.ping.unicast.hosts: [“pm-cdl-win-1.cdl.local”, “pm-cdl-win-2.cdl.local”, “pm-cdl-win-3.cdl.local”]

#discovery.zen.ping.unicast.hosts: [“10.1.16.11”, “10.1.16.17”, “10.1.16.19”]

#discovery.zen.no_master_block: all

discovery.zen.minimum_master_nodes: 1

EC2 discovery allows to use AWS EC2 API in order to perform discovery.

You have to install the cloud-aws plugin for enabling the EC2 discovery.

For more information, see

http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-ec2.html

See http://elasticsearch.org/tutorials/elasticsearch-on-ec2/

for a step-by-step tutorial.

GCE discovery allows to use Google Compute Engine API in order to perform discovery.

You have to install the cloud-gce plugin for enabling the GCE discovery.

For more information, see https://github.com/elasticsearch/elasticsearch-cloud-gce.

Azure discovery allows to use Azure API in order to perform discovery.

You have to install the cloud-azure plugin for enabling the Azure discovery.

For more information, see https://github.com/elasticsearch/elasticsearch-cloud-azure.

################################## Slow Log ##################################

Shard level query and fetch threshold logging.

#index.search.slowlog.threshold.query.warn: 10s

#index.search.slowlog.threshold.query.info: 5s

#index.search.slowlog.threshold.query.debug: 2s

#index.search.slowlog.threshold.query.trace: 500ms

#index.search.slowlog.threshold.fetch.warn: 1s

#index.search.slowlog.threshold.fetch.info: 800ms

#index.search.slowlog.threshold.fetch.debug: 500ms

#index.search.slowlog.threshold.fetch.trace: 200ms

#index.indexing.slowlog.threshold.index.warn: 10s

#index.indexing.slowlog.threshold.index.info: 5s

#index.indexing.slowlog.threshold.index.debug: 2s

#index.indexing.slowlog.threshold.index.trace: 500ms

################################## GC Logging ################################

#monitor.jvm.gc.young.warn: 1000ms

#monitor.jvm.gc.young.info: 700ms

#monitor.jvm.gc.young.debug: 400ms

#monitor.jvm.gc.old.warn: 10s

#monitor.jvm.gc.old.info: 5s

#monitor.jvm.gc.old.debug: 2s

################################## Security ################################

Uncomment if you want to enable JSONP as a valid return transport on the

http server. With this enabled, it may pose a security risk, so disabling

it unless you need it is recommended (it is disabled by default).

#http.jsonp.enable: true

#############################################################################################

SEARCH GUARD

Configuration

#############################################################################################

#Note: All waffle related options are only valid if your ES node is running on windows OS

Enable or disable the complete Searchguard plugin functionality

searchguard.enabled: true

Path where to write/read the searchguard master key file

#searchguard.key_path: .

When using DLS or FLS and a get or mget is performed then rewrite it as search request

#searchguard.rewrite_get_as_search: true

The index name where Searchguard will store its configuration and various other informations related to Searchguard itself

This index can only be access from localhost

searchguard.config_index_name: searchguard

Enable or disable HTTP session which caches the authentication and authorization informations in a cookie

#searchguard.http.enable_sessions: false

Enable or disable audit logging

#searchguard.auditlog.enabled: true

If this is true (default is false) then Searchguard will check if elasticsearch is running as root/windows admin and if so then abort.

searchguard.check_for_root: false

If this is true (default is false) then allow all HTTP REST requests from nodes loopback (e.g. localhost)

searchguard.allow_all_from_loopback: true

If this is true (default: false) then enable authenticated transports requests (e.g. TransportClient authentication)

This can be done in that way (for example):

TransportClient.get(new GetRequest(“marketing”, “customer”, “tp_3”).putHeader(“searchguard_transport_creds”, “c2FseWg6c2VjcmV0”))

Add a header “searchguard_transport_creds”

base64(username":"password) is the credentials string

base64(spock:secret) → c3BvY2s6c2VjcmV0

searchguard.transport_auth.enabled: false

#############################################################################################

Transport layer SSL

#############################################################################################

Enable or disable node-to-node ssl encryption

searchguard.ssl.transport.node.enabled: true

JKS or PKCS12

searchguard.ssl.transport.node.keystore_type: JKS

Absolute path to the keystore file (this stores the server certificates)

searchguard.ssl.transport.node.keystore_filepath: C:\Program Files\Hewlett-Packard\CSA/jboss-as/standalone/configuration/.keystore

Keystore password

searchguard.ssl.transport.node.keystore_password: changeit

Do other nodes have to authenticate themself to the cluster, default is true

searchguard.ssl.transport.node.enforce_clientauth: false

JKS or PKCS12

searchguard.ssl.transport.node.truststore_type: JKS

Absolute path to the truststore file (this stores the client certificates)

searchguard.ssl.transport.node.truststore_filepath: C:\Program Files\Hewlett-Packard\CSA/openjre/lib/security/cacerts

Truststore password

searchguard.ssl.transport.node.truststore_password: changeit

#############################################################################################

REST layer SSL

#############################################################################################

Enable or disable rest layer security (https)

searchguard.ssl.transport.http.enabled: true

JKS or PKCS12

searchguard.ssl.transport.http.keystore_type: JKS

Absolute path to the keystore file (this stores the server certificates)

searchguard.ssl.transport.http.keystore_filepath: C:\Program Files\Hewlett-Packard\CSA/jboss-as/standalone/configuration/.keystore

Keystore password

searchguard.ssl.transport.http.keystore_password: changeit

Do the clients (typically the browser or the proxy) have to authenticate themself to the http server, default is false

searchguard.ssl.transport.http.enforce_clientauth: false

JKS or PKCS12

searchguard.ssl.transport.http.truststore_type: JKS

Absolute path to the truststore file (this stores the client certificates)

searchguard.ssl.transport.http.truststore_filepath: C:\Program Files\Hewlett-Packard\CSA/openjre/lib/security/cacerts

Truststore password

searchguard.ssl.transport.http.truststore_password: changeit

#############################################################################################

X-Forwarded-For (XFF) header

#############################################################################################

X-Forwarded-For (XFF) header

If you have a http proxy in front of elasticsearch you have to configure this options to handle XFF properly

#searchguard.http.xforwardedfor.header: X-Forwarded-For

#searchguard.http.xforwardedfor.trustedproxies: null

#searchguard.http.xforwardedfor.enforce: false

#############################################################################################

Authentication backend

#############################################################################################

Validates the username and credentials

searchguard.authentication.authentication_backend.impl: com.floragunn.searchguard.authentication.backend.simple.SettingsBasedAuthenticationBackend

#searchguard.authentication.authentication_backend.impl: com.floragunn.searchguard.authentication.backend.ldap.LDAPAuthenticationBackend

#searchguard.authentication.authentication_backend.impl: com.floragunn.searchguard.authentication.backend.simple.AlwaysSucceedAuthenticationBackend

#searchguard.authentication.authentication_backend.impl: com.floragunn.searchguard.authentication.backend.waffle.WaffleAuthenticationBackend

If caching is enabled then the authentication succeed for 24 h since the first successful login without hitting the backend again and again

#searchguard.authentication.authentication_backend.cache.enable: true

#############################################################################################

Authorization backend (authorizer)

#############################################################################################

searchguard.authentication.authorizer.impl: com.floragunn.searchguard.authorization.simple.SettingsBasedAuthorizator

#searchguard.authentication.authorizer.impl: com.floragunn.searchguard.authorization.ldap.LDAPAuthorizator

#searchguard.authentication.authorizer.impl: com.floragunn.searchguard.authorization.waffle.WaffleAuthorizator

If caching is enabled then the role informations will be cached for 24 h without hitting the backend again and again

#searchguard.authentication.authorizer.cache.enable: true

#############################################################################################

HTTP authentication method

#############################################################################################

Define HTTP authentication method. In future we will here have more like NTLM, SPNEGO/Kerberos and Digest.

searchguard.authentication.http_authenticator.impl: com.floragunn.searchguard.authentication.http.basic.HTTPBasicAuthenticator

HTTPProxyAuthenticator assume there is kind of proxy in front of elasticsearch which handles the authentication and stores the

username of the authenticated user in a http header

#searchguard.authentication.http_authenticator.impl: com.floragunn.searchguard.authentication.http.proxy.HTTPProxyAuthenticator

SSL mutual authentication (works only if searchguard.ssl.transport.http.enabled is ‘true’ with client auth enabled)

#searchguard.authentication.http_authenticator.impl: com.floragunn.searchguard.authentication.http.clientcert.HTTPSClientCertAuthenticator

SPNEGO

#searchguard.authentication.http_authenticator.impl: com.floragunn.searchguard.authentication.http.spnego.HTTPSpnegoAuthenticator

Absolute file path to jaas login config file

#searchguard.authentication.spnego.login_config_filepath: null

Absolute file path to krb5 config file

#searchguard.authentication.spnego.krb5_config_filepath: null

Name of the login entry in jaas login config file which represents the acceptor (server)

#searchguard.authentication.spnego.login_config_name: com.sun.security.jgss.krb5.accept

Strip the realmname from username (hnelson@EXAMPLE.COM → hnelson)

#searchguard.authentication.spnego.strip_realm: true

Authenticates always a user with username ‘searchguard_unauthenticated_user’

#searchguard.authentication.http_authenticator.impl: com.floragunn.searchguard.authentication.http.HTTPUnauthenticatedAuthenticator

Waffle (Windows only, must be used with WaffleAuthorizator)

#searchguard.authentication.http_authenticator.impl: com.floragunn.searchguard.authentication.http.waffle.HTTPWaffleAuthenticator

Strip domain name from user (COMPANY\spock → spock)

#searchguard.authentication.waffle.strip_domain: true

#####################################################

Settings based authentication (define users and password directly here in the settings. Note: this is per node)

#searchguard.authentication.settingsdb.user.: password

searchguard.authentication.settingsdb.user.admin: cloud

#searchguard.authentication.settingsdb.user.user1: password

#searchguard.authentication.settingsdb.user.michaeljackson: neverland

If plain text password should be hashed use this. Supported digests are: SHA1 SHA256 SHA384 SHA512 MD5

#searchguard.authentication.settingsdb.digest: SHA1

#searchguard.authentication.settingsdb.user.admin: 000e793db70c59309fa6f0f36d0046d110f3be3c

#searchguard.authentication.settingsdb.user.michaeljackson: 824d55e7a62b7ca8751dff346ffab845a8f26d08

#####################################################

#####################################################

Settings based authorization (define users and their roles directly here in the settings. Note: this is per node)

#searchguard.authentication.authorization.settingsdb.roles.:

searchguard.authentication.authorization.settingsdb.roles.admin: [“admin”]

#searchguard.authentication.authorization.settingsdb.roles.user1: [“admin”]

#searchguard.authentication.authorization.settingsdb.roles.michaeljackson: [“admin”]

#####################################################

#####################################################

LDAP authentication backend (authenticate users against a LDAP or Active Directory)

The defaults are sufficient for Active Directory

#searchguard.authentication.ldap.host: [“localhost:389”]

#searchguard.authentication.ldap.ldaps.ssl.enabled: false

#searchguard.authentication.ldap.ldaps.starttls.enabled: false

JKS or PKCS12

#searchguard.authentication.ldap.ldaps.truststore_type: JKS

#searchguard.authentication.ldap.ldaps.truststore_filepath: null

#searchguard.authentication.ldap.ldaps.truststore_password: null

#searchguard.authentication.ldap.bind_dn: null

#searchguard.authentication.ldap.password: null

Default is root dse (“”)

#searchguard.authentication.ldap.userbase: “”

Filter to search for users (currently in the whole subtree beneath userbase)

{0} is substituted with the username

#searchguard.authentication.ldap.usersearch: (sAMAccountName={0})

Use this attribute from the user as username (if not set then DN is used)

#searchguard.authentication.ldap.username_attribute: null

#####################################################

#####################################################

LDAP authorization backend (gather roles from a LDAP or Active Directory, you have to configure the above LDAP authentication backend settings too)

The defaults are sufficient for Active Directory

Default is root dse (“”)

#searchguard.authentication.authorization.ldap.rolebase: “”

Filter to search for roles (currently in the whole subtree beneath rolebase)

{0} is substituted with the DN of the user

{1} is substituted with the username

{2} is substituted with an attribute value from user’s directory entry, of the authenticated user. Use userroleattribute to specify the name of the attribute

#searchguard.authentication.authorization.ldap.rolesearch: (member={0})

Specify the name of the attribute which value should be substituted with {2} above

#searchguard.authentication.authorization.ldap.userroleattribute: null

Roles as an attribute of the user entry

#searchguard.authentication.authorization.ldap.userrolename: memberOf

The attribute in a role entry containing the name of that role

#searchguard.authentication.authorization.ldap.rolename: name

Resolve nested roles transitive (roles which are members of other roles and so on …)

#searchguard.authentication.authorization.ldap.resolve_nested_roles: false

#####################################################

#####################################################

HTTP proxy authenticator configuration

Header name which contains the username

#searchguard.authentication.proxy.header:X-Authenticated-User

Array of trusted IP addresses (this are typically your proxy server(s))

#searchguard.authentication.proxy.trusted_ips: null

#####################################################

#####################################################

HTTP SSL mutual authentication configuration

Attribute of that attribute in the certificate dn which holds the username

#searchguard.authentication.https.clientcert.attributename: cn

#####################################################

##############################################################################################

Below here you configure what authenticated and authorized users are allowed to do (or not)#

This maps to the acl defined in the searchguard configuration index

#############################################################################################

Configure the restactionfilter to allow or forbid action

#searchguard.restactionfilter.names: [“readonly”]

#searchguard.restactionfilter.readonly.allowed_actions: [“*SearchAction”, “RestSearchScrollAction”, “RestClearScrollAction”, “RestGetAction”, “RestGetSourceAction”, “*MainAction”, “RestValidateQueryAction”, “RestMoreLikeThisAction”, “RestPercolateAction”]

#searchguard.restactionfilter.readonly.forbidden_actions: […]

Configure the actionrequestfilter to allow or forbid action

searchguard.actionrequestfilter.names: [“readonly”]

searchguard.actionrequestfilter.readonly.allowed_actions: [“*”]

#searchguard.actionrequestfilter.readonly.forbidden_actions: [“cluster:", "indices:admin”]

Configure document level security (dls) filter

Warning: All this (with the exception of “exists”) only works with not_analyzed fields because a term filter is used internally

Free and Open Search: The Creators of Elasticsearch, ELK & Kibana | Elastic

#searchguard.dlsfilter.names: [“a”, “b”, “c”, “d”, “e”, “f”, “g”]

#searchguard.dlsfilter.a: [“exists”,“field”, “false”] # if field exists (or not) match → false means field must exist

#searchguard.dlsfilter.b: [“term”, “field”,“value”, “false”] # if field==value (or not) match

#searchguard.dlsfilter.d: [“user_name”,“field”, “false”] # if field==username (or not) match

#searchguard.dlsfilter.e: [“user_roles”,“field”, “false”] # if field contaions a user role (or not) match

#searchguard.dlsfilter.f: [“ldap_user_attribute”,“field”, “attribute”, “false”] # if field==userldapattribute(attribute) (or not) match

#searchguard.dlsfilter.g: [“ldap_user_roles”,“field”, “attribute”, “false”] # if field contains ldaprole(attribute) (or not) match

Configure the field level security (fls) filter to filter _source

#searchguard.flsfilter.names: [“stripsensitive”]

#searchguard.flsfilter.stripsensitive.source_includes:

#searchguard.flsfilter.stripsensitive.source_excludes: [“sensitive*”, “public.sensitive*.sub”]

NO CHANGES BELOW THIS LINE !

#############################################################################################

Below there is list of all actionsrequests in elasticsearch 1.4 (for reference) .

Do not uncomment them here, they are configured above: searchguard.actionrequestfilter

#############################################################################################

#cluster:monitor/health

#cluster:admin/nodes/restart

#cluster:admin/nodes/shutdown

#cluster:admin/repository/delete

#cluster:admin/repository/get

#cluster:admin/repository/put

#cluster:admin/repository/verify

#cluster:admin/reroute

#cluster:admin/settings/update

#cluster:admin/snapshot/create

#cluster:admin/snapshot/delete

#cluster:admin/snapshot/get

#cluster:admin/snapshot/restore

#cluster:admin/snapshot/status

#cluster:monitor/nodes/hot_threads

#cluster:monitor/nodes/info

#cluster:monitor/nodes/liveness

#cluster:monitor/nodes/stats

#cluster:monitor/state

#cluster:monitor/stats

#cluster:monitor/task

#indices:admin/aliases

#indices:admin/aliases/exists

#indices:admin/aliases/get

#indices:admin/analyze

#indices:admin/cache/clear

#indices:admin/close

#indices:admin/create

#indices:admin/delete

#indices:admin/exists

#indices:admin/flush

#indices:admin/get

#indices:admin/mapping/delete

#indices:admin/mapping/put

#indices:admin/mappings/fields/get

#indices:admin/mappings/get

#indices:admin/open

#indices:admin/optimize

#indices:admin/refresh

#indices:admin/settings/update

#indices:admin/shards/search_shards

#indices:admin/template/delete

#indices:admin/template/get

#indices:admin/template/put

#indices:admin/types/exists

#indices:admin/validate/query

#indices:admin/warmers/delete

#indices:admin/warmers/get

#indices:admin/warmers/put

#indices:data/benchmark/abort

#indices:data/benchmark/start

#indices:data/benchmark/status

#indices:data/read/count

#indices:data/read/exists

#indices:data/read/explain

#indices:data/read/get

#indices:data/read/mget

#indices:data/read/mlt

#indices:data/read/mpercolate

#indices:data/read/msearch

#indices:data/read/mtv

#indices:data/read/percolate

#indices:data/read/script/get

#indices:data/read/scroll

#indices:data/read/scroll/clear

#indices:data/read/search

#indices:data/read/suggest

#indices:data/read/tv

#indices:data/write/bulk

#indices:data/write/delete

#indices:data/write/delete/by_query

#indices:data/write/index

#indices:data/write/script/delete

#indices:data/write/script/put

#indices:data/write/update

#indices:monitor/recovery

#indices:monitor/segments

#indices:monitor/settings/get

#indices:monitor/stats

#############################################################################################

Below there is list of all restactions in elasticsearch 1.4 (for reference) .

Do not uncomment them here, they are configured above: searchguard.restactionfilter

#############################################################################################

#RestMainAction

#RestNodesInfoAction

#RestNodesStatsAction

#RestNodesHotThreadsAction

#RestNodesShutdownAction

#RestNodesRestartAction

#RestClusterStatsAction

#RestClusterStateAction

#RestClusterHealthAction

#RestClusterUpdateSettingsAction

#RestClusterGetSettingsAction

#RestClusterRerouteAction

#RestClusterSearchShardsAction

#RestPendingClusterTasksAction

#RestPutRepositoryAction

#RestGetRepositoriesAction

#RestDeleteRepositoryAction

#RestVerifyRepositoryAction

#RestGetSnapshotsAction

#RestCreateSnapshotAction

#RestRestoreSnapshotAction

#RestDeleteSnapshotAction

#RestSnapshotsStatusAction

#RestIndicesExistsAction

#RestTypesExistsAction

#RestGetIndicesAction

#RestIndicesStatsAction

#RestIndicesStatusAction

#RestIndicesSegmentsAction

#RestGetAliasesAction

#RestAliasesExistAction

#RestIndexDeleteAliasesAction

#RestIndexPutAliasAction

#RestIndicesAliasesAction

#RestGetIndicesAliasesAction

#RestCreateIndexAction

#RestDeleteIndexAction

#RestCloseIndexAction

#RestOpenIndexAction

#RestUpdateSettingsAction

#RestGetSettingsAction

#RestAnalyzeAction

#RestGetIndexTemplateAction

#RestPutIndexTemplateAction

#RestDeleteIndexTemplateAction

#RestHeadIndexTemplateAction

#RestPutWarmerAction

#RestDeleteWarmerAction

#RestGetWarmerAction

#RestPutMappingAction

#RestDeleteMappingAction

#RestGetMappingAction

#RestGetFieldMappingAction

#RestRefreshAction

#RestFlushAction

#RestOptimizeAction

#RestUpgradeAction

#RestClearIndicesCacheAction

#RestIndexAction

#RestGetAction

#RestGetSourceAction

#RestHeadAction

#RestMultiGetAction

#RestDeleteAction

#RestDeleteByQueryAction

#org.elasticsearch.rest.action.count.RestCountAction

#RestSuggestAction

#RestTermVectorAction

#RestMultiTermVectorsAction

#RestBulkAction

#RestUpdateAction

#RestPercolateAction

#RestMultiPercolateAction

#RestSearchAction

#RestSearchScrollAction

#RestClearScrollAction

#RestMultiSearchAction

#RestValidateQueryAction

#RestMoreLikeThisAction

#RestExplainAction

#RestRecoveryAction

Templates API

#RestGetSearchTemplateAction

#RestPutSearchTemplateAction

#RestDeleteSearchTemplateAction

Scripts API

#RestGetIndexedScriptAction

#RestPutIndexedScriptAction

#RestDeleteIndexedScriptAction

Cat API

#RestAllocationAction

#RestShardsAction

#RestMasterAction

#RestNodesAction

#RestIndicesAction

#RestSegmentsAction

Fully qualified to prevent interference with rest.action.count.RestCountAction

#org.elasticsearch.rest.action.cat.RestCountAction

Fully qualified to prevent interference with rest.action.indices.RestRecoveryAction

#org.elasticsearch.rest.action.cat.RestRecoveryAction

#RestHealthAction

#org.elasticsearch.rest.action.cat.RestPendingClusterTasksAction

#RestAliasAction

#RestThreadPoolAction

#RestPluginsAction

#RestFielddataAction

#RestCatAction

I see the following error in the log

[2015-08-03 11:12:43,137][INFO ][transport ] [pm-cdl-win-2] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/169.254.39.184:9300]}

[2015-08-03 11:12:43,158][INFO ][discovery ] [pm-cdl-win-2] sample/hqX-VUDvQLmQUSjEGobNEw

[2015-08-03 11:12:47,769][ERROR][com.floragunn.searchguard.service.SearchGuardConfigService] [pm-cdl-win-2] Try to refresh security configuration but it failed due to org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];

org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];

at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:151)

at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.checkGlobalBlock(TransportShardSingleOperationAction.java:103)

at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.<init>(TransportShardSingleOperationAction.java:132)

at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.<init>(TransportShardSingleOperationAction.java:116)

at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:89)

at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:55)

at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)

at com.floragunn.searchguard.filter.SearchGuardActionFilter.apply0(SearchGuardActionFilter.java:141)

at com.floragunn.searchguard.filter.SearchGuardActionFilter.apply(SearchGuardActionFilter.java:89)

at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)

at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)

at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)

at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)

at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)

at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:105)

at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)

at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82)

at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:98)

at org.elasticsearch.client.support.AbstractClient.get(AbstractClient.java:193)

at org.elasticsearch.action.get.GetRequestBuilder.doExecute(GetRequestBuilder.java:201)

at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91)

at com.floragunn.searchguard.service.SearchGuardConfigService.reloadConfig(SearchGuardConfigService.java:81)

at com.floragunn.searchguard.service.SearchGuardConfigService.access$0(SearchGuardConfigService.java:80)

at com.floragunn.searchguard.service.SearchGuardConfigService$Reload.run(SearchGuardConfigService.java:111)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)
···

On Monday, August 3, 2015 at 9:55:07 AM UTC-7, mjs...@gmail.com wrote:

I have three instances of ES. I have followed following steps to form cluster configuration as follows

  1. I have used same cluster name
  1. I have used different node names
  1. I have marked all nodes and master and data nodes
  1. I have disabled multicaset
  1. Provided unicast ip’s
  1. Disabled searchguard
  1. When I restarted ES and verified the o/p of http://localhost:9201/_cluster/health?pretty=true. I have noticed that ES has formed cluster. It’s all good upto this point

Now,

  1. I have stopped ES.
  1. I have enabled search guard
  1. I have copied searchguard key from one node to other nodes
  1. I have imported SSL certificates of one node to others truststore
  1. restarted ES
  1. The ES didn’t form the cluster. The ES is started with 503 error. If I access any ES urls’s it is giving error as {“error”:“MasterNotDiscoveredException[waited for [30s]]”,“status”:503}

fyi; The above configuration used to work however it is not working now all of sudden.

I have the following searchguard rules stored in ES

{

“acl”: [

{

Comment”: “Default is to execute all filters”,

“filters_bypass”: [

“*”

],

“filters_execute”: [

“*”

]

},

{

Comment”: “Any authenticated user do anything on the ‘_all’, ‘searchguard’, ‘inventory’ index - no filter will be executed”,

“indices”: [

“_all”,

“searchguard”,

“inventory”

],

“filters_bypass”: [

“*”

],

“filters_execute”: [

“*”

]

},

{

Comment”: “For admin role all filters are bypassed (so none will be executed) for all indices. This means unrestricted access at all for this role.”,

“roles”: [

“admin”

],

“filters_bypass”: [

“*”

],

“filters_execute”:

}

]

}

Elasticsearch.yml file

##################### Elasticsearch Configuration Example #####################

This file contains an overview of various configuration settings,

targeted at operations staff. Application developers should

consult the guide at <http://elasticsearch.org/guide>.

The installation procedure is covered at

<http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html>.

Elasticsearch comes with reasonable defaults for most settings,

so you can try it out without bothering with configuration.

Most of the time, these defaults are just fine for running a production

cluster. If you’re fine-tuning your cluster, or wondering about the

effect of certain configuration option, please do ask on the

mailing list or IRC channel [http://elasticsearch.org/community].

Any element in the configuration can be replaced with environment variables

by placing them in ${…} notation. For example:

#node.rack: ${RACK_ENV_VAR}

For information on supported formats and syntax for the config file, see

<http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html>

################################### Cluster ###################################

Cluster name identifies your cluster for auto-discovery. If you’re running

multiple clusters on the same network, make sure you’re using unique names.

#cluster.name: “elasticsearch”

cluster.name: “sample”

#################################### Node #####################################

Node names are generated dynamically on startup, so you’re relieved

from configuring them manually. You can tie this node to a specific name:

node.name: “pm-cdl-win-2”

Every node can be configured to allow or deny being eligible as the master,

and to allow or deny to store the data.

Allow this node to be eligible as a master node (enabled by default):

node.master: true

Allow this node to store data (enabled by default):

node.data: true

You can exploit these settings to design advanced cluster topologies.

1. You want this node to never become a master node, only to hold data.

This will be the “workhorse” of your cluster.

#node.master: true

#node.data: true

2. You want this node to only serve as a master: to not store any data and

to have free resources. This will be the “coordinator” of your cluster.

#node.master: true

#node.data: false

3. You want this node to be neither master nor data node, but

to act as a “search load balancer” (fetching data from nodes,

aggregating results, etc.)

#node.master: false

#node.data: false

Use the Cluster Health API [http://localhost:9200/_cluster/health], the

Node Info API [http://localhost:9200/_nodes] or GUI tools

such as <http://www.elasticsearch.org/overview/marvel/>,

<http://github.com/karmi/elasticsearch-paramedic>,

<http://github.com/lukas-vlcek/bigdesk> and

<http://mobz.github.com/elasticsearch-head> to inspect the cluster state.

A node can have generic attributes associated with it, which can later be used

for customized shard allocation filtering, or allocation awareness. An attribute

is a simple key value pair, similar to node.key: value, here is an example:

#node.rack: rack314

By default, multiple nodes are allowed to start from the same installation location

to disable it, set the following:

#node.max_local_storage_nodes: 1

node.local: false

#################################### Index ####################################

You can set a number of options (such as shard/replica options, mapping

or analyzer definitions, translog settings, …) for indices globally,

in this file.

Note, that it makes more sense to configure index settings specifically for

a certain index, either when creating it or by using the index templates API.

See <http://elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules.html> and

<http://elasticsearch.org/guide/en/elasticsearch/reference/current/indices-create-index.html>

for more information.

Set the number of shards (splits) of an index (5 by default):

#index.number_of_shards: 5

Set the number of replicas (additional copies) of an index (1 by default):

#index.number_of_replicas: 1

Note, that for development on a local machine, with small indices, it usually

makes sense to “disable” the distributed features:

#index.number_of_shards: 1

#index.number_of_replicas: 0

These settings directly affect the performance of index and search operations

in your cluster. Assuming you have enough machines to hold shards and

replicas, the rule of thumb is:

1. Having more shards enhances the indexing performance and allows to

distribute a big index across machines.

2. Having more replicas enhances the search performance and improves the

cluster availability.

The “number_of_shards” is a one-time setting for an index.

The “number_of_replicas” can be increased or decreased anytime,

by using the Index Update Settings API.

Elasticsearch takes care about load balancing, relocating, gathering the

results from nodes, etc. Experiment with different settings to fine-tune

your setup.

Use the Index Status API (<http://localhost:9200/A/_status>) to inspect

the index status.

#################################### Paths ####################################

Path to directory containing configuration (this file and logging.yml):

#path.conf: /path/to/conf

Path to directory where to store index data allocated for this node.

#path.data: /path/to/data

Can optionally include more than one location, causing data to be striped across

the locations (a la RAID 0) on a file level, favouring locations with most free

space on creation. For example:

#path.data: /path/to/data1,/path/to/data2

Path to temporary files:

#path.work: /path/to/work

Path to log files:

#path.logs: /path/to/logs

Path to where plugins are installed:

#path.plugins: /path/to/plugins

#################################### Plugin ###################################

If a plugin listed here is not installed for current node, the node will not start.

#plugin.mandatory: mapper-attachments,lang-groovy

################################### Memory ####################################

Elasticsearch performs poorly when JVM starts swapping: you should ensure that

it never swaps.

Set this property to true to lock the memory:

#bootstrap.mlockall: true

Make sure that the ES_MIN_MEM and ES_MAX_MEM environment variables are set

to the same value, and that the machine has enough memory to allocate

for Elasticsearch, leaving enough memory for the operating system itself.

You should also make sure that the Elasticsearch process is allowed to lock

the memory, eg. by using ulimit -l unlimited.

############################## Network And HTTP ###############################

Elasticsearch, by default, binds itself to the 0.0.0.0 address, and listens

on port [9200-9300] for HTTP traffic and on port [9300-9400] for node-to-node

communication. (the range means that if the port is busy, it will automatically

try the next port).

Set the bind address specifically (IPv4 or IPv6):

#network.bind_host: localhost

Set the address other nodes will use to communicate with this node. If not

set, it is automatically derived. It must point to an actual IP address.

#network.publish_host: 192.168.0.1

Set both ‘bind_host’ and ‘publish_host’:

#network.host: 192.168.0.1

Set a custom port for the node to node communication (9300 by default):

#transport.tcp.port: 9300

transport.tcp.port: 9300

Enable compression for all communication between nodes (disabled by default):

#transport.tcp.compress: true

Set a custom port to listen for HTTP traffic:

http.port: 9201

Set a custom allowed content length:

#http.max_content_length: 100mb

Disable HTTP completely:

http.enabled: true

################################### Gateway ###################################

The gateway allows for persisting the cluster state between full cluster

restarts. Every change to the state (such as adding an index) will be stored

in the gateway, and when the cluster starts up for the first time,

it will read its state from the gateway.

There are several types of gateway implementations. For more information, see

<http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-gateway.html>.

The default gateway type is the “local” gateway (recommended):

#gateway.type: local

Settings below control how and when to start the initial recovery process on

a full cluster restart (to reuse as much local data as possible when using shared

gateway).

Allow recovery process after N nodes in a cluster are up:

#gateway.recover_after_nodes: 1

Set the timeout to initiate the recovery process, once the N nodes

from previous setting are up (accepts time value):

#gateway.recover_after_time: 5m

Set how many nodes are expected in this cluster. Once these N nodes

are up (and recover_after_nodes is met), begin recovery process immediately

(without waiting for recover_after_time to expire):

#gateway.expected_nodes: 2

############################# Recovery Throttling #############################

These settings allow to control the process of shards allocation between

nodes during initial recovery, replica allocation, rebalancing,

or when adding and removing nodes.

Set the number of concurrent recoveries happening on a node:

1. During the initial recovery

#cluster.routing.allocation.node_initial_primaries_recoveries: 4

2. During adding/removing nodes, rebalancing, etc

#cluster.routing.allocation.node_concurrent_recoveries: 2

Set to throttle throughput when recovering (eg. 100mb, by default 20mb):

#indices.recovery.max_bytes_per_sec: 20mb

Set to limit the number of open concurrent streams when

recovering a shard from a peer:

#indices.recovery.concurrent_streams: 5

################################## Discovery ##################################

Discovery infrastructure ensures nodes can be found within a cluster

and master node is elected. Multicast discovery is the default.

Set to ensure a node sees N other master eligible nodes to be considered

operational within the cluster. This should be set to a quorum/majority of

the master-eligible nodes in the cluster.

#discovery.zen.minimum_master_nodes: 1

Set the time to wait for ping responses from other nodes when discovering.

Set this option to a higher value on a slow or congested network

to minimize discovery failures:

discovery.zen.ping.timeout: 5s

For more information, see

<http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-zen.html>

Unicast discovery allows to explicitly control which nodes will be used

to discover the cluster. It can be used when multicast is not present,

or to restrict the cluster communication-wise.

1. Disable multicast discovery (enabled by default):

discovery.zen.ping.multicast.enabled: false

2. Configure an initial list of master nodes in the cluster

to perform discovery when new nodes (master or data) are started:

#discovery.zen.ping.unicast.hosts: [“host”: “host:port”]

discovery.zen.ping.unicast.hosts: [“pm-cdl-win-1.cdl.local”, “pm-cdl-win-2.cdl.local”, “pm-cdl-win-3.cdl.local”]

#discovery.zen.ping.unicast.hosts: [“10.1.16.11”, “10.1.16.17”, “10.1.16.19”]

#discovery.zen.no_master_block: all

discovery.zen.minimum_master_nodes: 1

EC2 discovery allows to use AWS EC2 API in order to perform discovery.

You have to install the cloud-aws plugin for enabling the EC2 discovery.

For more information, see

<http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-ec2.html>

See <http://elasticsearch.org/tutorials/elasticsearch-on-ec2/>

for a step-by-step tutorial.

GCE discovery allows to use Google Compute Engine API in order to perform discovery.

You have to install the cloud-gce plugin for enabling the GCE discovery.

For more information, see <https://github.com/elasticsearch/elasticsearch-cloud-gce>.

Azure discovery allows to use Azure API in order to perform discovery.

You have to install the cloud-azure plugin for enabling the Azure discovery.

For more information, see <https://github.com/elasticsearch/elasticsearch-cloud-azure>.

################################## Slow Log ##################################

Shard level query and fetch threshold logging.

#index.search.slowlog.threshold.query.warn: 10s

#index.search.slowlog.threshold.query.info: 5s

#index.search.slowlog.threshold.query.debug: 2s

#index.search.slowlog.threshold.query.trace: 500ms

#index.search.slowlog.threshold.fetch.warn: 1s

#index.search.slowlog.threshold.fetch.info: 800ms

#index.search.slowlog.threshold.fetch.debug: 500ms

#index.search.slowlog.threshold.fetch.trace: 200ms

#index.indexing.slowlog.threshold.index.warn: 10s

#index.indexing.slowlog.threshold.index.info: 5s

#index.indexing.slowlog.threshold.index.debug: 2s

#index.indexing.slowlog.threshold.index.trace: 500ms

################################## GC Logging ################################

#monitor.jvm.gc.young.warn: 1000ms

#monitor.jvm.gc.young.info: 700ms

#monitor.jvm.gc.young.debug: 400ms

#monitor.jvm.gc.old.warn: 10s

#monitor.jvm.gc.old.info: 5s

#monitor.jvm.gc.old.debug: 2s

################################## Security ################################

Uncomment if you want to enable JSONP as a valid return transport on the

http server. With this enabled, it may pose a security risk, so disabling

it unless you need it is recommended (it is disabled by default).

#http.jsonp.enable: true

#############################################################################################

SEARCH GUARD

Configuration

#############################################################################################

#Note: All waffle related options are only valid if your ES node is running on windows OS

Enable or disable the complete Searchguard plugin functionality

searchguard.enabled: true

Path where to write/read the searchguard master key file

#searchguard.key_path: .

When using DLS or FLS and a get or mget is performed then rewrite it as search request

#searchguard.rewrite_get_as_search: true

The index name where Searchguard will store its configuration and various other informations related to Searchguard itself

This index can only be access from localhost

searchguard.config_index_name: searchguard

Enable or disable HTTP session which caches the authentication and authorization informations in a cookie

#searchguard.http.enable_sessions: false

Enable or disable audit logging

#searchguard.auditlog.enabled: true

If this is true (default is false) then Searchguard will check if elasticsearch is running as root/windows admin and if so then abort.

searchguard.check_for_root: false

If this is true (default is false) then allow all HTTP REST requests from nodes loopback (e.g. localhost)

searchguard.allow_all_from_loopback: true

If this is true (default: false) then enable authenticated transports requests (e.g. TransportClient authentication)

This can be done in that way (for example):

TransportClient.get(new GetRequest(“marketing”, “customer”, “tp_3”).putHeader(“searchguard_transport_creds”, “c2FseWg6c2VjcmV0”))

Add a header “searchguard_transport_creds”

base64(username":"password) is the credentials string

base64(spock:secret) → c3BvY2s6c2VjcmV0

searchguard.transport_auth.enabled: false

#############################################################################################

Transport layer SSL

#############################################################################################

Enable or disable node-to-node ssl encryption

searchguard.ssl.transport.node.enabled: true

JKS or PKCS12

searchguard.ssl.transport.node.keystore_type: JKS

Absolute path to the keystore file (this stores the server certificates)

searchguard.ssl.transport.node.keystore_filepath: C:\Program Files\Hewlett-Packard\CSA/jboss-as/standalone/configuration/.keystore

Keystore password

searchguard.ssl.transport.node.keystore_password: changeit

Do other nodes have to authenticate themself to the cluster, default is true

searchguard.ssl.transport.node.enforce_clientauth: false

JKS or PKCS12

searchguard.ssl.transport.node.truststore_type: JKS

Absolute path to the truststore file (this stores the client certificates)

searchguard.ssl.transport.node.truststore_filepath: C:\Program Files\Hewlett-Packard\CSA/openjre/lib/security/cacerts

Truststore password

searchguard.ssl.transport.node.truststore_password: changeit

#############################################################################################

REST layer SSL

#############################################################################################

Enable or disable rest layer security (https)

searchguard.ssl.transport.http.enabled: true

JKS or PKCS12

searchguard.ssl.transport.http.keystore_type: JKS

Absolute path to the keystore file (this stores the server certificates)

searchguard.ssl.transport.http.keystore_filepath: C:\Program Files\Hewlett-Packard\CSA/jboss-as/standalone/configuration/.keystore

Keystore password

searchguard.ssl.transport.http.keystore_password: changeit

Do the clients (typically the browser or the proxy) have to authenticate themself to the http server, default is false

searchguard.ssl.transport.http.enforce_clientauth: false

JKS or PKCS12

searchguard.ssl.transport.http.truststore_type: JKS

Absolute path to the truststore file (this stores the client certificates)

searchguard.ssl.transport.http.truststore_filepath: C:\Program Files\Hewlett-Packard\CSA/openjre/lib/security/cacerts

Truststore password

searchguard.ssl.transport.http.truststore_password: changeit

#############################################################################################

X-Forwarded-For (XFF) header

#############################################################################################

X-Forwarded-For (XFF) header

If you have a http proxy in front of elasticsearch you have to configure this options to handle XFF properly

#searchguard.http.xforwardedfor.header: X-Forwarded-For

#searchguard.http.xforwardedfor.trustedproxies: null

#searchguard.http.xforwardedfor.enforce: false

#############################################################################################

Authentication backend

#############################################################################################

Validates the username and credentials

searchguard.authentication.authentication_backend.impl: com.floragunn.searchguard.authentication.backend.simple.SettingsBasedAuthenticationBackend

#searchguard.authentication.authentication_backend.impl: com.floragunn.searchguard.authentication.backend.ldap.LDAPAuthenticationBackend

#searchguard.authentication.authentication_backend.impl: com.floragunn.searchguard.authentication.backend.simple.AlwaysSucceedAuthenticationBackend

#searchguard.authentication.authentication_backend.impl: com.floragunn.searchguard.authentication.backend.waffle.WaffleAuthenticationBackend

If caching is enabled then the authentication succeed for 24 h since the first successful login without hitting the backend again and again

#searchguard.authentication.authentication_backend.cache.enable: true

#############################################################################################

Authorization backend (authorizer)

#############################################################################################

searchguard.authentication.authorizer.impl: com.floragunn.searchguard.authorization.simple.SettingsBasedAuthorizator

#searchguard.authentication.authorizer.impl: com.floragunn.searchguard.authorization.ldap.LDAPAuthorizator

#searchguard.authentication.authorizer.impl: com.floragunn.searchguard.authorization.waffle.WaffleAuthorizator

If caching is enabled then the role informations will be cached for 24 h without hitting the backend again and again

#searchguard.authentication.authorizer.cache.enable: true

#############################################################################################

HTTP authentication method

#############################################################################################

Define HTTP authentication method. In future we will here have more like NTLM, SPNEGO/Kerberos and Digest.

searchguard.authentication.http_authenticator.impl: com.floragunn.searchguard.authentication.http.basic.HTTPBasicAuthenticator

HTTPProxyAuthenticator assume there is kind of proxy in front of elasticsearch which handles the authentication and stores the

username of the authenticated user in a http header

#searchguard.authentication.http_authenticator.impl: com.floragunn.searchguard.authentication.http.proxy.HTTPProxyAuthenticator

SSL mutual authentication (works only if searchguard.ssl.transport.http.enabled is ‘true’ with client auth enabled)

#searchguard.authentication.http_authenticator.impl: com.floragunn.searchguard.authentication.http.clientcert.HTTPSClientCertAuthenticator

SPNEGO

#searchguard.authentication.http_authenticator.impl: com.floragunn.searchguard.authentication.http.spnego.HTTPSpnegoAuthenticator

Absolute file path to jaas login config file

#searchguard.authentication.spnego.login_config_filepath: null

Absolute file path to krb5 config file

#searchguard.authentication.spnego.krb5_config_filepath: null

Name of the login entry in jaas login config file which represents the acceptor (server)

#searchguard.authentication.spnego.login_config_name: com.sun.security.jgss.krb5.accept

Strip the realmname from username (hnelson@EXAMPLE.COM → hnelson)

#searchguard.authentication.spnego.strip_realm: true

Authenticates always a user with username ‘searchguard_unauthenticated_user’

#searchguard.authentication.http_authenticator.impl: com.floragunn.searchguard.authentication.http.HTTPUnauthenticatedAuthenticator

Waffle (Windows only, must be used with WaffleAuthorizator)

#searchguard.authentication.http_authenticator.impl: com.floragunn.searchguard.authentication.http.waffle.HTTPWaffleAuthenticator

Strip domain name from user (COMPANY\spock → spock)

#searchguard.authentication.waffle.strip_domain: true

#####################################################

Settings based authentication (define users and password directly here in the settings. Note: this is per node)

#searchguard.authentication.settingsdb.user.: password

searchguard.authentication.settingsdb.user.admin: cloud

#searchguard.authentication.settingsdb.user.user1: password

#searchguard.authentication.settingsdb.user.michaeljackson: neverland

If plain text password should be hashed use this. Supported digests are: SHA1 SHA256 SHA384 SHA512 MD5

#searchguard.authentication.settingsdb.digest: SHA1

#searchguard.authentication.settingsdb.user.admin: 000e793db70c59309fa6f0f36d0046d110f3be3c

#searchguard.authentication.settingsdb.user.michaeljackson: 824d55e7a62b7ca8751dff346ffab845a8f26d08

#####################################################

#####################################################

Settings based authorization (define users and their roles directly here in the settings. Note: this is per node)

#searchguard.authentication.authorization.settingsdb.roles.:

searchguard.authentication.authorization.settingsdb.roles.admin: [“admin”]

#searchguard.authentication.authorization.settingsdb.roles.user1: [“admin”]

#searchguard.authentication.authorization.settingsdb.roles.michaeljackson: [“admin”]

#####################################################

#####################################################

LDAP authentication backend (authenticate users against a LDAP or Active Directory)

The defaults are sufficient for Active Directory

#searchguard.authentication.ldap.host: [“localhost:389”]

#searchguard.authentication.ldap.ldaps.ssl.enabled: false

#searchguard.authentication.ldap.ldaps.starttls.enabled: false

JKS or PKCS12

#searchguard.authentication.ldap.ldaps.truststore_type: JKS

#searchguard.authentication.ldap.ldaps.truststore_filepath: null

#searchguard.authentication.ldap.ldaps.truststore_password: null

#searchguard.authentication.ldap.bind_dn: null

#searchguard.authentication.ldap.password: null

Default is root dse (“”)

#searchguard.authentication.ldap.userbase: “”

Filter to search for users (currently in the whole subtree beneath userbase)

{0} is substituted with the username

#searchguard.authentication.ldap.usersearch: (sAMAccountName={0})

Use this attribute from the user as username (if not set then DN is used)

#searchguard.authentication.ldap.username_attribute: null

#####################################################

#####################################################

LDAP authorization backend (gather roles from a LDAP or Active Directory, you have to configure the above LDAP authentication backend settings too)

The defaults are sufficient for Active Directory

Default is root dse (“”)

#searchguard.authentication.authorization.ldap.rolebase: “”

Filter to search for roles (currently in the whole subtree beneath rolebase)

{0} is substituted with the DN of the user

{1} is substituted with the username

{2} is substituted with an attribute value from user’s directory entry, of the authenticated user. Use userroleattribute to specify the name of the attribute

#searchguard.authentication.authorization.ldap.rolesearch: (member={0})

Specify the name of the attribute which value should be substituted with {2} above

#searchguard.authentication.authorization.ldap.userroleattribute: null

Roles as an attribute of the user entry

#searchguard.authentication.authorization.ldap.userrolename: memberOf

The attribute in a role entry containing the name of that role

#searchguard.authentication.authorization.ldap.rolename: name

Resolve nested roles transitive (roles which are members of other roles and so on …)

#searchguard.authentication.authorization.ldap.resolve_nested_roles: false

#####################################################

#####################################################

HTTP proxy authenticator configuration

Header name which contains the username

#searchguard.authentication.proxy.header:X-Authenticated-User

Array of trusted IP addresses (this are typically your proxy server(s))

#searchguard.authentication.proxy.trusted_ips: null

#####################################################

#####################################################

HTTP SSL mutual authentication configuration

Attribute of that attribute in the certificate dn which holds the username

#searchguard.authentication.https.clientcert.attributename: cn

#####################################################

##############################################################################################

Below here you configure what authenticated and authorized users are allowed to do (or not)#

This maps to the acl defined in the searchguard configuration index

#############################################################################################

Configure the restactionfilter to allow or forbid action

#searchguard.restactionfilter.names: [“readonly”]

#searchguard.restactionfilter.readonly.allowed_actions: [“*SearchAction”, “RestSearchScrollAction”, “RestClearScrollAction”, “RestGetAction”, “RestGetSourceAction”, “*MainAction”, “RestValidateQueryAction”, “RestMoreLikeThisAction”, “RestPercolateAction”]

#searchguard.restactionfilter.readonly.forbidden_actions: […]

Configure the actionrequestfilter to allow or forbid action

searchguard.actionrequestfilter.names: [“readonly”]

searchguard.actionrequestfilter.readonly.allowed_actions: [“*”]

#searchguard.actionrequestfilter.readonly.forbidden_actions: [“cluster:", "indices:admin”]

Configure document level security (dls) filter

Warning: All this (with the exception of “exists”) only works with not_analyzed fields because a term filter is used internally

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-term-filter.html

#searchguard.dlsfilter.names: [“a”, “b”, “c”, “d”, “e”, “f”, “g”]

#searchguard.dlsfilter.a: [“exists”,“field”, “false”] # if field exists (or not) match → false means field must exist

#searchguard.dlsfilter.b: [“term”, “field”,“value”, “false”] # if field==value (or not) match

#search

It’s network issue in my lab. The cluster is working fine. No issues and thus closing the thread.

···

On Monday, August 3, 2015 at 9:55:07 AM UTC-7, mjs...@gmail.com wrote:

I have three instances of ES. I have followed following steps to form cluster configuration as follows

  1. I have used same cluster name
  1. I have used different node names
  1. I have marked all nodes and master and data nodes
  1. I have disabled multicaset
  1. Provided unicast ip’s
  1. Disabled searchguard
  1. When I restarted ES and verified the o/p of http://localhost:9201/_cluster/health?pretty=true. I have noticed that ES has formed cluster. It’s all good upto this point

Now,

  1. I have stopped ES.
  1. I have enabled search guard
  1. I have copied searchguard key from one node to other nodes
  1. I have imported SSL certificates of one node to others truststore
  1. restarted ES
  1. The ES didn’t form the cluster. The ES is started with 503 error. If I access any ES urls’s it is giving error as {“error”:“MasterNotDiscoveredException[waited for [30s]]”,“status”:503}

fyi; The above configuration used to work however it is not working now all of sudden.

I have the following searchguard rules stored in ES

{

“acl”: [

{

Comment”: “Default is to execute all filters”,

“filters_bypass”: [

“*”

],

“filters_execute”: [

“*”

]

},

{

Comment”: “Any authenticated user do anything on the ‘_all’, ‘searchguard’, ‘inventory’ index - no filter will be executed”,

“indices”: [

“_all”,

“searchguard”,

“inventory”

],

“filters_bypass”: [

“*”

],

“filters_execute”: [

“*”

]

},

{

Comment”: “For admin role all filters are bypassed (so none will be executed) for all indices. This means unrestricted access at all for this role.”,

“roles”: [

“admin”

],

“filters_bypass”: [

“*”

],

“filters_execute”:

}

]

}

Elasticsearch.yml file

##################### Elasticsearch Configuration Example #####################

This file contains an overview of various configuration settings,

targeted at operations staff. Application developers should

consult the guide at <http://elasticsearch.org/guide>.

The installation procedure is covered at

<http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html>.

Elasticsearch comes with reasonable defaults for most settings,

so you can try it out without bothering with configuration.

Most of the time, these defaults are just fine for running a production

cluster. If you’re fine-tuning your cluster, or wondering about the

effect of certain configuration option, please do ask on the

mailing list or IRC channel [http://elasticsearch.org/community].

Any element in the configuration can be replaced with environment variables

by placing them in ${…} notation. For example:

#node.rack: ${RACK_ENV_VAR}

For information on supported formats and syntax for the config file, see

<http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html>

################################### Cluster ###################################

Cluster name identifies your cluster for auto-discovery. If you’re running

multiple clusters on the same network, make sure you’re using unique names.

#cluster.name: “elasticsearch”

cluster.name: “sample”

#################################### Node #####################################

Node names are generated dynamically on startup, so you’re relieved

from configuring them manually. You can tie this node to a specific name:

node.name: “pm-cdl-win-2”

Every node can be configured to allow or deny being eligible as the master,

and to allow or deny to store the data.

Allow this node to be eligible as a master node (enabled by default):

node.master: true

Allow this node to store data (enabled by default):

node.data: true

You can exploit these settings to design advanced cluster topologies.

1. You want this node to never become a master node, only to hold data.

This will be the “workhorse” of your cluster.

#node.master: true

#node.data: true

2. You want this node to only serve as a master: to not store any data and

to have free resources. This will be the “coordinator” of your cluster.

#node.master: true

#node.data: false

3. You want this node to be neither master nor data node, but

to act as a “search load balancer” (fetching data from nodes,

aggregating results, etc.)

#node.master: false

#node.data: false

Use the Cluster Health API [http://localhost:9200/_cluster/health], the

Node Info API [http://localhost:9200/_nodes] or GUI tools

such as <http://www.elasticsearch.org/overview/marvel/>,

<http://github.com/karmi/elasticsearch-paramedic>,

<http://github.com/lukas-vlcek/bigdesk> and

<http://mobz.github.com/elasticsearch-head> to inspect the cluster state.

A node can have generic attributes associated with it, which can later be used

for customized shard allocation filtering, or allocation awareness. An attribute

is a simple key value pair, similar to node.key: value, here is an example:

#node.rack: rack314

By default, multiple nodes are allowed to start from the same installation location

to disable it, set the following:

#node.max_local_storage_nodes: 1

node.local: false

#################################### Index ####################################

You can set a number of options (such as shard/replica options, mapping

or analyzer definitions, translog settings, …) for indices globally,

in this file.

Note, that it makes more sense to configure index settings specifically for

a certain index, either when creating it or by using the index templates API.

See <http://elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules.html> and

<http://elasticsearch.org/guide/en/elasticsearch/reference/current/indices-create-index.html>

for more information.

Set the number of shards (splits) of an index (5 by default):

#index.number_of_shards: 5

Set the number of replicas (additional copies) of an index (1 by default):

#index.number_of_replicas: 1

Note, that for development on a local machine, with small indices, it usually

makes sense to “disable” the distributed features:

#index.number_of_shards: 1

#index.number_of_replicas: 0

These settings directly affect the performance of index and search operations

in your cluster. Assuming you have enough machines to hold shards and

replicas, the rule of thumb is:

1. Having more shards enhances the indexing performance and allows to

distribute a big index across machines.

2. Having more replicas enhances the search performance and improves the

cluster availability.

The “number_of_shards” is a one-time setting for an index.

The “number_of_replicas” can be increased or decreased anytime,

by using the Index Update Settings API.

Elasticsearch takes care about load balancing, relocating, gathering the

results from nodes, etc. Experiment with different settings to fine-tune

your setup.

Use the Index Status API (<http://localhost:9200/A/_status>) to inspect

the index status.

#################################### Paths ####################################

Path to directory containing configuration (this file and logging.yml):

#path.conf: /path/to/conf

Path to directory where to store index data allocated for this node.

#path.data: /path/to/data

Can optionally include more than one location, causing data to be striped across

the locations (a la RAID 0) on a file level, favouring locations with most free

space on creation. For example:

#path.data: /path/to/data1,/path/to/data2

Path to temporary files:

#path.work: /path/to/work

Path to log files:

#path.logs: /path/to/logs

Path to where plugins are installed:

#path.plugins: /path/to/plugins

#################################### Plugin ###################################

If a plugin listed here is not installed for current node, the node will not start.

#plugin.mandatory: mapper-attachments,lang-groovy

################################### Memory ####################################

Elasticsearch performs poorly when JVM starts swapping: you should ensure that

it never swaps.

Set this property to true to lock the memory:

#bootstrap.mlockall: true

Make sure that the ES_MIN_MEM and ES_MAX_MEM environment variables are set

to the same value, and that the machine has enough memory to allocate

for Elasticsearch, leaving enough memory for the operating system itself.

You should also make sure that the Elasticsearch process is allowed to lock

the memory, eg. by using ulimit -l unlimited.

############################## Network And HTTP ###############################

Elasticsearch, by default, binds itself to the 0.0.0.0 address, and listens

on port [9200-9300] for HTTP traffic and on port [9300-9400] for node-to-node

communication. (the range means that if the port is busy, it will automatically

try the next port).

Set the bind address specifically (IPv4 or IPv6):

#network.bind_host: localhost

Set the address other nodes will use to communicate with this node. If not

set, it is automatically derived. It must point to an actual IP address.

#network.publish_host: 192.168.0.1

Set both ‘bind_host’ and ‘publish_host’:

#network.host: 192.168.0.1

Set a custom port for the node to node communication (9300 by default):

#transport.tcp.port: 9300

transport.tcp.port: 9300

Enable compression for all communication between nodes (disabled by default):

#transport.tcp.compress: true

Set a custom port to listen for HTTP traffic:

http.port: 9201

Set a custom allowed content length:

#http.max_content_length: 100mb

Disable HTTP completely:

http.enabled: true

################################### Gateway ###################################

The gateway allows for persisting the cluster state between full cluster

restarts. Every change to the state (such as adding an index) will be stored

in the gateway, and when the cluster starts up for the first time,

it will read its state from the gateway.

There are several types of gateway implementations. For more information, see

<http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-gateway.html>.

The default gateway type is the “local” gateway (recommended):

#gateway.type: local

Settings below control how and when to start the initial recovery process on

a full cluster restart (to reuse as much local data as possible when using shared

gateway).

Allow recovery process after N nodes in a cluster are up:

#gateway.recover_after_nodes: 1

Set the timeout to initiate the recovery process, once the N nodes

from previous setting are up (accepts time value):

#gateway.recover_after_time: 5m

Set how many nodes are expected in this cluster. Once these N nodes

are up (and recover_after_nodes is met), begin recovery process immediately

(without waiting for recover_after_time to expire):

#gateway.expected_nodes: 2

############################# Recovery Throttling #############################

These settings allow to control the process of shards allocation between

nodes during initial recovery, replica allocation, rebalancing,

or when adding and removing nodes.

Set the number of concurrent recoveries happening on a node:

1. During the initial recovery

#cluster.routing.allocation.node_initial_primaries_recoveries: 4

2. During adding/removing nodes, rebalancing, etc

#cluster.routing.allocation.node_concurrent_recoveries: 2

Set to throttle throughput when recovering (eg. 100mb, by default 20mb):

#indices.recovery.max_bytes_per_sec: 20mb

Set to limit the number of open concurrent streams when

recovering a shard from a peer:

#indices.recovery.concurrent_streams: 5

################################## Discovery ##################################

Discovery infrastructure ensures nodes can be found within a cluster

and master node is elected. Multicast discovery is the default.

Set to ensure a node sees N other master eligible nodes to be considered

operational within the cluster. This should be set to a quorum/majority of

the master-eligible nodes in the cluster.

#discovery.zen.minimum_master_nodes: 1

Set the time to wait for ping responses from other nodes when discovering.

Set this option to a higher value on a slow or congested network

to minimize discovery failures:

discovery.zen.ping.timeout: 5s

For more information, see

<http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-zen.html>

Unicast discovery allows to explicitly control which nodes will be used

to discover the cluster. It can be used when multicast is not present,

or to restrict the cluster communication-wise.

1. Disable multicast discovery (enabled by default):

discovery.zen.ping.multicast.enabled: false

2. Configure an initial list of master nodes in the cluster

to perform discovery when new nodes (master or data) are started:

#discovery.zen.ping.unicast.hosts: [“host”: “host:port”]

discovery.zen.ping.unicast.hosts: [“pm-cdl-win-1.cdl.local”, “pm-cdl-win-2.cdl.local”, “pm-cdl-win-3.cdl.local”]

#discovery.zen.ping.unicast.hosts: [“10.1.16.11”, “10.1.16.17”, “10.1.16.19”]

#discovery.zen.no_master_block: all

discovery.zen.minimum_master_nodes: 1

EC2 discovery allows to use AWS EC2 API in order to perform discovery.

You have to install the cloud-aws plugin for enabling the EC2 discovery.

For more information, see

<http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-ec2.html>

See <http://elasticsearch.org/tutorials/elasticsearch-on-ec2/>

for a step-by-step tutorial.

GCE discovery allows to use Google Compute Engine API in order to perform discovery.

You have to install the cloud-gce plugin for enabling the GCE discovery.

For more information, see <https://github.com/elasticsearch/elasticsearch-cloud-gce>.

Azure discovery allows to use Azure API in order to perform discovery.

You have to install the cloud-azure plugin for enabling the Azure discovery.

For more information, see <https://github.com/elasticsearch/elasticsearch-cloud-azure>.

################################## Slow Log ##################################

Shard level query and fetch threshold logging.

#index.search.slowlog.threshold.query.warn: 10s

#index.search.slowlog.threshold.query.info: 5s

#index.search.slowlog.threshold.query.debug: 2s

#index.search.slowlog.threshold.query.trace: 500ms

#index.search.slowlog.threshold.fetch.warn: 1s

#index.search.slowlog.threshold.fetch.info: 800ms

#index.search.slowlog.threshold.fetch.debug: 500ms

#index.search.slowlog.threshold.fetch.trace: 200ms

#index.indexing.slowlog.threshold.index.warn: 10s

#index.indexing.slowlog.threshold.index.info: 5s

#index.indexing.slowlog.threshold.index.debug: 2s

#index.indexing.slowlog.threshold.index.trace: 500ms

################################## GC Logging ################################

#monitor.jvm.gc.young.warn: 1000ms

#monitor.jvm.gc.young.info: 700ms

#monitor.jvm.gc.young.debug: 400ms

#monitor.jvm.gc.old.warn: 10s

#monitor.jvm.gc.old.info: 5s

#monitor.jvm.gc.old.debug: 2s

################################## Security ################################

Uncomment if you want to enable JSONP as a valid return transport on the

http server. With this enabled, it may pose a security risk, so disabling

it unless you need it is recommended (it is disabled by default).

#http.jsonp.enable: true

#############################################################################################

SEARCH GUARD

Configuration

#############################################################################################

#Note: All waffle related options are only valid if your ES node is running on windows OS

Enable or disable the complete Searchguard plugin functionality

searchguard.enabled: true

Path where to write/read the searchguard master key file

#searchguard.key_path: .

When using DLS or FLS and a get or mget is performed then rewrite it as search request

#searchguard.rewrite_get_as_search: true

The index name where Searchguard will store its configuration and various other informations related to Searchguard itself

This index can only be access from localhost

searchguard.config_index_name: searchguard

Enable or disable HTTP session which caches the authentication and authorization informations in a cookie

#searchguard.http.enable_sessions: false

Enable or disable audit logging

#searchguard.auditlog.enabled: true

If this is true (default is false) then Searchguard will check if elasticsearch is running as root/windows admin and if so then abort.

searchguard.check_for_root: false

If this is true (default is false) then allow all HTTP REST requests from nodes loopback (e.g. localhost)

searchguard.allow_all_from_loopback: true

If this is true (default: false) then enable authenticated transports requests (e.g. TransportClient authentication)

This can be done in that way (for example):

TransportClient.get(new GetRequest(“marketing”, “customer”, “tp_3”).putHeader(“searchguard_transport_creds”, “c2FseWg6c2VjcmV0”))

Add a header “searchguard_transport_creds”

base64(username":"password) is the credentials string

base64(spock:secret) → c3BvY2s6c2VjcmV0

searchguard.transport_auth.enabled: false

#############################################################################################

Transport layer SSL

#############################################################################################

Enable or disable node-to-node ssl encryption

searchguard.ssl.transport.node.enabled: true

JKS or PKCS12

searchguard.ssl.transport.node.keystore_type: JKS

Absolute path to the keystore file (this stores the server certificates)

searchguard.ssl.transport.node.keystore_filepath: C:\Program Files\Hewlett-Packard\CSA/jboss-as/standalone/configuration/.keystore

Keystore password

searchguard.ssl.transport.node.keystore_password: changeit

Do other nodes have to authenticate themself to the cluster, default is true

searchguard.ssl.transport.node.enforce_clientauth: false

JKS or PKCS12

searchguard.ssl.transport.node.truststore_type: JKS

Absolute path to the truststore file (this stores the client certificates)

searchguard.ssl.transport.node.truststore_filepath: C:\Program Files\Hewlett-Packard\CSA/openjre/lib/security/cacerts

Truststore password

searchguard.ssl.transport.node.truststore_password: changeit

#############################################################################################

REST layer SSL

#############################################################################################

Enable or disable rest layer security (https)

searchguard.ssl.transport.http.enabled: true

JKS or PKCS12

searchguard.ssl.transport.http.keystore_type: JKS

Absolute path to the keystore file (this stores the server certificates)

searchguard.ssl.transport.http.keystore_filepath: C:\Program Files\Hewlett-Packard\CSA/jboss-as/standalone/configuration/.keystore

Keystore password

searchguard.ssl.transport.http.keystore_password: changeit

Do the clients (typically the browser or the proxy) have to authenticate themself to the http server, default is false

searchguard.ssl.transport.http.enforce_clientauth: false

JKS or PKCS12

searchguard.ssl.transport.http.truststore_type: JKS

Absolute path to the truststore file (this stores the client certificates)

searchguard.ssl.transport.http.truststore_filepath: C:\Program Files\Hewlett-Packard\CSA/openjre/lib/security/cacerts

Truststore password

searchguard.ssl.transport.http.truststore_password: changeit

#############################################################################################

X-Forwarded-For (XFF) header

#############################################################################################

X-Forwarded-For (XFF) header

If you have a http proxy in front of elasticsearch you have to configure this options to handle XFF properly

#searchguard.http.xforwardedfor.header: X-Forwarded-For

#searchguard.http.xforwardedfor.trustedproxies: null

#searchguard.http.xforwardedfor.enforce: false

#############################################################################################

Authentication backend

#############################################################################################

Validates the username and credentials

searchguard.authentication.authentication_backend.impl: com.floragunn.searchguard.authentication.backend.simple.SettingsBasedAuthenticationBackend

#searchguard.authentication.authentication_backend.impl: com.floragunn.searchguard.authentication.backend.ldap.LDAPAuthenticationBackend

#searchguard.authentication.authentication_backend.impl: com.floragunn.searchguard.authentication.backend.simple.AlwaysSucceedAuthenticationBackend

#searchguard.authentication.authentication_backend.impl: com.floragunn.searchguard.authentication.backend.waffle.WaffleAuthenticationBackend

If caching is enabled then the authentication succeed for 24 h since the first successful login without hitting the backend again and again

#searchguard.authentication.authentication_backend.cache.enable: true

#############################################################################################

Authorization backend (authorizer)

#############################################################################################

searchguard.authentication.authorizer.impl: com.floragunn.searchguard.authorization.simple.SettingsBasedAuthorizator

#searchguard.authentication.authorizer.impl: com.floragunn.searchguard.authorization.ldap.LDAPAuthorizator

#searchguard.authentication.authorizer.impl: com.floragunn.searchguard.authorization.waffle.WaffleAuthorizator

If caching is enabled then the role informations will be cached for 24 h without hitting the backend again and again

#searchguard.authentication.authorizer.cache.enable: true

#############################################################################################

HTTP authentication method

#############################################################################################

Define HTTP authentication method. In future we will here have more like NTLM, SPNEGO/Kerberos and Digest.

searchguard.authentication.http_authenticator.impl: com.floragunn.searchguard.authentication.http.basic.HTTPBasicAuthenticator

HTTPProxyAuthenticator assume there is kind of proxy in front of elasticsearch which handles the authentication and stores the

username of the authenticated user in a http header

#searchguard.authentication.http_authenticator.impl: com.floragunn.searchguard.authentication.http.proxy.HTTPProxyAuthenticator

SSL mutual authentication (works only if searchguard.ssl.transport.http.enabled is ‘true’ with client auth enabled)

#searchguard.authentication.http_authenticator.impl: com.floragunn.searchguard.authentication.http.clientcert.HTTPSClientCertAuthenticator

SPNEGO

#searchguard.authentication.http_authenticator.impl: com.floragunn.searchguard.authentication.http.spnego.HTTPSpnegoAuthenticator

Absolute file path to jaas login config file

#searchguard.authentication.spnego.login_config_filepath: null

Absolute file path to krb5 config file

#searchguard.authentication.spnego.krb5_config_filepath: null

Name of the login entry in jaas login config file which represents the acceptor (server)

#searchguard.authentication.spnego.login_config_name: com.sun.security.jgss.krb5.accept

Strip the realmname from username (hnelson@EXAMPLE.COM → hnelson)

#searchguard.authentication.spnego.strip_realm: true

Authenticates always a user with username ‘searchguard_unauthenticated_user’

#searchguard.authentication.http_authenticator.impl: com.floragunn.searchguard.authentication.http.HTTPUnauthenticatedAuthenticator

Waffle (Windows only, must be used with WaffleAuthorizator)

#searchguard.authentication.http_authenticator.impl: com.floragunn.searchguard.authentication.http.waffle.HTTPWaffleAuthenticator

Strip domain name from user (COMPANY\spock → spock)

#searchguard.authentication.waffle.strip_domain: true

#####################################################

Settings based authentication (define users and password directly here in the settings. Note: this is per node)

#searchguard.authentication.settingsdb.user.: password

searchguard.authentication.settingsdb.user.admin: cloud

#searchguard.authentication.settingsdb.user.user1: password

#searchguard.authentication.settingsdb.user.michaeljackson: neverland

If plain text password should be hashed use this. Supported digests are: SHA1 SHA256 SHA384 SHA512 MD5

#searchguard.authentication.settingsdb.digest: SHA1

#searchguard.authentication.settingsdb.user.admin: 000e793db70c59309fa6f0f36d0046d110f3be3c

#searchguard.authentication.settingsdb.user.michaeljackson: 824d55e7a62b7ca8751dff346ffab845a8f26d08

#####################################################

#####################################################

Settings based authorization (define users and their roles directly here in the settings. Note: this is per node)

#searchguard.authentication.authorization.settingsdb.roles.:

searchguard.authentication.authorization.settingsdb.roles.admin: [“admin”]

#searchguard.authentication.authorization.settingsdb.roles.user1: [“admin”]

#searchguard.authentication.authorization.settingsdb.roles.michaeljackson: [“admin”]

#####################################################

#####################################################

LDAP authentication backend (authenticate users against a LDAP or Active Directory)

The defaults are sufficient for Active Directory

#searchguard.authentication.ldap.host: [“localhost:389”]

#searchguard.authentication.ldap.ldaps.ssl.enabled: false

#searchguard.authentication.ldap.ldaps.starttls.enabled: false

JKS or PKCS12

#searchguard.authentication.ldap.ldaps.truststore_type: JKS

#searchguard.authentication.ldap.ldaps.truststore_filepath: null

#searchguard.authentication.ldap.ldaps.truststore_password: null

#searchguard.authentication.ldap.bind_dn: null

#searchguard.authentication.ldap.password: null

Default is root dse (“”)

#searchguard.authentication.ldap.userbase: “”

Filter to search for users (currently in the whole subtree beneath userbase)

{0} is substituted with the username

#searchguard.authentication.ldap.usersearch: (sAMAccountName={0})

Use this attribute from the user as username (if not set then DN is used)

#searchguard.authentication.ldap.username_attribute: null

#####################################################

#####################################################

LDAP authorization backend (gather roles from a LDAP or Active Directory, you have to configure the above LDAP authentication backend settings too)

The defaults are sufficient for Active Directory

Default is root dse (“”)

#searchguard.authentication.authorization.ldap.rolebase: “”

Filter to search for roles (currently in the whole subtree beneath rolebase)

{0} is substituted with the DN of the user

{1} is substituted with the username

{2} is substituted with an attribute value from user’s directory entry, of the authenticated user. Use userroleattribute to specify the name of the attribute

#searchguard.authentication.authorization.ldap.rolesearch: (member={0})

Specify the name of the attribute which value should be substituted with {2} above

#searchguard.authentication.authorization.ldap.userroleattribute: null

Roles as an attribute of the user entry

#searchguard.authentication.authorization.ldap.userrolename: memberOf

The attribute in a role entry containing the name of that role

#searchguard.authentication.authorization.ldap.rolename: name

Resolve nested roles transitive (roles which are members of other roles and so on …)

#searchguard.authentication.authorization.ldap.resolve_nested_roles: false

#####################################################

#####################################################

HTTP proxy authenticator configuration

Header name which contains the username

#searchguard.authentication.proxy.header:X-Authenticated-User

Array of trusted IP addresses (this are typically your proxy server(s))

#searchguard.authentication.proxy.trusted_ips: null

#####################################################

#####################################################

HTTP SSL mutual authentication configuration

Attribute of that attribute in the certificate dn which holds the username

#searchguard.authentication.https.clientcert.attributename: cn

#####################################################

##############################################################################################

Below here you configure what authenticated and authorized users are allowed to do (or not)#

This maps to the acl defined in the searchguard configuration index

#############################################################################################

Configure the restactionfilter to allow or forbid action

#searchguard.restactionfilter.names: [“readonly”]

#searchguard.restactionfilter.readonly.allowed_actions: [“*SearchAction”, “RestSearchScrollAction”, “RestClearScrollAction”, “RestGetAction”, “RestGetSourceAction”, “*MainAction”, “RestValidateQueryAction”, “RestMoreLikeThisAction”, “RestPercolateAction”]

#searchguard.restactionfilter.readonly.forbidden_actions: […]

Configure the actionrequestfilter to allow or forbid action

searchguard.actionrequestfilter.names: [“readonly”]

searchguard.actionrequestfilter.readonly.allowed_actions: [“*”]

#searchguard.actionrequestfilter.readonly.forbidden_actions: [“cluster:", "indices:admin”]

Configure document level security (dls) filter

Warning: All this (with the exception of “exists”) only works with not_analyzed fields because a term filter is used internally

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-term-filter.html

#searchguard.dlsfilter.names: [“a”, “b”, “c”, “d”, “e”, “f”, “g”]

#searchguard.dlsfilter.a: [“exists”,“field”, “false”] # if field exists (or not) match → false means field must exist

#searchguard.dlsfilter.b: [“term”, “field”,“value”, “false”] # if field==value (or not) match

#search

Hi,
Its great to know you fixed this issue. I have recently started evaluating this product on ES version 2.2.0. I am looking majorly for 2 use cases in terms of security and hope you might provide some helpful insights to it.

  • HTTP Rest security (HTTPS with username:password type authentication)
  • Secured inter node communication over SSL.

I tried first by just installing the search guard SSL plugin by giving correct path to

···

On Tuesday, August 4, 2015 at 12:55:07 AM UTC+8, mjs...@gmail.com wrote:

I have three instances of ES. I have followed following steps to form cluster configuration as follows

  1. I have used same cluster name
  1. I have used different node names
  1. I have marked all nodes and master and data nodes
  1. I have disabled multicaset
  1. Provided unicast ip’s
  1. Disabled searchguard
  1. When I restarted ES and verified the o/p of http://localhost:9201/_cluster/health?pretty=true. I have noticed that ES has formed cluster. It’s all good upto this point

Now,

  1. I have stopped ES.
  1. I have enabled search guard
  1. I have copied searchguard key from one node to other nodes
  1. I have imported SSL certificates of one node to others truststore
  1. restarted ES
  1. The ES didn’t form the cluster. The ES is started with 503 error. If I access any ES urls’s it is giving error as {“error”:“MasterNotDiscoveredException[waited for [30s]]”,“status”:503}

fyi; The above configuration used to work however it is not working now all of sudden.

I have the following searchguard rules stored in ES

{

“acl”: [

{

Comment”: “Default is to execute all filters”,

“filters_bypass”: [

“*”

],

“filters_execute”: [

“*”

]

},

{

Comment”: “Any authenticated user do anything on the ‘_all’, ‘searchguard’, ‘inventory’ index - no filter will be executed”,

“indices”: [

“_all”,

“searchguard”,

“inventory”

],

“filters_bypass”: [

“*”

],

“filters_execute”: [

“*”

]

},

{

Comment”: “For admin role all filters are bypassed (so none will be executed) for all indices. This means unrestricted access at all for this role.”,

“roles”: [

“admin”

],

“filters_bypass”: [

“*”

],

“filters_execute”:

}

]

}

Elasticsearch.yml file

##################### Elasticsearch Configuration Example #####################

This file contains an overview of various configuration settings,

targeted at operations staff. Application developers should

consult the guide at <http://elasticsearch.org/guide>.

The installation procedure is covered at

<http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html>.

Elasticsearch comes with reasonable defaults for most settings,

so you can try it out without bothering with configuration.

Most of the time, these defaults are just fine for running a production

cluster. If you’re fine-tuning your cluster, or wondering about the

effect of certain configuration option, please do ask on the

mailing list or IRC channel [http://elasticsearch.org/community].

Any element in the configuration can be replaced with environment variables

by placing them in ${…} notation. For example:

#node.rack: ${RACK_ENV_VAR}

For information on supported formats and syntax for the config file, see

<http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html>

################################### Cluster ###################################

Cluster name identifies your cluster for auto-discovery. If you’re running

multiple clusters on the same network, make sure you’re using unique names.

#cluster.name: “elasticsearch”

cluster.name: “sample”

#################################### Node #####################################

Node names are generated dynamically on startup, so you’re relieved

from configuring them manually. You can tie this node to a specific name:

node.name: “pm-cdl-win-2”

Every node can be configured to allow or deny being eligible as the master,

and to allow or deny to store the data.

Allow this node to be eligible as a master node (enabled by default):

node.master: true

Allow this node to store data (enabled by default):

node.data: true

You can exploit these settings to design advanced cluster topologies.

1. You want this node to never become a master node, only to hold data.

This will be the “workhorse” of your cluster.

#node.master: true

#node.data: true

2. You want this node to only serve as a master: to not store any data and

to have free resources. This will be the “coordinator” of your cluster.

#node.master: true

#node.data: false

3. You want this node to be neither master nor data node, but

to act as a “search load balancer” (fetching data from nodes,

aggregating results, etc.)

#node.master: false

#node.data: false

Use the Cluster Health API [http://localhost:9200/_cluster/health], the

Node Info API [http://localhost:9200/_nodes] or GUI tools

such as <http://www.elasticsearch.org/overview/marvel/>,

<http://github.com/karmi/elasticsearch-paramedic>,

<http://github.com/lukas-vlcek/bigdesk> and

<http://mobz.github.com/elasticsearch-head> to inspect the cluster state.

A node can have generic attributes associated with it, which can later be used

for customized shard allocation filtering, or allocation awareness. An attribute

is a simple key value pair, similar to node.key: value, here is an example:

#node.rack: rack314

By default, multiple nodes are allowed to start from the same installation location

to disable it, set the following:

#node.max_local_storage_nodes: 1

node.local: false

#################################### Index ####################################

You can set a number of options (such as shard/replica options, mapping

or analyzer definitions, translog settings, …) for indices globally,

in this file.

Note, that it makes more sense to configure index settings specifically for

a certain index, either when creating it or by using the index templates API.

See <http://elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules.html> and

<http://elasticsearch.org/guide/en/elasticsearch/reference/current/indices-create-index.html>

for more information.

Set the number of shards (splits) of an index (5 by default):

#index.number_of_shards: 5

Set the number of replicas (additional copies) of an index (1 by default):

#index.number_of_replicas: 1

Note, that for development on a local machine, with small indices, it usually

makes sense to “disable” the distributed features:

#index.number_of_shards: 1

#index.number_of_replicas: 0

These settings directly affect the performance of index and search operations

in your cluster. Assuming you have enough machines to hold shards and

replicas, the rule of thumb is:

1. Having more shards enhances the indexing performance and allows to

distribute a big index across machines.

2. Having more replicas enhances the search performance and improves the

cluster availability.

The “number_of_shards” is a one-time setting for an index.

The “number_of_replicas” can be increased or decreased anytime,

by using the Index Update Settings API.

Elasticsearch takes care about load balancing, relocating, gathering the

results from nodes, etc. Experiment with different settings to fine-tune

your setup.

Use the Index Status API (<http://localhost:9200/A/_status>) to inspect

the index status.

#################################### Paths ####################################

Path to directory containing configuration (this file and logging.yml):

#path.conf: /path/to/conf

Path to directory where to store index data allocated for this node.

#path.data: /path/to/data

Can optionally include more than one location, causing data to be striped across

the locations (a la RAID 0) on a file level, favouring locations with most free

space on creation. For example:

#path.data: /path/to/data1,/path/to/data2

Path to temporary files:

#path.work: /path/to/work

Path to log files:

#path.logs: /path/to/logs

Path to where plugins are installed:

#path.plugins: /path/to/plugins

#################################### Plugin ###################################

If a plugin listed here is not installed for current node, the node will not start.

#plugin.mandatory: mapper-attachments,lang-groovy

################################### Memory ####################################

Elasticsearch performs poorly when JVM starts swapping: you should ensure that

it never swaps.

Set this property to true to lock the memory:

#bootstrap.mlockall: true

Make sure that the ES_MIN_MEM and ES_MAX_MEM environment variables are set

to the same value, and that the machine has enough memory to allocate

for Elasticsearch, leaving enough memory for the operating system itself.

You should also make sure that the Elasticsearch process is allowed to lock

the memory, eg. by using ulimit -l unlimited.

############################## Network And HTTP ###############################

Elasticsearch, by default, binds itself to the 0.0.0.0 address, and listens

on port [9200-9300] for HTTP traffic and on port [9300-9400] for node-to-node

communication. (the range means that if the port is busy, it will automatically

try the next port).

Set the bind address specifically (IPv4 or IPv6):

#network.bind_host: localhost

Set the address other nodes will use to communicate with this node. If not

set, it is automatically derived. It must point to an actual IP address.

#network.publish_host: 192.168.0.1

Set both ‘bind_host’ and ‘publish_host’:

#network.host: 192.168.0.1

Set a custom port for the node to node communication (9300 by default):

#transport.tcp.port: 9300

transport.tcp.port: 9300

Enable compression for all communication between nodes (disabled by default):

#transport.tcp.compress: true

Set a custom port to listen for HTTP traffic:

http.port: 9201

Set a custom allowed content length:

#http.max_content_length: 100mb

Disable HTTP completely:

http.enabled: true

################################### Gateway ###################################

The gateway allows for persisting the cluster state between full cluster

restarts. Every change to the state (such as adding an index) will be stored

in the gateway, and when the cluster starts up for the first time,

it will read its state from the gateway.

There are several types of gateway implementations. For more information, see

<http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-gateway.html>.

The default gateway type is the “local” gateway (recommended):

#gateway.type: local

Settings below control how and when to start the initial recovery process on

a full cluster restart (to reuse as much local data as possible when using shared

gateway).

Allow recovery process after N nodes in a cluster are up:

#gateway.recover_after_nodes: 1

Set the timeout to initiate the recovery process, once the N nodes

from previous setting are up (accepts time value):

#gateway.recover_after_time: 5m

Set how many nodes are expected in this cluster. Once these N nodes

are up (and recover_after_nodes is met), begin recovery process immediately

(without waiting for recover_after_time to expire):

#gateway.expected_nodes: 2

############################# Recovery Throttling #############################

These settings allow to control the process of shards allocation between

nodes during initial recovery, replica allocation, rebalancing,

or when adding and removing nodes.

Set the number of concurrent recoveries happening on a node:

1. During the initial recovery

#cluster.routing.allocation.node_initial_primaries_recoveries: 4

2. During adding/removing nodes, rebalancing, etc

#cluster.routing.allocation.node_concurrent_recoveries: 2

Set to throttle throughput when recovering (eg. 100mb, by default 20mb):

#indices.recovery.max_bytes_per_sec: 20mb

Set to limit the number of open concurrent streams when

recovering a shard from a peer:

#indices.recovery.concurrent_streams: 5

################################## Discovery ##################################

Discovery infrastructure ensures nodes can be found within a cluster

and master node is elected. Multicast discovery is the default.

Set to ensure a node sees N other master eligible nodes to be considered

operational within the cluster. This should be set to a quorum/majority of

the master-eligible nodes in the cluster.

#discovery.zen.minimum_master_nodes: 1

Set the time to wait for ping responses from other nodes when discovering.

Set this option to a higher value on a slow or congested network

to minimize discovery failures:

discovery.zen.ping.timeout: 5s

For more information, see

<http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-zen.html>

Unicast discovery allows to explicitly control which nodes will be used

to discover the cluster. It can be used when multicast is not present,

or to restrict the cluster communication-wise.

1. Disable multicast discovery (enabled by default):

discovery.zen.ping.multicast.enabled: false

2. Configure an initial list of master nodes in the cluster

to perform discovery when new nodes (master or data) are started:

#discovery.zen.ping.unicast.hosts: [“host”: “host:port”]

discovery.zen.ping.unicast.hosts: [“pm-cdl-win-1.cdl.local”, “pm-cdl-win-2.cdl.local”, “pm-cdl-win-3.cdl.local”]

#discovery.zen.ping.unicast.hosts: [“10.1.16.11”, “10.1.16.17”, “10.1.16.19”]

#discovery.zen.no_master_block: all

discovery.zen.minimum_master_nodes: 1

EC2 discovery allows to use AWS EC2 API in order to perform discovery.

You have to install the cloud-aws plugin for enabling the EC2 discovery.

For more information, see

<http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-ec2.html>

See <http://elasticsearch.org/tutorials/elasticsearch-on-ec2/>

for a step-by-step tutorial.

GCE discovery allows to use Google Compute Engine API in order to perform discovery.

You have to install the cloud-gce plugin for enabling the GCE discovery.

For more information, see <https://github.com/elasticsearch/elasticsearch-cloud-gce>.

Azure discovery allows to use Azure API in order to perform discovery.

You have to install the cloud-azure plugin for enabling the Azure discovery.

For more information, see <https://github.com/elasticsearch/elasticsearch-cloud-azure>.

################################## Slow Log ##################################

Shard level query and fetch threshold logging.

#index.search.slowlog.threshold.query.warn: 10s

#index.search.slowlog.threshold.query.info: 5s

#index.search.slowlog.threshold.query.debug: 2s

#index.search.slowlog.threshold.query.trace: 500ms

#index.search.slowlog.threshold.fetch.warn: 1s

#index.search.slowlog.threshold.fetch.info: 800ms

#index.search.slowlog.threshold.fetch.debug: 500ms

#index.search.slowlog.threshold.fetch.trace: 200ms

#index.indexing.slowlog.threshold.index.warn: 10s

#index.indexing.slowlog.threshold.index.info: 5s

#index.indexing.slowlog.threshold.index.debug: 2s

#index.indexing.slowlog.threshold.index.trace: 500ms

################################## GC Logging ################################

#monitor.jvm.gc.young.warn: 1000ms

#monitor.jvm.gc.young.info: 700ms

#monitor.jvm.gc.young.debug: 400ms

#monitor.jvm.gc.old.warn: 10s

#monitor.jvm.gc.old.info: 5s

#monitor.jvm.gc.old.debug: 2s

################################## Security ################################

Uncomment if you want to enable JSONP as a valid return transport on the

http server. With this enabled, it may pose a security risk, so disabling

it unless you need it is recommended (it is disabled by default).

#http.jsonp.enable: true

#############################################################################################

SEARCH GUARD

Configuration

#############################################################################################

#Note: All waffle related options are only valid if your ES node is running on windows OS

Enable or disable the complete Searchguard plugin functionality

searchguard.enabled: true

Path where to write/read the searchguard master key file

#searchguard.key_path: .

When using DLS or FLS and a get or mget is performed then rewrite it as search request

#searchguard.rewrite_get_as_search: true

The index name where Searchguard will store its configuration and various other informations related to Searchguard itself

This index can only be access from localhost

searchguard.config_index_name: searchguard

Enable or disable HTTP session which caches the authentication and authorization informations in a cookie

#searchguard.http.enable_sessions: false

Enable or disable audit logging

#searchguard.auditlog.enabled: true

If this is true (default is false) then Searchguard will check if elasticsearch is running as root/windows admin and if so then abort.

searchguard.check_for_root: false

If this is true (default is false) then allow all HTTP REST requests from nodes loopback (e.g. localhost)

searchguard.allow_all_from_loopback: true

If this is true (default: false) then enable authenticated transports requests (e.g. TransportClient authentication)

This can be done in that way (for example):

TransportClient.get(new GetRequest(“marketing”, “customer”, “tp_3”).putHeader(“searchguard_transport_creds”, “c2FseWg6c2VjcmV0”))

Add a header “searchguard_transport_creds”

base64(username":"password) is the credentials string

base64(spock:secret) → c3BvY2s6c2VjcmV0

searchguard.transport_auth.enabled: false

#############################################################################################

Transport layer SSL

#############################################################################################

Enable or disable node-to-node ssl encryption

searchguard.ssl.transport.node.enabled: true

JKS or PKCS12

searchguard.ssl.transport.node.keystore_type: JKS

Absolute path to the keystore file (this stores the server certificates)

searchguard.ssl.transport.node.keystore_filepath: C:\Program Files\Hewlett-Packard\CSA/jboss-as/standalone/configuration/.keystore

Keystore password

searchguard.ssl.transport.node.keystore_password: changeit

Do other nodes have to authenticate themself to the cluster, default is true

searchguard.ssl.transport.node.enforce_clientauth: false

JKS or PKCS12

searchguard.ssl.transport.node.truststore_type: JKS

Absolute path to the truststore file (this stores the client certificates)

searchguard.ssl.transport.node.truststore_filepath: C:\Program Files\Hewlett-Packard\CSA/openjre/lib/security/cacerts

Truststore password

searchguard.ssl.transport.node.truststore_password: changeit

#############################################################################################

REST layer SSL

#############################################################################################

Enable or disable rest layer security (https)

searchguard.ssl.transport.http.enabled: true

JKS or PKCS12

searchguard.ssl.transport.http.keystore_type: JKS

Absolute path to the keystore file (this stores the server certificates)

searchguard.ssl.transport.http.keystore_filepath: C:\Program Files\Hewlett-Packard\CSA/jboss-as/standalone/configuration/.keystore

Keystore password

searchguard.ssl.transport.http.keystore_password: changeit

Do the clients (typically the browser or the proxy) have to authenticate themself to the http server, default is false

searchguard.ssl.transport.http.enforce_clientauth: false

JKS or PKCS12

searchguard.ssl.transport.http.truststore_type: JKS

Absolute path to the truststore file (this stores the client certificates)

searchguard.ssl.transport.http.truststore_filepath: C:\Program Files\Hewlett-Packard\CSA/openjre/lib/security/cacerts

Truststore password

searchguard.ssl.transport.http.truststore_password: changeit

#############################################################################################

X-Forwarded-For (XFF) header

#############################################################################################

X-Forwarded-For (XFF) header

If you have a http proxy in front of elasticsearch you have to configure this options to handle XFF properly

#searchguard.http.xforwardedfor.header: X-Forwarded-For

#searchguard.http.xforwardedfor.trustedproxies: null

#searchguard.http.xforwardedfor.enforce: false

#############################################################################################

Authentication backend

#############################################################################################

Validates the username and credentials

searchguard.authentication.authentication_backend.impl: com.floragunn.searchguard.authentication.backend.simple.SettingsBasedAuthenticationBackend

#searchguard.authentication.authentication_backend.impl: com.floragunn.searchguard.authentication.backend.ldap.LDAPAuthenticationBackend

#searchguard.authentication.authentication_backend.impl: com.floragunn.searchguard.authentication.backend.simple.AlwaysSucceedAuthenticationBackend

#searchguard.authentication.authentication_backend.impl: com.floragunn.searchguard.authentication.backend.waffle.WaffleAuthenticationBackend

If caching is enabled then the authentication succeed for 24 h since the first successful login without hitting the backend again and again

#searchguard.authentication.authentication_backend.cache.enable: true

#############################################################################################

Authorization backend (authorizer)

#############################################################################################

searchguard.authentication.authorizer.impl: com.floragunn.searchguard.authorization.simple.SettingsBasedAuthorizator

#searchguard.authentication.authorizer.impl: com.floragunn.searchguard.authorization.ldap.LDAPAuthorizator

#searchguard.authentication.authorizer.impl: com.floragunn.searchguard.authorization.waffle.WaffleAuthorizator

If caching is enabled then the role informations will be cached for 24 h without hitting the backend again and again

#searchguard.authentication.authorizer.cache.enable: true

#############################################################################################

HTTP authentication method

#############################################################################################

Define HTTP authentication method. In future we will here have more like NTLM, SPNEGO/Kerberos and Digest.

searchguard.authentication.http_authenticator.impl: com.floragunn.searchguard.authentication.http.basic.HTTPBasicAuthenticator

HTTPProxyAuthenticator assume there is kind of proxy in front of elasticsearch which handles the authentication and stores the

username of the authenticated user in a http header

#searchguard.authentication.http_authenticator.impl: com.floragunn.searchguard.authentication.http.proxy.HTTPProxyAuthenticator

SSL mutual authentication (works only if searchguard.ssl.transport.http.enabled is ‘true’ with client auth enabled)

#searchguard.authentication.http_authenticator.impl: com.floragunn.searchguard.authentication.http.clientcert.HTTPSClientCertAuthenticator

SPNEGO

#searchguard.authentication.http_authenticator.impl: com.floragunn.searchguard.authentication.http.spnego.HTTPSpnegoAuthenticator

Absolute file path to jaas login config file

#searchguard.authentication.spnego.login_config_filepath: null

Absolute file path to krb5 config file

#searchguard.authentication.spnego.krb5_config_filepath: null

Name of the login entry in jaas login config file which represents the acceptor (server)

#searchguard.authentication.spnego.login_config_name: com.sun.security.jgss.krb5.accept

Strip the realmname from username (hne...@EXAMPLE.COM → hnelson)

#searchguard.authentication.spnego.strip_realm: true

Authenticates always a user with username ‘searchguard_unauthenticated_user’

#searchguard.authentication.http_authenticator.impl: com.floragunn.searchguard.authentication.http.HTTPUnauthenticatedAuthenticator

Waffle (Windows only, must be used with WaffleAuthorizator)

#searchguard.authentication.http_authenticator.impl: com.floragunn.searchguard.authentication.http.waffle.HTTPWaffleAuthenticator

Strip domain name from user (COMPANY\spock → spock)

#searchguard.authentication.waffle.strip_domain: true

#####################################################

Settings based authentication (define users and password directly here in the settings. Note: this is per node)

#searchguard.authentication.settingsdb.user.: password

searchguard.authentication.settingsdb.user.admin: cloud

#searchguard.authentication.settingsdb.user.user1: password

#searchguard.authentication.settingsdb.user.michaeljackson: neverland

If plain text password should be hashed use this. Supported digests are: SHA1 SHA256 SHA384 SHA512 MD5

#searchguard.authentication.settingsdb.digest: SHA1

#searchguard.authentication.settingsdb.user.admin: 000e793db70c59309fa6f0f36d0046d110f3be3c

#searchguard.authentication.settingsdb.user.michaeljackson: 824d55e7a62b7ca8751dff346ffab845a8f26d08

#####################################################

#####################################################

Settings based authorization (define users and their roles directly here in the settings. Note: this is per node)

#searchguard.authentication.authorization.settingsdb.roles.:

searchguard.authentication.authorization.settingsdb.roles.admin: [“admin”]

#searchguard.authentication.authorization.settingsdb.roles.user1: [“admin”]

#searchguard.authentication.authorization.settingsdb.roles.michaeljackson: [“admin”]

#####################################################

#####################################################

LDAP authentication backend (authenticate users against a LDAP or Active Directory)

The defaults are sufficient for Active Directory

#searchguard.authentication.ldap.host: [“localhost:389”]

#searchguard.authentication.ldap.ldaps.ssl.enabled: false

#searchguard.authentication.ldap.ldaps.starttls.enabled: false

JKS or PKCS12

#searchguard.authentication.ldap.ldaps.truststore_type: JKS

#searchguard.authentication.ldap.ldaps.truststore_filepath: null

#searchguard.authentication.ldap.ldaps.truststore_password: null

#searchguard.authentication.ldap.bind_dn: null

#searchguard.authentication.ldap.password: null

Default is root dse (“”)

#searchguard.authentication.ldap.userbase: “”

Filter to search for users (currently in the whole subtree beneath userbase)

{0} is substituted with the username

#searchguard.authentication.ldap.usersearch: (sAMAccountName={0})

Use this attribute from the user as username (if not set then DN is used)

#searchguard.authentication.ldap.username_attribute: null

#####################################################

#####################################################

LDAP authorization backend (gather roles from a LDAP or Active Directory, you have to configure the above LDAP authentication backend settings too)

The defaults are sufficient for Active Directory

Default is root dse (“”)

#searchguard.authentication.authorization.ldap.rolebase: “”

Filter to search for roles (currently in the whole subtree beneath rolebase)

{0} is substituted with the DN of the user

{1} is substituted with the username

{2} is substituted with an attribute value from user’s directory entry, of the authenticated user. Use userroleattribute to specify the name of the attribute

#searchguard.authentication.authorization.ldap.rolesearch: (member={0})

Specify the name of the attribute which value should be substituted with {2} above

#searchguard.authentication.authorization.ldap.userroleattribute: null

Roles as an attribute of the user entry

#searchguard.authentication.authorization.ldap.userrolename: memberOf

The attribute in a role entry containing the name of that role

#searchguard.authentication.authorization.ldap.rolename: name

Resolve nested roles transitive (roles which are members of other roles and so on …)

#searchguard.authentication.authorization.ldap.resolve_nested_roles: false

#####################################################

#####################################################

HTTP proxy authenticator configuration

Header name which contains the username

#searchguard.authentication.proxy.header:X-Authenticated-User

Array of trusted IP addresses (this are typically your proxy server(s))

#searchguard.authentication.proxy.trusted_ips: null

#####################################################

#####################################################

HTTP SSL mutual authentication configuration

Attribute of that attribute in the certificate dn which holds the username

#searchguard.authentication.https.clientcert.attributename: cn

#####################################################

##############################################################################################

Below here you configure what authenticated and authorized users are allowed to do (or not)#

This maps to the acl defined in the searchguard configuration index

#############################################################################################

Configure the restactionfilter to allow or forbid action

#searchguard.restactionfilter.names: [“readonly”]

#searchguard.restactionfilter.readonly.allowed_actions: [“*SearchAction”, “RestSearchScrollAction”, “RestClearScrollAction”, “RestGetAction”, “RestGetSourceAction”, “*MainAction”, “RestValidateQueryAction”, “RestMoreLikeThisAction”, “RestPercolateAction”]

#searchguard.restactionfilter.readonly.forbidden_actions: […]

Configure the actionrequestfilter to allow or forbid action

searchguard.actionrequestfilter.names: [“readonly”]

searchguard.actionrequestfilter.readonly.allowed_actions: [“*”]

#searchguard.actionrequestfilter.readonly.forbidden_actions: [“cluster:", "indices:admin”]

Configure document level security (dls) filter

Warning: All this (with the exception of “exists”) only works with not_analyzed fields because a term filter is used internally

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-term-filter.html

#searchguard.dlsfilter.names: [“a”, “b”, “c”, “d”, “e”, “f”, “g”]

#searchguard.dlsfilter.a: [“exists”,“field”, “false”] # if field exists (or not) match → false means field must exist

#searchguard.dlsfilter.b: [“term”, “field”,“value”, “false”] # if field==value (or not) match

#search