Permissions issue with indices:data/write/bulk[s]

Search Guard
1
  • Search Guard and Elasticsearch version

{

“name” : “vK2vBkK”,

“cluster_name” : “elasticsearch”,

“cluster_uuid” : “t_EwPLkSRDWMrUAcaae9Uw”,

“version” : {

“number” : “5.3.0”,

“build_hash” : “3adb13b”,

“build_date” : “2017-03-23T03:31:50.652Z”,

“build_snapshot” : false,

“lucene_version” : “6.4.1”

},

“tagline” : “You Know, for Search”

}

search-guard-5-5.3.0-11.jar

  • JVM version and operating system version

java version “1.8.0_121”

Java™ SE Runtime Environment (build 1.8.0_121-b13)

Java HotSpot™ 64-Bit Server VM (build 25.121-b13, mixed mode)

  • Number of nodes in your cluster

3

  • Description of the bug

After upgrading to 5.3.0 we started getting errors on our ES servers for permissions errors for data/write/bulk[s] when logging in from our Kibana servers:

[2017-04-06T23:57:08,636][INFO ][c.f.s.c.PrivilegesEvaluator] No perm match for User [name=user, roles=] [IndexType [index=.kibana-367, type=*]] [Action [indices:data/write/bulk[s]]] [RolesChecked [sg_kibana_optimizely, sg_public]]

Here is an example of the permissions set in sg_roles.yml.

sg_kibana_user:

cluster:

  • ‘*’

  • indices:data/write/bulk* ← Added after finding an article that seemed related but no change in behavior

indices:

‘?kibana-367’:

‘index-pattern’:

  • KIBANA_INDEX_PATTERNS

‘*’:

  • KIBANA_INDEX_OTHER

‘367-*’:

‘*’:

  • SPARK_ORG_USER

  • READ

  • SEARCH

  • indices:admin/mappings/fields/get*

‘travelers-367-*’:

‘*’:

  • SPARK_ORG_USER

  • READ

  • SEARCH

  • indices:admin/mappings/fields/get*

Unsure if it’s related but there was an ES crash before this started happening. I tried to open an issue with ES as well but they closed it immediately citing lack of information.

[2017-04-06T20:10:15,489][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] fatal error in thread [elasticsearch[MkZ0lPb][bulk][T#1]], exiting

java.lang.StackOverflowError: null

I don’t have much else to give you. There’s nothing else in the logs that seems related or interesting.

2

Can you please post your complete sg_roles.yml and sg_action_groups.yml?

Is the ES crash reproducible? (I guess thats the github issue: Issues · elastic/elasticsearch · GitHub)

Elasticsearch does have a breaking change in 5.3 regarding how index is handled, see also Make index and delete operation execute as single bulk item (backport of #21964) by areek · Pull Request #22812 · elastic/elasticsearch · GitHub
and https://groups.google.com/d/msg/search-guard/pgwf1VsUL2s/jYfL7dFnAgAJ

···

Am 07.04.2017 um 03:46 schrieb lance via Search Guard <search-guard@googlegroups.com>:

* Search Guard and Elasticsearch version
{
  "name" : "vK2vBkK",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "t_EwPLkSRDWMrUAcaae9Uw",
  "version" : {
    "number" : "5.3.0",
    "build_hash" : "3adb13b",
    "build_date" : "2017-03-23T03:31:50.652Z",
    "build_snapshot" : false,
    "lucene_version" : "6.4.1"
  },
  "tagline" : "You Know, for Search"
}

search-guard-5-5.3.0-11.jar

* JVM version and operating system version
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

* Number of nodes in your cluster
3

* Description of the bug
After upgrading to 5.3.0 we started getting errors on our ES servers for permissions errors for data/write/bulk[s] when logging in from our Kibana servers:

[2017-04-06T23:57:08,636][INFO ][c.f.s.c.PrivilegesEvaluator] No perm match for User [name=user, roles=] [IndexType [index=.kibana-367, type=*]] [Action [indices:data/write/bulk[s]]] [RolesChecked [sg_kibana_optimizely, sg_public]]

Here is an example of the permissions set in sg_roles.yml.
sg_kibana_user:
  cluster:
    - '*'
    - indices:data/write/bulk* <-- Added after finding an article that seemed related but no change in behavior
  indices:
    '?kibana-367':
      'index-pattern':
        - KIBANA_INDEX_PATTERNS
      '*':
        - KIBANA_INDEX_OTHER
    '367-*':
      '*':
        - SPARK_ORG_USER
        - READ
        - SEARCH
        - indices:admin/mappings/fields/get*
    'travelers-367-*':
      '*':
        - SPARK_ORG_USER
        - READ
        - SEARCH
        - indices:admin/mappings/fields/get*

Unsure if it's related but there was an ES crash before this started happening. I tried to open an issue with ES as well but they closed it immediately citing lack of information.

[2017-04-06T20:10:15,489][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] fatal error in thread [elasticsearch[MkZ0lPb][bulk][T#1]], exiting
java.lang.StackOverflowError: null

I don't have much else to give you. There's nothing else in the logs that seems related or interesting.

--
You received this message because you are subscribed to the Google Groups "Search Guard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/ed478e17-65e5-4bf8-937f-f31ab2694a62%40googlegroups.com\.
For more options, visit https://groups.google.com/d/optout\.

3

do you use regex patterns somewhere? seems the crash (caused by a stackoverflow) is related to regex pattern matching

	[2017-04-06T20:10:15,489][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[MkZ0lPb][bulk][T#1]], exiting
java.lang.StackOverflowError: null
at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658) ~[?:1.8.0_121]
at java.util.regex.Pattern$Loop.match(Pattern.java:4785) ~[?:1.8.0_121]
at java.util.regex.Pattern$GroupTail.match(Pattern.java:4717) ~[?:1.8.0_121]
at java.util.regex.Pattern$BranchConn.match(Pattern.java:4568) ~[?:1.8.0_121]
at java.util.regex.Pattern$CharProperty.match(Pattern.java:3777) ~[?:1.8.0_121]
at java.util.regex.Pattern$Branch.match(Pattern.java:4604) ~[?:1.8.0_121]
at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658) ~[?:1.8.0_121]
at java.util.regex.Pattern$Loop.match(Pattern.java:4785) ~[?:1.8.0_121]
at java.util.regex.Pattern$GroupTail.match(Pattern.java:4717) ~[?:1.8.0_121]
at java.util.regex.Pattern$BranchConn.match(Pattern.java:4568) ~[?:1.8.0_121]
at java.util.regex.Pattern$CharProperty.match(Pattern.java:3777) ~[?:1.8.0_121]
at java.util.regex.Pattern$Branch.match(Pattern.java:4604) ~[?:1.8.0_121]
at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658) ~[?:1.8.0_121]
···

On Friday, 7 April 2017 03:46:21 UTC+2, lxxx@xxxxxnd.com wrote:

  • Search Guard and Elasticsearch version

{

“name” : “vK2vBkK”,

“cluster_name” : “elasticsearch”,

“cluster_uuid” : “t_EwPLkSRDWMrUAcaae9Uw”,

“version” : {

“number” : “5.3.0”,

“build_hash” : “3adb13b”,

“build_date” : “2017-03-23T03:31:50.652Z”,

“build_snapshot” : false,

“lucene_version” : “6.4.1”

},

“tagline” : “You Know, for Search”

}

search-guard-5-5.3.0-11.jar

  • JVM version and operating system version

java version “1.8.0_121”

Java™ SE Runtime Environment (build 1.8.0_121-b13)

Java HotSpot™ 64-Bit Server VM (build 25.121-b13, mixed mode)

  • Number of nodes in your cluster

3

  • Description of the bug

After upgrading to 5.3.0 we started getting errors on our ES servers for permissions errors for data/write/bulk[s] when logging in from our Kibana servers:

[2017-04-06T23:57:08,636][INFO ][c.f.s.c.PrivilegesEvaluator] No perm match for User [name=user, roles=] [IndexType [index=.kibana-367, type=*]] [Action [indices:data/write/bulk[s]]] [RolesChecked [sg_kibana_optimizely, sg_public]]

Here is an example of the permissions set in sg_roles.yml.

sg_kibana_user:

cluster:

  • ‘*’
  • indices:data/write/bulk* ← Added after finding an article that seemed related but no change in behavior

indices:

‘?kibana-367’:

‘index-pattern’:

  • KIBANA_INDEX_PATTERNS

‘*’:

  • KIBANA_INDEX_OTHER

‘367-*’:

‘*’:

  • SPARK_ORG_USER
  • READ
  • SEARCH
  • indices:admin/mappings/fields/get*

‘travelers-367-*’:

‘*’:

  • SPARK_ORG_USER
  • READ
  • SEARCH
  • indices:admin/mappings/fields/get*

Unsure if it’s related but there was an ES crash before this started happening. I tried to open an issue with ES as well but they closed it immediately citing lack of information.

[2017-04-06T20:10:15,489][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] fatal error in thread [elasticsearch[MkZ0lPb][bulk][T#1]], exiting

java.lang.StackOverflowError: null

I don’t have much else to give you. There’s nothing else in the logs that seems related or interesting.

4

I’ve attached the roles and action groups files. I’m discussing your other questions with my coworkers. Appreciate the quick response.

sg_action_groups.yml (1.54 KB)

sg_roles.yml (3.93 KB)

···

On Thursday, April 6, 2017 at 11:56:24 PM UTC-7, Search Guard wrote:

Can you please post your complete sg_roles.yml and sg_action_groups.yml?

Is the ES crash reproducible? (I guess thats the github issue: https://github.com/elastic/elasticsearch/issues/23955)

Elasticsearch does have a breaking change in 5.3 regarding how index is handled, see also https://github.com/elastic/elasticsearch/pull/22812

and https://groups.google.com/d/msg/search-guard/pgwf1VsUL2s/jYfL7dFnAgAJ

Am 07.04.2017 um 03:46 schrieb lance via Search Guard search...@googlegroups.com:

  • Search Guard and Elasticsearch version

{

“name” : “vK2vBkK”,

“cluster_name” : “elasticsearch”,

“cluster_uuid” : “t_EwPLkSRDWMrUAcaae9Uw”,

“version” : {

"number" : "5.3.0",

"build_hash" : "3adb13b",

"build_date" : "2017-03-23T03:31:50.652Z",

"build_snapshot" : false,

"lucene_version" : "6.4.1"

},

“tagline” : “You Know, for Search”

}

search-guard-5-5.3.0-11.jar

  • JVM version and operating system version

java version “1.8.0_121”

Java™ SE Runtime Environment (build 1.8.0_121-b13)

Java HotSpot™ 64-Bit Server VM (build 25.121-b13, mixed mode)

  • Number of nodes in your cluster

3

  • Description of the bug

After upgrading to 5.3.0 we started getting errors on our ES servers for permissions errors for data/write/bulk[s] when logging in from our Kibana servers:

[2017-04-06T23:57:08,636][INFO ][c.f.s.c.PrivilegesEvaluator] No perm match for User [name=user, roles=] [IndexType [index=.kibana-367, type=*]] [Action [indices:data/write/bulk[s]]] [RolesChecked [sg_kibana_optimizely, sg_public]]

Here is an example of the permissions set in sg_roles.yml.

sg_kibana_user:

cluster:

- '*'

- indices:data/write/bulk* <-- Added after finding an article that seemed related but no change in behavior

indices:

'?kibana-367':

  'index-pattern':

    - KIBANA_INDEX_PATTERNS

  '*':

    - KIBANA_INDEX_OTHER

'367-*':

  '*':

    - SPARK_ORG_USER

    - READ

    - SEARCH

    - indices:admin/mappings/fields/get*

'travelers-367-*':

  '*':

    - SPARK_ORG_USER

    - READ

    - SEARCH

    - indices:admin/mappings/fields/get*

Unsure if it’s related but there was an ES crash before this started happening. I tried to open an issue with ES as well but they closed it immediately citing lack of information.

[2017-04-06T20:10:15,489][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] fatal error in thread [elasticsearch[MkZ0lPb][bulk][T#1]], exiting

java.lang.StackOverflowError: null

I don’t have much else to give you. There’s nothing else in the logs that seems related or interesting.

–
You received this message because you are subscribed to the Google Groups “Search Guard” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/ed478e17-65e5-4bf8-937f-f31ab2694a62%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

5

Appreciate you taking a look. We have rolled back to 5.2.2 after three consecutive crashes. I’ll see if we can reproduce in a test environment.

···

On Friday, April 7, 2017 at 12:26:19 AM UTC-7, Search Guard wrote:

do you use regex patterns somewhere? seems the crash (caused by a stackoverflow) is related to regex pattern matching

[2017-04-06T20:10:15,489][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[MkZ0lPb][bulk][	T#1]], exiting
java.lang.StackOverflowError: null
at java.util.regex.Pattern$GroupHead.match(Pattern.java:	4658) ~[?:1.8.0_121]
at java.util.regex.Pattern$Loop.	match(Pattern.java:4785) ~[?:1.8.0_121]
at java.util.regex.Pattern$GroupTail.match(Pattern.java:	4717) ~[?:1.8.0_121]
at java.util.regex.Pattern$BranchConn.match(Pattern.java:	4568) ~[?:1.8.0_121]
at java.util.regex.Pattern$CharProperty.match(Pattern.	java:3777) ~[?:1.8.0_121]
at java.util.regex.Pattern$Branch.match(Pattern.java:	4604) ~[?:1.8.0_121]
at java.util.regex.Pattern$GroupHead.match(Pattern.java:	4658) ~[?:1.8.0_121]
at java.util.regex.Pattern$Loop.	match(Pattern.java:4785) ~[?:1.8.0_121]
at java.util.regex.Pattern$GroupTail.match(Pattern.java:	4717) ~[?:1.8.0_121]
at java.util.regex.Pattern$BranchConn.match(Pattern.java:	4568) ~[?:1.8.0_121]
at java.util.regex.Pattern$CharProperty.match(Pattern.	java:3777) ~[?:1.8.0_121]
at java.util.regex.Pattern$Branch.match(Pattern.java:	4604) ~[?:1.8.0_121]
at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658) ~[?:1.8.0_121]

http://www.regular-expressions.info/catastrophic.html

On Friday, 7 April 2017 03:46:21 UTC+2, lx...@xxxxxnd.com wrote:

  • Search Guard and Elasticsearch version

{

“name” : “vK2vBkK”,

“cluster_name” : “elasticsearch”,

“cluster_uuid” : “t_EwPLkSRDWMrUAcaae9Uw”,

“version” : {

“number” : “5.3.0”,

“build_hash” : “3adb13b”,

“build_date” : “2017-03-23T03:31:50.652Z”,

“build_snapshot” : false,

“lucene_version” : “6.4.1”

},

“tagline” : “You Know, for Search”

}

search-guard-5-5.3.0-11.jar

  • JVM version and operating system version

java version “1.8.0_121”

Java™ SE Runtime Environment (build 1.8.0_121-b13)

Java HotSpot™ 64-Bit Server VM (build 25.121-b13, mixed mode)

  • Number of nodes in your cluster

3

  • Description of the bug

After upgrading to 5.3.0 we started getting errors on our ES servers for permissions errors for data/write/bulk[s] when logging in from our Kibana servers:

[2017-04-06T23:57:08,636][INFO ][c.f.s.c.PrivilegesEvaluator] No perm match for User [name=user, roles=] [IndexType [index=.kibana-367, type=*]] [Action [indices:data/write/bulk[s]]] [RolesChecked [sg_kibana_optimizely, sg_public]]

Here is an example of the permissions set in sg_roles.yml.

sg_kibana_user:

cluster:

  • ‘*’
  • indices:data/write/bulk* ← Added after finding an article that seemed related but no change in behavior

indices:

‘?kibana-367’:

‘index-pattern’:

  • KIBANA_INDEX_PATTERNS

‘*’:

  • KIBANA_INDEX_OTHER

‘367-*’:

‘*’:

  • SPARK_ORG_USER
  • READ
  • SEARCH
  • indices:admin/mappings/fields/get*

‘travelers-367-*’:

‘*’:

  • SPARK_ORG_USER
  • READ
  • SEARCH
  • indices:admin/mappings/fields/get*

Unsure if it’s related but there was an ES crash before this started happening. I tried to open an issue with ES as well but they closed it immediately citing lack of information.

[2017-04-06T20:10:15,489][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] fatal error in thread [elasticsearch[MkZ0lPb][bulk][T#1]], exiting

java.lang.StackOverflowError: null

I don’t have much else to give you. There’s nothing else in the logs that seems related or interesting.