Alias Filter do not work with Search Guard on ES 7.8.0

Lucas_Bremgartner · August 7, 2020, 10:09am

Elasticsearch version: 7.8.0

Server OS version: RHEL 7

Describe the issue:

We upgraded from ES 6.8.9 (with SG) to ES 7.8.0 (with SG) when we noticed, that filters on index alias (https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html#filtered) have stopped working (the queries no longer get filtered). To narrow down the problem, we also performed a test with ES 7.8.0 without SG and then the alias filter worked as expected.

Steps to reproduce:

With ES 7.8.0 and SG enabled
Create an index, insert some data and add a alias with a filter on on of the fields
Query the data via the index alias, only the filtered fraction of the data should be returned

Expected behavior:

Index alias should work the same with and without SG.

srgbnd · August 7, 2020, 10:19am

Do you see any error in the Elasticsearch log?

Lucas_Bremgartner · August 7, 2020, 10:21am

No, I do not see an error in elasticsearch.log.

Lucas_Bremgartner · August 7, 2020, 10:29am

I have an alias called logstash-aa with the following filter:

{
  "term": {
    "umgebung": "aa"
  }
}

If I execute the following query:

GET /logstash-aa/_search?pretty
{
    "size": 0,
    "query": {
        "bool": {
            "must": [],
            "filter": [
                {
                    "match_all": {}
                },
                {
                    "range": {
                        "@timestamp": {
                            "gte": "2020-08-07T07:00:00.000Z",
                            "lte": "2020-08-07T07:15:00.000Z",
                            "format": "strict_date_optional_time"
                        }
                    }
                }
            ],
            "should": [],
            "must_not": []
        }
    },
    "track_total_hits": true
}

I get:

{
  "took" : 56,
  "timed_out" : false,
  "_shards" : {
    "total" : 269,
    "successful" : 269,
    "skipped" : 257,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 421724,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}

If I execute the same query with the additional filter on umgebung, I get a different result:

GET /logstash-aa/_search?pretty
{
    "size": 0,
    "query": {
        "bool": {
            "must": [],
            "filter": [
                {
                    "match_all": {}
                },
                {
                    "term": {
                        "umgebung": "aa"
                    }
                },
                {
                    "range": {
                        "@timestamp": {
                            "gte": "2020-08-07T07:00:00.000Z",
                            "lte": "2020-08-07T07:15:00.000Z",
                            "format": "strict_date_optional_time"
                        }
                    }
                }
            ],
            "should": [],
            "must_not": []
        }
    },
    "track_total_hits": true
}

Result

{
  "took" : 66,
  "timed_out" : false,
  "_shards" : {
    "total" : 269,
    "successful" : 269,
    "skipped" : 257,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 125658,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}

But I expect both queries to return the same result for the total hits.

srgbnd · August 7, 2020, 10:42am

Hm. I can’t reproduce the issue.

My steps

Go to the Kibana sample data https://kibana.example.com:5601/kzn/app/kibana#/home/tutorial_directory/sampleData. Add Sample flight data.
Search the index:

GET kibana_sample_data_flights/_search
{
  "size" : 0,
  "query": {
    "bool": {
      "filter": [
        { "match_all": {} }
      ]
    }
  }
}

The result:

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}

Add alias with filter:

POST /_aliases
{
  "actions": [
    {
      "add": {
        "index": "kibana_sample_data_flights",
        "alias": "kibana_sample_data_flights_alias_0",
        "filter": { "term": { "Carrier": "Kibana Airlines" } }
      }
    }
  ]
}

Search the alias:

GET kibana_sample_data_flights_alias_0/_search
{
  "size" : 0,
  "query": {
    "bool": {
      "filter": [
        { "match_all": {} }
      ]
    }
  }
}

Result:

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 3234,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}

Add filter to the search query and search the alias again:

GET kibana_sample_data_flights_alias_0/_search
{
  "size" : 0,
  "query": {
    "bool": {
      "filter": [
        { "match_all": {} },
        { "term": {
          "Carrier": "Kibana Airlines"
        }}
      ]
    }
  }
}

The result:

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 3234,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}

See, the hits.total.value is equal to the previous result.

Lucas_Bremgartner · August 7, 2020, 11:57am

I also loaded the Kibana demo dataset and I executed your queries. In step 4 I get a different result, that is:

GET kibana_sample_data_flights_alias_0/_search
{
  "size" : 0,
  "query": {
    "bool": {
      "filter": [
        { "match_all": {} }
      ]
    }
  }
}

Result

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}

If I add the "track_total_hits": true (just to be sure), I get:

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 13059,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}

I have the following versions:

ES: 7.8.0
SG: 7.8.0-42.0.0
Kibana: 7.8.0
SG Kibana Plugin: 7.8.0-42.0.0

In SG encryption as well as authentication is enabled. As mentioned in the initial post, it worked with ES 6.8.9 (with SG) and it also works without SG (ES 7.8.0).

Lucas_Bremgartner · August 7, 2020, 12:28pm

I tested the following two settings, but they did not make any difference:

sg_config.dynamic.filtered_alias_mode (default value warn and nowarn)
searchguard.filter_sgindex_from_all_requests (true and false)

srgbnd · August 10, 2020, 9:18am

It looks like your alias doesn’t have a filter. Are you sure you created the “kibana_sample_data_flights_alias_0” alias with the filter I used in my example? This one:

POST /_aliases
{
  "actions": [
    {
      "add": {
        "index": "kibana_sample_data_flights",
        "alias": "kibana_sample_data_flights_alias_0",
        "filter": { "term": { "Carrier": "Kibana Airlines" } }
      }
    }
  ]
}

If I create the alias like that and then execute the search

GET kibana_sample_data_flights_alias_0/_search
{
  "size" : 0,
  "query": {
    "bool": {
      "filter": [
        { "match_all": {} }
      ]
    }
  }
}

the number of values is less than your number

{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 3234,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}

Lucas_Bremgartner · August 10, 2020, 6:10pm

Yes, this is exactly what I did. I just double checked and the result is the same. I also executed:

GET /kibana_sample_data_flights/_alias

and I get the following response:

{
  "kibana_sample_data_flights" : {
    "aliases" : {
      "kibana_sample_data_flights_alias_0" : {
        "filter" : {
          "term" : {
            "Carrier" : "Kibana Airlines"
          }
        }
      }
    }
  }
}

So this clearly tells me, that the term-filter is there on the alias.

srgbnd · August 11, 2020, 11:19am

This is weird. It works in my setup but doesn’t work for you. Do you see any error in the Elasticearch log when POST the filtered alias or do a search on this alias?

Send the following config files:

elasticsearch.yml
sg_config.yml

Also, show your cluster info:

curl -k -u admin:admin -X GET https://localhost:9200/_cluster/stats?human
curl -k -u admin:admin -X GET https://localhost:9200/_cluster/pending_tasks

jkressin · August 12, 2020, 2:19pm

Hi Lucas,

I am also unable to reproduce the issue, and all unit- and integration-tests for filtered aliases are working as expected, both for 6.8.9 and 7.8.0. So we need to check for differences in your setup and ours. Can you please share the configuration files Sergii has requested? We can also take this to the support portal.

Lucas_Bremgartner · August 13, 2020, 5:47am

Hi Jochen,

Thanks for your response. I got the access for the support portal and we decided internally to move this case there. I will share the requested details in the support portal.

xdzhou12 · August 16, 2020, 3:48pm

Just add my two cents here and we suddenly see the same issue today. We have elasticsearch 6.5.4 with opendistro version. The alias with filter was working fine until recently. One of the settings we changed is “opendistro_security.dynamic.do_not_fail_on_forbidden”, so the filter works before this setting is false. We turn it to true a few days ago and now we hear filter not working.

After some digging around, we turned the setting back to false, and filter works again. It seems the query sent to elasticsearch will remove this filter condition. That is what we observed, maybe we should look deeper into the source code if this is related.

srgbnd · August 17, 2020, 7:59am

Hi @xdzhou12. Thank you very much for the feedback.

Lucas_Bremgartner · August 20, 2020, 5:47am

@xdzhou12 Thanks for your feedback. You hit the nail, this setting has been the issue in our case ase well.

system · September 10, 2020, 5:47am

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.