Followed instructions here Upgrading from 6.x to 7.x | Security for Elasticsearch | Search Guard to upgrade from 6.8 to 7.3. The local admin user was able to login and see the Kibana dashboard, etc. The LDAP user from the authorized group (Elk-Admins) was presented with a 403 error in json format in plain text. I was able to move past this by modifying the sg_roles.yml to what is below.
Now, the admin is presented with a 403 when attempting to save changes within Advanced Settings in the Kibana dashboard. It looks like two tenants were created during the upgrade process. According to the log message, the tenant SGS_GLOBAL_TENANT is the issue.
Tenant SGS_GLOBAL_TENANT is not allowed to write (user: LDAP User)
Initially, the authinfo looked like this:
"sg_tenants" : {
"LDAP User" : true,
"SGS_GLOBAL_TENANT" : false,
"admin_tenant" : false
but, I was able to add admin_tenant to the role and that resulted in:
"sg_tenants" : {
"LDAP User" : true,
"SGS_GLOBAL_TENANT" : false,
"admin_tenant" : true
Unfortunately, because the error is related to SGS_GLOBAL_TENANT, it doesn’t change the issue that I am experiencing. Debug log from elasticsearch when the 403:
[2019-08-06T09:40:03,243][DEBUG][c.f.s.p.PrivilegesEvaluator] [hostname] sgr: [sg_own_index, Elk-Admins]
[2019-08-06T09:40:03,243][DEBUG][c.f.s.c.PrivilegesInterceptorImpl] [hostname] raw requestedTenant: 'null'
[2019-08-06T09:40:03,243][DEBUG][c.f.s.c.PrivilegesInterceptorImpl] [hostname] request class org.elasticsearch.action.update.UpdateRequest
[2019-08-06T09:40:03,243][WARN ][c.f.s.c.PrivilegesInterceptorImpl] [hostname] Tenant SGS_GLOBAL_TENANT is not allowed to write (user: LDAP User)
I’m hoping someone can give direction on this one rather than having to go by trial and error. Relevant configuration files below:
Search Guard configuration files
sg_action_groups.yml:
---
_sg_meta:
type: "actiongroups"
config_version: 2
CLUSTER_ALL:
reserved: true
hidden: false
allowed_actions:
- "cluster:*"
type: "cluster"
description: "Migrated from v6"
static: false
ALL:
reserved: true
hidden: false
allowed_actions:
- "INDICES_ALL"
type: "index"
description: "Migrated from v6"
static: false
CRUD:
reserved: true
hidden: false
allowed_actions:
- "READ"
- "WRITE"
type: "index"
description: "Migrated from v6"
static: false
SEARCH:
reserved: true
hidden: false
allowed_actions:
- "indices:data/read/search*"
- "indices:data/read/msearch*"
- "SUGGEST"
type: "index"
description: "Migrated from v6"
static: false
MONITOR:
reserved: true
hidden: false
allowed_actions:
- "INDICES_MONITOR"
type: "index"
description: "Migrated from v6"
static: false
DATA_ACCESS:
reserved: true
hidden: false
allowed_actions:
- "indices:data/*"
- "CRUD"
type: "index"
description: "Migrated from v6"
static: false
CREATE_INDEX:
reserved: true
hidden: false
allowed_actions:
- "indices:admin/create"
- "indices:admin/mapping/put"
type: "index"
description: "Migrated from v6"
static: false
WRITE:
reserved: true
hidden: false
allowed_actions:
- "indices:data/write*"
- "indices:admin/mapping/put"
type: "index"
description: "Migrated from v6"
static: false
MANAGE_ALIASES:
reserved: true
hidden: false
allowed_actions:
- "indices:admin/aliases*"
type: "index"
description: "Migrated from v6"
static: false
READ:
reserved: true
hidden: false
allowed_actions:
- "indices:data/read*"
- "indices:admin/mappings/fields/get*"
type: "index"
description: "Migrated from v6"
static: false
INDICES_ALL:
reserved: true
hidden: false
allowed_actions:
- "indices:*"
type: "index"
description: "Migrated from v6"
static: false
DELETE:
reserved: true
hidden: false
allowed_actions:
- "indices:data/write/delete*"
type: "index"
description: "Migrated from v6"
static: false
CLUSTER_COMPOSITE_OPS:
reserved: true
hidden: false
allowed_actions:
- "indices:data/write/bulk"
- "indices:admin/aliases*"
- "indices:data/write/reindex"
- "CLUSTER_COMPOSITE_OPS_RO"
type: "cluster"
description: "Migrated from v6"
static: false
CLUSTER_COMPOSITE_OPS_RO:
reserved: true
hidden: false
allowed_actions:
- "indices:data/read/mget"
- "indices:data/read/msearch"
- "indices:data/read/mtv"
- "indices:data/read/coordinate-msearch*"
- "indices:admin/aliases/exists*"
- "indices:admin/aliases/get*"
- "indices:data/read/scroll"
type: "cluster"
description: "Migrated from v6"
static: false
GET:
reserved: true
hidden: false
allowed_actions:
- "indices:data/read/get*"
- "indices:data/read/mget*"
type: "index"
description: "Migrated from v6"
static: false
MANAGE:
reserved: true
hidden: false
allowed_actions:
- "indices:monitor/*"
- "indices:admin/*"
type: "index"
description: "Migrated from v6"
static: false
CLUSTER_MONITOR:
reserved: true
hidden: false
allowed_actions:
- "cluster:monitor/*"
type: "cluster"
description: "Migrated from v6"
static: false
MANAGE_SNAPSHOTS:
reserved: true
hidden: false
allowed_actions:
- "cluster:admin/snapshot/*"
- "cluster:admin/repository/*"
type: "index"
description: "Migrated from v6"
static: false
INDEX:
reserved: true
hidden: false
allowed_actions:
- "indices:data/write/index*"
- "indices:data/write/update*"
- "indices:admin/mapping/put"
- "indices:data/write/bulk*"
type: "index"
description: "Migrated from v6"
static: false
UNLIMITED:
reserved: true
hidden: false
allowed_actions:
- "*"
type: "index"
description: "Migrated from v6"
static: false
INDICES_MONITOR:
reserved: true
hidden: false
allowed_actions:
- "indices:monitor/*"
type: "index"
description: "Migrated from v6"
static: false
SUGGEST:
reserved: true
hidden: false
allowed_actions:
- "indices:data/read/suggest*"
type: "index"
description: "Migrated from v6"
static: false
sg_roles_mapping.yml:
---
_sg_meta:
type: "rolesmapping"
config_version: 2
Elk-Users:
reserved: false
hidden: false
backend_roles:
- "kibanauser"
- "readall"
hosts: []
users: []
and_backend_roles: []
description: "Migrated from v6"
sg_all_access:
reserved: true
hidden: false
backend_roles:
- "admin"
hosts: []
users: []
and_backend_roles: []
description: "Migrated from v6"
sg_own_index:
reserved: false
hidden: false
backend_roles: []
hosts: []
users:
- "*"
and_backend_roles: []
description: "Migrated from v6"
sg_logstash:
reserved: false
hidden: false
backend_roles:
- "logstash"
hosts: []
users: []
and_backend_roles: []
description: "Migrated from v6"
Elk-Admins:
reserved: true
hidden: false
backend_roles:
- "admin"
hosts: []
users: []
and_backend_roles: []
description: "Migrated from v6"
sg_kibana_user:
reserved: false
hidden: false
backend_roles:
- "kibanauser"
hosts: []
users: []
and_backend_roles: []
description: "Migrated from v6"
sg_readall:
reserved: true
hidden: false
backend_roles:
- "readall"
hosts: []
users: []
and_backend_roles: []
description: "Migrated from v6"
sg_manage_snapshots:
reserved: true
hidden: false
backend_roles:
- "snapshotrestore"
hosts: []
users: []
and_backend_roles: []
description: "Migrated from v6"
sg_kibana_server:
reserved: true
hidden: false
backend_roles: []
hosts: []
users:
- "kibanaserver"
and_backend_roles: []
description: "Migrated from v6"
sg_roles.yml:
---
_sg_meta:
type: "roles"
config_version: 2
Elk-Users:
reserved: true
hidden: false
description: "Migrated from v6 (all types mapped)"
cluster_permissions:
- "INDICES_MONITOR"
- "CLUSTER_COMPOSITE_OPS"
index_permissions:
- index_patterns:
- "?kibana"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "READ"
- "DELETE"
- "MANAGE"
- "INDEX"
- index_patterns:
- "?kibana-6"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "READ"
- "DELETE"
- "MANAGE"
- "INDEX"
- index_patterns:
- "*"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "indices:data/read/*"
tenant_permissions: []
static: false
sg_own_index:
reserved: false
hidden: false
description: "Migrated from v6 (all types mapped)"
cluster_permissions:
- "CLUSTER_COMPOSITE_OPS"
index_permissions:
- index_patterns:
- "${user_name}"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "INDICES_ALL"
tenant_permissions: []
static: false
sg_xp_monitoring:
reserved: true
hidden: false
description: "Migrated from v6 (all types mapped)"
cluster_permissions:
- "cluster:monitor/xpack/info"
- "cluster:monitor/main"
- "cluster:admin/xpack/monitoring/bulk"
index_permissions:
- index_patterns:
- "?monitor*"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "INDICES_ALL"
tenant_permissions: []
static: false
sg_kibana_user:
reserved: true
hidden: false
description: "Migrated from v6 (all types mapped)"
cluster_permissions:
- "INDICES_MONITOR"
- "CLUSTER_COMPOSITE_OPS"
index_permissions:
- index_patterns:
- "?kibana"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "READ"
- "DELETE"
- "MANAGE"
- "INDEX"
- index_patterns:
- "?kibana-6"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "READ"
- "DELETE"
- "MANAGE"
- "INDEX"
- index_patterns:
- "?kibana_*"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "READ"
- "DELETE"
- "MANAGE"
- "INDEX"
- index_patterns:
- "?tasks"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "INDICES_ALL"
- index_patterns:
- "?management-beats"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "INDICES_ALL"
- index_patterns:
- "*"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "indices:data/read/xpack/rollup*"
- "indices:admin/mappings/get*"
- "indices:admin/get"
- "indices:data/read/field_caps*"
- index_patterns:
- "?.kibana_*"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "INDICES_ALL"
tenant_permissions: []
static: false
sg_manage_snapshots:
reserved: false
hidden: false
description: "Migrated from v6 (all types mapped)"
cluster_permissions:
- "MANAGE_SNAPSHOTS"
index_permissions:
- index_patterns:
- "*"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "indices:data/write/index"
- "indices:admin/create"
tenant_permissions: []
static: false
sg_xp_alerting:
reserved: true
hidden: false
description: "Migrated from v6 (all types mapped)"
cluster_permissions:
- "indices:data/read/scroll"
- "cluster:admin/xpack/watcher*"
- "cluster:monitor/xpack/watcher*"
index_permissions:
- index_patterns:
- "?watches*"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "INDICES_ALL"
- index_patterns:
- "?watcher-history-*"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "INDICES_ALL"
- index_patterns:
- "?triggered_watches"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "INDICES_ALL"
- index_patterns:
- "*"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "READ"
- "indices:admin/aliases/get"
tenant_permissions: []
static: false
sg_xp_machine_learning:
reserved: true
hidden: false
description: "Migrated from v6 (all types mapped)"
cluster_permissions:
- "cluster:admin/persistent*"
- "cluster:internal/xpack/ml*"
- "indices:data/read/scroll*"
- "cluster:admin/xpack/ml*"
- "cluster:monitor/xpack/ml*"
index_permissions:
- index_patterns:
- "*"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "READ"
- "indices:admin/get*"
- index_patterns:
- "?ml-*"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "*"
tenant_permissions: []
static: false
sg_kibana_server:
reserved: true
hidden: false
description: "Migrated from v6 (all types mapped)"
cluster_permissions:
- "CLUSTER_MONITOR"
- "CLUSTER_COMPOSITE_OPS"
- "cluster:admin/xpack/monitoring*"
- "indices:admin/template*"
- "indices:data/read/scroll*"
index_permissions:
- index_patterns:
- "?kibana"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "INDICES_ALL"
- index_patterns:
- "?kibana-6"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "INDICES_ALL"
- index_patterns:
- "?kibana_*"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "INDICES_ALL"
- "CREATE_INDEX"
- index_patterns:
- "?reporting*"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "INDICES_ALL"
- index_patterns:
- "?monitoring*"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "INDICES_ALL"
- index_patterns:
- "?tasks"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "INDICES_ALL"
- index_patterns:
- "?management-beats*"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "INDICES_ALL"
- index_patterns:
- "*"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "indices:admin/aliases*"
tenant_permissions: []
static: false
sg_all_access:
reserved: true
hidden: false
description: "Migrated from v6 (all types mapped)"
cluster_permissions:
- "UNLIMITED"
index_permissions:
- index_patterns:
- "*"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "UNLIMITED"
tenant_permissions:
- tenant_patterns:
- "admin_tenant"
allowed_actions:
- "SGS_KIBANA_ALL_WRITE"
static: false
sg_logstash:
reserved: false
hidden: false
description: "Migrated from v6 (all types mapped)"
cluster_permissions:
- "CLUSTER_MONITOR"
- "CLUSTER_COMPOSITE_OPS"
- "indices:admin/template/get"
- "indices:admin/template/put"
index_permissions:
- index_patterns:
- "*"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "CRUD"
- "CREATE_INDEX"
- index_patterns:
- "*beat*"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "CRUD"
- "CREATE_INDEX"
tenant_permissions: []
static: false
Elk-Admins:
reserved: true
hidden: false
description: "Migrated from v6 (all types mapped)"
cluster_permissions:
- "UNLIMITED"
index_permissions:
- index_patterns:
- "*"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "UNLIMITED"
tenant_permissions:
- tenant_patterns:
- "*"
allowed_actions:
- "UNLIMITED"
- tenant_patterns:
- "admin_tenant"
allowed_actions:
- "SGS_KIBANA_ALL_WRITE"
static: false
sg_readall_and_monitor:
reserved: false
hidden: false
description: "Migrated from v6 (all types mapped)"
cluster_permissions:
- "CLUSTER_MONITOR"
- "CLUSTER_COMPOSITE_OPS_RO"
index_permissions:
- index_patterns:
- "*"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "READ"
tenant_permissions: []
static: false
sg_readall:
reserved: true
hidden: false
description: "Migrated from v6 (all types mapped)"
cluster_permissions:
- "CLUSTER_COMPOSITE_OPS_RO"
index_permissions:
- index_patterns:
- "*"
dls: null
fls: null
masked_fields: null
allowed_actions:
- "READ"
tenant_permissions: []
static: false
elasticsearch.yml configuration file
elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: ELK
cluster.max_shards_per_node: 20000
cluster.initial_master_nodes:
- host1
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: ${HOSTNAME}
node.attr.box_type: hot
node.master: true
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /home/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# --------------------------------- Processor ----------------------------------
thread_pool:
warmer:
core: 1
max: 6
keep_alive: 2m
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: host1
#network.host: localhost
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#discovery.zen.ping.multicast.enabled: false
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes: 3
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
#xpack.security.enabled: false
#xpack.security.authc:
# anonymous:
# username: anonymous_user
# roles: role1, role2
# authz_exception: true
searchguard.allow_unsafe_democertificates: false
searchguard.audit.type: internal_elasticsearch
searchguard.enable_snapshot_restore_privilege: true
searchguard.check_snapshot_restore_write_privileges: true
searchguard.restapi.roles_enabled: ["sg_all_access","Elk-Admins"]
cluster.routing.allocation.disk.threshold_enabled: false
discovery.zen.minimum_master_nodes: 1
node.max_local_storage_nodes: 3
xpack.security.enabled: false
..SG LDAP SETTINGS SCRUBBED..
#Enable Enterprise features
searchguard.enterprise_modules_enabled: true
#Map LDAP roles to SG
searchguard.roles_mapping_resolution: BOTH
kibana.yml configuration file:
kibana.yml:
searchguard.readonly_mode.roles: ["sg_kibana_user", "Elk-Users"]
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601
# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "host1"
# Enables you to specify a path to mount Kibana at if you are running behind a proxy. This only affects
# the URLs generated by Kibana, your proxy is expected to remove the basePath value before forwarding requests
# to Kibana. This setting cannot end in a slash.
#server.basePath: ""
# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576
# The Kibana server's name. This is used for display purposes.
#server.name: "your-hostname"
# The URL of the Elasticsearch instance to use for all your queries.
# Use HTTPS instead of HTTP
elasticsearch.hosts: "https://host1:9200"
# When this setting's value is true Kibana uses the hostname specified in the server.host
# setting. When the value of this setting is false, Kibana uses the hostname of the host
# that connects to this Kibana instance.
#elasticsearch.preserveHost: true
# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
kibana.index: ".kibana"
# The default application to load.
#kibana.defaultAppId: "discover"
# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
# Configure the Kibana internal server user
elasticsearch.username: "kibanaserver"
elasticsearch.password: "kibanaserver"
# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key
# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files validate that your Elasticsearch backend uses the same key files.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key
# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]
# To disregard the validity of SSL certificates, change this setting's value to 'none'.
# Disable SSL verification because we use self-signed demo certificates
elasticsearch.ssl.verificationMode: none
# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500
# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
elasticsearch.requestTimeout: 60000
# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
# Whitelist the Search Guard Multi Tenancy Header
elasticsearch.requestHeadersWhitelist: [ "Authorization", "sgtenant" ]
# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}
# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 0
# Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.
#elasticsearch.startupTimeout: 5000
# Specifies the path where Kibana creates the process ID file.
#pid.file: /var/run/kibana.pid
# Enables you specify a file where Kibana stores log output.
logging.dest: /var/log/kibana.log
# Set the value of this setting to true to suppress all logging output.
#logging.silent: false
# Set the value of this setting to true to suppress all logging output other than error messages.
#logging.quiet: false
# Set the value of this setting to true to log all events, including system usage information
# and all requests.
#logging.verbose: false
# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000
# The default locale. This locale can be used in certain circumstances to substitute any missing
# translations.
#i18n.defaultLocale: "en"
xpack.security.enabled: false
