To be honest updating ELK Stack with searchguard with each version is getting more irritating and troublesome instead of getting easier. Yet again errors after update that I have no idea how to fix. Hopefully someone can help and tell me what is going on here. Updating cluster was yet again not possible because sgadmin update does not work with red clusters even if forced so I tried to do it with 1 node setup as below. result is that elasticsearch throws SSL errors, cannot connect to other nodes in cluster and sometimes crashes. Kibana tires to lauch but crases every single time because of permissions issues.
ELK 6.5.1 | SG 6.5.1-23.2 & 6.5.1-16
sg_roles:
Allows everything, but no changes to searchguard configuration index
sg_all_access:
readonly: true
cluster:
- UNLIMITED
indices:
‘':
'’:
- UNLIMITED
tenants:
admin_tenant: RW
Read all, but no write permissions
sg_readall:
readonly: true
cluster:
- CLUSTER_COMPOSITE_OPS_RO
indices:
‘':
'’:
- READ
Read all and monitor, but no write permissions
sg_readall_and_monitor:
cluster:
- CLUSTER_MONITOR
- CLUSTER_COMPOSITE_OPS_RO
indices:
‘':
'’:
- READ
For users which use kibana, access to indices must be granted separately
sg_kibana_user:
readonly: true
cluster:
- INDICES_MONITOR
- CLUSTER_COMPOSITE_OPS
indices:
‘?kibana’:
‘':
- MANAGE
- INDEX
- READ
- DELETE
‘?kibana-6’:
'’:
- MANAGE
- INDEX
- READ
- DELETE
‘':
'’:
- indices:data/read/field_caps*
For the kibana server
sg_kibana_server:
readonly: true
cluster:
- CLUSTER_MONITOR
- CLUSTER_COMPOSITE_OPS
- cluster:admin/xpack/monitoring*
- indices:admin/template*
indices:
‘?kibana’:
‘':
- INDICES_ALL
‘?kibana-6’:
'’:
- INDICES_ALL
‘?reporting*’:
‘':
- INDICES_ALL
'?monitoring’:
‘*’:
- INDICES_ALL
For logstash and beats
sg_logstash:
cluster:
- CLUSTER_MONITOR
- CLUSTER_COMPOSITE_OPS
- indices:admin/template/get
- indices:admin/template/put
indices:
‘logstash-':
'’:
- CRUD
- CREATE_INDEX
‘beat’:
‘*’:
- CRUD
- CREATE_INDEX
Allows adding and modifying repositories and creating and restoring snapshots
sg_manage_snapshots:
cluster:
- MANAGE_SNAPSHOTS
indices:
‘':
'’:
- “indices:data/write/index”
- “indices:admin/create”
Allows each user to access own named index
sg_own_index:
cluster:
- CLUSTER_COMPOSITE_OPS
indices:
‘${user_name}’:
‘*’:
- INDICES_ALL
``
elasticsearch config:
···
Default Elasticsearch configuration from elasticsearch-docker.
from https://github.com/elastic/elasticsearch-docker/blob/master/build/elasticsearch/elasticsearch.yml
#cluster.name: “some-cluster”
network.host: 0.0.0.0
node.name: “client1”
#network.publish_host: 172.X.X.102
minimum_master_nodes need to be explicitly set when bound on a public IP
set to 1 to allow single node clusters
Details: Enforce `discovery.zen.minimum_master_nodes` is set when bound to a public ip by bleskes · Pull Request #17288 · elastic/elasticsearch · GitHub
#discovery.zen.minimum_master_nodes: 1
#discovery.zen.ping.unicast.hosts: [“172.XX.XX.XX”,“172.XX.XX.XX”,“172.XX.XX.XX”]
Use single node discovery in order to disable production mode and avoid bootstrap checks
see Bootstrap Checks | Elasticsearch Guide [8.4] | Elastic
discovery.type: single-node
Search Guard
#xpack.security.enabled: false // if enabled throws erros in SG6.5.1 worked fine in older version
searchguard.enterprise_modules_enabled: false
searchguard.ssl.transport.pemcert_filepath: sg/client1.pem
searchguard.ssl.transport.pemkey_filepath: sg/client1.key
searchguard.ssl.transport.pemkey_password: XXXXXXXXX
searchguard.ssl.transport.pemtrustedcas_filepath: sg/root-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: false
searchguard.nodes_dn:
- CN=client1,OU=sigh,O=blabla,DC=blabla,DC=com
- CN=client2,OU=sigh,O=blabla,DC=blabla,DC=com
- CN=client3,OU=sigh,O=blabla,DC=blabla,DC=com
searchguard.authcz.admin_dn: - CN=admin,OU=sigh,O=blabla,DC=blabla,DC=com
- CN=admin2,OU=sigh,O=blabla,DC=blabla,DC=com
``
elasticsearch logs:
Kibana logs:
issue: [security_exception] no permissions for [indices:admin/create] , this also appears for other users and was working fine in older SG versions.
Unavailable",“prevState”:“red”,“prevMsg”:“Unable to connect to Elasticsearch at http://elasticsearch:9200/.”}
{“type”:“log”,“@timestamp”:“2018-12-03T12:53:41Z”,“tags”:[“status”,“plugin:elasticsearch@6.5.1”,“info”],“pid”:1,“state”:“green”,“message”:“Status changed from red to green - Ready”,“prevState”:“red”,“prevMsg”:“Service Unavailable”}
{“type”:“log”,“@timestamp”:“2018-12-03T12:53:41Z”,“tags”:[“info”,“migrations”],“pid”:1,“message”:“Creating index .kibana_2.”}
{“type”:“error”,“@timestamp”:“2018-12-03T12:53:41Z”,“tags”:[“fatal”,“root”],“pid”:1,“level”:“fatal”,“error”:{“message”:“[security_exception] no permissions for [indices:admin/create] and User [name=kibanaserver, roles=, requestedTenant=null]”,“name”:“Error”,“stack”:“[security_exception] no permissions for [indices:admin/create] and User [name=kibanaserver, roles=, requestedTenant=null] :: {"path":"/.kibana_2","query":{},"body":"{\"mappings\":{\"doc\":{\"dynamic\":\"strict\",\"properties\":{\"config\":{\"dynamic\":\"true\",\"properties\":{\"buildNum\":{\"type\":\"keyword\"}}},\"dashboard\":{\"properties\":{\"description\":{\"type\":\"text\"},\"hits\":{\"type\":\"integer\"},\"kibanaSavedObjectMeta\":{\"properties\":{\"searchSourceJSON\":{\"type\":\"text\"}}},\"optionsJSON\":{\"type\":\"text\"},\"panelsJSON\":{\"type\":\"text\"},\"refreshInterval\":{\"properties\":{\"display\":{\"type\":\"keyword\"},\"pause\":{\"type\":\"boolean\"},\"section\":{\"type\":\"integer\"},\"value\":{\"type\":\"integer\"}}},\"timeFrom\":{\"type\":\"keyword\"},\"timeRestore\":{\"type\":\"boolean\"},\"timeTo\":{\"type\":\"keyword\"},\"title\":{\"type\":\"text\"},\"uiStateJSON\":{\"type\":\"text\"},\"version\":{\"type\":\"integer\"}}},\"graph-workspace\":{\"properties\":{\"description\":{\"type\":\"text\"},\"kibanaSavedObjectMeta\":{\"properties\":{\"searchSourceJSON\":{\"type\":\"text\"}}},\"numLinks\":{\"type\":\"integer\"},\"numVertices\":{\"type\":\"integer\"},\"title\":{\"type\":\"text\"},\"version\":{\"type\":\"integer\"},\"wsState\":{\"type\":\"text\"}}},\"index-pattern\":{\"properties\":{\"fieldFormatMap\":{\"type\":\"text\"},\"fields\":{\"type\":\"text\"},\"intervalName\":{\"type\":\"keyword\"},\"notExpandable\":{\"type\":\"boolean\"},\"sourceFilters\":{\"type\":\"text\"},\"timeFieldName\":{\"type\":\"keyword\"},\"title\":{\"type\":\"text\"},\"type\":{\"type\":\"keyword\"},\"typeMeta\":{\"type\":\"keyword\"}}},\"search\":{\"properties\":{\"columns\":{\"type\":\"keyword\"},\"description\":{\"type\":\"text\"},\"hits\":{\"type\":\"integer\"},\"kibanaSavedObjectMeta\":{\"properties\":{\"searchSourceJSON\":{\"type\":\"text\"}}},\"sort\":{\"type\":\"keyword\"},\"title\":{\"type\":\"text\"},\"version\":{\"type\":\"integer\"}}},\"server\":{\"properties\":{\"uuid\":{\"type\":\"keyword\"}}},\"timelion-sheet\":{\"properties\":{\"description\":{\"type\":\"text\"},\"hits\":{\"type\":\"integer\"},\"kibanaSavedObjectMeta\":{\"properties\":{\"searchSourceJSON\":{\"type\":\"text\"}}},\"timelion_chart_height\":{\"type\":\"integer\"},\"timelion_columns\":{\"type\":\"integer\"},\"timelion_interval\":{\"type\":\"keyword\"},\"timelion_other_interval\":{\"type\":\"keyword\"},\"timelion_rows\":{\"type\":\"integer\"},\"timelion_sheet\":{\"type\":\"text\"},\"title\":{\"type\":\"text\"},\"version\":{\"type\":\"integer\"}}},\"type\":{\"type\":\"keyword\"},\"updated_at\":{\"type\":\"date\"},\"url\":{\"properties\":{\"accessCount\":{\"type\":\"long\"},\"accessDate\":{\"type\":\"date\"},\"createDate\":{\"type\":\"date\"},\"url\":{\"type\":\"text\",\"fields\":{\"keyword\":{\"type\":\"keyword\",\"ignore_above\":2048}}}}},\"visualization\":{\"properties\":{\"description\":{\"type\":\"text\"},\"kibanaSavedObjectMeta\":{\"properties\":{\"searchSourceJSON\":{\"type\":\"text\"}}},\"savedSearchId\":{\"type\":\"keyword\"},\"title\":{\"type\":\"text\"},\"uiStateJSON\":{\"type\":\"text\"},\"version\":{\"type\":\"integer\"},\"visState\":{\"type\":\"text\"}}},\"migrationVersion\":{\"dynamic\":\"true\",\"type\":\"object\"},\"namespace\":{\"type\":\"keyword\"},\"kql-telemetry\":{\"properties\":{\"optInCount\":{\"type\":\"long\"},\"optOutCount\":{\"type\":\"long\"}}}}}},\"settings\":{\"number_of_shards\":1,\"auto_expand_replicas\":\"0-1\"}}","statusCode":403,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"no permissions for [indices:admin/create] and User [name=kibanaserver, roles=, requestedTenant=null]\"}],\"type\":\"security_exception\",\"reason\":\"no permissions for [indices:admin/create] and User [name=kibanaserver, roles=, requestedTenant=null]\"},\"status\":403}"}\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:308:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:267:7)\n at HttpConnector. (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:165:7)\n at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4949:19)\n at emitNone (events.js:111:20)\n at IncomingMessage.emit (events.js:208:7)\n at endReadableNT (_stream_readable.js:1064:12)\n at _combinedTickCallback (internal/process/next_tick.js:138:11)\n at process._tickCallback (internal/process/next_tick.js:180:9)”},“message”:“[security_exception] no permissions for [indices:admin/create] and User [name=kibanaserver, roles=, requestedTenant=null]”}
``
elasticsearch:
issue: SSL Problem Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16), was not occuring in older version of SG
[2018-12-03T15:17:21,471][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [client1] SSL Problem Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
at sun.security.ssl.Alert.createSSLException(Alert.java:128) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:321) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:264) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:259) ~[?:?]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:129) ~[?:?]
at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:672) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:627) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:443) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:422) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:634) ~[?:?]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:294) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1297) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-common-4.1.30.Final.jar:4.1.30.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: javax.crypto.BadPaddingException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
at sun.security.ssl.SSLCipher$T13GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1852) ~[?:?]
at sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:240) ~[?:?]
at sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:197) ~[?:?]
at sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:160) ~[?:?]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:108) ~[?:?]
… 26 more
``