The Search Guard license information could not be loaded. Please contact your system administrator.

Search Guard
1

Hello when i connect to my kibana interface, i have this message : The Search Guard license information could not be loaded. Please contact your system administrator.

I have 3 nodes, (1 master, 2 slaves) all my config run with the ELK suite version 6.3.2, it’s the same for searchguard. Both are running as community edition (free - no paying module).

All thoose software are running on debian 8.

The error message is different when i see the Kibana log.

Kibana Log :

[“license”,“warning”,“xpack”],“pid”:28818,“message”:“License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. Error: No Living connections”}

[“warning”,“elasticsearch”,“data”],“pid”:28818,“message”:“No living connections”}

[“warning”,“elasticsearch”,“data”],“pid”:28818,“message”:“Unable to revive connection: https://xxxxxxxxxxxxx:9200/”}

``

Thanks for help and sorry for my bad english.

2

The underlying problem here is that Kibana is either not able to connect to Elasticsearch, or the configured kibanaserver user is not able to authenticate (wrong username / password).

[“warning”,“elasticsearch”,“data”],“pid”:28818,“message”:“No living connections”}

[“warning”,“elasticsearch”,“data”],“pid”:28818,“message”:“Unable to revive connection: https://xxxxxxxxxxxxx:9200/”}

The “no living connections” indicates the connection problem.

Can you please post your kibana.yml and also the Elasticsearch logfiles?

If Kibana is generally able to connect to Elasticsearch, but there are problems regarding the Kibana server user or other issues like TLS certificate problems, you should see messages in the ES logs.

···

On Tuesday, December 4, 2018 at 11:28:02 AM UTC+1, Voortexx wrote:

Hello when i connect to my kibana interface, i have this message : The Search Guard license information could not be loaded. Please contact your system administrator.

I have 3 nodes, (1 master, 2 slaves) all my config run with the ELK suite version 6.3.2, it’s the same for searchguard. Both are running as community edition (free - no paying module).

All thoose software are running on debian 8.

The error message is different when i see the Kibana log.

Kibana Log :

[“license”,“warning”,“xpack”],“pid”:28818,“message”:“License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. Error: No Living connections”}

[“warning”,“elasticsearch”,“data”],“pid”:28818,“message”:“No living connections”}

[“warning”,“elasticsearch”,“data”],“pid”:28818,“message”:“Unable to revive connection: https://xxxxxxxxxxxxx:9200/”}

``

Thanks for help and sorry for my bad english.

3

Hello Jochen, Thanks for replying.

Here is my kibana.yml :

Kibana is served by a back end server. This setting specifies the port to use.

server.port: 5601

Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.

The default is ‘localhost’, which usually means remote machines will not be able to connect.

To allow connections from remote users, set this parameter to a non-loopback address.

server.host: “10.15.20.160”

Enables you to specify a path to mount Kibana at if you are running behind a proxy.

Use the server.rewriteBasePath setting to tell Kibana if it should remove the basePath

from requests it receives, and to prevent a deprecation warning at startup.

This setting cannot end in a slash.

#server.basePath: “”

Specifies whether Kibana should rewrite requests that are prefixed with

server.basePath or require that they are rewritten by your reverse proxy.

This setting was effectively always false before Kibana 6.3 and will

default to true starting in Kibana 7.0.

#server.rewriteBasePath: false

The maximum payload size in bytes for incoming server requests.

#server.maxPayloadBytes: 1048576

The Kibana server’s name. This is used for display purposes.

#server.name: “your-hostname”

The URL of the Elasticsearch instance to use for all your queries.

elasticsearch.url: “https://xxxxxxxxxx:9200”

#elasticsearch.ssl.certificateAuthorities: [ “root-ca.cer” ]

#elasticsearch.ssl.verify: certificate

elasticsearch.ssl.verificationMode: none

When this setting’s value is true Kibana uses the hostname specified in the server.host

setting. When the value of this setting is false, Kibana uses the hostname of the host

that connects to this Kibana instance.

#elasticsearch.preserveHost: true

Kibana uses an index in Elasticsearch to store saved searches, visualizations and

dashboards. Kibana creates a new index if the index doesn’t already exist.

#kibana.index: “.kibana”

The default application to load.

#kibana.defaultAppId: “home”

If your Elasticsearch is protected with basic authentication, these settings provide

the username and password that the Kibana server uses to perform maintenance on the Kibana

index at startup. Your Kibana users still need to authenticate with Elasticsearch, which

is proxied through the Kibana server.

elasticsearch.username: “kibanaserver”

elasticsearch.password: “kibanaserver”

Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.

These settings enable SSL for outgoing requests from the Kibana server to the browser.

#server.ssl.enabled: true

#server.ssl.certificate: /path/to/your/server.crt

#server.ssl.key: /path/to/your/server.key

Optional settings that provide the paths to the PEM-format SSL certificate and key files.

These files validate that your Elasticsearch backend uses the same key files.

#elasticsearch.ssl.certificate: /path/to/your/client.crt

#elasticsearch.ssl.key: /path/to/your/client.key

Optional setting that enables you to specify a path to the PEM file for the certificate

authority for your Elasticsearch instance.

#elasticsearch.ssl.certificateAuthorities: [ “/path/to/your/CA.pem” ]

To disregard the validity of SSL certificates, change this setting’s value to ‘none’.

#elasticsearch.ssl.verificationMode: full

Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of

the elasticsearch.requestTimeout setting.

#elasticsearch.pingTimeout: 1500

Time in milliseconds to wait for responses from the back end or Elasticsearch. This value

must be a positive integer.

#elasticsearch.requestTimeout: 30000

List of Kibana client-side headers to send to Elasticsearch. To send no client-side

headers, set this value to (an empty list).

#elasticsearch.requestHeadersWhitelist: [ authorization ]

Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten

by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.

#elasticsearch.customHeaders: {}

Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.

#elasticsearch.shardTimeout: 30000

Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.

#elasticsearch.startupTimeout: 5000

Logs queries sent to Elasticsearch. Requires logging.verbose set to true.

#elasticsearch.logQueries: false

Specifies the path where Kibana creates the process ID file.

#pid.file: /var/run/kibana.pid

Enables you specify a file where Kibana stores log output.

#logging.dest: stdout

Set the value of this setting to true to suppress all logging output.

#logging.silent: false

Set the value of this setting to true to suppress all logging output other than error messages.

#logging.quiet: false

Set the value of this setting to true to log all events, including system usage information

and all requests.

#logging.verbose: false

Set the interval in milliseconds to sample system and process performance

metrics. Minimum is 100ms. Defaults to 5000.

#ops.interval: 5000

The default locale. This locale can be used in certain circumstances to substitute any missing

translations.

#i18n.defaultLocale: “en”

xpack.security.enabled: false

xpack.monitoring.enabled : false

xpack.graph.enabled : false

xpack.watcher.enabled : false

``

My root certificate is in the kibana file.

Elasticsearch Master Log :

[2018-12-05T09:18:11,609][INFO ][o.e.n.Node ] [Master] stopping …

[2018-12-05T09:18:12,144][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/23176] [Main.cc@148] Ml controller exiting

[2018-12-05T09:18:12,161][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started

[2018-12-05T09:18:12,276][INFO ][o.e.n.Node ] [Master] stopped

[2018-12-05T09:18:12,276][INFO ][o.e.n.Node ] [Master] closing …

[2018-12-05T09:18:12,380][INFO ][o.e.n.Node ] [Master] closed

[2018-12-05T09:18:21,483][INFO ][o.e.n.Node ] [Master] initializing …

[2018-12-05T09:18:21,657][INFO ][o.e.e.NodeEnvironment ] [Master] using [1] data paths, mounts [[/ (/dev/mapper/SDEVFRTLD1–vg-root)]], net usable_space [24.5gb], net total_space [29.8gb], types [ext4]

[2018-12-05T09:18:21,658][INFO ][o.e.e.NodeEnvironment ] [Master] heap size [990.7mb], compressed ordinary object pointers [true]

[2018-12-05T09:18:22,059][INFO ][o.e.n.Node ] [Master] node name [Master], node ID [iK6G-EppTjisNv1gI2M66w]

[2018-12-05T09:18:22,059][INFO ][o.e.n.Node ] [Master] version[6.3.2], pid[32442], build[default/deb/053779d/2018-07-20T05:20:23.451332Z], OS[Linux/4.9.0-8-amd64/amd64], JVM[Oracle Corporation/Java HotSpot™ 64-Bit Server VM/1.8.0_111/25.111-b14]

[2018-12-05T09:18:22,060][INFO ][o.e.n.Node ] [Master] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.vYqhgD90, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb]

[2018-12-05T09:18:26,985][INFO ][c.f.s.SearchGuardPlugin ] ES Config path is /etc/elasticsearch

[2018-12-05T09:18:27,097][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] OpenSSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSL

[2018-12-05T09:18:27,116][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Config directory is /etc/elasticsearch/, from there the key- and truststore files are resolved relatively

[2018-12-05T09:18:27,297][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] AES-256 not supported, max key length for AES is 128 bit. (This is not an issue, it just limits possible encryption strength. To enable AES 256, install ‘Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files’)

[2018-12-05T09:18:27,297][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Client Provider : JDK

[2018-12-05T09:18:27,297][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Server Provider : JDK

[2018-12-05T09:18:27,297][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS HTTP Provider : JDK

[2018-12-05T09:18:27,297][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for transport layer : [TLSv1.2, TLSv1.1]

[2018-12-05T09:18:27,297][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for HTTP layer : [TLSv1.2, TLSv1.1]

[2018-12-05T09:18:27,588][INFO ][c.f.s.SearchGuardPlugin ] Clustername: ClustBase

[2018-12-05T09:18:27,597][WARN ][c.f.s.SearchGuardPlugin ] File /etc/elasticsearch/.elasticsearch.keystore.initial_md5sum has insecure file permissions (should be 0600)

[2018-12-05T09:18:27,598][WARN ][c.f.s.SearchGuardPlugin ] File /etc/elasticsearch/87.cer has insecure file permissions (should be 0600)

[2018-12-05T09:18:27,598][WARN ][c.f.s.SearchGuardPlugin ] File /etc/elasticsearch/sdevfrekham.reyesholdings.com_http.key has insecure file permissions (should be 0600)

[2018-12-05T09:18:27,598][WARN ][c.f.s.SearchGuardPlugin ] File /etc/elasticsearch/root-ca.cer has insecure file permissions (should be 0600)

[2018-12-05T09:18:27,598][WARN ][c.f.s.SearchGuardPlugin ] File /etc/elasticsearch/88.cer has insecure file permissions (should be 0600)

[2018-12-05T09:18:27,598][WARN ][c.f.s.SearchGuardPlugin ] File /etc/elasticsearch/sdevfrekham.reyesholdings.com.key has insecure file permissions (should be 0600)

[2018-12-05T09:18:27,660][INFO ][o.e.p.PluginsService ] [Master] loaded module [aggs-matrix-stats]

[2018-12-05T09:18:27,661][INFO ][o.e.p.PluginsService ] [Master] loaded module [analysis-common]

[2018-12-05T09:18:27,661][INFO ][o.e.p.PluginsService ] [Master] loaded module [ingest-common]

[2018-12-05T09:18:27,661][INFO ][o.e.p.PluginsService ] [Master] loaded module [lang-expression]

[2018-12-05T09:18:27,661][INFO ][o.e.p.PluginsService ] [Master] loaded module [lang-mustache]

[2018-12-05T09:18:27,661][INFO ][o.e.p.PluginsService ] [Master] loaded module [lang-painless]

[2018-12-05T09:18:27,661][INFO ][o.e.p.PluginsService ] [Master] loaded module [mapper-extras]

[2018-12-05T09:18:27,662][INFO ][o.e.p.PluginsService ] [Master] loaded module [parent-join]

[2018-12-05T09:18:27,662][INFO ][o.e.p.PluginsService ] [Master] loaded module [percolator]

[2018-12-05T09:18:27,662][INFO ][o.e.p.PluginsService ] [Master] loaded module [rank-eval]

[2018-12-05T09:18:27,662][INFO ][o.e.p.PluginsService ] [Master] loaded module [reindex]

[2018-12-05T09:18:27,663][INFO ][o.e.p.PluginsService ] [Master] loaded module [repository-url]

[2018-12-05T09:18:27,663][INFO ][o.e.p.PluginsService ] [Master] loaded module [transport-netty4]

[2018-12-05T09:18:27,663][INFO ][o.e.p.PluginsService ] [Master] loaded module [tribe]

[2018-12-05T09:18:27,663][INFO ][o.e.p.PluginsService ] [Master] loaded module [x-pack-core]

[2018-12-05T09:18:27,663][INFO ][o.e.p.PluginsService ] [Master] loaded module [x-pack-deprecation]

[2018-12-05T09:18:27,663][INFO ][o.e.p.PluginsService ] [Master] loaded module [x-pack-graph]

[2018-12-05T09:18:27,663][INFO ][o.e.p.PluginsService ] [Master] loaded module [x-pack-logstash]

[2018-12-05T09:18:27,663][INFO ][o.e.p.PluginsService ] [Master] loaded module [x-pack-ml]

[2018-12-05T09:18:27,663][INFO ][o.e.p.PluginsService ] [Master] loaded module [x-pack-monitoring]

[2018-12-05T09:18:27,663][INFO ][o.e.p.PluginsService ] [Master] loaded module [x-pack-rollup]

[2018-12-05T09:18:27,663][INFO ][o.e.p.PluginsService ] [Master] loaded module [x-pack-security]

[2018-12-05T09:18:27,663][INFO ][o.e.p.PluginsService ] [Master] loaded module [x-pack-sql]

[2018-12-05T09:18:27,663][INFO ][o.e.p.PluginsService ] [Master] loaded module [x-pack-upgrade]

[2018-12-05T09:18:27,663][INFO ][o.e.p.PluginsService ] [Master] loaded module [x-pack-watcher]

[2018-12-05T09:18:27,664][INFO ][o.e.p.PluginsService ] [Master] loaded plugin [search-guard-6]

[2018-12-05T09:18:27,713][INFO ][c.f.s.SearchGuardPlugin ] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting ‘http.compression: true’ in elasticsearch.yml

[2018-12-05T09:18:31,654][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/32508] [Main.cc@109] controller (64 bit): Version 6.3.2 (Build 903094f295d249) Copyright (c) 2018 Elasticsearch BV

[2018-12-05T09:18:32,504][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin com.floragunn.searchguard.SearchGuardPlugin

[2018-12-05T09:18:33,045][INFO ][o.e.d.DiscoveryModule ] [Master] using discovery type [zen]

[2018-12-05T09:18:33,947][INFO ][c.f.s.SearchGuardPlugin ] 0 Search Guard modules loaded so far:

[2018-12-05T09:18:33,948][INFO ][o.e.n.Node ] [Master] initialized

[2018-12-05T09:18:33,948][INFO ][o.e.n.Node ] [Master] starting …

[2018-12-05T09:18:34,203][INFO ][o.e.t.TransportService ] [Master] publish_address {10.15.20.160:9300}, bound_addresses {10.15.20.160:9300}

[2018-12-05T09:18:34,247][INFO ][o.e.b.BootstrapChecks ] [Master] bound or publishing to a non-loopback address, enforcing bootstrap checks

[2018-12-05T09:18:34,261][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Check if searchguard index exists …

[2018-12-05T09:18:34,270][DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [Master] no known master node, scheduling a retry

[2018-12-05T09:18:37,383][INFO ][o.e.c.s.MasterService ] [Master] zen-disco-elected-as-master ([2] nodes joined)[, ], reason: new_master {Master}{iK6G-EppTjisNv1gI2M66w}{3d1e1txwS-GN8Fj9wqUQoA}{10.15.20.160}{10.15.20.160:9300}{ml.machine_memory=2101407744, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, added {{Slave1}{YZWJNQ2TRT6-A9GxN11wGA}{7DkcOH5zS2a_h3lfp3eEnA}{10.15.20.167}{10.15.20.167:9300}{ml.machine_memory=2101407744, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},{Slave2}{gdJsaoHBTRms7DXsqGO_nA}{QfHA9vzCTj6YAvvVbDFirg}{10.15.20.168}{10.15.20.168:9300}{ml.machine_memory=2101407744, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}

[2018-12-05T09:18:37,593][INFO ][o.e.c.s.ClusterApplierService] [Master] new_master {Master}{iK6G-EppTjisNv1gI2M66w}{3d1e1txwS-GN8Fj9wqUQoA}{10.15.20.160}{10.15.20.160:9300}{ml.machine_memory=2101407744, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, added {{Slave1}{YZWJNQ2TRT6-A9GxN11wGA}{7DkcOH5zS2a_h3lfp3eEnA}{10.15.20.167}{10.15.20.167:9300}{ml.machine_memory=2101407744, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},{Slave2}{gdJsaoHBTRms7DXsqGO_nA}{QfHA9vzCTj6YAvvVbDFirg}{10.15.20.168}{10.15.20.168:9300}{ml.machine_memory=2101407744, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {Master}{iK6G-EppTjisNv1gI2M66w}{3d1e1txwS-GN8Fj9wqUQoA}{10.15.20.160}{10.15.20.160:9300}{ml.machine_memory=2101407744, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([2] nodes joined)[, ]]])

[2018-12-05T09:18:37,749][INFO ][c.f.s.h.SearchGuardHttpServerTransport] [Master] publish_address {10.15.20.160:9200}, bound_addresses {10.15.20.160:9200}

[2018-12-05T09:18:37,749][INFO ][o.e.n.Node ] [Master] started

[2018-12-05T09:18:39,348][INFO ][o.e.l.LicenseService ] [Master] license [db54b8dd-65b0-4145-b692-300a0901a879] mode [basic] - valid

[2018-12-05T09:18:39,354][INFO ][o.e.g.GatewayService ] [Master] recovered [18] indices into cluster_state

[2018-12-05T09:18:40,802][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Info: No license needed because enterprise modules are not enabled

[2018-12-05T09:18:40,802][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Node ‘Master’ initialized

``

I found thoose command in my linux command history (ctrl+r)

bin/elasticsearch-plugin remove x-pack

``

bin/kibana-plugin remove x-pack

``

x-pack still there for elasticsearch (path : /usr/share/elasticsearch/bin/x-pack)

But for Kibana, there is no x-pack files in /plugin path. Is it normal that x-pack module are only located in /usr/share/kibana/node_modules/x-pack/plugins/ ?

Regards

4

up

5

I guess you need to set (in kibana.yml):

elasticsearch.url: “[https://10.15.20.160:9200](https://10.15.20.160:9200)”

elasticsearch.ssl.certificateAuthorities: [ “root-ca.cer” ]

``

···

On Friday, 21 December 2018 12:04:20 UTC+1, Voortexx wrote:

up

6

I returned to a snapshoot where there was no SearchGuard plugins and the ELK suite was working. I will tell you if there are some changes.

Thanks for help

7

It finally works !!!

I moved my root certificate to the /tmp directory, then i add

elasticsearch.url: “[https://10.15.20.160:9200](https://10.15.20.160:9200/)”

elasticsearch.ssl.certificateAuthorities: [ “root-ca.cer” ]

#elasticsearch.ssl.verify: certificate

#elasticsearch.ssl.verificationMode: none

In the kibana.yml and it works.

Thanks for help