I am getting a interrupted while the page was loading message from kibana.

Search Guard
1

I am trying to add searchguard to a running test ELK stack (one node for everything).

I have a running ELK stack and have created a CA with a valid cert for my server (elk-1.mdtsoft.com)
I added Search Guard SSL to the Elasticsearch and can access the :9200 without problem seeing
the following for https://elk-1.mdtsoft.com:9200/_cat/indices?v
(after doing basic auth to a user with ADMIN rights

health status index pri rep docs.count docs.deleted store.size pri.store.size green open filebeat-2016.09.16 5 0 11329 0 3.6mb 3.6mb yellow open filebeat-2016.09.17 5 1 12999 0 4.3mb 4.3mb green open filebeat-2016.09.14 5 0 12445 0 4.1mb 4.1mb green open filebeat-2016.09.15 5 0 12665 0 4.3mb 4.3mb yellow open filebeat-2016.09.18 5 1 11190 0 3.5mb 3.5mb yellow open filebeat-2016.09.19 5 1 4430 0 1.5mb 1.5mb green open filebeat-2016.09.12 5 0 22990 0 6.1mb 6.1mb green open filebeat-2016.09.13 5 0 11179 0 3.5mb 3.5mb green open filebeat-2016.09.11 5 0 1635 0 1.1mb 1.1mb green open .kibana 1 0 12 2 42.6kb 42.6kb yellow open topbeat-2016.09.19 5 1 514250 0 125.9mb 125.9mb yellow open topbeat-2016.09.17 5 1 1340098 0 331.5mb 331.5mb yellow open topbeat-2016.09.18 5 1 1322749 0 326.4mb 326.4mb green open topbeat-2016.09.15 5 0 1301458 0 322.5mb 322.5mb green open topbeat-2016.09.16 5 0 1313893 0 324.3mb 324.3mb green open topbeat-2016.09.13 5 0 1287888 0 314.2mb 314.2mb green open topbeat-2016.09.14 5 0 1287656 0 314.1mb 314.1mb green open topbeat-2016.09.12 5 0 717677 0 174.7mb 174.7mb green open filebeat-2016.09.08 5 0 45156 0 7.4mb 7.4mb green open packetbeat-2016.09.14 5 0 23 0 208.7kb 208.7kb green open packetbeat-2016.09.15 5 0 553 0 467.6kb 467.6kb green open packetbeat-2016.09.16 5 0 8 0 78.4kb 78.4kb yellow open packetbeat-2016.09.17 5 1 6 0 51kb 51kb green open packetbeat-2016.09.12 5 0 1362 0 774.3kb 774.3kb green open packetbeat-2016.09.13 5 0 8 0 70.2kb 70.2kb green open searchguard 1 0 0 0 112.8kb 112.8kb yellow open packetbeat-2016.09.18 5 1 9 0 79.5kb 79.5kb yellow open packetbeat-2016.09.19 5 1 63 0 269.4kb 269.4kb

https://elk-1.mdtsoft.com:9200/_searchguard/sslinfo?pretty

shows

  {
"principal" : "CN=elk-1.mdtsoft.com,OU=SSL,O=ELK,L=ELK,C=US",
"peer_certificates" : "2",
"ssl_protocol" : "TLSv1.2",
"ssl_cipher" : "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA",
"ssl_openssl_available" : true,
"ssl_openssl_version" : 268439647,
"ssl_openssl_version_string" : "OpenSSL 1.0.1e-fips 11 Feb 2013",
"ssl_openssl_non_available_cause" : "",
"ssl_provider_http" : "OPENSSL",
"ssl_provider_transport_server" : "OPENSSL",
"ssl_provider_transport_client" : "OPENSSL"
}

<details class='elided'>
<summary title='Show trimmed content'>&#183;&#183;&#183;</summary>

-----
So it looks like I ahve the elasticsearch stuff set up (including OpenSSL correctly)

I have added to the kibana.yml

# The Elasticsearch instance to use for all your queries.
elasticsearch.url: "https://elk-1.mdtsoft.com:9200"
elasticsearch.ssl.ca: "/etc/pki/tls/elk-root-ca.pem"
elasticsearch.username: "kibanaserver"
elasticsearch.password: MY-PASSWORD-FOR-KIBANASERVER-HERE-IN-QUOTES

and place the elk-root-ca.pem in the location show above.

I set the kibana logging to verbose with
# Set this to true to log all events, including system usage information and all requests.
logging.verbose: true

but the tail of the log only shows normal stuff (status to green, "Checking Elasticsearch version" that sort of
thing.

Kibana was working (although unsecure) before I set up Search Guard and SG SSL.

I am not sure what to do next to troubleshoot.

I tried both Firefox and Chrome both talk about the server being interrupted. (tried on two different computers)
2

In addition the Elasticsearch logs don’t show any traffic when I re-try the 5601 access. I restarted Elasticsearch and see the end of the log looks like.

[2016-09-20 08:31:45,285][INFO ][node ] [elk-1] stopping …
[2016-09-20 08:31:45,807][INFO ][node ] [elk-1] stopped
[2016-09-20 08:31:45,810][INFO ][node ] [elk-1] closing …
[2016-09-20 08:31:45,819][INFO ][node ] [elk-1] closed
[2016-09-20 08:31:47,250][INFO ][node ] [elk-1] version[2.4.0], pid[9798], build[ce9f0c7/2016-08-29T09:14:17Z]
[2016-09-20 08:31:47,252][INFO ][node ] [elk-1] initializing …
[2016-09-20 08:31:48,277][INFO ][com.floragunn.searchguard.ssl.SearchGuardSSLPlugin] Search Guard 2 plugin also available
[2016-09-20 08:31:48,286][INFO ][com.floragunn.searchguard.SearchGuardPlugin] Node [elk-1] is a transportClient: false/tribeNode: false/tribeNodeClient: false
[2016-09-20 08:31:48,287][INFO ][plugins ] [elk-1] modules [reindex, lang-expression, lang-groovy], plugins [search-guard-ssl, search-guard-2], sites
[2016-09-20 08:31:48,329][INFO ][env ] [elk-1] using [1] data paths, mounts [[/ (/dev/mapper/fedora_elk–1-root)]], net usable_space [34gb], net total_space [44.4gb], spins? [possibly], types [xfs]
[2016-09-20 08:31:48,330][INFO ][env ] [elk-1] heap size [1015.6mb], compressed ordinary object pointers [true]
[2016-09-20 08:31:48,400][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Open SSL OpenSSL 1.0.1e-fips 11 Feb 2013 available
[2016-09-20 08:31:48,405][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Open SSL OpenSSL 1.0.1e-fips 11 Feb 2013 available
[2016-09-20 08:31:48,874][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Config directory is /etc/elasticsearch/, from there the key- and truststore files are resolved relatively
[2016-09-20 08:31:49,030][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] HTTPS client auth mode OPTIONAL
[2016-09-20 08:31:49,057][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslTransportClientProvider:OPENSSL with ciphers [TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA]
[2016-09-20 08:31:49,058][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslTransportServerProvider:OPENSSL with ciphers [TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA]
[2016-09-20 08:31:49,058][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslHTTPProvider:OPENSSL with ciphers [TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA]
[2016-09-20 08:31:49,058][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslTransport protocols [TLSv1.2, TLSv1.1]
[2016-09-20 08:31:49,058][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslHTTP protocols [TLSv1.2, TLSv1.1]
[2016-09-20 08:31:49,313][INFO ][http ] [elk-1] Using [org.elasticsearch.http.netty.NettyHttpServerTransport] as http transport, overridden by [search-guard2]
[2016-09-20 08:31:49,406][INFO ][com.floragunn.searchguard.configuration.ConfigurationModule] FLS/DLS valve not bound (noop)
[2016-09-20 08:31:49,408][INFO ][com.floragunn.searchguard.auditlog.AuditLogModule] Auditlog not available
[2016-09-20 08:31:49,504][INFO ][transport ] [elk-1] Using [com.floragunn.searchguard.transport.SearchGuardTransportService] as transport service, overridden by [search-guard2]
[2016-09-20 08:31:49,505][INFO ][transport ] [elk-1] Using [com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] as transport, overridden by [search-guard-ssl]
[2016-09-20 08:31:52,226][INFO ][node ] [elk-1] initialized
[2016-09-20 08:31:52,226][INFO ][node ] [elk-1] starting …
[2016-09-20 08:31:52,297][INFO ][com.floragunn.searchguard.transport.SearchGuardTransportService] [elk-1] publish_address {50.59.199.121:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}, {50.59.199.121:9300}
[2016-09-20 08:31:52,301][INFO ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [elk-1] Check if searchguard index exists …
[2016-09-20 08:31:52,307][DEBUG][action.admin.indices.exists.indices] [elk-1] no known master node, scheduling a retry
[2016-09-20 08:31:52,313][INFO ][discovery ] [elk-1] elasticsearch/dYjmwfs6Q-qrdSGyFNV1Yg
[2016-09-20 08:31:55,522][INFO ][cluster.service ] [elk-1] new_master {elk-1}{dYjmwfs6Q-qrdSGyFNV1Yg}{50.59.199.121}{50.59.199.121:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2016-09-20 08:31:55,556][INFO ][http ] [elk-1] publish_address {50.59.199.121:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}, {50.59.199.121:9200}
[2016-09-20 08:31:55,558][INFO ][node ] [elk-1] started
[2016-09-20 08:31:56,326][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-09-20 08:31:56,375][INFO ][gateway ] [elk-1] recovered [28] indices into cluster_state
[2016-09-20 08:31:57,874][INFO ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [elk-1] Node ‘elk-1’ initialized
[2016-09-20 08:32:06,780][INFO ][cluster.routing.allocation] [elk-1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana][0]] …]).
(END)

Still not sure what I am doing wrong.