Elasticsearch 6.8.6, green cluster, yellow Kibana

Search Guard
1

I recently adjusted JVM heap size on all nodes in my cluster. After significant disturbance it’s back to behaving and is balancing shards.

{
“cluster_name” : “elasticsearch”,
“status” : “green”,
“timed_out” : false,
“number_of_nodes” : 13,
“number_of_data_nodes” : 9,
“active_primary_shards” : 762,
“active_shards” : 2113,
“relocating_shards” : 4,
“initializing_shards” : 0,
“unassigned_shards” : 0,
“delayed_unassigned_shards” : 0,
“number_of_pending_tasks” : 0,
“number_of_in_flight_fetch” : 0,
“task_max_waiting_in_queue_millis” : 0,
“active_shards_percent_as_number” : 100.0
}

But my old nemesis is back:

plugin:searchguard@6.8.6-19.0 Setting up index template.

I shut down Kibana, wiped log, restarted, and this is all it says:

{“type”:“log”,“@timestamp”:“2020-08-15T01:59:21Z”,“tags”:[“listening”,“info”],“pid”:14866,“message”:“Server running at http://0.0.0.0:5601”}

The console log doesn’t provide me any clues.

kibana:1 Failed to load resource: the server responded with a status of 503 ()
kibana:186 Refused to execute inline script because it violates the following Content Security Policy directive: “script-src ‘unsafe-eval’ ‘self’”. Either the ‘unsafe-inline’ keyword, a hash (‘sha256-SHHSeLc0bp6xt4BoVVyUy+3IbVqp3ujLaR+s+kSP5UI=’), or a nonce (‘nonce-…’) is required to enable inline execution.

/bundles/app/status_page/bootstrap.js:10 ^ A single error about an inline script not firing due to content security policy is expected!
kibana#?_g=():1 Failed to load resource: the server responded with a status of 503 ()

This system has a Cloudflare front end, it passes through a firewall as a port forward, lands on an nginx reverse proxy, and then gets dispatched to the actual Kibana instance. This is the configuration we’ve used from the start, so in production a year or more.

So … how do I get this running again? No Kibana doesn’t put us out of production, but it does blind us at a very busy time :frowning:

2

Let’s troubleshoot it.

  1. Enable the verbose mode in Kibana https://github.com/NetwarSystem/elktrouble/blob/master/h0-kibana.yml#L109
  2. Enable the SG debug mode.
curl -u admin:admin --insecure -X PUT "https://localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d '{
  "transient": {
    "logger.com.floragunn": "debug"
  }
}'
  1. Open your browser and navigate to Kibana. Make a screenshot of your browser. Paste it here.
  2. Additionally, show me (screenshot) what you have in the browser dev tools Network. Specifically, it’d be helpful to see the response headers and the response body of the failing requests.
  3. Send me logs from Kibana, Elasticsearch, and NGINX proxy.
  4. Send me sg_config.yml
  5. Reveal more cluster stats.
curl -k -u admin:admin -X GET https://localhost:9200/_cat/indices
curl -k -u admin:admin -X GET https://localhost:9200/_cluster/stats?human
curl -k -u admin:admin -X GET https://localhost:9200/_cluster/pending_tasks
3

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.