Kibana fails to migrade tenant index

Search Guard
1

Hi,

I had to restart kibana, and one of my indices is unhappy:

Aug 06 15:59:35 ccosvms0210 kibana[17940]: {"type":"log","@timestamp":"2020-08-06T13:59:35Z","tags":["info","migrations"],"pid":17940,"message":"Creating index .kibana_-1854767153_support_2."}


Aug 06 16:00:04 ccosvms0210 kibana[17940]: {"type":"error","@timestamp":"2020-08-06T14:00:04Z","tags":["warning","process"],"pid":17940,"level":"error","error":{"message":"Error: Service Unavailable\n    at setupIndexTemplate (/usr/share/kibana/plugins/searchguard/lib/elasticsearch/setup_index_template.js:42:19)","name":"UnhandledPromiseRejectionWarning","stack":"UnhandledPromiseRejectionWarning: Error: Service Unavailable\n    at setupIndexTemplate (/usr/share/kibana/plugins/searchguard/lib/elasticsearch/setup_index_template.js:42:19)\n    at emitWarning (internal/process/promises.js:81:15)\n    at emitPromiseRejectionWarnings (internal/process/promises.js:120:9)\n    at process._tickCallback (internal/process/next_tick.js:69:34)"},"message":"Error: Service Unavailable\n    at setupIndexTemplate (/usr/share/kibana/plugins/searchguard/lib/elasticsearch/setup_index_template.js:42:19)"}
Aug 06 16:00:04 ccosvms0210 kibana[17940]: {"type":"error","@timestamp":"2020-08-06T14:00:04Z","tags":["warning","process"],"pid":17940,"level":"error","error":{"message":"Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)","name":"UnhandledPromiseRejectionWarning","stack":"Error: Service Unavailable\n    at setupIndexTemplate (/usr/share/kibana/plugins/searchguard/lib/elasticsearch/setup_index_template.js:42:19)"},"message":"Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)"}
Aug 06 16:00:05 ccosvms0210 kibana[17940]: {"type":"error","@timestamp":"2020-08-06T14:00:05Z","tags":["error","migration"],"pid":17940,"level":"error","error":{"message":"[process_cluster_event_timeout_exception] failed to process cluster event (create-index [.kibana_-1854767153_support_2], cause [api]) within 30s","name":"Error","stack":"[process_cluster_event_timeout_exception] failed to process cluster event (create-index [.kibana_-1854767153_support_2], cause [api]) within 30s :: {\"path\":\"/.kibana_-1854767153_support_2\",\"query\":{\"include_type_name\":true},\"body\":\"{\\\"mappings\\\":{\\\"doc\\\":{\\\"dynamic\\\":\\\"strict\\\",\\\"properties\\\":{\\\"config\\\":{\\\"dynamic\\\":\\\"true\\\",\\\"properties\\\":{\\\"buildNum\\\":{\\\"type\\\":\\\"keyword\\\"}}},\\\"dashboard\\\":{\\\"properties\\\":{\\\"description\\\":{\\\"type\\\":\\\"text\\\"},\\\"hits\\\":{\\\"type\\\":\\\"integer\\\"},\\\"kibanaSavedObjectMeta\\\":{\\\"properties\\\":{\\\"searchSourceJSON\\\":{\\\"type\\\":\\\"text\\\"}}},\\\"optionsJSON\\\":{\\\"type\\\":\\\"text\\\"},\\\"panelsJSON\\\":{\\\"type\\\":\\\"text\\\"},\\\"refreshInterval\\\":{\\\"properties\\\":{\\\"display\\\":{\\\"type\\\":\\\"keyword\\\"},\\\"pause\\\":{\\\"type\\\":\\\"boolean\\\"},\\\"section\\\":{\\\"type\\\":\\\"integer\\\"},\\\"value\\\":{\\\"type\\\":\\\"integer\\\"}}},\\\"timeFrom\\\":{\\\"type\\\":\\\"keyword\\\"},\\\"timeRestore\\\":{\\\"type\\\":\\\"boolean\\\"},\\\"timeTo\\\":{\\\"type\\\":\\\"keyword\\\"},\\\"title\\\":{\\\"type\\\":\\\"text\\\"},\\\"uiStateJSON\\\":{\\\"type\\\":\\\"text\\\"},\\\"version\\\":{\\\"type\\\":\\\"integer\\\"}}},\\\"index-pattern\\\":{\\\"properties\\\":{\\\"fieldFormatMap\\\":{\\\"type\\\":\\\"text\\\"},\\\"fields\\\":{\\\"type\\\":\\\"text\\\"},\\\"intervalName\\\":{\\\"type\\\":\\\"keyword\\\"},\\\"notExpandable\\\":{\\\"type\\\":\\\"boolean\\\"},\\\"sourceFilters\\\":{\\\"type\\\":\\\"text\\\"},\\\"timeFieldName\\\":{\\\"type\\\":\\\"keyword\\\"},\\\"title\\\":{\\\"type\\\":\\\"text\\\"},\\\"type\\\":{\\\"type\\\":\\\"keyword\\\"},\\\"typeMeta\\\":{\\\"type\\\":\\\"keyword\\\"}}},\\\"kql-telemetry\\\":{\\\"properties\\\":{\\\"optInCount\\\":{\\\"type\\\":\\\"long\\\"},\\\"optOutCount\\\":{\\\"type\\\":\\\"long\\\"}}},\\\"migrationVersion\\\":{\\\"dynamic\\\":\\\"true\\\",\\\"type\\\":\\\"object\\\"},\\\"namespace\\\":{\\\"type\\\":\\\"keyword\\\"},\\\"search\\\":{\\\"properties\\\":{\\\"columns\\\":{\\\"type\\\":\\\"keyword\\\"},\\\"description\\\":{\\\"type\\\":\\\"text\\\"},\\\"hits\\\":{\\\"type\\\":\\\"integer\\\"},\\\"kibanaSavedObjectMeta\\\":{\\\"properties\\\":{\\\"searchSourceJSON\\\":{\\\"type\\\":\\\"text\\\"}}},\\\"sort\\\":{\\\"type\\\":\\\"keyword\\\"},\\\"title\\\":{\\\"type\\\":\\\"text\\\"},\\\"version\\\":{\\\"type\\\":\\\"integer\\\"}}},\\\"server\\\":{\\\"properties\\\":{\\\"uuid\\\":{\\\"type\\\":\\\"keyword\\\"}}},\\\"timelion-sheet\\\":{\\\"properties\\\":{\\\"description\\\":{\\\"type\\\":\\\"text\\\"},\\\"hits\\\":{\\\"type\\\":\\\"integer\\\"},\\\"kibanaSavedObjectMeta\\\":{\\\"properties\\\":{\\\"searchSourceJSON\\\":{\\\"type\\\":\\\"text\\\"}}},\\\"timelion_chart_height\\\":{\\\"type\\\":\\\"integer\\\"},\\\"timelion_columns\\\":{\\\"type\\\":\\\"integer\\\"},\\\"timelion_interval\\\":{\\\"type\\\":\\\"keyword\\\"},\\\"timelion_other_interval\\\":{\\\"type\\\":\\\"keyword\\\"},\\\"timelion_rows\\\":{\\\"type\\\":\\\"integer\\\"},\\\"timelion_sheet\\\":{\\\"type\\\":\\\"text\\\"},\\\"title\\\":{\\\"type\\\":\\\"text\\\"},\\\"version\\\":{\\\"type\\\":\\\"integer\\\"}}},\\\"type\\\":{\\\"type\\\":\\\"keyword\\\"},\\\"updated_at\\\":{\\\"type\\\":\\\"date\\\"},\\\"url\\\":{\\\"properties\\\":{\\\"accessCount\\\":{\\\"type\\\":\\\"long\\\"},\\\"accessDate\\\":{\\\"type\\\":\\\"date\\\"},\\\"createDate\\\":{\\\"type\\\":\\\"date\\\"},\\\"url\\\":{\\\"type\\\":\\\"text\\\",\\\"fields\\\":{\\\"keyword\\\":{\\\"type\\\":\\\"keyword\\\",\\\"ignore_above\\\":2048}}}}},\\\"visualization\\\":{\\\"properties\\\":{\\\"description\\\":{\\\"type\\\":\\\"text\\\"},\\\"kibanaSavedObjectMeta\\\":{\\\"properties\\\":{\\\"searchSourceJSON\\\":{\\\"type\\\":\\\"text\\\"}}},\\\"savedSearchId\\\":{\\\"type\\\":\\\"keyword\\\"},\\\"title\\\":{\\\"type\\\":\\\"text\\\"},\\\"uiStateJSON\\\":{\\\"type\\\":\\\"text\\\"},\\\"version\\\":{\\\"type\\\":\\\"integer\\\"},\\\"visState\\\":{\\\"type\\\":\\\"text\\\"}}}}}},\\\"settings\\\":{\\\"number_of_shards\\\":1,\\\"auto_expand_replicas\\\":\\\"0-1\\\"}}\",\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"process_cluster_event_timeout_exception\\\",\\\"reason\\\":\\\"failed to process cluster event (create-index [.kibana_-1854767153_support_2], cause [api]) within 30s\\\"}],\\\"type\\\":\\\"process_cluster_event_timeout_exception\\\",\\\"reason\\\":\\\"failed to process cluster event (create-index [.kibana_-1854767153_support_2], cause [api]) within 30s\\\"},\\\"status\\\":503}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:308:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:267:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:166:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n    at IncomingMessage.emit (events.js:194:15)\n    at endReadableNT (_stream_readable.js:1103:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"message":"[process_cluster_event_timeout_exception] failed to process cluster event (create-index [.kibana_-1854767153_support_2], cause [api]) within 30s"}
2

What version of Search Guard do you use?

Creating index .kibana_-1854767153_support_2.
…
failed to process cluster event (create-index [.kibana_-1854767153_support_2], cause [api]) within 30s

I see the migration failed to create the index. Show me your cluster stats.

GET _cluster/stats?human

Show if there are any pending tasks.

GET _cluster/pending_tasks

Also, send the following configuration files

  • elasticsearch.yml
  • kibana.yml

Do you have any errors in the Elasticsearch log?

3

Hi,

No pending tasks.
That being said, the problematic kibana index contained only an index pattern, so I DELETEd it.
There was no alias by the way.
Now Kibana is happy.

Here’s the info nevertheless:

{
    "_nodes": {
        "failed": 0,
        "successful": 12,
        "total": 12
    },
    "cluster_name": "coloss_v2",
    "cluster_uuid": "_9RpiQOYRgyiyPaRuMQw0A",
    "indices": {
        "completion": {
            "size": "0b",
            "size_in_bytes": 0
        },
        "count": 1672,
        "docs": {
            "count": 49828846370,
            "deleted": 10299
        },
        "fielddata": {
            "evictions": 0,
            "memory_size": "142.8mb",
            "memory_size_in_bytes": 149837992
        },
        "query_cache": {
            "cache_count": 703832,
            "cache_size": 110855,
            "evictions": 592977,
            "hit_count": 64523513,
            "memory_size": "11.8gb",
            "memory_size_in_bytes": 12748601194,
            "miss_count": 154232611,
            "total_count": 218756124
        },
        "segments": {
            "count": 111842,
            "doc_values_memory": "136.3mb",
            "doc_values_memory_in_bytes": 142989200,
            "file_sizes": {},
            "fixed_bit_set": "0b",
            "fixed_bit_set_memory_in_bytes": 0,
            "index_writer_memory": "4.6gb",
            "index_writer_memory_in_bytes": 4955970600,
            "max_unsafe_auto_id_timestamp": 1596787381650,
            "memory": "62.2gb",
            "memory_in_bytes": 66837912756,
            "norms_memory": "92.3mb",
            "norms_memory_in_bytes": 96787648,
            "points_memory": "2gb",
            "points_memory_in_bytes": 2168881425,
            "stored_fields_memory": "6.4gb",
            "stored_fields_memory_in_bytes": 6945728240,
            "term_vectors_memory": "0b",
            "term_vectors_memory_in_bytes": 0,
            "terms_memory": "53.5gb",
            "terms_memory_in_bytes": 57483526243,
            "version_map_memory": "0b",
            "version_map_memory_in_bytes": 0
        },
        "shards": {
            "index": {
                "primaries": {
                    "avg": 6.313995215311005,
                    "max": 12,
                    "min": 1
                },
                "replication": {
                    "avg": 1.005980861244019,
                    "max": 11.0,
                    "min": 1.0
                },
                "shards": {
                    "avg": 12.633971291866029,
                    "max": 24,
                    "min": 2
                }
            },
            "primaries": 10557,
            "replication": 1.0009472387989011,
            "total": 21124
        },
        "store": {
            "size": "27.7tb",
            "size_in_bytes": 30560605300167
        }
    },
    "nodes": {
        "count": {
            "coordinating_only": 0,
            "data": 12,
            "ingest": 12,
            "master": 12,
            "total": 12
        },
        "fs": {
            "available": "17.8tb",
            "available_in_bytes": 19660695683072,
            "free": "17.8tb",
            "free_in_bytes": 19660695683072,
            "total": "45.7tb",
            "total_in_bytes": 50356552753152
        },
        "jvm": {
            "max_uptime": "143.6d",
            "max_uptime_in_millis": 12408360250,
            "mem": {
                "heap_max": "278.6gb",
                "heap_max_in_bytes": 299150606336,
                "heap_used": "163gb",
                "heap_used_in_bytes": 175066115016
            },
            "threads": 2991,
            "versions": [
                {
                    "count": 3,
                    "version": "1.8.0_242",
                    "vm_name": "OpenJDK 64-Bit Server VM",
                    "vm_vendor": "Oracle Corporation",
                    "vm_version": "25.242-b08"
                },
                {
                    "count": 9,
                    "version": "1.8.0_252",
                    "vm_name": "OpenJDK 64-Bit Server VM",
                    "vm_vendor": "Oracle Corporation",
                    "vm_version": "25.252-b09"
                }
            ]
        },
        "network_types": {
            "http_types": {
                "com.floragunn.searchguard.http.SearchGuardHttpServerTransport": 12
            },
            "transport_types": {
                "com.floragunn.searchguard.ssl.http.netty.SearchGuardSSLNettyTransport": 12
            }
        },
        "os": {
            "allocated_processors": 144,
            "available_processors": 144,
            "mem": {
                "free": "9.1gb",
                "free_in_bytes": 9796419584,
                "free_percent": 2,
                "total": "559.2gb",
                "total_in_bytes": 600528420864,
                "used": "550.1gb",
                "used_in_bytes": 590732001280,
                "used_percent": 98
            },
            "names": [
                {
                    "count": 12,
                    "name": "Linux"
                }
            ],
            "pretty_names": [
                {
                    "count": 12,
                    "pretty_name": "CentOS Linux 7 (Core)"
                }
            ]
        },
        "plugins": [
            {
                "classname": "com.floragunn.searchguard.SearchGuardPlugin",
                "description": "Provide access control related features for Elasticsearch 6",
                "elasticsearch_version": "6.8.6",
                "extended_plugins": [],
                "has_native_controller": false,
                "java_version": "1.8",
                "name": "search-guard-6",
                "version": "6.8.6-25.5"
            }
        ],
        "process": {
            "cpu": {
                "percent": 381
            },
            "open_file_descriptors": {
                "avg": 7751,
                "max": 10007,
                "min": 6736
            }
        },
        "versions": [
            "6.8.6"
        ]
    },
    "status": "green",
    "timestamp": 1596793450106
}

I’ll post the other two files if you’re still interested.
Thanks for looking into this !

4

You deleted .kibana_-1854767153_support_2 and now Kibana works, right? Did Kibana create .kibana_-1854767153_support_2 again?

I saw multiple time Kibana failed with migration. Deleting the new index helped.

5

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.