es version: 5.5.1
search guard: 5-5.5.1-15
jdk version: 1.8.0_131-b11
logstash version: 2.4.1
My es cluster installed search guard has 5 nodes, logstash can’t send message to es, and error info is: host name does not match certificate subject. Why?
elasticsearch.yml
cluster.name: honeycomb-es-guard-5.5.1
node 1 to 5
node.name: node-1
path.data: /data01/elasticsearch-guard
path.logs: /opt/yrd_logs/elasticsearch-guard
network.host: 0.0.0.0
http.cors.enabled: true
http.cors.allow-origin: “*”
thread_pool.bulk.queue_size: 50
searchguard.ssl.transport.keystore_filepath: node-1-keystore.jks
searchguard.ssl.transport.keystore_password: node_es
searchguard.ssl.transport.truststore_filepath: truststore.jks
searchguard.ssl.transport.truststore_password: r_ca_honeycomb
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.enable_openssl_if_available: true
searchguard.ssl.transport.resolve_hostname: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.keystore_filepath: node-1-keystore.jks
searchguard.ssl.http.keystore_password: node_es
searchguard.ssl.http.truststore_filepath: truststore.jks
searchguard.ssl.http.truststore_password: r_ca_honeycomb
searchguard.authcz.admin_dn:
-
CN=*,OU=client,O=client,L=test, C=de
-
CN=kirk,OU=client,O=client,L=test,C=DE
searchguard info, as follows
[elk@orz-core-elk-05 config]$ curl -k -u admin:admin ‘https://10.134.84.33:9200/_searchguard/authinfo?pretty’
{
“user” : “User [name=admin, roles=]”,
“user_name” : “admin”,
“user_requested_tenant” : null,
“remote_address” : “10.134.84.36:49843”,
“sg_roles” : [
“sg_all_access”,
“sg_own_index”,
“sg_public”
],
“sg_tenants” : {
“test_tenant_ro” : true,
“admin” : true,
“adm_tenant” : true
},
“principal” : null,
“peer_certificates” : “0”
}
cluster info, as follows
curl -k -u admin:admin ‘https://10.134.84.33:9200/_cat/health?v’
epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1503395742 17:55:42 honeycomb-es-guard-5.5.1 green 5 5 5 1 0 0 0 0 - 100.0%
logstash configuration
output {
elasticsearch {
hosts => [“https://10.134.84.32:9200”]
index => “%{log_project}-%{+YYYY-MM-dd}”
ssl => true
ssl_certificate_verification => true
truststore => “/opt/yrd_soft/elasticsearch-5.5.1-guard/config/truststore.jks”
truststore_password => r_ca_honeycomb
user => logstash
password => logstash
}
}
start logstash
[elk@orz-core-elk-02 conf]$ /opt/yrd_soft/logstash-2.4.1-guard/bin/logstash -f /opt/yrd_soft/logstash-2.4.1-guard/conf/honeycomb-logstash.conf &
[1] 1691
[elk@orz-core-elk-02 conf]$ Settings: Default pipeline workers: 56
Host name ‘10.134.84.32’ does not match the certificate subject provided by the peer (CN=*, OU=SSL, O=Test, L=Test, C=DE) {:class=>“Manticore::UnknownException”, :level=>:error}
Pipeline main started
Attempted to send a bulk request to Elasticsearch configured at ‘[“https://10.134.84.32:9200”]’, but an error occurred and it failed! Are you sure you can reach elasticsearch from this machine using the configuration provided? {:error_message=>“Host name ‘10.134.84.32’ does not match the certificate subject provided by the peer (CN=*, OU=SSL, O=Test, L=Test, C=DE)”, :error_class=>“Manticore::UnknownException”, :backtrace=>[“/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:37:in initialize'", "org/jruby/RubyProc.java:281:in call’”, “/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:79:in call'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:256:in call_once’”, “/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:153:in code'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/http/manticore.rb:84:in perform_request’”, “org/jruby/RubyProc.java:281:in call'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/base.rb:257:in perform_request’”, “/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/http/manticore.rb:67:in perform_request'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/client.rb:128:in perform_request’”, “/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.1.0/lib/elasticsearch/api/actions/bulk.rb:93:in bulk'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:53:in non_threadsafe_bulk’”, “/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in bulk'", "org/jruby/ext/thread/Mutex.java:149:in synchronize’”, “/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in bulk'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:172:in safe_bulk’”, “/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:101:in submit'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:86:in retrying_submit’”, “/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:29:in multi_receive'", "org/jruby/RubyArray.java:1653:in each_slice’”, “/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:28:in multi_receive'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/output_delegator.rb:130:in worker_multi_receive’”, “/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/output_delegator.rb:114:in multi_receive'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:301:in output_batch’”, “org/jruby/RubyHash.java:1342:in each'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:301:in output_batch’”, “/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:232:in worker_loop'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:201:in start_workers’”], :level=>:error}
Host name ‘10.134.84.32’ does not match the certificate subject provided by the peer (CN=*, OU=SSL, O=Test, L=Test, C=DE) {:class=>“Manticore::UnknownException”, :backtrace=>[“/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:37:in initialize'", "org/jruby/RubyProc.java:281:in call’”, “/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:79:in call'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:256:in call_once’”, “/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:153:in code'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/http/manticore.rb:84:in perform_request’”, “org/jruby/RubyProc.java:281:in call'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/base.rb:257:in perform_request’”, “/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/http/manticore.rb:67:in perform_request'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/client.rb:128:in perform_request’”, “/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.1.0/lib/elasticsearch/api/actions/bulk.rb:93:in bulk'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:53:in non_threadsafe_bulk’”, “/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in bulk'", "org/jruby/ext/thread/Mutex.java:149:in synchronize’”, “/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in bulk'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:172:in safe_bulk’”, “/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:101:in submit'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:86:in retrying_submit’”, “/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:29:in multi_receive'", "org/jruby/RubyArray.java:1653:in each_slice’”, “/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:28:in multi_receive'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/output_delegator.rb:130:in worker_multi_receive’”, “/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/output_delegator.rb:114:in multi_receive'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:301:in output_batch’”, “org/jruby/RubyHash.java:1342:in each'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:301:in output_batch’”, “/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:232:in worker_loop'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:201:in start_workers’”], :level=>:warn}