Offline-tls-tool -crt doesn't generate key

Hey,

ElasticSearch version: 7.13.3
SearchGuard version: 7.13.3-52.1.0

I’m trying to use your provided offline certificate generation tool and here are couple of issues I’ve faced. I guess the tool is working correctly, just the documentation needs to be more clear in couple of places.

  1. Without having anything in out/ directory I’m running ./sgtlstool.sh -c ../config/example.yml -crt -v as I assume this should generate everything for me, but when I run it I get:
[root@test-searchguard1 tools]# ./sgtlstool.sh -c ../config/example.yml -crt -v
WARNING: JAVA_HOME not set, will use /usr/bin/java
Executing: com.floragunn.searchguard.tools.tlstool.tasks.Validate@750e2b97
Executing: com.floragunn.searchguard.tools.tlstool.tasks.LoadCa@2eae8e6e
File out/signing-ca.key is encrypted but no password is given. Please specify a password in the configuration file.
No files have been written                                                                                                                                                                                     Exception:                                                                                                                                                                                                      com.floragunn.searchguard.tools.tlstool.ToolException: File out/signing-ca.key is encrypted but no password is given. Please specify a password in the configuration file.
        at com.floragunn.searchguard.tools.tlstool.tasks.Task.tryConvertObjectToExpectedType(Task.java:267) ~[search-guard-tlstool-1.8.jar:1.8]
        at com.floragunn.searchguard.tools.tlstool.tasks.Task.readObjectFromPem(Task.java:232) ~[search-guard-tlstool-1.8.jar:1.8]
        at com.floragunn.searchguard.tools.tlstool.tasks.Task.readObjectFromPem(Task.java:212) ~[search-guard-tlstool-1.8.jar:1.8]
        at com.floragunn.searchguard.tools.tlstool.tasks.LoadCa.run(LoadCa.java:68) ~[search-guard-tlstool-1.8.jar:1.8]
        at com.floragunn.searchguard.tools.tlstool.SearchGuardTlsTool.run(SearchGuardTlsTool.java:213) ~[search-guard-tlstool-1.8.jar:1.8]                                                                             at com.floragunn.searchguard.tools.tlstool.SearchGuardTlsTool.main(SearchGuardTlsTool.java:61) [search-guard-tlstool-1.8.jar:1.8]
  1. To solve that I first run ./sgtlstool.sh -c ../config/example.yml -ca -v to get the key generated and after that I run previous command. But then I face another issue. I’ve set pkPassword to auto both for root and intermediate ca certs and in defaults. But then I get error: File out/signing-ca.key is encrypted but no password is given. Please specify a password in the configuration file..
    Which again I was able to fix just by setting the same password instead of auto in all fields.

  2. So I’ve finally copied all certificates, added settings to elasticsearch.yml, but now upon startup I’m getting following error:

java.lang.IllegalArgumentException: unknown setting [searchguard.ssl.http.pemkey_filepath] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

(I’ve installed the plugin before)

So either I misread something in the docs or it’s partly working or there’s something missing in docs.

@vaisov Could you share your elasticsearch.yml file?

Do you have search-guard-7 folder in plugins folder?

Please share the command which you used to install the SG plugin.

@vaisov Your command has only -crt option. That indicates that you already have Root CA generated.

The simple solution is to add -ca option to your command.

./sgtlstool.sh -c ../config/example.yml -ca -crt -v

I’ve executed that command using just unzipped offline tool with no additional configuration.

Yes, the folder is present and not empty at /usr/share/elasticsearch/plugins/search-guard-7/

Here’s the elasticsearch.yml:

cluster.name: elastic-test
node.name: "test-searchguard01"
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
transport.host: [ _eth1:ipv4_ ]
http.port: 19200
http.max_content_length: 1024mb
cluster.initial_master_nodes:
  - 10.1.0.4
  - 10.1.0.2
  - 10.1.0.3
http.host: [ _eth1:ipv4_ ]
discovery.seed_hosts:
  - 10.1.0.4
  - 10.1.0.2
  - 10.1.0.3
cluster.max_shards_per_node: 100000
cluster.info.update.interval: 1m
cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.flood_stage: 5gb
cluster.routing.allocation.disk.watermark.low: 30gb
cluster.routing.allocation.disk.watermark.high: 20gb
searchguard.ssl.transport.pemcert_filepath: test-searchguard1.pem
searchguard.ssl.transport.pemkey_filepath: test-searchguard1.key
searchguard.ssl.transport.pemkey_password: xxx
searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: test-searchguard1_http.pem
searchguard.ssl.http.pemkey_filepath: test-searchguard1_http.key
searchguard.ssl.http.pemkey_password: xxx
searchguard.ssl.http.pemtrustedcas_filepath: root-ca.pem
searchguard.nodes_dn:
- CN=sg.domain.com,OU=Ops,O=xxx,DC=domain,DC=com
- CN=sg.domain.com,OU=Ops,O=xxx,DC=domain,DC=com
- CN=sg.domain.com,OU=Ops,O=xxx,DC=domain,DC=com
searchguard.authcz.admin_dn:
- CN=sebas.domain.com,OU=Ops,O=xxx,DC=domain,DC=com

You’re right, looks like I was too exhausted yesterday to see that I was missing both arguments in one command :slight_smile: Thank you!

@vaisov What is the output of ./bin/elasticsearch-plugin list?

@vaisov Also you’re missing xpack.security.enabled: false in elasticsearch.yml file.

[root@test-searchguard1 elasticsearch]# ./bin/elasticsearch-plugin list -v
Plugins directory: /usr/share/elasticsearch/plugins
search-guard-7
- Plugin information:
Name: search-guard-7
Description: Provide access control related features for Elasticsearch 7
Version: 7.13.3-52.1.0
Elasticsearch Version: 7.13.3
Java Version: 1.8
Native Controller: false
Licensed: false
Type: isolated
Extended Plugins: [lang-painless]
 * Classname: com.floragunn.searchguard.SearchGuardPlugin

I’ve added xpack.security.enabled: false, but I still get the same error:

java.lang.IllegalArgumentException: unknown setting [searchguard.ssl.http.pemkey_filepath]

and nodes won’t start.

@vaisov I’ve tested your config and I could repro reported error when the SG plug-in was not installed.
Can you try to remove the SG plug-in and install again?

Yes, I already tried that. But it’s good that I’ve tried it again after your last reply, because I was distracted enough to forget to install the plugin to ALL nodes :slight_smile: So now I’ve installed it to all nodes and now I’m getting this error on all nodes:

[2022-05-30T15:14:30,185][WARN ][o.e.h.AbstractHttpServerTransport] [test-searchguard01] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.1.0.2:19200, remoteAddress=/10.1.0.2:36046}
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f5f6e6f6465733f66696c7465725f706174683d6e6f6465732e2a2e76657273696f6e2532436e6f6465732e2a2e687474702e7075626c6973685f616464726573732532436e6f6465732e2a2e697020485454502f312e310d0a617574686f72697a6174696f6e3a2042617369632061326c695957356858334e356333526c62546f356432746162586c695255784a55334a78516e513453334a5763413d3d0d0a757365722d6167656e743a20656c61737469637365617263682d6a732f372e31332e302d63616e6172792e3120286c696e757820332e31302e302d313136302e36322e312e656c372e7838365f36342d7836343b204e6f64652e6a73207631342e31372e32290d0a782d656c61737469632d70726f647563742d6f726967696e3a206b6962616e610d0a782d656c61737469632d636c69656e742d6d6574613a2065733d372e31332e30702c6a733d31342e31372e322c743d372e31332e30702c68633d31342e31372e320d0a486f73743a2031302e312e302e323a31393230300d0a436f6e6e656374696f6e3a206b6565702d616c6976650d0a0d0a
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at java.lang.Thread.run(Thread.java:831) [?:?]
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f5f6e6f6465733f66696c7465725f706174683d6e6f6465732e2a2e76657273696f6e2532436e6f6465732e2a2e687474702e7075626c6973685f616464726573732532436e6f6465732e2a2e697020485454502f312e310d0a617574686f72697a6174696f6e3a2042617369632061326c695957356858334e356333526c62546f356432746162586c695255784a55334a78516e513453334a5763413d3d0d0a757365722d6167656e743a20656c61737469637365617263682d6a732f372e31332e302d63616e6172792e3120286c696e757820332e31302e302d313136302e36322e312e656c372e7838365f36342d7836343b204e6f64652e6a73207631342e31372e32290d0a782d656c61737469632d70726f647563742d6f726967696e3a206b6962616e610d0a782d656c61737469632d636c69656e742d6d6574613a2065733d372e31332e30702c6a733d31342e31372e322c743d372e31332e30702c68633d31342e31372e320d0a486f73743a2031302e312e302e323a31393230300d0a436f6e6e656374696f6e3a206b6565702d616c6976650d0a0d0a
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
        ... 16 more

I think that’s coming from somewhere where http instead of https is used, let me check that.

Ok, this was coming from Kibana. I’ll just leave it her in case if someone else will face the same error. So now it looks like SG plugin started. Now I keep seeing this error:

[2022-05-30T15:30:39,689][ERROR][c.f.s.a.BackendRegistry  ] [test-searchguard01] Not yet initialized (you may need to run sgadmin)

So I’ve tried to initialize it with:

./sgadmin.sh --enable-shard-allocation -cert /root/tls/tools/out/sebas.pem -key /root/tls/tools/out/sebas.key -cacert /root/tls/tools/out/root-ca.pem -h 10.1.0.2 -p 19200

But now I’m getting:

Caused by: ElasticsearchSecurityException[Error while initializing transport SSL layer from PEM: org.bouncycastle.pkcs.PKCSException: unable to read encrypted data: Error finalising cipher]; nested: PKCSExce
ption[unable to read encrypted data: Error finalising cipher]; nested: InvalidCipherTextIOException[Error finalising cipher]; nested: BadPaddingException[Given final block not properly padded. Such issues ca
n arise if a bad key is used during decryption.];

I’ve used the key generated with tls tool, but apparently it doesn’t work.

@vaisov That will enable shard allocation. Have you applied configuration to the cluster?

No, not yet. Based on Installation | Security for Elasticsearch | Search Guard there isn’t any information about configuration which needs to be applied. In our case we need this plugin just to have alerts and nothing else, how can I set that up?

Also regarding sgadmin tool, I’ve added -keypass parameter to the command and it looks like it worked this time, but I still get error:

ERR: Cannot connect to Elasticsearch. Please refer to elasticsearch logfile for more information
Trace:
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{wI8tlO9yRVWUEjrLzup4ZA}{10.1.0.2}{10.1.0.2:19200}]]
        at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:240)
        at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:44)
        at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:380)
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:402)
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:390)
        at com.floragunn.searchguard.tools.SearchGuardAdmin.execute(SearchGuardAdmin.java:614)
        at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:156)

Probably because no configuration is applied? I’ve tried applying default configuration with ./plugins/search-guard-7/tools/sgadmin.sh -nhnv -icl -cert /root/tls/tools/out/sebas.pem -key /root/tls/tools/out/sebas.key -cacert /root/tls/tools/out/root-ca.pem -h 10.1.0.2 -p 19200 -keypass xxx, but I’m still getting the same error.

@vaisov The sgadmin.sh requires access to transport layer and not to http. Transport layer ports are assigned from the pool 9300 to 9400 by Elasticsearch.
You need to NAT or expose transport port and then use that port as an option with sgadmin.sh.
By default sgadmin.sh uses port 9300 if none is configured.

Ok, I’ve finally made it work. Now I’m trying to connect Kibana to ES cluster, but have some errors. I’ll let you know how it goes. Thanks for the tips, they were really helpful!

Also here is the wrong kibana_auth_domain config: Authentication | Security for Elasticsearch | Search Guard

It worked only after I’ve replaced enabled: true to http_enabled: true

However I’m still getting this error when I try to access Kibana:

{"statusCode":401,"error":"Unauthorized","message":"[Response Error]: undefined"}

Here’s what I have in sg_config.yml (other configs weren’t changed):

_sg_meta:
  type: "config"
  config_version: 2

sg_config:
  dynamic:
      http:
        anonymous_auth_enabled: false
        xff:
          enabled: false
          internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
      auth_token_provider: # NOT FREE FOR COMMERCIAL USE
        enabled: false
        jwt_signing_key_hs512: "xxx"
        max_validity: "1y"
        max_tokens_per_user: 100
      authc:
        kibana_auth_domain:
          http_enabled: true
          order: 0
          http_authenticator:
            type: basic
            challenge: false
          authentication_backend:
            type: internal
        basic_internal_auth_domain:
          description: "Authenticate via HTTP Basic against internal users database"
          http_enabled: true
          transport_enabled: true
          order: 4
          http_authenticator:
            type: basic
            challenge: true
          authentication_backend:
            type: intern
        clientcert_auth_domain:
          description: "Authenticate via SSL client certificates"
          http_enabled: false
          transport_enabled: false
          order: 2
          http_authenticator:
            type: clientcert
            config:
              username_attribute: cn #optional, if omitted DN becomes username
            challenge: false
          authentication_backend:
            type: noop

I can curl using kibanaserver user though, so the credentials are correct.

[root@test-searchguard1 sgconfig]# curl -u kibanaserver:kibanaserver -Ss -XGET 'https://10.1.0.2:19200/_searchguard/authinfo?pretty' -k
{
  "user" : "User kibanaserver <basic/internal>",
  "user_name" : "kibanaserver",
  "user_requested_tenant" : null,
  "remote_address" : "10.1.0.2:41732",
  "backend_roles" : [ ],
  "custom_attribute_names" : [ ],
  "attribute_names" : [ ],
  "sg_roles" : [
    "SGS_KIBANA_SERVER",
    "SGS_OWN_INDEX"
  ],
  "sg_tenants" : {
    "kibanaserver" : true
  },
  "principal" : null,
  "peer_certificates" : "0",
  "sso_logout_url" : null
}

What else am I missing?

@vaisov Try this instead. Your original config had basic authentication duplicated. Also one of them had challenge set to false.


_sg_meta:
  type: "config"
  config_version: 2

sg_config:
  dynamic:
      http:
        anonymous_auth_enabled: false
        xff:
          enabled: false
          internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
      auth_token_provider: # NOT FREE FOR COMMERCIAL USE
        enabled: false
        jwt_signing_key_hs512: "xxx"
        max_validity: "1y"
        max_tokens_per_user: 100
      authc:
        basic_internal_auth_domain:
          description: "Authenticate via HTTP Basic against internal users database"
          http_enabled: true
          transport_enabled: true
          order: 0
          http_authenticator:
            type: basic
            challenge: true
          authentication_backend:
            type: intern

Unfortunately that didn’t help, still getting the same error from Kibana. I’ve applied the config, even restarted Kibana, but that didn’t help unfortunately. This is from Kibana logs:

{"type":"log","@timestamp":"2022-06-03T07:48:47+00:00","tags":["debug","elasticsearch","query","data"],"pid":22240,"message":"401\nPOST /.kibana_7.13.3/_search?from=0&rest_total_hits_as_int=true&size=100\n{\"size\":100,\"seq_no_primary_term\":true,\"from\":0,\"query\":{\"bool\":{\"filter\":[{\"bool\":{\"should\":[{\"match\":{\"ingest-package-policies.package.name\":\"endpoint\"}}],\"minimum_should_match\":1}},{\"bool\":{\"should\":[{\"bool\":{\"must\":[{\"term\":{\"type\":\"ingest-package-policies\"}}],\"must_not\":[{\"exists\":{\"field\":\"namespace\"}},{\"exists\":{\"field\":\"namespaces\"}}]}}],\"minimum_should_match\":1}}]}},\"sort\":[{\"ingest-package-policies.updated_at\":{\"order\":\"desc\",\"unmapped_type\":\"date\"}}]} [undefined]: Response Error"}
{"type":"log","@timestamp":"2022-06-03T07:48:47+00:00","tags":["debug","elasticsearch","query","data"],"pid":22240,"message":"401\nPOST /.kibana_7.13.3/_search?from=0&rest_total_hits_as_int=true&size=20\n{\"size\":20,\"seq_no_primary_term\":true,\"from\":0,\"query\":{\"bool\":{\"filter\":[{\"bool\":{\"should\":[{\"bool\":{\"must\":[{\"term\":{\"type\":\"ingest_manager_settings\"}}],\"must_not\":[{\"exists\":{\"field\":\"namespace\"}},{\"exists\":{\"field\":\"namespaces\"}}]}}],\"minimum_should_match\":1}}]}}} [undefined]: Response Error"}
{"type":"log","@timestamp":"2022-06-03T07:48:47+00:00","tags":["error","plugins","fleet"],"pid":22240,"message":"ResponseError: Response Error\n    at onBody (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:337:23)\n    at IncomingMessage.onEnd (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:264:11)\n    at IncomingMessage.emit (events.js:387:35)\n    at endReadableNT (internal/streams/readable.js:1317:12)\n    at processTicksAndRejections (internal/process/task_queues.js:82:21) {\n  meta: {\n    body: 'Unauthorized',\n    statusCode: 401,\n    headers: {\n      'x-opaque-id': '2d1a4ca5-62dc-4596-ac44-905a223454d9',\n      'www-authenticate': 'Basic realm=\"Search Guard\"',\n      'content-type': 'text/plain; charset=UTF-8',\n      'content-length': '12'\n    },\n    meta: {\n      context: null,\n      request: [Object],\n      name: 'elasticsearch-js',\n      connection: [Object],\n      attempts: 0,\n      aborted: false\n    }\n  },\n  isBoom: true,\n  isServer: false,\n  data: null,\n  output: {\n    statusCode: 401,\n    payload: {\n      statusCode: 401,\n      error: 'Unauthorized',\n      message: 'Response Error'\n    },\n    headers: {}\n  },\n  [Symbol(SavedObjectsClientErrorCode)]: 'SavedObjectsClient/notAuthorized'\n}"}

This is my kibana.yml:

server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: [ "https://10.1.0.2:19200", "https://10.1.0.3:19200", "https://10.1.0.4:19200" ]
elasticsearch.username: "kibanaserver"
elasticsearch.password: "kibanaserver"
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/root-ca.pem" ]
elasticsearch.ssl.verificationMode: none
logging.verbose: true

@vaisov Your config is missing the following line.

xpack.security.enabled: false

Also, if you’d like to use multitenancy in Kibana, you need to add the following lines.

elasticsearch.requestHeadersWhitelist: [ "authorization", "sgtenant" ]
searchguard.multitenancy.enabled: true

Thanks, I’ll try this out if we decide to continue using Search Guard. For now we started to use Praeco - which is a GUI for Elastalert. As we needed only alertins, Search Guard was too much for us. Thank you for your support!

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.