JWT configuration issue

laurent_gaydon · February 13, 2018, 4:54pm

Search Guard: 5.6.6-23
Elasticsearch version: 5.6.6
Kibana: 5.6.6
Installed and used enterprise modules, if any:
Multi-Tenancy 5.4.4
JWT 5.0.6
JVM version: 1.8.0_162
operating system version RedHat 3.10.0-693.11.6.el7.x86_64
S3 plugin 5.6.6
Plugin Kibana 5.6.6-5

Hello,

I’m trying to configure JWT module using AWS Cognito to produce JWT tokens for us.

I have configured sg_config.yml to enable jwt:

searchguard:

dynamic:

Set filtered_alias_mode to ‘disallow’ to forbid more than 2 filtered aliases per index

Set filtered_alias_mode to ‘warn’ to allow more than 2 filtered aliases per index but warns about it (default)

Set filtered_alias_mode to ‘nowarn’ to allow more than 2 filtered aliases per index silently

#filtered_alias_mode: warn

kibana:

Kibana multitenancy - NOT FREE FOR COMMERCIAL USE

see https://github.com/floragunncom/search-guard-docs/blob/master/multitenancy.md

To make this work you need to install Home · floragunncom/search-guard-module-kibana-multitenancy Wiki · GitHub

multitenancy_enabled: true

server_username: kibanaserver

index: ‘.kibana’

do_not_fail_on_forbidden: true

http:

anonymous_auth_enabled: false

xff:

enabled: false

internalProxies: ‘192.168.0.10|192.168.0.11’ # regex pattern

#internalProxies: ‘.*’ # trust all internal proxies, regex pattern

remoteIpHeader: ‘x-forwarded-for’

proxiesHeader: ‘x-forwarded-by’

#trustedProxies: ‘.*’ # trust all external proxies, regex pattern

see Pattern (Java Platform SE 7 ) for regex help

more information about XFF X-Forwarded-For - Wikipedia

and here RFC 7239: Forwarded HTTP Extension

and Apache Tomcat 8 Configuration Reference (8.0.53) - The Valve Component

authc:

jwt_auth_domain:

enabled: true

order: 0

http_authenticator:

type: jwt

challenge: false

config:

signing_key: |-

-----BEGIN PUBLIC KEY-----

MIIBC… My Public key

-----END PUBLIC KEY-----

jwt_header: “Authorization”

jwt_url_parameter: null

roles_key: “family_name”

subject_key: “cognito:username”

authentication_backend:

type: noop

basic_internal_auth_domain:

enabled: true

order: 1

http_authenticator:

type: basic

challenge: false

authentication_backend:

type: intern

In elasticsearch.yml I have disabled the user cache

Disable the SG User Cache for JWT

searchguard.cache.ttl_minutes: 0

in kibana.yml I add the following searchguard parameters:

elasticsearch.requestHeadersWhitelist: [ “sg_tenant”, “Authorization”, “authorization”, “jwtparam” ]

searchguard.multitenancy.enabled: true

searchguard.multitenancy.tenants.enable_global: false

searchguard.multitenancy.tenants.enable_private: false

searchguard.basicauth.enabled: false

searchguard.jwt.enabled: true

searchguard.jwt.url_param: jwtparam

On elasticsearch log I have the following error:

[2018-02-13T15:08:02,130][ERROR][c.f.d.a.h.j.HTTPJwtAuthenticator] Error creating JWT authenticator: java.lang.NoClassDefFoundError: io/jsonwebtoken/impl/TextCodec. JWT authentication will not work

java.lang.NoClassDefFoundError: io/jsonwebtoken/impl/TextCodec

at com.floragunn.dlic.auth.http.jwt.HTTPJwtAuthenticator.(HTTPJwtAuthenticator.java:77) [dlic-search-guard-auth-http-jwt-5.0-6.jar:5.0-6]

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [?:1.8.0_162]

at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) [?:1.8.0_162]

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) [?:1.8.0_162]

at java.lang.reflect.Constructor.newInstance(Unknown Source) [?:1.8.0_162]

at com.floragunn.searchguard.auth.BackendRegistry.newInstance(BackendRegistry.java:168) [search-guard-5-5.6.6-18.jar:?]

at com.floragunn.searchguard.auth.BackendRegistry.onChange(BackendRegistry.java:218) [search-guard-5-5.6.6-18.jar:?]

at com.floragunn.searchguard.configuration.IndexBaseConfigurationRepository.notifyAboutChanges(IndexBaseConfigurationRepository.java:305) [search-guard-5-5.6.6-18.jar:?]

at com.floragunn.searchguard.configuration.IndexBaseConfigurationRepository.reloadConfiguration(IndexBaseConfigurationRepository.java:271) [search-guard-5-5.6.6-18.jar:?]

at com.floragunn.searchguard.configuration.IndexBaseConfigurationRepository$1$1.run(IndexBaseConfigurationRepository.java:138) [search-guard-5-5.6.6-18.jar:?]

at java.lang.Thread.run(Unknown Source) [?:1.8.0_162]

Caused by: java.lang.ClassNotFoundException: io.jsonwebtoken.impl.TextCodec

at java.net.URLClassLoader.findClass(Unknown Source) ~[?:1.8.0_162]

at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:1.8.0_162]

at java.net.FactoryURLClassLoader.loadClass(Unknown Source) ~[?:1.8.0_162]

at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:1.8.0_162]

… 11 more

On Kibana side, it doesn’t work. When I try to connect and I can authenticate with user authentication (the chrome one) and tenants menu doesn’t show us any existing tenants.

Note: Multitenant was working fine before to enable JWT.

regards

Laurent

searchguard_google_group · February 13, 2018, 5:44pm

you need to install the jwt module with its dependencies (jar-with-dependencies).

http://oss.sonatype.org/service/local/artifact/maven/content?c=jar-with-dependencies&r=releases&g=com.floragunn&a=dlic-search-guard-auth-http-jwt&v=5.0-6

···

Am 13.02.2018 um 17:54 schrieb laurent gaydon <laurentgaydon@gmail.com>:

* Search Guard: 5.6.6-23
* Elasticsearch version: 5.6.6
* Kibana: 5.6.6
* Installed and used enterprise modules, if any:
   * Multi-Tenancy 5.4.4
   * JWT 5.0.6
* JVM version: 1.8.0_162
* operating system version RedHat 3.10.0-693.11.6.el7.x86_64
* S3 plugin 5.6.6
* Plugin Kibana 5.6.6-5

Hello,

I'm trying to configure JWT module using AWS Cognito to produce JWT tokens for us.

I have configured sg_config.yml to enable jwt:
searchguard:
  dynamic:
    # Set filtered_alias_mode to 'disallow' to forbid more than 2 filtered aliases per index
    # Set filtered_alias_mode to 'warn' to allow more than 2 filtered aliases per index but warns about it (default)
    # Set filtered_alias_mode to 'nowarn' to allow more than 2 filtered aliases per index silently
    #filtered_alias_mode: warn
    kibana:
      # Kibana multitenancy - NOT FREE FOR COMMERCIAL USE
      # see https://github.com/floragunncom/search-guard-docs/blob/master/multitenancy.md
      # To make this work you need to install Home · floragunncom/search-guard-module-kibana-multitenancy Wiki · GitHub
      multitenancy_enabled: true
      server_username: kibanaserver
      index: '.kibana'
      do_not_fail_on_forbidden: true
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
        internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
        #internalProxies: '.*' # trust all internal proxies, regex pattern
        remoteIpHeader: 'x-forwarded-for'
        proxiesHeader: 'x-forwarded-by'
        #trustedProxies: '.*' # trust all external proxies, regex pattern
        ###### see Pattern (Java Platform SE 7 ) for regex help
        ###### more information about XFF X-Forwarded-For - Wikipedia
        ###### and here RFC 7239 - Forwarded HTTP Extension
        ###### and Apache Tomcat 8 Configuration Reference (8.0.53) - The Valve Component
    authc:
      jwt_auth_domain:
        enabled: true
        order: 0
        http_authenticator:
          type: jwt
          challenge: false
          config:
            signing_key: |-
              -----BEGIN PUBLIC KEY-----
              MIIBC... My Public key
              -----END PUBLIC KEY-----
            jwt_header: "Authorization"
            jwt_url_parameter: null
            roles_key: "family_name"
            subject_key: "cognito:username"
        authentication_backend:
          type: noop
      basic_internal_auth_domain:
        enabled: true
        order: 1
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: intern

In elasticsearch.yml I have disabled the user cache

# Disable the SG User Cache for JWT
searchguard.cache.ttl_minutes: 0

in kibana.yml I add the following searchguard parameters:

elasticsearch.requestHeadersWhitelist: [ "sg_tenant", "Authorization", "authorization", "jwtparam" ]

searchguard.multitenancy.enabled: true
searchguard.multitenancy.tenants.enable_global: false
searchguard.multitenancy.tenants.enable_private: false

searchguard.basicauth.enabled: false
searchguard.jwt.enabled: true
searchguard.jwt.url_param: jwtparam

On elasticsearch log I have the following error:
[2018-02-13T15:08:02,130][ERROR][c.f.d.a.h.j.HTTPJwtAuthenticator] Error creating JWT authenticator: java.lang.NoClassDefFoundError: io/jsonwebtoken/impl/TextCodec. JWT authentication will not work
java.lang.NoClassDefFoundError: io/jsonwebtoken/impl/TextCodec
        at com.floragunn.dlic.auth.http.jwt.HTTPJwtAuthenticator.<init>(HTTPJwtAuthenticator.java:77) [dlic-search-guard-auth-http-jwt-5.0-6.jar:5.0-6]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [?:1.8.0_162]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) [?:1.8.0_162]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) [?:1.8.0_162]
        at java.lang.reflect.Constructor.newInstance(Unknown Source) [?:1.8.0_162]
        at com.floragunn.searchguard.auth.BackendRegistry.newInstance(BackendRegistry.java:168) [search-guard-5-5.6.6-18.jar:?]
        at com.floragunn.searchguard.auth.BackendRegistry.onChange(BackendRegistry.java:218) [search-guard-5-5.6.6-18.jar:?]
        at com.floragunn.searchguard.configuration.IndexBaseConfigurationRepository.notifyAboutChanges(IndexBaseConfigurationRepository.java:305) [search-guard-5-5.6.6-18.jar:?]
        at com.floragunn.searchguard.configuration.IndexBaseConfigurationRepository.reloadConfiguration(IndexBaseConfigurationRepository.java:271) [search-guard-5-5.6.6-18.jar:?]
        at com.floragunn.searchguard.configuration.IndexBaseConfigurationRepository$1$1.run(IndexBaseConfigurationRepository.java:138) [search-guard-5-5.6.6-18.jar:?]
        at java.lang.Thread.run(Unknown Source) [?:1.8.0_162]
Caused by: java.lang.ClassNotFoundException: io.jsonwebtoken.impl.TextCodec
        at java.net.URLClassLoader.findClass(Unknown Source) ~[?:1.8.0_162]
        at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:1.8.0_162]
        at java.net.FactoryURLClassLoader.loadClass(Unknown Source) ~[?:1.8.0_162]
        at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:1.8.0_162]
        ... 11 more

On Kibana side, it doesn't work. When I try to connect and I can authenticate with user authentication (the chrome one) and tenants menu doesn't show us any existing tenants.

Note: Multitenant was working fine before to enable JWT.

regards
Laurent

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/44cc8c48-df38-413d-9927-2db0c9603b4d%40googlegroups.com\.
For more options, visit https://groups.google.com/d/optout\.

laurent_gaydon · February 14, 2018, 2:08pm

Hello,

Thanks for the reply it’s working better with the dependencies.

Now we have a configuration issue, we configure sg_role like that:

authc:

jwt_auth_domain:

enabled: true

order: 0

http_authenticator:

type: jwt

challenge: false

config:

signing_key: |-

-----BEGIN PUBLIC KEY-----

MIIBCgKCAQEAokkT24WSNnOf0x6Y+C93qDnsweyC13QFddDBwoxXrZ6z+9GVvZTr

+LQ05Vwyh8GE80fzfyqFvdk5ghXbKcEKXqInq60/ljy7L+Q1iVnev3RAFBHaSH6/

revQLHMTW/Bjukelm258hniKPfLMtmNq3XhGPIz6VMKJ8DBQ2XZH0mAGRFqhU3fM

T7C+yt2oO4KV01vA4Zl1hS/SR+RKNdSPjPHihe1mRSG80ZnPeGbDdLmLZ6sj6DBk

HoDAeXrRwyijSsIlGkAb+BQxxzG/2DdiZOKbLodxVk5/ORcAzxOCONxN1mZQ9Nny

ghutASoP1v59Ys7U2avG4igeU6W9HgXCXwIDAQAB

-----END PUBLIC KEY-----

jwt_header: null

jwt_url_parameter: “Authorization”

roles_key: “family_name”

subject_key: “cognito:username”

authentication_backend:

type: noop

and sg_role_mapping.yml:

sg_bband:

users:

bband

backendroles:

tenant1

we try to connect to ES

https://<ES_IP>:9200/?Authorization=eyJraWQiOiJoQ2ZcLzI0OEUwM2FWb1lRZ2Npa0t4YWtSc3FkdmpWMkYrR2xOR2wxOEpiVT0iLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJkNTUxYTE0NC0yM2M2LTQxNmYtOThhMC0xMTlmZjI3NmUzMGMiLCJlbWFpbF92ZXJpZmllZCI6dHJ1ZSwiaXNzIjoiaHR0cHM6XC9cL2NvZ25pdG8taWRwLmV1LXdlc3QtMS5hbWF6b25hd3MuY29tXC9ldS13ZXN0LTFfbkxSSXFvTUtyIiwiY29nbml0bzp1c2VybmFtZSI6ImJsYWlzZS5sdWdlb25AbmFncmEuY29tIiwiZ2l2ZW5fbmFtZSI6InRvdG8iLCJhdWQiOiI2cHBvdjlpZTdhcTk3NHV2c2F2cGZtamN2MSIsImV2ZW50X2lkIjoiYWNjZDNkMDItMTE4ZS0xMWU4LTgzYjYtYTk4ZTJkNDJhM2E5IiwidG9rZW5fdXNlIjoiaWQiLCJhdXRoX3RpbWUiOjE1MTg2MTY1MTEsImV4cCI6MTUxODYyMDExMSwiaWF0IjoxNTE4NjE2NTExLCJmYW1pbHlfbmFtZSI6InRlbmFudDEiLCJlbWFpbCI6ImJsYWlzZS5sdWdlb25AbmFncmEuY29tIn0.n_n_EFqXW2suyw0gcC2TdjzPpQSqOB9MjufiVKFfPpgOnzpnfU0Pys03kxGLdNlJYmkc7Um8cMeniXUXSNZ34n2IknoHHlg3b_VZGdUkdZASpziHnco6Mgl6ZEMteac9bfUvjmpXC3AUB9K1j9HDS9s-cS09qHkClajlxQ8w9SpAK3ffCZh2hPGtJnrDbAUkkYqGNOvK8mVYAJMoScP59hQlBW-1_TegPDyIAe_Pmee_v0QjQzeLPIOyxHuJtBiY1I0H0XnAEkxtQrLYUrBEkU4n8XpCLDdO6J38ePmlH2QjG9jYQJsMCNc-bqFQYSfiKSpPNoGX1G5e2xlJt5dyXg

And I have the following error

Authentication finally failed.

Regards,

Laurent.

jkressin · February 14, 2018, 2:23pm

This can have several reasons, we need to see the logfile I guess. Can you please enable debug logging as described here:

Then, issue the failing request again and post the logs here. Thanks!

···

On Wednesday, February 14, 2018 at 3:08:49 PM UTC+1, laurent gaydon wrote:

Hello,

Thanks for the reply it’s working better with the dependencies.

Now we have a configuration issue, we configure sg_role like that:

authc:

jwt_auth_domain:

enabled: true

order: 0

http_authenticator:

type: jwt

challenge: false

config:

signing_key: |-

-----BEGIN PUBLIC KEY-----

MIIBCgKCAQEAokkT24WSNnOf0x6Y+C93qDnsweyC13QFddDBwoxXrZ6z+9GVvZTr

+LQ05Vwyh8GE80fzfyqFvdk5ghXbKcEKXqInq60/ljy7L+Q1iVnev3RAFBHaSH6/

revQLHMTW/Bjukelm258hniKPfLMtmNq3XhGPIz6VMKJ8DBQ2XZH0mAGRFqhU3fM

T7C+yt2oO4KV01vA4Zl1hS/SR+RKNdSPjPHihe1mRSG80ZnPeGbDdLmLZ6sj6DBk

HoDAeXrRwyijSsIlGkAb+BQxxzG/2DdiZOKbLodxVk5/ORcAzxOCONxN1mZQ9Nny

ghutASoP1v59Ys7U2avG4igeU6W9HgXCXwIDAQAB

-----END PUBLIC KEY-----

jwt_header: null

jwt_url_parameter: “Authorization”

roles_key: “family_name”

subject_key: “cognito:username”

authentication_backend:

type: noop

and sg_role_mapping.yml:

sg_bband:

users:

bband

backendroles:

tenant1

we try to connect to ES

https://<ES_IP>:9200/?Authorization=eyJraWQiOiJoQ2ZcLzI0OEUwM2FWb1lRZ2Npa0t4YWtSc3FkdmpWMkYrR2xOR2wxOEpiVT0iLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJkNTUxYTE0NC0yM2M2LTQxNmYtOThhMC0xMTlmZjI3NmUzMGMiLCJlbWFpbF92ZXJpZmllZCI6dHJ1ZSwiaXNzIjoiaHR0cHM6XC9cL2NvZ25pdG8taWRwLmV1LXdlc3QtMS5hbWF6b25hd3MuY29tXC9ldS13ZXN0LTFfbkxSSXFvTUtyIiwiY29nbml0bzp1c2VybmFtZSI6ImJsYWlzZS5sdWdlb25AbmFncmEuY29tIiwiZ2l2ZW5fbmFtZSI6InRvdG8iLCJhdWQiOiI2cHBvdjlpZTdhcTk3NHV2c2F2cGZtamN2MSIsImV2ZW50X2lkIjoiYWNjZDNkMDItMTE4ZS0xMWU4LTgzYjYtYTk4ZTJkNDJhM2E5IiwidG9rZW5fdXNlIjoiaWQiLCJhdXRoX3RpbWUiOjE1MTg2MTY1MTEsImV4cCI6MTUxODYyMDExMSwiaWF0IjoxNTE4NjE2NTExLCJmYW1pbHlfbmFtZSI6InRlbmFudDEiLCJlbWFpbCI6ImJsYWlzZS5sdWdlb25AbmFncmEuY29tIn0.n_n_EFqXW2suyw0gcC2TdjzPpQSqOB9MjufiVKFfPpgOnzpnfU0Pys03kxGLdNlJYmkc7Um8cMeniXUXSNZ34n2IknoHHlg3b_VZGdUkdZASpziHnco6Mgl6ZEMteac9bfUvjmpXC3AUB9K1j9HDS9s-cS09qHkClajlxQ8w9SpAK3ffCZh2hPGtJnrDbAUkkYqGNOvK8mVYAJMoScP59hQlBW-1_TegPDyIAe_Pmee_v0QjQzeLPIOyxHuJtBiY1I0H0XnAEkxtQrLYUrBEkU4n8XpCLDdO6J38ePmlH2QjG9jYQJsMCNc-bqFQYSfiKSpPNoGX1G5e2xlJt5dyXg

And I have the following error

Authentication finally failed.

Regards,

Laurent.

laurent_gaydon · February 14, 2018, 3:20pm

Here is log:
[2018-02-14T15:15:23,087][DEBUG][c.f.s.a.BackendRegistry ] Try to extract auth creds from http jwt

[2018-02-14T15:15:23,089][DEBUG][c.f.d.a.h.j.HTTPJwtAuthenticator] Invalid or expired JWT token.

java.lang.IllegalArgumentException: Key bytes can only be specified for HMAC signatures. Please specify a PublicKey or PrivateKey instance.

at io.jsonwebtoken.lang.Assert.isTrue(Assert.java:38) ~[dlic-search-guard-auth-http-jwt-5.0-6-jar-with-dependencies.jar:?]

at io.jsonwebtoken.impl.DefaultJwtParser.parse(DefaultJwtParser.java:324) ~[dlic-search-guard-auth-http-jwt-5.0-6-jar-with-dependencies.jar:?]

at io.jsonwebtoken.impl.DefaultJwtParser.parse(DefaultJwtParser.java:481) ~[dlic-search-guard-auth-http-jwt-5.0-6-jar-with-dependencies.jar:?]

at io.jsonwebtoken.impl.DefaultJwtParser.parseClaimsJws(DefaultJwtParser.java:541) ~[dlic-search-guard-auth-http-jwt-5.0-6-jar-with-dependencies.jar:?]

at com.floragunn.dlic.auth.http.jwt.HTTPJwtAuthenticator.extractCredentials0(HTTPJwtAuthenticator.java:157) [dlic-search-guard-auth-http-jwt-5.0-6-jar-with-dependencies.jar:?]

at com.floragunn.dlic.auth.http.jwt.HTTPJwtAuthenticator.access$000(HTTPJwtAuthenticator.java:46) [dlic-search-guard-auth-http-jwt-5.0-6-jar-with-dependencies.jar:?]

at com.floragunn.dlic.auth.http.jwt.HTTPJwtAuthenticator$1.run(HTTPJwtAuthenticator.java:122) [dlic-search-guard-auth-http-jwt-5.0-6-jar-with-dependencies.jar:?]

at com.floragunn.dlic.auth.http.jwt.HTTPJwtAuthenticator$1.run(HTTPJwtAuthenticator.java:119) [dlic-search-guard-auth-http-jwt-5.0-6-jar-with-dependencies.jar:?]

at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0_162]

at com.floragunn.dlic.auth.http.jwt.HTTPJwtAuthenticator.extractCredentials(HTTPJwtAuthenticator.java:119) [dlic-search-guard-auth-http-jwt-5.0-6-jar-with-dependencies.jar:?]

at com.floragunn.searchguard.auth.BackendRegistry.authenticate(BackendRegistry.java:428) [search-guard-5-5.6.6-18.jar:?]

at com.floragunn.searchguard.filter.SearchGuardRestFilter.checkAndAuthenticateRequest(SearchGuardRestFilter.java:113) [search-guard-5-5.6.6-18.jar:?]

at com.floragunn.searchguard.filter.SearchGuardRestFilter.access$000(SearchGuardRestFilter.java:45) [search-guard-5-5.6.6-18.jar:?]

at com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:69) [search-guard-5-5.6.6-18.jar:?]

at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:262) [elasticsearch-5.6.6.jar:5.6.6]

at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:200) [elasticsearch-5.6.6.jar:5.6.6]

at com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:60) [search-guard-ssl-5.6.6-23.jar:5.6.6-23]

at org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:505) [transport-netty4-client-5.6.6.jar:5.6.6]

at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:80) [transport-netty4-client-5.6.6.jar:5.6.6]

at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.13.Final.jar:4.1.13.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]

at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68) [transport-netty4-client-5.6.6.jar:5.6.6]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]

at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.13.Final.jar:4.1.13.Final]

at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.13.Final.jar:4.1.13.Final]