I have a cluster of 3 master nodes and 8 data nodes. I’d like SG to give me some simple role based access to the cluster (including Kibana) but I am having a hard time getting it configured. It doesn’t help that my knowledge of SSL and certificates is thin!
I use Ansible to build an AMI for master nodes and another AMI for the data nodes. I then use SpotInst to launch 3 copies of the master in 1 group and 8 copies of the data nodes in another group. This builds an unprotected cluster that is resilient to outages but I need some authentication.
After following the basic SG install instructions (using the offline TLS tool) I’ve finally managed to get the nodes to actually start but I’m getting the error:
“SSL Problem Received fatal alert: certificate_unknown”
I think this is (at least) because I have incorrectly configured the nodes section in the tlstool/config (and probably because I’m creating different root-ca files on master and data nodes) e.g.
But here I am little stuck on how to progress. At the time of creation of the AMI I do not know what IP or DNS will be given to any node (master or data) when it is started by SpotInst. I can get this information when the node is started and run the script at that point, but I will not have any of the information of any of the other nodes at that time. Further at any time one of my nodes may be removed and replaced by a new node and I need that to be attached to the cluster.
I am starting to wonder if this is even possible with SG. Perhaps I should just stick nginx in front and let that do the authentication.
— System details
AWS M3, M4 or M5 either large, xlarge or 2xlarge (dynamically allocated by SpotInst)
The DNS or IP in the SAN section of the certificate is optional. If it is there, you can use advanced features like hostname verification (verifying the hostname against the DNS entries in the SAN section) or DNS lookups (resolving the hostname against the DNS). But these are not mandatory, so you don’t need to add IPs or hostnames in the SAN section.
“and probably because I’m creating different root-ca files on master and data nodes”
This, of course, is not possible. The whole idea of TLS is to establish a chain of trust, which means all participating TLS certificates need to be trusted. If your master node connects to your data node, it sends its certificate. The data node verifies this certificate against its configured root CA. If this is not the root CA the certificate was signed with, it will fail.
Why do you want to have different root CAs on your master and your data nodes? If this is a hard requirement, you will need to add both root CAs in a single root CA PEM file and configure this combined one in Search Guard.
···
On Wednesday, September 26, 2018 at 11:59:29 AM UTC+2, Martin wrote:
I’m using ES6.4.0 and SG tlstool 1.5.
I have a cluster of 3 master nodes and 8 data nodes. I’d like SG to give me some simple role based access to the cluster (including Kibana) but I am having a hard time getting it configured. It doesn’t help that my knowledge of SSL and certificates is thin!
I use Ansible to build an AMI for master nodes and another AMI for the data nodes. I then use SpotInst to launch 3 copies of the master in 1 group and 8 copies of the data nodes in another group. This builds an unprotected cluster that is resilient to outages but I need some authentication.
After following the basic SG install instructions (using the offline TLS tool) I’ve finally managed to get the nodes to actually start but I’m getting the error:
“SSL Problem Received fatal alert: certificate_unknown”
I think this is (at least) because I have incorrectly configured the nodes section in the tlstool/config (and probably because I’m creating different root-ca files on master and data nodes) e.g.
But here I am little stuck on how to progress. At the time of creation of the AMI I do not know what IP or DNS will be given to any node (master or data) when it is started by SpotInst. I can get this information when the node is started and run the script at that point, but I will not have any of the information of any of the other nodes at that time. Further at any time one of my nodes may be removed and replaced by a new node and I need that to be attached to the cluster.
I am starting to wonder if this is even possible with SG. Perhaps I should just stick nginx in front and let that do the authentication.
— System details
AWS M3, M4 or M5 either large, xlarge or 2xlarge (dynamically allocated by SpotInst)
“and probably because I’m creating different root-ca files on master and data nodes”
This, of course, is not possible. The whole idea of TLS is to establish a chain of trust, which means all participating TLS certificates need to be trusted. If your master node connects to your data node, it sends its certificate. The data node verifies this certificate against its configured root CA. If this is not the root CA the certificate was signed with, it will fail.
Why do you want to have different root CAs on your master and your data nodes? If this is a hard requirement, you will need to add both root CAs in a single root CA PEM file and configure this combined one in Search Guard.
Thanks for your reply.
This is mostly due to my novice approach to SG (all my previous experience with ES has ignored security entirely). I worked through a number of fairly obvious config issues and left this one because I was having trouble running the offline TLS tool with a pre-existing root-ca.pem and root-ca.key. As it turns out I think this may now be the main (only, fingers crossed) reason things aren’t working. I don’t want to have different a root CA.
The easiest way is to generate the root CA (and intermediate, but that is optional) and then as many other certs as you want.
First change your configuration and set a password for the root CA (pkPassword). You can also use “auto” here, but then you will need to copy and paste the auto generated password to the config after the first run. So let’s just set a fixed password for the moment.
Then run the tool with the -ca option to generate the root CA:
./sgtlstool.sh -ca -c …/config/example.yml
``
This will generate the root CA (and optional intermediate CA) in the output folder.
Now configure your node-, admin- and client certificates in the configuration file and let the tool generate them:
./sgtlstool.sh -crt -c …/config/example.yml
``
You can run this command as many times as you want. If you add certificates to the config, they will be generated. Already generated certificates will be left untouched. If you want to overwrite them you can do so by using the -o option.
PM UTC+2, Martin wrote:
···
“and probably because I’m creating different root-ca files on master and data nodes”
This, of course, is not possible. The whole idea of TLS is to establish a chain of trust, which means all participating TLS certificates need to be trusted. If your master node connects to your data node, it sends its certificate. The data node verifies this certificate against its configured root CA. If this is not the root CA the certificate was signed with, it will fail.
Why do you want to have different root CAs on your master and your data nodes? If this is a hard requirement, you will need to add both root CAs in a single root CA PEM file and configure this combined one in Search Guard.
Thanks for your reply.
This is mostly due to my novice approach to SG (all my previous experience with ES has ignored security entirely). I worked through a number of fairly obvious config issues and left this one because I was having trouble running the offline TLS tool with a pre-existing root-ca.pem and root-ca.key. As it turns out I think this may now be the main (only, fingers crossed) reason things aren’t working. I don’t want to have different a root CA.
Thanks. That got it working … sadly wasn’t the last issue but I’m moving forward.
···
On Friday, 28 September 2018 10:27:53 UTC+1, Jochen Kressin wrote:
The easiest way is to generate the root CA (and intermediate, but that is optional) and then as many other certs as you want.
First change your configuration and set a password for the root CA (pkPassword). You can also use “auto” here, but then you will need to copy and paste the auto generated password to the config after the first run. So let’s just set a fixed password for the moment.
Then run the tool with the -ca option to generate the root CA:
./sgtlstool.sh -ca -c …/config/example.yml
``
This will generate the root CA (and optional intermediate CA) in the output folder.
Now configure your node-, admin- and client certificates in the configuration file and let the tool generate them:
./sgtlstool.sh -crt -c …/config/example.yml
``
You can run this command as many times as you want. If you add certificates to the config, they will be generated. Already generated certificates will be left untouched. If you want to overwrite them you can do so by using the -o option.
PM UTC+2, Martin wrote:
“and probably because I’m creating different root-ca files on master and data nodes”
This, of course, is not possible. The whole idea of TLS is to establish a chain of trust, which means all participating TLS certificates need to be trusted. If your master node connects to your data node, it sends its certificate. The data node verifies this certificate against its configured root CA. If this is not the root CA the certificate was signed with, it will fail.
Why do you want to have different root CAs on your master and your data nodes? If this is a hard requirement, you will need to add both root CAs in a single root CA PEM file and configure this combined one in Search Guard.
Thanks for your reply.
This is mostly due to my novice approach to SG (all my previous experience with ES has ignored security entirely). I worked through a number of fairly obvious config issues and left this one because I was having trouble running the offline TLS tool with a pre-existing root-ca.pem and root-ca.key. As it turns out I think this may now be the main (only, fingers crossed) reason things aren’t working. I don’t want to have different a root CA.
What especially are you struggling with? We are always looking for ways to improve Search Guard and/or the documentation. So any input here is most welcome!
···
On Friday, September 28, 2018 at 11:35:19 AM UTC+2, Martin wrote:
Thanks. That got it working … sadly wasn’t the last issue but I’m moving forward.
On Friday, 28 September 2018 10:27:53 UTC+1, Jochen Kressin wrote:
The easiest way is to generate the root CA (and intermediate, but that is optional) and then as many other certs as you want.
First change your configuration and set a password for the root CA (pkPassword). You can also use “auto” here, but then you will need to copy and paste the auto generated password to the config after the first run. So let’s just set a fixed password for the moment.
Then run the tool with the -ca option to generate the root CA:
./sgtlstool.sh -ca -c …/config/example.yml
``
This will generate the root CA (and optional intermediate CA) in the output folder.
Now configure your node-, admin- and client certificates in the configuration file and let the tool generate them:
./sgtlstool.sh -crt -c …/config/example.yml
``
You can run this command as many times as you want. If you add certificates to the config, they will be generated. Already generated certificates will be left untouched. If you want to overwrite them you can do so by using the -o option.
PM UTC+2, Martin wrote:
“and probably because I’m creating different root-ca files on master and data nodes”
This, of course, is not possible. The whole idea of TLS is to establish a chain of trust, which means all participating TLS certificates need to be trusted. If your master node connects to your data node, it sends its certificate. The data node verifies this certificate against its configured root CA. If this is not the root CA the certificate was signed with, it will fail.
Why do you want to have different root CAs on your master and your data nodes? If this is a hard requirement, you will need to add both root CAs in a single root CA PEM file and configure this combined one in Search Guard.
Thanks for your reply.
This is mostly due to my novice approach to SG (all my previous experience with ES has ignored security entirely). I worked through a number of fairly obvious config issues and left this one because I was having trouble running the offline TLS tool with a pre-existing root-ca.pem and root-ca.key. As it turns out I think this may now be the main (only, fingers crossed) reason things aren’t working. I don’t want to have different a root CA.
As I have stated before security is fairly new to me, before now I was only using ES on internal projects. My most common use of security other than user/pass, is ssh rsa keys.
The SG issues I am having I can sum up with “How I do something I could do before now there’s a layer of security?”.
e.g.
It took me some googling to find this github answer (last comment, probably by yourself). That’s one of the most basic ES commands curl localhost:9200/_cluster/health but I was not able to work it out or find it in your docs.
I access the 3 master nodes via an AWS load balancer. The target group needs a health check. Without SG it worked with protocol=http, port=9200 url=/_cat/health. Now I need to change the protocol to https but I am struggling to work out how to get the certificate passed in as I could with curl. I am trying to get the right sgconfig to acknowledge the proxy and then make only this call be allowed anonymously.
Most of your docs assume a fairly good understanding of what certificates are and how they are used. Even if you don’t spell everything out in your docs, links at relevant places to docs/wiki you trust that explain the concepts concisely would be invaluable. For example I don’t understand why when using curl I need to have both the admin key and the admin cert and also specify -k to make it unsecure. I don’t really understand what a PEM file is or even if I need to know to get my cluster working correctly. I’ve just been using them like a big user/pass that’s stored in a file. I was happy to note that even though they both look like any other RSA key they can’t be used in the wrong places. I have absolutely no idea to test if the final config is actually secure beyond the fact that I am having so much trouble talking to it.
The other issues are mostly self inflicted! I have a reasonably complex configuration and there are bits that are quite new to me.
I am configuring everything using Ansible. This is good but adds a layer of abstraction.
We use SpotInst. This manages the allocation of instances within our VPC but means there is little scope for fixed ips and adds another layer of abstraction.
I have configured ES with 3 master nodes and 8 data nodes, sat behind a load balancer (which is using SSL). I want to attach Logstash and Kibana (and add some user roles) and have them talk through the LB.
As always there’s never enough time, so I’m trying to get it working without fully understanding everything as that takes too long.
Specifically, if you could tell me how to fix the AWS Target Group Health Checks issue that would be amazing as I am really unsure of my approach, let alone the specifics of the config files.
Thanks for the support. I may persuade management that an SG support contract is the best way to go, but that will not be easy!
···
On Monday, 1 October 2018 15:17:14 UTC+1, Jochen Kressin wrote:
What especially are you struggling with? We are always looking for ways to improve Search Guard and/or the documentation. So any input here is most welcome!
thanks very much for your input, this is most valuable. Yes, TLS in itself can be quite complex, and you need some understanding of it to configure it correctly. This is where most users that are not familiar with it struggle … I think gathering some more information sources like other websites or blogs and adding them to the documentation is a good idea, we will work on that.
We have also started a TLS series on our own blog, the first one came out last month:
We will follow up with articles regarding certificates and how to use OpenSSL to generate and/or check certificates.
Regarding the AWS question: Instead of the _cluster/health endpoint, which is protected by Search Guard, you can simply use the SG built-in health check which is accessible without any credentials like user/pass or a certificate:
In order to improve our docs: Did you find the piece about the SG health check in the docs, e.g. by using the search? Was the documentation unclear or too vague?
···
On Monday, October 1, 2018 at 6:17:20 PM UTC+2, Martin wrote:
As I have stated before security is fairly new to me, before now I was only using ES on internal projects. My most common use of security other than user/pass, is ssh rsa keys.
The SG issues I am having I can sum up with “How I do something I could do before now there’s a layer of security?”.
e.g.
It took me some googling to find this github answer (last comment, probably by yourself). That’s one of the most basic ES commands curl localhost:9200/_cluster/health but I was not able to work it out or find it in your docs.
I access the 3 master nodes via an AWS load balancer. The target group needs a health check. Without SG it worked with protocol=http, port=9200 url=/_cat/health. Now I need to change the protocol to https but I am struggling to work out how to get the certificate passed in as I could with curl. I am trying to get the right sgconfig to acknowledge the proxy and then make only this call be allowed anonymously.
Most of your docs assume a fairly good understanding of what certificates are and how they are used. Even if you don’t spell everything out in your docs, links at relevant places to docs/wiki you trust that explain the concepts concisely would be invaluable. For example I don’t understand why when using curl I need to have both the admin key and the admin cert and also specify -k to make it unsecure. I don’t really understand what a PEM file is or even if I need to know to get my cluster working correctly. I’ve just been using them like a big user/pass that’s stored in a file. I was happy to note that even though they both look like any other RSA key they can’t be used in the wrong places. I have absolutely no idea to test if the final config is actually secure beyond the fact that I am having so much trouble talking to it.
The other issues are mostly self inflicted! I have a reasonably complex configuration and there are bits that are quite new to me.
I am configuring everything using Ansible. This is good but adds a layer of abstraction.
We use SpotInst. This manages the allocation of instances within our VPC but means there is little scope for fixed ips and adds another layer of abstraction.
I have configured ES with 3 master nodes and 8 data nodes, sat behind a load balancer (which is using SSL). I want to attach Logstash and Kibana (and add some user roles) and have them talk through the LB.
As always there’s never enough time, so I’m trying to get it working without fully understanding everything as that takes too long.
Specifically, if you could tell me how to fix the AWS Target Group Health Checks issue that would be amazing as I am really unsure of my approach, let alone the specifics of the config files.
Thanks for the support. I may persuade management that an SG support contract is the best way to go, but that will not be easy!
On Monday, 1 October 2018 15:17:14 UTC+1, Jochen Kressin wrote:
What especially are you struggling with? We are always looking for ways to improve Search Guard and/or the documentation. So any input here is most welcome!
Regarding the AWS question: Instead of the _cluster/health endpoint, which is protected by Search Guard, you can simply use the SG built-in health check which is accessible without any credentials like user/pass or a certificate:
In order to improve our docs: Did you find the piece about the SG health check in the docs, e.g. by using the search? Was the documentation unclear or too vague?
I did find that doc and used that end point to prove SG was configured correctly, I just did not think to use it as the health check! Sometimes the obvious answer is not the easiest to come to.
Thanks for your help. If I have other questions arising from the configuration of the roles/proxy I will start a new thread. The responsiveness from SG has given me confidence that it is the right choice and not just because the basic security is free.