Let's Encrypt renewal - restart required?

Search Guard
1

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version

  • Used enterprise modules, if any

  • JVM version and operating system version

  • Search Guard configuration files

  • Elasticsearch log messages on debug level

Hi,

I’m working on upgrading our 2.x cluster to 5.5.1, and searchguard-ssl therefore to 5.5.1-23, and I see that PEM certs are now usable in recent versions of searchguard-ssl (with a mention of letsencrypt in the example config).

Will cert renewals require a restart of each Elasticsearch instance? Or will the file changing on disk cause searchguard-ssl to re-read?

If there are any other good practices and/or gotchas to think about here, I’d love to hear them.

Thanks in advance,

Tom

2

Yes, any changes to the certificates require a restart of your node(s) in order for Search Guard to pick up the new certificates. The certs are loaded on start up of the node(s).

···

On Wednesday, August 16, 2017 at 7:50:19 AM UTC+2, Tom Ryan wrote:

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version
  • Used enterprise modules, if any
  • JVM version and operating system version
  • Search Guard configuration files
  • Elasticsearch log messages on debug level

Hi,

I’m working on upgrading our 2.x cluster to 5.5.1, and searchguard-ssl therefore to 5.5.1-23, and I see that PEM certs are now usable in recent versions of searchguard-ssl (with a mention of letsencrypt in the example config).

Will cert renewals require a restart of each Elasticsearch instance? Or will the file changing on disk cause searchguard-ssl to re-read?

If there are any other good practices and/or gotchas to think about here, I’d love to hear them.

Thanks in advance,

Tom

3

Thanks Jochen.

In that case, I’ll stick with using our internal PKI. I trust it a bit more than Let’s Encrypt anyway tbh. I do think it would be great ti have a way to reread/reload either the cert or searchguard-ssl itself, without doing a full node restart.

Regards,

Tom

···

On Thursday, 17 August 2017 07:00:51 UTC+10, Jochen Kressin wrote:

Yes, any changes to the certificates require a restart of your node(s) in order for Search Guard to pick up the new certificates. The certs are loaded on start up of the node(s).

On Wednesday, August 16, 2017 at 7:50:19 AM UTC+2, Tom Ryan wrote:

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version
  • Used enterprise modules, if any
  • JVM version and operating system version
  • Search Guard configuration files
  • Elasticsearch log messages on debug level

Hi,

I’m working on upgrading our 2.x cluster to 5.5.1, and searchguard-ssl therefore to 5.5.1-23, and I see that PEM certs are now usable in recent versions of searchguard-ssl (with a mention of letsencrypt in the example config).

Will cert renewals require a restart of each Elasticsearch instance? Or will the file changing on disk cause searchguard-ssl to re-read?

If there are any other good practices and/or gotchas to think about here, I’d love to hear them.

Thanks in advance,

Tom

4

Just out of curiosity - what is your use case for auto-reloading the certificates? I understand that the Let’s Encrypt certificates have an expiration date and you would need to renew them from time to time. Which would also mean a rolling restart of the cluster. However, automatically reloading the certificates would open a potential attack vector, which is why we decided against it, and rather support certificate revocation lists. How would you use an automatic certificate reload feature?

Thanks!

···

On Thursday, August 17, 2017 at 2:04:35 AM UTC+2, Tom Ryan wrote:

Thanks Jochen.

In that case, I’ll stick with using our internal PKI. I trust it a bit more than Let’s Encrypt anyway tbh. I do think it would be great ti have a way to reread/reload either the cert or searchguard-ssl itself, without doing a full node restart.

Regards,

Tom

On Thursday, 17 August 2017 07:00:51 UTC+10, Jochen Kressin wrote:

Yes, any changes to the certificates require a restart of your node(s) in order for Search Guard to pick up the new certificates. The certs are loaded on start up of the node(s).

On Wednesday, August 16, 2017 at 7:50:19 AM UTC+2, Tom Ryan wrote:

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version
  • Used enterprise modules, if any
  • JVM version and operating system version
  • Search Guard configuration files
  • Elasticsearch log messages on debug level

Hi,

I’m working on upgrading our 2.x cluster to 5.5.1, and searchguard-ssl therefore to 5.5.1-23, and I see that PEM certs are now usable in recent versions of searchguard-ssl (with a mention of letsencrypt in the example config).

Will cert renewals require a restart of each Elasticsearch instance? Or will the file changing on disk cause searchguard-ssl to re-read?

If there are any other good practices and/or gotchas to think about here, I’d love to hear them.

Thanks in advance,

Tom

5

Let’s Encrypt certs expire after 90 days, and are typically renewed after 60. The key, as you might expect, doesn’t change.

I mostly asked because I saw Let’s Encrypt mentioned in the latest example configs. After I thought about it a bit more, I decided it wasn’t for me because of the immense trust it places in public DNS (which mostly cannot be avoided, but since we control all the clients, in this case it can).

But to answer your questions: reload could avoid having to rolling restart every ~60 days, and I’d expect it to either watch the files on disk or be called after successful renewal by the certbot renewal client.

hth,

tom

···

On Saturday, 19 August 2017 06:44:16 UTC+10, Jochen Kressin wrote:

Just out of curiosity - what is your use case for auto-reloading the certificates? I understand that the Let’s Encrypt certificates have an expiration date and you would need to renew them from time to time. Which would also mean a rolling restart of the cluster. However, automatically reloading the certificates would open a potential attack vector, which is why we decided against it, and rather support certificate revocation lists. How would you use an automatic certificate reload feature?

Thanks!

On Thursday, August 17, 2017 at 2:04:35 AM UTC+2, Tom Ryan wrote:

Thanks Jochen.

In that case, I’ll stick with using our internal PKI. I trust it a bit more than Let’s Encrypt anyway tbh. I do think it would be great ti have a way to reread/reload either the cert or searchguard-ssl itself, without doing a full node restart.

Regards,

Tom

On Thursday, 17 August 2017 07:00:51 UTC+10, Jochen Kressin wrote:

Yes, any changes to the certificates require a restart of your node(s) in order for Search Guard to pick up the new certificates. The certs are loaded on start up of the node(s).

On Wednesday, August 16, 2017 at 7:50:19 AM UTC+2, Tom Ryan wrote:

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version
  • Used enterprise modules, if any
  • JVM version and operating system version
  • Search Guard configuration files
  • Elasticsearch log messages on debug level

Hi,

I’m working on upgrading our 2.x cluster to 5.5.1, and searchguard-ssl therefore to 5.5.1-23, and I see that PEM certs are now usable in recent versions of searchguard-ssl (with a mention of letsencrypt in the example config).

Will cert renewals require a restart of each Elasticsearch instance? Or will the file changing on disk cause searchguard-ssl to re-read?

If there are any other good practices and/or gotchas to think about here, I’d love to hear them.

Thanks in advance,

Tom

6

I have a similar use case. I’m building an internal PKI but intend to issue short-lived certs (like Let’s Encrypt); the idea is that by automating cert renewal and doing it often you can make sure the actual renewal process works and is tested, and it reduces the window of vulnerability if a cert is compromised. In the long run that also cuts down on the risk of “the certs expired and the cluster died all of a sudden since nobody noticed” scenarios.

The problem is that if cert renewals require a full restart, then that needs to be coordinated between nodes. If reloads can be done with no downtime, that means servers can be responsible for requesting renewal of their own cert on their own schedule. If downtime is involved then at minimum I’d have to use a fixed cron schedule to make sure nodes never restart at the same time, which is less ideal.

Why do you think cert reloads would open an attack vector? Surely if an attacker can change the certs on disk they can also restart Elasticsearch, assuming no egregious filesystem permissions issues.

···

On Tuesday, August 22, 2017 at 10:31:44 AM UTC+9, Tom Ryan wrote:

Let’s Encrypt certs expire after 90 days, and are typically renewed after 60. The key, as you might expect, doesn’t change.

I mostly asked because I saw Let’s Encrypt mentioned in the latest example configs. After I thought about it a bit more, I decided it wasn’t for me because of the immense trust it places in public DNS (which mostly cannot be avoided, but since we control all the clients, in this case it can).

But to answer your questions: reload could avoid having to rolling restart every ~60 days, and I’d expect it to either watch the files on disk or be called after successful renewal by the certbot renewal client.

hth,

tom

On Saturday, 19 August 2017 06:44:16 UTC+10, Jochen Kressin wrote:

Just out of curiosity - what is your use case for auto-reloading the certificates? I understand that the Let’s Encrypt certificates have an expiration date and you would need to renew them from time to time. Which would also mean a rolling restart of the cluster. However, automatically reloading the certificates would open a potential attack vector, which is why we decided against it, and rather support certificate revocation lists. How would you use an automatic certificate reload feature?

Thanks!

On Thursday, August 17, 2017 at 2:04:35 AM UTC+2, Tom Ryan wrote:

Thanks Jochen.

In that case, I’ll stick with using our internal PKI. I trust it a bit more than Let’s Encrypt anyway tbh. I do think it would be great ti have a way to reread/reload either the cert or searchguard-ssl itself, without doing a full node restart.

Regards,

Tom

On Thursday, 17 August 2017 07:00:51 UTC+10, Jochen Kressin wrote:

Yes, any changes to the certificates require a restart of your node(s) in order for Search Guard to pick up the new certificates. The certs are loaded on start up of the node(s).

On Wednesday, August 16, 2017 at 7:50:19 AM UTC+2, Tom Ryan wrote:

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version
  • Used enterprise modules, if any
  • JVM version and operating system version
  • Search Guard configuration files
  • Elasticsearch log messages on debug level

Hi,

I’m working on upgrading our 2.x cluster to 5.5.1, and searchguard-ssl therefore to 5.5.1-23, and I see that PEM certs are now usable in recent versions of searchguard-ssl (with a mention of letsencrypt in the example config).

Will cert renewals require a restart of each Elasticsearch instance? Or will the file changing on disk cause searchguard-ssl to re-read?

If there are any other good practices and/or gotchas to think about here, I’d love to hear them.

Thanks in advance,

Tom

7

We are currently looking into that and maybe we come up with certificate hot reloading, so stay tuned ...
At the moment it is not possible.

···

Am 13.09.2017 um 14:36 schrieb Hector Martin <marcan@marcan.st>:

I have a similar use case. I'm building an internal PKI but intend to issue short-lived certs (like Let's Encrypt); the idea is that by automating cert renewal and doing it often you can make sure the actual renewal process works and is tested, and it reduces the window of vulnerability if a cert is compromised. In the long run that also cuts down on the risk of "the certs expired and the cluster died all of a sudden since nobody noticed" scenarios.

The problem is that if cert renewals require a full restart, then that needs to be coordinated between nodes. If reloads can be done with no downtime, that means servers can be responsible for requesting renewal of their own cert on their own schedule. If downtime is involved then at minimum I'd have to use a fixed cron schedule to make sure nodes never restart at the same time, which is less ideal.

Why do you think cert reloads would open an attack vector? Surely if an attacker can change the certs on disk they can also restart Elasticsearch, assuming no egregious filesystem permissions issues.

On Tuesday, August 22, 2017 at 10:31:44 AM UTC+9, Tom Ryan wrote:
Let's Encrypt certs expire after 90 days, and are typically renewed after 60. The key, as you might expect, doesn't change.

I mostly asked because I saw Let's Encrypt mentioned in the latest example configs. After I thought about it a bit more, I decided it wasn't for me because of the immense trust it places in public DNS (which mostly cannot be avoided, but since we control all the clients, in this case it can).

But to answer your questions: reload could avoid having to rolling restart every ~60 days, and I'd expect it to either watch the files on disk or be called after successful renewal by the `certbot` renewal client.

hth,
tom

On Saturday, 19 August 2017 06:44:16 UTC+10, Jochen Kressin wrote:
Just out of curiosity - what is your use case for auto-reloading the certificates? I understand that the Let's Encrypt certificates have an expiration date and you would need to renew them from time to time. Which would also mean a rolling restart of the cluster. However, automatically reloading the certificates would open a potential attack vector, which is why we decided against it, and rather support certificate revocation lists. How would you use an automatic certificate reload feature?

Thanks!

On Thursday, August 17, 2017 at 2:04:35 AM UTC+2, Tom Ryan wrote:
Thanks Jochen.

In that case, I'll stick with using our internal PKI. I trust it a bit more than Let's Encrypt anyway tbh. I do think it would be great ti have a way to reread/reload either the cert or searchguard-ssl itself, without doing a full node restart.

Regards,
Tom

On Thursday, 17 August 2017 07:00:51 UTC+10, Jochen Kressin wrote:
Yes, any changes to the certificates require a restart of your node(s) in order for Search Guard to pick up the new certificates. The certs are loaded on start up of the node(s).

On Wednesday, August 16, 2017 at 7:50:19 AM UTC+2, Tom Ryan wrote:
When asking questions, please provide the following information:

* Search Guard and Elasticsearch version
* Used enterprise modules, if any
* JVM version and operating system version
* Search Guard configuration files
* Elasticsearch log messages on debug level

Hi,

I'm working on upgrading our 2.x cluster to 5.5.1, and searchguard-ssl therefore to 5.5.1-23, and I see that PEM certs are now usable in recent versions of searchguard-ssl (with a mention of letsencrypt in the example config).

Will cert renewals require a restart of each Elasticsearch instance? Or will the file changing on disk cause searchguard-ssl to re-read?

If there are any other good practices and/or gotchas to think about here, I'd love to hear them.

Thanks in advance,
Tom

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/d692488a-434b-4ab0-a2b9-193ed7ebfc6c%40googlegroups.com\.
For more options, visit https://groups.google.com/d/optout\.

8

Any news on this?

···

Den torsdag 28 december 2017 kl. 13:05:36 UTC+1 skrev Search Guard:

We are currently looking into that and maybe we come up with certificate hot reloading, so stay tuned …

At the moment it is not possible.

Am 13.09.2017 um 14:36 schrieb Hector Martin mar...@marcan.st:

I have a similar use case. I’m building an internal PKI but intend to issue short-lived certs (like Let’s Encrypt); the idea is that by automating cert renewal and doing it often you can make sure the actual renewal process works and is tested, and it reduces the window of vulnerability if a cert is compromised. In the long run that also cuts down on the risk of “the certs expired and the cluster died all of a sudden since nobody noticed” scenarios.

The problem is that if cert renewals require a full restart, then that needs to be coordinated between nodes. If reloads can be done with no downtime, that means servers can be responsible for requesting renewal of their own cert on their own schedule. If downtime is involved then at minimum I’d have to use a fixed cron schedule to make sure nodes never restart at the same time, which is less ideal.

Why do you think cert reloads would open an attack vector? Surely if an attacker can change the certs on disk they can also restart Elasticsearch, assuming no egregious filesystem permissions issues.

On Tuesday, August 22, 2017 at 10:31:44 AM UTC+9, Tom Ryan wrote:

Let’s Encrypt certs expire after 90 days, and are typically renewed after 60. The key, as you might expect, doesn’t change.

I mostly asked because I saw Let’s Encrypt mentioned in the latest example configs. After I thought about it a bit more, I decided it wasn’t for me because of the immense trust it places in public DNS (which mostly cannot be avoided, but since we control all the clients, in this case it can).

But to answer your questions: reload could avoid having to rolling restart every ~60 days, and I’d expect it to either watch the files on disk or be called after successful renewal by the certbot renewal client.

hth,

tom

On Saturday, 19 August 2017 06:44:16 UTC+10, Jochen Kressin wrote:

Just out of curiosity - what is your use case for auto-reloading the certificates? I understand that the Let’s Encrypt certificates have an expiration date and you would need to renew them from time to time. Which would also mean a rolling restart of the cluster. However, automatically reloading the certificates would open a potential attack vector, which is why we decided against it, and rather support certificate revocation lists. How would you use an automatic certificate reload feature?

Thanks!

On Thursday, August 17, 2017 at 2:04:35 AM UTC+2, Tom Ryan wrote:

Thanks Jochen.

In that case, I’ll stick with using our internal PKI. I trust it a bit more than Let’s Encrypt anyway tbh. I do think it would be great ti have a way to reread/reload either the cert or searchguard-ssl itself, without doing a full node restart.

Regards,

Tom

On Thursday, 17 August 2017 07:00:51 UTC+10, Jochen Kressin wrote:

Yes, any changes to the certificates require a restart of your node(s) in order for Search Guard to pick up the new certificates. The certs are loaded on start up of the node(s).

On Wednesday, August 16, 2017 at 7:50:19 AM UTC+2, Tom Ryan wrote:

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version
  • Used enterprise modules, if any
  • JVM version and operating system version
  • Search Guard configuration files
  • Elasticsearch log messages on debug level

Hi,

I’m working on upgrading our 2.x cluster to 5.5.1, and searchguard-ssl therefore to 5.5.1-23, and I see that PEM certs are now usable in recent versions of searchguard-ssl (with a mention of letsencrypt in the example config).

Will cert renewals require a restart of each Elasticsearch instance? Or will the file changing on disk cause searchguard-ssl to re-read?

If there are any other good practices and/or gotchas to think about here, I’d love to hear them.

Thanks in advance,

Tom

–
You received this message because you are subscribed to the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/d692488a-434b-4ab0-a2b9-193ed7ebfc6c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

9

So far we have not decided to support certificate hot reloading. In our opinion, this would introduce major security risks and we still think that root certificate rollover should be a deliberate, manual process.

···

On Thursday, May 17, 2018 at 11:47:20 AM UTC+2, Joakim Sundqvist wrote:

Any news on this?

Den torsdag 28 december 2017 kl. 13:05:36 UTC+1 skrev Search Guard:

We are currently looking into that and maybe we come up with certificate hot reloading, so stay tuned …

At the moment it is not possible.

Am 13.09.2017 um 14:36 schrieb Hector Martin mar...@marcan.st:

I have a similar use case. I’m building an internal PKI but intend to issue short-lived certs (like Let’s Encrypt); the idea is that by automating cert renewal and doing it often you can make sure the actual renewal process works and is tested, and it reduces the window of vulnerability if a cert is compromised. In the long run that also cuts down on the risk of “the certs expired and the cluster died all of a sudden since nobody noticed” scenarios.

The problem is that if cert renewals require a full restart, then that needs to be coordinated between nodes. If reloads can be done with no downtime, that means servers can be responsible for requesting renewal of their own cert on their own schedule. If downtime is involved then at minimum I’d have to use a fixed cron schedule to make sure nodes never restart at the same time, which is less ideal.

Why do you think cert reloads would open an attack vector? Surely if an attacker can change the certs on disk they can also restart Elasticsearch, assuming no egregious filesystem permissions issues.

On Tuesday, August 22, 2017 at 10:31:44 AM UTC+9, Tom Ryan wrote:

Let’s Encrypt certs expire after 90 days, and are typically renewed after 60. The key, as you might expect, doesn’t change.

I mostly asked because I saw Let’s Encrypt mentioned in the latest example configs. After I thought about it a bit more, I decided it wasn’t for me because of the immense trust it places in public DNS (which mostly cannot be avoided, but since we control all the clients, in this case it can).

But to answer your questions: reload could avoid having to rolling restart every ~60 days, and I’d expect it to either watch the files on disk or be called after successful renewal by the certbot renewal client.

hth,

tom

On Saturday, 19 August 2017 06:44:16 UTC+10, Jochen Kressin wrote:

Just out of curiosity - what is your use case for auto-reloading the certificates? I understand that the Let’s Encrypt certificates have an expiration date and you would need to renew them from time to time. Which would also mean a rolling restart of the cluster. However, automatically reloading the certificates would open a potential attack vector, which is why we decided against it, and rather support certificate revocation lists. How would you use an automatic certificate reload feature?

Thanks!

On Thursday, August 17, 2017 at 2:04:35 AM UTC+2, Tom Ryan wrote:

Thanks Jochen.

In that case, I’ll stick with using our internal PKI. I trust it a bit more than Let’s Encrypt anyway tbh. I do think it would be great ti have a way to reread/reload either the cert or searchguard-ssl itself, without doing a full node restart.

Regards,

Tom

On Thursday, 17 August 2017 07:00:51 UTC+10, Jochen Kressin wrote:

Yes, any changes to the certificates require a restart of your node(s) in order for Search Guard to pick up the new certificates. The certs are loaded on start up of the node(s).

On Wednesday, August 16, 2017 at 7:50:19 AM UTC+2, Tom Ryan wrote:

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version
  • Used enterprise modules, if any
  • JVM version and operating system version
  • Search Guard configuration files
  • Elasticsearch log messages on debug level

Hi,

I’m working on upgrading our 2.x cluster to 5.5.1, and searchguard-ssl therefore to 5.5.1-23, and I see that PEM certs are now usable in recent versions of searchguard-ssl (with a mention of letsencrypt in the example config).

Will cert renewals require a restart of each Elasticsearch instance? Or will the file changing on disk cause searchguard-ssl to re-read?

If there are any other good practices and/or gotchas to think about here, I’d love to hear them.

Thanks in advance,

Tom

–
You received this message because you are subscribed to the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/d692488a-434b-4ab0-a2b9-193ed7ebfc6c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

10

Ok, thanks for the response.

In our case we have decided to go for letsencrypt certificates on all our servers so for us it means that we will have to restart the server every 60 days to reload the https certificate, for the transport protocol we are using a self-signed cert with an expiration date far in the future.

···

Den fredag 18 maj 2018 kl. 01:11:25 UTC+2 skrev Jochen Kressin:

So far we have not decided to support certificate hot reloading. In our opinion, this would introduce major security risks and we still think that root certificate rollover should be a deliberate, manual process.

On Thursday, May 17, 2018 at 11:47:20 AM UTC+2, Joakim Sundqvist wrote:

Any news on this?

Den torsdag 28 december 2017 kl. 13:05:36 UTC+1 skrev Search Guard:

We are currently looking into that and maybe we come up with certificate hot reloading, so stay tuned …

At the moment it is not possible.

Am 13.09.2017 um 14:36 schrieb Hector Martin mar...@marcan.st:

I have a similar use case. I’m building an internal PKI but intend to issue short-lived certs (like Let’s Encrypt); the idea is that by automating cert renewal and doing it often you can make sure the actual renewal process works and is tested, and it reduces the window of vulnerability if a cert is compromised. In the long run that also cuts down on the risk of “the certs expired and the cluster died all of a sudden since nobody noticed” scenarios.

The problem is that if cert renewals require a full restart, then that needs to be coordinated between nodes. If reloads can be done with no downtime, that means servers can be responsible for requesting renewal of their own cert on their own schedule. If downtime is involved then at minimum I’d have to use a fixed cron schedule to make sure nodes never restart at the same time, which is less ideal.

Why do you think cert reloads would open an attack vector? Surely if an attacker can change the certs on disk they can also restart Elasticsearch, assuming no egregious filesystem permissions issues.

On Tuesday, August 22, 2017 at 10:31:44 AM UTC+9, Tom Ryan wrote:

Let’s Encrypt certs expire after 90 days, and are typically renewed after 60. The key, as you might expect, doesn’t change.

I mostly asked because I saw Let’s Encrypt mentioned in the latest example configs. After I thought about it a bit more, I decided it wasn’t for me because of the immense trust it places in public DNS (which mostly cannot be avoided, but since we control all the clients, in this case it can).

But to answer your questions: reload could avoid having to rolling restart every ~60 days, and I’d expect it to either watch the files on disk or be called after successful renewal by the certbot renewal client.

hth,

tom

On Saturday, 19 August 2017 06:44:16 UTC+10, Jochen Kressin wrote:

Just out of curiosity - what is your use case for auto-reloading the certificates? I understand that the Let’s Encrypt certificates have an expiration date and you would need to renew them from time to time. Which would also mean a rolling restart of the cluster. However, automatically reloading the certificates would open a potential attack vector, which is why we decided against it, and rather support certificate revocation lists. How would you use an automatic certificate reload feature?

Thanks!

On Thursday, August 17, 2017 at 2:04:35 AM UTC+2, Tom Ryan wrote:

Thanks Jochen.

In that case, I’ll stick with using our internal PKI. I trust it a bit more than Let’s Encrypt anyway tbh. I do think it would be great ti have a way to reread/reload either the cert or searchguard-ssl itself, without doing a full node restart.

Regards,

Tom

On Thursday, 17 August 2017 07:00:51 UTC+10, Jochen Kressin wrote:

Yes, any changes to the certificates require a restart of your node(s) in order for Search Guard to pick up the new certificates. The certs are loaded on start up of the node(s).

On Wednesday, August 16, 2017 at 7:50:19 AM UTC+2, Tom Ryan wrote:

When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version
  • Used enterprise modules, if any
  • JVM version and operating system version
  • Search Guard configuration files
  • Elasticsearch log messages on debug level

Hi,

I’m working on upgrading our 2.x cluster to 5.5.1, and searchguard-ssl therefore to 5.5.1-23, and I see that PEM certs are now usable in recent versions of searchguard-ssl (with a mention of letsencrypt in the example config).

Will cert renewals require a restart of each Elasticsearch instance? Or will the file changing on disk cause searchguard-ssl to re-read?

If there are any other good practices and/or gotchas to think about here, I’d love to hear them.

Thanks in advance,

Tom

–
You received this message because you are subscribed to the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/d692488a-434b-4ab0-a2b9-193ed7ebfc6c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

11

Why is a restart every 60 days a problem? You can do it in a rolling manner and should not experience any downtime.
Normally you need anyhow to take down nodes down from time to time to apply patches for the OS or the JVM.

···

Am 18.05.2018 um 08:33 schrieb Joakim Sundqvist <pokerjocke70@gmail.com>:

Ok, thanks for the response.

In our case we have decided to go for letsencrypt certificates on all our servers so for us it means that we will have to restart the server every 60 days to reload the https certificate, for the transport protocol we are using a self-signed cert with an expiration date far in the future.

Den fredag 18 maj 2018 kl. 01:11:25 UTC+2 skrev Jochen Kressin:
So far we have not decided to support certificate hot reloading. In our opinion, this would introduce major security risks and we still think that root certificate rollover should be a deliberate, manual process.

On Thursday, May 17, 2018 at 11:47:20 AM UTC+2, Joakim Sundqvist wrote:
Any news on this?

Den torsdag 28 december 2017 kl. 13:05:36 UTC+1 skrev Search Guard:
We are currently looking into that and maybe we come up with certificate hot reloading, so stay tuned ...
At the moment it is not possible.

> Am 13.09.2017 um 14:36 schrieb Hector Martin <mar...@marcan.st>:
>
> I have a similar use case. I'm building an internal PKI but intend to issue short-lived certs (like Let's Encrypt); the idea is that by automating cert renewal and doing it often you can make sure the actual renewal process works and is tested, and it reduces the window of vulnerability if a cert is compromised. In the long run that also cuts down on the risk of "the certs expired and the cluster died all of a sudden since nobody noticed" scenarios.
>
> The problem is that if cert renewals require a full restart, then that needs to be coordinated between nodes. If reloads can be done with no downtime, that means servers can be responsible for requesting renewal of their own cert on their own schedule. If downtime is involved then at minimum I'd have to use a fixed cron schedule to make sure nodes never restart at the same time, which is less ideal.
>
> Why do you think cert reloads would open an attack vector? Surely if an attacker can change the certs on disk they can also restart Elasticsearch, assuming no egregious filesystem permissions issues.
>
> On Tuesday, August 22, 2017 at 10:31:44 AM UTC+9, Tom Ryan wrote:
> Let's Encrypt certs expire after 90 days, and are typically renewed after 60. The key, as you might expect, doesn't change.
>
> I mostly asked because I saw Let's Encrypt mentioned in the latest example configs. After I thought about it a bit more, I decided it wasn't for me because of the immense trust it places in public DNS (which mostly cannot be avoided, but since we control all the clients, in this case it can).
>
> But to answer your questions: reload could avoid having to rolling restart every ~60 days, and I'd expect it to either watch the files on disk or be called after successful renewal by the `certbot` renewal client.
>
> hth,
> tom
>
> On Saturday, 19 August 2017 06:44:16 UTC+10, Jochen Kressin wrote:
> Just out of curiosity - what is your use case for auto-reloading the certificates? I understand that the Let's Encrypt certificates have an expiration date and you would need to renew them from time to time. Which would also mean a rolling restart of the cluster. However, automatically reloading the certificates would open a potential attack vector, which is why we decided against it, and rather support certificate revocation lists. How would you use an automatic certificate reload feature?
>
> Thanks!
>
> On Thursday, August 17, 2017 at 2:04:35 AM UTC+2, Tom Ryan wrote:
> Thanks Jochen.
>
> In that case, I'll stick with using our internal PKI. I trust it a bit more than Let's Encrypt anyway tbh. I do think it would be great ti have a way to reread/reload either the cert or searchguard-ssl itself, without doing a full node restart.
>
> Regards,
> Tom
>
> On Thursday, 17 August 2017 07:00:51 UTC+10, Jochen Kressin wrote:
> Yes, any changes to the certificates require a restart of your node(s) in order for Search Guard to pick up the new certificates. The certs are loaded on start up of the node(s).
>
>
>
> On Wednesday, August 16, 2017 at 7:50:19 AM UTC+2, Tom Ryan wrote:
> When asking questions, please provide the following information:
>
> * Search Guard and Elasticsearch version
> * Used enterprise modules, if any
> * JVM version and operating system version
> * Search Guard configuration files
> * Elasticsearch log messages on debug level
>
> Hi,
>
> I'm working on upgrading our 2.x cluster to 5.5.1, and searchguard-ssl therefore to 5.5.1-23, and I see that PEM certs are now usable in recent versions of searchguard-ssl (with a mention of letsencrypt in the example config).
>
> Will cert renewals require a restart of each Elasticsearch instance? Or will the file changing on disk cause searchguard-ssl to re-read?
>
> If there are any other good practices and/or gotchas to think about here, I'd love to hear them.
>
> Thanks in advance,
> Tom
>
> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/d692488a-434b-4ab0-a2b9-193ed7ebfc6c%40googlegroups.com\.
> For more options, visit https://groups.google.com/d/optout\.

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search-guard@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/b89eb71c-5a58-4dfc-8606-a9190a6158b9%40googlegroups.com\.
For more options, visit https://groups.google.com/d/optout\.

12

Well, it does not have to be a problem.
We will for sure do automatic restarts to patch the servers in a rolling manner and roll the certificates on restart but sometimes the only thing changed is the certificate and then it would be nice to have a hot reload of the cert.

···

Den fredag 18 maj 2018 kl. 09:16:37 UTC+2 skrev Search Guard:

Why is a restart every 60 days a problem? You can do it in a rolling manner and should not experience any downtime.

Normally you need anyhow to take down nodes down from time to time to apply patches for the OS or the JVM.

Am 18.05.2018 um 08:33 schrieb Joakim Sundqvist pokerj...@gmail.com:

Ok, thanks for the response.

In our case we have decided to go for letsencrypt certificates on all our servers so for us it means that we will have to restart the server every 60 days to reload the https certificate, for the transport protocol we are using a self-signed cert with an expiration date far in the future.

Den fredag 18 maj 2018 kl. 01:11:25 UTC+2 skrev Jochen Kressin:

So far we have not decided to support certificate hot reloading. In our opinion, this would introduce major security risks and we still think that root certificate rollover should be a deliberate, manual process.

On Thursday, May 17, 2018 at 11:47:20 AM UTC+2, Joakim Sundqvist wrote:

Any news on this?

Den torsdag 28 december 2017 kl. 13:05:36 UTC+1 skrev Search Guard:

We are currently looking into that and maybe we come up with certificate hot reloading, so stay tuned …
At the moment it is not possible.

Am 13.09.2017 um 14:36 schrieb Hector Martin mar...@marcan.st:

I have a similar use case. I’m building an internal PKI but intend to issue short-lived certs (like Let’s Encrypt); the idea is that by automating cert renewal and doing it often you can make sure the actual renewal process works and is tested, and it reduces the window of vulnerability if a cert is compromised. In the long run that also cuts down on the risk of “the certs expired and the cluster died all of a sudden since nobody noticed” scenarios.

The problem is that if cert renewals require a full restart, then that needs to be coordinated between nodes. If reloads can be done with no downtime, that means servers can be responsible for requesting renewal of their own cert on their own schedule. If downtime is involved then at minimum I’d have to use a fixed cron schedule to make sure nodes never restart at the same time, which is less ideal.

Why do you think cert reloads would open an attack vector? Surely if an attacker can change the certs on disk they can also restart Elasticsearch, assuming no egregious filesystem permissions issues.

On Tuesday, August 22, 2017 at 10:31:44 AM UTC+9, Tom Ryan wrote:
Let’s Encrypt certs expire after 90 days, and are typically renewed after 60. The key, as you might expect, doesn’t change.

I mostly asked because I saw Let’s Encrypt mentioned in the latest example configs. After I thought about it a bit more, I decided it wasn’t for me because of the immense trust it places in public DNS (which mostly cannot be avoided, but since we control all the clients, in this case it can).

But to answer your questions: reload could avoid having to rolling restart every ~60 days, and I’d expect it to either watch the files on disk or be called after successful renewal by the certbot renewal client.

hth,
tom

On Saturday, 19 August 2017 06:44:16 UTC+10, Jochen Kressin wrote:
Just out of curiosity - what is your use case for auto-reloading the certificates? I understand that the Let’s Encrypt certificates have an expiration date and you would need to renew them from time to time. Which would also mean a rolling restart of the cluster. However, automatically reloading the certificates would open a potential attack vector, which is why we decided against it, and rather support certificate revocation lists. How would you use an automatic certificate reload feature?

Thanks!

On Thursday, August 17, 2017 at 2:04:35 AM UTC+2, Tom Ryan wrote:
Thanks Jochen.

In that case, I’ll stick with using our internal PKI. I trust it a bit more than Let’s Encrypt anyway tbh. I do think it would be great ti have a way to reread/reload either the cert or searchguard-ssl itself, without doing a full node restart.

Regards,
Tom

On Thursday, 17 August 2017 07:00:51 UTC+10, Jochen Kressin wrote:
Yes, any changes to the certificates require a restart of your node(s) in order for Search Guard to pick up the new certificates. The certs are loaded on start up of the node(s).

On Wednesday, August 16, 2017 at 7:50:19 AM UTC+2, Tom Ryan wrote:
When asking questions, please provide the following information:

  • Search Guard and Elasticsearch version
  • Used enterprise modules, if any
  • JVM version and operating system version
  • Search Guard configuration files
  • Elasticsearch log messages on debug level

Hi,

I’m working on upgrading our 2.x cluster to 5.5.1, and searchguard-ssl therefore to 5.5.1-23, and I see that PEM certs are now usable in recent versions of searchguard-ssl (with a mention of letsencrypt in the example config).

Will cert renewals require a restart of each Elasticsearch instance? Or will the file changing on disk cause searchguard-ssl to re-read?

If there are any other good practices and/or gotchas to think about here, I’d love to hear them.

Thanks in advance,
Tom

–
You received this message because you are subscribed to the Google Groups “Search Guard Community Forum” group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/d692488a-434b-4ab0-a2b9-193ed7ebfc6c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups “Search Guard Community Forum” group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/b89eb71c-5a58-4dfc-8606-a9190a6158b9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.