Rotating Google-managed SSL certificates with zero downtime

Google, Cloud, DevOps Zero downtime rotation for google managed ssl certificates Posted on

Google-managed SSL certificates are TLS certificates that Google Cloud obtains and manages for your domains, renewing them automatically. These are Domain Validation (DV) certificates which support multiple hostnames (via SNI) in each certificate. The process is conceptually similar to Let's Encrypt.

Suppose I have a website like example.com that's using Google-managed SSL certificates on a Google Cloud Load Balancer, following these steps.

Now suppose I want to add another domain, example.co, to the certificate. The certificates are immutable, so we have to create another certificate with both of our domains on the certificate:

gcloud compute ssl-certificates create "example-com-certs-2" \
  --domains "example.com,example.co"

This will create the SSL certificate, but the SSL certificate will remain in the PROVISIONING state. In order for the DV validation to succeed, the example.co domain needs to point to the IP address of the load balancer which will serve that traffic. Update the DNS in your upstream provider for example.co to point to the IP address of the load balancer.

However, we cannot simply update the load balancer to use the new certificate. That would cause downtime as it can take up to 60 minutes to provision the certificate and up to 30 more minutes for the certificate to be available on the load balancer. That means changing the certificate could result in up to 90 minutes of downtime.

Instead, we need to update the target HTTPS proxy to use both certificates simultaneously. This will give the DV validation time to succeed without removing the already-provisioned certificates.

gcloud compute target-https-proxies update "example-com" \
  --ssl-certificates "projects/MY_PROJECT/global/sslCertificates/example-com-certs,projects/MY_PROJECT/global/sslCertificates/example-com-certs-2"

During this time, the initial certificate will remain green while the new certificate provisions. After the new certificate is finished provisioning, you can safely update the HTTPS proxy to remove the old one:

gcloud compute target-https-proxies update "example-com" \
  --ssl-certificates "projects/MY_PROJECT/global/sslCertificates/example-com-certs-2"

About Seth

Seth Vargo is an engineer at Google Cloud. Previously he worked at HashiCorp, Chef Software, CustomInk, and some Pittsburgh-based startups. He is the author of Learning Chef and is passionate about reducing inequality in technology. When he is not writing, working on open source, teaching, or speaking at conferences, Seth advises non-profits.