LetsEncrypt change breaks the internet: Quick fix is available for web servers (not obvious, though — cert.pem vs. fullchain.pem)


Something weird happened on the internet on Wednesday afternoon. Millions of people found out that their devices and websites were being affected by security problems.

A company called LetsEncrypt seems to be the world’s largest provider of SSL security certificates. And for some reason, they allowed one of their main certificates to expire. I won’t pretend to know what all the ins and out are, but for us, we were having problems on some devices but not others. More specifically, Windows computers were not showing any signs of a problem, but all of our Apple devices were giving us warnings that our connection was not secure.

I thought I had done something wrong, since I had updated our Ubuntu server just about 10 hours earlier that morning. But it did not add up as to why something I did at 5am broke something at around 3pm. I tried many solutions, but realized it couldn’t be my fault. So, I punched LetsEncrypt into the News search instead and — wow, “Internet goes down for millions, tech companies scramble as key encryption service expires.” Ah, at least it wasn’t my fault.

So, the problem area for us was a subdomain for a wildcard certificate that we were using to refer traffic to a 3rd Party service called FlipHtml5 that provides magazine-like presentations of PDFs.

www.OurDomain.com was working fine on all devices, but MAGAZINE.OurDomain.com was not working properly — not on iPhones, iPads or Mac laptops and computers. Windows devices and an Android were just fine.

We were using a web server at Linode with Ubuntu and Apache.

The solution was actually rather simple, but it was not at all obvious.

When using CertBot to create a LetsEncrypt certificate, it creates 4 files. In our situation, the references to the files were located at /etc/letsencrypt/live/OurDoman.com/

  • cert.pem
  • privkey.pem
  • chain.pem
  • fullchain.pem

And previously we only had to refer to the certificate under cert.pem.

Ah, but that’s not good enough anymore, and we needed to instead refer to the fullchain.pem instead.

So, to do that, there is an Apache2 configuration file at: /etc/apache2/sites-available/OurDomain.com.conf

Using pico to edit the file, there is a <VirtualHost> section for the SSL, and it contained this line:

SSLCertificateFile /etc/letsencrypt/live/OurDomain.com/cert.pem

so, change that to:


Then restart the Apache server with either or both:

systemctl restart apache2


/etc/init.d/apache2 restart

And all should be better and working.

But for us, we still had this 3rd party service, FlipHTML5, which wasn’t working right.

It turns out, we had to go into the Settings section, and change the “Certificate” section from the contents of the cert.pem file to the contents of the fullchain.pem file also.

And, “hooray! it works again!”