29 Jun, 2020
What Happens When Your SSL Certificate's Trust Chain Breaks
Detecting breaks in your SSL certificate's trust chain can be tricky, as only some of the visitors to your site may be affected. Here's how to get to the root of the problem.
When SSL certificates stop working, it's usually easy to find the reason. Two causes account for the large majority of the cases. Either the certificate has expired and wasn't renewed, or it's incorrectly configured for the domain it covers. The fix is to get a new certificate or install the existing one correctly.
Sometimes, though, it's more of a mystery. We encountered an odd case recently with one of our customers, a financial institution. We'll call it "Website X" for privacy reasons. A visitor reported a browser warning that the customer's certificate was expired. Most visitors had no problem.
Having an expired SSL certificate hurts a site's business. Customers who see a warning mostly don't know what it means, but it looks frightening. Has the site been hijacked? Is it dangerous to access it? To continue to the site, they have to click a button or checkbox that says they're willing to take the risk. Many of them won't.
Sometimes, though, it's more of a mystery. We encountered an odd case recently with one of our customers, a financial institution. We'll call it "Website X" for privacy reasons. A visitor reported a browser warning that the customer's certificate was expired. Most visitors had no problem.
Having an expired SSL certificate hurts a site's business. Customers who see a warning mostly don't know what it means, but it looks frightening. Has the site been hijacked? Is it dangerous to access it? To continue to the site, they have to click a button or checkbox that says they're willing to take the risk. Many of them won't.
The Chain of Trust
To understand what happened in this case, you need to know about the SSL/TLS chain of trust. To prove that your certificate belongs to you, it has to be signed by a certificate authority (CA). The system uses public-private key pairs. The private key is carefully protected, while the public key is embedded in its certificate, which is available to everyone.
A CA uses its private key to sign a certificate, and you can verify the signature using its public key. But this just pushes the problem back a level. How do you know that the CA's certificate is authentic? Its certificate can be signed by another CA, but the same question remains.
It can't be "turtles all the way down." Ultimately, there has to be a certificate you just trust. It's called a root certificate. They're issued by trusted parties that have other ways of verifying their identity. There may be three or four levels, sometimes even more, on the chain of trust from a site's certificate to the root.
Root certificates are self-signed. This may come as a surprise since you've doubtless heard that self-signed certificates have no value. Root certificates, though, are delivered securely, so you can be confident about their authenticity. Every browser has a set of root certificates stored with it.
A CA uses its private key to sign a certificate, and you can verify the signature using its public key. But this just pushes the problem back a level. How do you know that the CA's certificate is authentic? Its certificate can be signed by another CA, but the same question remains.
It can't be "turtles all the way down." Ultimately, there has to be a certificate you just trust. It's called a root certificate. They're issued by trusted parties that have other ways of verifying their identity. There may be three or four levels, sometimes even more, on the chain of trust from a site's certificate to the root.
Root certificates are self-signed. This may come as a surprise since you've doubtless heard that self-signed certificates have no value. Root certificates, though, are delivered securely, so you can be confident about their authenticity. Every browser has a set of root certificates stored with it.
How the Trust Chain Breaks
As the proverb says, a chain is only as strong as its weakest link. If any of the certificates along the chain from a website go bad, it's no longer possible to verify the site's certificate. This doesn't happen very often, but it can. Even root certificates can stop being valid.
All SSL certificates have expiration dates. A CA can cease operations or stop issuing certificates. They usually give ample advance warning, but not everyone keeps up. When their certificates expire and aren't renewed, all the certificates signed with them become invalid.
This doesn't mean they'll stop working everywhere at once. Not all software is equally strict. Each client has its own set of root certificates. When the trust chain breaks, some clients may continue to accept the certificate while others reject it.
Modern browsers don't generally check the full chain. If they know that the first signing certificate is valid, they'll continue to trust it. This saves a lot of interaction with servers. Some users, though, will be affected and see browser warnings. This can be a confusing situation to diagnose. Quttera ThreatSign checks certificates thoroughly so that you know which ones have become invalid before your customers complain.
All SSL certificates have expiration dates. A CA can cease operations or stop issuing certificates. They usually give ample advance warning, but not everyone keeps up. When their certificates expire and aren't renewed, all the certificates signed with them become invalid.
This doesn't mean they'll stop working everywhere at once. Not all software is equally strict. Each client has its own set of root certificates. When the trust chain breaks, some clients may continue to accept the certificate while others reject it.
Modern browsers don't generally check the full chain. If they know that the first signing certificate is valid, they'll continue to trust it. This saves a lot of interaction with servers. Some users, though, will be affected and see browser warnings. This can be a confusing situation to diagnose. Quttera ThreatSign checks certificates thoroughly so that you know which ones have become invalid before your customers complain.
What Happened to Website X
Website X passed on a report to us that one of its regular visitors saw a notice that the site was using an expired SSL certificate. We looked to figure out what had happened. Website X had reinstalled all its certificates just a few months ago. X's IT people were positive there was nothing wrong with them.
They checked the HTTPS pages on their website. It was all accessible and showed no problems.
The next day, though, the same visitor again reported he couldn't access the website because of an expired certificate. The problem was real, yet we weren't seeing it.
A deeper exploration revealed that there was a trust chain problem. The certificate depended on the AddTrust External CA Root by way of a couple of intermediaries. That certificate was a legacy one that expired on May 30, 2020. It was still in use for the benefit of older browsers that don't have copies of the newest root certificates.
Certificates are commonly cross-signed by more than one CA. Up-to-date browsers can follow other chains of trust and validate the certificate. This is why X didn't get widespread reports of problems. The large majority of users were able to follow a more recent trust path and validate it. Older browsers knew only about the expired certificate.
In many cases, modern browsers don't routinely check the full trust chain; if the first stop on the chain was previously validated and hasn't expired, they don't check further. It's the customers with old computers whose browsers can no longer be updated who have a problem.
Only a small number of customers may be directly affected on their browsers, but the issue is bigger than that. Commercial Web filtering software and appliances check certificates more thoroughly. Office firewalls and intrusion prevention systems (IPS) will detect the expired root and report the certificate as untrusted.
They checked the HTTPS pages on their website. It was all accessible and showed no problems.
The next day, though, the same visitor again reported he couldn't access the website because of an expired certificate. The problem was real, yet we weren't seeing it.
A deeper exploration revealed that there was a trust chain problem. The certificate depended on the AddTrust External CA Root by way of a couple of intermediaries. That certificate was a legacy one that expired on May 30, 2020. It was still in use for the benefit of older browsers that don't have copies of the newest root certificates.
Certificates are commonly cross-signed by more than one CA. Up-to-date browsers can follow other chains of trust and validate the certificate. This is why X didn't get widespread reports of problems. The large majority of users were able to follow a more recent trust path and validate it. Older browsers knew only about the expired certificate.
In many cases, modern browsers don't routinely check the full trust chain; if the first stop on the chain was previously validated and hasn't expired, they don't check further. It's the customers with old computers whose browsers can no longer be updated who have a problem.
Only a small number of customers may be directly affected on their browsers, but the issue is bigger than that. Commercial Web filtering software and appliances check certificates more thoroughly. Office firewalls and intrusion prevention systems (IPS) will detect the expired root and report the certificate as untrusted.
Fixing the Problem
Certificates that used to depend on the AddTrust External CA Root have been updated to use a different legacy root, one that will remain valid for years. Simply updating and reinstalling any certificates that have the issue will fix it.
The hard part is detecting the problem. Most users won't notice anything wrong, and it's easy to dismiss occasional complaints as user problems. Quttera's ThreatSign service includes, among many other website security benefits, SSL validation to let you make sure your certificates are fully up to snuff. The best time to update problematic certificates is before customers notice. That avoids drops in traffic and questions about the site's reliability.
With ThreatSign, you get a comprehensive website security scanner, malware detection and removal, reports and removal of blacklisting, security metrics, and an API for automating security tasks. Contact us to learn how Quttera ThreatSign can make your site more reliable and secure.
The hard part is detecting the problem. Most users won't notice anything wrong, and it's easy to dismiss occasional complaints as user problems. Quttera's ThreatSign service includes, among many other website security benefits, SSL validation to let you make sure your certificates are fully up to snuff. The best time to update problematic certificates is before customers notice. That avoids drops in traffic and questions about the site's reliability.
With ThreatSign, you get a comprehensive website security scanner, malware detection and removal, reports and removal of blacklisting, security metrics, and an API for automating security tasks. Contact us to learn how Quttera ThreatSign can make your site more reliable and secure.