Let's Encrypt old root expiry and OpenSSL

[Angus Robertson 577]

For the last few years, Let;'s Encrypt issued free certificates whose intermediate was signed by an old root issued by Digital Signature Trust Co, that expires today.   In the ICS root bundles and certifica\te chain logs, it appears as follows: 

 

Issued to (CN): DST Root CA X3, (O): Digital Signature Trust Co.
Issuer: Self Signed
Expires: 2021-09-30T14:01:15, Signature: sha1WithRSAEncryption
Valid From: 2000-09-30T21:12:19, Serial Number: 44afb080d6a327ba893039862ef8406b
Fingerprint (sha256): 0687260331a72403d909f105e69bcf0d32e1bd2493ffc6d9206d11bcd6770739
Public Key: RSA Key Encryption 2048 bits, 112 security bits

 

Let;'s Encrypt distributed it's own root certificate,  Issued by (CN): ISRG Root X1, (O): Internet Security Research Group a few years ago, but older applications might not have been updated with it, so since then certificates issued by Let;'s Encrypt have two intermediates so that either root was acceptable.  Unfortunately not all application verify the chain correctly, including OpenSSL,  there was a blog about this two weeks ago, https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/

 

There are implications for servers running Let;'s Encrypt certificates and clients verifying the chains, last night one of my ICS client applications started giving chain verification errors on some of my servers, specifically those using Let;'s Encrypt certificates issued in July and due to expire in two weeks, but not those with certificates issued from mid August,  I'm still investigating what changed, I did make changes to the ICS functions that build certificate bundles, and Let;'s Encrypt periodically change the intermediates they issue. 

 

So this message is really a warning to watch out for failure to connect to web sites using Let;'s Encrypt certificates today.  More later.

 

Angus

[Angus Robertson 577]

After investigation, the main issue today was with the Windows IIS web server using Let;'s Encrypt certificates. 

 

The Windows Intermediate Certificate Authorities store had old certificates that it was still sending out with each request, according to the excellent SSL Labs test site.  Essentially, you only install new certificates in the store and old ones remain until removed manually using Admin Tools, Manage Computer Certificates, or the latest version of the ICS PemTools sampl;e which also allows deletion of certificates, which can now be done from applications as well.  IIS then sends any intermediates it finds matching for the server certificate. 

 

Browsers seem cleverer than OpenSSL in ignoring unwanted certificates, so the problem may not be that visible.  My IIS server has IPv4 and IPv6 binding on several IP addresses, and the issue did not appear on all bindings, possibly due to caching.  I had to reboot the server after deleting the unwanted certificates to stop IIS sending them, even after restarting IIS itself.

 

So if you have installed Let;'s Encrypt certificates into the Windows store, I'd recommend you deleted these old intermediates:

 

Issued to CN: R3, (O): Let's Encrypt

Issuer (CN): DST Root CA X3, (O): Digital Signature Trust Co.

Expires: 29/09/20213  

 

Issued to (CN): Let's Encrypt Authority X3, (O): Let's Encrypt
Issued by (CN): DST Root CA X3, (O): Digital Signature Trust Co.
Expires: 17/03/2021 16:40:46,

 

Issued to (CN): ISRG Root X1, (O): Internet Security Research Group
Issuer (CN): DST Root CA X3, (O): Digital Signature Trust Co.
Expires: 2024-09-30T18:14:03,

The last one is still being distributed by Let's Encrypt with new orders, and needs a change to ICS to remove it, but does not seem to give an error with OpenSSL.

 

Angus

 

 

 

 

[Vincent Parrett 763]

We had issues with some of our certificates, the fix was to renew them - the new certificates use the R3 root certificatge

 

https://techcrunch.com/2021/09/21/lets-encrypt-root-expiry/

[Angus Robertson 577]

Let's Encrypt started using R3 intermediates last December, there were three different versions since then, two signed by the expired root, which Windows IIS was still sending out, one expired this week but IIS still used it.

 

Angus

 

[Vincent Parrett 763]

I forgot there was somethings else I also did to fix this - install the new intermediate certificates on the servers. I don't remember having to do this before, but I was clutching at straws and trying things 😉 

[Angus Robertson 577]

Quote

I forgot there was somethings else I also did to fix this - install the new intermediate certificates on the servers.

This is all down how you install new certificates into the Windows Store, which has always been a black art.  You can double click on a PFX/P12 file, or do it from IIS Server Certificates which is better.  Both should install intermediates into the correct store, but may not, and won't remove old intermediates with the same name, that may still be sent with requests.

 

Which is one reason why ICS now has a new TMsCertTools class that allow installation of certificates to the Windows store. 

 

Angus

 

[Guest]

Well, you can grab the Microsoft official trusted root store directly using 

Quote

CertUtil –generateSSTFromWU RootStore.sst

or something like this

CertUtil –generateSSTFromWU D:\RootCerts\TrustedRootStore.sst

Then use the management console (MMC) to import them in trusted root or use the powershell to install them directly like this

https://news.ycombinator.com/item?id=10139967

or this

http://www.chawn.com/RootCerts.htm

[Guest]

OT: I told a client their wordpress used letsencrypt on the same domain while caring a lot about my certificate (on my recommendation). Granted, the information stored in my systems are way more sensitive but as it is the same domain i still felt i should inform them... when i read about this earlier this week i went in to check, and it seems they have now paid for a "named" certificate for their worpress subdomain.