[THTTPRIO, 10.4.2] WinHttpSendRequest + client certificate authentication

[jaenicke 13]

Hello,

 

I would like to use THTTPRIO / Soap and thus TWinHTTPClient with WinHttpSendRequest. Client side certificates are used for authentication. This all works in the old version under XE4 with Indy. There you can also load a certificate from a file etc. without installing it.

 

Now I want to update this to Delphi 10.4.2. Unfortunately I found out that TWinHTTPClient is declared under implementation without Rtti information and uses hard-coded the MY store. So I see three possibilities: Load the certificate to MY store, bypass MY store and load a certificate directly as context, bypass MY store and load a certificate as store and fetch the context.

 

So I have redirected the virtual method DoClientCertificateAccepted by VMT pointer, so that I can use WinHttpSetOption with WINHTTP_OPTION_CLIENT_CERT_CONTEXT. The fact that the reference to the request is not accessible from outside is no big deal either. No beautiful code, but at least it works and is easy to adapt for future versions of Delphi.

 

The problem now is to get a certificate context. I get it loaded with CertCreateCertificateContext from a .pfx file. Unfortunately the link to the private key is missing. I haven't found a way to add it with e.g. CertSetCertificateContextProperty. Consequently, I get the error message that the private key is missing when I make the request.

 

So I tried it with PFXImportCertStore. This way I can load the .pfx file too, can get the context, but unfortunately the private key seems not to be in it, at least after the call of CertSetCertificateContextProperty I get the value 80092004 when I call GetLastError. That means that the property was not found.

 

Also a try with the Clever Internet Suite did not work.

 

Long story short:
I have a .p12 file as source, I have a .pem file with the certificate and a .key file with the private key, I have also already made a .pfx out of it. With this I would like to do the authentication via client certificate. Which way would you recommend and how can I do it?

 

Thanks a lot
Greetings
Sebastian

Edited July 23, 2021 by jaenicke

[jaenicke 13]

No idea? Nobody?

 

Well, in the meanwhile I checked the certicate files using "openssl -v -info" and they contain the private key as they should.

 

I imported the .pfx file by using certlm.msc directly. The import works and the private key is visible there. The certificate is found by TWinHTTPClient correctly and I can select it in OnNeedClientCertificate.

 

But nevertheless the result is the same:

I get error 12185 (ERROR_WINHTTP_CLIENT_CERT_NO_PRIVATE_KEY).

 

Is there anything that needs to be done in addition?

Edited July 26, 2021 by jaenicke

[jaenicke 13]

SOLVED

 

Unfortunately I got no hint for an error when calling PFXImportCertStore, but the format of the parameter was wrong. This does not explain why the other way using the store did not work, but this would have been only a workaround anyway.

 

What I can say at this point:
The implementation of Embarcadero really is not good. There is already working code, but one cannot use it, because the functionality is completely unreachable.

Edited July 26, 2021 by jaenicke

[Angus Robertson 574]

I'm just adding import and export for certificates to and from the Windows store, for ICS. 

 

PFXImportCertStore and PFXExportCertStoreEx seem to be preferred solution to convert to and from a PFX/PCKS12 blob containing a certificate, private key and intermediates. 

 

Beware you won't be able to export from TPMs, only certificates saved with exportable private keys.

 

But since you have PFX and PEM files, not sure why you are using the Windows store.

 

Angus

 

[jaenicke 13]

14 hours ago, Angus Robertson said:

But since you have PFX and PEM files, not sure why you are using the Windows store.

Because I did not find another way to apply the certificate including the private key to the request.

 

Should CertCreateCertificateContext work with the contents of a .pfx file including the private key too? It did not work for me and I did not find anything except the hint to use PFXImportCertStore. Perhaps I did something wrong there too.

 

14 hours ago, Angus Robertson said:

PFXImportCertStore and PFXExportCertStoreEx seem to be preferred solution to convert to and from a PFX/PCKS12 blob containing a certificate, private key and intermediates.

That would be interesting. At the moment I use openssl.exe externally to convert .pem + .key --> .pfx. But I found no way to do this using the API.

 

If this conversion would not be necessary, it would be even better of course.

 

14 hours ago, Angus Robertson said:

I'm just adding import and export for certificates to and from the Windows store, for ICS. 

Could ICS be used somehow with THTTPRIO for SOAP communication including client certificate authentication? If this would work, it would be really great. But as there is no official way to change the engine behind THTTPRIO, it would not be easy (if possible at all). And as ICS works asynchronously (what I like), the process flow is so different, that it won't match the way THTTPRIO works anyway. So I have little hope. 😉

 

Thank you for your reply!

Sebastian

Edited July 27, 2021 by jaenicke

[Angus Robertson 574]

Currently ICS only uses the Certxx APIs to extract certificates from the Windows store and to validate certificate chains, I'm just adding bits to put certificates into the store, but won't be done this week.  My reading suggests CryptAcquireCertificatePrivateKey is how you get the private key for a certificate you find in the store, but not tried it yet. 

 

ICS has classes TX509Base and TX509List which allow certificates and bundles to be created, read and saved in various formats, there is sample OverbyteIcsPemtool that illustrates everything, and another OverbyteIcsX509CertsTst that acquires certificates from Let's Encrypt. including multi-domain wildcards. 

 

Sorry, never used THTTPRIO so no idea what it does, but the ICS HTTP client supports client certificates.  You can use ICS synchronously, there are methods for that.

 

Angus

 

 

[jaenicke 13]

I have one question left:

Is it possible to use ICS to build a HTTP server where clients authenticate themselves using client certificates? I had a short look, but did not see, whether this is possible.

 

On 7/27/2021 at 10:08 AM, Angus Robertson said:

My reading suggests CryptAcquireCertificatePrivateKey is how you get the private key for a certificate you find in the store, but not tried it yet.

I don't find a way using this function, but it already works, so it does not matter.

 

On 7/27/2021 at 10:08 AM, Angus Robertson said:

ICS has classes TX509Base and TX509List which allow certificates and bundles to be created, read and saved in various formats, there is sample OverbyteIcsPemtool that illustrates everything, and another OverbyteIcsX509CertsTst that acquires certificates from Let's Encrypt. including multi-domain wildcards.

This is very interesting indeed as I don't need to call openssl.exe externally anymore with this functionality. 

 

On 7/27/2021 at 10:08 AM, Angus Robertson said:

Sorry, never used THTTPRIO so no idea what it does, but the ICS HTTP client supports client certificates.  You can use ICS synchronously, there are methods for that.

THTTPRIO us used to use automatically generated client classes to communicate with a SOAP server. This way you do not need to build and analyze the XML contents manually. It would only be possible to change the engine behind by changing a few RTL units, so this is no option for general purpose.

 

Thank you very much!

Edited July 31, 2021 by jaenicke

[Angus Robertson 574]

Quote

Is it possible to use ICS to build a HTTP server where clients authenticate themselves using client certificates?

TSslWSocketServer has a property SslCliCertMethod which determines whether a client certificate is required or optional, you check the certificate in the OnSslHandshakeDone event and close the connection if invalid, it is documented on the wiki page, http://wiki.overbyte.eu/wiki/index.php/TWSocketServer.  Note I've not tested this for a while.

 

Angus

 

[jaenicke 13]

I found the demo application with TSslHttpAppSrv, where I have this property.

After I had a short look at the possibilities I have with those components, I think it is the best solution I have seen yet.

 

I'll have a deeper look at it soon.

 

Thank you again!

[Angus Robertson 574]

Quote

I'm just adding import and export for certificates to and from the Windows store, for ICS. 

ICS has new classes TMsCertTools and TMsX509List to write and read SSL/TLS certificates to and from the Windows Certificate Store, including private keys.  This is primarily so Let's Encrypt certificates can be installed automatically for use with the IIS web server. 

 

The PemTool sample includes new buttons to list all the Windows certificate and private key stores and allow old items to be deleted.

 

Most of this was straight forward, but Microsoft seems to have messed up the APIs when adding CNG support for ECDSA keys in Vista, keys and certificates are held in separate stores and the way they are linked together is badly documented and flaky, trying to set IIS site bindings often gives an error that means the key can not found.  I was only able to add certificates and private keys that can not be exported from Windows, the NCrypt functions fail. 

 

Angus

[jaenicke 13]

18 hours ago, Angus Robertson said:

ICS has new classes TMsCertTools and TMsX509List to write and read SSL/TLS certificates to and from the Windows Certificate Store, including private keys.

Thank you very much for your efforts. I'll take a look at it.

18 hours ago, Angus Robertson said:

keys and certificates are held in separate stores and the way they are linked together is badly documented and flaky

I had already begun to doubt myself...

 

In the meanwhile I found out, that the internal rest client of Delphi supports client based authentication... the problem is, that it can't be used, because there is no way to set the properties needed... So I made a quality entry:

https://quality.embarcadero.com/browse/RSP-34451

I redirected the virtual methods, so it works for me, but it is no good way, because everything is declared inside the implementation section. So having no RTTI I have to use the cpu view to find the VMT table and memory offsets to the addresses, which I have to redirect.

 

The next problem is, that I can use the .NET class X509Certificate2 to open a certificate, for which I can get the public key afterwards by simply using GetPublicKey(). But if I try this using Delphi and your ICS classes or using other ways, I always have to provide a password, which I do not have (and do not need when using .NET). I tried an empty password as well, but it did not work. I do not know, whether this does not work or whether I made an error when loading the certificate data.

[Angus Robertson 574]

X509 certificates never have a password or encryption, by definition they are public.  The private key used to sign an X509 certificate or use it in a server may be protected, so I assume you are opening a bundle file that contains both a certificate and a protected private key. 

 

For a PEM bundle file, the certificate and private key are separate blocks of text, so you only need the certificate and can get the public key from that.  A PFX/PKCS12 bundle, is a binary blob, and OpenSSL will try and read everything in it, and fail if the key is protected and you don't have the password.  I believe there are PKCS12 parsers to extract the contents of the file separately, but never looked for one.   The wincrypt API to read PKCS12 is the same, reads the lot and needs a password. 

 

Angus