How to use a certificate from the Windows certificate store

[Droesjba 0]

I have a client (Windows service) using TSslWSocket and a server (also a Windows service) using TSslWSocketServer. I'm trying to add SSL to the communication between the two services.

 

Step A:

In the server I'm using a certificate:

...SslContext.SslCertFile := 'C:\Temp\SSL\cert.pem';

So far, so good. In the OnSslVerifyPeer event of TSslWSocket I'm detecting that the certificate is self-signed which is correct.

 

Step B:

Instead of using ...SslContext.SslCertfile I'm using ...SslContext.SslCertX509

This SslCertX509 is retrieved from the Windows certificate store (see topic 

After finding the right certificate I define:

  FSslCertX509 := TX509Base.Create(nil);
  FSslCertX509 := TX509Base(lMsX509List);

  .....
  WSslSocketServer.SslContext.SslCertX509 := FSslCertX509;

 

The server gives an exception (access vioalation) at line 15182 of OverbyteIcsWSocket.pas (see code snippet below):

 

When I'm using the debugger and enter FSslCertX509.IsCertLoaded then the result is False, so no certificate seems to be loaded.

Nevertheless the statement at line 15812 seems to be executed. When I'm hoovering over "IsCertLoaded" at line 15811 I see:

"unknown exception 3402A6AC at 24748B14".

 

Any suggestions?

 

 

 

 

 

Edited October 24, 2022 by Droesjba

[Angus Robertson 526]

Can you please explain the purpose of using a certificate from the Windows store?  Is this in your client or server application, as a server or client certificate. Is your server or client giving an exception? 

 

Angus

 

[Droesjba 0]

The server gives an exception (access vioalation) at line 15182 of OverbyteIcsWSocket.pas
The certificate is a server certificate, the client uses no certificate.

[Angus Robertson 526]

So you are attempting to use a certificate from the Windows store as a server certificate? 

 

How did you set the private key for this certificate. You can not extract private keys from the Windows store?  Server certificates are useless without a private key.

 

Private keys are kept by Windows in a different store and linked to certificates, but not stored together.  Windows itself provides a means to export certificates with a private key, if allowed, but this is not implemented by ICS, not sure if the APIs allow it.  Most certificate have key export blocked to stop them being stolen. 

 

Angus

[Droesjba 0]

So you are attempting to use a certificate from the Windows store as a server certificate? 

Yes

 

How did you set the private key for this certificate. You can not extract private keys from the Windows store?  Server certificates are useless without a private key.

From the pem file I've created a pfx file including a private key and a password. I've imported this pfx file in Windows certificate store using MMC (Microsoft Management Console).

 

I use the LoadFromStore method to read from the Windows Certificate Store. This fills a TMsX509List in which I lookup the thumbprint (and find it).

That gives me the number of the certificate in the TMsX509List.

 

So the question is: What do I have to do to make use of the SslCertX509 property of TSslContext (from OverbyteIcsWSocket).

Making use of the SslCertFile property seems to work OK, why can't I reach the same result when using the SslCertX509 property?

 

[Angus Robertson 526]

No, the question is why if you already have a PFX file with a private key, you are installing it into the Windows Store in the first place, then reading it again without the private key and expecting it to work as a server certificate.  Why not use the PFX directly. 

 

The ICS pemtool loads certificates into the windows into the Windows Store correctly, I do it all the time to use Let's Encrypt certificates with IIS.  

 

The exception will be an error in your code.

 

Angus

 

[Droesjba 0]

Most likely the exception indeed is caused by an error in my code. I'm trying discover what I've done wrong. Have I forgotten something? 

The reason I don't use the pfx file is in the fact that our customers have to be able to use their own certificates. This is accomplished by specifying the thumbprint of the certificate the customer wants to use.

This is not something I have made up myself. Other programs from the company I work for act in the same way.

 

So specifying a thumbprint and looking it up in the Windows Certificate Store is imposed on me.

Perhaps you can answer if the following use of the SslCertX509 property should work or not:

 

After finding the right (i) certificate I define:

  FSslCertX509 := TX509Base.Create(nil);
  FSslCertX509 := TX509Base(lMsX509List);   sad but true: I've added square bracket i square bracket  here after lMsM509List but it disappears every time

   .....
  WSslSocketServer.SslContext.SslCertX509 := FSslCertX509;

Edited October 24, 2022 by Droesjba

[Angus Robertson 526]

We are going in circles here, even if you load the certificate into the context a subsequent error will happen since there is no private key with the certificate.  

 

The TX509Base and TMsX509List classes can hold a private key, but if you use PemTool to list a store you'll see it reports 'Private key in User Store, Could not export private key - The requested operation is not supported'.  Only the Microsoft crypto functions can access the private key store, and OpenSSL does not use them.

 

For your own code, you don't show a definition for lMsX509List nor how you are indexing into the list, but it should be something like FSslCertX509 := MyList{x]. or MyList.Items[x]

 

Angus

 

[Sander_ 0]

20 hours ago, Angus Robertson said:

No, the question is why if you already have a PFX file with a private key, you are installing it into the Windows Store in the first place, then reading it again without the private key and expecting it to work as a server certificate.  Why not use the PFX directly.

The certificate is in the Windows Certificate Store because that is the way our customers provide us the certificate on their servers. For developing purposes we created a selfsigned certificate with key, converted this into a pfx and imported it to the Windows Certificate Store. On a customer site they have already loaded a certificate in the store for us, we don't get the pfx or any other file.

 

19 hours ago, Angus Robertson said:

The TX509Base and TMsX509List classes can hold a private key, but if you use PemTool to list a store you'll see it reports 'Private key in User Store, Could not export private key - The requested operation is not supported'.  Only the Microsoft crypto functions can access the private key store, and OpenSSL does not use them.

I tried PemTool and from the six certificates I have in my Local Machine / Personal store I see that the Private Key can be exported of some of them.

Quote

#1 Friendly Name: <none>

#2 Friendly Name: Veeam self-signed certificate for Microsoft Azure plug-in, Key Name: fa2495c5-6738-4f68-8c49-0760d6b4f008 - Private key in User Store, Private key exported OK

#3 Friendly Name: Veeam Backup Server Certificate, Key Name: 1d3c416f-1d5f-4fdc-bb81-2a6e7fe28271 - Private key in User Store, Could not export private key - De gevraagde bewerking wordt niet ondersteund (#-2146893783)

#4 Friendly Name: Veeam self-signed certificate for AWS plug-in, Key Name: 9ead74f3-9372-46c7-92ff-ce2e39f7f6fb - Private key in User Store, Could not export private key - De gevraagde bewerking wordt niet ondersteund (#-2146893783)

#5 Friendly Name: IIS Express Development Certificate, Key Name: IIS Express Development Certificate Container - Private key in User Store, Private key exported OK

#6 Friendly Name: Veeam Backup Server Certificate, Key Name: 55f4cbd9-ba29-472a-a372-7c394cd9ce79 - Private key in User Store, Private key exported OK

These certificates had the Private Key marked as exportable is what I found out after checking them, the others not.

 

On 10/24/2022 at 12:12 PM, Angus Robertson said:

Private keys are kept by Windows in a different store and linked to certificates, but not stored together.  Windows itself provides a means to export certificates with a private key, if allowed, but this is not implemented by ICS, not sure if the APIs allow it.  Most certificate have key export blocked to stop them being stolen. 

You say that ICS cannot do this, but PemTool reports Private Key exported OK. Or are they exported in another way?

 

21 hours ago, Angus Robertson said:

Only the Microsoft crypto functions can access the private key store, and OpenSSL does not use them.

If we get the server certificate from the store and mark the private key as exportable it should be doable right?

[Angus Robertson 526]

ICS attempts to extract a private key from the Windows store, but I don't recall it working on any of my servers or PCs, so untested. 

 

So if it works, good, if not, as expected.  I don't plan on testing this again since it is not a core ICS function, not needed for any samples. 

 

Personally, I would not distribute an application that relies on extracting a private key from the Windows Store, you have to hope that whoever imported it ticked the correct box to allow export, and Microsoft has not changed the rules again.  I would tell the customer you need the PFX. to use with OpenSSL. 

 

But you easily test if your FSslCertX509 object has a matching key and can be used as a server certificate.  If you are using the recommend IcsHosts server properties, the certificate gets checked and reported before the server starts, so you know if it's going to work.

 

Angus

 

[uso 0]

On 10/25/2022 at 7:26 PM, Angus Robertson said:

ICS attempts to extract a private key from the Windows store, but I don't recall it working on any of my servers or PCs, so untested. 

 

So if it works, good, if not, as expected.  I don't plan on testing this again since it is not a core ICS function, not needed for any samples. 

 

Personally, I would not distribute an application that relies on extracting a private key from the Windows Store, you have to hope that whoever imported it ticked the correct box to allow export, and Microsoft has not changed the rules again.  I would tell the customer you need the PFX. to use with OpenSSL. 

 

But you easily test if your FSslCertX509 object has a matching key and can be used as a server certificate.  If you are using the recommend IcsHosts server properties, the certificate gets checked and reported before the server starts, so you know if it's going to work.

 

Angus

 

Hi Angus,

 

if a certificate is imported with the "Is Exportable" option into the Windows certificate storage, ICS is able to fetch the private keys for RSA/DSA certs.

For ECC certificates a workaround is required as they are missing the NCRYPT_ALLOW_PLAINTEXT_EXPORT_FLAG per default (no UI option to set it on import):

 

diff --git "a/overbyte/ics/Source/OverbyteIcsMsSslUtils.pas" "b/overbyte/ics/Source/OverbyteIcsMsSslUtils.pas"
index 7932c54..cce85e7 100644
--- "a/overbyte/ics/Source/OverbyteIcsMsSslUtils.pas"
+++ "b/overbyte/ics/Source/OverbyteIcsMsSslUtils.pas"
@@ -234,6 +234,23 @@ implementation
 {$IFDEF MSWINDOWS}
 {$IFDEF USE_SSL}
 
+const
+  RGB_SALT_SIZE = 8;
+
+type
+  CRYPT_PKCS12_PBE_PARAMS = record
+    iIterations: Int32;
+    cbSalt: Int32;
+  end;
+
+  PBE_PARAMS = record
+  const
+    RgbSaltSize: Int32 = RGB_SALT_SIZE;
+  var
+    Params: CRYPT_PKCS12_PBE_PARAMS;
+    rgbSalt: array [0..(RGB_SALT_SIZE-1)] of Byte;
+  end;
+
 const
   CRYPT_E_REVOKED     = $80092010;
   CRYPT_E_EXISTS      = $80092005;
@@ -241,6 +258,12 @@ const
   sCryptoApiError   = 'CryptoAPI Error. Code: %d.'+#13#10+'%s';
   sUnKnownCryptoApi = 'A call to a CryptoAPI function failed';
 
+  NTE_BAD_FLAGS: Cardinal = $80090009;
+  NTE_BAD_KEY_STATE: Cardinal = $8009000B;
+  NTE_FAIL: Cardinal = $80090020;
+  NTE_INVALID_HANDLE: Cardinal = $80090026;
+  NTE_NOT_SUPPORTED: Cardinal = $80090029;
+  PBE_SHA1_3DES: AnsiString = '1.2.840.113549.1.12.1.3'#0;
 
 {* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *}
 procedure RaiseOSSLError(AErrCode: LongWord);
@@ -769,6 +792,137 @@ end;
 { load a Windows store into the list of certificates, optionally not emptying it first }
 function TMsX509List.LoadFromStore(MsCertStoreType: TMsCertStoreType; MsCertLocation: TMsCertLocation;
                                                                          Empty: Boolean = True): Integer;
+  function CngTrySetEccPrivateKeyExportable(const ACertStore: HCERTSTORE; var AKeyHandle: NCRYPT_KEY_HANDLE; var AError: string): Boolean;
+  var
+    LKeyData: RawByteString;
+    LParameterList: TNCryptBufferDesc;
+    LCryptBuffers: array[0..4] of TNCryptBuffer;
+    LParams: PBE_PARAMS;
+    dwRes: DWORD;
+    dwFlags: DWORD;
+    LKeyLen: Cardinal;
+    LError: Cardinal;
+    LKeyPol: DWORD;
+    LhProvider: NCRYPT_PROV_HANDLE;
+  begin
+    Result := False;
+
+    // Prepare ECC private key export in a way that requirements of Crypto-API/CNG are statisfied
+    FillChar(LCryptBuffers[0], SizeOf(LCryptBuffers), #0);
+    FillChar(LParams, SizeOf(LParams), #0);
+    LParams.Params.cbSalt := LParams.RgbSaltSize;
+
+    LCryptBuffers[0].BufferType := NCRYPTBUFFER_PKCS_ALG_OID;
+    LCryptBuffers[0].cbBuffer := Length(PBE_SHA1_3DES);
+    LCryptBuffers[0].pvBuffer := @PBE_SHA1_3DES[1];
+
+    LCryptBuffers[1].BufferType := NCRYPTBUFFER_PKCS_SECRET;
+    LCryptBuffers[1].cbBuffer := 0;
+    LCryptBuffers[1].pvBuffer := nil;
+
+    LCryptBuffers[2].BufferType := NCRYPTBUFFER_PKCS_ALG_PARAM;
+    LCryptBuffers[2].cbBuffer := SizeOf(LParams);
+    LCryptBuffers[2].pvBuffer := @LParams;
+
+    dwRes := 0;
+    dwFlags := NCRYPT_SILENT_FLAG;
+
+    LParameterList.ulVersion := NCRYPTBUFFER_VERSION;
+    LParameterList.cBuffers := 3;
+    LParameterList.pBuffers := @LCryptBuffers[0];
+
+    // Get private key length
+    LError := NCryptExportKey(AKeyHandle, 0, NCRYPT_PKCS8_PRIVATE_KEY_BLOB, @LParameterList, nil, 0, dwRes, dwFlags);
+    if ERROR_SUCCESS <> LError then
+    begin
+      AError := AError + 'Could not export private key - ' + GetWindowsErr(LError);
+      Exit;
+    end;
+
+    // Get private key
+    LKeyLen := dwRes;
+    SetLength(LKeyData, LKeyLen);
+    FillChar(LKeyData[1], LKeyLen, #0);
+    LError := NCryptExportKey(AKeyHandle, 0, NCRYPT_PKCS8_PRIVATE_KEY_BLOB, @LParameterList, @LKeyData[1], LKeyLen, LKeyLen, dwFlags);
+    if ERROR_SUCCESS <> LError then
+    begin
+      SetLength(LKeyData, 0);
+      AError := AError + 'Could not export private key - ' + GetWindowsErr(LError);
+      Exit;
+    end;
+
+    // Get current key export policy
+    LKeyPol := 0;
+    LError := NCryptGetProperty(AKeyHandle, NCRYPT_EXPORT_POLICY_PROPERTY, @LKeyPol, SizeOf(LKeyPol), dwRes, dwFlags);
+    if ERROR_SUCCESS <> LError then
+    begin
+      SetLength(LKeyData, 0);
+      AError := AError + 'Error fetching private key properties - ' + GetWindowsErr(LError);
+      Exit;
+    end;
+
+    if NCRYPT_ALLOW_PLAINTEXT_EXPORT_FLAG = (LKeyPol and NCRYPT_ALLOW_PLAINTEXT_EXPORT_FLAG) then
+    begin
+      AError := AError + 'Key already marked as exportable. Don''t know what went wrong.';
+      Exit;
+    end;
+
+    LError := NCryptFreeObject(AKeyHandle);
+    if ERROR_SUCCESS <> LError then
+    begin
+      AError := AError + 'Error freeing previous key handle - ' + GetWindowsErr(LError);
+      Exit;
+    end;
+
+    LhProvider := 0;
+    try
+      LError := NCryptOpenStorageProvider(LhProvider, MS_KEY_STORAGE_PROVIDER, 0);
+      if ERROR_SUCCESS <> LError then
+      begin
+        AError := AError + 'Error opening key storage provider - ' + GetWindowsErr(LError);
+        Exit;
+      end;
+
+      // Reimport the private key
+      FillChar(LCryptBuffers[0], SizeOf(LCryptBuffers), #0);
+      LCryptBuffers[0].BufferType := NCRYPTBUFFER_PKCS_SECRET;
+      LCryptBuffers[0].cbBuffer := 0;
+      LCryptBuffers[0].pvBuffer := nil;
+      LParameterList.cBuffers := 1;
+      dwFlags := NCRYPT_SILENT_FLAG or  NCRYPT_SILENT_FLAG OR NCRYPT_OVERWRITE_KEY_FLAG OR NCRYPT_DO_NOT_FINALIZE_FLAG;
+      LError := NCryptImportKey(LhProvider, 0, NCRYPT_PKCS8_PRIVATE_KEY_BLOB, @LParameterList, AKeyHandle, @LKeyData[1], LKeyLen, dwFlags);
+      if ERROR_SUCCESS <> LError then
+      begin
+        AError := AError + 'Error reimporting private key - ' + GetWindowsErr(LError);
+        Exit;
+      end;
+
+      // Allow plain text export.
+      // It seems the NCRYPT_ALLOW_PLAINTEXT_EXPORT_FLAG is volatile in the context of ECC private keys.
+      dwFlags := NCRYPT_SILENT_FLAG;
+      LKeyPol := LKeyPol or NCRYPT_ALLOW_PLAINTEXT_EXPORT_FLAG;
+      LError := NCryptSetProperty(AKeyHandle, NCRYPT_EXPORT_POLICY_PROPERTY, @LKeyPol, SizeOf(LKeyPol), dwFlags);
+      if ERROR_SUCCESS <> LError then
+      begin
+        AError := AError + 'Error adjusting private key properties - ' + GetWindowsErr(LError);
+        Exit;
+      end;
+
+      LError := NCryptFinalizeKey(AKeyHandle, dwFlags);
+      if ERROR_SUCCESS <> LError then
+      begin
+        AError := AError + 'Error finalizing private key - ' + GetWindowsErr(LError);
+        Exit;
+      end;
+    finally
+      if 0 <> LhProvider then
+      begin
+        NCryptFreeObject(LhProvider);
+      end;
+    end;
+    Result := True;
+  end;
+
 var
     hSystemStore: HCERTSTORE;
     pCertContext: PCCERT_CONTEXT;
@@ -859,6 +1013,14 @@ begin
                                 ParameterList.cBuffers := 1;
                                 ParameterList.pBuffers := @CryptBuffers[0];
                                 Ret := NCryptExportKey(hKey, 0, NCRYPT_PKCS8_PRIVATE_KEY_BLOB, @ParameterList, Nil, 0, dwResLen, dwFlags);
+                                if NTE_NOT_SUPPORTED = Cardinal(Ret) then
+                                begin
+                                    if CngTrySetEccPrivateKeyExportable(hSystemStore, hKey, PKeyInfo) then
+                                    begin
+                                        Ret := NCryptExportKey(hKey, 0, NCRYPT_PKCS8_PRIVATE_KEY_BLOB, @ParameterList, Nil, 0, dwResLen, dwFlags);
+                                    end;
+                                end;
+
                                 if Ret <> ERROR_SUCCESS then
                                     PKeyInfo := PKeyInfo + 'Could not export private key - ' + GetWindowsErr(Ret)
                                 else begin

 

I have to admit that this code is not testet with the ICS server components

(haven't figured out how to get ICS proxy in reverse mode to load certs from the MS cert storage until now - but that is another story).

It is more or less a proof of concept with the apparent result, that TMsX509List was able to fetch an ECC private key from the MS cert store.

 

I also hope that it is of any help for all the folks out there that have the problem to figure out how to NCryptExportKey the private part of an ECC based certificate from the Microsoft Certificate Storage.
 

Regards,

 

uso

Edited June 12, 2023 by uso

[Angus Robertson 526]

Can you please email me the complete edited unit, pasting from this forum often leaves unicode characters where we don't want them.

 

Once you have the cert and key in a TX509Base object, that can be used for ICS servers, although less easily with IcsHosts which is designed to read files. 

 

Angus

 

 

 

[Angus Robertson 526]

Thanks for the unit, should be in SVN next week with an updated PemTools sample.

 

Angus

 

[Angus Robertson 526]

SVN and the overnight zip include changes that allow ICS SSL servers to use certificates from the Windows Certificate Store instead of using disk files. 

 

Apart from rebuilding, no code changes are needed, simply add one line to the server configuration file. 

 

The HTTP Rest sample will be updated shortly to use a client certificate from the Windows Store. 

 

There is now a simple method LoadOneFromStore that creates a bundle in a TX509Base with certificate, private key and intermediates that can be used in any ICS projects, it can be tested in the PemTool sample. 

 

Angus

 

Edited July 5, 2023 by Angus Robertson