bug Odd (OpenSSL?) bug and types in X509 certs

[merijnb 4]

note: We have seen these issues with multiple versions of the OpenSSL binaries, including the ones bundled with ICS 8.64.

 

Please consider the following code snippet:

 var s := TSslContext.Create(nil);
 try
  s.SslCALines.Text := sslRootCACertsBundle();
  try
   s.InitContext();
  except
   on e: exception do
    ShowMessage(e.Message);
  end;
 finally
  s.Free();
 end;

This snippet works on most machines, but we have a few machines at client sites, where this code raises an exception: 'Error reading info file "Lines"'. (We haven't figured out yet why these specific machines have this issue).

 

This exception is raised on OverbyteIcsWSocket.pas:14667 (ICS 8.64).

 

    InfoStack := PStack(f_PEM_X509_INFO_read_bio(InBIO, nil, nil, nil));
    if not Assigned(InfoStack) then
        raise ESslContextException.CreateFmt('Error reading info file "%s"', [FileName]);

Apparently the call to f_PEM_X509_INFO_read_bio() failed, but no information is give why, so we've added some calls to f_ERR_get_error() to see what is going wrong, and then it turns out the call to f_PEM_X509_INFO_read_bio() sets the error code 151580774 (get_header_and_data).

To find out which of the hard coded certificates returned by sslRootCACertsBundle() were making this error happening, we've passed each of the certificates (sslRootCACerts001, sslRootCACerts002, etc) one by one to an TSSLContext to see which ones failed. This showed 2 interesting things:

 

There is a 'typo' in sslRootCACerts009 and sslRootCACerts011:
 

        'AGAT/3B+XxFNSRuzFVJ7yVTav52Vr2ua2J7p8eRDjeIRRDq/r72DQnNSi6q7pynP' + #13#10 +
        '9WQcCk3RvKqsnyrQ/39/2n3qse0wJcGE2jTSW3iDVuycNsMm4hH2Z0kdkquM++v/' + #13#10 +
        'eu6FSqdQgPCnXEqULl8FmTxSQeDNtGPPAUO6nIPcj2A781q0tHuu2guQOHXvgR1m' + #13#10 +
        '0vdXcDazv/wor3ElhVsT/h5/WrQ8' + #13#10;
    sslRootCACerts010 =
        '-----END CERTIFICATE-----' + #13#10 +
        '# X509 SSL Certificate' + #13#10 +

and

 

        'S0Zj/gA0QHDBw7mh3aZw4gSzQbzpgJHqZJx64SIDqZxubw5lT2yHh17zbqD5daWb' + #13#10 +
        'QOhTsiedSrnAdyGN/4fy3ryM7xfft0kL0fJuMAsaDk527RH89elWsn2/x20Kk4yl' + #13#10 +
        '0MC2Hb46TpSi125sC8KKfPog88Tk5c0NqMuRkrF8hey1FGlmDoLnzc7ILaZRfyHB' + #13#10 +
        'NVOFBkpdn627G190' + #13#10;
    sslRootCACerts012 =
        '-----END CERTIFICATE-----' + #13#10 +
        '# X509 SSL Certificate' + #13#10 +

(the END CERTIFICATE line is part of the wrong constant), which isn't a problem when you call sslRootCACertsBundle() since all constants are concatenated, but which shows up if you try to use these certificates separately.

 

More interestingly, one these specific machines there seems to be an issue with sslRootCACerts033 and sslRootCACerts034.

 

This is sslRootCACerts033:

 

# X509 SSL Certificate
# Subject Common Name: GlobalSign
# Subject Organisation: GlobalSign
# Subject Organisation Unit: GlobalSign Root CA - R2
# GlobalSign Root R2 SHA1 • RSA • 2048
# Issuer: Self Signed
# Expires: 15/12/2021
-----BEGIN CERTIFICATE-----
MIIDujCCAqKgAwIBAgILBAAAAAABD4Ym5g0wDQYJKoZIhvcNAQEFBQAwTDEgMB4G
A1UECxMXR2xvYmFsU2lnbiBSb290IENBIC0gUjIxEzARBgNVBAoTCkdsb2JhbFNp
Z24xEzARBgNVBAMTCkdsb2JhbFNpZ24wHhcNMDYxMjE1MDgwMDAwWhcNMjExMjE1
MDgwMDAwWjBMMSAwHgYDVQQLExdHbG9iYWxTaWduIFJvb3QgQ0EgLSBSMjETMBEG
A1UEChMKR2xvYmFsU2lnbjETMBEGA1UEAxMKR2xvYmFsU2lnbjCCASIwDQYJKoZI
hvcNAQEBBQADggEPADCCAQoCggEBAKbPJA6+Lm8omUVCxKs+IVSbC9N/hHD6ErPL
v4dfxn+G07IwXNb9rfF73OX4YJYJkhD10FPe+3t+c4isUoh7SqbKSaZeqKeMWhG8
eoLrvozps6yWJQeXSpkqBy+0Hne/ig+1AnwblrjFuTosvNYSuetZfeLQBoZfXklq
tTleiDTsvHgMCJiEbKjNS7SgfQx5TfC4LcshytVsW33hoCmEofnTlEnLJGKRILzd
C9XZzPnqJworc5HGnRusyMvo4KD0L5CLTfuwNhv2GXqF4G3yYROIXJ/gkwpRl4pa
zq+r1feqCapgvdzZX99yqWATXgAByUr6P6TqBwMhAo6CygPCm48CAwEAAaOBnDCB
mTAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUm+IH
V2ccHsBqBt5ZtJot39wZhi4wNgYDVR0fBC8wLTAroCmgJ4YlaHR0cDovL2NybC5n
bG9iYWxzaWduLm5ldC9yb290LXIyLmNybDAfBgNVHSMEGDAWgBSb4gdXZxwewGoG
3lm0mi3f3BmGLjANBgkqhkiG9w0BAQUFAAOCAQEAmYFThxxol4aR7OBKuEQLq4Gs
J0/WwbgcQ3izDJr86iw8bmEbTUsp9Z8FHSbBuOmDAGJFtqkIk7mpM0sYmsL4h4hO
291xNBrBVNpGP+DTKqttVCL1OmLNIG+6KYnX3ZHu01yiPqFbQfXf5WRDLenVOavS
ot+3i9DAgBkcRcAtjOj4LaR0VknFBbVPFd5uRHg5h6h+u/N5GJG79G+dwfCMNYxd
AfvDbbnvRG15RjF+Cv6pgsH/76tuIMRQyV+dTZsXjAzlAcmgQWpzU/qlULRuJQ/7
TBj0/VLZjmmx6BEP3ojY+x1J96relc8geMJgEtslQIxq/H5COEBkEveegeGTLg==
-----END CERTIFICATE-----

When processing this certificate on these specific machines OpenSSL gives this error, unless:

- I remove one of the • in the 5th line from the top. (As long as there is only 1 • in the file it's ok, as soon as I add a 2nd • anywhere in the header the error returns)

- I add a CRLF at the bottom of the file

 

The same thing applies to sslRootCACerts034. I realize that this is probably a bug in OpenSSL itself, but it might be wise to remove the •'s from the hard coded certificates to work around this issue.

 

Curious if anyone can shed some light on the cause of this 🙂

 

Merijn

[Angus Robertson 574]

Thanks, I've fixed certs 10 and 12, that was me on auto pilot copying selected certificates from the automatically generated bundle files and manually converting them into constants with global replace, really need to automate that to avoid errors.  But since all the certificates are concatenated in order, it should not make any difference. in theory. 

 

Not sure how those • 8-bit characters got into the literals, but I've removed them, and from TrustedCABundle, the larger bundles are built in a different way. 

 

Very strange that errors don't show up consistently.  I have seen a problem with PSTACK on servers, where stacks were being continually rebuilt to validate certificates,, but that went away when I simplified the code to avoid rebuilding so often.  

 

These fixes should be in SVN tomorrow, when I've finished some other stuff.

 

Angus

[Guest]

1 hour ago, merijnb said:

Curious if anyone can shed some light on the cause of this 🙂

I saw something very similar to this but it wasn't OpenSSL but different libraries, the reason for this wrong parsing is coming from Language for Non-Unicode program settings in Windows Language settings, you can confirm if this is the case by finding what settings on those few machines, away from that removing those characters might help but the better solution is to convert this bundle file to Unicode or UTF8 ,...(whatsoever) as long you don't leave it to Windows and OpenSSL.

[merijnb 4]

4 minutes ago, Kas Ob. said:

I saw something very similar to this but it wasn't OpenSSL but different libraries, the reason for this wrong parsing is coming from Language for Non-Unicode program settings in Windows Language settings, you can confirm if this is the case by finding what settings on those few machines, away from that removing those characters might help but the better solution is to convert this bundle file to Unicode or UTF8 ,...(whatsoever) as long you don't leave it to Windows and OpenSSL.

 

I can confirm that this checkbox was checked, after unchecking and rebooting the problem is gone.

[Angus Robertson 574]

The safest way to read bundle files with unicode characters will be to strip out all the comments I add before passing them to OpenSSL. 

 

For maintenance, adding certificate fields makes life easy, I hate long blocks of base64, and previously OpenSSL has seemed good at ignoring added text.  But if the comments are causing trouble, they should go.

 

Angus

 

[merijnb 4]

57 minutes ago, Angus Robertson said:

The safest way to read bundle files with unicode characters will be to strip out all the comments I add before passing them to OpenSSL. 

 

For maintenance, adding certificate fields makes life easy, I hate long blocks of base64, and previously OpenSSL has seemed good at ignoring added text.  But if the comments are causing trouble, they should go.

 

Angus

 

Or just convert it to ascii of unicode gives the issues

[Angus Robertson 574]

There is a fix for the internal root certificate bundle in SVN, I simply suppressed literal comments from the constants which saves space in binaries and avoids any 8-bit characters being read.   I added a button in the PemTool sample that loads and displays the various ICS bundles, to make sure they work. 

 

I need to look more closely at the various functions in ICS that save and read certificates to make sure they all support UTF8, and so that only real base64 PEM content is sent to OpenSSL to avoid any problems reading existing files, not just roots.  That will all need careful testing. 

 

Angus

 

[Guest]

@Angus Robertson May i suggest to skip windows cert store and use Mozilla's, https://curl.haxx.se/docs/caextract.html , it does include only the name of the certificates, but still better than raw Base64.

[Guest]

Above i pasted the wrong link this from Mozilla, this is the right one

https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/

[Angus Robertson 574]

Our root stores were taken from the Windows store on one of my servers until last year, but Windows now only includes certificates needed by the server, and downloads new ones it needs. 

 

http://wiki.overbyte.eu/wiki/index.php/FAQ_SSL/TLS_Certificate_Authority_Root_Stores explains how we build them now.  But all the textual comments are mine, to keep track of them.

 

Angus

 

[Guest]

The comments are very helpful to keep tracking which is which.

[m227 2]

Hi, I have exactly the same problem, in ICS 8.58 and ICS 8.64 as well. I have 4 signatures in my XAdES file.

Two of them are decoded well, but two don't. ("Text Lines")... Line 14667 (OverbyteIcsWSocket.pas).

It happens when I have 2 OpenSSL libraries in my program folder (libeay32.dll + ssleay32.dll  and  libcrypto-1_1.dll + libssl-1_1.dll).

When i remove libcrypto and libssl the problem vanishes. But I need these libraries for Indy 10 to be used when reading some data from some HTTPS URLS. If they are missing, the Indy reports problems.

 

I extract data from XAdES Signature/KeyInfo/X509Data/X509Certificate nodes and envelope them with:

    -----BEGIN CERTIFICATE----- #CRLF

base64data
#CRLF -----END CERTIFICATE-----

 

I will try to direct ICS to subfolder with properly selected DLL-s but still have no clue how.

 

Edit: I did change:

OverbyteIcsSSLEAY.GSSL_DLL_DIR := ExtractFilePath(ParamStr(0)) + 'ICS_DLL\';

  or

 OverbyteIcsSSLEAY.GSSLEAY_DLL_IgnoreNew := True;

 

But any of the solutions above made also Indy work improperly (loading SSL). It seems that once selected Lib for ICS influences lib used by Indy... Horrible.

Edited June 18, 2020 by m227
New information

[Angus Robertson 574]

The problem being discussed here is PEM certificate files with extra content outside the base64 blocks, where there may be text outside the 7-bit ASCII range, and something gets confused deciding whether this is UTF8 or ANSI, not quite sure what yet,  or why the new Windows 10 force UTF8 APIs setting would have any effect on content embedded in a De[phi application and not being read from a file. 

 

So unless your XAdES file has extra stuff you did not post, it's not this issue.

 

ICS currently supports three versions of OpenSSL with two different DLL names,  If you set the global GSSLEAY_DLL_IgnoreNew to true before calling any SSL functions, it will load the old unsupported libeay32.dll instead of the modern one, and it's possible they do not understand UTF8 so work for you.  But Indy still uses the older DLLs, so confused as to why removing the new ones will upset it.  

 

There is another global variable GSSL_DLL_DIR you can set to the directory from which to load OpenSSL DLLs, again set before calling anything, needs trailing backslash.

 

Angus

[Guest]

39 minutes ago, Angus Robertson said:

not quite sure what yet,  or why the new Windows 10 force UTF8 APIs setting would have any effect on content embedded in a De[phi application and not being read from a file. 

This not Windows 10 problem, this behaviour was consistent since Windows XP at least.

 

The core problem is between these functions WideCharToMultiByte and MultiByteToWideChar, and what Ansi CodePage (ACP) to use , as the default is CP_ACP, even threads has their own ACP. 

[m227 2]

I use Windows 7/64 so it is not Windows 10 issue. I solved problem but had to act carefully. Once I put new DLL-s from:

https://indy.fulgan.com/SSL/Archive/Experimental/openssl-1.1.0h-x32-VC2017.zip

 

i had also to add:

OverbyteIcsSSLEAY.GSSLEAY_DLL_IgnoreOld := True;

and leave libeay32.dll and ssleay32.dll intact, as their lack made my Indy to throw such exceptions using SSL http communication:         

Quote

 

EIdOSSLUnderlyingCryptoError with message 'Error connecting with SSL. error:1409442E:SSL routines:SSL3_READ_BYTES:tlsv1 alert protocol version'.

EIdOSSLUnderlyingCryptoError with message 'Error connecting with SSL. error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version'.

 

dependently of used TIdSSLIOHandlerSocketOpenSSL.SSLOptions.Method.

 

Angus, surely I have no "extra content outside the base64 blocks" but the same error in the same lines of ICS as OP's lead me here.

Edited June 18, 2020 by m227
misspelling

[Angus Robertson 574]

OpenSSL 1.1.0 is also old and no longer supported, you should be using 1.1.1g from http://wiki.overbyte.eu/wiki/index.php/ICS_Download which is the latest, but is unlikely to make much difference to certificate errors. 

 

I am currently making certificate reading more robust, but the offending certificates in the bundles have been there a long time so there is no clear reason why we are now seeing errors on just a few PCs. 

 

Angus

 

[m227 2]

2 hours ago, Angus Robertson said:

OpenSSL 1.1.0 is also old and no longer supported, you should be using 1.1.1g from http://wiki.overbyte.eu/wiki/index.php/ICS_Download which is the latest, but is unlikely to make much difference to certificate errors. 

 

I am currently making certificate reading more robust, but the offending certificates in the bundles have been there a long time so there is no clear reason why we are now seeing errors on just a few PCs. 

 

Angus

 

Unfortunately version 1.1.1 backs to ESslContextException: Error reading "Text Lines" (Line 14667, OverbyteIcsWSocket.pas). So I cannot use it.

[Angus Robertson 574]

Okay, so something changed in 1,1,1 reading certificates.

 

Unfortunately we are using an old undocumented API PEM_X509_INFO_read_bio which has worked for 15 years, must see if there is something better.

 

Angus

 

[m227 2]

If there is a need for a sample file to test, I can prepare and send it. I'd be glad to see ICS working with OpenSSL 1.1.1.

[Angus Robertson 574]

Yes, please email your bad certificate file, my address is in readme8.txt. 

 

Angus

 

[m227 2]

24 minutes ago, Angus Robertson said:

Yes, please email your bad certificate file, my address is in readme8.txt. 

 

Angus

 

Done

[Angus Robertson 574]

Thanks for the files.  ICS will read two of them OK, but not the other two. 

 

Nor can the openssl.exe tool read the bad files using the x509 command. 

 

The asn1parse command is unable to read any of your files, but is fine with all mine.  So there is something strange about the construction of these Certum certificates. 

 

Angus

 

[m227 2]

2 hours ago, Angus Robertson said:

 

I'd like to remind that all four files are read using OpenSSL 1.1.0 (so I use it now instead of 1.1.1). This makes me suggest that something went wrong in 1.1.1 (if the error communicate does not clarify what).

[Angus Robertson 574]

Probably the certificate files are incorrectly formatted but previously the error was ignored, and this is corrected. 

 

There is a web site that parses ASN1 https://lapo.it/asn1js/ and it reads all the files OK, but that does not mean they are valid X509 certificates. 

 

ICS is not the only server using OpenSSL 1.1.1, Certum should make their certificates are compatible with the latest and greatest or others will complain as well. 

 

Angus

 

[m227 2]

My clients notify that other gates/programs read these signatures with no errors (probably they use older OpenSSL) and I cannot argue with it. If the certificate is spoiled it should be read as it is with 1.1.0 and some annotation could be added. Current 1.1.1 is in this field useless. If you find any reliable site which will revoke these certificates, I eagerly cite it to my clients.

Edited June 19, 2020 by m227