ICS SSL under Linux x64

[ap2021 0]

I know that some parts of ICS compile for Linux and so I have tried using ICS SSL, but ran into issues:

1) Compile errors referring to somethingRCDATAsomething led me to remove all included resources, including all OpenSSL stuff - executables and bundles, etc. Now it compiles and links Ok.

2) The project is an Apache module and without ICS it works, but with the minimum SSL functionality that I need, it starts alright (so Apache is happy at this stage) and immediatly exits (at which stage Apache hangs and eventually errors out 1min or so later), without any errors at any stage anywhere in between.

 

I'm using a RHEL clone - Oracle Linux, v. 9.2, which is recent, with whatever the latest Apache it ships with, Delphi Alexandria and WebBroker. The combination works, as I have another module working.

 

Since I cannot debug into it, I rely on logs. I can see it ticking through nicely until Application.Run, then the immediatate next line, then the last line in the DPR and then some code used in one of my units finalization section, non-stop. WebModuleCreate is never called, but maybe it's only kicking in on actual client connections, so it probably means nothing.

 

Apache logs nothing useful either. The last entry I can see is this: "[http2:debug] [pid 1002119:tid 1002119] mod_http2.c(112): AH03089: initializing post config dry run", which does not look like an error.

 

I have exception handlers all around - nothing is logged from there. No SIG's are reported anywhere, apart from the very end, after the timeout triggers cleanup in Apache:

 

Starting The Apache HTTP Server...

AH01574: module session_module is already loaded, skipping

AH01574: module wsgi_module is already loaded, skipping

httpd.service: start operation timed out. Terminating.

httpd.service: State 'stop-sigterm' timed out. Killing.

httpd.service: Killing process 1002119 (httpd) with signal SIGKILL.

httpd.service: Main process exited, code=killed, status=9/KILL

httpd.service: Failed with result 'timeout'.

systemd[1]: Failed to start The Apache HTTP Server.

 

Anyone tried any of this before? Any tips or tricks, or hopefully solutions you can share?

 

[Angus Robertson 577]

I'm not clear which ICS version you are using, only V9.3 released yesterday builds for Linux with SSL, but only utility functions work, like creating and reading SSL/TLS certificates.  I've only tested against Ubuntu 22.04, not Red Hat. 

 

ICS does not yet support TWSocket for Linux, so no protocols will work.  A new cross platform message pump supporting Windows and Linus has been written, but has not yet been added to ICS V9, probably within the next six months.  I'm planning to make this conditional, so Windows applications can use the existing or new message pumps. 

 

Previously ICS had a second message pump that was only supported MacOS, not Windows, which I've just removed ready for the new pump.  Another ICS user tried to make the MacOS pump work under Linux, but I never managed to build ICS with the changes he submitted.     

 

Angus

 

[ap2021 0]

Angus, it's "V9.3 beta" - as of maybe 2-3 days ago. I can re-test.

 

I'm not using any protocols there: it's all WebBroker, I only need JOSE stuff - read certificates, sign, validate signature, etc. I could have extra units included, though.

 

Delphi code seemingly works the same under either Ubuntu or RHEL. I'm actually still surprised so many developers default to Ubuntu, when most companies use RHEL, i.e.: who would they be developing for?

 

But this issue is something else. I suspect it could be some init/finalization in some unit(s), but without direct debugging, I cannot see that far...

[Angus Robertson 577]

I fixed all the Linux resource file build issues and wincrypt problems two months ago, and built the Linux test package yesterday with V9.3 without errors, so it sounds like your version is V9.2, try the final release. 

 

Provided your Linux project does not include TWSocket, it should work.  Try building the new IcsPemTest FMX sample on Red Hat provided it has OpenSSL 3 libraries in the same path as Ubuntu.  I will install Red Hat at some point, but only once most of ICS is working on Ubuntu.  

 

My testing showed putting TWSocket on a Linux format prevented that form appearing in Linux, some library function it brings in that just stops apps running, which is why I created the new

IcsPemTest sample so at least something would run.

 

Angus

 

[ap2021 0]

Ok, I'll try, hold on...

 

Where are the SSL libraries are, in your case? For me, they are under /usr/lib64 and I did not have this directory explicitly listed in any of my environment vars, so I can try that next...

[ap2021 0]

I do have /lib64 listed and that does point to /usr/lib64, so that should good. It has v. 3.0 installed, but that should be Ok too, right?

[ap2021 0]

Trying the PemTest project, got:

But I do have the library:

Although, the name in my case is .3.0.7, not just .3

 

But there's also a softlink .3 pointing to the right lib as well, so it should certainly be able to find the file.

 

3.0.7 is the latest available for this OS now.

 

Does this error explain what's going on for you? Maybe I just need to dial the security level up or down to make it work?

 

Edited September 28 by ap2021

[Angus Robertson 577]

That exception happens when ICS attempts to load all the OpenSSL exports, it would appear the OpenSSL version is missing certain EC_GROUP exports that are available in the standard 3.0 library.   Unfortunately, the screen short truncates the error list, there may be lots more missing.  They may be logged, can not recall if that sample has a log. 

 

Not sure if ICS uses EC_GROUPs, if not we can simply avoid loading them and the problem goes away.  I've done that before.  

 

I guess I need to build a Hyper-V RedHat VM to test in my copious spare time, the issue is what other important stuff do I ignore to do that?

 

Angus

 

[ap2021 0]

That's always the catch, isn't it? I'm in the same boat. I can only spend so much time on these things and when I hit a wall, I soon give up and then maybe circle back a few months later, and so on.

 

When ICS checks these functions, maybe you should skip those it doesn't use/need?

 

Interesting how a seemingly "standard" SSL distro may be missing bits and pieces. Isn't that weird? It's the latest Oracle Linux OpenSSL update for v. 9.2 of Oracle Linux. Seems nobody creates common distros, so every vendor is shipping their own, just doing the best they can.

 

Is there a way to programmatically ask ICS to be more lenient with that stuff?

 

Or maybe you can start building a Linux distro of SSL yourself, like you do for Windows? Or maybe I'll bite the bullet and build one myself, like I do for Windows - safer than relying on others at any rate.

 

I could give you remote access to this VM. Or ship you a 100G BD with a VDI disk (which Hyper-V may be able to import?..

[Angus Robertson 577]

It seems six (or more) EC_Group functions were deprecated in OpenSSL 3, and some builds exclude deprecated functions. 

 

In OverbyteIcsLIBEAY.pas, suppress four lines in the table with @@EC_GROUP_set_curve_GF , @@EC_GROUP_new and EC_GROUP_clear_free and change the array size to [0..841], that will stop these four exports loading.  If you get new names appearing in the exception, do the same thing.  ICS now only uses EC_GROUP_get_curve_name, not the dozens of others we load.  

 

I agree only loading those exports we use would be ideal, but working out which of those 842 are not used is not trivial, and many are there for future use, or use by end user applications.  

 

Angus

 

Edited September 28 by Angus Robertson

[ap2021 0]

Ok, cool, many thanks. It works now 😉

 

But it turns out I do use ICS HttpClient, so for the original issue I had with Apache, it's still inconclusive: it still fails, whether due to HttpClient or not.

 

I'll try replacing HttpClient with something else next...

 

 

[ap2021 0]

Just reporting my findings as I go along:

 

I'm certainly on 9.3 now and that "multiple definitions" link error about RCDATA was gone for me for a while. But it's back now, I can't see how yet.

 

I also figured that 1) the library actually loads Ok, with no errors, I could load it from my own test project, then map an exported field and read it - the loading app, when run in a debugger, would typically pick up any errors around missing dependencies and such, plus 2) Apache module is actually supposed to exit right after it starts, because TApacheApplication does not implement a Run method and parent's does nothing, so that's normal - my other module actually does the same, weird as it sounds.

 

A few builds ago, when I had no "multiple definitions" error, I could also confirm that completely removing ICS HttpClient & related units did not fix the original Apache issue, it kept timing out with the same symptoms. So I'm now experimenting with different initialization units' clauses.

[Angus Robertson 577]

As I said earlier, ICS has never had a working message pump for Linux, so no functions dependent on messages will work.

 

Some changes were made to TIcsEventQueue to try and support Linux in V9.2 but did not work and were disabled for V9.3 so at least the non-message stuff loaded. 

 

So making THttpClient work is not trivial with V9.3. 

 

Angus

 

[ap2021 0]

I can see that this issue starts for me with OverbyteIcsSslJose, which includes Ics.Posix.PXMessages and there, it hangs on FreeAndNil(GlobalSync) in finalization. Even though GlobalSync was seemingly assigned when I debugged it.

 

Adding try/except around does not change anything.

 

You should be able to reproduce it in your PemTest project, if you add OverbyteIcsSslJose, maybe call something from there so the compiler does not remove it for you, and try closing the application after it starts - it happens on exit.

 

Actually, that unit is not even used there, I could comment it out and see this issue disappear. Please remove Ics.Posix.PXMessages unit reference from OverbyteIcsSslJose.

 

OverbyteIcsSslX509Certs unit also includes Ics.Posix.PXMessages, but probably only not for POSIX, because it does not seem to be causing this issue for me.

 

PS: The RCDATA issue goes away, if the project has FMX defined.

 

Looks like I'm past this issue for now 😉 Thanks for your help, much appreciated!

Edited September 30 by ap2021

[Angus Robertson 577]

The Ics.Posix.PXMessages unit was written many years ago for 32-bit MacOS, and has never been updated for 64-bit, nor tested properly with Linux.  It is now obsolete, so I'll remove it from ICS shortly.  No idea why I put it into the Jose unit, no messages there.

 

Angus

 

 

 

[ap2021 0]

Thanks for your help, I'm now directly at the stage where I'm trying to validate a JWT, but:

 

Signature validation failed error:02000077:rsa routines::wrong signature length
error:1C880004:Provider routines::RSA lib
error:02000077:rsa routines::wrong signature length
error:1C880004:Provider routines::RSA lib

 

Unfortunately, your PemTest project has no comparable functionality to try the vanilla implementation. I'll try adding mine there next. I am using known certs and proven code (under Win64), but still not quite sure, since it's Linux...

[Angus Robertson 577]

One common issue using the Jose functions is they use binary arguments, originally AnsiString, now TBytes, not Base64, so you need to decode Base64 first, that might explain the wrong length. 

 

I'll put an FMX version of the Jose sample on my list. 

 

Angus

 

[ap2021 0]

Yep, been there, done that, and it's not it here.

 

Looking at IcsAsymVerifyDigestTB function, I think under Linux, TBytes is somehow different (it's shown in the debugger as JSON, with Length as the first element) and PAnsiChar(OldDigest) does not work as intended.

 

For instance, if I do:

 

        hash := PAnsiChar(OldDigest); // To AnsiString
        bhash := TEncoding.ANSI.GetBytes(hash); // To TBytes

 

I end up with garbage, plus it's shorter than it was. Although it could be something else.

 

Trying to figure out what to do about it now. Have you seen anything like this before?

[Angus Robertson 577]

You don't say what type OldDigest is, or which Jose function you are using, but casting and TEncoding is probably corrupting it.  Try using IcsStringToTBytes instead, which is what the Jose unit uses.

 

Angus

 

[ap2021 0]

Sorry, it was TBytes. And ICS casts it as PAnsiChar(OldDigest). I think mine has #0 @ 53rd position and I think EVP_DigestVerify only receives the 1st 52 chars as a result, hence this "wrong signature length" error.

 

So it's not StringToTbytes, it's the other way around, but I think this is where the issue is...

[ap2021 0]

Incidentally, this is how debugger shows the TByates value in the inspector, under Linux x64:

 

{length = 256, [0] = 133 '#$85', [1] = 21 '#$15', [2] = 119 'w', [3] = 190 '#$be', [4] = 104 'h', [5] = 226 '#$e2', [6] = 237 '#$ed', [7] = 121 'y', [8] = 59 ';', [9] = 71 'G', [10] = 91 '[', [11] = 109 'm', [12] = 252 '#$fc', [13] = 29 '#$1d', [14] = 9 '#$09', [15] = 120 'x', [16] = 163 '#$a3', [17] = 229 '#$e5', [18] = 66 'B', [19] = 46 '.', [20] = 238 '#$ee', [21] = 3 '#$03', [22] = 84 'T', [23] = 236 '#$ec', [24] = 152 '#$98', [25] = 91 '[', [26] = 104 'h', [27] = 33 '!', [28] = 142 '#$8e', [29] = 4 '#$04', [30] = 72 'H', [31] = 180 '#$b4', [32] = 60 '<', [33] = 148 '#$94', [34] = 50 '2', [35] = 207 '#$cf', [36] = 102 'f', [37] = 233 '#$e9', [38] = 226 '#$e2', [39] = 97 'a', [40] = 227 '#$e3', [41] = 53 '5', [42] = 88 'X', [43] = 141 '#$8d', [44] = 32 ' ', [45] = 17 '#$11', [46] = 52 '4', [47] = 109 'm', [48] = 69 'E', [49] = 149 '#$95', [50] = 172 '#$ac', [51] = 45 '-', [52] = 100 'd', [53] = 0 '#$00', [54] = 161 '#$a1', [55] = 197 '#$c5', [56] = 147 '#$93', [57] = 179 '#$b3', [58] = 45 '-', [59] = 185 '#$b9', [60] = 141 '#$8d', [61] = 240 '#$f0', [62] = 119 'w', [63] = 107 'k', [64] = 111 'o', [65] = 35 '#', [66] = 27 '#$1b', [67] = 17 '#$11', [68] = 102 'f', [69] = 122 'z', [70] = 19 '#$13', [71] = 139 '#$8b', [72] = 233 '#$e9', [73] = 15 '#$0f', [74] = 56 '8', [75] = 33 '!', [76] = 105 'i', [77] = 183 '#$b7', [78] = 151 '#$97', [79] = 83 'S', [80] = 19 '#$13', [81] = 37 '%', [82] = 116 't', [83] = 81 'Q', [84] = 169 '#$a9', [85] = 149 '#$95', [86] = 208 '#$d0', [87] = 247 '#$f7', [88] = 211 '#$d3', [89] = 189 '#$bd', [90] = 59 ';', [91] = 51 '3', [92] = 35 '#', [93] = 61 '=', [94] = 227 '#$e3', [95] = 180 '#$b4', [96] = 42 '*', [97] = 105 'i', [98] = 249 '#$f9'}

 

That's all, it cuts it at 99 bytes. I just hope it's only for display, because there are 256 chars there, or supposed to be (Ok, it actually says so in the 1st element of that JSON, if you believe that). Very hard to debug under Linux ;-(

 

I also suspect that strings under Linux may be 0-based and ICS does a lot of "from i := 1 to ..."s all around...

Edited October 4 by ap2021

[ap2021 0]

Here's a simple test:

(OldDigest is TBytes, lll is integer)

        lll := Length(PAnsiChar(OldDigest)); // 52
        lll := Length(OldDigest); // 256

 

That cast does kill it...

[Angus Robertson 577]

I missed the IcsAsymVerifyDigestTB name, it takes a TBytes binary digest (base 0) and casts it to PAnsiChar to pass to an OpenSSL API.

 

ICS does not use TEncoding anywhere since it's not in old compilers, so I've no idea of the purpose of that line.  ICS only uses base 1 for strings, not TBytes. 

 

Angus

 

 

 

[ap2021 0]

Yes, but as per my earlier message, that one cast may be sufficient to kill it.

 

Now I have rewritten all parts involved in this one atomic process to replace pansichar with pointer and I'm copying raw data, so there can be no character conversions anywhere here, including in the EVP_DigestVerify definition and I'm watching memory and can see it all makes through as intended. And yet, the same error.

 

It works under Windows, BTW, with the same values.

[Lajos Juhász 295]

1 hour ago, ap2021 said:

Yes, but as per my earlier message, that one cast may be sufficient to kill it.

Yes it is, PAnsiChar is a zero terminated string....