SSL Hanging on close

[Eric Bonilha 3]

Hello

 

We are using ICS 8.68 (I just saw that its old now...) and we have encountered a problem with OpenSSL that is hanging the thread where its running (In this case, the main thread). I'm not sure if this is something that was already fixed in newer versions, but it looks like it hangs inside the OpenSSL library

 

I was able to catch it with EurekaLog (image attached). Does anyone have any idea if this can still happen? Looks like this happened when the client connection was already broken (This is on SERVER SIDE) and the server tried to send a packet and ICS closed the connection on the server side after identifying it was already dead

 

Thanks

[Kas Ob. 91]

58 minutes ago, Eric Bonilha said:

Does anyone have any idea if this can still happen?

I can't answer that, sorry.

58 minutes ago, Eric Bonilha said:

I was able to catch it with EurekaLog (image attached).

This is really nice catch !, and it will be useful for everyone if you share with us, how you did it and what setting EurekaLog did you have.

1 hour ago, Eric Bonilha said:

Looks like this happened when the client connection was already broken (This is on SERVER SIDE) and the server tried to send a packet and ICS closed the connection on the server side after identifying it was already dead

This should not be important, i mean if the connection was broken or not or who initialized the closing.

 

My thought on this based on the screenshot:

1) ResetSSL is the last call from Delphi application the rest was in OpenSSL, while freeing a memory allocation.

2) The hang was caused by WER (Windows Error Reporting), ZwWaitForMutipleObjects is waiting for WerReportExceptionWorker.

3) The application is Windows Service which means it is not allowed to have GUI elements, so the system in theory should have froze the service then WER kicked in and asked for allowing to report (to upload to Microsoft), of course WER generated a dump log somewhere, but may be WER was configured to automatically upload the report.

 

So in my opinion, you should check your WER setting, (there is many resources on this, simple search will help), the most important is was WER generating the dump and it took very long time?, or it was generated and was uploading it without you interaction ?, or it was blocked from by firewall from uploading?.....

That on one hand, on the other hand EurekaLog can capture (raised by 3rdparty/system libraries), the exception raised by a DLL, and the system stepped instead of handling this differently by returning the execution to the application (to the last try-except), the Windows kernel decided that exception is critical and in need to be reported, so if you can produce that specific exception, then i suggest to train on it, have fun capturing it with EurekaLog before WER interfere, you will gain experience with EL and system exotic exceptions.

 

Another thought, i see that you are using timers, so what is the probability of that Send (in your code) being called while another thread is releasing the socket or OpenSSL resources, is there a chance that your application somehow calling ResetSSL on already freed/released context ?, in other words is there a threads race to reset/close/free that socket ?

 

 

Hope that helps.

[Angus Robertson 526]

ICS v9 has a lot of low level stability changes, many for Win64 applications, in particular errors during close and within finally statements, where inherited functions were sometimes skipped.

 

I only built my own public servers as Win64 about a year ago, which means that is when Win64 got a lot of extra live testing.   And I'm still uncertain about the OpenSSL DLLs in some very rare circumstances with Win64. 

 

Angus

[Kas Ob. 91]

On 9/12/2023 at 7:44 AM, Eric Bonilha said:

Does anyone have any idea if this can still happen?

Now I have a theory,

See, that RtlFreeHeap, this one since i saw it and it lingered in my brain, it makes less sense every time i think of scenario for it, RtlxxHeap functions belongs to Windows Driver realm (WDK), and they have very specific usage to communicate and share memory with drivers, so why is LibCrypto is freeing such a heap ? and what could have went wrong?

 

The Answer for these might be simple, your OpenSSL library used a hardware token or loaded a certificate from some hardware most likely a USB devices, this device was either a malfunctioned driver, a conflict with more than one driver, or simply it was unplugged ( also here comes some interference from other factors, like i have a rare problem, when i have my mobile doing heavy wifi traffic and i put it close to the monitor some times the monitor will flash to black in this second i notice that my back lighted keyboard also flash to black like it was unplugged and re-plugged) this will cause the driver to stop and restart.

 

Also a fact everyone should know, almost all Windows drivers will not upload form memory, they will stay there disabled and removed, they will linger there, taking resources by design as performance wise, next windows restart they will not be loaded.

So if you didn't restart your PC for long time, also you have changed or switched or updated your drivers, then as silly as it might sound a restart might fix this for good.

Try to remember what did happen with your hardware and reproduce that exception, are you using hardware certificate storage? did you unplugged it while you application was running ?.....

 

Also it is unlikely to be an OpenSSL bug, because it might be something you configured it or copied some settings from the net, if you are not using hardware storage then make sure OpenSSL is not looking for one.

 

Hope that help, and good luck.

[Eric Bonilha 3]

Thank you for all the answers, we will give it a try with ICS 9 and we will also check WER if we can disable it and check the configurations.

 

I will keep you updated with our testing, but you gave some very good insights