Can ICS thread cause memory violation? (EOutOfResources error)

[PizzaProgram 9]

I have a huge app (pizzaprogram) running for 20+ years rock stable on hundreds of PCs, written in Delphi7, using:

 AlphaSkin + Indy + UIB + TVirtualStringTree

also had 5 background threads running without any problems. (Some of them are using UIB.)

 

This year I've started to work with `OverbyteIcsSslHttpRest, OverbyteIcsLogger, OverbyteIcsSslJose, OverbyteIcsSslX509Utils, OverbyteIcsWSocket, OverbyteIcsSuperObject, OverbyteIcsHttpProt, OverbyteIcsUtils, OverbyteIcsSSLEAY` units, running in a 6th separated thread.

 

Since then, every 1-6 hours my program is crashing with EOutOfResources whenever AlphaSkin is trying to create or resize a bigger Bitmap.

I've upgraded AlphaSkin from 2019 version to latest, but it did not help.
 (Actually it made it even worth, because the new version is generating 32bit bitmaps for every form and panel and button forehead.)

 

The private memory consumption of my EXE is always under <150MB , usually 75MB

Peak Virtual memory size < 300MB

GDI number < 900

 

Please give me some hint / advice, how could I solve this mystery?

[PizzaProgram 9]

The new TThread is doing nothing special, just reading max first 64 lines from the firebird database via fresh created TUIBDatabase + TUIBQuery, creating a JSON from it, and sending to the government server, via special formed, (non standard JOSE signed) https REST PUT.
The only interaction with main-thread is a thread-safe array record for logging and a few Booleans.

(I've used the same logging array class for other threads before, without any problem.)

 

Using latest Version: 8.70 (2022-11-09)

 

 

Edited August 19, 2023 by PizzaProgram
add version

[aehimself 396]

What OS the application is running on? Also, are you reusing your threads or creating and freeing them up as needed?

On Windows 2000 creating and freeing up threads lead to the same error for me (probably due to memory fragmentation, idk) and the solution was to simply reuse threads.

[Brian Evans 105]

What memory manager are you using? Failing larger allocations when memory seems available is a sign it could be memory fragmentation. A memory manager that uses buckets (groups allocations by size) usually avoids the scattered small allocations blocking large allocations.  

[Angus Robertson 574]

Delphi 7 is end of life, ICS support has now ceased, I won't be running it up again now ICS v9 has been tested on it.

 

Also, running TSslHttpRest in a thread is not tested heavily (or atall), being async you can download hundreds of files in parallel in the main thread.  

 

Angus

 

[FPiette 383]

EOutOfResource happens when you don't free resources after use and yet allocate a new one. This can come from any resource.

Put the thread task in a separate exe to ease testing.

[PizzaProgram 9]

Thank you for all the answers so far!

 

I'm using latest FastMM4 memory manager. (declared at the top of the project.)

If I close the project, it is listing the "non-released resources" pretty well. There are basically none!

 

I'm starting every necessary thread at program start, and stopping at the end. Each one is running a simple:

While true do begin 
  try 
    ...
  except on e: Exception do LogMyError(...); 
  end;
  // free all resources
end;

So there is no "recreating" problem at all.

 

Also I've bought MadExcept module, so I get great error-debug reports from everywhere. Like:

allocated memory   : 150,98 MB
largest free block : 6,05 MB
exception number   : 1
exception class    : EOutOfResources
...
main thread ($d7c):
0049126f +077 TermiPRO.exe Graphics                GDIError
004912a7 +007 TermiPRO.exe Graphics                GDICheck
0049545a +2d2 TermiPRO.exe Graphics                CopyBitmap
00495c5b +063 TermiPRO.exe Graphics                TBitmap.CopyImage
00497010 +03c TermiPRO.exe Graphics                TBitmap.SetHeight
0063be2b +103 TermiPRO.exe acSBUtils     1629  +20 PrepareCache

 

I've tested to recreate such errors: by changing AlphaSkin's acSBUtils.pas to set a TBitmap.Width. Found out, it is normally impossible, except a value over 600000 (x 1000 height).

 

My EXE is running on all kinds of platforms everywhere: XP + Win7/10/11 32+64bit. Different brands Dell / HP / unique / etc ... all different CPUs ... different VGAs.
Luckily nobody is using Win2000 any more. (I had 1 client still had 1 client PC just 4 years ago! 😄 )

This new problem occurs on all platforms similar way.

[PizzaProgram 9]

This is the 2 main functions I'm using ICS with in this thread:

function TNtakThread.JWS_Sign(const payload, key1: String): AnsiString;
var
  SObj    : ISuperObject;
  JoseAlg : OverbyteIcsSslJose.TJoseAlg;
  PrivKey : TSslCertTools;
begin
    Result  := '';
    JoseAlg := OverbyteIcsSslJose.jsigRsa256;
    PrivKey := OverbyteIcsSslX509Utils.TSslCertTools.Create(nil);
    try
        try
            PrivKey.ClearAll;
            PrivKey.PrivateKeyLoadFromText(key1, '') ;
            SObj   := SO(IcsJoseJWSJson(JoseAlg, payload, '', PrivKey.PrivateKey, '', '', '', ''));
            Result := SObj.AsObject.S['protected'] + '..' + SObj.AsObject.S['signature']; 
        except on E:Exception do
            ntLOG('JWS ERR: ' + E.Message);
        end;
    finally
        PrivKey.Free;
        Sobj := nil;
    end;
end;

 

Sending REST requests:

function TNtakThread.NTAK_Send1(const url: string; const json: SOString; var hiba: string; var valasz: WideString): Boolean;
// "hiba" means: error ,   "valasz" means: answer
var
    SslHttpCli  : TSslHttpCli;
    SslContext  : TSslContext;
    JsonUTF8    : UTF8String;
    mySign      : AnsiString;
    lg          : integer; // length
begin
    Result := False;
    hiba   := '';
    valasz := '';
    OverbyteIcsSSLEAY.GSSL_DLL_DIR := SSl3_.path; // '...\Bin\SSL\20211217\';
    SslContext      := nil;

    SslHttpCli      := TSslHttpCli.Create(nil);
    SslHttpCli.URL  := url;
    SslHttpCli.ContentTypePost  := 'application/jose+json; charset=UTF-8';
    SslHttpCli.Accept           := 'application/json';
    SslHttpCli.SendStream := TMemoryStream.Create;
    SslHttpCli.RcvdStream := TMemoryStream.Create;

    try
        try
            SslHttpCli.Timeout := 10; // Must be seconds! Not MS !!! ... TODO: ask the author to create a default property: TimeoutMs  

            SslContext := TSslContext.Create(nil);
            SslContext.SslMinVersion := sslVerTLS1_2;
            SslContext.SslMaxVersion := sslVerTLS1_3;
            SSlContext.SslVerifyPeer := False; // := SSL_VERIFY_NONE;
            SslContext.SslCertLines.Text    := cer1;
            SslContext.SslPrivKeyLines.Text := key1;

            SslHttpCli.SslContext := SslContext;

            if cer6464 = '' then // enough to encode once
                cer6464 := Base64Encode(cer1);
            SslHttpCli.ExtraHeaders.Add( 'x-certificate: ' + cer6464);
            mySign := JWS_Sign( json, key1 );
            SslHttpCli.ExtraHeaders.Add( 'x-jws-signature: ' + mySign  );

            JsonUTF8 := UTF8Encode(json);
            lg       := Length(JsonUTF8);
            SslHttpCli.SendStream.Write(JsonUTF8[1], lg);
            SslHttpCli.SendStream.Seek(0, soFromBeginning);

            SslHttpCli.Post;
            ntLOG( 'Successful sending :-)', -1);
        except on E:Exception do begin // 400-as válasznál is exception-re ugrik. Attól még a JSON válasz ott van a Stream-ben!
            hiba := E.Message;
            if SslHttpCli.StatusCode <> 400 then
                ntLOG( '! HIBA a küldéskor. Vélhetően net-hiba: ' + CRLF + hiba, 1 );
            end;
        end;
    finally
        Result := (SslHttpCli.StatusCode = 200) or (SslHttpCli.StatusCode = 400);  // goverment is sending error messages if = 400

        try
            if SslHttpCli.RcvdStream.Size > 0 then begin
                SslHttpCli.RcvdStream.Seek(0, soFromBeginning);
                SetLength(JsonUTF8, SslHttpCli.RcvdStream.Size);
                SslHttpCli.RcvdStream.Read(JsonUTF8[1], SslHttpCli.RcvdStream.Size);
                valasz := UTF8Decode( JsonUTF8 );
                ntLOG( 'Answer:' + CRLF + valasz + CRLF + 'Header:' + SslHttpCli.RcvdHeader.Text, -1 );
            end
            else
                hiba := 'ERROR! 0 byte answer.';
        except on E: Exception do
            hiba := ifThen(hiba<>'', hiba + CRLF)  + 'Stream error: ' + CRLF + E.Message;
        end;

        SslHttpCli.SslContext := nil;
        try
            if Assigned(SslHttpCli.SendStream) then SslHttpCli.SendStream.Free;
            if Assigned(SslHttpCli.RcvdStream) then SslHttpCli.RcvdStream.Free;
            if Assigned(SslContext) then SslContext.Free;
            if Assigned(SslHttpCli) then SslHttpCli.Free;
        except end;
    end;
end;

 

[PizzaProgram 9]

Quote

 

Delphi 7 is end of life, ICS support has now ceased, I won't be running it up again now ICS v9 has been tested on it.

Also, running TSslHttpRest in a thread is not tested heavily (or atall), being async you can download hundreds of files in parallel in the main thread.  

 

 

Dear Angus,

 

- What do you mean by "now ICS v9 has been tested on it" ?
 That is the opposite of "now ceased". Also I'm still using v8.70, did not find any "v9" yet.

 

- As it turned out, I'm not using TSslHttpRest unit at all, only httpCli.  (It was a leftover from a test.)

Was the Cli part tested? (see my prev. post)

 

Thank you very much for all the answers and help for everyone !!!

[Angus Robertson 574]

Current versions of OpenSSL do not work on Windows XP, nor is ICS supported on XP.

 

I would suggest you remove all the GUI interaction except to a log file, so you know if the problem is ICS or the GUI.

 

Angus

 

[Angus Robertson 574]

ICS v9 is about to be released, see comments a week ago here,

 

Part of that release process is building ICS on Delphi 7 and I had to change several new components due to missing language features and components in Delphi 7.  I will not be doing that again, a waste of my time and potentially causing problems for newer compilers if I introduce new errors.

 

Angus

 

[PizzaProgram 9]

Quote

 

Current versions of OpenSSL do not work on Windows XP, nor is ICS supported on XP.

 

 

Sorry for mentioning XP. My App is checking for min. OS, and does not start the ICS background thread on those client PC. All "server-PCs" are min. Win7.

I just listed it to prove:

 - on those PCs, where ICS thread is not running: there are NO problems at all ! (ca. 100+ PCs)

 - but on those Win7+ OSs, where ICS thread is running in background, (50+ PCs) every few hours there is a GUI error.

 

Quote

I would suggest you remove all the GUI interaction except to a log file, so you know if the problem is ICS or the GUI.

Can you please explain, what do you mean by that?

The background thread does not interact with the GUI at all! None of my thread do ever.

(Except if ICS is doing it somehow without my knowledge?)

 

Again: Thanks for trying to help! 🙂

[Brian Evans 105]

Anything in the Windows event logs? 

 

For example an Event 4266, Tcpip  A request to allocate an ephemeral port number from the global UDP port space has failed due to all such ports being in use.

[Dalija Prasnikar 1396]

52 minutes ago, PizzaProgram said:

Can you please explain, what do you mean by that?

The background thread does not interact with the GUI at all! None of my thread do ever.

(Except if ICS is doing it somehow without my knowledge?)

It means that either code in the GUI or the code in thread is exhausting the memory. So you need to know which one causes the problem if you want to locate it. 

 

For start, just comment out all the code in thread and see how application behaves, then you can start turning on parts of the code in thread until you hit the issue. There is plenty of code we don't see here so it is hard to say what is the problem.

 

Just because FASTMM does not show the leak that does not mean that there is no memory leak, it is just that it is not classical memory leak where you allocate memory and you don't release it. Another form of a leak is that you keep allocating memory and holding on to it even after you no longer need it. such memory will be properly released on application exit, but while running application can still exhaust all available memory.

 

BTW, some of your code in thread is not properly handling allocations/deallocations. 

 

You have try...finally blocks that do more than they should, you don't protect all allocations with try block - the other option, would be to initialize all object instances to nil first, and you have unnecessary Assigned checks before calling Free which can safely run on nil object instances. All that tells me that your memory management code is not exactly stellar and that you probably similarly poor code in other parts of your application. So chances are rather high that your code somewhere in your application is causing you a trouble and that it has nothing to do with ICS alone.

 

However, EOutOfResources error is not about a memory allocation but leaking GDI objects. Those leaks will not be reported by FASTMM. Because it is wrapped inside GDICheck it is possible that there is no leak at all, but some other GDI error.

[PizzaProgram 9]

55 minutes ago, Brian Evans said:

Anything in the Windows event logs? 

Good idea!

 ... 30 min later:

Checked a few thousands of log entries, on 2 PCs, but nothing special. Only a few Schannel 40 and 70 errors, but found no evidence it would be my app related.

[aehimself 396]

According to the stack trace it is indeed GDI object exhaustion. Check where and how you are manipulating images and make sure you are disposing of them properly.

We had this when there was an image list on a frame which was created thousands of times. Moving the imagelist to a common place solved our issue immediately.

 

@Dalija Prasnikar we also received EOutOfResources when our application used up all available user handles so it’s not strictly GDI-related. But yes, leaking handles often pop up as Delphi classes in the memory leak report.

[PizzaProgram 9]

Dear Dalija,

Thank you very much for the long and detailed explanation !! 🙂

48 minutes ago, Dalija Prasnikar said:

It means that either code in the GUI or the code in thread is exhausting the memory.

But if that would be true, than a simple "process explorer" would show memory consumption of > 2GB, and not just 150MB. Am I wrong?

(Actually I'm using a more advanced program for that: SystemInformer which shows every tiny byte of memory allocation.)

 

Also MadExcept has a:

ShowLeakReport(GetLeakReport() );

function, I have used before the whole APP is closing.

There are no big leaks at my program!

 

46 minutes ago, Dalija Prasnikar said:

However, EOutOfResources error is not about a memory allocation but leaking GDI objects. Those leaks will not be reported by FASTMM. Because it is wrapped inside GDICheck it is possible that there is no leak at all, but some other GDI error.

Exactly !

And it only happens, if the ICS background thread is RUNNING. Never if it's stopped, or not started at all.

While if I check the GDI object number with a separate program, it is always less than 1000! (The default max is 10.000 in windows.)

 

That's what I do not understand.

[PizzaProgram 9]

I think I have to apologize:

I've re-checked everything, and realised:

 - Yes, the "private bytes" used are minimal,

 - but after just 2 hours of app running the Virtual Memory consumption grows heavily !

 (And I see continuously growing)

 

Private bytes, 85,72 MB
Peak private bytes, 116,15 MB
Virtual size, 1,18 GB
Page faults, 878 816
Hard faults, 2 114
Working set, 90,04 MB
Peak working set, 119,59 MB
Private WS, 73,84 MB
Shareable WS, 16,2 MB
Shared WS, 9,78 MB

 

GDI < 900

Handles < 400

 

So logically in every 2-4 hours it can reach 32bit limit. (2GB)

 

I'll investigate further.

There are ca 100 "Private: Reserved" 7-8MB segments.

[FPiette 383]

5 hours ago, PizzaProgram said:

Also I've bought MadExcept module, so I get great error-debug reports from everywhere. Like:

allocated memory   : 150,98 MB
largest free block : 6,05 MB
exception number   : 1
exception class    : EOutOfResources
...
main thread ($d7c):
0049126f +077 TermiPRO.exe Graphics                GDIError
004912a7 +007 TermiPRO.exe Graphics                GDICheck
0049545a +2d2 TermiPRO.exe Graphics                CopyBitmap
00495c5b +063 TermiPRO.exe Graphics                TBitmap.CopyImage
00497010 +03c TermiPRO.exe Graphics                TBitmap.SetHeight
0063be2b +103 TermiPRO.exe acSBUtils     1629  +20 PrepareCache

 

This stacktrace tells that the EOutOfResources is triggered when you manipulate a TBitMap. This doesn't mean that the TBitMap is the culprit.

[Dalija Prasnikar 1396]

4 hours ago, PizzaProgram said:

But if that would be true, than a simple "process explorer" would show memory consumption of > 2GB, and not just 150MB. Am I wrong?

Yes. I started with explanation how to approach memory allocation issues (since you mentioned FASTMM), but I should have been more clear that this part is most likely not related to your current issue.

Edited August 20, 2023 by Dalija Prasnikar

[Dalija Prasnikar 1396]

4 hours ago, PizzaProgram said:

And it only happens, if the ICS background thread is RUNNING. Never if it's stopped, or not started at all.

While if I check the GDI object number with a separate program, it is always less than 1000! (The default max is 10.000 in windows.)

If the issue happens only when background thread is running, then background thread creates a problem.  

 

4 hours ago, PizzaProgram said:

GDI < 900

Handles < 400

 

So logically in every 2-4 hours it can reach 32bit limit. (2GB)

This sounds like memory issue, if FASTMM is not registering the leak, you are probably allocating and holding on to some memory that accumulates inside the thread. Another potential issue without a visible leak could be a memory fragmentation and then you have an issue when you try to allocate larger block of memory like bitmap.

[uso 0]

15 hours ago, FPiette said:

This stacktrace tells that the EOutOfResources is triggered when you manipulate a TBitMap. This doesn't mean that the TBitMap is the culprit.

I once had this sort of problem (leaking GDI handles) by creating (and releasing)  > 46 TBitmap(s) per second.

The problem was not the TBitmap itself, the problem was somehow based on GDI ressources created in non UI threads have not been released propperly.

The only solution I found in my particular case was to redesign the code. I created a pool of bitmaps in the VCL main thread and reused the TBitmap instances in the other threads.

Edited August 21, 2023 by uso

[Angus Robertson 574]

There are two fundamental misuses of ICS in the code snippets supplied.

 

1  - the code is said to be running in a thread, but the MultiThreaded property of TSslHttpCli is never set, so messages for the thread will be processed using Application.ProcessMessages in a different thread.

 

2 - More seriously, the ICS components are being created and perhaps destroyed for each HTTPS request made, which is probably the cause of the memory leak, and is also highly inefficient. 

 

Specifically, OpenSSL is being loaded automatically by the components when the SslContext is automatically iniitialised by the request starting, and perhaps being unloaded when the request ends.  The SslContext is designed as something to be shared by components, initialised once and then reused.  Or OpenSSL can be loaded once when the program starts, to allow use with multiple SslContexts, in servers for instance that use multiple certificates.  Many of the ICS samples show how to load OpenSSL early.

 

ICS v9 has various improvements relating to freeing and destroying components, particularly when exceptions happen during that process, to ensure that inherited destroys are still called and not skipped which can cause memory leaks.

 

Having said that, reports of memory leaks using ICS are very rare, and many ICS applications run for weeks or months without a problem.

 

Angus

 

 

Angus

 

[PizzaProgram 9]

5 minutes ago, Angus Robertson said:

but the MultiThreaded property of TSslHttpCli is never set

Wow ! This could be it !!!

Thank your VERY much for looking into the code!

 

7 minutes ago, Angus Robertson said:

the ICS components are being created and perhaps destroyed for each HTTPS request made,

I've always thought this is safest method, instead of leaving a (most probably connection-error broken) component open for possible re-use.

There are normally 1-2 request-sends pro 1 minute, and the SSL header always changes by goverment request, so SSL has to be re-re-re-initialized anyway.

Some of my users have only 3G or DSL (max 40Kbit, instable) net connections, impossible to get better.

 

10 minutes ago, Angus Robertson said:

ICS v9 has various improvements

And You also said: it won't be compatible with Delphi 7 any more.

A new Delphi price is impossible high, and rewriting the whole codebase (1M+ lines) would be an impossible job. (For start: no IBX component available.)

(Would be easier to drop delphi completely, what I don't really want to.)

[Dalija Prasnikar 1396]

22 minutes ago, Angus Robertson said:

the code is said to be running in a thread, but the MultiThreaded property of TSslHttpCli is never set, so messages for the thread will be processed using Application.ProcessMessages in a different thread.

Well, that is rather unfortunate design choice as it can be easy for users of a library to overlook this requirement. 

 

Wouldn't it be better to autodetect whether component is created within background thread and set that property automatically rather than manually?