memory; Memory not freed in Linux but freed in Windows

[michel.seicon 1]

Hello,

The memory is not freed and the program keeps the memory, this does not happen in Windows and only in Linux. Below is a simple example,

but in everyday if the system is running for a few days the memory usage ends up being excessive,

this does not happen in Windows Is necessary restar the program everday.

 

Affected

Debian 12, Delphi 11.2 Path 1

Ubuntu 22.04 Delphi 12.1.

 

Sample simple program

 

program Project1;

{$APPTYPE CONSOLE}
{$R *.res}

uses
  System.SysUtils,
  System.Classes;

type
  TListaString = class
    V_String:String;
  end;

var
  V_Listas: TList;
  ii, i   : integer;
  V_Lista : TListaString;
  V_Run   : Boolean;

begin
  V_Run := true;
  V_Listas:=TList.Create;
  try
    while true do
    begin
      if V_Run then
      begin
        V_Run := false;

        for i := 0 to 3000 do
        begin
          V_Lista            := TListaString.Create;
          for ii             := 0 to 10000 do
            V_Lista.V_String := V_Lista.V_String + IntToStr(Random(99999));
          V_Listas.Add(V_Lista);
        end;

        writeln('create 1');
        sleep(5000);

        for i := 0 to 3000 do
        begin
          V_Lista            := TListaString.Create;
          for ii             := 0 to 10000 do
            V_Lista.V_String := V_Lista.V_String + IntToStr(Random(99999));
          V_Listas.Add(V_Lista);
        end;

        writeln('create 2');
        sleep(5000);

        for i := 0 to 3000 do
        begin
          V_Lista            := TListaString.Create;
          for ii             := 0 to 10000 do
            V_Lista.V_String := V_Lista.V_String + IntToStr(Random(99999));
          V_Listas.Add(V_Lista);
        end;

        writeln('create 3');
        sleep(5000);

        while V_Listas.Count > 0 do
        begin
          TListaString(V_Listas[0]).V_String := '';
          TListaString(V_Listas[0]).Free;
          V_Listas.Delete(0);
        end;
        V_Listas.Free;
      end;

      writeln('free, but not free');
      sleep(5000);

    end;

    { TODO -oUser -cConsole Main : Insert code here }
  except
    on E: Exception do
      writeln(E.ClassName, ': ', E.Message);
  end;

end.

Edited April 29 by michel.seicon

[Lars Fosdal 1793]

The while true causes an infinte loop, which doesn't do anything but looping and sleeping after the first run, so why this can't run for days and days witout running out of memory, is a bit of a mystery.

How do you monitor the memory usage?

 

Building these fairly large strings will have the MemoryManager allocate memory blocks which will remain allocated (and reusable) after you free your classes. These will be allocated from system memory until the application exits. 

In some corner cases, you can have memory fragmentation, which is caused by the elements in the list of released blocks of memory, being to small to be reused to hold the new objects, but in you example, that is not the issue.

 

The line

TListaString(V_Listas[0]).V_String := '';

is not necessary, as the string type is reference counted and automatically will be released.

Apart from that,  I guess there is a point to why you are NOT using TStringList, but using other containers and methods in classes to encapsulate reused code, could greatly reduce the amount of code duplication - which again reduces the potential for errors.

See System.Contnrs TObjectList and System.Generics.Collections TObjectList for examples of lists that can manage your objects.

 

Tip - For the purpose of testing, you would want each time you test to be identical - hence setting RandSeed to a fixed value at the start will ensure you have repeatable execution.

 

[michel.seicon 1]

Hello Lars Fosdal

Thank you for your lesson, but this does not explain the reason for the memory not being released while the program is running.

Of course, this was a small example that we managed to diagnose in the middle of our source.

In the example above, if you increase the loop so that it consumes 4gb, and at the end release the objects, the correct thing would be to release this memory to the OS

The problem is that in our system, if we manipulate images, files, and send them via Sockets then over time the system simply consumes a lot of memory and never releases it.

As I mentioned, this problem simply does not occur on Windows and only on Linux.

Obs: In Lazarus/Freepascal  the problema not occur.

Edited April 30 by michel.seicon

[Remy Lebeau 1436]

3 hours ago, michel.seicon said:

In the example above, if you increase the loop so that it consumes 4gb, and at the end release the objects, the correct thing would be to release this memory to the OS

Unfortunately, that is not how Delphi's memory manager works, on any OS.  Freed memory is held onto and reused, it is rarely returned to the OS until the app is exited.

3 hours ago, michel.seicon said:

The problem is that in our system, if we manipulate images, files, and send them via Sockets then over time the system simply consumes a lot of memory and never releases it.

You should consider re-writing your code to make better use of memory buffers so they are (re-)allocated and freed less often.

3 hours ago, michel.seicon said:

As I mentioned, this problem simply does not occur on Windows and only on Linux.

That is not true.  Memory management and fragmentation are important issues to be aware of on Windows too, or any OS for that matter.

[michel.seicon 1]

Hey guys,
Note that this example I posted is a simple code that shows that in Linux the memory is simply not freed even if the object is freed, 
in Windows this simply does not happen, anyone who has doubts can do the test and check.
Unused memory must be returned to the OS, because if this did not happen, imagine how database systems or systems that need to be running constantly would increase memory usage due to fragmentation, 
this does not make any sense. 
As I said , test the code on Linux and Windows to see the result.

Obs: In (Lazarus/Freepascal/Linux/Windows) , (Delphi/Windows) , the problema not occur.

Only Delphi/Linux

 

I posted

https://embt.atlassian.net/servicedesk/customer/portal/1/RSS-744

 

 

Edited May 1 by michel.seicon

[michel.seicon 1]

I don't need to use Google, 
I'm testing it in practice, 
please use the example like this and test it on Linux, Windows with Delphi and Lazarus so you can prove what I say. 
I prefer to check what I'm seeing than messages on Google that don't reflect reality.

[Lars Fosdal 1793]

Another artifact I find alarming: change it to a Windows 64-bit build, and the performance drops like a rock.

32-bit (around 6k ms)

 

64-bit (around 34k ms per loop)

[DelphiUdIT 187]

2 hours ago, Lars Fosdal said:

Another artifact I find alarming: change it to a Windows 64-bit build, and the performance drops like a rock.

32-bit (around 6k ms)

 

64-bit (around 34k ms per loop)

Mine ...

Edited May 1 by DelphiUdIT

[Lars Fosdal 1793]

That is weird! Maybe it is EurekaLog?

 

[DelphiUdIT 187]

11 minutes ago, Lars Fosdal said:

That is weird! Maybe it is EurekaLog?

 

I don't know, I use a laptop with I7 12700H RAM DDR5.

[Lars Fosdal 1793]

It was Eurekalog. 

32-bit


64-bit

 

Lenovo P16
Processor    12th Gen Intel(R) Core(TM) i7-12850HX, 2100 Mhz, 16 Core(s), 24 Logical Processor(s)
 

[DelphiUdIT 187]

This is with the use of EmptyWorkingSet ... the working set is the same value of used memory that TaskManager report ...

 

Edited May 1 by DelphiUdIT

[michel.seicon 1]

I think Embarcadero didn't do a good job on the compiler for Linux.

Does anyone have Kylix to test it with an example.

So we can isolate the problem in the Compiler or in the Memory Manager used for Linux.

There must be few systems developed for Linux in Delphi running as a service.

So these bugs should start to appear as more systems appear, like my case.

Function like

SetProcessWorkingSetSize(MainHandle, $FFFFFFFF, $FFFFFFFF) ;

 

They are not necessary

The tests I carried out were
Delphi Windows ok
Delphi Linux problem
Lazarus Windows ok
Lazarus Linux ok

Kylix Linux untested(need help)

 

 

 

[michel.seicon 1]

I just did an interesting test.
Using wine on Linux to run the program made on Windows, the memory is freed normally as expected.

The problem is isolating itself in Delphi/Linux

[Lars Fosdal 1793]

@michel.seicon In theory, it could be related to

Ref. https://stackoverflow.com/questions/48651432/glibc-application-holding-onto-unused-memory-until-just-before-exit

Delphi memory manager on Linux wraps malloc, so the question then becomes - is it possible to invoke malloc_trim from Delphi for Linux?

 

I suggest you clean up the test program to run correctly, and report it at https://qp.embarcadero.com with a proper description of the differences in behavior.

That way we might get some information from EMBT.

[Lars Fosdal 1793]

EmptyWorkingSet discussion moved to

on request from @DelphiUdIT

[michel.seicon 1]

@Lars Fosdal

Dear Lars Fosdal

Thank you very much for your tip.

To solve the problem, simply call the "libc" library function.

Below if you have the same problem

 type
  TMalloc_trim = function: Integer; cdecl;

 var
 LibHandle: THandle;
 Malloc_trim:TMalloc_trim;
 const
 SOName = 'libc.so.6';
begin
 LibHandle := LoadLibrary(PChar(SOName));
 Malloc_trim := GetProcAddress(LibHandle, PChar('malloc_trim'));
 
if Malloc_trim=1 then
showmessage('Success')
esle
showmessage('Memory freeing not allowed').

Please close this case.

 

[Lars Fosdal 1793]

6 hours ago, michel.seicon said:

Please close this case.

Awesome!