FastMM - Fragmentation Analysis?

[Lars Fosdal 1792]

We are fighting a memory leak, but neither FastMM nor EurekaLog reports what is leaking.

This is a service without a GUI, which uses COM objects, sockets and FireDAC databases.

 

Is there a non-UI FastMMUsageTracker anywhere that could shine light on if this is a fragmentation issue?

Are there other things that I should be looking at using?

[Guest]

Hope those are not wasting your time

 

1) COM objects, is there system/3rdparty provided COM or all are yours ? 

2) Some of those are very hard to track, and the only way is to modify FastMM to log everything in simple one line per allocate/free then with write small parser to parse that log file and list the only the culprit, now to make sure it is done right you need to take snapshot of it and don't wait until full finish (exit the application), this will only give you hint of what is going on and what size's to refine the search for, then go ahead and expand the log detail for specific size, or figure out the what size belong to what and increase the class/ interface size to pin point the culprit, or at least find a good start point in understanding the issue, now i talked about size and you can think of different things too, you got the idea and i believe you can expand on it.

3) I don't know how much you are familiar with exotic tools, for me anything that can hold water and take heat i might call it a cookware, so if you are experienced in Cheat Engine to peek in the memory of the application then you can use with VMMap to look at those segment you suspect are accumulating like dangling pointers, separated blocks...etc , but here a great deal of the your experience and knowledge of how a class or an interface looks raw in memory, this is the shortest method when the leak is big enough or repeated many time, but it the hardest to master.

[Guest]

Two extra things, 

4) Your application is multithreaded, log the ThreadID in the log file.

5) to shorten things a little, modify FastMM to allocate huge shared memory holding an array of integers, so for 1024*1024 *4 size you have an array that can hold size from 1 to 1mb increase the allocate item per size on get memory for that size and decrease it on free in atomic way, this can give you a look from outside in realtime of what is used and how they increase and decrease, i used this method and it does work like charm, don't know why it didn't came to mind sooner .

[Lars Fosdal 1792]

1) The COM objects are ancient and provided by the ERP system.

No new versions have been introduced for years.

We have very little code that uses interfaces.

 

2) We have an object hierarchy that tracks allocations and deallocations - and on app close, we report the stuff that has not been properly disposed of.

There is nothing in the list that stands out as gobbling memory.

 

3) I am familiar with tools for inspecting memory, but there are simply so much movement in memory that it is not viable.

 

4) We are logging threadid and FastMM allocation size throughout the logfile.

using this function

 

function MemoryUsed: cardinal; inline;
var
  MMS: TMemoryManagerState;
  Block: TSmallBlockTypeState;
begin
  GetMemoryManagerState(MMS);
  Result := MMS.TotalAllocatedMediumBlockSize + MMS.TotalAllocatedLargeBlockSize;
  for Block in MMS.SmallBlockTypeStates
   do Result := Result + (Block.UseableBlockSize * Block.AllocatedBlockCount);
end;

LineNo {Time (ThreadId) Memory} Logged text

   4331737 {10:44:39,145 (10488) 970150k} TPSDTask_HandleServerNotifications - End (id:113072)

 

The allocated memory that FastMM reports stays approx in tune with what the system reports.

 

 

5) We use TObjectList<T> - not arrays, and mostly for actual objects.

We have control over initial size and growth strategy, and adapt it to the respective usages by overriding virtual methods.

 

function TPSDBaseList<T>.DefaultCapacity: Integer;
begin
  Result := 32;
end;

function TPSDBaseList<T>.DefaultGrowBy: Integer;
begin
  Result := 32;
end;

 

 

It seems that the problem appeared when we started using 10.3.3 - but then again - we made a LOT of other changes for that release, so it may not be relevant.

[Guest]

28 minutes ago, Lars Fosdal said:

5) We use TObjectList<T> - not arrays, and mostly for actual objects.

I might be not clear there, and i think you didn't get, goes like this, 

modify FastMM to allocate big array in shard memory can be accessed from other application, this big chunk memory will hold an array simple array of integer, an item index represent the size, and that is it, allocation will increase that item in atomic manner and decrease it on free.

after that write small application that access that shared block and list what is not holding 0, means a block of size = index of the item, is allocated and used X times the value at that item, you can even expand that simple array to hold two integers, one for current and one for just increasing usage as counter.

[Guest]

Static array, that is the word. 

Allocate shared memory and use it as static array of integer.

[Erwin Mouthaan 7]

Nexus Quality Suite comes to mind. A trial version is available.

[Cristian Peța 103]

1 hour ago, Lars Fosdal said:

neither FastMM nor EurekaLog reports what is leaking

But reports something or doesn't report anything?

[Lars Fosdal 1792]

Both report something, but it is just a few K data which are all known leaks related to startup configuration.

[Guest]

I forgot to mention a powerful too.

http://www.rohitab.com/apimonitor

Grab that APIMonitor select all memory handling API's like Virtual, Heap, Global ..etc , there is many groups but the more you get familiar with it the more you will use it efficiently, the stack is not very friendly with Delphi application so you need to find calls by address, and the filter does have limitation, so if you can pin the type of those unallocated calls then you can parse the logs yourself for allocate and free 

[Cristian Peța 103]

Then it looks like it's deallocated at close. Maybe if you can debug and look in task manager. Put a break point in System._Halt0 and maybe you will see where is deallocated.

Maybe deallocation is in finalization sections or some dll's.

Maybe this MemoryUsed function can be called from watch list and compared with Task manager.

Edited July 23, 2020 by Cristian Peța

[Mahdi Safsafi 225]

FastMM is not aware about external allocated resource such VirtualAlloc/. A non closed handle can leak too.

Try to use madExcept as its more aware about many external allocation.

[A.M. Hoornweg 144]

2 hours ago, Lars Fosdal said:

We are fighting a memory leak, but neither FastMM nor EurekaLog reports what is leaking.

This is a service without a GUI, which uses COM objects, sockets and FireDAC databases.

 

Is there a non-UI FastMMUsageTracker anywhere that could shine light on if this is a fragmentation issue?

Are there other things that I should be looking at using?

Maybe you should look at allocations that bypass Delphi's memory manager and hence not get caught by FastMM4. I mean stuff like VirtualAlloc, GlobalAlloc, LocalAlloc and HeapAlloc. Are you using any external DLLs written in C ?.   

 

You max need a tool like AQTime or Deleaker to track down such leaks.

 

[Guest]

This 

Quote

The allocated memory that FastMM reports stays approx in tune with what the system reports.

Eliminate external memory allocation, so it is within the application, tracking FastMM internally as i described would be the best solution.

Unlike the IDE and its memory problem, when using APIMonitor you can see tens of thousands of allocation on heap coming from .Net stuff, but who knows Lars might found the core problem of his application and pin point the problem if the IDE itself.

 

After thinking, if the same method i suggested can be used on the IDE with Detours to track the heap functions too !

 

Where are you @Mahdi Safsafi ?, and how to tackle this right ? 

May be expanding/injecting BorlandMM with logger not only for the allocated memory but to capture the heaps operations too.

And under no circumstances i am asking you to do it, just your insight if this is doable, after all it is Embarcadero guys to figure it out and solve it, they been paid to that.

[Mahdi Safsafi 225]

54 minutes ago, Kas Ob. said:

Eliminate external memory allocation, so it is within the application, tracking FastMM internally as i described would be the best solution.

Yes possibly true.

Quote

After thinking, if the same method i suggested can be used on the IDE with Detours to track the heap functions too !

Where are you @Mahdi Safsafi ?, and how to tackle this right ? 

 May be expanding/injecting BorlandMM with logger not only for the allocated memory but to capture the heaps operations too.

Knowing if there is a leak is somehow easy to implement but tracing it is the problem. To give a nice report about the leak you need to hook every external function that may allocate memory such CreateSolidBrush and hook every function that release the memory such DeleteObject ... How many Api are there ? 

Besides, your engine needs to allocate data too. For each called function that allocates memory, an X bytes of stack must be saved for later report (trace)! 

As you can see this is not something easy to implement if you just consider the amount of Win32Api function that may allocate !

Quote

after all it is Embarcadero guys to figure it out and solve it, they been paid to that.

I truly respect the way you're thinking but in my opinion, no ! they're not obliged to do that ! Memory leak report/detection is just a sugar for the RTL. They're paid to maintain/extend standard not to implement sugar stuff(that's why 3rd party exists). 

[Vincent Parrett 750]

Perhaps using something like VMMap might help - https://docs.microsoft.com/en-us/sysinternals/downloads/vmmap

[Lars Fosdal 1792]

We don't use direct Windows heap allocations at all. Nor do we use C DLLs.

I starting to suspect that we have stale objects lingering in lists, which gets cleared at app exit - hence no leaks.

[A.M. Hoornweg 144]

You could use OutputDebugString in conjunction with Sysinternals' DebugView  to monitor the size of your lists in certain time intervals. 

That way you can observe if one of your lists is growing out of control.

 

DebugView can do tcpip client/server by the way, that way you can observe the debugging output on a different computer.

[Guest]

39 minutes ago, Mahdi Safsafi said:

I truly respect the way you're thinking but in my opinion, no ! they're not obliged to do that ! Memory leak report/detection is just a sugar for the RTL. They're paid to maintain/extend standard not to implement sugar stuff(that's why 3rd party exists). 

I think you missed my point here, i was talking about the fragmented memory of the IDE itself, it is their job to deliver stable IDE that doesn't disappear out of the blue due memory problem, it is their doing and their job.

42 minutes ago, Mahdi Safsafi said:

Knowing if there is a leak is somehow easy to implement but tracing it is the problem. To give a nice report about the leak you need to hook every external function that may allocate memory such CreateSolidBrush and hook every function that release the memory such DeleteObject ... How many Api are there ? 

Also here the talk about is about the IDE itself by hooking Heap Alloc/Free API's them self first to identify if they are the cause of ever growing usage causing memory fragmentation reserved by the IDE and its components, then try to make them use the faster memory manager FastMM itself, this might even increasing the compiling speed and fix the stability of the IDE.

Going after handles and user objects is hard and there is CodeWatch from Nexus Quality Suite does that, but i merely want to track size's of allocation and see if there is ever grow of usage of specific sizes to narrow the window of the cause.

 

Let me show you something in this

Those are four sample from that API monitor some un describable behaviour

what dcc32 is trying to do with freeing that memory like that, and that may be right or not, not my call, but the other screen shot which is coming from using .NET (most likely the fossilized v2, i may be wrong though), but look at the timestamp and size and see if that is right behaviour or acceptable, now the question is will redirecting those calls to FastMM help the performance of the IDE and the compiler by boost the speed of the .NET stuff ?  or even catch the dangling/stale objects?

 

around 2 years ago i started this thing, i mean tried to redirect all calls from the OS API itself to FastMM, and had difficulty in handling lock and unlock, as i skipped the locking mechanism, it didn't work and then i got busy and bored with it, then dropped the idea, now i can't find my scratch project, because simply can't recall what i named that project !

 

54 minutes ago, Vincent Parrett said:

Perhaps using something like VMMap might help

It does provide limited help, i think i pointed that somewhere in this forum (may be wrong though and it was somewhere else, can't remember), and the point was those 4k and 64k which are visible in the above screenshot.

 

51 minutes ago, Lars Fosdal said:

I starting to suspect that we have stale objects lingering in lists, which gets cleared at app exit - hence no leaks.

Most likely it is, and if the reported memory usage by MM is almost the same from the TaskManager or VMMap then i suggest to probe FMM as i suggested, and wish you good luck.

[Mahdi Safsafi 225]

12 minutes ago, Kas Ob. said:

I think you missed my point here, i was talking about the fragmented memory of the IDE itself, it is their job to deliver stable IDE that doesn't disappear out of the blue due memory problem, it is their doing and their job.

My bad ... I must be very slow at understanding.

Quote

what dcc32 is trying to do with freeing that memory like that, and that may be right or not, not my call, but the other screen shot which is coming from using .NET (most likely the fossilized v2, i may be wrong though), but look at the timestamp and size and see if that is right behaviour or acceptable

Yes the timestamp/size is interesting here !

Quote

around 2 years ago i started this thing, i mean tried to redirect all calls from the OS API itself to FastMM

I wan't do such a thing! After all FastMM is relying on OS Api. Beside do you have any guarantee that those calls do not imply some constraints like expecting the allocated memory to be filled with zero or expecting the memory to be allocated at certain range ? or assuming some memory access right ?, or ...

Moreover, the purpose of memory management is to handle small/medium size. for large block, you should call OS Api instead. 

Its likely that doing this will ends up on having an unpredictable/undefined behavior. So you should leave OS stuff to the OS memory manager.

[Guest]

15 minutes ago, Mahdi Safsafi said:

I wan't do such a thing! After all FastMM is relying on OS Api. Beside do you have any guarantee that those calls do not imply some constraints like expecting the allocated memory to be filled with zero or expecting the memory to be allocated at certain range ? or assuming some memory access right ?, or ...

Moreover, the purpose of memory management is to handle small/medium size. for large block, you should call OS Api instead. 

Its likely that doing this will ends up on having an unpredictable/undefined behavior. So you should leave OS stuff to the OS memory manager.

It was just an idea for amusement and research, but i might try it again, what worse thing can happen fail to run .net stuff, because the hooking then redirecting will be selective based on the return address, if the call came is from those few specific .Net core DLL's, such thing can't be predicted i agree but there is a chance they will give better understanding what is going on, and there is small chance it might work, i really want my favorite IDE's ( they all are old) to work better and may more stable.

 

As you saw, thousands of calls just like allocate 6 bytes just to free them ! , they must picked the best .NET framework version to build on, and forgot to check if newer will be better.

[Mahdi Safsafi 225]

4 minutes ago, Kas Ob. said:

such thing can't be predicted i agree but there is a chance they will give better understanding what is going on, and there is small chance it might work, i really want my favorite IDE's ( they all are old) to work better and may more stable.

Literally you're no longer coding ... you're gambling. 

If some function X expects 32 byte alignment from OS and somehow you managed to redirect the call to FastMM that yields 16B aligned memory ... what would be you expectation ? 

Don't waste you time on doing something that is theoretically invalid.

[Fr0sT.Brutal 900]

FastMM can dump the whole heap, this could help in those "pseudo-leaking" objects that grow in number at runtime but are correctly disposed on exit

[Guest]

6 minutes ago, Mahdi Safsafi said:

Literally you're no longer coding ... you're gambling. 

If some function X expects 32 byte alignment from OS and somehow you managed to redirect the call to FastMM that yields 16B aligned memory ... what would be you expectation ? 

You are right here, on other hand if the target is HeapAlloc only, then as per documentation which doesn't mention alignment it might work, the bigger and harder problem to solve is with locking (HeapLock/UnLock) if it is essential functionality for MSBuild within .NET framework to work.

 

Anyway, it was just food for thought.

[Mahdi Safsafi 225]

19 minutes ago, Kas Ob. said:

You are right here, on other hand if the target is HeapAlloc only, then as per documentation which doesn't mention alignment it might work, the bigger and harder problem to solve is with locking (HeapLock/UnLock) if it is essential functionality for MSBuild within .NET framework to work.

Even so, there are other requirements like LastErrorCode. a logic may call GetLastError after calling HeapAlloc. FastMM doesn't probably use SetLastError and even if it does ... does it emulate HeapAlloc behavior ?

 

 

Quote

Anyway, it was just food for thought.

Having some thought is really good. You just need to do it right in a way that you don't waste your valuable time.