Generics and Classes on Windows 2000 = OOM

[aehimself 396]

Hello all,

 

After switching from D7 to D10.2 I re-wrote one of my multithreaded apps with proper OOP, using the "new" language features. Each thread has a work queue, which used to be a Array Of String, storing the properties separated with tabs. Now, the work queue is a TObjectList, and work queue items are classes, created upon adding, freed after successful processing. Everything works like a charm, there are no memory leaks reported whatsoever even after a stress-test of the work item part.

 

After about a week of uptime my Windows 2000 test system reaches 0% of free memory and becomes unstable, requiring a reboot. Unfortunately there are a lot of factors to consider here:

- The ESX host the test VM is running on was patched a couple of times (and because of that vmWare tools got updated on the guests)

- The obvious, Array to TList and String to Class conversion

- Only a Windows 2000 system is affected, Windows 2003 and above accepted the change well

 

I will attempt to create a new Windows 2000 machine without vmWare tools to see if it makes any difference, however I believe that TList will be the root cause here; which got reinforced after seeing

 

Did anyone met this symptom before? How can I be sure the root cause is my application and not the prehistoric OS? Maybe the combination of both? Is there a (even a hacky) way to force a Delphi application to release any currently not used memory?

 

Thanks all!

[David Heffernan 2345]

Didn't you read that topic? The entire topic was based on misconception. 

 

[David Heffernan 2345]

It's very plausible that you have a memory leak. Just because fastmm4 says you don't doesn't mean you don't. It's one thing returning memory when the program closes, another returning it in a timely fashion during execution. 

Edited March 8, 2020 by David Heffernan

[aehimself 396]

2 hours ago, David Heffernan said:

Didn't you read that topic? The entire topic was based on misconception. 

 

I did, this is why I linked a post, not the topic itself. I personally don't have this deep knowledge of how the language works but as noone opposed Pawel's statement I considered it true.

 

2 hours ago, David Heffernan said:

It's very plausible that you have a memory leak. Just because fastmm4 says you don't doesn't mean you don't. It's one thing returning memory when the program closes, another returning it in a timely fashion during execution. 

I not just went back to ensure all blocks are surrounded by a Try ... Finally ... FreeAndNil ... End; FastMM and MadExcept both says there are no leaks at all. The average alive time of a work queue item is a couple of seconds, and as I'm using TObjectList.Create(True) as the queue, I can be sure that work items are disposed upon removal.

Furthermore, as I mentioned the problem only appears on a Windows 2000 machine with (currently) 1 GB of memory (previous solution ran fine on 64 MB); a Windows 2003 R2 with 128 MB and a 2008 with 512 MB is running the tool fine for weeks now (since the latest patch).

 

In the mean time, I set up a basic VM with 128 MB of RAM and no vmWare tools. We'll see how long it will last.

 

P.s.: I don't want anyone to find the issue for me; I'm looking for directions on how to find these by myself. So any tips are highly welcome.

Edited March 8, 2020 by aehimself

[aehimself 396]

One more important detail. OOM is given by the OS; not by the application itself.

 

[David Heffernan 2345]

Smells like a memory leak, or perhaps the loss of contiguous address space due to fragmentation from reallocation. 

[aehimself 396]

Just now, David Heffernan said:

Smells like a memory leak, or perhaps the loss of contiguous address space due to fragmentation from reallocation. 

I do think the same way; but correct me if I'm wrong: if an application wants to allocate a specific size of memory and that space is not available, shouldn't the application throw the error...? Or again, I'm expecting too much from an ancient technology?

 

I already have experience with memory, handle leaks and memory corruption, but never had to deal with fragmentation before. Is there a way to detect it from the application?

[Anders Melander 1775]

FWIW, Windows 2000 isn't supported by the version of Delphi you're using.

 

Also, please read the event log message again. It has all the clues:

• It isn't your application that is experiencing an "out of memory" error.
• It's Windows that is experiencing an "out of virtual memory" error.
• To fix it: Increase the size of the page file.

My guess is that the W2K system has a fixed size page file - or no page file at all. Otherwise it would just increase the size automatically - or maybe that feature was added later. I forget. and who cares anymore.

 

If you're really interested in why you get a windows error and not an application error, read some books on Windows internals and the virtual memory manager, or just google it.

[aehimself 396]

Windows 2000 indeed has a configurable page file size. According to Microsoft's standards I'm always setting it to 1,5 times the physical memory size (adjusted every time the memory was expanded).

And again - before generics, 64 MB RAM and 96 MB of pagefile was sufficient. After Generics Windows reaches OOM with 512 MB RAM and 768 MB of page file.

 

Guess I'll have to wait for my secondary VM to produce (or not to produce) the symptoms. I'm thinking if this will be related to desktop heap (I'll try it once any VMs produce the same symptoms).

Still - is there a way for a Delphi application to forcefully release unused heap memory somehow?

[RonaldK 18]

1 hour ago, aehimself said:

Still - is there a way for a Delphi application to forcefully release unused heap memory somehow?

Maybe you can track down the memory usage with FastMM4

 

uses
  FastMM4;
...
var
  LMemoryStart : TMemoryManagerUsageSummary;
  LMemoryNow   : TMemoryManagerUsageSummary;
begin
  GetMemoryManagerUsageSummary(LMemoryStart);
...
  GetMemoryManagerUsageSummary(LMemoryNow);
  LMemoryNow.AllocatedBytes - LMemoryStart.AllocatedBytes;

 

[David Heffernan 2345]

Strange suggestions given that all symptoms point to memory leak in your code. 

[Sherlock 663]

2 hours ago, aehimself said:

And again - before generics, 64 MB RAM and 96 MB of pagefile was sufficient. After Generics Windows reaches OOM with 512 MB RAM and 768 MB of page file.

Wait. What? Megabytes? Really? I do recall W2k being quite thrifty with memory but that does not sound very likely.

[aehimself 396]

Yes. Considering built-in devices (robotic arms, controllers, etc.) where you can not upgrade the hardware and/or the software, I like to experiment with the absolute minimum.

At the moment my 1 GB Win2K is actively using ~104 MB, while the 128 MB Win2k is using only 11 MB of RAM:

 

[Anders Melander 1775]

8 minutes ago, aehimself said:

Yes. Considering built-in devices (robotic arms, controllers, etc.) where you can not upgrade the hardware and/or the software, I like to experiment with the absolute minimum

That makes good sense, but why limit virtual memory? Doesn't the devices have hard disks?

[aehimself 396]

Hard drives as you might think - not necessarily. They often have a chip soldered on the board 🙂

[Anders Melander 1775]

2 minutes ago, aehimself said:

Hard drives as you might think - not necessarily

Okay. Now the use of W2K makes much better sense. Interesting constraints.

 

Anyway, back to the problem. If you can observe that virtual and physical memory consumption increases during execution, but all resources are released before the process is terminated (thus no leaks detected), then you can force the allocated memory to be reported as leaks by terminating the application prematurely with a call to Halt.

[Bill Meyer 337]

4 hours ago, aehimself said:

Yes. Considering built-in devices (robotic arms, controllers, etc.) where you can not upgrade the hardware and/or the software, I like to experiment with the absolute minimum.

At the moment my 1 GB Win2K is actively using ~104 MB, while the 128 MB Win2k is using only 11 MB of RAM:

You appear to assume that this environment will continue to be viable as new versions of Delphi roll out. Unfortunately, the prevailing assumption in the development world is that a) memory is cheap, and b) it is measured in GB. Even without generics, the difference in an application built in Delphi XE and Delphi 10.2 is huge. In the case of a small app of my own, XE produced a 3MB executable, while 10.2 was over 70MB. These numbers are not horrible, but in the context of Windows 2000 treated as an embedded system, I think you are nearing the end of the line for those old systems. You may buy some time by avoiding generics, but sooner or later, you will hit a wall. In a system where your available resources are soldered into the product, you may well need to support a version for those systems which builds in an earlier version of Delphi, and perhaps cannot acquire the same new features as your main application.

[Fr0sT.Brutal 899]

19 hours ago, Sherlock said:

Wait. What? Megabytes? Really? I do recall W2k being quite thrifty with memory but that does not sound very likely.

I used to work on PC win Win2k having only 200Mb RAM, it was enough for D7, surfing with Opera, ICQ and some other apps.

14 hours ago, Bill Meyer said:

In the case of a small app of my own, XE produced a 3MB executable, while 10.2 was over 70MB.

You are likely comparing release mode with debug mode with full debug info compiled in. F.ex., full test suite of Zeos DB framework with all supported DB drivers being compiled with 10.1 occupies just 6.5M with 5.5M map file and 27M rsm file.

[aehimself 396]

One more thing which comes with generics is how I enumerate through them. And I always had issues understanding dynamic arrays.

 

Let's say I have a method:

Function TMyClass.Method: TArray<String>;
Begin
 Result := _dictionary.Keys.ToArray;
End;

And then I do a:

For s In MyClassInstance.Method Do

It is allocating a TArray, which is not finalized. Does it mean that the memory will be released when the application closes, or when the caller method ends...?

 

This basically applies to all dynamic arrays; and I think I never finalized them until now.

 

Could this "leak"?

[David Heffernan 2345]

49 minutes ago, aehimself said:

Could this "leak"?

no

[Fr0sT.Brutal 899]

17 hours ago, aehimself said:

 And I always had issues understanding dynamic array

 

Consider them behave like interfaces. A variable keeps reference counter which is increased on assignment and decreased when exiting from scope. There are only a few cases when you need to finalize string/array manually. One of them is to free a valuable amount of memory occupied by a variable ASAP without waiting for subroutine end.

[aehimself 396]

4 hours ago, Fr0sT.Brutal said:

Consider them behave like interfaces. A variable keeps reference counter which is increased on assignment and decreased when exiting from scope. There are only a few cases when you need to finalize string/array manually. One of them is to free a valuable amount of memory occupied by a variable ASAP without waiting for subroutine end.

Yep, I guess my question would have been better if I ask what exactly the scope is. I quickly discarded this option though as if it would be a memory leak, it would appear on all systems, not only on Win2k.

I started a perfmon on the two machines to see how the processes and system memory is changing. Resource usage of my executable did not grow with a single byte in the past 24 hours. In fact, Working Set dramatically decreased from ~14 MB to 1,2 - guess some stuff was paged out.

 

I really suspect this will be Windows- or ESX related. Not Delphi or my code.

[Stefan Glienke 2001]

Dynamic arrays cannot leak unless you circumvent the compiler generated code controlling their reference counting.

[aehimself 396]

I was running a close resource monitoring of the named application, having no unexplainable jumps whatsoever. It's only been a week so cannot be sure, but I strongly believe that whatever caused the OOM was / is out of my control. I set the affected W2K machines memory back to 128MB to try to reproduce the issue in the mean time.

 

Theoretical question. Is seems the scope of local variables is the end of the method. So if I have

Procedure TMyThread.Execute;
Var
 tdictkeys: TArray<String>;
Begin
 While Not Terminated Do
  tdictkeys := _privatedict.Keys.ToArray;
End;

...where the amount of items are growing in the dictionary (by an other thread for example, imagining thread safety non-existent for this example), does it mean that the array will keep relocating (without releasing the previous memory area); without a detectable leak but effectively causing OOM?

 

On 3/8/2020 at 11:58 PM, Anders Melander said:

FWIW, Windows 2000 isn't supported by the version of Delphi you're using.

Correct me if I'm wrong but a Win32 binary is a Win32 binary. A Win32 compatible OS should be able to run it, no? It's a bunch of assembly instructions at the end, after all.

[Vandrovnik 211]

5 minutes ago, aehimself said:

Correct me if I'm wrong but a Win32 binary is a Win32 binary. A Win32 compatible OS should be able to run it, no? It's a bunch of assembly instructions at the end, after all.

New versions of compiler may use new functions in Windows API, that are not present in Windows 2000. So application will start and then complain about missing something (or start and immediatelly die silently).