aehimself

Hello all, After switching from D7 to D10.2 I re-wrote one of my multithreaded apps with proper OOP, using the "new" language features. Each thread has a work queue, which used to be a Array Of String, storing the properties separated with tabs. Now, the work queue is a TObjectList, and work queue items are classes, created upon adding, freed after successful processing. Everything works like a charm, there are no memory leaks reported whatsoever even after a stress-test of the work item part. After about a week of uptime my Windows 2000 test system reaches 0% of free memory and becomes unstable, requiring a reboot. Unfortunately there are a lot of factors to consider here: - The ESX host the test VM is running on was patched a couple of times (and because of that vmWare tools got updated on the guests) - The obvious, Array to TList and String to Class conversion - Only a Windows 2000 system is affected, Windows 2003 and above accepted the change well I will attempt to create a new Windows 2000 machine without vmWare tools to see if it makes any difference, however I believe that TList will be the root cause here; which got reinforced after seeing Did anyone met this symptom before? How can I be sure the root cause is my application and not the prehistoric OS? Maybe the combination of both? Is there a (even a hacky) way to force a Delphi application to release any currently not used memory? Thanks all!

So after weeks of attempting to find the guilty code piece I found nothing. Every allocation had a release pair, memory usage was perfectly consistent for close to a month. This combined with the fact that only one machine was affected means that whatever happened, it was not caused by my code (and therefore generics and classes) and was only a coincidence. And I think I already know what happened. So thanks all for the tips; at least we know that (even if it's unsupported) D10.3 applications are running happily (and correctly) on Windows 2000 too 🙂

Sorry, true that; it's physical cores not logical ones. Please disregard my previous comment.

How exactly are you spawning the threads? Afaik, TThread.Create will always use the same logical CPU core as your main application. To make Windows to spread workers on multiple cores you need to use BeginThread... at least this is how it worked back in D7.

I wrote a beautifier - minimizer for XML; HTML should work with minor adjustments. If you wish I can share the source.

I completely agree with you on this one. this is just how I started, and later continued to think. We still have to agree that in a Client-Server architecture, where you have to create the filter on the Client (and have no access to dataset parameters) working with a number is easier. I already had enough of TO_DATE and CONVERT for a lifetime. Well, yes - that is my idea. Convince, but still controllable. And as an end user I like to have control

True, from this perspective it's useless. My problem with TDateTime values is that they are local; once you decide to go international you'll have to change the codebase. I mean for example a record was created at 21:00:00 in China. If someone from the US checks it, it will only tell that the record was created at 21:00:00 but that will be incorrect as 21:00:00 in China is 03:00:00 in the US. As I'm thinking in UNIX timestamps only lately I have a library to take care of these conversions for me in all of my applications. As for the database storage, the only important factor is that your app should know how to handle this data; but most DB engines have UNIX encoding functions (e.g. FROM_UNIXTIME in MySQL). I also wrote my own database explorer with these built-in: But, I don't want to hijack the topic. @Ian Branch If you can not rely on the OS, the easiest solution is if you add a drop-down menu in your settings to choose the appropriate location. If you want an automatic solution, my idea would be to use an IP to location API to detect the country the PC is connecting to the Internet from. This method won't work if they are using VPN, though; so maybe you can just add "Automatic" in the drop-down. Normally they are being detected, but they can override it in case.

Spot on, did not think about this. It would be pretty easy to diagnose, though.

I was running a close resource monitoring of the named application, having no unexplainable jumps whatsoever. It's only been a week so cannot be sure, but I strongly believe that whatever caused the OOM was / is out of my control. I set the affected W2K machines memory back to 128MB to try to reproduce the issue in the mean time. Theoretical question. Is seems the scope of local variables is the end of the method. So if I have Procedure TMyThread.Execute; Var tdictkeys: TArray<String>; Begin While Not Terminated Do tdictkeys := _privatedict.Keys.ToArray; End; ...where the amount of items are growing in the dictionary (by an other thread for example, imagining thread safety non-existent for this example), does it mean that the array will keep relocating (without releasing the previous memory area); without a detectable leak but effectively causing OOM? Correct me if I'm wrong but a Win32 binary is a Win32 binary. A Win32 compatible OS should be able to run it, no? It's a bunch of assembly instructions at the end, after all.

I'm paranoid. Even for IDs expected to be in the million range I use UInt64 🙂

This is why I use UNIX timestamps adjusted to GMT without DST (where applicable) as DateTime values in databases lately. Store them as numbers, read them as numbers, handle them as numbers.

Yep, I guess my question would have been better if I ask what exactly the scope is. I quickly discarded this option though as if it would be a memory leak, it would appear on all systems, not only on Win2k. I started a perfmon on the two machines to see how the processes and system memory is changing. Resource usage of my executable did not grow with a single byte in the past 24 hours. In fact, Working Set dramatically decreased from ~14 MB to 1,2 - guess some stuff was paged out. I really suspect this will be Windows- or ESX related. Not Delphi or my code.

One more thing which comes with generics is how I enumerate through them. And I always had issues understanding dynamic arrays. Let's say I have a method: Function TMyClass.Method: TArray<String>; Begin Result := _dictionary.Keys.ToArray; End; And then I do a: For s In MyClassInstance.Method Do It is allocating a TArray, which is not finalized. Does it mean that the memory will be released when the application closes, or when the caller method ends...? This basically applies to all dynamic arrays; and I think I never finalized them until now. Could this "leak"?

Hard drives as you might think - not necessarily. They often have a chip soldered on the board 🙂

Yes. Considering built-in devices (robotic arms, controllers, etc.) where you can not upgrade the hardware and/or the software, I like to experiment with the absolute minimum. At the moment my 1 GB Win2K is actively using ~104 MB, while the 128 MB Win2k is using only 11 MB of RAM:

Windows 2000 indeed has a configurable page file size. According to Microsoft's standards I'm always setting it to 1,5 times the physical memory size (adjusted every time the memory was expanded). And again - before generics, 64 MB RAM and 96 MB of pagefile was sufficient. After Generics Windows reaches OOM with 512 MB RAM and 768 MB of page file. Guess I'll have to wait for my secondary VM to produce (or not to produce) the symptoms. I'm thinking if this will be related to desktop heap (I'll try it once any VMs produce the same symptoms). Still - is there a way for a Delphi application to forcefully release unused heap memory somehow?

I do think the same way; but correct me if I'm wrong: if an application wants to allocate a specific size of memory and that space is not available, shouldn't the application throw the error...? Or again, I'm expecting too much from an ancient technology? I already have experience with memory, handle leaks and memory corruption, but never had to deal with fragmentation before. Is there a way to detect it from the application?

One more important detail. OOM is given by the OS; not by the application itself.

I did, this is why I linked a post, not the topic itself. I personally don't have this deep knowledge of how the language works but as noone opposed Pawel's statement I considered it true. I not just went back to ensure all blocks are surrounded by a Try ... Finally ... FreeAndNil ... End; FastMM and MadExcept both says there are no leaks at all. The average alive time of a work queue item is a couple of seconds, and as I'm using TObjectList.Create(True) as the queue, I can be sure that work items are disposed upon removal. Furthermore, as I mentioned the problem only appears on a Windows 2000 machine with (currently) 1 GB of memory (previous solution ran fine on 64 MB); a Windows 2003 R2 with 128 MB and a 2008 with 512 MB is running the tool fine for weeks now (since the latest patch). In the mean time, I set up a basic VM with 128 MB of RAM and no vmWare tools. We'll see how long it will last. P.s.: I don't want anyone to find the issue for me; I'm looking for directions on how to find these by myself. So any tips are highly welcome.

Nice! I used HxD before and I really liked it. I just always have VSCode open and it's easier with it's addon. It doesn't do as much as your program does, though. Good job, you just created an awesome tool in an awesome language!

Could not agree more. I spent a week, 12 hours a day to XMLDoc document a mid-sized project. The result was so ugly and unreadable I pushed it to a separate branch only. It is really good to see tooltip insights, or a PasDoc created documentation, but it creates a painful mess for the eye. I'm glad I'm not the only one seeing it this way.

For way too long this is haunting my coding experience. A simple Hello World is becoming thousands of lines due to encryption, obfuscation and filler NoOps. At the end of the day you are only making the life of the attacker harder. Never impossible. True and not true. If password is sent as a parameter and the application is stored in %APPDATA% for example, you can replace the secondary .EXE to just dump the password. Command line passwords are a really bad idea from this perspective.

Hmm, seems I messed up the declarations all along: TSSHConnection = Class(TConnection) strict private _puttyhandle: THandle; At the end of the day Delphi considers them the same, this is why my code works: WinApi.Windows.THandle = System.THandle = System.NativeUInt WinApi.Windows.HWND = WinApi.Windows.UINT_PTR = System.UIntPtr = System.NativeUInt I'll correct it anyway 🙂 Thanks for the heads-up!

Thank you for sharing the looks of before-after! I love to see the dramatic change from using colors to monochrome is causing... it indeed looks cleaner and more professional. I do have troubles with a good UI design and it's hard to come by with useful tips and tricks. Good luck with your application 🙂

Docking is easy. _puttyhandle is a THandle, value comes from FindWindow. _owner should be a TWinControl, where you want your external application to be docked. I docked Putty, but Notepad also works fine 🙂 procedure TSSHConnection.Dock; var WindowStyle : Integer; FAppThreadID: Cardinal; begin WindowStyle := GetWindowLong(_puttyhandle, GWL_STYLE); WindowStyle := WindowStyle - WS_CAPTION - WS_BORDER - WS_OVERLAPPED - WS_THICKFRAME; SetWindowLong(_puttyhandle,GWL_STYLE,WindowStyle); /// Attach container app input thread to the running app input thread, so that the running app receives user input. FAppThreadID := GetWindowThreadProcessId(_puttyhandle, nil); AttachThreadInput(GetCurrentThreadId, FAppThreadID, True); /// Changing parent of the running app to our provided container control Windows.SetParent(_puttyhandle, TWinControl(_owner).Handle); SendMessage(TWinControl(_owner).Handle, WM_UPDATEUISTATE, UIS_INITIALIZE, 0); UpdateWindow(_puttyhandle); /// This prevents the parent control to redraw on the area of its child windows (the running app) SetWindowLong(TWinControl(_owner).Handle, GWL_STYLE, GetWindowLong(TWinControl(_owner).Handle,GWL_STYLE) or WS_CLIPCHILDREN); /// Make the running app to fill all the client area of the container SetWindowPos(_puttyhandle,0,0,0,TWinControl(_owner).ClientWidth,TWinControl(_owner).ClientHeight,SWP_NOZORDER); // SetForegroundWindow(WindowHandle); // SetFocus(WindowHandle); end; Importing MSTSCAX as a VCL component is tricky, but I found a solution a couple of years ago which still works. I don't have the link anymore, so I don't know who to credit for it 😞 - Component -> Import Component -> Import ActiveX Control -> Select Terminal Services (mstscax.dll) - Palette page: MsTSC, unit dir name: Delphi Lib folder, Search path: empty. Check Generate component wrappers - Install to New package - Enter the same name (MsTSCLib_TLB.dproj) under Delphi Lib folder - Try to build package just to fail - Change Property type of ConnectWithEndpoint to "OleVariant", build package again - Right-click on MsTSCLib_TLB.bpl in Project manager and select "Install" As for the modified PageControl, you can check it out at https://www.aecentral.org/index.php?post/35. Idk why I used a global variable as the OnClose event... feel free to change it, you can even add a Register procedure to have a VCL component.