aehimself

Sorry, true that; it's physical cores not logical ones. Please disregard my previous comment.

How exactly are you spawning the threads? Afaik, TThread.Create will always use the same logical CPU core as your main application. To make Windows to spread workers on multiple cores you need to use BeginThread... at least this is how it worked back in D7.

I wrote a beautifier - minimizer for XML; HTML should work with minor adjustments. If you wish I can share the source.

11 hours ago, Fr0sT.Brutal said:

First, nothing in TDateTime could be "local". It's just the same shift from some constant point as any other timestamp. It's up to client app to generate this shift either in local timezone or in UTC (which is what I always recommend, just like UTF8 in string encoding).

Second, there are TIMESTAMP_WITH_TIMEZONE type in most of DB engines. It requires some additional handling though and not mapped as-is to TDateTime for obvious reasons.

I completely agree with you on this one. this is just how I started, and later continued to think. We still have to agree that in a Client-Server architecture, where you have to create the filter on the Client (and have no access to dataset parameters) working with a number is easier. I already had enough of TO_DATE and CONVERT for a lifetime.

11 hours ago, Fr0sT.Brutal said:

Just like most of web-sites behave today. "We determined your city as %CityName, is that correct?" and leave ability to set whatever other city user wants

Well, yes - that is my idea. Convince, but still controllable. And as an end user I like to have control

20 minutes ago, Fr0sT.Brutal said:

*scratching my head* any database internally uses its own timestamp format that converts without problems to Delphi TDateTime by DB libs. I see no sense in replacing this mechanism by custom one that only your apps know how to deal with, so f.ex. examining table data with an admin tool becomes a real pain.

True, from this perspective it's useless. My problem with TDateTime values is that they are local; once you decide to go international you'll have to change the codebase. I mean for example a record was created at 21:00:00 in China. If someone from the US checks it, it will only tell that the record was created at 21:00:00 but that will be incorrect as 21:00:00 in China is 03:00:00 in the US. As I'm thinking in UNIX timestamps only lately I have a library to take care of these conversions for me in all of my applications.

As for the database storage, the only important factor is that your app should know how to handle this data; but most DB engines have UNIX encoding functions (e.g. FROM_UNIXTIME in MySQL). I also wrote my own database explorer with these built-in:

But, I don't want to hijack the topic. @Ian Branch If you can not rely on the OS, the easiest solution is if you add a drop-down menu in your settings to choose the appropriate location. If you want an automatic solution, my idea would be to use an IP to location API to detect the country the PC is connecting to the Internet from. This method won't work if they are using VPN, though; so maybe you can just add "Automatic" in the drop-down. Normally they are being detected, but they can override it in case.

Just now, Vandrovnik said:

New versions of compiler may use new functions in Windows API, that are not present in Windows 2000. So application will start and then complain about missing something (or start and immediatelly die silently).

Spot on, did not think about this. It would be pretty easy to diagnose, though.

I was running a close resource monitoring of the named application, having no unexplainable jumps whatsoever. It's only been a week so cannot be sure, but I strongly believe that whatever caused the OOM was / is out of my control. I set the affected W2K machines memory back to 128MB to try to reproduce the issue in the mean time.

Theoretical question. Is seems the scope of local variables is the end of the method. So if I have

Procedure TMyThread.Execute;
Var
 tdictkeys: TArray<String>;
Begin
 While Not Terminated Do
  tdictkeys := _privatedict.Keys.ToArray;
End;

...where the amount of items are growing in the dictionary (by an other thread for example, imagining thread safety non-existent for this example), does it mean that the array will keep relocating (without releasing the previous memory area); without a detectable leak but effectively causing OOM?

On 3/8/2020 at 11:58 PM, Anders Melander said:

FWIW, Windows 2000 isn't supported by the version of Delphi you're using.

Correct me if I'm wrong but a Win32 binary is a Win32 binary. A Win32 compatible OS should be able to run it, no? It's a bunch of assembly instructions at the end, after all.

58 minutes ago, Vandrovnik said:

And what data type do you use? https://en.wikipedia.org/wiki/Year_2038_problem

I'm paranoid. Even for IDs expected to be in the million range I use UInt64 🙂

This is why I use UNIX timestamps adjusted to GMT without DST (where applicable) as DateTime values in databases lately. Store them as numbers, read them as numbers, handle them as numbers.

4 hours ago, Fr0sT.Brutal said:

Consider them behave like interfaces. A variable keeps reference counter which is increased on assignment and decreased when exiting from scope. There are only a few cases when you need to finalize string/array manually. One of them is to free a valuable amount of memory occupied by a variable ASAP without waiting for subroutine end.

Yep, I guess my question would have been better if I ask what exactly the scope is. I quickly discarded this option though as if it would be a memory leak, it would appear on all systems, not only on Win2k.

I started a perfmon on the two machines to see how the processes and system memory is changing. Resource usage of my executable did not grow with a single byte in the past 24 hours. In fact, Working Set dramatically decreased from ~14 MB to 1,2 - guess some stuff was paged out.

I really suspect this will be Windows- or ESX related. Not Delphi or my code.

One more thing which comes with generics is how I enumerate through them. And I always had issues understanding dynamic arrays.

Let's say I have a method:

Function TMyClass.Method: TArray<String>;
Begin
 Result := _dictionary.Keys.ToArray;
End;

And then I do a:

For s In MyClassInstance.Method Do

It is allocating a TArray, which is not finalized. Does it mean that the memory will be released when the application closes, or when the caller method ends...?

This basically applies to all dynamic arrays; and I think I never finalized them until now.

Could this "leak"?

Hard drives as you might think - not necessarily. They often have a chip soldered on the board 🙂

Yes. Considering built-in devices (robotic arms, controllers, etc.) where you can not upgrade the hardware and/or the software, I like to experiment with the absolute minimum.

At the moment my 1 GB Win2K is actively using ~104 MB, while the 128 MB Win2k is using only 11 MB of RAM:

Windows 2000 indeed has a configurable page file size. According to Microsoft's standards I'm always setting it to 1,5 times the physical memory size (adjusted every time the memory was expanded).

And again - before generics, 64 MB RAM and 96 MB of pagefile was sufficient. After Generics Windows reaches OOM with 512 MB RAM and 768 MB of page file.

Guess I'll have to wait for my secondary VM to produce (or not to produce) the symptoms. I'm thinking if this will be related to desktop heap (I'll try it once any VMs produce the same symptoms).

Still - is there a way for a Delphi application to forcefully release unused heap memory somehow?

Just now, David Heffernan said:

Smells like a memory leak, or perhaps the loss of contiguous address space due to fragmentation from reallocation. 

I do think the same way; but correct me if I'm wrong: if an application wants to allocate a specific size of memory and that space is not available, shouldn't the application throw the error...? Or again, I'm expecting too much from an ancient technology?

I already have experience with memory, handle leaks and memory corruption, but never had to deal with fragmentation before. Is there a way to detect it from the application?

One more important detail. OOM is given by the OS; not by the application itself.

2 hours ago, David Heffernan said:

Didn't you read that topic? The entire topic was based on misconception. 

 

I did, this is why I linked a post, not the topic itself. I personally don't have this deep knowledge of how the language works but as noone opposed Pawel's statement I considered it true.

2 hours ago, David Heffernan said:

It's very plausible that you have a memory leak. Just because fastmm4 says you don't doesn't mean you don't. It's one thing returning memory when the program closes, another returning it in a timely fashion during execution. 

I not just went back to ensure all blocks are surrounded by a Try ... Finally ... FreeAndNil ... End; FastMM and MadExcept both says there are no leaks at all. The average alive time of a work queue item is a couple of seconds, and as I'm using TObjectList.Create(True) as the queue, I can be sure that work items are disposed upon removal.

Furthermore, as I mentioned the problem only appears on a Windows 2000 machine with (currently) 1 GB of memory (previous solution ran fine on 64 MB); a Windows 2003 R2 with 128 MB and a 2008 with 512 MB is running the tool fine for weeks now (since the latest patch).

In the mean time, I set up a basic VM with 128 MB of RAM and no vmWare tools. We'll see how long it will last.

P.s.: I don't want anyone to find the issue for me; I'm looking for directions on how to find these by myself. So any tips are highly welcome.

Nice!

I used HxD before and I really liked it. I just always have VSCode open and it's easier with it's addon. It doesn't do as much as your program does, though.

Good job, you just created an awesome tool in an awesome language!

Hello all,

After switching from D7 to D10.2 I re-wrote one of my multithreaded apps with proper OOP, using the "new" language features. Each thread has a work queue, which used to be a Array Of String, storing the properties separated with tabs. Now, the work queue is a TObjectList, and work queue items are classes, created upon adding, freed after successful processing. Everything works like a charm, there are no memory leaks reported whatsoever even after a stress-test of the work item part.

After about a week of uptime my Windows 2000 test system reaches 0% of free memory and becomes unstable, requiring a reboot. Unfortunately there are a lot of factors to consider here:

- The ESX host the test VM is running on was patched a couple of times (and because of that vmWare tools got updated on the guests)

- The obvious, Array to TList and String to Class conversion

- Only a Windows 2000 system is affected, Windows 2003 and above accepted the change well

I will attempt to create a new Windows 2000 machine without vmWare tools to see if it makes any difference, however I believe that TList will be the root cause here; which got reinforced after seeing

Did anyone met this symptom before? How can I be sure the root cause is my application and not the prehistoric OS? Maybe the combination of both? Is there a (even a hacky) way to force a Delphi application to release any currently not used memory?

Thanks all!

On 3/4/2020 at 1:39 PM, A.M. Hoornweg said:

I find formats like Xmldoc totally obnoxious because it interrupts the source code and the text is never in the place where I need to consult it.

Could not agree more. I spent a week, 12 hours a day to XMLDoc document a mid-sized project. The result was so ugly and unreadable I pushed it to a separate branch only. It is really good to see tooltip insights, or a PasDoc created documentation, but it creates a painful mess for the eye.

I'm glad I'm not the only one seeing it this way.

On 2/27/2020 at 8:53 AM, Fr0sT.Brutal said:

Anyway if a user has admin rights in your system, he is able to dump process memory or even debug it.

For way too long this is haunting my coding experience. A simple Hello World is becoming thousands of lines due to encryption, obfuscation and filler NoOps. At the end of the day you are only making the life of the attacker harder. Never impossible.

On 2/27/2020 at 8:53 AM, Fr0sT.Brutal said:

If he hasn't, encrypt your data and transfer it in any manner.

True and not true. If password is sent as a parameter and the application is stored in %APPDATA% for example, you can replace the secondary .EXE to just dump the password. Command line passwords are a really bad idea from this perspective.

16 hours ago, David Heffernan said:

Nope. It's an HWND. A THandle is quite different. 

 

Also, https://devblogs.microsoft.com/oldnewthing/20130412-00/?p=4683

Hmm, seems I messed up the declarations all along:

 TSSHConnection = Class(TConnection)
 strict private
  _puttyhandle: THandle;

At the end of the day Delphi considers them the same, this is why my code works:

WinApi.Windows.THandle = System.THandle = System.NativeUInt

WinApi.Windows.HWND = WinApi.Windows.UINT_PTR = System.UIntPtr = System.NativeUInt

I'll correct it anyway 🙂 Thanks for the heads-up!

Thank you for sharing the looks of before-after! I love to see the dramatic change from using colors to monochrome is causing... it indeed looks cleaner and more professional. I do have troubles with a good UI design and it's hard to come by with useful tips and tricks.

Good luck with your application 🙂

Docking is easy. _puttyhandle is a THandle, value comes from FindWindow. _owner should be a TWinControl, where you want your external application to be docked. I docked Putty, but Notepad also works fine 🙂

procedure TSSHConnection.Dock;
var
 WindowStyle : Integer;
 FAppThreadID: Cardinal;
begin
 WindowStyle := GetWindowLong(_puttyhandle, GWL_STYLE);
 WindowStyle := WindowStyle - WS_CAPTION - WS_BORDER - WS_OVERLAPPED - WS_THICKFRAME;
 SetWindowLong(_puttyhandle,GWL_STYLE,WindowStyle);

 /// Attach container app input thread to the running app input thread, so that the running app receives user input.
 FAppThreadID := GetWindowThreadProcessId(_puttyhandle, nil);
 AttachThreadInput(GetCurrentThreadId, FAppThreadID, True);

 /// Changing parent of the running app to our provided container control
 Windows.SetParent(_puttyhandle, TWinControl(_owner).Handle);
 SendMessage(TWinControl(_owner).Handle, WM_UPDATEUISTATE, UIS_INITIALIZE, 0);
 UpdateWindow(_puttyhandle);

 /// This prevents the parent control to redraw on the area of its child windows (the running app)
 SetWindowLong(TWinControl(_owner).Handle, GWL_STYLE, GetWindowLong(TWinControl(_owner).Handle,GWL_STYLE) or WS_CLIPCHILDREN);

 /// Make the running app to fill all the client area of the container

 SetWindowPos(_puttyhandle,0,0,0,TWinControl(_owner).ClientWidth,TWinControl(_owner).ClientHeight,SWP_NOZORDER);
// SetForegroundWindow(WindowHandle);
// SetFocus(WindowHandle);
end;

Importing MSTSCAX as a VCL component is tricky, but I found a solution a couple of years ago which still works. I don't have the link anymore, so I don't know who to credit for it 😞

- Component -> Import Component -> Import ActiveX Control -> Select Terminal Services (mstscax.dll)
- Palette page: MsTSC, unit dir name: Delphi Lib folder, Search path: empty. Check Generate component wrappers
- Install to New package
- Enter the same name (MsTSCLib_TLB.dproj) under Delphi Lib folder
- Try to build package just to fail
- Change Property type of ConnectWithEndpoint to "OleVariant", build package again
- Right-click on MsTSCLib_TLB.bpl in Project manager and select "Install"

As for the modified PageControl, you can check it out at https://www.aecentral.org/index.php?post/35. Idk why I used a global variable as the OnClose event... feel free to change it, you can even add a Register procedure to have a VCL component.

As time progressed, more and more new possible solutions were introduced, but since it's 20+ year old legacy code which was designed to have their filters this way... changing the frameworks filtering would require us to check and modify ~1000 frames on the Windows client and I guess it would break the web app too. There's close to no chance on changing this anymore 😞

I'm not a DB expert, I personally never used temporary tables; but I have a slight memory having issues / limitations with temporary tables on some engines. I could be wrong on this one, though.