aehimself

Nice! I also played around with Arduino boards (this is how I fixed my smoking Christmas tree lights on Christmas eve) and I love them a lot. I even minified a project to a single ATMega chip for extra efficiency.

But there was one thing I was always wondered but never took the time to dig myself into... can I talk to this board with Delphi...?

I don't want you to share mission critical parts, but can you give some hints on how you did it? Are you using standard Serial to USB and text commands? Or a custom interface?

While we were working from home the office building decided that it's a good time to test the electric breakers. This resulted someone having to travel on-site and turning all PCs on one-by-one on the whole floor.

I started to experiment; my plan is to write a simple service application which is discovering PCs in a pre-defined range and provides a web interface where you can log on with your domain credentials and choose which PC you would like to wake up with WOL. If I get the green light from management I think this is going to be our solution.

I'll have to test though, if a WOL broadcast packet actually works out-of-subnet or not; or what configuration is required on the network devices as I have no personal experience with this.

7 minutes ago, Alexander Sviridenkov said:

This is only one of hundreds problems. Parsing HTML is much harder than XML.

Thank god we use XMLs in this case 🙂

1 hour ago, limelect said:

I never said just one there can be many explorer.

But my software chooses the first in list.

And that is an issue, if you ask me. To be honest I lost track a long time ago, but for a while the Windows Taskbar and the Desktop was hosted by an instance of the same explorer.exe process what you launched when you clicked on "My computer"; maybe Windows is using it for even more undocumented things (like specific background processes; which would answer why they just come and go). If your application requires to interact with an Explorer window, you should let your users to choose which one. But then again - this is how I think.

4 hours ago, limelect said:

But a hidden process does not have any name just a number.

Wrong. A process has multiple granted properties; like an executable NAME and a PID for example. All processes have that, having visible windows or not, being service applications or not, being launched under a svchost process or not. Depending on how you enumerate those, you get different properties available. I personally never used ShellWindows as I always went for EnumWindows or FindWindow what @Fr0sT.Brutal advised to you already. It's a bit more low level, but opens up tons of opportunities.

2 hours ago, limelect said:

Maybe you should download my software and see what it dose.

There are a lot of people using it and it is quite useful for us.

Just unzip and use. No installation. you can always delete.

And this part; I must totally have to agree with @David Heffernan. If someone anonymous on the Internet would tell you "Download my app, it's useful, you always can delete"... would you trust him/her?

At this point noone really understands why you are doing what you are doing so they are unable to give you alternatives. Without an understandable code snipplet (e.g. I have no damn clue what "Do not forget Dispatch := ShellWindows.Item(i); is nil" means and I think I'm not alone) or a clear goal set... it just sounds like an other million dollar idea what SW developers are usually getting.

10 hours ago, Alexander Sviridenkov said:

@aehimself 95% of real world HTMLs cannot be parsed by XML parser. Also you cannot correctly modify HTML without calculating CSS styles. F.e. in this case <p class="p1">This is text</p>   <p class="p1">This is text2</p>  spaces between p can be removed, but in other case <p class="p2">This is text</p>   <p class="p2">This is text2</p> they should be preserved.

I quickly realized. It threw a nullpointer exception anyway because of embedded JavaScript. And XML does not have that 🙂

9 hours ago, Fr0sT.Brutal said:

Btw, what you need XML minification for? As far as I'm aware you can rarely see plain XML in the wild; most of formats suppose Zipped XML (docx, fb2.zip). DB engines often use compression internally as well, HTTP protocol allows deflate compression.

Our overgrown legacy system is talking to an other, overgrown legacy system through a custom protocol, transporting the XMLs in a beautified form. Now the issue is, that sometimes these two systems start to get misaligned because either of a 3rd overgrown legacy system, a bug in their software or a bug in ours. So after fixing a bug these information packets should be re-processed or re-sent. And as the protocol was designed you always need a reference to an earlier packet as well... quite complicated, resulting us having to save all communication between our software instances and their system. Originally, someone decided to keep the XMLs in a BLOB field in a table. Unfortunately the code is so big and disorganized (20+ years old) that changing the saving scheme would need a re-writing half our application plus the web interface. We know it should be done but we also know that it never will be done as the clients do not pay for refactoring.

The good thing in legacy systems are the gems what you can find. My favorite goes like this:

Function GetUserID: Integer;
Begin
 Case MenuID Of
  0: Result := -1;
  1: Result := -1;
  2: Result := -1;
  [...]
  100: Result := -1;
  Else Result := -1;
 End;
End;

... and oooooh, those sweet comments 🙂 Too bad that most of them are in Hungarian, otherwise I'd have posted them on DevHumor.

12 hours ago, Fr0sT.Brutal said:

Word has never been a good HTML generator )) CSS styles in all pages it created were a nightmare.

I know, just wanted bring a really negative example. Most of the CMS WYSIWYG editors are no exception though and since they love to use shared libraries I suppose webmail interfaces are guilty as well.

12 hours ago, Fr0sT.Brutal said:

Well, I tried the first online minifier Google has given me and the source of this very page. Initial 76217, minified 63707, "compression" 16%. On the one hand, that's more than nothing. On the other hand, that makes the page source unreadable, could cause errors (f.ex. in embedded scripts or styles), and finally saves only 13 kB.

As I mentioned I mainly work with XML, which is similar (but not the same) as HTML; and in XML the results are surprising already. I’ll go ahead and call minifying a compression method (a fairly inefficient one, though) from now on.

As with most compressions the output heavily depends on the input; but usually the larger the input, the better the ratio becomes. The purpose of a compression is not to make it readable, and if it causes errors it’s a bug in the code.

All I’m saying is whether if it’s efficient or not - minifying is still viable as this is the only way the output is still accepted by the target software.

Just out of curiosity I’ll run my XML minifyer on a HTML page and will post the results.

On 4/3/2020 at 6:27 PM, Fr0sT.Brutal said:

Will it worth the efforts?

Oh, you would be surprised. The last WYSIWYG program that wrote clean HTML code was Front Page Express around 2000. My favorite was creating a simple page, saving it in Word 2003 as HTML and then manually trimming the waste out. I usually could save aywhere between 5 or 20 kilobytes, keeping the original design.

Now imagine trimming the "useless" characters too, used only to make the code readable.

We deal with XMLs mostly at work. As an example, a beautified XML is 687722 characters, the same minimized is 455528; that's about 33% "compression".

So after weeks of attempting to find the guilty code piece I found nothing. Every allocation had a release pair, memory usage was perfectly consistent for close to a month. This combined with the fact that only one machine was affected means that whatever happened, it was not caused by my code (and therefore generics and classes) and was only a coincidence.

And I think I already know what happened.

So thanks all for the tips; at least we know that (even if it's unsupported) D10.3 applications are running happily (and correctly) on Windows 2000 too 🙂

Sorry, true that; it's physical cores not logical ones. Please disregard my previous comment.

How exactly are you spawning the threads? Afaik, TThread.Create will always use the same logical CPU core as your main application. To make Windows to spread workers on multiple cores you need to use BeginThread... at least this is how it worked back in D7.

I wrote a beautifier - minimizer for XML; HTML should work with minor adjustments. If you wish I can share the source.

11 hours ago, Fr0sT.Brutal said:

First, nothing in TDateTime could be "local". It's just the same shift from some constant point as any other timestamp. It's up to client app to generate this shift either in local timezone or in UTC (which is what I always recommend, just like UTF8 in string encoding).

Second, there are TIMESTAMP_WITH_TIMEZONE type in most of DB engines. It requires some additional handling though and not mapped as-is to TDateTime for obvious reasons.

I completely agree with you on this one. this is just how I started, and later continued to think. We still have to agree that in a Client-Server architecture, where you have to create the filter on the Client (and have no access to dataset parameters) working with a number is easier. I already had enough of TO_DATE and CONVERT for a lifetime.

11 hours ago, Fr0sT.Brutal said:

Just like most of web-sites behave today. "We determined your city as %CityName, is that correct?" and leave ability to set whatever other city user wants

Well, yes - that is my idea. Convince, but still controllable. And as an end user I like to have control

20 minutes ago, Fr0sT.Brutal said:

*scratching my head* any database internally uses its own timestamp format that converts without problems to Delphi TDateTime by DB libs. I see no sense in replacing this mechanism by custom one that only your apps know how to deal with, so f.ex. examining table data with an admin tool becomes a real pain.

True, from this perspective it's useless. My problem with TDateTime values is that they are local; once you decide to go international you'll have to change the codebase. I mean for example a record was created at 21:00:00 in China. If someone from the US checks it, it will only tell that the record was created at 21:00:00 but that will be incorrect as 21:00:00 in China is 03:00:00 in the US. As I'm thinking in UNIX timestamps only lately I have a library to take care of these conversions for me in all of my applications.

As for the database storage, the only important factor is that your app should know how to handle this data; but most DB engines have UNIX encoding functions (e.g. FROM_UNIXTIME in MySQL). I also wrote my own database explorer with these built-in:

But, I don't want to hijack the topic. @Ian Branch If you can not rely on the OS, the easiest solution is if you add a drop-down menu in your settings to choose the appropriate location. If you want an automatic solution, my idea would be to use an IP to location API to detect the country the PC is connecting to the Internet from. This method won't work if they are using VPN, though; so maybe you can just add "Automatic" in the drop-down. Normally they are being detected, but they can override it in case.

Just now, Vandrovnik said:

New versions of compiler may use new functions in Windows API, that are not present in Windows 2000. So application will start and then complain about missing something (or start and immediatelly die silently).

Spot on, did not think about this. It would be pretty easy to diagnose, though.

I was running a close resource monitoring of the named application, having no unexplainable jumps whatsoever. It's only been a week so cannot be sure, but I strongly believe that whatever caused the OOM was / is out of my control. I set the affected W2K machines memory back to 128MB to try to reproduce the issue in the mean time.

Theoretical question. Is seems the scope of local variables is the end of the method. So if I have

Procedure TMyThread.Execute;
Var
 tdictkeys: TArray<String>;
Begin
 While Not Terminated Do
  tdictkeys := _privatedict.Keys.ToArray;
End;

...where the amount of items are growing in the dictionary (by an other thread for example, imagining thread safety non-existent for this example), does it mean that the array will keep relocating (without releasing the previous memory area); without a detectable leak but effectively causing OOM?

On 3/8/2020 at 11:58 PM, Anders Melander said:

FWIW, Windows 2000 isn't supported by the version of Delphi you're using.

Correct me if I'm wrong but a Win32 binary is a Win32 binary. A Win32 compatible OS should be able to run it, no? It's a bunch of assembly instructions at the end, after all.

58 minutes ago, Vandrovnik said:

And what data type do you use? https://en.wikipedia.org/wiki/Year_2038_problem

I'm paranoid. Even for IDs expected to be in the million range I use UInt64 🙂

This is why I use UNIX timestamps adjusted to GMT without DST (where applicable) as DateTime values in databases lately. Store them as numbers, read them as numbers, handle them as numbers.

4 hours ago, Fr0sT.Brutal said:

Consider them behave like interfaces. A variable keeps reference counter which is increased on assignment and decreased when exiting from scope. There are only a few cases when you need to finalize string/array manually. One of them is to free a valuable amount of memory occupied by a variable ASAP without waiting for subroutine end.

Yep, I guess my question would have been better if I ask what exactly the scope is. I quickly discarded this option though as if it would be a memory leak, it would appear on all systems, not only on Win2k.

I started a perfmon on the two machines to see how the processes and system memory is changing. Resource usage of my executable did not grow with a single byte in the past 24 hours. In fact, Working Set dramatically decreased from ~14 MB to 1,2 - guess some stuff was paged out.

I really suspect this will be Windows- or ESX related. Not Delphi or my code.

One more thing which comes with generics is how I enumerate through them. And I always had issues understanding dynamic arrays.

Let's say I have a method:

Function TMyClass.Method: TArray<String>;
Begin
 Result := _dictionary.Keys.ToArray;
End;

And then I do a:

For s In MyClassInstance.Method Do

It is allocating a TArray, which is not finalized. Does it mean that the memory will be released when the application closes, or when the caller method ends...?

This basically applies to all dynamic arrays; and I think I never finalized them until now.

Could this "leak"?

Hard drives as you might think - not necessarily. They often have a chip soldered on the board 🙂

Yes. Considering built-in devices (robotic arms, controllers, etc.) where you can not upgrade the hardware and/or the software, I like to experiment with the absolute minimum.

At the moment my 1 GB Win2K is actively using ~104 MB, while the 128 MB Win2k is using only 11 MB of RAM:

Windows 2000 indeed has a configurable page file size. According to Microsoft's standards I'm always setting it to 1,5 times the physical memory size (adjusted every time the memory was expanded).

And again - before generics, 64 MB RAM and 96 MB of pagefile was sufficient. After Generics Windows reaches OOM with 512 MB RAM and 768 MB of page file.

Guess I'll have to wait for my secondary VM to produce (or not to produce) the symptoms. I'm thinking if this will be related to desktop heap (I'll try it once any VMs produce the same symptoms).

Still - is there a way for a Delphi application to forcefully release unused heap memory somehow?

Just now, David Heffernan said:

Smells like a memory leak, or perhaps the loss of contiguous address space due to fragmentation from reallocation. 

I do think the same way; but correct me if I'm wrong: if an application wants to allocate a specific size of memory and that space is not available, shouldn't the application throw the error...? Or again, I'm expecting too much from an ancient technology?

I already have experience with memory, handle leaks and memory corruption, but never had to deal with fragmentation before. Is there a way to detect it from the application?

One more important detail. OOM is given by the OS; not by the application itself.

2 hours ago, David Heffernan said:

Didn't you read that topic? The entire topic was based on misconception. 

 

I did, this is why I linked a post, not the topic itself. I personally don't have this deep knowledge of how the language works but as noone opposed Pawel's statement I considered it true.

2 hours ago, David Heffernan said:

It's very plausible that you have a memory leak. Just because fastmm4 says you don't doesn't mean you don't. It's one thing returning memory when the program closes, another returning it in a timely fashion during execution. 

I not just went back to ensure all blocks are surrounded by a Try ... Finally ... FreeAndNil ... End; FastMM and MadExcept both says there are no leaks at all. The average alive time of a work queue item is a couple of seconds, and as I'm using TObjectList.Create(True) as the queue, I can be sure that work items are disposed upon removal.

Furthermore, as I mentioned the problem only appears on a Windows 2000 machine with (currently) 1 GB of memory (previous solution ran fine on 64 MB); a Windows 2003 R2 with 128 MB and a 2008 with 512 MB is running the tool fine for weeks now (since the latest patch).

In the mean time, I set up a basic VM with 128 MB of RAM and no vmWare tools. We'll see how long it will last.

P.s.: I don't want anyone to find the issue for me; I'm looking for directions on how to find these by myself. So any tips are highly welcome.