TDirectory - file lock out on Win 10 LTSC

[Roger Cigol 114]

VCL - RAD Studio 12.2 C++ Classic Compiler

 

I have a windows C++ (Classic compiler) VCL application that uses TDirectory::GetDirectories() and TDirectory::GetFiles() to get lists of files from the Windows OS file structure.

These files are then opened for read or write using stl ofstream and ifstream along with >> and << operators.

The same code has been in use on different instances of the application on different machines (including Win XP and Windows 7).

I have a new instance where the operating system is Windows 10 LTSC version 10.0.19044 build 19044. I have not run it on this version of windows before.

 

Problem; Directory and file lists, reading and writing work ok for a longish time (24 hours sometimes, 30 hours sometimes, 10 hours sometimes) and then

the calls to TDirectory::GetDirectories() and TDirectory::GetFiles() both return empty lists. Writing to a log file (with a fixed file name) also stops working at

this time. It's as if my application has been locked out of the file system somehow.

 

At this point it is possible to use File Explorer to see that there still are files and directories present. It is possible to open the files by other applications.

(some are plain text files and can be opened with notepad). The files are stored in a directory inside C:\Users\Public\Documents\ so

should be available to all applications all of the time.

 

I cannot offer a reproducible example - all my code works fine on my development machine (which is Windows 10 Pro build 19045.5371 22H2).

This is true if I run it under the IDE debugger or as a free standing exe.

 

I wonder if any of my fellow Delphi-Praxis colleagues might have any similar experiences or can offer some ideas of what I should try next ?

 

[Der schöne Günther 323]

10 minutes ago, Roger Cigol said:

Writing to a log file also stops working at this time. 

It's as if my application has been locked out of the file system somehow.

To me, it sounds more like your application has run out of resources. Not necessarily memory, but probably too many handles, like still opened files.

 

Check Windows Event Viewer for resource exhaustion events and other errors and warnings.

[Angus Robertson 608]

I wrote an open source component that indexes Windows directories and files recursively, written 20 years ago and used on every version of Windows and Delphi since, it just works, never fails. 

 

It currently builds a list of 1.2 million files on my c drive which it compares with the same number on a second drive, for backup on Windows 11.

 

Angus

 

[Roger Cigol 114]

29 minutes ago, Der schöne Günther said:

Check Windows Event Viewer for resource exhaustion events and other errors and warnings

@Der schöne GüntherMany thanks for this suggestion. Sadly (for me) I have already checked windows events - no sign of any problem error or warning messages.

[Kas Ob. 126]

Well, i don't have a idea about such case per se, but i witnessed many of these when i broke my OS kernel debugging and fooling around, so i have thoughts here

 

But first let me say what is different in LSTC from the normal, they are the same, except LTSC comes with slightly different default policies, policies that are not even listed in GP editor, some of them need to be added using ADMX files to be accessible, to have an idea look here these 

https://learn.microsoft.com/en-us/windows/client-management/mdm/policy-csp-storage

They mostly are documented though, yet many need to be added, so your OS might need some restoring its default.

 

That being said, now to what i think might be the cause, also while we shooting darts in the dark:

1) Storage handling layer and its drivers run in two layers User mode part and Kernel mode part, in some cases i caused a corruption in user mode, this triggered an exception, yet that exception wasn't critical to crash the system as it happened after returning form the kernel system calls (drivers), the exception marked the file handle as corrupted and left it in locked mode, or just the handle tables were faulty and the system couldn't add another handle or even try to read part of that table, sometimes many files locked, so no more file access with strange errors or simply freeze, yet the file(s) were accessible from from different processes, this due the user process sandboxing which start in kernel and extend to user mode.

2) LTSC does have delayed update policy, meaning fewer fixes will be pushed, so if there is a bug it could be simply fixed by update your LTSC to the latest and in case you can't then really running SFC 🙂 (as they always suggest) can help, the more info about running SFC is literally everywhere !

3) Your disk have a problem, i saw these on Server 2003, also have them on my old XP, the disk wasn't aligned, yes it is a thing and there is few tools to check disk aligning and fix it, 

https://superuser.com/questions/132296/how-to-check-the-partition-alignment-on-an-ssd-drive

Notice that searching the net now gives me only SSD result, but that is not the only case, it might happen with any disk type as it with mine, and also it increase the speed, and linger for trouble in accessing disks, which might be your case, 

One thing though don't use any non official application or method from any where on the net to align your disk, first check your disk manufacturer if they have such a tool, WD, Intel, Samsung ... they wither have tools or their software will check and prompt you to fix the alignment.

And with 3rdpary tools like the one mentioned here

https://www.diskpart.com/windows-10/ssd-alignment-windows-10-3889.html

It could be fine yet i wouldn't recommended it.

 

Hope that helps !

[Der schöne Günther 323]

Can't you just spin up a fresh VM with the same windows image and see if the problem persists?

[Remy Lebeau 1501]

4 hours ago, Roger Cigol said:

the calls to TDirectory::GetDirectories() and TDirectory::GetFiles() both return empty lists.

I don't trust the IOUtils unit. I feel it's functionality is bloated and error reporting is lacking/wonky. I prefer to stick with SysUtils::Find[First|Next]() or even ::Find[First|Next]File() directly. Have you tried that yet? Are there any error codes being reported when the problem happens?

 

Similarly with std::[i|o]fstream, they lack good error reporting, too. Do you get an error code from SysUtils::File[Create|Open]() or ::CreateFile()? 

[Roger Cigol 114]

On 1/29/2025 at 12:36 PM, Der schöne Günther said:

Can't you just spin up a fresh VM with the same windows image and see if the problem persists?

Thank you @Der schöne Günther for this suggestion. It's not quite so easy...

a) I am not sure if I have the IT skills to get an exact copy of Win 10 LTSC from the customer's machine on to a VM (I did not supply the customer's PC).

b) The software talks to significant complex hardware (it's a special purpose machine control system) which is hard to replicate / simulate on a VM system.

 

[Roger Cigol 114]

On 1/29/2025 at 3:58 PM, Remy Lebeau said:

Do you get an error code from SysUtils::File[Create|Open]() or ::CreateFile()?

Doing more error checking is of course a good idea. Perhaps I shouldn't have had to be pushed to do this. 

The software does log some errors to the log file. But of course if the log file falls over then I don't get a record of the errors.

There are practical difficulties working with the customer on site (if you do too many version changes they start to feel (with some justification) that you are using them as the debugging team...

However I agree with your helpful hint : I need to see if I can find out what is being reported when this problem occurs. I have created a version that will push error messages into the GUI thread and this thread will display them. 

I have asked the customer to take screen shots of any error boxes that pop up. 

Thanks for your help @Remy Lebeau I will post here again if I solve the problem or if I get more error information that I need help understanding !

[Roger Cigol 114]

An update: by adding code to get the Windows error code following a failed attempt to open a text file I have now got an error number : 1450

 

A quick google search reveals 

 

Error 1450 = Insufficient system resources exist to complete the requested service

 

So what resources is it short of, I wonder? I am currently trying to get a remote connection to the PC in question before they close down (and restart) the application....

[Roger Cigol 114]

When I open up task manager I see 0% for CPU, disk and network, very low power useage and only 14.7MB of memory useage - and it's been running for a week and a half, so doesn't look like a memory leak.

 

[Kas Ob. 126]

Well, this clear few things

14 minutes ago, Roger Cigol said:

Error 1450 = Insufficient system resources exist to complete the requested service

Service ! , so the cause is not your application, i am of course assuming your application is not a service, as you didn't mention that and that is very relevant.

 

My logic thinking about this, is that a service very relevant to your application and in this case very relevant to reading/accessing files failed to resources or other causes, here keep in mind many service error are mistakenly reported as resources deficiency due how it is structured to communicate by direct IPC or other method, these belongs to OS IPC designs.

 

So an OS service caused this, how this can be ? 

OS Services interact and interfere in [a/any] application in directly or indirectly 

1) Directly, as example, DNS Cache, TWAIN, or some .NET freak service,.... if your application is depending on something like OS DNS resolving and the service is misconfigured (something broken hosts file) it could lead all sort of such unexplained errors, yes i know i am bringing dns example to file but the idea i want to convey.

2) Indirectly, and this is the most relevant and may be the cause of your problem, see, there is services within OS are built to hook and intercept IO, example Volume Shadow Copy, System Remedy (or something), System Restore, Defender..etc all of these capable to stop your IO from completion, and on top of that if they failed for some reason, and your application is done and gone, and you are left with cryptic error message like yours.

 

Also there is one in particular not mentioned in 1 and 2, Application Verifier, though it is low probability to be your problem, but checking does worth it, https://learn.microsoft.com/en-us/windows-hardware/drivers/devtest/application-verifier

Make sure if it is exist on the target device, if yes then open it and remove/delete any thing in that list, yes delete them and you will know how once you see it.

 

Anyway back to suggestions and shooting darts in the dark, try to find the service attached to that error, and delete system restore check points, disable then enable... just try to figure what service is interfering and try to remedy it, one of them could have broken policy or setting or just really out of resources. 

[Roger Cigol 114]

When I open resource monitor I see 2 threads and 15 handle names (13 are files, one is the application exe and one is the application mutex created and checked by our application (we use this to prevent two instances being run at the same time)

So this doesn't show any problem.

Hard Disk still has 43GB used of a total capacity of 930GB

 

[Kas Ob. 126]

Use Process Monitor without filter(s), them see what is trying figure who reported the Resource Error.

[Der schöne Günther 323]

I've had this with faulty drivers from garbage companies like Intel. Their resource usage is nowhere to be seen in Task Manager, over time, they clearly eat up more and more system resources until the system starts collapsing.

 

Once again, stuff like this is clearly logged in Windows Event Viewer. It might provide additional info.

[Lajos Juhász 311]

My bet is on file handles?

[Roger Cigol 114]

25 minutes ago, Kas Ob. said:

Service ! , so the cause is not your application,

Are you sure that "service" means a windows service in this context? I am not so sure that MS error messages are so precise! I think this just means a "general service provided by the operating system" - but I could be wrong.

@Kas Ob. I am a bit out of my depth with this one - so your expertise is really welcome and appreciated. So above question is just me being ultra cautious!

 

[Roger Cigol 114]

4 minutes ago, Lajos Juhász said:

My bet is on file handles?

@Lajos Juhász  How can I check this ?

 

[Roger Cigol 114]

I have remote access to the machine, My application is still running and I can confirm that it is still not able to create / write to any files.

I have used the sysinternal handle.exe utility. When run with -a -p nnnn >MyOuptutTextFile.txt                       //  (where nnnn = my application process ID)

 

When I look at the resulting output file it is 400MB long and almost all of it is taken up with the same entries, here's an extract

 

 3B4: <Unknown type> \Sessions\1\BaseNamedObjects\MyMutex1
  3B8: <Unknown type> \Sessions\1\BaseNamedObjects\MyMutex1
 

// loads and loads and loads of entries here

//    -      and then the file ends

 

 F71D78: <Unknown type> \Sessions\1\BaseNamedObjects\MyMutex1
 F71D7C: <Unknown type> \Sessions\1\BaseNamedObjects\MyMutex1
 F71D80: <Unknown type> \Sessions\1\BaseNamedObjects\MyMutex1
 F71D84: <Unknown type> \Sessions\1\BaseNamedObjects\MyMutex1
 F71D88: <Unknown type> \Sessions\1\BaseNamedObjects\MyMutex1
 F71D8C: <Unknown type> \Sessions\1\BaseNamedObjects\MyMutex1
 F71D90: <Unknown type> \Sessions\1\BaseNamedObjects\MyMutex1
 F71D94: <Unknown type> \Sess
 

This doesn't look healthy to me - but what does this line mean?

[Kas Ob. 126]

1 minute ago, Roger Cigol said:

Are you sure that "service" means a windows service in this context? I am not so sure that MS error messages are so precise! I think this just means a "general service provided by the operating system" - but I could be wrong.

@Kas Ob. I am a bit out of my depth with this one - so your expertise is really welcome and appreciated. So above question is just me being ultra cautious!

 

TO be honest it is not exactly a service but stapled to one, see this cryptic error most likely caused and reported from User-Mode Driver, as kernel mode driver are more detailed error, yet in both cases these drivers belong and part of to an OS service as mentioned above, i think it is between System Restore, Defender or some network mapped drive.

[Roger Cigol 114]

35 minutes ago, Kas Ob. said:

Use Process Monitor without filter(s), them see what is trying figure who reported the Resource Error.

I have sysinternals process monitor running on the machine. I get a long list of processes - but can't see my application mentioned anywhere (note - may application it is not actually doing anything at present - but it is still running)

Any tips on how to drive this ?

[Kas Ob. 126]

Just now, Roger Cigol said:

This doesn't look healthy to me - but what does this line mean?

No not healthy at all.

 

But this one could be the running out resource !

Yes handles has limit and still the same as i mentioned above, the handle is wrongly handled, 

 

Please, download WinObjEx64 https://github.com/hfiref0x/WinObjEx64/releases

Then share screenshots of the three tabs like this 

 

[Kas Ob. 126]

Just now, Roger Cigol said:

Any tips on how to drive this ?

Not mentioned is enough for this one, it does confirm it is coming form a driver, low level one.

[Kas Ob. 126]

 

Just now, Kas Ob. said:

Please, download WinObjEx64 https://github.com/hfiref0x/WinObjEx64/releases

Then share screenshots of the three tabs like this 

Sorry i forgot to mention to search and find your MyMutex1

[Roger Cigol 114]

@Kas Ob.   My actual app is a 32 bit exe - will the 64bit WinObjEx64 still be ok to help us (you!) get to the bottom of this ?