Need help investigating an app crash

[Der schöne Günther 336]

I have a Delphi app that is running 24/7, around the globe. On one very important installation, it will just crash and terminate, with no sign of an error message. Of course, I do log full stack traces on any exception as well as Windows resource exhaustion and other things. But this time, I am really stuck.

 

Here is what is happening:

1. The app is constantly polling another device through TCP. I am using the delphi-modbus library, which is, of course, based on Indy.
2. At some point, the main thread will also talk to the device, and fail with a custom exception type. What bothers me, is at the exact same time, the background thread will also fail polling the device. That points to a resource being used by two threads at once, but the connection is entirely guarded by a Critical Section.
3. Now the really interesting part: A third thread (one of Indy's scheduler threads) will now crash with an Access Violation. And at that moment, Windows shuts down my entire application.

 

I have not attached the callstacks for (2) because I don't think they provide any useful information.

 

As for (3), my app was able to log the following callstack:

24/07/2025 14:31:32
[63.4]
Thread-ID 9052 ("TIdThreadWithTask")EAccessViolation: "Access violation at address 0098638C in module 'MyStuff.exe'. Write of address 00000004"
[0098638C] IdTask.TIdTask.DoBeforeRun
[00CA9106] JclHookExcept.DoExceptFilter
[00CA92D9] JclHookExcept.HookedExceptObjProc
[0040BAA3] System.@HandleAnyException
[004E2399] System.Classes.ThreadProc
[0040C6F8] System.ThreadWrapper

Then, Windows will log the following two entries in the event viewer:

 

First, there is an exception of 0xc0000409, which, according to Raymon Chen, just means "that the application decided to terminate itself with great haste"

Event[1]:
  Log Name: Application
  Source: Application Error
  Date: 2025-07-24T14:31:32.4810000Z
  Event ID: 1000
  Task: Application Crashing Events
  Level: Error
  Opcode: Info
  Keyword: Classic
  User: N/A
  User Name: N/A
  Description: 
Faulting application name: MyStuff.exe, version: 63.4.0.0, time stamp: 0x67b5aae0
Faulting module name: MyStuff.exe, version: 63.4.0.0, time stamp: 0x67b5aae0
Exception code: 0xc0000409
Fault offset: 0x020b025d
Faulting process id: 0x1e5c
Faulting application start time: 0x01dbfc922ab483ef
Faulting application path: C:\Users\Public\MyStuff.exe
Faulting module path: C:\Users\Public\MyStuff.exe
Report Id: 2a531142-03f7-461e-9fa5-9f50df86ef33

A few milliseconds afterwards, the access violation is also logged

Event[0]:
  Log Name: Application
  Source: Application Error
  Date: 2025-07-24T14:31:32.9150000Z
  Event ID: 1000
  Task: Application Crashing Events
  Level: Error
  Opcode: Info
  Keyword: Classic
  User: N/A
  User Name: N/A
  Description: 
Faulting application name: MyStuff.exe, version: 63.4.0.0, time stamp: 0x67b5aae0
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0xf0a79e3d
Faulting process id: 0x1e5c
Faulting application start time: 0x01dbfc922ab483ef
Faulting application path: C:\Users\Public\MyStuff.exe
Faulting module path: unknown
Report Id: 9b3d67f1-32be-44df-bae9-466d2f36d1b8
Faulting package full name: 
Faulting package-relative application ID: 

For me, it looks like 

1. "Something" triggers stack corruption, which windows logs as a "Stack Buffer Overrun"
2. Possibly within Indy's code (TIdTask)
3. Windows will then shut the process down, wrapping it in an unhandled "Access violation"

 

My dilemma:

How on earth can I get behind this? I see I can use tools like procdump to automatically create a dump when the application gets shut down, but even then, I don't know what I would to with it.

I have looked up the "fault offset" in the first exception (adding the base address to 0x020b025d), and I don't see any useful code there.

 

Of course there is probably something wrong with my code, but I am rattled by Indy code being the final code running before the lights go out. Maybe @Remy Lebeau does have an idea?

My Indy version (10.6.2.5311) is an older one. I have tried looking up changes since then, but Github doesn't go that far back. Where else can I check?

Edited 13 hours ago by Der schöne Günther

[DelphiUdIT 255]

I use Indy Bundle and the same Modbus component to read a PLC (Encoder conuter) every 10 / 15 ms. and write on them every 150 ms. (EDIT... I do this since 8 years and there are dozen of machines that work in this way 24h/24h).

I never had such problem.

TAKE CARE that you cannot use Indy (and ModBus too of course) from differnt Threads (you spoke about use it in Main Thread too).

 

Try to modify the logic and work from single Thread (where you can create and  use ModBus) ... untill you are sure the Indy operation don't interfer between threads.

Edited 13 hours ago by DelphiUdIT

[PeaShooter_OMO 36]

@Der schöne Günther

Do you call the Connected function anywhere?

 

Indy can be used from multiple threads but how you use it will drive that decision.

 

I created a TCP framework using Indy and my Client component has a ListenerThread. So the ListenerThread obviously will be polling for incoming data. Sending will be from other threads, obviously the threads that calls the SendXXX commands. The issue was that inside the sending threads (inside the SendXXX commands)  I used to ask via Connected if the connection was still there and that caused anomalies. The problem is that Connected actually reads from the socket and thus you will have multiple threads reading from the socket. Not a good thing to do.

 

So in my framework's functionality one thread read/receives all data and hands it off to an event where whoever implements the event can sync or do whatever it pleases with the data but on the sending side I make sure I don't read the socket.

 

If in your case a thread follows the traditional way of Sending-Reading-Sending-Reading-[repeat] all in itself then make sure you lock that so that only one thread does that at a time but also take note of Connected's consequence.

Edited 12 hours ago by PeaShooter_OMO

[Der schöne Günther 336]

24 minutes ago, DelphiUdIT said:

you cannot use Indy (and ModBus too of course) from differnt Threads (you spoke about use it in Main Thread too)

Are you 100 % certain? I have been doing so for many years. I am aware of things like Windows GDI resources which can only be used from the thread they were created (something like this), but I suppose Indy should absolutely be fine, as long as everything is properly serialized through things like critical sections...

 

21 minutes ago, PeaShooter_OMO said:

Do you call the Connected function anywhere?

Yes, I eventually do.

21 minutes ago, PeaShooter_OMO said:

I used to ask via Connected if the connection was still there and that caused issues

That sounds exactly what my code is doing. Thank you! I will investigate.

[PeaShooter_OMO 36]

I basically got rid of worrying about the connected state via Connected and just did try...except around the socket access parts and then react based on that. I do keep a state flag internal for myself but that just indicates if there was a successful connection made in this session.

[Remy Lebeau 1623]

1 hour ago, Der schöne Günther said:

But this time, I am really stuck.

There is really no way for anyone to diagnose this problem for you with such limited information.

Quote

As for (3), my app was able to log the following callstack:

An AV near memory address 0 almost always means a nil pointer was accessed. But you would need to debug your program to figure out what code was actually running at memory address 009863BC when the AV occurred. The fact that Indy's TIdTask and TIdThreadWithTask classes appear on the call stack could be related or benign, there's just no way to tell from the error message alone.

Quote

My Indy version (10.6.2.5311) is an older one.

Potentially quite old.

Quote

I have tried looking up changes since then, but Github doesn't go that far back.

Actually, it does. GitHub has Indy 10's full change history going back to 2005 (Indy migrated from SVN to GitHub in 2019).  Indy 10 rolled to 10.6.2.0 in commit f60999f on April 22 2013, and rolled to 10.6.3.0 in commit b079cb2 on Mar 2 2024. That gives you an 11 year range of history, available on GitHub, for the entire 10.6.2 run. Sorry, I couldn't tell you exactly which commits belong to 10.6.2.5311 specifically. The build numbers were generated by SVN scripts until the GitHub migration. If it was a version shipped with the IDE, you can use the IDE's release date to narrow down the commits. Or, if not a shipped version, then the date the files were downloaded.

Edited 12 hours ago by Remy Lebeau

[Der schöne Günther 336]

1 minute ago, Remy Lebeau said:

There is really no way for anyone to diagnose this problem for you with such limited information.

Is there anything else I can include?

 

I've tried running a debug build with "Debug DCUs", Range and Overflow Checking enabled, but that didn't change anything.

 

As of now, I had to disable integration of that one device, and the software has been running completely fine since then.

[Remy Lebeau 1623]

15 minutes ago, Der schöne Günther said:

I suppose Indy should absolutely be fine, as long as everything is properly serialized through things like critical sections...

Correct.

9 minutes ago, PeaShooter_OMO said:

I basically got rid of worrying about the connected state via Connected and just did try...except around the socket access parts and then react based on that.

Agreed. Do the I/O normally and let Indy tell you when the connection is closed.

[Remy Lebeau 1623]

3 minutes ago, Der schöne Günther said:

Is there anything else I can include?

Not really, not without knowing what code actually crashed, and what it was actually trying to do at the time of the crash. Debugging via forum isn't a good way to handle this. You need to actually debug the program directly and try to reproduce the issue.

3 minutes ago, Der schöne Günther said:

As of now, I had to disable integration of that one device, and the software has been running completely fine since then.

You say you have 2 threads trying to talk to the same device, presumably over a single connection. Best to give each thread its own connection if possible. Otherwise, I would suggest delegating all I/O on that connection to a single thread and let it accept queries from, and return responses to, other threads as needed.

[Kas Ob. 148]

Just now, Der schöne Günther said:

Is there anything else I can include?

1) Windows version

2) Confirm if this crash happen with the same exact addresses/parameters , specially the "Fault Offset", it is important.

3) How frequent this happen ? can you reproduce or predict the moment of this crash ?

4) Get the ProcDump with full dump like this "procdump -ma -e 1 -t MyStuff.exe MyStuffCrash.dmp" , this is loadable with WinDBG then try to understand what exist at 0xf0a79e3d with "!address"

 

Theories:

1) That fault offset is not out of thin air, it might be coming from hidden WOW64, yet the looks like 32bit hence the unexplained address, these emulator DLLs are 64bit and loaded in 32bit, though impossible to see or interact with them from the process itself, only you see their exception when they raised, as always their exceptions are fatal.

2) handling the exception did in fact corrupt the stack and triggered cascade of problems in Windows trying to unwind , so i don't like these two calls appears on the call stack 

Quote

[00CA9106] JclHookExcept.DoExceptFilter
[00CA92D9] JclHookExcept.HookedExceptObjProc

they looks the cause, just remove them and see if things changes, but this has to do with the extra information above (3)

 

how the above will help, well, you can't solve such crash without pinpoint specific operation, those are (form my experience) a conflict in IO operation combined with stack usage for buffer (or some return values) from different thread, missing or forgetting about these return values, so review your code and make sure your local variable are not passed to different threads, or handling the exception itself triggered extra corruption.

[Kas Ob. 148]

Yes i don't like seeing these on the call stack captured and reported by OS

Quote

[00CA9106] JclHookExcept.DoExceptFilter
[00CA92D9] JclHookExcept.HookedExceptObjProc

This interpreted as the exception handling did fail and returned to the OS, hence the shut down.

I don't have any experience with JclHookExcept, never used it, and looking at its code 

https://github.com/bematech/jcl/blob/master/windows/JclHookExcept.pas#L383

leading to 

https://github.com/bematech/jcl/blob/master/windows/JclHookExcept.pas#L322

 

Now what, did you tried to resolve or access the custom exception ? may be trying to report some buffer, return values, or any thing was on a thread stack that might be already gone ? 

This is very plausible scenario.

[DelphiUdIT 255]

About the Connected "property.

In my code I used to do this:

//PLcBot is TIdModBusClient
//THIS IS PRIVATE procedure used ONLY INSIDE A THREAD
procedure TPlcComm.fprocBotIP(value: string);
begin
  if PlcBot.Connected then   <--- connetcted tested like normaly should be
    PlcBot.Disconnect;
  PlcBot.Host := Value;
  try
      PlcBot.Connect;
  except on e:exception do
    ;
  end;
end;

//This is the test connection used OUTSIDE the THREAD, via a READ ONLY property
function TPlcComm.fGetPlcConnection: boolean;
begin
  if assigned(PlcBot.IOHandler) then				<---- Test CONNECTED used in another way !!!
    result := PlcBot.IOHandler.Connected			<----
  else
    result := false;
end;

The test about connection state is used in different way if used outside the thread or inside the thread, may be at that time (many and many years ago) there was something wrong about that.

It has been working for at least 8 years (this part was last updated almost 8 years ago).

Edited 11 hours ago by DelphiUdIT

[PeaShooter_OMO 36]

@DelphiUdIT

PlcBot.Connected and PlcBot.IOHandler.Connected is the same. Inside Connected it checks against IOHandler.Connected.

 

Checking against Connected is not wrong but be mindful that you might get issues when doing so at the same time as reading the socket from another thread.

Edited 11 hours ago by PeaShooter_OMO

[Der schöne Günther 336]

57 minutes ago, Remy Lebeau said:

Best to give each thread its own connection if possible.

This is deliberate, because the device is a simple controller that cannot serve multiple requests. It can only keep one socket connection open. Therefore, the Delphi app is only opening one connection and sharing it throughout the code.

 

57 minutes ago, Remy Lebeau said:

Debugging via forum isn't a good way to handle this. You need to actually debug the program directly and try to reproduce the issue.

I know. It's not that easy to debug software on an industrial line that is processing tons of raw materials while the line is also on display for an exhibition. I'm doing what I can 😳

 

52 minutes ago, Kas Ob. said:

How frequent this happen ? can you reproduce or predict the moment of this crash ?

I haven't been able to reproduce it. There are other lines running the exact same software and configuration, and they're fine too. I am currently investigating a different software version on said microcontroller, but I am still at loss how this could affect the Delphi app so deeply.

 

51 minutes ago, Kas Ob. said:

Confirm if this crash happen with the same exact addresses/parameters , specially the "Fault Offset", it is important.

Thank you, I will check.

 

 

Many thanks for all your replies so far! ❤️

Edited 11 hours ago by Der schöne Günther

[PeaShooter_OMO 36]

8 minutes ago, Der schöne Günther said:

1 hour ago, Remy Lebeau said:

Best to give each thread its own connection if possible.

This is deliberate, because the device is a simple controller that cannot serve multiple requests. It can only keep one socket connection open. Therefore, the Delphi app is only opening one connection and sharing it throughout the code. 

Remy's second suggestion would be a good idea... 

Quote

Otherwise, I would suggest delegating all I/O on that connection to a single thread and let it accept queries from, and return responses to, other threads as needed

My framework also has a thread for sending. It is working on a single socket after all .

 

Edited 11 hours ago by PeaShooter_OMO

[PeaShooter_OMO 36]

11 minutes ago, Der schöne Günther said:

I haven't been able to reproduce it

I could not reproduce it precisley either when I had my issue. It would just happen whenever the gremlins were bored. I suggest you take a look at either a single thread that handles all comms or perhaps a single sending thread paired with a single receiving thread for all comms like I do and keep all the Indy/ModBus calls in that thread.

[DelphiUdIT 255]

27 minutes ago, PeaShooter_OMO said:

@DelphiUdIT

PlcBot.Connected and PlcBot.IOHandler.Connected is the same. Inside Connected it checks against IOHandler.Connected.

 

Checking against Connected is not wrong but be mindful that you might get issues when doing so at the same time as reading the socket from another thread.

I don't know. I worked always like I wrote with 1 thread for ModBus connection.

But I had what I show changed at that time, with no comment (my fault  ) and there were no pratical reasons to had done this.

 

May be like you said something about multithreading with access to IoHandler, and that only change something obscure ...

 

To do some datas, this week (7 days) a line that works near 8 hours / day have done more than 15 milions of reading (MODBUS) from PLC with only 5 failures and more than 1 milions of writing with no failure (that machine were in test with monitoring all working parameters, also double check about write ModBus register and read back the values).