Access violation on DataModule := TDataModuleMain.Create(nil);

[ioan 45]

I have a multithreaded service that creates a TDataModule for each thread. The DataModule has the logic for some light work on a firebird database.

My service runs, depending of the hour of the day, between 0 and 300 consecutive threads and about 50-70 thousand total threads in a day. Some threads can be alive for an hour or more, some for few seconds.  Most of the time the threads are just idle, waiting for a file on disk (each thread waits for a file with a specific name, no two threads try to read the same file). 

 

There are no memory leaks in the service, if I run the service as an application for a whole day in production with the 

ReportMemoryLeaksOnShutdown := true;

and there are no leaks reported. 

There are no threads that remain hanging, after a certain while, they'll terminate automatically if they are alive for too long (a timeout value depending of what the job is). 

 

The problem is that after a random amount of time of running, the thread creation fails at with Access Violation

DataModule := TDataModuleMain.Create(nil);

and when this starts, no more threads can be created, all of them fail at the same line, the line above.

 

The TThread.Create is as follows:

constructor TPFXIncomingMessage.Create;
begin
  inherited Create(true);
  FreeOnTerminate := true;
  DataModule := TDataModuleMain.Create(nil);
  FIsRecovered := false;
end;

 

Here is the call stack:

EAccessViolation: Access violation at address 0000000000921298 in module 'pfxout.exe'. Read of address FFFFFFFFFFFFFFFF
----------------------------
[0000000000F1EB51] JclDebug.TJclStackInfoList.Create + $151
[0000000000F1E698] JclDebug.JclCreateStackList + $48
[0000000000F1E5A6] JclDebug.DoExceptionStackTrace + $76
[0000000000F20BA6] JclDebug.DoExceptNotify + $86
[0000000000F116C5] JclHookExcept.TNotifierItem.DoNotify + $35
[0000000000F1190B] JclHookExcept.DoExceptNotify + $BB
[0000000000F11A65] JclHookExcept.HookedRaiseException + $75
[00000000008F21D3] System.@RaiseAtExcept + $103
[00000000008F2238] System.@RaiseAgain + $38
[00000000015FC969] threaddoit.TPFXIncomingMessage.Create (Line 59, "threaddoit.pas" + 5) + $42
[00000000015FFC76] outthread.TOutThread.ProcessRecRequest (Line 182, "outthread.pas" + 15) + $E
[00000000015FF4CB] outthread.TOutThread.Execute (Line 88, "outthread.pas" + 6) + $0
[0000000000A1A1E3] System.Classes.ThreadProc + $43
[00000000008F2DAD] System.ThreadWrapper + $3D
[00007FF95AA713D2] BaseThreadInitThunk + $22
[00007FF95C455504] RtlUserThreadStart + $34

 

The service is compiled for 64 bits.

Before I start modifying everything, trying to fix a problem that I have no idea where it comes from, before trying to rewrite a bunch of code that should work and has no memory leaks, my question is: Are there some kind of rules that I don't know about on creating and freeing so many DataModules? I'm asking this because I had the same problem with a RemObjects SOAP service and the answer I got on the remobjects forum was: 

it is failed at creating _Impl forms.
it is weird error and something is happened in delphi’s standard library

 

Any ideas or advice?

 

Thanks.

 

Edited May 27, 2022 by ioan

[PeterBelow 238]

The VCL is not thread-safe in itsself, creating top-level objects like forms, frames or datamodules accesses some global stuff in vcl.forms connected to reading components from the dfm file.

 

Try to replace your datamodules with a class (not derived from TCustomForm) that creates and configures all components you now place at design-time in code. I dimly remember a tool that could create such code from a dfm file, never used it myself, though.

 

Are all your threads created in the same host thread? If so that should actually mitigate the problem since the thread constructor runs in the host thread, so all datamodules would be created in the same thread. But that may cause another problem: database connections are usually bound to the thread that creates them. If the datamodule creation already creates the connection, which is then later used in the context of the thread that owns the datamodule that may cause problems since the db access library may try to solve the conflict by serializing all DB access in a separate thread.

 

Multithreading using a database can be even trickier than multithreading itself...

[Vandrovnik 214]

What if you create the datamodule using Synchronize?

[ioan 45]

26 minutes ago, Vandrovnik said:

What if you create the datamodule using Synchronize?

I'll try but I'm not sure if it makes any difference. The code in TThread.Create where the datamodule is created is executed in the main thread. 

Edited May 28, 2022 by ioan

[Dalija Prasnikar 1396]

12 hours ago, ioan said:

I'll try but I'm not sure if it makes any difference. The code in TThread.Create where the datamodule is created is executed in the main thread. 

Are you sure it is executed in the context of the main thread? Because that is not what the stack trace says. It shows it is called from within TOutThread.Execute method.

 

Anyway, whatever the problem is, there is not enough code to determine what is the root cause. Data modules can be constructed in the background threads, but only and only if all components used are thread safe in that regard. In other words if they support being constructed in background thread. How they are configured and what other components are linked as properties also impacts the thread safety.

 

Additional comment. That application does not have memory leaks is good, but not having memory leaks does not mean that code is thread-safe and that it will run correctly.

Edited May 29, 2022 by Dalija Prasnikar

[Attila Kovacs 629]

@ioan I'd check what exactly is on 0x0000000000921298 this could give you more hints what is failing. For this reason I'm always archiving the map file for a release, otherwise you have to look up the asm in the (failing) release and try to identify the place with the latest src/build.

[David Heffernan 2345]

I mean you could just use madExcept and have a proper stack trace for all such occurrences

[ioan 45]

On 5/29/2022 at 12:29 AM, Dalija Prasnikar said:

Are you sure it is executed in the context of the main thread? Because that is not what the stack trace says. It shows it is called from within TOutThread.Execute method.

 

Anyway, whatever the problem is, there is not enough code to determine what is the root cause. Data modules can be constructed in the background threads, but only and only if all components used are thread safe in that regard. In other words if they support being constructed in background thread. How they are configured and what other components are linked as properties also impacts the thread safety.

 

Additional comment. That application does not have memory leaks is good, but not having memory leaks does not mean that code is thread-safe and that it will run correctly.

TOutThread is the main thread that starts lots of worker threads. The problem is in the worker threads. The only global variables I'm using are some strings that I initialize at the application start (before any threads are running) with the initial paths and the worker only read the values. All the other components are standard Delphi firedac database access. 

 

[ioan 45]

22 hours ago, David Heffernan said:

I mean you could just use madExcept and have a proper stack trace for all such occurrences

I tried madExcept several years ago and I couldn't get it to work right in a service, so I ended up using JCL Debug, which works fine.  I'll probably have to look into it again.

[David Heffernan 2345]

6 minutes ago, ioan said:

I tried madExcept several years ago and I couldn't get it to work right in a service, so I ended up using JCL Debug, which works fine.  I'll probably have to look into it again.

madExcept works fine a service for me. But if you have JCL Debug working then that's fine. In which case can you get a proper stack trace? 

[ioan 45]

12 minutes ago, David Heffernan said:

madExcept works fine a service for me. But if you have JCL Debug working then that's fine. In which case can you get a proper stack trace? 

The stack trace I posted is the only thing I get when this access violation occurs. Usually when there is an exception, I get more detailed stack trace, but with this problem, that's the only thing I get.

[ioan 45]

I started from scratch (had in plan to trim some fat anyway) and now I'm accessing the database directly from the TThread, inspired by the example from Embarcadero: 

https://docwiki.embarcadero.com/RADStudio/Sydney/en/Multithreading_(FireDAC) 

The service now works very well, no more access violations. 

I wish I had the time and patience to actually find exactly what exactly caused the problem, but the problem only appeared in production and I was pressed to fix it fast so I pretty much made it from scratch.