TMethodImplementationIntercept/__dbk_fcall_wrapper called infinite and high cpu

[mvanrijnen 121]

We have one application which since a few months gives problems on most of the workstation here. 

(not all, ofcourse not on the dev machines 🙂 )

 

The application goes into high cpu constantly without doing anything (wanted).

We noticed that this was causes by a thread which uses/started because TMethodImplementationIntercept

so we fixed thats this method was not exported, then the troubles moved to __dbk_fcall_wrapper.

 

It's only in one of the many applications we have here that this problem occurs, anyone any idea or direction we should search in?

 

[Kas Ob. 91]

On 8/10/2023 at 6:16 PM, mvanrijnen said:

We noticed that this was causes by a thread which uses/started because TMethodImplementationIntercept

so we fixed thats this method was not exported, then the troubles moved to __dbk_fcall_wrapper.

I think screenshot would helped earlier, so others here could spot the problem for you, look at these

 

is that what do you see ?

The first one is an new project while the second has this

Quote

procedure TForm10.FormCreate(Sender: TObject);
begin
  while True do
end;

 

So i think you are following the wrong thing, as many tools are build for Windows binaries with PDB debug information, in this case it could find a suitable address (previous with an export or known from the debug info) so it grabbed the closest before the address and resolved it as entry point.

Find the culprit address by calculating the address of TMethodImplementationIntercept +xxxxxx then follow it in you debugger, or attach a debugger or... may be try Andres PDB debug info tool see if that works with your watching tool, or find another tool that does show an address instead of a relative address ...

 

Hope that helps, and good luck.

[mvanrijnen 121]

Yes, indeed thats what we see, difference is that there are more threads which also take cpu. 

gonna take a look into your tip of finding the real method/thread you mentioned.

 

quick question, the Displayed ThreadID should always be correct ?

 

Edited August 21, 2023 by mvanrijnen

[Kas Ob. 91]

50 minutes ago, mvanrijnen said:

Displayed ThreadID should always be correct ?

Yes.

 

Using remote Delphi debugger or using different debugger to get these addresses will be way more accurate and easier.

This one is the best out there 

https://x64dbg.com/

[aehimself 388]

@mvanrijnen Did you find an effective way to debug this? My application started to do the same, with a similar call stack in the offending thread:

 

 

It's important to note that this thread is not a worker of mine, I don't know what or when it was created. It also safely can be killed and won't cause any (noticable) disturbances.

I also couldn't find a way to reproduce the issue, one time it just thinks it's time and then it locks up.

[Kas Ob. 91]

15 hours ago, aehimself said:



 

It's important to note that this thread is not a worker of mine, I don't know what or when it was created. It also safely can be killed and won't cause any (noticable) disturbances.

Evidently it is from Delphi code and started with CreateThread , see the UserThreadStart ? that is it.

 

So this thread in particular is create from library you are using, it could be VCL or RTL or 3rd party or some unit you included, but without any doubt it is started from CreateThread there is high chance to be from TThread somewhere, but this is an encapsulation after all.

 

There is also a chance it is from a DLL your code is calling, but again this thread is created from user space code not the kernel, also from the stack itself it is deep nested with 7 levels in your EXE, so it should be easy to identify.

[Anders Melander 1639]

1 hour ago, Kas Ob. said:

without any doubt it is started from CreateThread there is high chance to be from TThread somewhere, but this is an encapsulation after all.

There are other ways for a thread to get started, but ignoring that and assuming it was created with CreateThread, one could simply place a breakpoint on the CreateThread import in Winapi.Windows and investigate the call stack as the threads gets crated.

[Stefan Glienke 1905]

The GetSystemTimes call reminds me of System.Threading.TThreadPool.TThreadPoolMonitor.Execute which calls TThread.GetCPUUsage all the time.

See https://quality.embarcadero.com/browse/RSP-43540

Edited February 16 by Stefan Glienke

[Kas Ob. 91]

11 minutes ago, Anders Melander said:

There are other ways for a thread to get started,

Yes there is, but in this case the RtlUserThreadStart from the kernel user mode (ntdll) is the one supplied with ThreadProc from CreateThread from this running process.

[Anders Melander 1639]

32 minutes ago, Kas Ob. said:

Yes there is, but in this case the RtlUserThreadStart from the kernel user mode (ntdll) is the one supplied with ThreadProc from CreateThread from this running process.

How can you tell?

[Kas Ob. 91]

@aehimself use ApiMonitor to find these not-yours threads http://www.rohitab.com/apimonitor

 

 

NT Native -> Process and Threads -> Ntdll.dll for lower level functions

and

Process and Threads -> Thread -> Kernel32.dll for higher level function (your usual user mode functions aka RTL)

 

[Kas Ob. 91]

Just now, Anders Melander said:

How can you tell?

Hours and hours digging into Windows kernel, also the name of that first in the stack function is very specific and very familiar RtlUserThreadStart, as example, CreateRemoteThread doesn't invoke this one.

Lastly from old readings, i can't find many resources but have a look here http://www.nynaeve.net/?p=200

[Anders Melander 1639]

4 minutes ago, Kas Ob. said:

use ApiMonitor to find these not-yours threads http://www.rohitab.com/apimonitor

Nice! I didn't know that one.

Too bad about the XP skin though 🙂

[Anders Melander 1639]

2 minutes ago, Kas Ob. said:

Hours and hours digging into Windows kernel, also the name of that first in the stack function is very specific and very familiar RtlUserThreadStart, as example, CreateRemoteThread doesn't invoke this one.

Hmm. Okay, I'll defer to your expertise then 🙂 - but I can't tell, from looking at the call stack, if the thread was created from Delphi code or from something injected into the process from the outside. I was hoping there was some obvious clue that I had missed.

 

5 minutes ago, Kas Ob. said:

Lastly from old readings, i can't find many resources but have a look here http://www.nynaeve.net/?p=200

Quote

RtlUserThreadStart is used on Windows Vista and Windows Server 2008 (and later OS’s) to form the initial entrypoint context for a thread started with NtCreateThreadEx (this is a marked departure from the approach taken by NtCreateThread, wherein the user mode caller supplies the initial thread context).

More hmmm. Doesn't really tell me much with regard to the source of the thread.

[Kas Ob. 91]

8 minutes ago, Anders Melander said:

Nice! I didn't know that one.

What to say, i have pasted the link and mentioned many times 😎

For more powerful tool i use, different but more invasive/intrusive i use CheatEngine 

https://www.cheatengine.org/

https://github.com/cheat-engine/cheat-engine

 

Written mostly in FreePascal, and it is for games, yet it is so much powerful with its monitoring and even capturing low level event like executing a specific assembly code or passing though (executing) specific address or even accessing a block of memory (read or write), also there is LUA scripting...

In short it is really useful to master and use.

[aehimself 388]

 

So the process of finding these is not that easy but not that hard either, I'll put it here so future visitors don't have to research / experiment.

- Download @Anders Melander's Map2pdb, compile it and run it against your applications .map file. You'll get a .pdb file.

- Download WinDbg and install it. Once done, go to File -> Settings -> Debugging settings and add the folder where your .pdb file is to the Symbol path list

- Launch your application and when it's in the state you want to examine, go to File -> Attach to process and select your application.

- In the lower-right pane select the "Threads" tab and double-click the offending one (TID = ThreadID in HEX)

- Now go to the Stack tab and if everything is good, you get readable stack traces

 

In my case @Stefan Glienke was absolutely right:

 

 

As I am not using thread pools, I need to find out what does... and get rid of it, somehow...

 

[Anders Melander 1639]

11 minutes ago, Anders Melander said:

Too bad about the XP skin though 🙂

Turns out there's good explanation for that 😕

 

But at least the UI has a "Windows 7" skin.

 

[Kas Ob. 91]

3 minutes ago, Anders Melander said:

if the thread was created from Delphi code or from something injected into the process from the outside.

Good point, but remember invoking/calling CreateThread from local thread doesn't require specific privileges but while injecting (almost always ) with CreateRemoteThreads does require security privileges.

 

This is interesting https://github.com/stephenfewer/ReflectiveDLLInjection/pull/17

Quote

Firefox on Windows hooks kernel32!BaseThreadInitThunk which prevents the remote thread to start in the target, see https://dxr.mozilla.org/mozilla-central/source/mozglue/build/WindowsDllBlocklist.cpp#821:

FireFox indeed tries (tried in the past i don't know the current code) to protect itself from remote injection by hooking the BaseThreadInitThunk not the RtlUserThreadStart, for the same reason that RtlUserThreadStart is not always the start point.

 

 

16 minutes ago, Anders Melander said:

Quote

RtlUserThreadStart is used on Windows Vista and Windows Server 2008 (and later OS’s) to form the initial entrypoint context for a thread started with NtCreateThreadEx (this is a marked departure from the approach taken by NtCreateThread, wherein the user mode caller supplies the initial thread context).

More hmmm. Doesn't really tell me much with regard to the source of the thread.

Well you are diving deeper into OS kernel, so to make sure we are on the same page first let clear the separation of the functions in the OS as whole Kernel part and kernel user part.

 

In Windows there is 3 levels of functions, and they are named little differently, sometimes the difference is only with Nt or Zw against nothing, or completely different name encapsulating multi functionality.

eg CreateThread is for RTL user mode, this will internally call NtCreateThread we still in the kernel but in the user part which is lower than user process but higher than the kernel itself (the hidden and protected one), then comes ZwCreateThread which reside in the kernel and this one is system call not system function, meaning the execution is not done by simple assembly branching instruction like JMP or CALL, no this is done by SYSCALL and SYSENTER 

https://www.felixcloutier.com/x86/syscall

https://www.felixcloutier.com/x86/sysenter

 

This page https://learn.microsoft.com/en-us/windows-hardware/drivers/kernel/using-nt-and-zw-versions-of-the-native-system-services-routines explain the difference but still hard to grasp or understand it from one reading, hence i am trying (or failing) to make clearer a little.

With each level different checks are performed for security, errors, ... Zw calls are essentially to be called directly and exclusively form drivers and the kernel, Nt calls are less strict yet these Nt call are the ones that will check for privileges to perform/acces from User more process, while Zw are the ultimate to decide as there is many of them will simply refuse to execute because the calling thread is not kernel one, Nt will refuse to execute if you don't have user mode privileges.

 

Take as example CloseHandle, this function does close almost everything yet it called CloseHandle, there is NtCloseHandle, but there is no ZwCloseHandle, there is ZwClose that perform all the closing in the kernel.

 

Now i drifted far form the question and your comment (but for IMO good reason), NtCreateThreadEx is the real function behind CreateThread (which in fact is calling NtCreateThread) from the User Mode and will perform the same functionality but it does have the last check for privileges and context to execute or invoke a new thread.

 

Not sure if this was clear, i just hope.

[Anders Melander 1639]

3 minutes ago, Kas Ob. said:

Not sure if this was clear, i just hope.

Very clear. Thank you.