Find exception location from MAP file?

[A.M. Hoornweg 144]

Hello all,

 

one of my (heavily multithreaded) Windows services crashed today with an access violation and Windows was decent enough to write the "fault offset" in the event log.  
I have a detailed linker MAP file of the application.  Does anyone know a tool that will parse the MAP file to help me find the approximate error location in the source?

[David Heffernan 2345]

3 minutes ago, A.M. Hoornweg said:

Hello all,

 

one of my (heavily multithreaded) Windows services crashed today with an access violation and Windows was decent enough to write the "fault offset" in the event log.  
I have a detailed linker MAP file of the application.  Does anyone know a tool that will parse the MAP file to help me find the approximate error location in the source?

madExcept, EurekaLog, etc. are the best ways to do this. But obviously you don't have that yet, so you can just subtract $00401000 from the address and look up that value in the Publics by Value section of the map file.

[Lars Fosdal 1792]

Eurekalog is invaluable for services. 

Eliminates the guesswork.

Worth every €/$/£.

[A.M. Hoornweg 144]

 

33 minutes ago, Lars Fosdal said:

Eurekalog is invaluable for services. 

Eliminates the guesswork.

Worth every €/$/£.

My service is heavily multi-threaded (see attachment) , has hundreds of threads running which pull data from a multitude of oil rigs.
The communication threads use Remobjects Remoting which is also multithreaded. The service itself has a builtin http management console that is based on Intraweb, which is also a multithreaded framework.

 

 

There is sooo much multithreading going on, can a tool like Eurekalog produce a stack frame if one of these threads produces an AV?

 

 

 

[David Heffernan 2345]

17 minutes ago, A.M. Hoornweg said:

There is sooo much multithreading going on, can a tool like Eurekalog produce a stack frame if one of these threads produces an AV?

EurekaLog and madExcept can both do this. It's no big deal at all.

[Lars Fosdal 1792]

@A.M. Hoornweg
The Eurekalog stack trace is for thread where the problem occurs.

There also is an option to collect stacks from multiple threads at an access violation or exception, but it is usually not recommended since it is very resource intensive.

 

Our service is heavily multithreaded as well, doing database work, multi-connection TCP comms, as well as serving a web UI to multiple concurrent users.

[Vincent Parrett 750]

I haven't tried this, but if you have the map file, you could use map2pdb and then use windbg to figure out where the exception occurred. 

 

As @David Heffernan mentioned, madEcept or Eurekalog are the way to go - they are invaluable for getting usable stack traces, I've fixed countless bugs  in my code thanks to madExcept.   

[David Heffernan 2345]

2 hours ago, Vincent Parrett said:

I haven't tried this, but if you have the map file, you could use map2pdb and then use windbg to figure out where the exception occurred. 

Far quicker at this point to subtract $00401000 from the address (assuming it's an exe loaded at $00400000)

[Stefan Glienke 2002]

5 minutes ago, David Heffernan said:

Far quicker at this point to subtract $00401000 from the address (assuming it's an exe loaded at $00400000)

No need to assume or guess anything - the base addresses are in the map file.

[Remy Lebeau 1393]

29 minutes ago, Stefan Glienke said:

No need to assume or guess anything - the base addresses are in the map file.

The *preferred* base addresses are in the map file, as well as in exe/dll PE headers, but it is not guaranteed that exe/dll modules will actually use those base addresses at runtime (typically, they will, but the OS is free to move them somewhere else).  You can use the Win32 GetModuleHandle() function, or the RTL's SysInti.HInstance, to get the actual base address at runtime.

[Der schöne Günther 316]

We just set System.ExceptObjProc (and one or two others, I think) to point to our own exception handler and acquire a textual representation of the callstack with FastMM_Fulldebugmode.dll. Never saw the need for additional tools like Eurekalog or MadExcept.

 

As far as I recall, precompiled FastMM dll internally uses JCL for looking up the .map file, but can be recompiled to use Eureka or MadExcept as well.

[eivindbakkestuen 47]

On 4/30/2021 at 9:17 PM, A.M. Hoornweg said:

hundreds of threads running which pull data from a multitude of oil rigs.

Since this appears to be a low budget project, you could start using the free exception stack trace routines in the JCL library.

[A.M. Hoornweg 144]

Thanks  to everybody who responded in this thread.  I think I'll acquire Eurekalog or MadExcept for this project. Is any of these two particularly suited for multithreading NT services? 

 

 

[Lars Fosdal 1792]

Since I've not used MadExcept, I can't speak for that, but we use Eurekalog in several multithreaded NT services.

[Vincent Parrett 750]

I think you should evaluate both and decide which suits your needs best. I've mostly used madexcept but when I have used Eurekalog it's worked fine. 

[David Heffernan 2345]

12 hours ago, A.M. Hoornweg said:

Is any of these two particularly suited for multithreading NT services? 

Neither is more suited than the other for multi-threaded programs. I don't think your usage scenario is particularly challenging or unusual as you seem to think. 

[Remy Lebeau 1393]

19 hours ago, eivindbakkestuen said:

Since this appears to be a low budget project, you could start using the free exception stack trace routines in the JCL library.

Last time I tried that (last year), I couldn't get it to work.  So I had to write my own stack tracing code from scratch, which "works" but it doesn't display function names yet, only memory addresses relative to the process' base address.  And that project has since been EOL'ed, so I can't go back and update it.

[ioan 45]

I also have several multi threaded windows services and I'm using JCL Library to track exceptions. I'm very happy with it and never had problems. You'll find my code in this post:

 

Make sure to follow the instructions from the code comments.

 

[A.M. Hoornweg 144]

12 hours ago, ioan said:

I also have several multi threaded windows services and I'm using JCL Library to track exceptions. I'm very happy with it and never had problems. You'll find my code in this post:

 

Make sure to follow the instructions from the code comments.

 

This is very interesting, I'll certainly look into this. 

 

I'm also currently evaluating Eurekalog.

Edited May 5, 2021 by A.M. Hoornweg

[eivindbakkestuen 47]

On 5/4/2021 at 8:46 AM, Remy Lebeau said:

Last time I tried that (last year), I couldn't get it to work.  So I had to write my own stack tracing code from scratch, which "works" but it doesn't display function names yet, only memory addresses relative to the process' base address.  And that project has since been EOL'ed, so I can't go back and update it.

Not sure what is going on for you, it works well enough that we use it in NexusDB for the binaries we provide.

[aehimself 396]

I just want to make sure my logic is correct here. I built a really simple test, see .dpr and the generated .map file attached. I used this method to extract method addresses.

 

Stack trace was as following:

$0000000000429A84
$0000000000425327
$0000000000425C1B
$000000000040A5C6
$000000000040A601
$0000000000429BE5
$0000000000429C0D
$0000000000429C2D
$0000000000429C82
$00007FFD09E97034
$00007FFD0A7A2651

 

Now, from each address I substracted image base ($400000) and $1000, which resulted the following relative addresses:

 

$28A84
$24327
$24C1B
$95C6
$9601
$28BE5
$28C0D
$28C2D
$28C82
$7FFD09A96034
$7FFD0A3A1651

 

In the map file "Address - Publics by name" section I looked for the address which is the closest but below each relative address. This gives me the method names.

In the map file "Line numbers for xxx" section I searched for the exact relative address. This gives me the line numbers, if any.

 

Based on the above approach my reconstructed stack trace is the following:

 

StackTrace.GetExceptionStackInfo ( StackTrace:44 )
System.SysUtils.Exception.RaisingException ( ??? )
System.SysUtils.ExceptHandler ( System.SysUtils:23574 )
??? ( System.pas:22018 )
??? ( System.pas:22108 )
Project1.Three ( Project1:14 )
Project1.Two ( Project1:19 )
Project1.One ( Project1:24 )
Project1.Project1 ( ??? )
???
???

 

It resembles what went on, but I have some missing information which I was unable to find.

I have two questions... is this the way to reconstruct the call stack from a map file? And if yes, how I can find the missing information?

Project1.zip