Jump to content
A.M. Hoornweg

Find exception location from MAP file?

Recommended Posts

Hello all,

 

one of my (heavily multithreaded) Windows services crashed today with an access violation and Windows was decent enough to write the "fault offset" in the event log.  
I have a detailed linker MAP file of the application.  Does anyone know a tool that will parse the MAP file to help me find the approximate error location in the source?

Share this post


Link to post
3 minutes ago, A.M. Hoornweg said:

Hello all,

 

one of my (heavily multithreaded) Windows services crashed today with an access violation and Windows was decent enough to write the "fault offset" in the event log.  
I have a detailed linker MAP file of the application.  Does anyone know a tool that will parse the MAP file to help me find the approximate error location in the source?

madExcept, EurekaLog, etc. are the best ways to do this. But obviously you don't have that yet, so you can just subtract $00401000 from the address and look up that value in the Publics by Value section of the map file.

Share this post


Link to post

Eurekalog is invaluable for services. 

Eliminates the guesswork.

Worth every €/$/£.

Share this post


Link to post

 

33 minutes ago, Lars Fosdal said:

Eurekalog is invaluable for services. 

Eliminates the guesswork.

Worth every €/$/£.

My service is heavily multi-threaded (see attachment) , has hundreds of threads running which pull data from a multitude of oil rigs.
The communication threads use Remobjects Remoting which is also multithreaded. The service itself has a builtin http management console that is based on Intraweb, which is also a multithreaded framework.

 

 

There is sooo much multithreading going on, can a tool like Eurekalog produce a stack frame if one of these threads produces an AV?

 

 

 

wellsync.png

Share this post


Link to post
17 minutes ago, A.M. Hoornweg said:

There is sooo much multithreading going on, can a tool like Eurekalog produce a stack frame if one of these threads produces an AV?

EurekaLog and madExcept can both do this. It's no big deal at all.

Share this post


Link to post

@A.M. Hoornweg
The Eurekalog stack trace is for thread where the problem occurs.

There also is an option to collect stacks from multiple threads at an access violation or exception, but it is usually not recommended since it is very resource intensive.

 

Our service is heavily multithreaded as well, doing database work, multi-connection TCP comms, as well as serving a web UI to multiple concurrent users.

Share this post


Link to post

I haven't tried this, but if you have the map file, you could use map2pdb and then use windbg to figure out where the exception occurred. 

 

As @David Heffernan mentioned, madEcept or Eurekalog are the way to go - they are invaluable for getting usable stack traces, I've fixed countless bugs :classic_blush: in my code thanks to madExcept.   

  • Like 1

Share this post


Link to post
2 hours ago, Vincent Parrett said:

I haven't tried this, but if you have the map file, you could use map2pdb and then use windbg to figure out where the exception occurred. 

Far quicker at this point to subtract $00401000 from the address (assuming it's an exe loaded at $00400000)

Share this post


Link to post
5 minutes ago, David Heffernan said:

Far quicker at this point to subtract $00401000 from the address (assuming it's an exe loaded at $00400000)

No need to assume or guess anything - the base addresses are in the map file.

Share this post


Link to post
29 minutes ago, Stefan Glienke said:

No need to assume or guess anything - the base addresses are in the map file.

The *preferred* base addresses are in the map file, as well as in exe/dll PE headers, but it is not guaranteed that exe/dll modules will actually use those base addresses at runtime (typically, they will, but the OS is free to move them somewhere else).  You can use the Win32 GetModuleHandle() function, or the RTL's SysInti.HInstance, to get the actual base address at runtime.

Share this post


Link to post

We just set System.ExceptObjProc (and one or two others, I think) to point to our own exception handler and acquire a textual representation of the callstack with FastMM_Fulldebugmode.dll. Never saw the need for additional tools like Eurekalog or MadExcept.

 

As far as I recall, precompiled FastMM dll internally uses JCL for looking up the .map file, but can be recompiled to use Eureka or MadExcept as well.

Share this post


Link to post
On 4/30/2021 at 9:17 PM, A.M. Hoornweg said:

hundreds of threads running which pull data from a multitude of oil rigs.

Since this appears to be a low budget project, you could start using the free exception stack trace routines in the JCL library.

Share this post


Link to post

Thanks  to everybody who responded in this thread.  I think I'll acquire Eurekalog or MadExcept for this project. Is any of these two particularly suited for multithreading NT services? 

 

 

Share this post


Link to post

Since I've not used MadExcept, I can't speak for that, but we use Eurekalog in several multithreaded NT services.

Share this post


Link to post

I think you should evaluate both and decide which suits your needs best. I've mostly used madexcept but when I have used Eurekalog it's worked fine. 

Share this post


Link to post
12 hours ago, A.M. Hoornweg said:

Is any of these two particularly suited for multithreading NT services? 

Neither is more suited than the other for multi-threaded programs. I don't think your usage scenario is particularly challenging or unusual as you seem to think. 

Share this post


Link to post
19 hours ago, eivindbakkestuen said:

Since this appears to be a low budget project, you could start using the free exception stack trace routines in the JCL library.

Last time I tried that (last year), I couldn't get it to work.  So I had to write my own stack tracing code from scratch, which "works" but it doesn't display function names yet, only memory addresses relative to the process' base address.  And that project has since been EOL'ed, so I can't go back and update it.

Share this post


Link to post

I also have several multi threaded windows services and I'm using JCL Library to track exceptions. I'm very happy with it and never had problems. You'll find my code in this post:

 

Make sure to follow the instructions from the code comments.

 

Share this post


Link to post
12 hours ago, ioan said:

I also have several multi threaded windows services and I'm using JCL Library to track exceptions. I'm very happy with it and never had problems. You'll find my code in this post:

 

Make sure to follow the instructions from the code comments.

 

This is very interesting, I'll certainly look into this. 

 

I'm also currently evaluating Eurekalog.

Edited by A.M. Hoornweg

Share this post


Link to post
On 5/4/2021 at 8:46 AM, Remy Lebeau said:

Last time I tried that (last year), I couldn't get it to work.  So I had to write my own stack tracing code from scratch, which "works" but it doesn't display function names yet, only memory addresses relative to the process' base address.  And that project has since been EOL'ed, so I can't go back and update it.

Not sure what is going on for you, it works well enough that we use it in NexusDB for the binaries we provide.

Share this post


Link to post

I just want to make sure my logic is correct here. I built a really simple test, see .dpr and the generated .map file attached. I used this method to extract method addresses.

 

Stack trace was as following:

$0000000000429A84
$0000000000425327
$0000000000425C1B
$000000000040A5C6
$000000000040A601
$0000000000429BE5
$0000000000429C0D
$0000000000429C2D
$0000000000429C82
$00007FFD09E97034
$00007FFD0A7A2651

 

Now, from each address I substracted image base ($400000) and $1000, which resulted the following relative addresses:

 

$28A84
$24327
$24C1B
$95C6
$9601
$28BE5
$28C0D
$28C2D
$28C82
$7FFD09A96034
$7FFD0A3A1651

 

In the map file "Address - Publics by name" section I looked for the address which is the closest but below each relative address. This gives me the method names.

In the map file "Line numbers for xxx" section I searched for the exact relative address. This gives me the line numbers, if any.

 

Based on the above approach my reconstructed stack trace is the following:

 

StackTrace.GetExceptionStackInfo ( StackTrace:44 )
System.SysUtils.Exception.RaisingException ( ??? )
System.SysUtils.ExceptHandler ( System.SysUtils:23574 )
??? ( System.pas:22018 )
??? ( System.pas:22108 )
Project1.Three ( Project1:14 )
Project1.Two ( Project1:19 )
Project1.One ( Project1:24 )
Project1.Project1 ( ??? )
???
???

 

It resembles what went on, but I have some missing information which I was unable to find.

I have two questions... is this the way to reconstruct the call stack from a map file? And if yes, how I can find the missing information?

Project1.zip

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×