MAP2PDB - Profiling with VTune

[Anders Melander 1782]

It took me a bit longer than expected to get here but I believe I've finally reached the goal.

The following shows VTune profiling a Delphi application, with symbol, line number and source code resolution:

 

Download

Get the source here: https://bitbucket.org/anders_melander/map2pdb/

And a precompiled exe here: https://bitbucket.org/anders_melander/map2pdb/downloads/

The source has only been tested with Delphi 10.3 - uses inline vars so it will not compile with older versions.

 

Usage

map2pdb - Copyright (c) 2021 Anders Melander
Version 2.0

Parses the map file produced by Delphi and writes a PDB file.

Usage: map2pdb [options] <map-filename>

Options:
  -v                         Verbose output
  -pdb[:<output-filename>]   Writes a PDB (default)
  -yaml[:<output-filename>]  Writes an YAML file that can be used with llvm-pdbutil
  -bind[:<exe-filename>]     Patches a Delphi compiled exe file to include a reference to the pdb file
  -test                      Works on test data. Ignores the input file

 

Example:

1. Configure your project linker options to output a Detailed map file.
2. Compile the project.
3. Execute map2pdb <map-filename> -bind
4. Profile the application with VTune (or whatever)

 

Known issues

• The -bind switch must occur after the filename contrary to the usage instructions.
• PDB files larger than 16Mb are not valid. This is currently by design.
• 64-bit PE files are not yet supported by the -bind option.

 

As should be evident I decided not to go the DWARF route after all. After using a few days to read the DWARF specification and examine the FPC source I decided that it would be easier to leverage the PDB knowledge I had already acquired. Not that this has been easy. Even though I've been able to use the LLVM PDB implementation and Microsoft's PDB source as a reference LLVM's implementation is incomplete and buggy and the LLVM source  is "modern C++" which means that it's close to unreadable in places. Microsoft's source, while written in clean C and guaranteed to be correct, doesn't compile and is poorly commented. Luckily it was nothing a few all-nighters with a disassembler and a hex editor couldn't solve.

 

Enjoy!

Edited April 3, 2021 by Anders Melander

[David Heffernan 2345]

Amazing work. Well done!! Thank you. 

 

Although it can't be compiled in older versions, presumably it works with programs built with older versions of Delphi? And do you have plans to support 64 bit? 

[Attila Kovacs 629]

1 hour ago, Anders Melander said:

• PDB files larger than 16Mb are not valid. This is currently by design.

okay, my first try was 26Mb pdb. Where is the problem with larges sizes as 16Mb?

 

 

[58555] Failed to resolve symbol to module: 0005:00000098 SysInit.TlsLast
[58557] Failed to resolve symbol to module: 0004:FE72B000 SysInit.__ImageBase

 

Is that bad?

 

exe was not patched, not sure if because of the messages above or because of the 26Mb limit, no real information.

 

edit: ok, tested with another project, same messages, deleted the lines from the map, exe was patched fine with pure -bind and exe was not ptached with -bind:filename.exe

 

 

Edited April 3, 2021 by Attila Kovacs

[Anders Melander 1782]

1 hour ago, David Heffernan said:

presumably it works with programs built with older versions of Delphi?

Yes it should. As long as the layout of the map file is the same.

It's easy enough to fix if there's minor differences but I don't think there are.

 

1 hour ago, David Heffernan said:

And do you have plans to support 64 bit?

Yes. I need it for 64-bit myself.

The only thing needed to support for 64-bit is the ability to parse and modify a PE64 image. Right now I don't what the difference is between PE32 and PE64.

Here's the relevant code:

https://bitbucket.org/anders_melander/map2pdb/src/27bb0daa1b4a7e0159b646f4b0cd0d34ffc72fd3/Source/debug.info.pdb.

 

[Anders Melander 1782]

52 minutes ago, Attila Kovacs said:

Where is the problem with larges sizes as 16Mb?

The problem is the MSF container format.

It's a blocked format with 4096 bytes in each block and up to 4096 blocks in each "interval" (4096*4096 = 16Mb). Each interval starts with a Free Page Map (FPM, a bit like a FAT).

 

Since I wanted to concentrate on getting the PDB layout right my MSF abstraction doesn't implement intervals so once I pass 4096 blocks the file becomes invalid.

Here's LLVMs explanation of the MSF format: https://llvm.org/docs/PDB/MsfFile.html

 

A proper implementation of intervals will require a bit of work. At present I can assume that a physical stream offset value equals the logical offset value. Once I add intervals I will have to take into the FPMs into account since the blocks are no longer contiguous. Also when I write I will have to consider that a write can start on once side of the FPMs and continue on the other side of them. It isn't rocket science but it does complicate the IO layer considerably.

 

1 hour ago, Attila Kovacs said:

[58555] Failed to resolve symbol to module: 0005:00000098 SysInit.TlsLast
[58557] Failed to resolve symbol to module: 0004:FE72B000 SysInit.__ImageBase

Is that bad?

No. I get those too.

I'm not sure what they are for but they are in segment 4 (TLS) and 5 (PDATA= exception data) so you'll not miss them.

 

The warning just means that the address of a symbol couldn't be matched to any of the modules (units).

 

1 hour ago, Attila Kovacs said:

exe was not patched, not sure if because of the messages above or because of the 26Mb limit, no real information.

It should be. How do you come to the conclusion that it wasn't patched?

Try the -v switch.

 

1 hour ago, Attila Kovacs said:

exe was patched fine with pure -bind and exe was not ptached with -bind:filename.exe

Okay. There's probably a bug in the command line interface then. I discovered the problem with the position of the -bind switch just before release so there's probably more.

[Stefan Glienke 2002]

Very nice! - Just a thing about the map file parsing - and actually that's an issue that I've seen with most map file analyzing tools as they do it wrong - relating symbols from generics to the correct type and unit - example:

 

Spring.{System.Generics.Collections}TDictionary<System.Pointer,System.Classes.TList>.GetBucketIndex

 

This means the code for this method because its a generic was compiled into the Spring.dcu because I used that TDictionary from the RTL in Spring.pas - but the source will not be found there but in the unit in the curly brackets: System.Generics.Collections

Edited April 3, 2021 by Stefan Glienke

[Anders Melander 1782]

26 minutes ago, Stefan Glienke said:

Very nice! - Just a thing about the map file parsing - and actually that's an issue that I've seen with most map file analyzing tools as they do it wrong - realating symbols to the correct file - example:

 

Spring.{System.Generics.Collections}TDictionary<System.Pointer,System.Classes.TList>.GetBucketIndex

 

This means the code for this method because its a generic was compiled into the Spring.dcu because that TDictionary in Spring.pas - but the source will not be found there but in the unit in the curly brackets: System.Generics.Collections

Thanks. Naturally, as something like that always is, the map parser is a mess

 

However I'm not actually associating symbols with source files. I'm associating symbols with modules (i.e. units) based on the symbol address and the module address range.

So for example:

 0001:001236E0       System.Classes.{System.Generics.Collections}TDictionary<System.Integer,System.Classes.IInterfaceList>.Add

is associated with System.Classes because the offset $001236E0 locates it in the range of that module: $000C7730 - $0015F654 ($000C7730+$0009F24):

 0001:000C7730 00097F24 C=CODE     S=.text    G=(none)   M=System.Classes ALIGN=4

 

Later when I parse the line number information I get the correct source file and can associate that, source file and line number+offset, with the module. I.e. System.Generics.Collection.pas in the module SystemClasses.

Line numbers for System.Classes(System.Generics.Collections.pas) segment .text

  7088 0001:001236E0  7089 0001:001236F1  7090 0001:001236F9  7092 0001:00123701
  7093 0001:0012370E  7094 0001:0012371B  7095 0001:0012371F  7097 0001:0012373C
  7098 0001:00123755

This way you get both the symbol resolved to the correct module/unit and the code resolved to the correct source lines.

[Stefan Glienke 2002]

All I can say is that it did not work properly for me - when clicking on the method I mentioned above it wanted to open Spring.pas and marked an obviously completely wrong line.

[Anders Melander 1782]

51 minutes ago, Anders Melander said:

Okay. There's probably a bug in the command line interface then. I discovered the problem with the position of the -bind switch just before release so there's probably more.

Fixed.

https://bitbucket.org/anders_melander/map2pdb/commits/8c94db07a2da1b503ce6f00c39792ee0b5479387

[Vincent Parrett 750]

I was seriously excited about this, until I realized I don't really know how to drive vtune 🙄 (terrible user interface!). 

 

So I was able to generate a pdb (9.5MB) and it appears to bind, however when running with vtune  I don't see function names

CPU Time
1 of 1: 100.0% (0.010s of 0.010s)

FinalBuilder9.exe ! func@0x77c258 - [unknown source file]
FinalBuilder9.exe ! func@0x4de4bf + 0xca - [unknown source file]
FinalBuilder9.exe ! func@0x4ea8f8 + 0x176 - [unknown source file]
FinalBuilder9.exe ! func@0x4ea518 + 0x10c - [unknown source file]
FinalBuilder9.exe ! func@0x4ea8f8 + 0xf5 - [unknown source file]
FinalBuilder9.exe ! func@0x4ea518 + 0x10c - [unknown source file]
FinalBuilder9.exe ! func@0x4ea8f8 + 0xf5 - [unknown source file]
FinalBuilder9.exe ! func@0x4ea518 + 0x10c - [unknown source file]
FinalBuilder9.exe ! func@0x4ea8f8 + 0xf5 - [unknown source file]
FinalBuilder9.exe ! func@0x4ea518 + 0x10c - [unknown source file]
.....

 

How can I see if vtune is actually using the pdb? 

 

Also I'm using runtime packages, however when I tried to create a pdb for packages  map2pdb crashed

 

E:\map2pdb>map2pdb.exe I:\FBAT_HG9\Output\FB9\VSoft.Core.map -bind:I:\FBAT_HG9\Output\FB9\VSoft.Core.bpl -v
map2pdb - Copyright (c) 2021 Anders Melander
Version 2.0

Output filename not specified. Defaulting to VSoft.Core.pdb
Reading MAP file
- Segments
- Modules
An error occurred in the application.

date/time          : 2021-04-04, 08:53:18, 171ms
computer name      : SR22
user name          : vincent
registered owner   : Vincent
operating system   : Windows 10 x64 build 19042
system language    : English
system up time     : 42 minutes 16 seconds
program up time    : 185 milliseconds
processors         : 8x Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
physical memory    : 52813/65477 MB (free/total)
free disk space    : (C:) 78.24 GB (E:) 233.56 GB
display mode       : 2560x1440, 32 bit
process id         : $55d0
allocated memory   : 118.39 MB
largest free block : 1.47 GB
command line       : map2pdb.exe I:\FBAT_HG9\Output\FB9\VSoft.Core.map -bind:I:\FBAT_HG9\Output\FB9\VSoft.Core.bpl -v
executable         : map2pdb.exe
exec. date/time    : 2021-04-04 08:00
compiled with      : Delphi 10.3 Rio
madExcept version  : 5.1.0
callstack crc      : $7846b8db, $f263fd00, $f263fd00
exception number   : 1
exception class    : EArgumentOutOfRangeException
exception message  : Argument out of range.

thread $5a5c:
0046c541 +029 map2pdb.exe  debug.info                    TDebugInfoSegments.FindByIndex
0047e90f +58b map2pdb.exe  debug.info.reader.map         TDebugInfoMapReader.LoadFromStream
0047d844 +034 map2pdb.exe  debug.info.reader             TDebugInfoReader.LoadFromFile
004aa13e +2e6 map2pdb.exe  map2pdb               140 +62 initialization
76dffa27 +017 KERNEL32.DLL                               BaseThreadInitThunk

thread $5c8:
76dffa27 +17 KERNEL32.DLL  BaseThreadInitThunk

thread $55d8:
76dffa27 +17 KERNEL32.DLL  BaseThreadInitThunk

thread $5b2c:
76dffa27 +17 KERNEL32.DLL  BaseThreadInitThunk

thread $4fac:
76dffa27 +17 KERNEL32.DLL  BaseThreadInitThunk

thread $5bc0:
76dffa27 +17 KERNEL32.DLL  BaseThreadInitThunk

 

[Attila Kovacs 629]

24 minutes ago, Vincent Parrett said:

I don't see function names

me neither, everything func@addr

there must be something with the pdb as I can't even use it in a debugger, I'll try with simpler ones

Edited April 3, 2021 by Attila Kovacs

[Anders Melander 1782]

18 minutes ago, Vincent Parrett said:

How can I see if vtune is actually using the pdb? 

Sometimes it writes in the log if it can't find the pdb. Sometimes it doesn't...

I guess Intel are focusing their efforts on making hardware

 

I had to use Process Monitor to verify that it actually read the pdb file. That was also the way I discovered that it didn't look for the pdb unless there was a reference to it in the exe file.

 

28 minutes ago, Vincent Parrett said:

Also I'm using runtime packages, however when I tried to create a pdb for packages  map2pdb crashed

I think I will need the map file to determine the cause of that. You can mail it to me if that's ok.

I haven't thought about dlls at all but I guess there shouldn't be any difference.

 

[Anders Melander 1782]

27 minutes ago, Attila Kovacs said:

me neither

everything func@addr

Can you try with a tiny console application. I've used the following for test:

program test;

{$APPTYPE CONSOLE}

{$R *.res}

uses
  System.Classes,
  System.SysUtils;

  procedure WasteSomeTime;
  begin
    var Strings := TStringList.Create;
    try

      for var i := 1 to 100000 do
        Strings.Add(i.ToString);

      Strings.Sort;

    finally
      Strings.Free;
    end;
  end;

begin
  try

    for var i := 1 to 10 do
    begin
      Writeln(i.ToString);
      WasteSomeTime;
    end;

  except
    on E: Exception do
      Writeln(E.ClassName, ': ', E.Message);
  end;
end.

 

If that doesn't work either then please zip the map file + exe produced by the above code and post it here or at the issue tracker on bitbucket.

[Vincent Parrett 750]

I have pm'd the map file, I didn't have your email address handy. 

 

[Vincent Parrett 750]

I suspect this has to do with the compiler version, using 10.4.2 even the test console app doesn't show function names. 

Project3.zip

Edited April 3, 2021 by Vincent Parrett

[Attila Kovacs 629]

@Anders Melander hohooo!!! The test project's pdb just works fine with OllyDbg!!! Is that cool! I'd contribute with pleasure to get that right with large projects too but I can't deal with inline vars.

[Vincent Parrett 750]

Well I tried the same project with 10.3 and got the same result, so it's not the compiler version. 

 

I'm using Vtune 2021.1.1 Gold - perhaps it's the vtune version? 

[Vincent Parrett 750]

I compiled map2pdb from source using 10.4.2 and vtune now loads the pdb from the test project. Sadly it didn't work for my real application

 

Cannot locate debugging information for file 'I:\FBAT_HG9\Output\FB9\FinalBuilder9.exe'. Cannot match the module with the symbol file 'I:\FBAT_HG9\Output\FB9\FinalBuilder9.pdb'. Make sure to specify the correct path to the symbol file in the Binary/Symbol Search list of directories.  

 

[Anders Melander 1782]

14 hours ago, Vincent Parrett said:

I have pm'd the map file, I didn't have your email address handy. 

The problem with your map file was just that my validation was a bit too aggressive. I've now turned some of the errors into warnings.

New version uploaded.

 

One of the problems was that I discarded zero size segments. E.g. the TLS segment:

 0005:00000000 00000000H .tls                    TLS

which meant that later on when a module or symbol tried to resolve segment #5 that turned into an error.

Detailed map of segments
 0005:00000000 00000100 C=TLS      S=.tls     G=(none)   M=OtlCommon.Utils ACBP=A9

  Address             Publics by Name
 0005:00000000       OtlCommon.Utils.LastThreadName
 0005:00000100       SysInit.TlsLast

I'm not not really sure of how I should handle these out of bounds situations. Right now I just warn about them and then ignore the module/symbol/line.

 

 

14 hours ago, Vincent Parrett said:

I suspect this has to do with the compiler version, using 10.4.2 even the test console app doesn't show function names. 

Project3.zip

I think you just forgot the -bind switch. Your project3.exe did not contain a reference to project3.pdb.

After I run map2pdb -bind project3,map I got VTune to resolve with your exe.

 

 

11 hours ago, Vincent Parrett said:

Sadly it didn't work for my real application

You're probably hitting the 16Mb limit,

I'm now outputting a warning so we're aware of it when it happens.

 

I was planning on adding support for PE32+ (64-bit) next but I guess solving the MSF/16Mb problem is more important. I can't really use it myself for anything serious until that's done.

[Attila Kovacs 629]

@Anders Melander did you update the exe? can I try it?

[Anders Melander 1782]

1 minute ago, Attila Kovacs said:

did you update the exe? can I try it?

Yes - and yes

[Anders Melander 1782]

15 hours ago, Stefan Glienke said:

All I can say is that it did not work properly for me - when clicking on the method I mentioned above it wanted to open Spring.pas and marked an obviously completely wrong line.

I don't think there's enough information in the map file to solve that problem properly but I'll investigate it some more when I've gotten these other things out of the way.

[Attila Kovacs 629]

16 minutes ago, Anders Melander said:

Right now I just warn about them and then ignore the module/symbol/line.

As for the lines, if they are empty (xxx:00000000) there is either no code in the unit or the unit was in the project but not used anywhere, thus it's not linked to the exe.

At least these are my observations. Sadly still no luck with a bigger project. OllyDbg loads the pdb without yielding any error, but can't really use it. Same in VTunes, obviously.

[Attila Kovacs 629]

@Anders Melander 

 

this tool works on the pdb of the test app but does not show anything on a bigger project

 

https://github.com/Microsoft/microsoft-pdb/blob/master/cvdump/cvdump.exe

 

there is somewhere just a tiny glitch I'm sure

[Guest]

48 minutes ago, Anders Melander said:

I'm not not really sure of how I should handle these out of bounds situations. Right now I just warn about them and then ignore the module/symbol/line.

You can fix this by finding the address (offset) from the PE header itself, don't depend on the section index number but use the section/segment name.

Also you can see why i called the start addresses (offests) by Delphi linker map file are wrong and should be relative to 0 instead of $400000.