MAP2PDB - Profiling with VTune

[Laurens 1]

This is a really nice topic and the first PDB-tool I have found that works fast. I am trying to get debugging working in VS Code for Win32 using LLDB. The debugger runs, but does not shows the correct callstack/symbols and I hoped to fix this using the PDB-output from this tool. However, I cannot get it working yet. Has anybody here tried this?

[Laurens 1]

Last night and this morning I continued working on this and I have some more information that hopefully can give some insights. Using LLDB and after running `map2pdb.exe -exclude:system*;dx*;cx* -bind Tester.map` I see that the stacktrace in my code always looks the same:

(lldb) thread backtrace 1
  thread #1
    frame #0: 0x7565113c win32u.dll`NtUserWaitMessage + 12
    frame #1: 0x00a605e6 Tester.exe`__dbk_fcall_wrapper + 2927526
    frame #2: 0x00a5f5b8 Tester.exe`__dbk_fcall_wrapper + 2923384
    frame #3: 0x02791c90 Tester.exe`__dbk_fcall_wrapper + 33538640
    frame #4: 0x02791cf7 Tester.exe`__dbk_fcall_wrapper + 33538743
    frame #5: 0x03e092d2 Tester.exe`__dbk_fcall_wrapper + 57096338
    frame #6: 0x03e0931f Tester.exe`__dbk_fcall_wrapper + 57096415
    frame #7: 0x05753071 Tester.exe`Tester + 93
    frame #8: 0x76d07d59 kernel32.dll`BaseThreadInitThunk + 25
    frame #9: 0x779ab74b ntdll.dll`RtlInitializeExceptionChain + 107
    frame #10: 0x779ab6cf ntdll.dll`RtlClearBits + 191

LLDB always refers to `__dbk_fcall_wrapper`, but not to the actual method. I also noticed that running the same program in WinDBG, gives no stackstrackes at all. Running in WinDBG with a DBG-file created with `map2dbg` gives stacktraces with correct method references. (Although the sourcecode references do not work yet).

 

I really love to use LLDB (from VS Code, because the editor performs so much better for large projects than the Delphi IDE) and I think we can get close to get it working, but correct callstacks and method/source references are absolute necessary for this. I hope someone can put me in the right direction.

[Allen@Grijjy 44]

I am little late to this party, but stumbled across this wonderful utility, so thank you for it!  At first map2pdb didn't consume my map files without throwing some exceptions, but I made a few code changes and it works wonderfully now.  I was able to use it with Intel VTune which is really nice!  The issues I encountered was that it complained about 'overlapping segments' between .tls and the .data segments.  It also wouldn't load source lines if I use -include:0001 or -include:0001;0002;etc..

 

I was able to modify the sources to exclude the .tls segment and if I changed to -include:* using a wildcard then it properly loaded all the source lines from the various segments.  Anyway, thanks for this!

[Anders Melander 1782]

I have just released version 3.1.2 with the following changes since 3.0.1:

• map2pdb can now consume JEDI jdbg-files.
• A rare overflow bug in the MSF writer has been fixed.
• The map parser can now handle the slightly different files produced by beta versions of Delphi.

The big change here is the ability to create pdb-files from jdbg-files. You can thank Stefan for being so annoying that I finally caved in and implemented it to get him to shut up about it.

 

This means that it is now possible to profile Delphi's run-time packages by converting the jdbg-files bundled with Delphi and binding the produced pdb-files to the bpl-files.
It's as easy as map2pdb -bind:rtl290.bpl rtl290.jdbg

 

Because of a bug in the JEDI tool Embarcadero uses to convert from map to jdbg, some symbol names produced from Embarcadero's jdbg-files may look a bit strange. It's a minor issue that has no impact on the functionality and there's nothing I can do about it since the original map files aren't available.

 

Get it while it's hot:

https://bitbucket.org/anders_melander/map2pdb/downloads/

 

Here's an example from Stefan showing an application using the rtl290.bpl run-time package being profiled with VTune:

[Vincent Parrett 750]

19 minutes ago, Anders Melander said:

map2pdb can now consume JEDI jdbg-files

This is awesome, embarcadero have no excuse for poor performance now!

 

19 minutes ago, Anders Melander said:

You can thank Stefan for being so annoying that I finally caved in and implemented it to get him to shut up about it.

🤣Nice one 

 

As someone who uses runtime packages (due to the plug architecture of my application) - this is a god send - I have profiled the crap out of my code, found a lot of small improvements and a couple of large ones - but quite often the profiler shows the majority of the per ends up somewhere inside the RTL - this will allow me to do another round of profiling. The only down side is I have an amd cpu, and uProf is nowhere near as good as VTune (although both suck when it comes to UI/UX).

Edited November 12, 2023 by Vincent Parrett
addition

[hukmac 5]

Dear Anders Melander. Thank you for your time developing map2pdb project. In my case it works great with Delphi 11 Community Edition and VTune 2024.0.1. It is a very useful tool 🙂

[Wagner Landgraf 43]

Has anybody tried to use VTune inside a VM with M1 (ARM MAC)?

I recall I was able to use it, but I rebuilt my VM and now I can't make it work. I get this message:

 

[Instrumentation Engine]: [tid:15552] Failed to allocate Injector, Error = INJECTOR_ERR Pin is exiting due to fatal error

 

I just couldn't find anything relevant about this issue when googling for it. Even the VTune matrix demo doesn't work (so not related to map2pdb). 

[Anders Melander 1782]

1 hour ago, Wagner Landgraf said:

Has anybody tried to use VTune inside a VM with M1 (ARM MAC)?

I recall I was able to use it, but I rebuilt my VM and now I can't make it work.

VTune only supports Intel hardware as it relies on certain CPU features that are only available on Intel CPUs. At least that what they claim:

https://www.intel.com/content/www/us/en/developer/articles/system-requirements/vtune-profiler-system-requirements.html

 

Maybe you can get an older version of VTune to work. For example the current version of VTune doesn't support hardware assisted profiling on my (admittedly pretty old) processor.

[Wagner Landgraf 43]

46 minutes ago, Anders Melander said:

Maybe you can get an older version of VTune to work.

Probably that's what is happening here. I had an older VTune version that worked, and now I installed Tuen 2024 which doesn't. (Of course, "working" was very limited, it didn't support any hardware assisted profiling, but at least the Hotspots with no hardware was working, just to profile application logic).

 

The problem is: where the heck to I find old VTune versions to install? I search everywhere, and I can't find any information. Closes I found was to register for Intel account and go to some download/registration center, but nothing is displayed there are it only lists "registered" products.

 

Does anyone happen to have an old offline installer of 2023, maybe 2022 VTune?

Edited February 24 by Wagner Landgraf

[Anders Melander 1782]

1 minute ago, Wagner Landgraf said:

where the heck to I find old VTune versions to install

There used to be a link to download previous versions (which is how I managed to use it with Windows 7 at that time), but apparently they've removed that ability:

https://community.intel.com/t5/Analyzers/where-can-I-download-an-older-version-vtune/m-p/1561574#M24281

 

Quote

Only Customer with access to priority support can download the older version of tools. Otherwise Intel provides the latest and greatest version for the public use.

😞

[Willicious 8]

Hi, can anyone help me make sense of the following:

Quote

• Configure your project linker options to output a Detailed map file.

In RAD 10.4.2, where can I find project linker options? And, do these expressly give the option to "output a detailed map file"?

Quote

• Execute map2pdb <map-filename> -bind

As in, from a command line? From within RAD? Do I need to type exactly what's here, or replace with a specific filename?

Quote

• Profile the application with VTune (or whatever)

So, I'm guessing this is done by running VTune whilst the application is running? Perhaps the other two steps are what link the project, so... all I need to do is run VTune?

 

Oh, also - how can I get VTune? 

 

 

Edited June 10 by Willicious

[Willicious 8]

Replying again because I forgot to notify

[Anders Melander 1782]

3 minutes ago, Willicious said:

where can I find project linker options? And, do these expressly give the option to "output a detailed map file"?

 

A google search would have told you that. Or the IDE insight:

 

or the help...

 

6 minutes ago, Willicious said:

As in, from a command line? From within RAD? Do I need to type exactly what's here, or replace with a specific filename?

Command line. Replace <map-filename> with the name of your map file.

 

7 minutes ago, Willicious said:

So, I'm guessing this is done by running VTune whilst the application is running?

Run VTune, configure your project there, launch the application from within VTune. But before you do anything, please read some VTune documentation.

 

5 minutes ago, Willicious said:

Replying again because I forgot to notify

You don't need to post anything to get notifications. Just click the Following dropdown.

[Willicious 8]

Thanks 🙂

[Jan Rysavy 8]

Hi,

Do we have any information about x64 RTL debug symbols under Delphi for Windows? For x86, MAP2PDB now supports reading from C:\Program Files (x86)\Embarcadero\Studio\23.0\bin\rtl290.jdbg, which is fantastic. However, for x64, this file doesn't seem to be present in the Delphi 11 or Delphi 12 distributions. I checked the Modules window in the debugger, and it appears that the debug symbols are being read from C:\Program Files (x86)\Embarcadero\Studio\23.0\lib\win64\debug\rtl.dcp. Do we have any details on the DCP structure, and is there any possibility of generating a .MAP file from it?

Thanks!

 

Edited August 23 by Jan Rysavy

[Stefan Glienke 2002]

Recompile the RTL yourself using source\rtl\buildrtl.bat and generate map file - that's how I used to do it until Anders added jdbg support.

[Anders Melander 1782]

36 minutes ago, Jan Rysavy said:

Do we have any details on the DCP structure

Not that I know of. The file format is completely undocumented and there's no known API to extract info from it.

 

A bit of googling found these:

• https://www.delphipraxis.net/48587-dcp-format.html
• http://hmelnov.icc.ru/DCU/
• https://gitlab.com/dcu32int/DCU32INT/-/blob/master/DCP.pas?ref_type=heads

The DCU32INT project looks like it could be a stepping stone.

[Jan Rysavy 8]

1 hour ago, Stefan Glienke said:

Recompile the RTL yourself using source\rtl\buildrtl.bat and generate map file - that's how I used to do it until Anders added jdbg support.

This works fine, but we also need debug symbols for vcl290.bpl and vclimg290.bpl, whose projects are missing in the distribution.

[Stefan Glienke 2002]

you can generate a dpk with the help of System.SysUtils.GetPackageInfo

[Jan Rysavy 8]

2 hours ago, Anders Melander said:

• https://www.delphipraxis.net/48587-dcp-format.html
• http://hmelnov.icc.ru/DCU/
• https://gitlab.com/dcu32int/DCU32INT/-/blob/master/DCP.pas?ref_type=heads

Also: https://wiert.me/2020/10/07/reverse-engineering-delphi-and-turbo-pascal-unit-interfaces/