Anders Melander

New version (2.5) uploaded. Changes since last upload: Include/exclude modules/units from pdb. This helps keep the size of the pdb down and thus reduces the symbol resolve time in VTune. You no longer need to link your projects with debug info. map2pdb will reuse the existing debug section in the exe/dll/bpl if there is one. Otherwise it will create a new one. https://bitbucket.org/anders_melander/map2pdb/downloads/ What's next: Refactoring of the logging code. The current logging is basically just some functions that calls WriteLn. This should be replaced with a pluggable log framework so the whole logging mechanism can be replaced. The end goal is to enable integration of the map2pdb core into other projects. A jdbg reader. Embarcadero does not supply map files for the RTL/VCL rune time packages. Instead they ship jdbg files that can be read with the JEDI debug functions. The jdbg are built from map files so supposedly they contains much, if not all, of the information we need. The task here is to write a reader for the jdbg file format so we can produce pdb files from them. Figure out why VTune is so slow. A never ending task it seems.

Ah Yes, that too. Not insignificant but once you know, you know.

While I completely agree with David's considerations it should be noted that the startup cost of an in-process COM server is considerably smaller than that of of an out-of-process COM server. For an in-process server it's basically just the cost of setting up the COM apartment. For an out-of-process server there's also the cost of launching the process. In the cases where I have chosen to use COM for interop with .NET it has mostly been because of the convenience of not having to worry about marshalling and because it's so easy to just wrap a Delphi object in an interface/TAutoIntfObject and pass that on to .NET

Too bad. It looks very nice. I guess I'll take a look at it when my VTune trial expires.

Any clues as to what goes wrong?

Yes, I think it is. It's been a while since I did it though so I could be wrong. Maybe start by reading the help http://docwiki.embarcadero.com/RADStudio/Sydney/en/Developing_Interoperable_Applications_Using_COM

Let's start with the map file. Zip it and PM it to me.

Just write your DLL as an in-process COM server (or out-of-process if that suits your need better). .NET can use that directly.

I don't know about C++Builder and map files but in any case there would be too many differences, caused by all the other stuff that VS outputs, to make that feasible. What I've done previously, when I had to figure out why something didn't work, was to use a hex editor to compare the pdb of VTune's matrix example with the output from "map2pdb -test". They are both sufficiently small. At this time I can pretty much parse pdb just by looking at the hex 🙂 I think the easiest way forward would be to just examine the key suspects (address table, hash tables) of matrix.pdb in a hex editor and verify that they're ordered and structured like we expect them to be. I'm using @mael's HxD editor so if would help if that supported structures. tap.tap... 🙂 Another way would be to write a drop-in replacement for the msdia140.dll in-process COM server, which is what VTune uses to access the PDB data (I considered doing that at one point since it would completely eliminate the need to write pdb). That would tell us exactly what API methods VTune is using and how.

Yes.

With an exclusion list that removed most of the VCL and RTL as well as DevExpress, TeeChart, Indy and Firedac I managed to reduced the size of my pdb from 200Mb to 35Mb. VTune now loads my project in "just" 5 minutes... It's still struggling though. Everything is incredible slow. I get the impression that Intel has never tried profiling VTune with VTune.... or maybe they tried and gave up because it was too slow. Here's my command line: map2pdb -v -bind "TurboFooProPlus.map" -exclude:dx*;cx*;system*;winapi*;vcl*;data*;firedac*;soap*;web*;id*

New version with @Stefan Glienke's improvements uploaded. https://bitbucket.org/anders_melander/map2pdb/downloads/ Notable changes: VTune now attributes samples to the correct source lines within a block. Removing RTTI from the exe has reduced the size by 15%. The pdb is now being built in a temporary memory stream which is significantly faster due to reduced I/O.

I too have that problem. So far I haven't been able to profile our main project because the symbol resolution step takes longer (hours) than I'm willing to wait. It's probably caused by a bug in map2pdb (e.g. some hash table is incorrect) but for now my plan is to add a white/black list option to include and exclude units from the pdb. So for example if I specify the switch -exclude:dx* then any unit that starts with "dx" (i.e. DevExpress) will be excluded from the pdb. It's hard to spot but there's one at the bottom of the download page: https://software.intel.com/content/www/us/en/develop/articles/oneapi-standalone-components.html#vtune I'm on Windows 7 so I had to find an older version of VTune. VTune 2019 was the last one to support Windows 7. Unfortunately it's a trial and it expires in a week...

N0n53N53. /733t d00d3

Fixed the problem where some symbols (in particular generics) either resolved to the calling unit or to a wrong line number in the implementing unit. Thanks to @Stefan Glienke for pestering me about this until it got solved. Source committed and binary uploaded. And now... Pizza! 🍕

Yes, but it's a bug on my part. I previously ignored empty segments (i.e. the PDATA segment) but that caused other problems so now I add them to the segment list. I just forgot that I still need to ignore them when checking for overlaps. You can probably just delete the line from you map file. If there are any references to the segment in the map file ten you will have to remove those as well (search for 000A:).

I don't know. And does the segments overlap?

Hmmm. Maybe one just has to be patient. Very patient...

Exactly my experience.

I have now implemented MSF intervals so the 16Mb pdb limit should be gone. I have however not been able to verify that it actually works beyond validating the files with cvdump and llvm-pdbutil because VTune hangs, burning CPU, when it tries to resolve symbols from the large pdb files I've thrown at it (the smaller ones still works). This could be a bug in VTune, it could be that I just need to wait longer than the 30 minutes I've tried or it could be that there's either something wrong with the pdb file or that I need to order the symbols in it differently. From examining the call stacks of the VTune threads it appears as if it's busy doing symbol lookup through msdia140.dll. Anyhow, source committed and new precompiled uploaded. Give it a go and let me know if you have better luck than me. P.S. It turned out to be just a tiny bit more complex than just some recursive stuff https://bitbucket.org/anders_melander/map2pdb/commits/95353cbf3ba5b284919e617ace970b02ca0bf0e2

Thank you for the nice feedback @Vincent Parrett. Nighty night!

Well, I'm not sure what I can do about that because the error message is correct. There is no debug info in the file: I could of course write a new debug directory instead of relying on there being one already but I don't know how to do that. I looked into it at one point and AFAIR I think I remember that it could become fairly complicated once you start adding stuff to a PE file.

Amen to that. Ask them to ship pdb files. Now there's no excuse not to Or even better; Publish an API to read their debug info so we don't have to rely on the map files. I'm sure they have one.

Priorities. They probably don't have the 240+ hours to spare plus the time it will take to integrate into the product and maintain it afterwards. Also it's only fairly recently that this has become possible to do at all, after MS released the PDB source and LLVM deciphered them. And even LLVM still has problems with it.

@Attila Kovacs I found and fixed the problem; The size of the MSF Directory stream had grown beyond 1 block and my implementation of the MSF Directory Index assumed that a one block directory would be enough (who needs more than 640K anyway?) so I only wrote one block pointer. A module stream is created for each unit so if there are many units in a project there will be many stream and thus many entries in the directory. In your case there were 1775 streams in total. @David Heffernan I've also implemented support for PE32+ (64 bit). I guess I'm procrastinating to avoid handling the 16Mb limit New version uploaded.