Anders Melander

Yes.

With an exclusion list that removed most of the VCL and RTL as well as DevExpress, TeeChart, Indy and Firedac I managed to reduced the size of my pdb from 200Mb to 35Mb. VTune now loads my project in "just" 5 minutes... It's still struggling though. Everything is incredible slow. I get the impression that Intel has never tried profiling VTune with VTune.... or maybe they tried and gave up because it was too slow. Here's my command line: map2pdb -v -bind "TurboFooProPlus.map" -exclude:dx*;cx*;system*;winapi*;vcl*;data*;firedac*;soap*;web*;id*

New version with @Stefan Glienke's improvements uploaded. https://bitbucket.org/anders_melander/map2pdb/downloads/ Notable changes: VTune now attributes samples to the correct source lines within a block. Removing RTTI from the exe has reduced the size by 15%. The pdb is now being built in a temporary memory stream which is significantly faster due to reduced I/O.

I too have that problem. So far I haven't been able to profile our main project because the symbol resolution step takes longer (hours) than I'm willing to wait. It's probably caused by a bug in map2pdb (e.g. some hash table is incorrect) but for now my plan is to add a white/black list option to include and exclude units from the pdb. So for example if I specify the switch -exclude:dx* then any unit that starts with "dx" (i.e. DevExpress) will be excluded from the pdb. It's hard to spot but there's one at the bottom of the download page: https://software.intel.com/content/www/us/en/develop/articles/oneapi-standalone-components.html#vtune I'm on Windows 7 so I had to find an older version of VTune. VTune 2019 was the last one to support Windows 7. Unfortunately it's a trial and it expires in a week...

N0n53N53. /733t d00d3

Fixed the problem where some symbols (in particular generics) either resolved to the calling unit or to a wrong line number in the implementing unit. Thanks to @Stefan Glienke for pestering me about this until it got solved. Source committed and binary uploaded. And now... Pizza! 🍕

Yes, but it's a bug on my part. I previously ignored empty segments (i.e. the PDATA segment) but that caused other problems so now I add them to the segment list. I just forgot that I still need to ignore them when checking for overlaps. You can probably just delete the line from you map file. If there are any references to the segment in the map file ten you will have to remove those as well (search for 000A:).

I don't know. And does the segments overlap?

Hmmm. Maybe one just has to be patient. Very patient...

Exactly my experience.

I have now implemented MSF intervals so the 16Mb pdb limit should be gone. I have however not been able to verify that it actually works beyond validating the files with cvdump and llvm-pdbutil because VTune hangs, burning CPU, when it tries to resolve symbols from the large pdb files I've thrown at it (the smaller ones still works). This could be a bug in VTune, it could be that I just need to wait longer than the 30 minutes I've tried or it could be that there's either something wrong with the pdb file or that I need to order the symbols in it differently. From examining the call stacks of the VTune threads it appears as if it's busy doing symbol lookup through msdia140.dll. Anyhow, source committed and new precompiled uploaded. Give it a go and let me know if you have better luck than me. P.S. It turned out to be just a tiny bit more complex than just some recursive stuff https://bitbucket.org/anders_melander/map2pdb/commits/95353cbf3ba5b284919e617ace970b02ca0bf0e2

Thank you for the nice feedback @Vincent Parrett. Nighty night!

Well, I'm not sure what I can do about that because the error message is correct. There is no debug info in the file: I could of course write a new debug directory instead of relying on there being one already but I don't know how to do that. I looked into it at one point and AFAIR I think I remember that it could become fairly complicated once you start adding stuff to a PE file.

Amen to that. Ask them to ship pdb files. Now there's no excuse not to Or even better; Publish an API to read their debug info so we don't have to rely on the map files. I'm sure they have one.

Priorities. They probably don't have the 240+ hours to spare plus the time it will take to integrate into the product and maintain it afterwards. Also it's only fairly recently that this has become possible to do at all, after MS released the PDB source and LLVM deciphered them. And even LLVM still has problems with it.

@Attila Kovacs I found and fixed the problem; The size of the MSF Directory stream had grown beyond 1 block and my implementation of the MSF Directory Index assumed that a one block directory would be enough (who needs more than 640K anyway?) so I only wrote one block pointer. A module stream is created for each unit so if there are many units in a project there will be many stream and thus many entries in the directory. In your case there were 1775 streams in total. @David Heffernan I've also implemented support for PE32+ (64 bit). I guess I'm procrastinating to avoid handling the 16Mb limit New version uploaded.

Please do. I'll need both the map file and the pdb file. I'm not sure what info you're referring to here. What table?

I think .NET pdb files are in a different format.

I can't just ignore the section offsets from the map file since all the other offsets in it are relative to them. I would have to relocate all offsets then. Edit: That was obviously nonsense. I wouldn't have to relocate the offsets exactly because they're relative. The TLS section offset is wrong but I believe all other offsets are correct. VTune seems to agree...

Yes. I've been using cvdump and llvm-pdbutil to validate my pdb files along the way. How large is the pdb file? llvm-pdbutil is fairly tolerant while cvdump mostly just gives up and does nothing if there is a problem. cvdump uses the DIA API to read information from the pdb, which is the same API that VTune uses. You can download llvm-pdbutil here: https://github.com/shaharv/llvm-pdbutil-builds/releases Use the command: llvm-pdbutil dump -all <pdb-filename> I suggest you pipe the output to a file and then examine the file.

I don't think there's enough information in the map file to solve that problem properly but I'll investigate it some more when I've gotten these other things out of the way.

Yes - and yes

The problem with your map file was just that my validation was a bit too aggressive. I've now turned some of the errors into warnings. New version uploaded. One of the problems was that I discarded zero size segments. E.g. the TLS segment: 0005:00000000 00000000H .tls TLS which meant that later on when a module or symbol tried to resolve segment #5 that turned into an error. Detailed map of segments 0005:00000000 00000100 C=TLS S=.tls G=(none) M=OtlCommon.Utils ACBP=A9 Address Publics by Name 0005:00000000 OtlCommon.Utils.LastThreadName 0005:00000100 SysInit.TlsLast I'm not not really sure of how I should handle these out of bounds situations. Right now I just warn about them and then ignore the module/symbol/line. I think you just forgot the -bind switch. Your project3.exe did not contain a reference to project3.pdb. After I run map2pdb -bind project3,map I got VTune to resolve with your exe. You're probably hitting the 16Mb limit, I'm now outputting a warning so we're aware of it when it happens. I was planning on adding support for PE32+ (64-bit) next but I guess solving the MSF/16Mb problem is more important. I can't really use it myself for anything serious until that's done.

Can you try with a tiny console application. I've used the following for test: program test; {$APPTYPE CONSOLE} {$R *.res} uses System.Classes, System.SysUtils; procedure WasteSomeTime; begin var Strings := TStringList.Create; try for var i := 1 to 100000 do Strings.Add(i.ToString); Strings.Sort; finally Strings.Free; end; end; begin try for var i := 1 to 10 do begin Writeln(i.ToString); WasteSomeTime; end; except on E: Exception do Writeln(E.ClassName, ': ', E.Message); end; end. If that doesn't work either then please zip the map file + exe produced by the above code and post it here or at the issue tracker on bitbucket.

Sometimes it writes in the log if it can't find the pdb. Sometimes it doesn't... I guess Intel are focusing their efforts on making hardware I had to use Process Monitor to verify that it actually read the pdb file. That was also the way I discovered that it didn't look for the pdb unless there was a reference to it in the exe file. I think I will need the map file to determine the cause of that. You can mail it to me if that's ok. I haven't thought about dlls at all but I guess there shouldn't be any difference.