Anders Melander

Thank you for the nice feedback @Vincent Parrett. Nighty night!

Well, I'm not sure what I can do about that because the error message is correct. There is no debug info in the file: I could of course write a new debug directory instead of relying on there being one already but I don't know how to do that. I looked into it at one point and AFAIR I think I remember that it could become fairly complicated once you start adding stuff to a PE file.

Amen to that. Ask them to ship pdb files. Now there's no excuse not to Or even better; Publish an API to read their debug info so we don't have to rely on the map files. I'm sure they have one.

Priorities. They probably don't have the 240+ hours to spare plus the time it will take to integrate into the product and maintain it afterwards. Also it's only fairly recently that this has become possible to do at all, after MS released the PDB source and LLVM deciphered them. And even LLVM still has problems with it.

@Attila Kovacs I found and fixed the problem; The size of the MSF Directory stream had grown beyond 1 block and my implementation of the MSF Directory Index assumed that a one block directory would be enough (who needs more than 640K anyway?) so I only wrote one block pointer. A module stream is created for each unit so if there are many units in a project there will be many stream and thus many entries in the directory. In your case there were 1775 streams in total. @David Heffernan I've also implemented support for PE32+ (64 bit). I guess I'm procrastinating to avoid handling the 16Mb limit New version uploaded.

Please do. I'll need both the map file and the pdb file. I'm not sure what info you're referring to here. What table?

I think .NET pdb files are in a different format.

I can't just ignore the section offsets from the map file since all the other offsets in it are relative to them. I would have to relocate all offsets then. Edit: That was obviously nonsense. I wouldn't have to relocate the offsets exactly because they're relative. The TLS section offset is wrong but I believe all other offsets are correct. VTune seems to agree...

Yes. I've been using cvdump and llvm-pdbutil to validate my pdb files along the way. How large is the pdb file? llvm-pdbutil is fairly tolerant while cvdump mostly just gives up and does nothing if there is a problem. cvdump uses the DIA API to read information from the pdb, which is the same API that VTune uses. You can download llvm-pdbutil here: https://github.com/shaharv/llvm-pdbutil-builds/releases Use the command: llvm-pdbutil dump -all <pdb-filename> I suggest you pipe the output to a file and then examine the file.

I don't think there's enough information in the map file to solve that problem properly but I'll investigate it some more when I've gotten these other things out of the way.

Yes - and yes

The problem with your map file was just that my validation was a bit too aggressive. I've now turned some of the errors into warnings. New version uploaded. One of the problems was that I discarded zero size segments. E.g. the TLS segment: 0005:00000000 00000000H .tls TLS which meant that later on when a module or symbol tried to resolve segment #5 that turned into an error. Detailed map of segments 0005:00000000 00000100 C=TLS S=.tls G=(none) M=OtlCommon.Utils ACBP=A9 Address Publics by Name 0005:00000000 OtlCommon.Utils.LastThreadName 0005:00000100 SysInit.TlsLast I'm not not really sure of how I should handle these out of bounds situations. Right now I just warn about them and then ignore the module/symbol/line. I think you just forgot the -bind switch. Your project3.exe did not contain a reference to project3.pdb. After I run map2pdb -bind project3,map I got VTune to resolve with your exe. You're probably hitting the 16Mb limit, I'm now outputting a warning so we're aware of it when it happens. I was planning on adding support for PE32+ (64-bit) next but I guess solving the MSF/16Mb problem is more important. I can't really use it myself for anything serious until that's done.

Can you try with a tiny console application. I've used the following for test: program test; {$APPTYPE CONSOLE} {$R *.res} uses System.Classes, System.SysUtils; procedure WasteSomeTime; begin var Strings := TStringList.Create; try for var i := 1 to 100000 do Strings.Add(i.ToString); Strings.Sort; finally Strings.Free; end; end; begin try for var i := 1 to 10 do begin Writeln(i.ToString); WasteSomeTime; end; except on E: Exception do Writeln(E.ClassName, ': ', E.Message); end; end. If that doesn't work either then please zip the map file + exe produced by the above code and post it here or at the issue tracker on bitbucket.

Sometimes it writes in the log if it can't find the pdb. Sometimes it doesn't... I guess Intel are focusing their efforts on making hardware I had to use Process Monitor to verify that it actually read the pdb file. That was also the way I discovered that it didn't look for the pdb unless there was a reference to it in the exe file. I think I will need the map file to determine the cause of that. You can mail it to me if that's ok. I haven't thought about dlls at all but I guess there shouldn't be any difference.

Fixed. https://bitbucket.org/anders_melander/map2pdb/commits/8c94db07a2da1b503ce6f00c39792ee0b5479387

Thanks. Naturally, as something like that always is, the map parser is a mess However I'm not actually associating symbols with source files. I'm associating symbols with modules (i.e. units) based on the symbol address and the module address range. So for example: 0001:001236E0 System.Classes.{System.Generics.Collections}TDictionary<System.Integer,System.Classes.IInterfaceList>.Add is associated with System.Classes because the offset $001236E0 locates it in the range of that module: $000C7730 - $0015F654 ($000C7730+$0009F24): 0001:000C7730 00097F24 C=CODE S=.text G=(none) M=System.Classes ALIGN=4 Later when I parse the line number information I get the correct source file and can associate that, source file and line number+offset, with the module. I.e. System.Generics.Collection.pas in the module SystemClasses. Line numbers for System.Classes(System.Generics.Collections.pas) segment .text 7088 0001:001236E0 7089 0001:001236F1 7090 0001:001236F9 7092 0001:00123701 7093 0001:0012370E 7094 0001:0012371B 7095 0001:0012371F 7097 0001:0012373C 7098 0001:00123755 This way you get both the symbol resolved to the correct module/unit and the code resolved to the correct source lines.

The problem is the MSF container format. It's a blocked format with 4096 bytes in each block and up to 4096 blocks in each "interval" (4096*4096 = 16Mb). Each interval starts with a Free Page Map (FPM, a bit like a FAT). Since I wanted to concentrate on getting the PDB layout right my MSF abstraction doesn't implement intervals so once I pass 4096 blocks the file becomes invalid. Here's LLVMs explanation of the MSF format: https://llvm.org/docs/PDB/MsfFile.html A proper implementation of intervals will require a bit of work. At present I can assume that a physical stream offset value equals the logical offset value. Once I add intervals I will have to take into the FPMs into account since the blocks are no longer contiguous. Also when I write I will have to consider that a write can start on once side of the FPMs and continue on the other side of them. It isn't rocket science but it does complicate the IO layer considerably. No. I get those too. I'm not sure what they are for but they are in segment 4 (TLS) and 5 (PDATA= exception data) so you'll not miss them. The warning just means that the address of a symbol couldn't be matched to any of the modules (units). It should be. How do you come to the conclusion that it wasn't patched? Try the -v switch. Okay. There's probably a bug in the command line interface then. I discovered the problem with the position of the -bind switch just before release so there's probably more.

Yes it should. As long as the layout of the map file is the same. It's easy enough to fix if there's minor differences but I don't think there are. Yes. I need it for 64-bit myself. The only thing needed to support for 64-bit is the ability to parse and modify a PE64 image. Right now I don't what the difference is between PE32 and PE64. Here's the relevant code: https://bitbucket.org/anders_melander/map2pdb/src/27bb0daa1b4a7e0159b646f4b0cd0d34ffc72fd3/Source/debug.info.pdb.

For those following this thread:

No but someone suggested doing so and as always that got the ball rolling.

First of all the application isn't hanging. It's just not processing messages while it's working. When you click on the form Windows detects that the application hasn't pumped the message queue for a while and assumes this is because the application is stuck in an endless loop somewhere. In your case that's a wrong conclusion because eventually your task finishes and everything is good again. The simple work around is to pump the message queue inside your loop - just don't use Application.ProcessMessages. I believe have previously posted some code here that can be used inside a loop to allow the user to interact with a progress dialog (e.g. move it, push an abort button, etc.) without the pitfalls of Application.ProcessMessages but if you can't find it you can at least pump the queue for WM_NULL messages. I think these are what Windows use to detect a hung application: var Msg: TMessage; while PeekMessage(0, Msg, WM_NULL, WM_NULL, PM_REMOVE) do begin if (Msg.message = WM_QUIT) then PostQuitMessage(Msg.wParam); end; If you can reproduce while running in the debugger then just press the Pause button in the debugger and examine the call stack. At run time you can use something like madExcept's freeze detection. It uses the same technique as Windows to detect a hung application and produces a nice call stack when it happens (after prompting the user).

Yes. You are wrong. It processes WM_PAINT messages (which is why it's updating) but leave everything else alone.

I don't think you understand how the message queue works... How are messages supposed to arrive through Application.OnMessage if the message queue isn't being pumped? And if it is being pumped then your code is unnecessary because the messages are already being dispatched and processed. In short: Your code does nothing that isn't already being done.

I'm not sure why that is relevant. I only mentioned PDATA because you claimed that the map file doesn't contain the ImageBase value; I pointed out that it does because in your map example ImageBase is the same as the start offset of the PDATA segment. Generally the ImageBase value is probably just the offset with the lowest value from the segment list. Everything is easy once you know how to do it. I understand the MSF container format but I have no clear understanding of the internal structure of the PDB format it contains. In hindsight it would have been faster to just learn what there is to know about the topic, but the fact that MS have stated that they'll change the format as they see fit, kinda turned me away from that idea. I've now also gotten the clear impression, from reading the LLVM source, there there are parts that even the LLVM project don't understand. For example the PublicsStream section which VTune apparently requires. Anyway, I'm now learning about DWARF and how FPC handles it. Thanks for bringing that to my attention.

Another interpretation could be that VTune's PDB importer extracts more details than its DWARF importer, even if the files contain the same information. I believe the roundtrip test with VTune's matrix example proved that yaml2pdb produces PDBs that cannot be used with VTune. The pdb worked with VTune before going through LLVM's pdb2yaml->yaml2pdb. It didn't work after. Definitely. I have no idea about how they affect performance but they are incredible convenient and I think they improve the readability of the code. Yes it is: 0006:00400000 00000000H .pdata PDATA This corresponds to the ImageBase field in the PE header. I don't know what the deal is with the .tls section. Anyway since all the offsets in the map file are relative, and the process can get reallocated to anywhere at run time, the section addresses are really just nice-to-know. As far as I can see, if you know the actual ImageBase of a process, then the map file contains all the information required to get from physical address to method/function and source code line and vice versa. That's not a duplicate. The map file contains two list of symbols. One is ordered by name and the other by address so if you just do a text search any symbol will occur twice in the map file. I only read the first list and ignore the other.