Anders Melander

First of all the application isn't hanging. It's just not processing messages while it's working. When you click on the form Windows detects that the application hasn't pumped the message queue for a while and assumes this is because the application is stuck in an endless loop somewhere. In your case that's a wrong conclusion because eventually your task finishes and everything is good again. The simple work around is to pump the message queue inside your loop - just don't use Application.ProcessMessages. I believe have previously posted some code here that can be used inside a loop to allow the user to interact with a progress dialog (e.g. move it, push an abort button, etc.) without the pitfalls of Application.ProcessMessages but if you can't find it you can at least pump the queue for WM_NULL messages. I think these are what Windows use to detect a hung application: var Msg: TMessage; while PeekMessage(0, Msg, WM_NULL, WM_NULL, PM_REMOVE) do begin if (Msg.message = WM_QUIT) then PostQuitMessage(Msg.wParam); end; If you can reproduce while running in the debugger then just press the Pause button in the debugger and examine the call stack. At run time you can use something like madExcept's freeze detection. It uses the same technique as Windows to detect a hung application and produces a nice call stack when it happens (after prompting the user).

Yes. You are wrong. It processes WM_PAINT messages (which is why it's updating) but leave everything else alone.

I don't think you understand how the message queue works... How are messages supposed to arrive through Application.OnMessage if the message queue isn't being pumped? And if it is being pumped then your code is unnecessary because the messages are already being dispatched and processed. In short: Your code does nothing that isn't already being done.

I'm not sure why that is relevant. I only mentioned PDATA because you claimed that the map file doesn't contain the ImageBase value; I pointed out that it does because in your map example ImageBase is the same as the start offset of the PDATA segment. Generally the ImageBase value is probably just the offset with the lowest value from the segment list. Everything is easy once you know how to do it. I understand the MSF container format but I have no clear understanding of the internal structure of the PDB format it contains. In hindsight it would have been faster to just learn what there is to know about the topic, but the fact that MS have stated that they'll change the format as they see fit, kinda turned me away from that idea. I've now also gotten the clear impression, from reading the LLVM source, there there are parts that even the LLVM project don't understand. For example the PublicsStream section which VTune apparently requires. Anyway, I'm now learning about DWARF and how FPC handles it. Thanks for bringing that to my attention.

Another interpretation could be that VTune's PDB importer extracts more details than its DWARF importer, even if the files contain the same information. I believe the roundtrip test with VTune's matrix example proved that yaml2pdb produces PDBs that cannot be used with VTune. The pdb worked with VTune before going through LLVM's pdb2yaml->yaml2pdb. It didn't work after. Definitely. I have no idea about how they affect performance but they are incredible convenient and I think they improve the readability of the code. Yes it is: 0006:00400000 00000000H .pdata PDATA This corresponds to the ImageBase field in the PE header. I don't know what the deal is with the .tls section. Anyway since all the offsets in the map file are relative, and the process can get reallocated to anywhere at run time, the section addresses are really just nice-to-know. As far as I can see, if you know the actual ImageBase of a process, then the map file contains all the information required to get from physical address to method/function and source code line and vice versa. That's not a duplicate. The map file contains two list of symbols. One is ordered by name and the other by address so if you just do a text search any symbol will occur twice in the map file. I only read the first list and ignore the other.

Accessing a string one char beyond the length of the string is fine because Delphi string contains an implicit zero terminating zero. Doing the same on an empty string once worked the same but now causes an AV because the string is nil. I don't know when this changed but it was before 10.3 var Foo: string; begin Foo := 'Hello World'; ShowMessage(IntToStr(Ord( Foo[Length(Foo)+1] ))); // No problem Foo := ''; ShowMessage(IntToStr(Ord( Foo[Length(Foo)+1] ))); // Access Violation end; My guess is the code predates Delphi 2009 since it uses WideString to support unicode.

So like @Stefan Glienke suggested I ran the VTune example application matrix.exe/matrix.pdb through llvm-pdbutil pdb2yaml to produce yaml and then again with yaml2pdb to produce a new pdb. VTune now failed to resolve addresses so I ran the updated pdb through llvm-pdbutil pdb2yaml once more and compared the first and second yaml file to see what had changed. The only significant change was that the PublicsStream section was now empty. I had observed this empty PublicsStream earlier when testing the roundtrip of the yaml I produce but since I assumed that the pdb produced by llvm-pdbutil can actually be used with VTune I concluded that this was a section that wasn't required and so I disabled the output of it (I can also see in the llvm source that they haven't implemented a writer for the section). I can now see that my assumption was incorrect. While LLVM might have the intention of producing pdb that can be used with VTune I haven't seen any proof that they have actually reached that goal. This means that at least half of this project has been a complete goose chase. Well at least I now know why it doesn't work. I looked at it early on and then decided that it was safer to use llvm-pdbutil than to try and port it to Delphi. Maybe I was wrong 🙂 Also it's cheating; It's using undocumented Visual Studio DLLs to manipulate the pdb files so it requires that VS is installed. If this was a part of the Win SDK I wouldn't have a problem with it but I think the dependency on VS makes it a no-go. I thought DWARF was for for Linux only and that VTune didn't support it on Windows...? If I have understood you correctly you're saying that VTune works with DWARF on Windows both when it's linked into the exe and when it's external (with a reference from the EXE). I haven't investigated the DWARF format yet but I know that it's at least documented (unlike LLVMs YAML) - and maybe I can even reuse some of FPCs code to handle it. I believe we can conclude that whatever values I write is moot since the PublicsStream section is missing from the pdb and that alone is enough to for VTune to not use the line number information that is there. Regardless of that; When you say RVA I assume you mean the PE section RVA as specified in the PE headers? The only place in the pdb that RVAs are references is in the "*Linker *" section. Since the map file doesn't give me the RVA values I simply write the segment offset instead. I don't really understand why the pdb would need the RVA but if that's how it is then I will just have to deal with it. I guess I will have to read the map file and then extract the RVA values from the PE header in order to get all the info required to produce a pdb. I discard duplicate symbols based on segment+offset. Duplicate segment+name are allowed but only one is emitted. I have seen any segment+offset duplicates.

Maybe. I'm a bit unsure about what they mean by symbols. In my implementation "symbols" are methods (name/offset/size) but I'm not including stack frame, parameter info, locals, return values etc. since I don't have that information available. SymChk reports that I don't have "global symbols" but while I assumed that to mean global exports (which I don't supply) it might well be meaning the globals mentioned in the post. Good idea. The round trip broke the pdb. VTune can no longer resolve and it doesn't report any errors. Just like with my pdb I'll do a diff on the old and new tomorrow. Maybe I should try with a newer version of VTune. I'm not too thrilled about asking for help on the VTune forum about an old unsupported version either (my community registration finally completed).

It was. This is what happened when I changed the YAML to include the module ObjName property with a reference to the dcu (of course the map file doesn't contain the obj filename so I'm just supplying a dummy filename): Looks like a bug in VTune.

I've tried that but for some reason I can't get past their community registration. I registered when I downloaded VTune but when I try to post to their forum I get stuck in a registration confirmation loop. Tried with different accounts, different browsers. No. Never used WinDbg. If someone else want to have a go at it I'll be happy to supply a zip with source, exe and pdb. I verified the exe/pdb with symchk from Debugging Tools for Windows and it passed with no problems: D:\Projects\map2pdb\Bin\Win32\Debug>"D:\Development\Debugging Tools for Windows (x86)\symchk.exe" -v map2yaml.exe -s D:\Projects\map2pdb\Bin\Win32\Debug /pf [SYMCHK] Searching for symbols to D:\Projects\map2pdb\Bin\Win32\Debug\map2yaml.exe in path D:\Projects\map2pdb\Bin\Win32\Debug DBGHELP: Symbol Search Path: D:\Projects\map2pdb\Bin\Win32\Debug [SYMCHK] Using search path "D:\Projects\map2pdb\Bin\Win32\Debug" DBGHELP: No header for D:\Projects\map2pdb\Bin\Win32\Debug\map2yaml.exe. Searching for image on disk DBGHELP: D:\Projects\map2pdb\Bin\Win32\Debug\map2yaml.exe - OK DBGHELP: map2yaml - public symbols & lines D:\Projects\map2pdb\Bin\Win32\Debug\map2yaml.pdb [SYMCHK] MODULE64 Info ---------------------- [SYMCHK] Struct size: 1680 bytes [SYMCHK] Base: 0x00400000 [SYMCHK] Image size: 6230016 bytes [SYMCHK] Date: 0x60526592 [SYMCHK] Checksum: 0x00000000 [SYMCHK] NumSyms: 0 [SYMCHK] SymType: SymPDB [SYMCHK] ModName: map2yaml [SYMCHK] ImageName: D:\Projects\map2pdb\Bin\Win32\Debug\map2yaml.exe [SYMCHK] LoadedImage: D:\Projects\map2pdb\Bin\Win32\Debug\map2yaml.exe [SYMCHK] PDB: "D:\Projects\map2pdb\Bin\Win32\Debug\map2yaml.pdb" [SYMCHK] CV: RSDS [SYMCHK] CV DWORD: 0x53445352 [SYMCHK] CV Data: map2yaml.pdb [SYMCHK] PDB Sig: 0 [SYMCHK] PDB7 Sig: {CBB17264-89FA-4AED-A2D7-814EE276EF3E} [SYMCHK] Age: 1 [SYMCHK] PDB Matched: TRUE [SYMCHK] DBG Matched: TRUE [SYMCHK] Line nubmers: TRUE [SYMCHK] Global syms: FALSE [SYMCHK] Type Info: FALSE [SYMCHK] ------------------------------------ SymbolCheckVersion 0x00000002 Result 0x000f0001 DbgFilename DbgTimeDateStamp 0x60526592 DbgSizeOfImage 0x005f1000 DbgChecksum 0x00000000 PdbFilename D:\Projects\map2pdb\Bin\Win32\Debug\map2yaml.pdb PdbSignature {CBB17264-89FA-4AED-A2D7-814EE276EF3E} PdbDbiAge 0x00000001 [SYMCHK] [ 0x00000000 - 0x000f0001 ] Checked "D:\Projects\map2pdb\Bin\Win32\Debug\map2yaml.exe" SYMCHK: FAILED files = 0 SYMCHK: PASSED + IGNORED files = 1 The above just verifies that the exe reference to the pdb is correct and that the pdb is valid and contain line number information. I don't think symchk can validate deeper than that. I get similar output when I run it on the matrix.exe sample provided with VTune, except that one also has symbol & type information in the pdb. Since I could see that VTune was looking in it's own bin directory for files with the same name (sans type) as the modules in my pdb I tried copying the source files there and rename them to match what it looked for. Apparently it wasn't source files it was looking for because it just read the first 56 bytes of each file and it made no difference in the end. My guess is that it was looking for obj files.

🙂 yeah they need to be fairly recent. Since LLVm only produce they "new" PDB format it doesn't need to support the older formats. I think anything older that 5 years will probably fail.

Historical reasons probably. https://softwareengineering.stackexchange.com/questions/171565/why-is-the-code-section-called-a-text-section https://stackoverflow.com/questions/1282506/where-did-the-text-segment-get-its-name

I'm not sure what saying - or asking... I know what the values in the map file means (they're all absolute [*]). It's the values that I need to put in the yaml/pdb that I'm unsure about. [*] Of course all addresses are relative in a sense, since it's only after the exe has been mapped into virtual memory that absolute addresses can be known. When I say "relative to the segment" I mean these from the map file: Start Length Name Class 0001:00401000 0011B0C0H .text CODE 0002:0051D000 00001144H .itext ICODE 0003:0051F000 00003BACH .data DATA 0004:00523000 0000645CH .bss BSS 0005:00000000 00000020H .tls TLS 0006:00400000 00000000H .pdata PDATA

Yes. As far as I can tell. I have also tried validating the pdb but there aren't really any tools (that I've found) that can analyze a pdb for validity. I think the problem is more about what values I need to provide in the pdb (for example I don't have type information or mangled names so I can't write those) and what the meaning of the values are. One of the things I've struggled with are address values. Are they absolute or relative and if relative, then relative to what. I currently assume the following: Segment: Absolute Module (Unit): Relative to Segment Line: Relative to Module Symbol (method/function): Relative to Module But I have tried just about all different combinations. Since VTune doesn't provide any feedback on the address resolution (other than crash/no crash) I have just to throw everything at the wall to see what sticks. Pretty frustrating. I even tried attaching to the VTune backend with the debugger to examine what it did but I quickly gave up on that as it (not surprising) is massively multi threaded and simply too complex for that approach.

Don't count your chickens before they hatch... I think you jinxed me there. Here's what works so far: Parse a MAP file and produce an YAML file. Convert the YAML to PDB. Update the EXE with a reference to the PDB. Not crash VTune while it's loading the PDB Here's what doesn't work: Getting VTune to use any of the information in the PDB I know that VTune reads the PDB because I can see in Process Monitor that it looks for the source files. Unfortunately the file names it looks for are the module (i.e. unit) names and not the file names and it doesn't search the source folders I have defined: I'm using VTune 2019 btw since that's the last version to support Windows 7. Anyway if you can stomach watching the sausages getting made then the current source is available here: https://bitbucket.org/anders_melander/map2pdb/ There are two projects: map2yaml and bindpdb. In addition to that the llvm-pdbutil tool is needed. The Tools folder contains a batch file that calls all three in order. Delphi 10.3 or later is required (inline vars).

I haven't looked at it at all (still fighting with the damned PDB stuff) but maybe that could have been done automatically when a new resource module is loaded? That would have made the change backward compatible.

Got it. Wow. Nice that they've added a hook but they really seem to have made that complicated. I'll look at it later.

I don't have time to investigate myself right now but can you see where LoadResStringFunc is assigned? I briefly scanned through the 10.4 source and I couldn't find anything there.

It turns out the null GUID was what kept VTune from looking for the pdb file. VTune now loads my pdb file but doesn't resolve the addresses to source lines. I will probably have to populate the pdb with method information for that to work. Makes sense. I guess I'm not getting any sleep tonight

As I wrote, VTune doesn't even look for the pdb file so the problem must be in the exe. Anyhow I have now managed to patch the exe so VTune at least logs that it is looking for a pdb (it didn't before) but can't find one. Process Monitor still shows that it isn't looking for the file though so I'm not there yet. My pdb file only contains module/unit and line number information as I couldn't find a way to pass the method names on to the yaml2pdb tool at that time. I think I have found out how to do that now but I haven't tried it yet. Yes, that is one of the resources I'm using - and as far as I can see it states exactly the opposite: So far I'm doing this: Set NtHeaders32.OptionalHeader.DataDirectory[IMAGE_DIRECTORY_ENTRY_DEBUG].Size = SizeOf(TImageDebugDirectory) Old value was 1 which is invalid. Set DebugDirectory._Type := IMAGE_DEBUG_TYPE_CODEVIEW Old value was IMAGE_DEBUG_TYPE_UNKNOWN (zero). A bit lazy on Borland's behalf since there is a IMAGE_DEBUG_TYPE_BORLAND defined. Replace the old TDS debug data with a CV_INFO_PDB70 structure containing the RSDS signature, a null GUID, 1 for the Age and the pdb filenamer as a zero terminated UTF8 string. Clear the checksum to zero. I have verified the patched exe with various PE editors and viewers and everything seems to be in place but still no cigar.

Not quite Game Over it seems. It appears that the Size being 1 is just because whomever wrote the linker has misunderstood the meaning of the field. 1 in this case means that there one entry, so I could replace the 1 with $1C and the entry would be valid. If I then assume that data in the debug directory is now valid then the one entry points to the .debug segment. I assume this segment contains the TDS debug data or something like (it starts with the TDS signature "FB09"). It's only present (for both 32- and 64-bit) if I link with debug info enabled. Now since this debug info isn't used anyway when profiling with VTune, I can just hijack the area occupied by it and store my IMAGE_DEBUG_TYPE_CODEVIEW structure there. This means that I won't have to deal with adding new sections and updating all the various offsets in the PE header. Should be doable with what I know so far. I have to some gardening to take care of now but I'll give it another go this evening. Stay (V)tuned...

No. By default VTune will look for the file in the same folder as the executable and I have also tried to add this folder manually to the search list but, as I said, VTune doesn't look for the file anywhere. I'm sure those two have their uses but for me they simply does not provide the level of detail that I need. I need call graphs and asm level analysis.

The content of a .NET pdb file is not the same as one produced by the native linker so that will not help. Besides I have already solved the pdb part of the problem and am able to generate a pdb file from the Delphi map file. The remaining problem is that VTune will not load the pdb file unless the file is referenced inside the exe file. I have verified with process monitor that VTune doesn't even look for the pdb file when loading an application compiled with Delphi. It's probably a fairly simple task to patch the exe with the required information but it involves altering the PE structures and that is something I know very little about - and I don't really have time to learn that stuff. The following is the PE layout of an application that does contain the required debug information (the tool used is PE View) : An entry in the PE header points to the debug directory (an array of IMAGE_DATA_DIRECTORY). The number of entries in the debug directory is Size div SizeOf( IMAGE_DATA_DIRECTORY). In this case the size is $1C so the count is one. The entry in the debug directory contains a value that specifies the type of the entry (in this case CodeView) and a pointer (relative address) to the debug data (a IMAGE_DEBUG_TYPE_CODEVIEW structure): Finally the IMAGE_DEBUG_TYPE_CODEVIEW structure contains the name of the PDB file: The IMAGE_DEBUG_TYPE_CODEVIEW structure looks like this: struct CV_INFO_PDB70 { DWORD CvSignature; GUID Signature; DWORD Age; BYTE PdbFileName[]; }; And this is the same information of a Delphi compiled application: Notice that the size of the debug directory is 1 (an invalid value incidentally), so the count is zero. Game over.

I know that VS refuses to load a PDB if the GUID doesn't match and I assume VTune does as well. They probably both use the DbgHelp API so maybe it's a feature of that. I think the flow here would be to generate a PDB with a GUID and then update the EXE with the GUID and the name of the PDB (sans the path). I can't see patching the EXE with a hex editor would buy me anything and it's not that easy when there isn't already room in the debug directory. It could be used to verify that the PDB is valid, but since the final tool chain will need to patch the EXE anyway one might as well get that out of the way first and only then worry about the validity of the PDB.

I have several tools that can edit PE files (among them PE Explorer), but I think it would be faster to just write a tool that does the job the right way. It seems there also a GUID that needs to be matched between the PDB and EXE.