Anders Melander 1795 Posted May 22, 2023 11 minutes ago, Anders Melander said: address bitness I meant what do you mean by this ^ Do you mean 32- vs 64-bit addresses? AFAIR there's no choice or ambiguities in the PDB format with regard to the size of an address value, relative or absolute, but I would have to take a look at the source to make sure. Share this post Link to post
Stefan Glienke 2009 Posted May 22, 2023 2 minutes ago, Anders Melander said: I meant what do you mean by this ^ Do you mean 32- vs 64-bit addresses? AFAIR there's no choice or ambiguities in the PDB format with regard to the size of an address value, relative or absolute, but I would have to take a look at the source to make sure. Yes - because apparently according to the warning, there is a problem with "match the module with the symbol file". It could be very well a VTune bug though - after all the 2023.1.0 we currently get is a "pre-release". Share this post Link to post
Anders Melander 1795 Posted May 22, 2023 51 minutes ago, Stefan Glienke said: Cannot locate debugging information for file `somepath\Tests.exe'. Cannot match the module with the symbol file `somepath\Tests.pdb'. Make sure to specify the correct path to the symbol file in the Binary/Symbol Search list of directories. I think that is a generic error message meaning "Something went wrong and our error handling sucks". As far as I remember you get a message like that regardless of what problem VTune encounters when resolving through the PDB file. Share this post Link to post
Jan Rysavy 8 Posted May 22, 2023 Created issue: https://bitbucket.org/anders_melander/map2pdb/issues/5/intel-vtune-20231-does-not-load-generated 1 Share this post Link to post
Jan Rysavy 8 Posted May 22, 2023 (edited) 2 hours ago, Anders Melander said: The problem is most likely caused by msdia140.dll; The library VTune uses to read pdb files. You can try replacing the one that VTune installs with an older version. I can confirm replacing C:\Program Files (x86)\Intel\oneAPI\vtune\2023.1.0\bin64\amplxe_msdia140.dll (version 14.34.31942.0) with version 14.28.29910.0 from VTune 2022.4.1 solves this problem! Edited May 22, 2023 by Jan Rysavy 1 Share this post Link to post
Anders Melander 1795 Posted May 22, 2023 8 minutes ago, Jan Rysavy said: I can confirm replacing C:\Program Files (x86)\Intel\oneAPI\vtune\2023.1.0\bin64\amplxe_msdia140.dll (version 14.34.31942.0) with version 14.28.29910.0 from VTune 2022.4.1 solves this problem! That means the bug is most likely in map2pdb because that DLL is Microsoft's API for reading PDB files. Share this post Link to post
Jan Rysavy 8 Posted May 22, 2023 This also probably explains why Profiler in Visual Studio (2022) stopped displaying function names in recent versions. It uses newer versions of msdia140.dll. Share this post Link to post
Jan Rysavy 8 Posted May 22, 2023 VTune 2023.1.0 works fine also with C:\Program Files\AMD\AMDuProf\bin\msdia140.dll (version 14.29.30035.0). Share this post Link to post
Jan Rysavy 8 Posted May 22, 2023 (edited) I tried to load the PDB using the current version of C:\Program Files\Microsoft Visual Studio\2022\Community\DIA SDK (14.36.32532.0) and the method loadDataFromPdb returns the error E_PDB_FORMAT. The same test with older msdia140.dll (14.29.30035.0) works. Edited May 22, 2023 by Jan Rysavy Share this post Link to post
Stefan Glienke 2009 Posted May 22, 2023 This could be some relevant info: https://randomascii.wordpress.com/2023/03/08/when-debug-symbols-get-large/ Share this post Link to post
Anders Melander 1795 Posted May 22, 2023 28 minutes ago, Stefan Glienke said: This could be some relevant info: https://randomascii.wordpress.com/2023/03/08/when-debug-symbols-get-large/ Looks like it. 1 hour ago, Jan Rysavy said: This also probably explains why Profiler in Visual Studio (2022) stopped displaying function names in recent versions. It uses newer versions of msdia140.dll. Ooooh, interesting. Maybe they've accidentally broken support for the older format and not noticed it because they're only testing the new format now. The article Stefan linked to makes me think that even though the PDB format supported large PDB files, the PDB reader (msdia140.dll) didn't. Otherwise, they would only have had to update their PDB writer to support large PDB files. Share this post Link to post
shineworld 73 Posted May 22, 2023 2 hours ago, Jan Rysavy said: I can confirm replacing C:\Program Files (x86)\Intel\oneAPI\vtune\2023.1.0\bin64\amplxe_msdia140.dll (version 14.34.31942.0) with version 14.28.29910.0 from VTune 2022.4.1 solves this problem! Can you attach the working DLL to try ? Share this post Link to post
Jan Rysavy 8 Posted May 22, 2023 26 minutes ago, shineworld said: Can you attach the working DLL to try ? msdia140.zip 1 Share this post Link to post
Jan Rysavy 8 Posted May 22, 2023 MS released debug symbols for msdia140.dll... nice Share this post Link to post
shineworld 73 Posted May 22, 2023 (edited) Seems to work.... Original version raise error on PDB file reading Edited May 22, 2023 by shineworld Share this post Link to post
Stefan Glienke 2009 Posted May 22, 2023 @Anders Melander Might be worth looking into https://github.com/llvm/llvm-project/blob/main/llvm/lib/DebugInfo/PDB/Native/PDBFileBuilder.cpp Share this post Link to post
mvanrijnen 123 Posted May 22, 2023 (edited) Just a quick question is the 16Mb limit still valid for the tool? i keep getting: "Cannot locate debugging information for file `D:\Data\company\Delphi\Projects\SVN\appname4\Clients\appname\Win32\Debug\appname.exe'. Cannot match the module with the symbol file `D:\Data\company\Delphi\Projects\SVN\appname4\Clients\appname\Win32\Debug\appname.pdb'. Make sure to specify the correct path to the symbol file in the Binary/Symbol Search list of directories." While the .pdb is there. (i replaced the msdia dll) works, replaced the wrong msdia.dll 🙂 Edited May 22, 2023 by mvanrijnen Share this post Link to post
shineworld 73 Posted May 22, 2023 13 minutes ago, mvanrijnen said: 8 minutes ago, mvanrijnen said: Just a quick question is the 16Mb limit still valid for the tool? I'm not an expert with PDB but that is what I've done and now seems to work: - Installed the latest version of VTune Profiler (2023.1.0). - Changed the original amplxe_media140.dll with an old version from Jan Rysavy (previous posts). - Enabled map in Detailed mode. - Compiled a project of 1.359.947 code lines (including library sources) which generate a map file of 49.696.190 bytes. - Executed map2pdb.exe rosettacncpph1.64.map -bind -v D:\x\develop\qem\rosetta_cnc_1>map2pdb.exe rosettacncpph1.64.map -bind -v map2pdb - Copyright (c) 2021 Anders Melander Version 2.8.0 Constructed a new PDB GUID: {F2D8CB4B-DA08-4BBD-A399-DBC449AF1364} Output filename not specified. Defaulting to rosettacncpph1.64.pdb Reading MAP file - Segments - Modules - Symbols Warning: [116390] Failed to resolve symbol to module: [0004:00000000000002C8] SysInit.TlsLast Warning: [116392] Failed to resolve symbol to module: [0003:00000000FE7F6000] SysInit.__ImageBase - Line numbers Collected 3.996 modules, 182.925 symbols, 525.223 lines, 985 source files Constructing PDB file - Collecting source file names - Module streams - Strings stream - PDB Info stream - TPI stream - Symbols stream - DBI stream - IPI stream - Finalizing PDB file - 9.068 blocks written in 3 intervals PE filename not specified. Defaulting to rosettacncpph1.64.exe Patching PE file - PE32+ image (64-bit) - Adding .debug section. - PDB file name has been stored in debug data. - PE file has been updated. Elapsed time: 00:00:00.895 This has generated a PDB file of 37.142.528 bytes - Started profiling of EXE in VTune profiler which generates this log at the profiling stop: Data collection is completed successfully May 22 2023 18:06:59 The result file 'XXX\r001hs\r001hs.vtune' is successfully created and added to the project . Finalization completed with warnings May 22 2023 18:08:33 Result finalization has completed with warnings that may affect the representation of the analysis data. Please see details below. Cannot locate debugging information for file `C:\WINDOWS\System32\msvcrt.dll'. Cannot locate debugging information for file `C:\WINDOWS\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.19041.1110_none_60b5254171f9507e\COMCTL32.dll'. Cannot locate debugging information for file `C:\WINDOWS\WinSxS\amd64_microsoft.windows.gdiplus_6595b64144ccf1df_1.1.19041.2251_none_91a40448cc8846c1\gdiplus.dll'. Cannot locate debugging information for file `C:\WINDOWS\SYSTEM32\ntdll.dll'. Cannot locate debugging information for file `C:\WINDOWS\System32\GDI32.dll'. Cannot locate debugging information for file `C:\WINDOWS\System32\SETUPAPI.DLL'. Cannot locate debugging information for file `C:\WINDOWS\System32\KERNEL32.DLL'. Cannot locate debugging information for file `C:\WINDOWS\System32\cfgmgr32.dll'. Cannot locate debugging information for file `C:\WINDOWS\System32\gdi32full.dll'. Cannot locate debugging information for file `C:\Program Files\Bitdefender\Endpoint Security\bdhkm\dlls_266262988153465131\bdhkm64.dll'. Cannot locate debugging information for file `C:\WINDOWS\System32\user32.dll'. Cannot locate debugging information for file `C:\WINDOWS\System32\combase.dll'. Cannot locate debugging information for file `C:\Program Files\Bitdefender\Endpoint Security\atcuf\dlls_266575548366517634\atcuf64.dll'. Cannot locate debugging information for file `C:\WINDOWS\system32\mswsock.dll'. Cannot locate debugging information for file `C:\WINDOWS\system32\uxtheme.dll'. Cannot locate debugging information for file `C:\WINDOWS\System32\ole32.dll'. Cannot locate debugging information for file `C:\WINDOWS\SYSTEM32\opengl32.dll'. Cannot locate debugging information for file `C:\WINDOWS\SYSTEM32\DEVOBJ.dll'. Cannot locate debugging information for file `C:\WINDOWS\System32\KERNELBASE.dll'. Cannot locate debugging information for file `C:\WINDOWS\SYSTEM32\TextShaping.dll'. Cannot locate debugging information for file `C:\Windows\System32\msxml6.dll'. Cannot locate debugging information for file `C:\WINDOWS\System32\WS2_32.dll'. Cannot locate debugging information for file `C:\WINDOWS\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_c1a085cc86772d3f\nvoglv64.dll'. Cannot locate debugging information for file `C:\WINDOWS\SYSTEM32\HID.DLL'. Cannot locate debugging information for file `C:\WINDOWS\System32\win32u.dll'. Cannot locate debugging information for file `C:\Program Files (x86)\Intel\oneAPI\vtune\latest\bin64\tpsstool.dll'.: And all symbols project symbols are visible (not for below list of DLL): Share this post Link to post
Anders Melander 1795 Posted May 22, 2023 45 minutes ago, Stefan Glienke said: @Anders Melander Might be worth looking into https://github.com/llvm/llvm-project/blob/main/llvm/lib/DebugInfo/PDB/Native/PDBFileBuilder.cpp Yeah... Not too keen on that as a first approach. The last time I tried using the llvm pdb support as a reference I wasted a lot of time before I found out that it was very incomplete to the point of being unusable by VTune. It has probably improved but since then it's hard to tell what state it's in. https://github.com/llvm/llvm-project/issues/37279 https://github.com/llvm/llvm-project/issues/28528 I will try to see if I can reproduce and spot the problem in the source before I go down that road. Thanks anyway. Share this post Link to post
Anders Melander 1795 Posted May 22, 2023 2 hours ago, Jan Rysavy said: MS released debug symbols for msdia140.dll... nice Neat. If you can spot where it gives up on the pdb file and returns an error that would be suuuuper nice. Does it produce any debug output while loading? Share this post Link to post
Anders Melander 1795 Posted May 22, 2023 3 hours ago, Stefan Glienke said: This could be some relevant info: https://randomascii.wordpress.com/2023/03/08/when-debug-symbols-get-large/ Quote we added a use_large_pdbs build setting which would switch the PDB page size to 8 KiB. However this setting initially needed to be off by default due to a lack of complete tool support. If the block size being 4096 is the only problem (I somehow doubt that I'm that lucky) then this is the line that needs to be changed to write 8192-byte blocks: https://bitbucket.org/anders_melander/map2pdb/src/2341200827af24f7dd75cb695a668dfa9564bcf5/Source/debug.info.writer.pdb.pas#lines-225 constructor TDebugInfoPdbWriter.Create; begin Create(4096); end; Share this post Link to post
Jan Rysavy 8 Posted May 22, 2023 4 hours ago, Anders Melander said: Does it produce any debug output while loading? Seems there is some logger framework, probably undocumented. In msdia140.zip is attached 'wt' command output for msdia140!CDiaDataSource::loadDataFromPdb. msdia140.zip 1 Share this post Link to post
Anders Melander 1795 Posted May 22, 2023 35 minutes ago, Jan Rysavy said: In msdia140.zip is attached 'wt' command output for msdia140!CDiaDataSource::loadDataFromPdb. Excellent! A few quick observations: First of all, it's strange that the error isn't logged. That would StrmTbl::internalSerializeBigMsf Lots of calls to this. I'm guessing it's reading MSF blocks and deblocking them into linear memory streams. This is probably the new code that supports the 8192-byte "big" MSF block size. MSF_HB::load Probably the code that loads the PDB tables from the memory streams. StrmTbl::~StrmTbl Lots of calls to this. Probably clean up after the load has been aborted. PortablePDB::PortablePDB Something wrong here. "Portable PDB" is the .NET PDB format. It's a completely different file format. I'm guessing it's falling back to that format after failing to validate the file as PDB. Share this post Link to post
Jan Rysavy 8 Posted May 22, 2023 15 minutes ago, Anders Melander said: PortablePDB::PortablePDB Something wrong here. "Portable PDB" is the .NET PDB format. It's a completely different file format. I'm guessing it's falling back to that format after failing to validate the file as PDB. Yes, see attached 'wt' output from old msdia140.dll 14.29.30035.0 for the same input PDB file. In this case loadDataFromPdb succeeds. msdia14_142930035.zip Share this post Link to post
Anders Melander 1795 Posted May 22, 2023 39 minutes ago, Jan Rysavy said: Yes, see attached 'wt' output from old msdia140.dll 14.29.30035.0 for the same input PDB file. In this case loadDataFromPdb succeeds. msdia14_142930035.zip Ew! It looks like they have done a complete rewrite. No wonder it's broken. So I guess this is an example of the main problem with the PDB format: Microsoft considers it their own internal format to do with what they like. They have their own (undocumented) writer, their own reader, and no documentation. Share this post Link to post