MAP2PDB - Profiling with VTune

[Anders Melander 1783]

28 minutes ago, Attila Kovacs said:

this tool works on the pdb of the test app but does not show anything on a bigger project

 

https://github.com/Microsoft/microsoft-pdb/blob/master/cvdump/cvdump.exe

 

there is somewhere just a tiny glitch I'm sure

Yes. I've been using cvdump and llvm-pdbutil to validate my pdb files along the way. How large is the pdb file?

 

llvm-pdbutil is fairly tolerant while cvdump mostly just gives up and does nothing if there is a problem.

cvdump uses the DIA API to read information from the pdb, which is the same API that VTune uses.

 

You can download llvm-pdbutil here: https://github.com/shaharv/llvm-pdbutil-builds/releases

Use the command: llvm-pdbutil dump -all <pdb-filename> I suggest you pipe the output to a file and then examine the file.

[Attila Kovacs 629]

@Anders Melander 8mb/14mb I can send you if you want. Jetbrains dotPeek says that it can not read beyond the stream. Also, offset size / files count mismatch.

The latter is at the end of the file right before the filenames, ~3rd field in the table.

 

 

[Anders Melander 1783]

52 minutes ago, Kas Ob. said:

You can fix this by finding the address (offset) from the PE header itself

I can't just ignore the section offsets from the map file since all the other offsets in it are relative to them. I would have to relocate all offsets then.

Edit: That was obviously nonsense. I wouldn't have to relocate the offsets exactly because they're relative.

 

52 minutes ago, Kas Ob. said:

Also you can see why i called the start addresses (offests) by Delphi linker map file are wrong and should be relative to 0 instead of $400000.

The TLS section offset is wrong but I believe all other offsets are correct. VTune seems to agree...

Edited April 4, 2021 by Anders Melander

[Anders Melander 1783]

4 minutes ago, Attila Kovacs said:

Jetbrains dotPeek says that it can not read beyond the stream.

I think .NET pdb files are in a different format.

[Attila Kovacs 629]

Just now, Anders Melander said:

I think .NET pdb files are in a different format.

It's true, that's why I tried to read all kind of pdb's with it to see if it fails on them but could not find any.

[Anders Melander 1783]

23 minutes ago, Attila Kovacs said:

8mb/14mb I can send you if you want.

Please do. I'll need both the map file and the pdb file.

 

24 minutes ago, Attila Kovacs said:

Also, offset size / files count mismatch.

The latter is at the end of the file right before the filenames, ~3rd field in the table.

I'm not sure what info you're referring to here. What table?

[Anders Melander 1783]

@Attila Kovacs I found and fixed the problem; The size of the MSF Directory stream had grown beyond 1 block and my implementation of the MSF Directory Index assumed that a one block directory would be enough (who needs more than 640K anyway?) so I only wrote one block pointer.

A module stream is created for each unit so if there are many units in a project there will be many stream and thus many entries in the directory. In your case there were 1775 streams in total.

 

@David Heffernan I've also implemented support for PE32+ (64 bit). I guess I'm procrastinating to avoid handling the 16Mb limit

 

New version uploaded.

[Attila Kovacs 629]

great! thank you. I'll test is right now. the 16Mb+ is "just" some recursive stuff, looking forward to it!

Edited April 4, 2021 by Attila Kovacs

[Attila Kovacs 629]

yes yes yes yes yes yes yes! finally we can debug too! @Anders Melander it works! \o/

[Wagner Landgraf 43]

Very interesting tool, I believe it opens lots of possibilities. I use AQTime and Sampling Profiler, is anyone able to tell the differences and advantages of VTune compared to those two tools?

[Vincent Parrett 750]

8 hours ago, Anders Melander said:

You're probably hitting the 16Mb limit,

It wasn't that, and I probably did forget the bind switch before I sent it to you, however with the new version.... we have lift off! Awesome work!

 

E:\map2pdb>map2pdb.exe I:\FBAT_HG9\Output\FB9\FinalBuilder9.map -v -bind:I:\FBAT_HG9\Output\FB9\FinalBuilder9.exe
map2pdb - Copyright (c) 2021 Anders Melander
Version 2.0

Constructed a new PDB GUID: {CE1E6D08-67E6-4B5B-B42A-3FC719254F1E}
Output filename not specified. Defaulting to FinalBuilder9.pdb
Reading MAP file
- Segments
Warning: [    8] Empty segment: .pdata [0006:00400000]
- Modules
- Symbols
Warning: [20671] Failed to resolve symbol to module: [0005:00000608] SysInit.TlsLast
Warning: [20673] Failed to resolve symbol to module: [0004:FF77A000] SysInit.__ImageBase
- Line numbers
Warning: [71563] Line number with zero offset ignored. Module:VSoft.IDE.Types, Segment:0001, Source:VSoft.IDE.Types.pas, Line:49
Warning: [71567] Line number with zero offset ignored. Module:VSoft.IDE.Interfaces, Segment:0001, Source:VSoft.IDE.Interfaces.pas, Line:328
Warning: [75481] Line number with zero offset ignored. Module:LMDRTLConst, Segment:0001, Source:LMDRTLConst.pas, Line:213
Warning: [75621] Line number with zero offset ignored. Module:intfLMDBase, Segment:0001, Source:intfLMDBase.pas, Line:44
Warning: [76980] Line number with zero offset ignored. Module:LMDDckCst, Segment:0001, Source:LMDDckCst.pas, Line:29
Warning: [79649] Line number with zero offset ignored. Module:VSoft.IDE.ActionListInterfaces, Segment:0001, Source:VSoft.IDE.ActionListInterfaces.pas, Line:136
Warning: [90056] Line number with zero offset ignored. Module:SynEditStrConst, Segment:0001, Source:SynEditStrConst.pas, Line:577
Warning: [98436] Line number with zero offset ignored. Module:VSoft.IDE.WizardManagerService, Segment:0001, Source:VSoft.IDE.WizardManagerService.pas, Line:61
Warning: [99799] Line number with zero offset ignored. Module:dwTaskbarList, Segment:0001, Source:dwTaskbarList.pas, Line:213
Warning: [103835] Line number with zero offset ignored. Module:RestException, Segment:0001, Source:RestException.pas, Line:26
Warning: [105461] Line number with zero offset ignored. Module:VSoft.Shared.UXLControlIntf, Segment:0001, Source:VSoft.Shared.UXLControlIntf.pas, Line:60
Warning: [107033] Line number with zero offset ignored. Module:VSoft.IDE.MS.ActionCollection.Interfaces, Segment:0001, Source:VSoft.IDE.MS.ActionCollection.Interfaces.pas, Line:1085
Constructing PDB file
- Collecting source file names
- Module streams
- Strings stream
- PDB Info stream
- TPI stream
- Symbols stream
- DBI stream
- IPI stream
- Finalizing PDB file
Patching PE file
- PE32 image (32-bit)
- PDB file name has been stored in debug data.
- Header checksum has been cleared.
- PE file has been updated.

 

I was also able to create a pdb for the package

E:\map2pdb>map2pdb.exe I:\FBAT_HG9\Output\FB9\VSoft.Core.map -bind:I:\FBAT_HG9\Output\FB9\VSoft.Core.bpl -v
map2pdb - Copyright (c) 2021 Anders Melander
Version 2.0

Constructed a new PDB GUID: {76323C0F-0787-44F8-BC72-2B6ACC9CBF82}
Output filename not specified. Defaulting to VSoft.Core.pdb
Reading MAP file
- Segments
Warning: [    7] Empty segment: .tls [0005:00000000]
Warning: [    8] Empty segment: .pdata [0006:00400000]
- Modules
Warning: [ 1682] Module exceed segment bounds - ignored: OtlCommon.Utils [0005:00000000+256]
- Symbols
Warning: [ 6403] Failed to resolve symbol to module: [0005:00000000] OtlCommon.Utils.LastThreadName
Warning: [ 8705] Failed to resolve symbol to module: [0005:00000100] SysInit.TlsLast
Warning: [ 8709] Failed to resolve symbol to module: [0004:FFBF8000] SysInit.__ImageBase
- Line numbers
Warning: [64147] Line number offset out of range for module. Offset:003DAEEF, Module:.VSoft.Core [003DAEF0 - 003E5738]
Constructing PDB file
- Collecting source file names
- Module streams
- Strings stream
- PDB Info stream
- TPI stream
- Symbols stream
- DBI stream
- IPI stream
- Finalizing PDB file
Patching PE file
- PE32 image (32-bit)
- PDB file name has been stored in debug data.
- Header checksum has been cleared.
- PE file has been updated.

Now to hook it into every project in the group and see how it goes 😃 Thanks again for this awesome effort.. my only question now is, if you can do this in your spare time, why TF can't embarcadero do it 🙄

[Anders Melander 1783]

6 minutes ago, Vincent Parrett said:

my only question now is, if you can do this in your spare time, why TF can't embarcadero do it 🙄

Priorities. They probably don't have the 240+ hours to spare plus the time it will take to integrate into the product and maintain it afterwards. Also it's only fairly recently that this has become possible to do at all, after MS released the PDB source and LLVM deciphered them. And even LLVM still has problems with it.

[Vincent Parrett 750]

6 minutes ago, Anders Melander said:

Priorities. They probably don't have the 240+ hours to spare plus the time it will take to integrate into the product and maintain it afterwards. Also it's only fairly recently that this has become possible to do at all, after MS released the PDB source and LLVM deciphered them. And even LLVM still has problems with it.

Well that's what happens when you strip the team down to the bare bones and then hire short term contractors, maintaining things becomes a problem when your teams domain knowledge is low. 

 

FWIW, 5 minutes in and I already fixed a perf issue in the area that I have been trying to optimise for the last week (replaced a list with a hashset) - a small win but I'll take any win I can get right now😃

 

Pity embarcadero don't ship map files for their bpl's - hmmm will log a qc for that - because as I dig deeper into my perf issues I end up in the weeds in rtl etc with func@xxx a lot. 

[Anders Melander 1783]

1 minute ago, Vincent Parrett said:

Well that's what happens when you strip the team down to the bare bones and then hire short term contractors, maintaining things becomes a problem when your teams domain knowledge is low

Amen to that.

 

2 minutes ago, Vincent Parrett said:

Pity embarcadero don't ship map files for their bpl's - hmmm will log a qc for that - because as I dig deeper into my perf issues I end up in the weeds in rtl etc with func@xxx a lot. 

Ask them to ship pdb files. Now there's no excuse not to

Or even better; Publish an API to read their debug info so we don't have to rely on the map files. I'm sure they have one.

[Vincent Parrett 750]

2 minutes ago, Anders Melander said:

Ask them to ship pdb files. Now there's no excuse not to

Or even better; Publish an API to read their debug info so we don't have to rely on the map files. I'm sure they have one.

Agree, there is already a few issues asking for pdb support that have lots of votes and have been open for years. @Marco Cantu @David Millington FYI. 

[Vincent Parrett 750]

This is an odd one, the package is compiled with debug info etc, but I get this output :

 

:\map2pdb>map2pdb.exe I:\FBAT_HG9\Output\FB9\VSoft.Core.DB.map -bind:I:\FBAT_HG9\Output\FB9\VSoft.Core.DB.bpl -v
map2pdb - Copyright (c) 2021 Anders Melander
Version 2.0

Constructed a new PDB GUID: {B1FCFCB0-3F09-4207-BB64-0D1D08101C3F}
Output filename not specified. Defaulting to VSoft.Core.DB.pdb
Reading MAP file
- Segments
Warning: [    7] Empty segment: .tls [0005:00000000]
Warning: [    8] Empty segment: .pdata [0006:00400000]
- Modules
Warning: [  791] Module exceed segment bounds - ignored: nxllConst [0005:00000000+4]
Warning: [  792] Module exceed segment bounds - ignored: nxllSync [0005:00000004+12]
Warning: [  793] Module exceed segment bounds - ignored: nxllUtils [0005:00000010+8]
Warning: [  794] Module exceed segment bounds - ignored: nxllException [0005:00000018+12]
Warning: [  795] Module exceed segment bounds - ignored: nxllPlatformInterface [0005:00000024+4]
Warning: [  796] Module exceed segment bounds - ignored: nxsdDataDictionary [0005:00000028+4]
Warning: [  797] Module exceed segment bounds - ignored: nxsdServerEngine [0005:0000002C+4]
Warning: [  798] Module exceed segment bounds - ignored: nxsrFileImplWin32 [0005:00000030+8]
Warning: [  799] Module exceed segment bounds - ignored: nxsrTimeoutHandling [0005:00000038+56]
Warning: [  800] Module exceed segment bounds - ignored: nxseHeapEngineCached [0005:00000070+4]
Warning: [  801] Module exceed segment bounds - ignored: nxsrServerEngine [0005:00000074+4]
Warning: [  802] Module exceed segment bounds - ignored: nxllTransport [0005:00000078+4]
Warning: [  803] Module exceed segment bounds - ignored: nxrbCallContext [0005:0000007C+4]
Warning: [  804] Module exceed segment bounds - ignored: nxptBasePooledTransport [0005:00000080+24]
Warning: [  805] Module exceed segment bounds - ignored: nxsqlTableExp [0005:00000098+4]
Warning: [  806] Module exceed segment bounds - ignored: nxsqlCondExp [0005:0000009C+4]
Warning: [  807] Module exceed segment bounds - ignored: nxsqlProxies [0005:000000A0+4]
Warning: [  808] Module exceed segment bounds - ignored: nxdb [0005:000000A4+4]
Warning: [  809] Module exceed segment bounds - ignored: nxllSocket [0005:000000A8+4]
- Symbols
Warning: [ 2213] Failed to resolve symbol to module: [0005:000000A4] nxdb._nxDefaultSession
Warning: [ 3236] Failed to resolve symbol to module: [0005:00000000] nxllConst._CurrentThreadNxVersion
Warning: [ 3391] Failed to resolve symbol to module: [0005:00000020] nxllException._CurrentExceptionHookInfo
Warning: [ 3395] Failed to resolve symbol to module: [0005:00000018] nxllException._LastError
Warning: [ 3396] Failed to resolve symbol to module: [0005:0000001C] nxllException._LastMessage
Warning: [ 4052] Failed to resolve symbol to module: [0005:00000024] nxllPlatformInterface.nxIsUIThread
Warning: [ 4216] Failed to resolve symbol to module: [0005:000000A8] nxllSocket.SocketErrorProc
Warning: [ 4455] Failed to resolve symbol to module: [0005:00000008] nxllSync.nxtv_BreathCallback
Warning: [ 4649] Failed to resolve symbol to module: [0005:00000078] nxllTransport.nxitvTransport
Warning: [ 4990] Failed to resolve symbol to module: [0005:00000014] nxllUtils._ThreadCodePage
Warning: [ 4991] Failed to resolve symbol to module: [0005:00000010] nxllUtils._ThreadLocale
Warning: [ 5545] Failed to resolve symbol to module: [0005:00000080] nxptBasePooledTransport.nxitvReplyInfos
Warning: [ 5732] Failed to resolve symbol to module: [0005:00000090] nxptBasePooledTransport._Interval
Warning: [ 5733] Failed to resolve symbol to module: [0005:00000094] nxptBasePooledTransport._Mask
Warning: [ 5734] Failed to resolve symbol to module: [0005:00000088] nxptBasePooledTransport._NetIdle
Warning: [ 5757] Failed to resolve symbol to module: [0005:0000007C] nxrbCallContext._CurrentCallContext
Warning: [ 7123] Failed to resolve symbol to module: [0005:00000028] nxsdDataDictionary._DontLoadID
Warning: [ 9598] Failed to resolve symbol to module: [0005:0000002C] nxsdServerEngine._SessionStatsCurrent
Warning: [10232] Failed to resolve symbol to module: [0005:00000070] nxseHeapEngineCached._FindBlockNumber
Warning: [12890] Failed to resolve symbol to module: [0005:0000009C] nxsqlCondExp._CurrentlyProcessedContains
Warning: [14747] Failed to resolve symbol to module: [0005:000000A0] nxsqlProxies.nxsqlNoProcessingCounter
Warning: [16463] Failed to resolve symbol to module: [0005:00000098] nxsqlTableExp._LockUnbindIfRequired
Warning: [17204] Failed to resolve symbol to module: [0005:00000030] nxsrFileImplWin32.FileBlocks
Warning: [17207] Failed to resolve symbol to module: [0005:00000034] nxsrFileImplWin32.HighestBlock
Warning: [18261] Failed to resolve symbol to module: [0005:00000074] nxsrServerEngine._ForRecover
Warning: [18471] Failed to resolve symbol to module: [0005:00000038] nxsrTimeoutHandling.nxtv_TimeoutHandling
Warning: [18617] Failed to resolve symbol to module: [0005:000000AC] SysInit.TlsLast
Warning: [18621] Failed to resolve symbol to module: [0004:FFC91000] SysInit.__ImageBase
- Line numbers
Constructing PDB file
- Collecting source file names
- Module streams
- Strings stream
- PDB Info stream
- TPI stream
- Symbols stream
- DBI stream
- IPI stream
- Finalizing PDB file
Patching PE file
- PE32 image (32-bit)
Image does not contain a debug directory address - please link with debug info enabled

bpl, map and pdb attached. 

VSoft_Core.DB.zip

[Anders Melander 1783]

7 minutes ago, Vincent Parrett said:

This is an odd one, the package is compiled with debug info etc, but I get this output :

Well, I'm not sure what I can do about that because the error message is correct. There is no debug info in the file:

 

I could of course write a new debug directory instead of relying on there being one already but I don't know how to do that.

I looked into it at one point and AFAIR I think I remember that it could become fairly complicated once you start adding stuff to a PE file.

[jbg 239]

43 minutes ago, Vincent Parrett said:

Pity embarcadero don't ship map files for their bpl's - hmmm will log a qc for that - because as I dig deeper into my perf issues I end up in the weeds in rtl etc with func@xxx a lot. 

They ship jdbg files with the BPLs and those are created with the JCL from the map files. So you can read them instead of the map file with the help of the JclDebug.pas (that also has a map file parser and a TD32 debug information parser).

 

There are only two problems with that approach:

1. The JclDebug API only has an "Address to ProcName/LineNum" function but doesn't give you access to the actual ProcNames and Lines lists. But you can change the code and access the private fields (TJclBinDebugScanner).
2. The jdbg files for some files are from the debug build and don't match the actual release code (but that got a lot better in recent releases)  or they are from a broken Borland C++ compiler that generates map files with just wrong information (e.g. Delphi compilers).

[Vincent Parrett 750]

2 minutes ago, Anders Melander said:

Well, I'm not sure what I can do about that because the error message is correct. There is no debug info in the file:

Weird since the file is definitely being compiled with debug info. 

 

I added this to test

 

{$IFOPT D+}
 Well debug info is turned on so this should not compile
{$ENDIF}

and it hits the error. I'll investigate further.

[Vincent Parrett 750]

I recreated the dproj file and now it works.. gotta love the mess that is dproj files upgraded through multiple delphi versions 🙄

[Renate Schaaf 64]

Is there any chance that AMD uProf can be used with this?

[Attila Kovacs 629]

20 minutes ago, Renate Schaaf said:

Is there any chance that AMD uProf can be used with this?

yep:

Debug info formats:
▪ PDB, COFF, DWARF, STABS

 

[Vincent Parrett 750]

I just had my first big win with VTune 😃

 

I was looking into improving the performance of loading projects in FinalBuilder 9 (in dev) - we have some customers with huge projects that were taking a while to load. Profiling with VTune showed that most of the time was spent in the project lexer/parser - the lexer created class based tokens (and some other associated classes) and a lot of time was spent in the lexer and the memory manager. So 3 hours later, the code has been converted to use records rather than classes - the unit tests all pass (I only had to comment out all the Assert.IsNotNull(token); calls) and the application runs normally (still need to code review to make sure I didn't break things).

 

The result is around a 30% improvement with that change alone! That's just me counting out loud as the project loads 😉 - I'll do more formal timing/testing tomorrow. I also just compared to FinalBuilder 8 and the total improvement is more like 60% - I'll put that down to some manual code review looking for possible hotspots over the last week, and also to switching from the rtl generic collections to using spring4d everywhere! 

Time for some sleep, but looking forward to more VTuning tomorrow.

 

Thanks @Anders Melander again for this amazing bit of work!

[Anders Melander 1783]

Thank you for the nice feedback @Vincent Parrett. Nighty night!

[Anders Melander 1783]

I have now implemented MSF intervals so the 16Mb pdb limit should be gone.

I have however not been able to verify that it actually works beyond validating the files with cvdump and llvm-pdbutil because VTune hangs, burning CPU, when it tries to resolve symbols from the large pdb files I've thrown at it (the smaller ones still works). This could be a bug in VTune, it could be that I just need to wait longer than the 30 minutes I've tried or it could be that there's either something wrong with the pdb file or that I need to order the symbols in it differently. From examining the call stacks of the VTune threads it appears as if it's busy doing symbol lookup through msdia140.dll.

 

Anyhow, source committed and new precompiled uploaded. Give it a go and let me know if you have better luck than me.

 

P.S.

On 4/4/2021 at 9:58 PM, Attila Kovacs said:

the 16Mb+ is "just" some recursive stuff

It turned out to be just a tiny bit more complex than just some recursive stuff

https://bitbucket.org/anders_melander/map2pdb/commits/95353cbf3ba5b284919e617ace970b02ca0bf0e2