Profiler for Delphi

[Anders Melander 1795]

20 minutes ago, Willicious said:

any suggestions?

 

[Willicious 8]

8 minutes ago, Anders Melander said:

 

I take it VTune is the Profiler... what is the MAP2PDB for?

 

Complete beginner here, I've been advised to Profile my project's codebase to look for permance issues. Not sure where to start.

Also, isn't VTune an expensive Intel app? Ideally looking for free, or inexpensive alternatives.

VTune installer says it can integrate with MS Visual Studio, but not RAD Studio - is this even a Delphi app?

Edited July 13, 2023 by Willicious

[Anders Melander 1795]

15 minutes ago, Willicious said:

I take it VTune is the Profiler... what is the MAP2PDB for?

Did you read the page linked to? First paragraph states:

Quote

map2pdb is a tool used to convert the MAP files produced by the Delphi and C++ Builder compilers to Microsoft PDB files for use in tools that support that format.

[edit] I guess you don't know what the PDB file is for.

The PDB file is used by the profiler to map the addresses in the application being profiled to source files, function names, and line numbers. Without that information, the profiler would only be able to show you the raw addresses.

 

15 minutes ago, Willicious said:

Not sure where to start.

1. Download and install VTune.
2. Download map2pdb, extract the exe, and save it somewhere of your choice. For example c:\tools\map2pdb\map2pdb.exe
3. Add a menu item in the Delphi IDE via Tools -> Configure Tools...
   
4. The parameters in the above are: -debug -v -pause -bind:$EXENAME $PATH($EXENAME)$NAMEONLY($EXENAME).map -include:0001
5. Make sure the compiler are generating a full map file:
   
6. Compile your project.
7. Execute the map2pdb tool action.
8. Launch VTune and create a profiler project for your exe.
9. Profile the project in VTune.
10. Profit!

Edited July 13, 2023 by Anders Melander

[Willicious 8]

Apologies, I appreciate the suggestion and the time you've taken to respond, but this isn't quite what I'm looking for.

 

I need a program that will basically "read" the entire codebase - ideally at source or within the IDE itself - and point out any areas which could be removed, simplified, or optimized to reduce performance bottlenecks.

 

Maybe I'm dreaming and such a thing doesn't exist yet...? I just know that AQTime used to be bundled with RAD Studio but isn't any longer (and, from what I've seen, this does exactly what I'm describing, but is currently mad expensive for someone who only does small open-source projects).

 

Also - I'm looking for somthing that will do the above, and which works straightaway with an absolute minimum of setup. If that doesn't exist, then that's fine. Maybe it will someday 😊

Edited July 16, 2023 by Willicious

[David Heffernan 2347]

That's not what AQtime does. Vtune is essentially analogous to AQtime but far more capable. 

 

You aren't going to find this magic solution that you want, whereby you wave a magic wand and your program is suddenly much faster. Optimisation though is going to take effort from you to understand your program. No shortcuts. 

[Darian Miller 365]

On 7/16/2023 at 4:21 PM, Willicious said:

I need a program that will basically "read" the entire codebase

 

Profilers measure application execution.  Two main types are Sampling and Instrumenting profilers.  You run the application and the profiler measures times for code to complete.  

 

It sounds like what you want is some sort of AI to read/parse your code and suggest improvements.  You could try feeding pieces to ChatGPT and see how well that performs...

 

Or, perhaps you want a static code analyzer, like Pascal Analyzer from Peganza....https://www.peganza.com/products_pal.html   it will parse your code and provide reports of things that could be improved.  (They offer a free, limited feature version to try....Pascal Analyzer Lite: https://www.peganza.com/download.html)

 

 

 

[Fr0sT.Brutal 900]

Why you want to modify the code? If you encounter perf issues then profiler will help. If you have buggy areas - try static analyzer or AI or a programmer. If you want some fragments just simplified - well, why? To be more maintainable? Then just ask the maintainers 🙂

[Willicious 8]

On 7/18/2023 at 3:14 AM, Darian Miller said:

Or, perhaps you want a static code analyzer, like Pascal Analyzer from Peganza....https://www.peganza.com/products_pal.html   it will parse your code and provide reports of things that could be improved.  (They offer a free, limited feature version to try....Pascal Analyzer Lite: https://www.peganza.com/download.html)

 

Yes, this is what I'm after. Thanks! 🙂👍

[chkaufmann 17]

On 7/13/2023 at 6:29 PM, Anders Melander said:

The parameters in the above are: -debug -v -pause -bind:$EXENAME $PATH($EXENAME)$NAMEONLY($EXENAME).map -include:0001

Thanks for your instructions. I did like this, but when I call map2pdb I get a long list of errors. For example I get many lines like "Debug: Module has no source files: xxxxxxx" where xxxxxxx is the name of a unit, that is part of the project.

Then the only thing I can start in VTune is "Hotspots". This runs fine and at the end I get a summary report. But this is not really helpfull since the "Top Hotspots" are Windows calls only.

Using the Caller/Callee I can find my units, but when I open I don't see a lot: Percentage is there, CPU Time is zero most of the times and I don't see the number of calls.

 

I used AQTime before, but support and I have no working version anymore. They had a call tree and a graphic where you could easily navigate down all routines.

 

Maybe I just didn't find the correct view in VTune yet.

 

My questions

- Should I care about these errors when creating the pdb file? How to understand the error and how to solve it?

- Where can I find a good tutorial for VTune in order to get the info I used before with AQTime?

 

Thanks for any help

 

Christian

[Allen@Grijjy 44]

1 hour ago, chkaufmann said:

Thanks for your instructions. I did like this, but when I call map2pdb I get a long list of errors. For example I get many lines like "Debug: Module has no source files: xxxxxxx" where xxxxxxx is the name of a unit

Guess we are both working on this at the same time!  I had the same issue yesterday, but I figured out that -include:0001 (or whatever code segments you are analyzing) doesn't actually load the source lines.  I haven't debugged it fully yet, but if I switch to -include:* (using a filename wildcard instead) it works.   The main thing is to make sure that the -debug output says it loaded more than 0 lines.

Also, it has some issue with overlapping segments from the map files, so I had to make a source code change to work around it, but that issue may be project specific.

Edited October 26, 2023 by Allen@Grijjy

[Stefan Glienke 2009]

fwiw all parameters but passing the mapfile name (since the exe name is most likely the same and directly next to it) and the -bind parameter (which causes the PE header in the binary to be updated in order for VTune to properly use the pdb) are optional.

 

3 hours ago, chkaufmann said:

Where can I find a good tutorial for VTune in order to get the info I used before with AQTime?

VTune has some very extensive documentation and even a tour inside the application that teaches the basics

Edited October 26, 2023 by Stefan Glienke

[Anders Melander 1795]

4 hours ago, chkaufmann said:

Should I care about these errors when creating the pdb file? How to understand the error and how to solve it?

They are not errors.

The "debug" prefix tells you that they are diagnostic messages produced because you specified the -debug switch.

 

4 hours ago, chkaufmann said:

Where can I find a good tutorial for VTune in order to get the info I used before with AQTime?

I have no idea. I guess there's probably something on the VTune site or maybe some help installed with VTune.

 

2 hours ago, Allen@Grijjy said:

I figured out that -include:0001 (or whatever code segments you are analyzing) doesn't actually load the source lines.

-include and -exclude are filters that are applied after the map file has been parsed.

• If neither is specified, everything is included.
• If -include is specified, everything that doesn't satisfy the include filter is ignored.
• If -exclude is specified, everything that does satisfies the exclude filter is ignored.

 

2 hours ago, Allen@Grijjy said:

if I switch to -include:* (using a filename wildcard instead) it works.

-include:* does nothing since it just includes everything - which is the default.

 

2 hours ago, Allen@Grijjy said:

Also, it has some issue with overlapping segments from the map files, so I had to make a source code change to work around it, but that issue may be project specific.

Well, overlapping segments should never occur so that would be a bug in Delphi if you have that in the map file.

In the first version of your your other message, you stated that you encountered some exceptions but were able to work around them by modifying the source. I'll of course be interested to know exactly what the problem was and what you changed, so I can get it merged into the source if there really is a problem.

 

I'll probably also need to see the map files to get to the bottom of this. You can either create a (public) issue at Bitbucket or PM them to me here.

Edited October 26, 2023 by Anders Melander

[Allen@Grijjy 44]

20 hours ago, Anders Melander said:

-include and -exclude are filters that are applied after the map file has been parsed.

• If neither is specified, everything is included.
• If -include is specified, everything that doesn't satisfy the include filter is ignored.
• If -exclude is specified, everything that does satisfies the exclude filter is ignored.

This is where I am confused (and I have read all the existing documentation).  You recommend configuring a tool in Delphi with these parameters:

-debug -v -pause -bind:$EXENAME $PATH($EXENAME)$NAMEONLY($EXENAME).map -include:0001

 

When I do this, it will parse my map file correctly, but it won't load any source lines regardless of the segments that I include.  If I I change to something like this:

-v -pause -bind:$EXENAME $PATH($EXENAME)$NAMEONLY($EXENAME).map -include:0001;0003;0004

It also doesn't load any source lines.  

 

I checked this within the debugger directly.  It enumerates the files correctly, it just doesn't consume them into the FSourceLines collection.   

 

Is my syntax correct or am I supposed to be doing this:

-debug -v -pause -bind:$EXENAME $PATH($EXENAME)$NAMEONLY($EXENAME).map -include:0001;0002;*

so I am indicating both filenames and segments to include?

20 hours ago, Anders Melander said:

Well, overlapping segments should never occur so that would be a bug in Delphi if you have that in the map file.

In the first version of your your other message, you stated that you encountered some exceptions but were able to work around them by modifying the source. I'll of course be interested to know exactly what the problem was and what you changed, so I can get it merged into the source if there really is a problem.

I'll probably also need to see the map files to get to the bottom of this. You can either create a (public) issue at Bitbucket or PM them to me here.

 

I would be happy to assist in any way I can.  I think it's a great tool and would love to do an Embarcadero Coderage MVP session on how to use some of the industry standard profilers with your utility in the future, if that's okay.

 

I noticed other people have encountered the same issue with consuming overlapping segments.   In my case I modified the code to skip the overlapping segments instead of raising an exception and stopping.  For example, Delphi emitted the map file segments as:

 

 Start                 Length     Name                   Class
 0001:0000000000401000 006CD7B8H .text                   CODE
 0002:0000000000ACF000 000A9208H .data                   DATA
 0003:0000000000B79000 00011F20H .bss                    BSS
 0004:0000000000400000 00008260H .tls                    TLS
 0005:0000000000BDE000 0004FF5CH .pdata                  PDATA

 

TDebugInfoSegment.CheckOverlap subsequently raised an exception:

Project map2pdb.exe raised exception class Exception with message 'Overlapping segments: .tls [0004:0000000000400000] and .text [0001:0000000000401000]'.

Edited October 27, 2023 by Allen@Grijjy

[Anders Melander 1795]

22 minutes ago, Allen@Grijjy said:

This is where I am confused (and I have read all the existing documentation).  You recommend configuring a tool in Delphi with these parameters:

-debug -v -pause -bind:$EXENAME $PATH($EXENAME)$NAMEONLY($EXENAME).map -include:0001

Yes. This is the exact default setup that I use.

 

I have an additional two setups; One that includes everything (without the -include filter) and one that excludes a bunch of units (-exclude:dx*;cx*;winapi.*;vcl.*;data.*;firedac.*;soap.*;web.*;datasnap.*;id*;vcltee.*) to keep the size of the pdb down.

On a related note, I've been considering making a GUI for this so that the filters can be specified visually (I believe AQTime does this). Often you know exactly what part of the code you want to profile and having PDB info generated for the rest is just overhead.

 

32 minutes ago, Allen@Grijjy said:

Is my syntax correct or am I supposed to be doing this:

-debug -v -pause -bind:$EXENAME $PATH($EXENAME)$NAMEONLY($EXENAME).map -include:0001;0002;*

so I am indicating both filenames and segments to include?

No, that's not necessary.

By default, everything is included. It's only when you specify an inclusion filter that everything by default becomes excluded.

 

36 minutes ago, Allen@Grijjy said:

I think it's a great tool and would love to do an Embarcadero Coderage MVP session on how to use some of the industry standard profilers with your utility in the future, if that's okay.

Yes, of course it is. We really need more awareness among Delphi developers of the tools available to them.

 

40 minutes ago, Allen@Grijjy said:

I noticed other people have encountered the same issue with consuming overlapping segments.   In my case I modified the code to skip the overlapping segments instead of raising an exception and stopping.

That was my initial way of dealing with overlaps but the problem with that is that you don't know if the segment you're skipping is the "correct" one. And you might need the segment later on if there are references to it elsewhere in the map file.

So far I don't have enough test cases of map files with these overlapping segments to be able to make a decision on how best to handle them, so if I could get my hand on the one you have that would be great.

Also, if you could examine the corresponding exe file (I recommend using PE Tools) to find out what the actual segment offsets and sizes are (because they're obviously not what the map file states) that would be a big help.

 

1 hour ago, Allen@Grijjy said:

Start                 Length     Name                   Class
 0001:0000000000401000 006CD7B8H .text                   CODE
 0002:0000000000ACF000 000A9208H .data                   DATA
 0003:0000000000B79000 00011F20H .bss                    BSS
 0004:0000000000400000 00008260H .tls                    TLS
 0005:0000000000BDE000 0004FF5CH .pdata                  PDATA

The .tls segment does seem to be a common factor among the various segment-related problems. For example:

Again, I could just ignore segments with Offset=0 but then I would get errors later on as there are symbols referencing it. Another solution would be to hardcode handling of the .tls segment but I would really like to avoid a fragile workaround like that and keep the handling generic.

 

Right now I'm mostly leaning toward marking segments causing overlap, or with offset=0, as invalid and then simply ignore all later references to them.

 

The best would of course be if Embarcadero fixed the map file but I'm not holding my breath for that, and I still need to be able to handle what's already out there.

[Allen@Grijjy 44]

4 hours ago, Anders Melander said:

Yes. This is the exact default setup that I use.

 

If I use that command line, I always get '0' source lines.  Here is the output for the following:

-v -pause -bind:C:\Users\Allen\Desktop\Test1.exe C:\Users\Allen\Desktop\Test1.map -include:0001

---->

map2pdb - Copyright (c) 2021 Anders Melander
Version 3.0.1

Constructed a new PDB GUID: {33C40D0A-BC13-43D4-A6E3-D7D23966F480}
Output filename not specified. Defaulting to Test1.pdb
Reading MAP file
- Segments
- Modules
- Symbols
Warning: [36728] Failed to resolve symbol to module: [0004:0000000000008260] SysInit.TlsLast
Warning: [36731] Failed to resolve symbol to module: [0003:00000000FF887000] SysInit.__ImageBase
- Line numbers
Include filter eliminated 1,114 module(s), 60,204 symbol(s)
Collected 0 modules, 0 symbols, 0 lines, 280 source files
Constructing PDB file
- Collecting source file names
- Module streams
- Strings stream
- PDB Info stream
- TPI stream
- Symbols stream
- DBI stream
- IPI stream
- Finalizing PDB file
- 15 blocks written in 1 intervals
Patching PE file
- PE32+ image (64-bit)
- PDB file name has been stored in debug data.
- PE file has been updated.
Elapsed time: 00:00:00.246

 

If I take the exact same map file and do the following instead, Here is the output:

-v -pause -bind:C:\Users\Allen\Desktop\Test1.exe C:\Users\Allen\Desktop\Test1.map

---->

map2pdb - Copyright (c) 2021 Anders Melander
Version 3.0.1

Constructed a new PDB GUID: {44F58444-CC5E-4CA2-B03E-288B255CC688}
Output filename not specified. Defaulting to Test1.pdb
Reading MAP file
- Segments
- Modules
- Symbols
Warning: [36728] Failed to resolve symbol to module: [0004:0000000000008260] SysInit.TlsLast
Warning: [36731] Failed to resolve symbol to module: [0003:00000000FF887000] SysInit.__ImageBase
- Line numbers
Collected 1,114 modules, 60,204 symbols, 155,944 lines, 280 source files
Constructing PDB file
- Collecting source file names
- Module streams
- Strings stream
- PDB Info stream
- TPI stream
- Symbols stream
- DBI stream
- IPI stream
- Finalizing PDB file
ERangeError: Range check error
Elapsed time: 00:00:03.714
Done - Press enter to continue

Notice (2) things.  First I get all the sources lines, but before I did not.  Also I get a Range Check Error (at the bottom).  I am really using the syntax correctly?  I guess I don't understand the benefit of generating a PDB without line information?

 

 

4 hours ago, Anders Melander said:

So far I don't have enough test cases of map files with these overlapping segments to be able to make a decision on how best to handle them, so if I could get my hand on the one you have that would be great.

Also, if you could examine the corresponding exe file (I recommend using PE Tools) to find out what the actual segment offsets and sizes are (because they're obviously not what the map file states) that would be a big help.

 

I am willing to send the .map file I am using for you to examine if there is a private means of doing so.  I don't want to share it publicly.  I grabbed to PE information for it.  It does seem quite different from Delphi's output:

 Start                 Length     Name                   Class
 0001:0000000000401000 006CD7B8H .text                   CODE
 0002:0000000000ACF000 000A9208H .data                   DATA
 0003:0000000000B79000 00011F20H .bss                    BSS
 0004:0000000000400000 00008260H .tls                    TLS
 0005:0000000000BDE000 0004FF5CH .pdata                  PDATA

 

 

4 hours ago, Anders Melander said:

The best would of course be if Embarcadero fixed the map file but I'm not holding my breath for that, and I still need to be able to handle what's already out there.

I totally agree.  If you have a QC report with Embarcadero I would be happy to vote for it and escalate it.

 

I really would like to see if we could get to the bottom of these things because it's an important capability and I would live to share it with the wider audience.  Happy to help in any way on this.

Edited October 27, 2023 by Allen@Grijjy

[Anders Melander 1795]

9 minutes ago, Allen@Grijjy said:

Notice (2) things.  First I get all the sources lines, but before I did not.  Also I get a Range Check Error (at the bottom).  I am really using the syntax correctly?

The syntax is correct. I fixed the Range Check Error yesterday. Pull the latest source.

 

10 minutes ago, Allen@Grijjy said:

I guess I don't understand the benefit of generating a PDB without line information? 

There's no point in having a PDB without line information and, given the parameters you use, you should get a usable PDB.

 

You can mail the map file to anders🍥melander.dk

 

16 minutes ago, Allen@Grijjy said:

I grabbed to PE information for it.  It does seem quite different from Delphi's output:

Actually, apart from the .tls problem, they look fine to me.

You have to take into account that the PE Tools segment offsets are relative to the image base (see the Optional Header dialog) while the map file offsets are absolute assuming an image base of $00400000.

 

[Anders Melander 1795]

Here's a quick update on the two issues @Allen@Grijjy encountered:

• The problem with the overlapping segments has been worked around in code by ignoring conflicts caused by the .tls segment.
  Incidentally, Allen found an old QC report about a similar issue (RSP-12824). Unfortunately, that issue was closed by someone who apparently doesn't understand what the map file segment offset values signify.
  Regardless, an overlapping .tls segment now only produces a warning message.
   
• The filter problem turned out to be a copy/paste problem.
  Allen had copied his map2pdb parameter list from a post here, and somewhere along the way, two invisible zero-width no-break space characters (#$F0FF) got inserted into the string he pasted. So what looked like '-include:0001' was actually '-include:000'#$F0FF#$F0FF'1'.
  I've been bitten by that one myself a few times when I've copied code from a browser.

[Stefan Glienke 2009]

14 hours ago, Anders Melander said:

• The filter problem turned out to be a copy/paste problem.
  Allen had copied his map2pdb parameter list from a post here, and somewhere along the way, two invisible zero-width no-break space characters (#$F0FF) got inserted into the string he pasted. So what looked like '-include:0001' was actually '-include:000'#$F0FF#$F0FF'1'.
  I've been bitten by that one myself a few times when I've copied code from a browser.

I reported that issue years ago but unfortunately it still exists - the report contains the exact steps that lead to this at the end and thus might help to avoid it in the future:

 

Edited October 29, 2023 by Stefan Glienke

[Pat Heuvel 1]

Gday,

Thanks for making this tool available! I've just tried to use it on a C++ Builder 32-bit binary and got the following:

 

map2pdb - Copyright (c) 2021 Anders Melander
Version 3.1.2

Fatal: [   44] Modules overlap: C:\PROGRAM, C:\PROGRAM
0001:0003AD33 00000297 C=CODE    S=_TEXT    G=(none)   M=C:\PROGRAM FILES (X86)\EMBARCADERO\STUDIO\22.0\LIB\WIN32\DEBUG\RTL.LIB|System.Internal.ExcUtils ACBP=A9

Am I doing something wrong? My command line is

D:\Devel\CPP\BlueVane\BVMultisport\Win32\Debug>map2pdb BVMultisport.map -bind

Executable was compiled under RS 11.3, with recommended options for the mapfile.

 

Thank you!

[Anders Melander 1795]

4 minutes ago, Pat Heuvel said:

Am I doing something wrong?

Probably not but I have never tested with a map file produced by C++ Builder and it looks like the format differs slightly from that of Delphi.

The segment/module list of a Delphi map file looks like this:

Detailed map of segments

 0001:00000000 0000FED4 C=CODE     S=.text    G=(none)   M=System   ACBP=A9
 0001:0000FED4 00000C9C C=CODE     S=.text    G=(none)   M=SysInit  ACBP=A9
 0001:00010B70 0000373C C=CODE     S=.text    G=(none)   M=System.Types ACBP=A9
 0001:000142AC 000007E8 C=CODE     S=.text    G=(none)   M=System.UITypes ACBP=A9
 0001:00014A94 00001E04 C=CODE     S=.text    G=(none)   M=Winapi.Windows ACBP=A9
 0001:00016898 000003A8 C=CODE     S=.text    G=(none)   M=System.SysConst ACBP=A9
[...]

As you can see there's no path in the module names.

 

If you create a bug report at the map2pdb issue tracker and attach the map file (zipped) I will take a look at it.