64 bit compiler running out of memory

[Dave Novo 51]

Hello,

We are using Delphi Seattle.

 

We have a ~ 2M LOC program with hundreds of units. The 64 bit compiler is running out of memory when compiling. Even when we set to "use MSBuild" to compile in a separate process. The separate process dies with an out of memory around 2GB of RAM usage.  The VM I am using has 8 GB of RAM and plenty of spare RAM when the process dies.

 

Aside from breaking the application into different packages/projects, does anyone have any bright ideas.

 

We are in the process of upgrading to Delphi 10.4.1.  Is the 64 bit compiler still a 32 bit app?

 

[FPiette 382]

2 minutes ago, Dave Novo said:

The 64 bit compiler

There is no Delphi 64 bit compiler. There is only a 32 bit compiler able to generate 64 bit executable 🙂

 

[Lajos Juhász 293]

The easiest way to kill the compiler is to have huge cyclic unit dependencies. Try to reduce that. Also if the project was created before unit scope names were introduced you should add the scope names to the units in the uses lists and remove from project options in Unit Scope Names. 

[Dave Novo 51]

@Lajos Juhász - we definately are missing the unit scope names. I did not think that would make such a big difference. Certainly easy (but tedious) to fix.

[Uwe Raabe 2057]

7 minutes ago, Dave Novo said:

Certainly easy (but tedious) to fix.

Perhaps this may  help a bit: 

UsesCleanerSource.zip

[Vincent Parrett 750]

1 hour ago, Uwe Raabe said:

Perhaps this may  help a bit: 

UsesCleanerSource.zip

I can definitely recommend this tool. I ran it over my 4M line project group and it did a fantastic job. The only things I had to manually fix where where I was referencing a type or function with the unit name, e.g SysUtils.StringReplace(....) had to change to System.SysUtils.StringReplace(....).

 

This will also speed up the compile time as the compiler spends less time doing namespace lookups. 

 

[Dave Novo 51]

@Uwe Raabe - thank you for the tool. After I made it compile for Delphi I ran it, and all it seemed to do was rearrange my uses clause. An example

 

If the unit did have the fully qualified name, it moved them to the top of the uses clause, but otherwise left things alone.  i.e.

 

Am I missing something to make it prefix the fully qualified namespaces?

[Vincent Parrett 750]

Perhaps this will help

 

 

 

 

[Uwe Raabe 2057]

9 hours ago, Dave Novo said:

 

Am I missing something to make it prefix the fully qualified namespaces?

Perhaps you are missing a proper config file (default is UsesCleaner.cfg next to the exe). If there is none, the program will use the default settings which are - well, somewhat special:

 

procedure TSourceFileUsesClauseFormatter.InitSettings;
begin
  UsesHelper.UnitAliases := 'WinTypes=Winapi.Windows;WinProcs=Winapi.Windows;DbiTypes=BDE;DbiProcs=BDE;DbiErrs=BDE;';
  UsesHelper.UnitScopeNames := 'Winapi;System.Win;Data.Win;Datasnap.Win;Web.Win;Soap.Win;Xml.Win;Bde;System;Xml;Data;Datasnap;Web;Soap;Vcl;Vcl.Imaging;Vcl.Touch;Vcl.Samples;Vcl.Shell';
  UsesHelper.SearchPath := 'c:\Program Files (x86)\Embarcadero\Studio\19.0\lib\win32\release';
  UsesHelper.GroupNames := '<UnitScopeNames>';
end;

In your case, the SearchPath may be the culprit, which may cause the standard units not to be found.

 

A decent config file can look like this (with an adjusted SearchPath entry of course):

 

[Settings]
Indentation=2
Compressed=0
MaxLineLength=130
SearchPath=c:\program files (x86)\embarcadero\studio\21.0\lib\Win32\release;
UnitAliases=WinTypes=Winapi.Windows;WinProcs=Winapi.Windows;DbiTypes=BDE;DbiProcs=BDE;DbiErrs=BDE;
UnitScopeNames=Winapi;System.Win;Data.Win;Datasnap.Win;Web.Win;Soap.Win;Xml.Win;System;Xml;Data;Datasnap;Bde;Web;Soap;Vcl;Vcl.Imaging;Vcl.Touch;Vcl.Samples;Vcl.Shell
GroupNames=@DelphiOTA;Winapi;System.Win;System;Data;FireDAC;Vcl

[Groups]
DelphiOTA=ToolsAPI;DesignIntf;DesignEditors

 

[Anders Melander 1782]

34 minutes ago, Uwe Raabe said:

Perhaps you are missing a proper config file

The config file helped for me

 

One thing though: The conversion doesn't preserve the file encoding; The new file is written as ANSI even if the source file was UTF-8 encoded.

[Uwe Raabe 2057]

17 minutes ago, Anders Melander said:

The conversion doesn't preserve the file encoding; The new file is written as ANSI even if the source file was UTF-8 encoded.

Good point!

[vfbb 285]

16 hours ago, Dave Novo said:

 

Aside from breaking the application into different packages/projects, does anyone have any bright ideas.

This is the correct solution and the Delphi ide/compiler 64 bits is far. 
 

I faced this problem on a daily basis, the project became too big, and a solution adopted by us was to divide the project into packages, so as not to overload with a single compilation. You can for example divide main project, FMX package and RTL package.

[Stefan Glienke 2002]

10 hours ago, vfbb said:

Delphi ide/compiler 64 bits is far

Kids these days - making them 64bit would be the lazy solution - being more reasonable with allocating memory would be the good one.

Especially with confirmed performance issues in the compiler that are caused by unnecessary heap allocations.

[Dave Novo 51]

@Uwe Raabe - thanks for the tip. Everything works great after using the config settings you suggested.