Uses clauses and ide performance - does it make a difference?

[ventiseis 2]

Delphi provides two uses clauses in every unit. The discussion, which uses clause is to be preferred, seems to be an old one. There are different justifications which unit clause should be used – regarding code style, circular references, or readability.

 

The information I am missing is: in terms of compilation time and memory use – where should I put my units if I am free to decide?

 

Is there a general rule for all Delphi versions or is the behavior different in different versions?  I do not know of an official statement from Embarcadero, but I tend to think that putting everything in the interface clause should improve compilation time and memory usage in the IDE.

 

I do not want to start another discussion about code style and which uses clause to prefer – this was already discussed at length at other locations. I want to reduce compilation time and memory use in large projects.

[Uwe Raabe 2089]

7 minutes ago, ventiseis said:

but I tend to think that putting everything in the interface clause should improve compilation time and memory usage in the IDE.

Can you explain how you came to this conclusion or is it just a gut feeling?

[ventiseis 2]

Of course there is no proof -- but for one project I moved the uses clauses to the top in a lot of units and got approx. 1s less build time (which is not much).

 

This project (now) consists of 1182 units and ca. 1.700.000 lines of code. The first build in the IDE needs 2.8GB of memory and one minute compilation time.

After the third build I usually have to restart the ide, because compiliation time would be too long.

Perhaps its time for a 64bit IDE. 

 

[Lajos Juhász 308]

12 minutes ago, ventiseis said:

Perhaps its time for a 64bit IDE. 

Or make fewer circular references. Moving everything in the code to interface section is wrong. It is well known fact that the uses are only for those units that are required in the interface section, everything else must go into the implementation section. Other crucial thing is to remove the Unit Scope names and in code use only fully qualified unit names. This was important in older versions of Delphi (there were problems with unit name caching). 

[ventiseis 2]

44 minutes ago, Lajos Juhász said:

Moving everything in the code to interface section is wrong. It is well known fact [...]

 

There is one thing that the documentation states:

 

Quote

When a change is made in the interface section of a unit, other units that depend on the change must be recompiled. But when changes are made only in the implementation or other sections of a unit, dependent units don't have to be recompiled. The compiler tracks these dependencies automatically and recompiles units only when necessary.

 

This would support your statement. But this behaviour is broken in our environment: if I compile the same project twice, every single unit is compiled again to dcu - even if I do not change a single line of code!

We use an alternative dcu output path in the project options. It is a virtual drive created by the subst utility. If I remove that virtual drive and set the dcu output path to a different path, the documented behavior works again - i have no explanation for this.

 

Edited 21 hours ago by ventiseis

[Uwe Raabe 2089]

19 minutes ago, ventiseis said:

But this behaviour is broken in our environment: if I compile the same project twice, every single unit is compiled again to dcu - even if I do not change a single line of code!

We use an alternative dcu output path in the project options. It is a virtual drive created by the subst utility. If I remove that virtual drive and set the dcu output path to a different path, the documented behavior works again - i have no explanation for this.

Seems like you were asking the wrong question then.

[Anders Melander 1861]

3 minutes ago, ventiseis said:

This would support your statement. But this behaviour is broken in our environment: if I compile the same project twice, every single unit is compiled again to dcu - even if I do not change a single line of code!

It sounds like you should focus on fixing your build environment first 🙂

 

4 minutes ago, ventiseis said:

We use an alternative dcu output path in the project options. It is a virtual drive created by the subst utility. If I remove that virtual drive and set the dcu output path to a different path, the documented behavior works again - i have no explanation for this. 

Well, there you have it. subst is a leftover from DOS and should be avoided.

 

A file in a virtual (subst) location has (at least) two paths: The virtual path and the physical path. Depending on what windows API a virtual filename is passed though you can either get a result that is relative to the virtual location or to the physical location. I can't give you a concrete example but I've seen it happen many times.

 

For example, I would guess that if you resolve a virtual filename to a PIDL and then convert that back to a path, then you will end up with the physical path.

[ventiseis 2]

Just for explanation: we use subst to have the identical paths on every dev machine. Before that, a ram drive was used for that - but in newer windows versions they seemed problematic, too.

[Uwe Raabe 2089]

3 minutes ago, ventiseis said:

we use subst to have the identical paths on every dev machine

This raises the question why you need absolute path names for the dcu output in the first place. IIRC that is a no-go for build systems and should be avoided in development systems, too.

 

Although I also use alternative dcu output paths (usually to separate dcu output for different projects in a project group in addition to separate platforms and build configurations), all of these are relative to the project. As I often have multiple work trees of a repo, I would open a can of worms when all of them were using the same dcu output folder.

[Anders Melander 1861]

2 hours ago, ventiseis said:

Before that, a ram drive was used for that - but in newer windows versions they seemed problematic, too.

More DOS legacy.

RAM-drives hasn't had any justification since 16-bit processors.

[ventiseis 2]

I understand your points and I will be happily convinced to use another dcu output path.

 

But back to the original question: can we reduce our memory usage by the compiler if we follow a specific strategy with the uses clauses?

Does the afore mentioned tracking of dependencies between included units in the interface uses clause increase memory consumption? From a naive point of view, the compiler must compile each unit at the first build - regardless of the order of the units. And I would assume that the binary result is cached somehow in the memory, leading to the high usage.

Another thing which I do not want to include in the discussion are the uncertainties with unit initialization order which is derived from the uses order of units in the uses clause.

 

[Uwe Raabe 2089]

1 hour ago, ventiseis said:

But back to the original question: can we reduce our memory usage by the compiler if we follow a specific strategy with the uses clauses?

I guess that question can only be answered by someone from Embarcadero familiar with the compiler internals.

 

On the other hand, cyclic dependencies between units do make a significant difference in performance and probably also in memory consumption. As moving all units into the interface uses clause will just not allow circular dependencies, so it may guide you to pure non-cyclic units and thus increase performance. That doesn't imply that this cannot be achieved with units in the implementation uses clause. Personally I didn't notice any drawbacks with the latter approach.

[Tom F 85]

You may be interested in this video: 

 

[Vandrovnik 216]

4 hours ago, Anders Melander said:

More DOS legacy.

RAM-drives hasn't had any justification since 16-bit processors.

Why?

I use RAM drive as target for all .dcu files. After reboot, no .dcus exists -> no problems from old .dcus, .dcus compiled with another settings etc. And it also saves some SSD writes. (With HDDs, RAM drive was also significantly faster.)

[Brandon Staggs 325]

3 minutes ago, Vandrovnik said:

And it also saves some SSD writes.

Interesting. I know this was a concern in the early days of SSD drives, but I've never replaced an SSD drive due to it "running out of writes."

 

Obviously your RAM is going to be your fastest storage, but just doing a full build and letting your OS cache do its work ought to be enough. Have you actually tested the speed of both methods? I have long since quit trying to do a better job than my OS cache does on its own, especially in these days of NVME solid state drives...

Edited 13 hours ago by Brandon Staggs

[Vandrovnik 216]

16 minutes ago, Brandon Staggs said:

Interesting. I know this was a concern in the early days of SSD drives, but I've never replaced an SSD drive due to it "running out of writes."

 

Obviously your RAM is going to be your fastest storage, but just doing a full build and letting your OS cache do its work ought to be enough. Have you actually tested the speed of both methods? I have long since quit trying to do a better job than my OS cache does on its own, especially in these days of NVME solid state drives...

I did not make speed tests, the speed will probably be almost the same, because of large file cache and fast NVME drives.

[DelphiUdIT 205]

1 hour ago, Vandrovnik said:

I did not make speed tests, the speed will probably be almost the same, because of large file cache and fast NVME drives.

https://www.kingston.com/it/blog/pc-performance/what-is-ram-disk

[Anders Melander 1861]

2 hours ago, Vandrovnik said:

I did not make speed tests, the speed will probably be almost the same, because of large file cache and fast NVME drives.

Exactly. Let the OS manage the RAM; It knows where it's needed the most.

There are some cases where a RAM disk makes sense but output from a compiler ain't it.

 

1 hour ago, DelphiUdIT said:

https://www.kingston.com/it/blog/pc-performance/what-is-ram-disk

TLDR; but I wouldn't trust a RAM manufacturer to be unbiased in this matter.

[DelphiUdIT 205]

Just now, Anders Melander said:

TLDR; but I wouldn't trust a RAM manufacturer to be unbiased in this matter

But they produce SSD too, and the test that they made were about their SSD.

I don't think they have any interest in "cheating" on the results. It could be that technology will move the "hand" one way or the other over time.

[Vincent Parrett 800]

I went thought the whole RAM Disk scenario a few years ago to try and speed up our CI builds - this was before we got new servers with nvme drives. I even did some rough benchmarking. 

 

https://forums.adug.org.au/t/anyone-using-ram-disks-these-days/59756

 

TLDR; don't bother with RAM Disks - they are no faster than nvme. The only scenario where they might be beneficial is avoiding writes to consumer grade SSD's - but even then you can work around that by buying bigger capcity SSD's (write leveling takes care of wear). 

[ventiseis 2]

15 hours ago, Uwe Raabe said:

On the other hand, cyclic dependencies between units do make a significant difference in performance and probably also in memory consumption. As moving all units into the interface uses clause will just not allow circular dependencies, so it may guide you to pure non-cyclic units and thus increase performance.

I also think that this a major advantage of putting all dependencies into the interface section- you are "forced" to write cycle-free dependencies.

 

Side note 1: Of course I tried the avaliable tools to detect cyclic dependencies (if there are any) - but the last time I tried they didn't work as well as expected: conditional compilation with compiler directives or generics lead to parser errors in this tools.

Side note 2: I don't think that there is an official statement in the documentation which warns you about these cyclic references... The only thing I could find is the following:

 

Quote

To reduce the chance of circular references, it's a good idea to list units in the implementation uses clause whenever possible. Only when identifiers from another unit are used in the interface section is it necessary to list that unit in the interface uses clause.

 

Is that misleading?

[Anders Melander 1861]

Just now, ventiseis said:

Is that misleading?

No. Why would you think that it was?

 

My own experience is that only having the necessary units in the implementation section improves compile time and reduces memory usage during compile.

It also helps with identifying and minimizing dependencies as I become conscious of them when a unit has to be moved to the implementation section.