Delphi Package Manager - choices?

[Darian Miller 366]

When poking around the web, I stumbled acorss "Delphi Package Installer (DelphiPI)" on BitBucket: https://bitbucket.org/idursun/delphipi   This project has commit history dating back to 2007

 

The latest product seems to be DPM from @Vincent Parrett (https://docs.delphipm.org/)

 

Just from casual observation, the most used on GitHub seems to be Delphinus  (https://memnarch.bplaced.net/blog/delphinus/)  and Boss (https://github.com/HashLoad/boss)

 

Another interesting one is from @pyscripter, an extension of MultiInstaller:  https://github.com/pyscripter/MultiInstaller

 

Of course, GetIt is available in later versions of RAD Studio.

 

 

Now that a new release is available, thoughts invariably wander back to this painful area again, so - what do you currently use for lessening the pain of upgrades?  Code is certainly in version control, but there are still some manual processes that I do which I'd rather not continue doing for the visual components.

 

 

[Vincent Parrett 764]

2 hours ago, Darian Miller said:

When poking around the web, I stumbled acorss "Delphi Package Installer (DelphiPI)" on BitBucket: https://bitbucket.org/idursun/delphipi   This project has commit history dating back to 2007

Interesting, I hadn't seen this before (and I thought I had seen every delphi package manager!). Can't really make heads or tails of it though - not quite sure how it works. 

2 hours ago, Darian Miller said:

The latest product seems to be DPM from @Vincent Parrett (https://docs.delphipm.org/)

The docs are a little out of date - the project is still actively being worked on (source code here) - I'm currently working on project group support, resolving package conflicts between projects  - anyone familiar with nuget would understand where I am heading

 

This is what working with an individual project looks like

 

2 hours ago, Darian Miller said:

Of course, GetIt is available in later versions of RAD Studio.

🤣 thought you were talking about package managers, not a promotional vehicle. 

 

2 hours ago, Darian Miller said:

Now that a new release is available, thoughts invariably wander back to this painful area again, so - what do you currently use for lessening the pain of upgrades?  Code is certainly in version control, but there are still some manual processes that I do which I'd rather not continue doing for the visual components.

I'm skipping D11 so no pain this time - but I am working hard to get DPM usable for everyone - that said, I have a day job that takes most of my time, so contributions and collaboration is welcomed 😉  

[Fr0sT.Brutal 900]

I'm looking at these tools from time to time but still haven't adopted any of them. One of the reasons is my approach:

- 3rd party packages are global (in Coding\Components)

- Packages and projects I modified or develop are global in separate folder (in Coding\Projects) for small projects that unlikely need consistent history

- Packages and projects I modified or develop are Git submodules inside big projects - this way I have exact pointers to repo of used packages.

 

I doubt any PM could handle all of this. Anyway updating a package is just a question of Git pull + package rebuild so not a big mess.

Edited September 17, 2021 by Fr0sT.Brutal

[Vincent Parrett 764]

The reality is that most people don't modify packages, they just use them. As for packages being global, well that's one of my pet peeves - because I work on different projects or versions of projects that use different versions of a package - can't do that is packages are global - at least not without a lot of hassle. That's something I working to resolve in my package manager.

 

The cool thing about any package manager is that it's optional - you can choose to use it or not, or use it for some packages and reference others manually like before.  

[Wagner Landgraf 43]

I think the "problem" with Package Manager is that is has to be widely adopted by the community to be very useful. Because the key is that relevant libraries and 3rd parties are available in the package manager. "Unfortunately" Embarcadero is pushing GetIt so it's unlikely we will have a different one being widely adopted, I guess.

[Fr0sT.Brutal 900]

1 hour ago, Vincent Parrett said:

The reality is that most people don't modify packages, they just use them. As for packages being global, well that's one of my pet peeves - because I work on different projects or versions of projects that use different versions of a package - can't do that is packages are global - at least not without a lot of hassle. That's something I working to resolve in my package manager.

 

The cool thing about any package manager is that it's optional - you can choose to use it or not, or use it for some packages and reference others manually like before.  

Sure! I'm not insisting in any form on my layout because I guess it's too personal. But storing big packages inside each project is not an option for me either (imagine 10 copies of JVCL...). It's not just a question of space - I perform "search in files in project dir" quite often and the more stuff is laying inside, the more irrelevant results will appear. Moreover, whole system of BPLs suppose global installation.

Probably you guys authors of all PMs should communicate and establish a single standard of config so that PM1 could utilize PM2-prepared package. This way you won't split the (currently) small % of projects and people using PMs into several parts but unite them.

 

To conclude: as for me, personally, what I'd use PM for:

- Management of global packages - backup from existing IDE instance, batch installation to a new one with all the library/search path stuff

- Packing of a project with ALL deps, including global packages

[Anders Melander 1815]

13 minutes ago, Fr0sT.Brutal said:

imagine 10 copies of JVCL...

Or even one 😱  ( 🙂 )

[Uwe Raabe 2064]

40 minutes ago, Fr0sT.Brutal said:

I perform "search in files in project dir" quite often and the more stuff is laying inside, the more irrelevant results will appear.

My layout is a root folder that contains all external libraries as sub repos in a separate sub folder named lib, while the project is located in the source sub folder. This keeps the lib stuff out of the "search in files in project dir".

[Wagner Landgraf 43]

1 hour ago, Fr0sT.Brutal said:

imagine 10 copies of JVCL...

If I'm not wrong, @Vincent Parrett's package manager puts the packages in a global cache. So if you have multiple projects using the same version of JVCL, only one copy will be in your storage. If a couple of projects use an older version, then you will have two copies of JVCL. So it's the ideal scenario. I guess that's how it works, but he can confirm it. 

[Vincent Parrett 764]

6 minutes ago, Wagner Landgraf said:

If I'm not wrong, @Vincent Parrett's package manager puts the packages in a global cache. So if you have multiple projects using the same version of JVCL, only one copy will be in your storage. If a couple of projects use an older version, then you will have two copies of JVCL. So it's the ideal scenario. I guess that's how it works, but he can confirm it. 

That's correct. 

 

When you install a package in a project, it first looks in the package cache (typically %APPDATA%\.dpm\packages - can be changed by the DPMPACKAGECACHE env variable)

If the package is there then that is used, otherwise it (and any missing dependencies) is downloaded from the registered package sources into the package source. 

Then using some msbuild variables the package and it's dependencies are added to the project's search path

 

This is an example from the dpm ide plugin for 10.4

<PropertyGroup>
  <DPMCompiler>10.4</DPMCompiler>
  <DPMCache Condition="'$(DPMCache)' == ''">$(APPDATA)\.dpm\packages</DPMCache>
  <DPM>$(DPMCache)\$(DPMCompiler)\$(Platform)</DPM>
  <DPMSearch Condition="'$(Platform)'=='Win32'">$(DPM)\VSoft.VirtualListView\0.3.0-rc8\lib;$(DPM)\VSoft.Uri\0.1.1\lib;$(DPM)\VSoft.SemanticVersion\0.2.2\lib;$(DPM)\VSoft.JsonDataObjects\0.1.1\lib;$(DPM)\VSoft.HttpClient\0.2.4\lib;$(DPM)\VSoft.CancellationToken\0.0.3\lib;$(DPM)\Gabr42.OmniThreadLibrary\3.7.8\lib;$(DPM)\VSoft.Awaitable\0.2.1\lib;$(DPM)\VSoft.AntPatterns\0.1.1\lib;$(DPM)\Spring4D.Extensions\2.0.0-dev7\lib;$(DPM)\Spring4D.Base\2.0.0-dev7\lib;$(DPM)\Spring4D.Core\2.0.0-dev7\lib;</DPMSearch>
</PropertyGroup>

Then the DPMSearch variable is added to the project search path

 

<PropertyGroup Condition="'$(Base)'!=''">
    <DCC_UnitSearchPath>$(DPMSearch);$(DCC_UnitSearchPath)</DCC_UnitSearchPath>
</PropertyGroup>

and lastly the package references are stored in the dproj so they can be restored (if missing) next time you open the project

 

<ProjectExtensions>
	<DPM>
		<PackageReference id="Spring4D.Core" platform="Win32" version="2.0.0-dev7">
			<PackageReference id="Spring4D.Base" platform="Win32" version="2.0.0-dev7" range="[2.0.0-dev7,]"/>
		</PackageReference>
		<PackageReference id="Spring4D.Extensions" platform="Win32" version="2.0.0-dev7">
			<PackageReference id="Spring4D.Base" platform="Win32" version="2.0.0-dev7" range="[2.0.0-dev7,]"/>
		</PackageReference>
		<PackageReference id="VSoft.AntPatterns" platform="Win32" version="0.1.1"/>
		<PackageReference id="VSoft.Awaitable" platform="Win32" version="0.2.1">
			<PackageReference id="Gabr42.OmniThreadLibrary" platform="Win32" version="3.7.8" range="[3.7.8,]"/>
			<PackageReference id="VSoft.CancellationToken" platform="Win32" version="0.0.3" range="[0.0.3,]"/>
		</PackageReference>
		<PackageReference id="VSoft.HttpClient" platform="Win32" version="0.2.4">
			<PackageReference id="VSoft.CancellationToken" platform="Win32" version="0.0.3" range="[0.0.3,]"/>
		</PackageReference>
		<PackageReference id="VSoft.JsonDataObjects" platform="Win32" version="0.1.1"/>
		<PackageReference id="VSoft.SemanticVersion" platform="Win32" version="0.2.2"/>
		<PackageReference id="VSoft.Uri" platform="Win32" version="0.1.1"/>
		<PackageReference id="VSoft.VirtualListView" platform="Win32" version="0.3.0-rc8"/>
	</DPM>
</ProjectExtensions>

Of course I have simplified the description above, there's a lot more to it but that is the basics.

 

I'm currently working on project group support (made some good progress today) and then design time component support.

[Vincent Parrett 764]

1 hour ago, Fr0sT.Brutal said:

Probably you guys authors of all PMs should communicate and establish a single standard of config so that PM1 could utilize PM2-prepared package

Nice in theory, not really doable though - they are all so different because we all have different ideas. 

 

Edited September 17, 2021 by Vincent Parrett

[Wagner Landgraf 43]

8 minutes ago, Vincent Parrett said:

they are all so different because we all have different ideas. 

That's exactly the reason why none of them will ever be widely adopted. There should be one single manager that applies all the different ideas. It's not impossible.

[dummzeuch 1517]

2 hours ago, Fr0sT.Brutal said:

imagine 10 copies of JVCL

... and JCL and Indy and TeeChart and Graphic32 and a few more that I tend to forget. And not just for 10 projects but for about 50 (not all of these active at the same time, of course). That's nothing I need to imagine, that's my actual working environment.

But similar to what @Uwe Raabe just described, all libraries are stored as subdirectories under a libs subfolder, while the project sources are in the src subfolder.

 

Edited September 17, 2021 by dummzeuch

[Lars Fosdal 1793]

Speaking of TChart - I wonder why TChart and DUnit isn't in GetIt instead of among the additional installation options?

[TurboMagic 92]

On 9/17/2021 at 12:49 PM, Wagner Landgraf said:

I think the "problem" with Package Manager is that is has to be widely adopted by the community to be very useful. Because the key is that relevant libraries and 3rd parties are available in the package manager. "Unfortunately" Embarcadero is pushing GetIt so it's unlikely we will have a different one being widely adopted, I guess.

There are concrete (but not public yet) plans to improve GetIt. One of the items will make it easier to update contents which has already been published via GetIt so it might attrack more publishers.

[RaelB 4]

On 9/17/2021 at 3:45 PM, dummzeuch said:

...  And not just for 10 projects but for about 50 (not all of these active at the same time, of course). That's nothing I need to imagine, that's my actual working environment.

So how do you manage working in an IDE with all those different lib versions?

 

- Do you use specific IDE's for specific projects?

- Are you using any tools to help you with that?..

 

Peter Below, once told me this:

Quote

It is possible to create project-specific IDE configurations if you really run into performance problems and the project in question really needs only a subset of your complete component set. You can disable specific design-time packages on a per-project basis, for instance. That does not solve the long search path (and browsing path, more relevant for code insight etc.) issue, though. But you can create a copy of the IDE's registry key tree (the one under HKEY_CURRENT_USER/Software) under a new key name, modify that to only get the packages needed for the project mentioned on the IDE pathes, and start the IDE with the -r command line parameter to make it use this key instead of the default. This requires a bit of fiddling with regedit, but can pay off if you have to maintain several big projects in parallel. You would have then one shortcut to the IDE for each of the projects and use the appropriate one to load a project with the associated component set.

Perhaps you are using this idea..

[dummzeuch 1517]

10 hours ago, RaelB said:

So how do you manage working in an IDE with all those different lib versions?

 

- Do you use specific IDE's for specific projects?

- Are you using any tools to help you with that?..

It's not as bad as it sounds. I have one version of each component package installed in the IDE and that's used at design time regardless of which version of the sources are compiled into the project. When I make any changes to the controls I keep an eye on the dfm files for properties that might not be compatible with the active version of the controls and revert them when committing to source control, but given the mature state of the components that's very rarely the case.

Source code and dcus of 3rd party libraries is not in the IDE's search paths but only in those of the project. The source code itself is always an svn:external to the project sources, so it's in a subdirectory of the project and all entries in the search path are relative to the project. We always compile from sources, including 3rd party sources, and don't use packages.

I have several installations of the IDEs on different computers, but that's for debugging purposes only (due to specialized hardware these programs need to access) not to keep different versions of the controls. We develop with Delphi 2007, XE2 and now 10.2. XE2 projects are slowly being upgraded to 10.2. Some 2007 projects need to stay with that version until we finally phase out the last Windows XP machines (planned for this winter), then these too will be moved to 10.2.

 

I blogged about this several years ago (when life was still a bit simpler, but the general principles still apply http://dummzeuch.de/delphi/subversion/english.html

Edited October 13, 2021 by dummzeuch

[Der schöne Günther 316]

10 hours ago, RaelB said:

So how do you manage working in an IDE with all those different lib versions?

Different VMs 😎

[RaelB 4]

Thanks for the info :).