Blogged : Introducing DPM - a Package Manager for Delphi

[Vincent Parrett 763]

Back in Feb 2019, I blogged about the need for a Package Manager for Delphi. The blog post garnered lots of mostly useful feedback and encouragement, but until recently I could never find a solid block of time to work on it. Over the last few weeks I've been working hard to get it to an mvp stage.

 

https://www.finalbuilder.com/resources/blogs/introducing-dpm-a-package-manager-for-delphi

 

 

[Fr0sT.Brutal 900]

Looks very promising! Haven't examined this PM yet but definitely will give it a try.

Quick notes on what I'd like to see in PM (probably they're implemented already):

- Option to choose global (configurable path that will be added to IDE Library path) or local (inside the project folder, within configurable subfolder, path will be added to project Search path) installation

- Packing the whole project with deps, with choice whether include global deps or not

- Some kind of Delphinus support, maybe just a config converter from Delphinus to DPM

- Installing from an archive from Internet, this way GitHub repos could be used as well. Installer should check archive's Last-Modified and don't download it if it wasn't changed

[Vincent Parrett 763]

Thanks

 

Packages are downloaded and extracted into a package cache folder, which is configurable, and then added to the search path of the project. I have no plans for using the global library path. 

Packing the whole project with deps - not planned.
Delphinus is very different to DPM so not sure how that would work. 

Installing from a repo.. maybe, but would need to figure out versioning. Not planning to look at this any time soon, but if someone wants to contribute I'm open to it. 

 

 

[timfrost 79]

Without managing the global library path, a package manager would not work for me.  I cannot think of any library we use that is not used by multiple Delphi projects; in some cases dozens.  So no project search path is ever used or needed.  Are you targeting just the type of development shop which builds only one or two huge applications?  But don't pay too much attention to this comment, because as I have said before I am very happy using a Finalbuilder project as a DPM.

[Vincent Parrett 763]

How do you manage upgrading libraries. Once you upgrade for one project, you have to upgrade for every project, that's the nature of global libraries. The "overhead" of installing packages is really quite small, and you can do it incrementally. DPM does not use the library path, but that doesn't mean you still cant.  

 

FWIW, one of my reasons for working on dpm is probably somewhat selfish, but 90% of the problems FinalBuilder customers have with compiling delphi projects are related to the library path. I'm hoping that dpm will provide a standard way to work with libraries, so the whole issue of compiles on one machine, not on the other becomes a thing of the past (I can spend my time working on other things!). 

 

There are other benefit to not using the library path is the compiler doesn't have to trawl through every installed library even when the project doesn't use them, which makes for slower compile times. 

 

It's early days, there's still a lot to do on the project, including IDE integration, which will make working with it a lot simpler. Stay tuned.

[Vincent Parrett 763]

I have created packages for most of my delphi open source projects (not DUnitX, that needs some tidying up)

 

They listed here :

https://github.com/DelphiPackageManager/DPM/issues/2

 

[Uwe Raabe 2064]

Cleaning the global library path from unwanted entries creeping in when installing a library with its own setup is a continuing task here. In my installations the global library path contains only what the Delphi installer writes to it. Every additional library path is added to the search path of the project that makes use of it. Option sets for each library not only simplifies this, but also give a visual clue in the project manager which libraries are used for the project. Also these paths are somehow included in the repository of this project. This way I can fire up a new build agent for Continua, install Delphi, FinalBuilder and whatever tools are needed - that's it. All my projects can be compiled on this machine. Gone are the days where hunting for missing libraries detected late in the build process - often when you are already in a hurry.

[David Heffernan 2353]

On 12/13/2019 at 11:27 PM, timfrost said:

Without managing the global library path, a package manager would not work for me.  I cannot think of any library we use that is not used by multiple Delphi projects; in some cases dozens.  So no project search path is ever used or needed.  Are you targeting just the type of development shop which builds only one or two huge applications?  But don't pay too much attention to this comment, because as I have said before I am very happy using a Finalbuilder project as a DPM.

In my world I need to build against multiple different versions of the same library. This is not an obscure requirement. It happens pretty much anytime you maintain development and release branches. When you maintain legacy versions. This is a mainstream scenario in any professional programming setup. 

 

Global settings are simply incompatible with that.

[dummzeuch 1517]

What David said!

 

My library path only contains libraries that were installed together with the IDE, and I even remove some of these (e.g. Indy) because I update to newer versions. Everything else goes to the project's search path.

[Fr0sT.Brutal 900]

OTOH, I think we all have several active projects with full set of tests/CI/auto deployment/doc system etc and much, much more little projects for which using a completely individual set of libraries is overhead. Having 100 copies of JCL only because all 100 projects use a couple of its functions? I don't think it's a good idea

[Wagner Landgraf 43]

23 minutes ago, Fr0sT.Brutal said:

OTOH, I think we all have several active projects with full set of tests/CI/auto deployment/doc system etc and much, much more little projects for which using a completely individual set of libraries is overhead. Having 100 copies of JCL only because all 100 projects use a couple of its functions? I don't think it's a good idea

Why do you assume there will be 100 copies of JCL? It should be one copy shared by all the 100 projects, which will just reference the same library.

[Anders Melander 1815]

7 minutes ago, Wagner Landgraf said:

It should be one copy shared by all the 100 projects, which will just reference the same library.

That's not my understanding. As I read it the package cache can be either global or per project.

 

Regardless I can't see the problem of each project having their own copy of dependent libraries - unless it's the JCL in which case even a single copy is a problem 🙂

Disk space is cheap. Time is not.

[dummzeuch 1517]

52 minutes ago, Fr0sT.Brutal said:

OTOH, I think we all have several active projects with full set of tests/CI/auto deployment/doc system etc and much, much more little projects for which using a completely individual set of libraries is overhead. Having 100 copies of JCL only because all 100 projects use a couple of its functions? I don't think it's a good idea

But that's exactly how we do it. Each project has its complete set of libraries stored in subdirectories of the project and referenced in subversion as svn:external. Compared to the size of the data we are working with (e.g. HD videos with several 10000 pictures each), that's not much space. And hard disks are cheap, especially if you buy them by the hundreds.

Edited December 17, 2019 by dummzeuch

[Vincent Parrett 763]

47 minutes ago, Fr0sT.Brutal said:

OTOH, I think we all have several active projects with full set of tests/CI/auto deployment/doc system etc and much, much more little projects for which using a completely individual set of libraries is overhead. Having 100 copies of JCL only because all 100 projects use a couple of its functions? I don't think it's a good idea

It's one copy (the default), unless you want it to be per project. As @David Heffernan said, using different versions if a library for different projects or different branches is quite common and something I do all the time (and with delphi it's painful to manage).

 

As for the size of the JCL, it could probably be split into smaller packages (I don't use it due to it's propensity to pull in a bunch of units for 1 function). 

[Wagner Landgraf 43]

14 minutes ago, Anders Melander said:

That's not my understanding. As I read it the package cache can be either global or per project.

 

Regardless I can't see the problem of each project having their own copy of dependent libraries - unless it's the JCL in which case even a single copy is a problem 🙂

Disk space is cheap. Time is not.

So, if the package cache can be global, then you can share the library with the 100 projects, right?

[Anders Melander 1815]

4 minutes ago, Vincent Parrett said:

It's one copy

I would have preferred it to be zero or one copies: The package cache would just contain the meta data and optionally a single copy (per version) of the source files.

On update the package manager would copy the specified source files to a local folder specified in the project options.

 

This way there would be no dependency between the project and DPM and the the project would be completely autonomous. As far as I can tell, the system you have now creates a dependency between the project and DPM because the project search path has to include DPM specific folders - local or not.

There's also a duplicate binding between the project and the package version because the version is specified in both the dpm.config file and in the project search path. It should be sufficient to specify the version in the config file.

[Anders Melander 1815]

13 minutes ago, Wagner Landgraf said:

So, if the package cache can be global, then you can share the library with the 100 projects, right?

Yes.

[Vincent Parrett 763]

8 hours ago, Anders Melander said:

I would have preferred it to be zero or one copies: The package cache would just contain the meta data and optionally a single copy (per version) of the source files.

 

The package cache has a single expanded (ie unzipped) copy of a package version, only for versions that you have used and only when you ask for them. 

 

8 hours ago, Anders Melander said:

On update the package manager would copy the specified source files to a local folder specified in the project options.

Then you would have 2 copies 😉

 

8 hours ago, Anders Melander said:

This way there would be no dependency between the project and DPM and the the project would be completely autonomous. As far as I can tell, the system you have now creates a dependency between the project and DPM because the project search path has to include DPM specific folders - local or not.

There's also a duplicate binding between the project and the package version because the version is specified in both the dpm.config file and in the project search path. It should be sufficient to specify the version in the config file.

 

At some point you have to reference the location of the source/lib files, I have tried to do it in a manner that will work across multiple machines, no matter where the project is. You are free to change the location of the PackageCache, it might actually be advisable if your C drive is like mine, always full!   

 

DPM adds this to the top of the project file :

 

    <PropertyGroup>
        <ProjectGuid>{D07E7230-728D-4BB3-AC4E-EBFA414C5503}</ProjectGuid>
        <!-- snipped -->
    </PropertyGroup>
    <PropertyGroup>
        <DPMCompiler>XE7</DPMCompiler>
        <DPMCache Condition="'$(DPMCache)' == ''">$(APPDATA)\.dpm\packages</DPMCache>
        <DPM>$(DPMCache)\$(DPMCompiler)\$(Platform)</DPM>
        <DPMSearch Condition="'$(Platform)'=='Win32'">$(DPM)\VSoft.A\0.1.5\src;$(DPM)\VSoft.B\0.0.4\src;$(DPM)\VSoft.D\0.0.8\src;$(DPM)\VSoft.C\0.0.7\src;</DPMSearch>
        <DPMSearch Condition="'$(Platform)'=='Win64'">$(DPM)\VSoft.A\0.1.5\src;$(DPM)\VSoft.B\0.0.4\src;$(DPM)\VSoft.D\0.0.8\src;$(DPM)\VSoft.C\0.0.7\src;</DPMSearch>
    </PropertyGroup>

This sets up the DPMSearch property for use in the search path. For now we are just adding it to the base config, not 100% if we can add it to all, that might be something we leave for the user. Note that the DPMCache property can be overriden by an environment variable as well. 

    <PropertyGroup Condition="'$(Base)'!=''">
        <DCC_UnitSearchPath>$(DPMSearch);$(DCC_UnitSearchPath)</DCC_UnitSearchPath>
    </PropertyGroup>

To store which package references we use, dpm adds this to the ProjectExtensions section

    <ProjectExtensions>
        <DPM>
            <PackageReference id="VSoft.A" platform="Win32" version="0.1.5"/>
            <PackageReference id="VSoft.A" platform="Win64" version="0.1.5"/>
        </DPM>
    </ProjectExtensions>

Note that we only have PackageReferences for VSoft.A, however the DPMSearch also has VSoft.B/C/D as those are dependencies of VSoft.A - the install/restore process works this out, downloads and extracts any missing packages and then updates the DPMSearch properties. 

 

There's still a lot of work to do with DPM - I'm working on compilation support (optional compile on install) so that builds are done using dcu's rather than the source (faster builds),  design time package installation, IDE integration (restore during project load), improving the dependency resolution algorithm.

 

I would love for people to jump in and help, even just code reviews looking for obvious bugs.

[Anders Melander 1815]

1 hour ago, Vincent Parrett said:

Then you would have 2 copies

I'm way past caring about disk space.

For example we use DevExpress in almost all desktop projects and in addition to the regular install, which we need for design time support, help, examples, etc., each project has a complete copy of the DevExpress source files to make sure the projects are autonomous, under full revision control and are always built with the correct version. That's 120 Mb in 3000 files - per project, per branch, etc.

 

Yes, a global package cache could reduce that to a single copy, but it would be pointless. The problem we have isn't disk space. It's creating a reliable, robust and foolproof development environment.

 

 

27 minutes ago, Vincent Parrett said:

At some point you have to reference the location of the source/lib files,

Yes, but I would prefer to have complete control over where that exact location is.

I'm not saying that it's unreasonable to require at least some adaptation of a project for use with DPM but for me at least it will be a showstopper if use of DPM becomes a requirement once the project has been adapted and having the package cache in the project search path, and the search path controlled by DPM, is such a requirement as far as I can tell.

 

I guess right now I'm focusing on how DPM could fit in with my existing projects where I already have all dependencies sorted out and folders and search paths assigned.

I understand the use cases of dependency resolution, the package cache and updating the search path, but I'm not sure the convenience they offer is worth sacrificing the autonomy I already have in place for the projects I manage.
Now if only DPM had been a standard part of Delphi then it would have been an entirely different matter.

 

[Vincent Parrett 763]

I don't expect that DPM will be for everyone or that everyone will switch every project to it. 

 

I spent months reviewing feedback from my RFC post, studying other package managers and working to understand the limitations that delphi imposes before really getting stuck into dpm (I originally started on it in 2013, but shelved it when getit was announced). I think that I have a reasonable solution (or at least it will be when it's done) that provides consistency (which is the main requirement) and is performant. 

 

I've answered a few emailed questions about performance, in the example dproj info I posted earlier, this is what a restore looks like when the package cache is empty :

 

I:\Github\DelphiPackageManager\DPM\TestPackages\TestProject>dpm restore .\TestProject.dproj
Delphi Package Manager - Version : 0.1.1.0
© 2019 Vincent Parrett and Contributors
License - http://www.apache.org/licenses/LICENSE-2.0
Attempting restore on [.\TestProject.dproj] for [Win32]
Starting dependency resolution...
Resolving dependency : VSoft.A.0.1.5->VSoft.B [0.0.1,0.0.4]
            selected : VSoft.B.0.0.4
Resolving dependency : VSoft.A.0.1.5->VSoft.C [0.0.3,0.0.7]
            selected : VSoft.C.0.0.7
Resolving dependency : VSoft.B.0.0.4->VSoft.D [0.0.6,0.0.10]
            selected : VSoft.D.0.0.9
Resolving dependency : VSoft.C.0.0.7->VSoft.D [0.0.1,0.0.8]
       conflict - selected version : VSoft.D-0.0.9 does not satisfy [0.0.1,0.0.8]
       overlapping range found : VSoft.D-[0.0.6, 0.0.8]
Resolving dependency : VSoft.C.0.0.7->VSoft.D [0.0.6, 0.0.8]
            selected : VSoft.D.0.0.8
Dependency resolution done in 220ms
Downloading package [VSoft.A-XE7-Win32-0.1.5] from [i:\dpmfeed]
Package  [C:\Users\vincent.OFFICE\AppData\Roaming\.dpm\packages\VSoft.A-XE7-Win32-0.1.5.dpkg] added to cache.
Downloading package [VSoft.B-XE7-Win32-0.0.4] from [i:\dpmfeed]
Package  [C:\Users\vincent.OFFICE\AppData\Roaming\.dpm\packages\VSoft.B-XE7-Win32-0.0.4.dpkg] added to cache.
Downloading package [VSoft.D-XE7-Win32-0.0.8] from [i:\dpmfeed]
Package  [C:\Users\vincent.OFFICE\AppData\Roaming\.dpm\packages\VSoft.D-XE7-Win32-0.0.8.dpkg] added to cache.
Downloading package [VSoft.C-XE7-Win32-0.0.7] from [i:\dpmfeed]
Package  [C:\Users\vincent.OFFICE\AppData\Roaming\.dpm\packages\VSoft.C-XE7-Win32-0.0.7.dpkg] added to cache.

 

This is what it looks like when the packages are already cached.

 

I:\Github\DelphiPackageManager\DPM\TestPackages\TestProject>dpm restore .\TestProject.dproj
Delphi Package Manager - Version : 0.1.1.0
© 2019 Vincent Parrett and Contributors
License - http://www.apache.org/licenses/LICENSE-2.0
Attempting restore on [.\TestProject.dproj] for [Win32]
Starting dependency resolution...
Resolving dependency : VSoft.A.0.1.5->VSoft.B [0.0.1,0.0.4]
            selected : VSoft.B.0.0.4
Resolving dependency : VSoft.A.0.1.5->VSoft.C [0.0.3,0.0.7]
            selected : VSoft.C.0.0.7
Resolving dependency : VSoft.B.0.0.4->VSoft.D [0.0.6,0.0.10]
            selected : VSoft.D.0.0.9
Resolving dependency : VSoft.C.0.0.7->VSoft.D [0.0.1,0.0.8]
       conflict - selected version : VSoft.D-0.0.9 does not satisfy [0.0.1,0.0.8]
       overlapping range found : VSoft.D-[0.0.6, 0.0.8]
Resolving dependency : VSoft.C.0.0.7->VSoft.D [0.0.6, 0.0.8]
            selected : VSoft.D.0.0.8
Dependency resolution done in 21ms

I only showed the Win32 platform for brevity, but each other platform takes the same time. Of course when we eventually have http based package sources, downloading packages will take a bit longer. At the moment it's all single threaded so there's plenty of room for improving performance in the future if needed.

[Vincent Parrett 763]

40 minutes ago, Anders Melander said:

Yes, but I would prefer to have complete control over where that exact location is.

I'm not saying that it's unreasonable to require at least some adaptation of a project for use with DPM but for me at least it will be a showstopper if use of DPM becomes a requirement once the project has been adapted and having the package cache in the project search path, and the search path controlled by DPM, is such a requirement as far as I can tell.

The only thing DPM adds to the search path (via a single variable) is the paths each package requires, as specified by the package authors, and only for the packages that project is actually using. How is that any different from manually managing the search paths for a library?    

[Fr0sT.Brutal 900]

Hmm, so, if I understood correctly, all packages installed by DPM will be global in fact. Then, I guess, packing the whole project would be a necessary feature

[Vincent Parrett 763]

3 hours ago, Fr0sT.Brutal said:

Hmm, so, if I understood correctly, all packages installed by DPM will be global in fact. Then, I guess, packing the whole project would be a necessary feature

Global as (by default at least) 1 copy of the package on the file system, but not global as installed on the ide library path. Not sure what you mean by packing the whole project? 

[Fr0sT.Brutal 900]

39 minutes ago, Vincent Parrett said:

Global as (by default at least) 1 copy of the package on the file system, but not global as installed on the ide library path. Not sure what you mean by packing the whole project? 

Yeah, one copy located in DPM cache. So this approach won't benefit neither from IDE-global layout (i.e. configured via Settings>Library path) (a library is accessible to all projects without any modifications to the projects themselves) nor from project-local layout (a project is stand-alone and consistent, could be easily imported/backed up).

By packing the project I mean something like Node's "npm pack" - creating a developer's package (archive) from a project folder. With all the above, I guess this feature becomes more necessary.

(Just my 2 Bitcoin cents :D )

Edited December 18, 2019 by Fr0sT.Brutal

[Vincent Parrett 763]

15 hours ago, Fr0sT.Brutal said:

Yeah, one copy located in DPM cache. So this approach won't benefit neither from IDE-global layout (i.e. configured via Settings>Library path) (a library is accessible to all projects without any modifications to the projects themselves) nor from project-local layout (a project is stand-alone and consistent, could be easily imported/backed up).

The package cache is accessible from projects without modification, but that's not my primary use case... in fact I can't see why you would want to do that. As for stand alone and being easily backed up.. I've not found making my vs.net projects stand alone necessary at all. I use distributed version control (git and mercurial) which is backed up (along with the copies already on our CI servers and other dev machines) - and our nuget packages are backed up (by nuget.org and our own backups for packages we haven't published). 

 

15 hours ago, Fr0sT.Brutal said:

By packing the project I mean something like Node's "npm pack" - creating a developer's package (archive) from a project folder. With all the above, I guess this feature becomes more necessary.

I've not tried using npm pack on a non package project, but does it really back up all the dependencies, I don't think it does.