Blogged : Delphi Package Manager RFC

[Vincent Parrett 763]

Delphi/Rad Studio desperately needs a proper package/library/component manager. A package manager provides a standardized way of consuming third party libraries. At the moment, use of third party libraries is very much adhoc, and in many cases this makes it difficult to move projects between machines, or to get a new hire up and running quickly. Other development eco systems, like Ruby, .net and Javascript, recognised and solved this problem many years ago. Getting a Ruby, .net or Javascript project up an running, in a new working folder or new machine is trivial.

 

Delphi Package Manager RFC

[Uwe Raabe 2064]

Thanks for this blog post. Instead of just ranting about GetIt you start taking real actions. I appreciate that very much.

 

Some ideas came already to my mind while reading, but I need to think a bit about them first.

 

Regarding the design time components, I had tried to get this covered with my Package Magician mentioned in this blog post: There Can Only Be One – At a Time! While it works in the majority of cases, it heavily relies on the packages to be unloaded cleanly - sometimes not even standard Delphi packages can guarantee that. Unfortunately some of the big players in the package market seem to ignore that their packages my be unloaded and reloaded in the IDE.

 

If you are interested I am open to contribute the code to your project.

 

BTW, the link to the GitHub repository is broken.

[sh17 26]

you should repair the github-link in your article

 

https://github.com/DelphiPackageManager/PackageManagerRFC

[Vincent Parrett 763]

47 minutes ago, sh17 said:

you should repair the github-link in your article

 

https://github.com/DelphiPackageManager/PackageManagerRFC

Thanks for the head up, I changed the project name at the last minute and forgot to update the link. 

[Vincent Parrett 763]

1 hour ago, Uwe Raabe said:

Thanks for this blog post. Instead of just ranting about GetIt you start taking real actions. I appreciate that very much.

 

Some ideas came already to my mind while reading, but I need to think a bit about them first.

 

Regarding the design time components, I had tried to get this covered with my Package Magician mentioned in this blog post: There Can Only Be One – At a Time! While it works in the majority of cases, it heavily relies on the packages to be unloaded cleanly - sometimes not even standard Delphi packages can guarantee that. Unfortunately some of the big players in the package market seem to ignore that their packages my be unloaded and reloaded in the IDE.

 

If you are interested I am open to contribute the code to your project.

 

BTW, the link to the GitHub repository is broken.

Uwe, I'm definitely open to contributions, there's no way I can achieve the desired result by myself (I still have to earn a living!).  It's a complex task, and there are a lot of moving parts and design decisions to be made.

 

I expected package unloading to be an issue. Package Magician looks interesting.. hopefully we can get you on board, I haven't really played with the open tools api much for many years.

 

Github link fixed. 

[pyscripter 694]

A related project, MultiInstaller, is not a package manager, but a good way to install multiple packages, directly from git repositories (e.g. Github), from zip files or from existing folders in one-step.  I can start using a new version of Delphi in minutes rather than days.  The project is fork of the Silverpoint MultiInstaller.

[Mahdi Safsafi 225]

Quote

Packages will not contain any scripts (ie to build during install) for security reasons (I don't want to be running random scripts). 

Please reconsider this point ! Nowadays it's really important to use scripts (we're no longer on 19xx).

Also for the package name, it's important to have a formal rule for name. e.g a package's name may look like this : TMS::VCL::SomeProduct. Also we can group names by category (like cpan packages which I really like its concept) e.g: XML::MyXmlLib. So all XML stuffs are under XML::.

 

 

[Georgge Bakh 29]

Very good initiative!

The absence of dependency manager is a big problem of Pascal ecosystem. Not only CI scenarios suffers from it but library developers - there is no option to make a library dependent on another library as it become too complicated for most people to install and use it. So most libraries have *commons.pas units which contains pretty same purpose routines and classes.

I thought about it a lot and the only reason why I not started such a project is that I believe it should be a community effort. Or we'll end up with another package manager without packages and users.

 

I'm familiar with Maven from Java ecosystem and it does it's job well.

 

Some quick thoughts:

I see no much sense binding it to a particular IDE or even compiler. Let it work for FPC too!

Components and packages are good but libraries are also good and much easier to manage.

I think such a tools should have a command line based core with plugins for IDEs when needed. In that case CI server will be able to build project from scratch as it should be.

Simple workflow - a project config file at input and a directory structure with library/package files at output.

A package can be identified with an ID - something domain name-like works good here and partially solves problems with copyright etc. Please no GUIDs here!

To identify a dependency seems to enough: (ID, version, compiler/platform, type)

type is source or binary (dcu/ppu).

Therefore, to use a library it's needed to create a config file and run the tool. After it ends its work there is a ready to use directory structure with all needed files and maybe even compiler config file with paths.

[Vincent Parrett 763]

I'm open to suggestions when it comes to package naming conventions, but it's going to take a lot to convince me to allow running of scripts. Running random scripts that haven't been vetted for security is major issue. Perhaps a better option would be to include some sort of dsl that has some pre-defined functionality (like copy files etc). 

 

[Eli M. 38]

What about Chocolatey? Already for Windows.

 

https://chocolatey.org/

 

It already exists.

It's already for Windows.

Adding Delphi components to their ecosystem would help spread Delphi to other developers.

Edited February 14, 2019 by Eli M.

[Vincent Parrett 763]

2 minutes ago, Eli M. said:

What about Chocolatey? Already for Windows.

 

https://chocolatey.org/

 

It already exists.

It's already for Windows.

Adding Delphi components to their ecosystem would help spread Delphi to other developers.

 

Chocolatey is Nuget repurposed for installing applications, not for installing code libraries. It's also a .net application. As much as I love .net/c#, I think a delphi package manager should be written in delphi. 

[Eli M. 38]

I can contribute an FMX client that ties into whatever command line package manager you come up with. I already have one that uses a local CodeCentral database.

Edited February 14, 2019 by Eli M.

[Vincent Parrett 763]

@Eli M. Thanks. We're some way from that yet, but when we get to the IDE integration we'll be looking for people with UI skills to contribute. Right now I want to focus on the structure of packages/libraries etc, how they are defined, naming conventions, etc. 

[Georgge Bakh 29]

44 minutes ago, Vincent Parrett said:

 

Chocolatey is Nuget repurposed for installing applications, not for installing code libraries. It's also a .net application. As much as I love .net/c#, I think a delphi package manager should be written in delphi. 

I'd say in Free Pascal as such tool should be crossplatform and work for FPC too.

[Vincent Parrett 763]

I'm open to contributions that would allow supporting FPC, but my primary target is what I use, Delphi.  

[Remy Lebeau 1436]

2 hours ago, Georgge Bakh said:

I'd say in Free Pascal as such tool should be crossplatform and work for FPC too.

FreePascal already has its own Online Package Manager built right into the Lazarus IDE.

Edited February 15, 2019 by Remy Lebeau

[Vincent Parrett 763]

55 minutes ago, Remy Lebeau said:

FreePascal already has its own Online Package Manager built right into the Lazarus IDE.

As far as I can tell, it's pretty much like GetIt, ie a global package installer? 

 

What I'm proposing is handling per project dependencies, ie nothing is global. 

[dummzeuch 1517]

I'm not sure that such a tool should support freepascal and Lazarus too. This would:

• Add a lot more complexity
• Get the maintainer / developer into unnecessary open vs. closed source discussions
• Would be a direct "competitor" to an already existing tool which would again cause endless discussions.

I'm not a fan of tying it into Github, btw.

[dummzeuch 1517]

16 hours ago, pyscripter said:

A related project, MultiInstaller, is not a package manager, but a good way to install multiple packages, directly from git repositories (e.g. Github), from zip files or from existing folders in one-step.  I can start using a new version of Delphi in minutes rather than days.  The project is fork of the Silverpoint MultiInstaller.

I just tried to download and compile the sources: MultiInstaller_Icon.ico is missng.

[pyscripter 694]

@dummzeuch Missing icon file added.

[Stefan Glienke 2019]

Wasted effort imo - the fundamental design and architecture of the IDE and how (designtime)packages work is flawed to get something like this working properly.

 

Unloading of packages often does not work properly because they are not written for that or some things that happen during loading cannot be easily undone (sloppy developer or simply because the API might be missing).

Adding component/library to the IDE is not just loading a module but adding paths in numerous places, like library path for each supported platform, browsing path for stepping into code or even special debug DCU path.

 

And last but not least because there is no strict rule how packages and files should be structured and if the library path should only point to DCUs or to pas files which often causes vendors to place their stuff all over the place and even require modifying the windows path variable (OMG how I hate that!)

[Thijs van Dien 9]

A proper way to make design-time packages part of my project rather than of my IDE is just about the only thing that could get me to upgrade Delphi ever again. I'm sure it will be technically challenging, and that any implementation would be buggy for a long time (as with most even mildly complicated new features), but done well it would be a big step forward. Yes, packages would have to be made compatible with it, but the same was true for GetIt. As long as the old style "IDE packages" keep working too, I don't see the problem. It could be a chance to streamline how packages must be written and hopefully adding any paths would no longer be necessary for such "project packages". If it will still be possible for such packages to make hard to reverse changes to the IDE at runtime, some kind of sandboxing would be needed.

Edited February 19, 2019 by Thijs van Dien

[Remy Lebeau 1436]

You SHOULD already be able to have per-project packages, in all versions of Delphi.  The trick is to install the packages but NOT ENABLE them globally.  With no project loaded, install the packages as needed, and then disable them so they don't appear in the Palettes when no project is loaded.  Then, load a project, enable only the installed packages it actually needs, and close the project.  Repeat as needed for other projects.  Then, from now on, the IDE should load and unload packages on a per-project basis.  At least, this is how it used to work years ago, I haven't tried it in recent years.  Hopefully it still works.

[Vincent Parrett 763]

@Remy Lebeau That does still work, however it still doesn't allow you to install multiple versions of the same package, which makes life difficult. 

 

For example, on v7 I might use packageA.1.0.0 and on v8 I might use packageA2.2.0 - I don't want to update v7 to use the newer packageA, for a bunch of possible reasons.

 

There are (as others have pointed out) problems with unloading packages, lots of third party packages don't clean up properly and unloading causes IDE instability. 

[Vincent Parrett 763]

@Stefan Glienke It's shame you feel that way, I was thinking the spring framework would be a perfect candidate for something like this, and in fact I've been using it while playing with ideas for the package specification. 

 

Yes, design time package unloading will be a problem, most likely what we'll do is on package restore, if there is any change in the design time components installed we'll offer to restart the IDE. Of course this is not an issue for command line or CI builds, as we don't need to install components, just ensure the project can be built with the correct search paths. 

 

This project will have lots of technical hurdles, but I still think we can do better than what we have now.