Can the packages setup be updated for ICS in new versions?

[Geoffrey Smith 45]

I would like to suggest for the next version of ICS, the naming of the packaging be updated to follow the document from @Vincent Parrett from Finalbuilder found at https://www.finalbuilder.com/resources/blogs/advice-for-delphi-library-authors

This would help with a component I use in project I am developing when new versions of Delphi come out and also help other developers who use ICS in packages.

https://github.com/geoffsmith82/delphi-mqtt

 

Most people will not come across this issue unless they create a component that uses ICS.

[FPiette 383]

9 minutes ago, Geoffrey Smith said:

Most people will not come across this issue unless they create a component that uses ICS.

I create a lot of components using ICS and I never had this problem. I always add ICS source code to whatever component/application I create and never use run time packages.

 

And by the way, ICS exist for the last 27 years and no one ever asked to change package naming. This being said, I'm not against it. It is just that I have better use of my spare time 😉

Try to convince Angus which does almost all changes for several years now. I'm sure he read this.

 

[Angus Robertson 574]

I accept the package structure for ICS is out of control, with over 280 dpk, dproj and cbproj files I have to update when I add a unit, even updating a single Delphi release is 10 or more packages.  But for the same reason, changing it will take days of effort if we continue to support all versions of Delphi back to 7. 

 

Ideally an application should have been written years ago to create packages from a master file, as I did for the SSL/TLS root certificate packages, but it never happened.

 

So changing it for ICS V8 is highly unlikely, but very probable for ICS V9 that will only support recent compilers so we can use new language features.  Vincent's suggested package rules all make sense, for new projects. 

 

I used to create separate packages for my own ICS components without problems, but they are nearly all now part of the ICS distribution. 

 

Your MQTT project appears to be branched from the 10 year old https://github.com/pjde/delphi-mqtt and the simple way to avoid package problems in the future would be for the units and sample to be included with the ICS distribution, as I did with the new WebSocket unit recently. 

 

Angus

 

 

 

 

 

 

 

[Vincent Parrett 750]

19 hours ago, Angus Robertson said:

Ideally an application should have been written years ago to create packages from a master file

This is something I have thought about doing - managing package files for multiple compiler versions is really tiresome. The dproj format has evolved a bit over the years.. but it shoud still be doable. A command line tool that takes in a definition file and spits out the dpk and dproj would be pretty awesome. I should have some free time in 2036 to actually do it🙄 

[Geoffrey Smith 45]

@Vincent Parrett  I thought I had seen someone write a tool that did this... but haven't been able to find it again after searching for it.

Edited February 1, 2023 by Geoffrey Smith

[FPiette 383]

42 minutes ago, Vincent Parrett said:

I should have some free time in 2036 to actually do it🙄 

Same problem for each of us.

[Vincent Parrett 750]

Actually it's probably not that much work to create a package generator - the hardest part will be creating the definition schema.

 

Something like this (YAML)

schemaVersion: 0.1
package:
	name: VSoft.MessagingR
	compilerVersions: XE2-11
	libSuffix: %compilerVersion%
	packagesFolder: .\packages
	folder-template: Rad Studio %ProductVersion% %CodeName%
	platforms : Win32, Win64
	
files:
	- .\src\VSoft.Messaging.pas
	- .\src\VSoft.Messaging.Internal.pas
	- .\src\VSoft.Messaging.Dispatchers.pas
	- .\src\VSoft.Messaging.Channel.pas

requires:
	- rtl

compilerOptions : |
		{$R *.res}
		{$IFDEF IMPLICITBUILDING This IFDEF should not be used by users}
		{$ALIGN 8}
		{$ASSERTIONS ON}
		{$BOOLEVAL OFF}
		{$DEBUGINFO ON}
		{$EXTENDEDSYNTAX ON}
		{$IMPORTEDDATA ON}
		{$IOCHECKS ON}
		{$LOCALSYMBOLS ON}
		{$LONGSTRINGS ON}
		{$OPENSTRINGS ON}
		{$OPTIMIZATION OFF}
		{$OVERFLOWCHECKS OFF}
		{$RANGECHECKS OFF}
		{$REFERENCEINFO ON}
		{$SAFEDIVIDE OFF}
		{$STACKFRAMES ON}
		{$TYPEDADDRESS OFF}
		{$VARSTRINGCHECKS ON}
		{$WRITEABLECONST OFF}
		{$MINENUMSIZE 1}
		{$IMAGEBASE $400000}
		{$DEFINE DEBUG}
		{$ENDIF IMPLICITBUILDING}
		{$DESCRIPTION 'VSoft.Messaging Runtime for %ProductVersion%'}
		{$LIBSUFFIX '%LIBSUFFIX%'}
		{$RUNONLY}
		{$IMPLICITBUILD OFF}
		

The idea being there would be a bunch of built in variables like CompilerVersion, ProductVersion etc that can be sprinkled in the template and then replaced when the dproj/dpk files are generated. Obviously there is more to add to the above, like output folders, configs etc.. but it should be pretty simple to build on it. 

I'll make a start and chuck it up on github, open to suggestions on the definition schema. 

 

I'll probably use https://github.com/neslib/Neslib.Yaml for the yaml parser (haven't tried it yet though). 

 

 

[FPiette 383]

18 minutes ago, Vincent Parrett said:

I'll make a start and chuck it up on github, open to suggestions on the definition schema.

That would be really cool!

I suggest using JSON because it is widely used nowadays and ICS contains a JSON builder/parser (OverbyteIcsSuperObject.pas taken from https://github.com/hgourvest/superobject).

[Geoffrey Smith 45]

Here are my thoughts on the tool - hopefully not too obvious for you @Vincent Parrett

- Have a default set of template dproj files that should suit most people (maybe different ones for handling different platforms or strip out unneeded stuff like project magician)

- Ability to download templates for new versions of Delphi - perhaps these templates could just be stored in a git repository that people could update with git

- Option for custom dproj's though also able to be downloaded from a git project

- Spit out files required for Delphi Package Manager

[Vincent Parrett 750]

29 minutes ago, Geoffrey Smith said:

Have a default set of template dproj files that should suit most people (maybe different ones for handling different platforms or strip out unneeded stuff like project magician)

The cool thing about generating the dproj files from scratch (rather than using a template) is we don't need to generate a lot of the sh1te that rad studio does. Using templates would be tricky due to the differences between delphi releases. So I'm not keen on templates as such.

 

31 minutes ago, Geoffrey Smith said:

Spit out files required for Delphi Package Manager

More likely I will implement this in dpm itself.. as a dpm package could have more than one delphi package (dpk) in it. There is already a stubbed out dpm spec command

 

Quote

spec                                   Generates a package.dspec for a new package. If this command is run in the
                                           same folder as a project file (.dproj), it will create a tokenized dspec
                                           file.

 

 

 

[Remy Lebeau 1397]

On 1/31/2023 at 3:18 AM, Angus Robertson said:

Ideally an application should have been written years ago to create packages from a master file

That is the approach that Indy took a long time ago.  It has a master database of its units and their characteristics, and then there is a Package Generator app that reads in that database and spits out DPK files (no DPROJ, CBPROJ, or GROUPPROJ files at this time) for all of the supported compiler versions (well, not all of them, as the app hasn't been updated in a long time - packages for modern compiler versions have largely just been copied from earlier versions and edited by hand).

Edited February 1, 2023 by Remy Lebeau

[Geoffrey Smith 45]

@FPiette and @Angus Robertson I spoke to the original Author of the component and he was fine having it included into the ICS library.  See below.  My version has a few minor improvements and you can include that as well.

 

Quote

 

Hi Geoff

 

I don't have any issues about having it included in the ICS base install.

I also have a websocket based on ICS.

 

Regards

 

Paul 

 

 

Edited February 2, 2023 by Geoffrey Smith

[FPiette 383]

2 hours ago, Geoffrey Smith said:

I spoke to the original Author of the component

Which component?

[Angus Robertson 574]

It was the MQTT protocol I was proposing to add to ICS, WebSockets is done and dusted.

 

Angus

 

[Fr0sT.Brutal 900]

12 hours ago, Remy Lebeau said:

spits out DPK files (no DPROJ, CBPROJ, or GROUPPROJ files at this time)

DPKs almost do not change across versions, it's the *PROJ that cause real pain... I believe most DPKs could be easily cloned 1:1

[Remy Lebeau 1397]

8 hours ago, Fr0sT.Brutal said:

DPKs almost do not change across versions, it's the *PROJ that cause real pain...

True, which is why they are not being auto-generated right now.

8 hours ago, Fr0sT.Brutal said:

I believe most DPKs could be easily cloned 1:1

Almost.  There are some small differences between older and newer DPKs.

[Vincent Parrett 750]

On 2/1/2023 at 7:28 PM, FPiette said:

That would be really cool!

I suggest using JSON because it is widely used nowadays and ICS contains a JSON builder/parser (OverbyteIcsSuperObject.pas taken from https://github.com/hgourvest/superobject).

I did end up using JSON - simply because it's easier to parse 

 

I have made a start on this - https://github.com/VSoftTechnologies/DPackGen

Example definition file. It's modelled on how I did the package spec files for my packag manager project (which I really need to get back to!) - so I was able to copy/pasta a lot of the code or at least borrow the design of the definition parsing and package generation. Supporting XE2-11.x I uses templates, you can define more than one if a compiler version needs different settings. I might also look at allowing templates to inherit from another template, so if a template only needs to change a few settings it can be a bit less verbose. 

 

Note that you don't have to use the template, you can define all the settings on the targetPlatform - just remove the template property and fill in the rest of the props like in the template - but using templates reduces verbosity. 

 

{
	definitionSchemaVersion: 0,
	name : "VSoft.MessagingR",
	packagesFolder: ".\\packages",
	projectGUID: "{8FF7EA44-009F-4370-BDA2-3C30D26F6FC9}",
	frameworkType : "VCL",
	packageType : "Runtime",
	targetPlatforms : [
		{
		  compiler : "XE2",
		  platforms : "Win32, Win64",
		  template : "default"
		},
		{
		  compiler : "XE3",
		  platforms : "Win32, Win64",
		  template : "default"
		},
		{
		  compiler : "XE4",
		  platforms : "Win32, Win64",
		  template : "default"
		},
		{
		  compiler : "XE5",
		  platforms : "Win32, Win64",
		  template : "default"
		},
		{
		  compiler : "XE6",
		  platforms : "Win32, Win64",
		  template : "default"
		},
		{
		  compiler : "XE7",
		  platforms : "Win32, Win64",
		  template : "default"
		},
		{
		  compiler : "XE8",
		  platforms : "Win32, Win64",
		  template : "default"
		},
		{
		  compiler : "D10.0",
		  platforms : "Win32, Win64",
		  template : "default"
		},
		{
		  compiler : "D10.1",
		  platforms : "Win32, Win64",
		  template : "default"
		},
		{
		  compiler : "D10.2",
		  platforms : "Win32, Win64",
		  template : "default"
		},
		{
		  compiler : "D10.3",
		  platforms : "Win32, Win64",
		  template : "default"
		},
		{
		  compiler : "D10.4",
		  platforms : "Win32, Win64",
		  template : "default"
		},
		{
		  compiler : "D11",
		  platforms : "Win32, Win64",
		  template : "default",
		  libSuffix : "%compilerVersion%"
		}

	],
	templates : [
		{   
			name : "default",
			libSuffix : "%compilerVersion%",
			folderNameTemplate: "Rad Studio %Compiler% %CompilerCodeName%",		
			descriptionTemplate: "VSoft.Messaging Runtime for Delphi %Compiler%",
			files : [
			".\\src\\VSoft.Messaging.pas",
			".\\src\\VSoft.Messaging.Internal.pas",
			".\\src\\VSoft.Messaging.Dispatchers.pas",
			".\\src\\VSoft.Messaging.Channel.pas"
			],
			requires : [
				"rtl"
			],
			dpkOptions: [
				"{$R *.res}",
				"{$IFDEF IMPLICITBUILDING This IFDEF should not be used by users}",
				"{$ALIGN 8}",
				"{$ASSERTIONS ON}",
				"{$BOOLEVAL OFF}",
				"{$DEBUGINFO ON}",
				"{$EXTENDEDSYNTAX ON}",
				"{$IMPORTEDDATA ON}",
				"{$IOCHECKS ON}",
				"{$LOCALSYMBOLS ON}",
				"{$LONGSTRINGS ON}",
				"{$OPENSTRINGS ON}",
				"{$OPTIMIZATION OFF}",
				"{$OVERFLOWCHECKS OFF}",
				"{$RANGECHECKS OFF}",
				"{$REFERENCEINFO ON}",
				"{$SAFEDIVIDE OFF}",
				"{$STACKFRAMES ON}",
				"{$TYPEDADDRESS OFF}",
				"{$VARSTRINGCHECKS ON}",
				"{$WRITEABLECONST OFF}",
				"{$MINENUMSIZE 1}",
				"{$IMAGEBASE $400000}",
				"{$DEFINE DEBUG}",
				"{$ENDIF IMPLICITBUILDING}",
				"{$DESCRIPTION '%description%'}",
				"{$LIBSUFFIX '%LIBSUFFIX%'}",
				"{$RUNONLY}",
				"{$IMPLICITBUILD OFF}"
			]
		}
	]
}

 

I used my package manager to install the dependencies (JsonDataObjects, Spring4D.Base & VSoft.CommandLineParser) so if you want to build it you would need to grab those projects from github/bitbucket and update the search paths. 

 

DPK generation mostly works - need to figure out how to deal with forms/frames. 

 

DPROJ generation is a work in progress - not there yet - the dproj format is not trivial to generate due to config inheritance etc. 

 

Hopefully I will find some time over the weekend to work on it and get it to a usable state. I need to flesh out the properties for the TargetPlatform/Template for things like search paths, namespace prefixes, output paths etc. 

 

BTW I have not made any provisions for generating CBuilder packages - I guess that can come later if someone wants to work on it - I don't have a need for it so not likely to do it myself.

 

Edited February 3, 2023 by Vincent Parrett
additional comment

[Lars Fosdal 1792]

2 hours ago, Vincent Parrett said:

I did end up using JSON - simply because it's easier to parse 

That is great, because
YAML - Yikes Awful Markup Language

[Vincent Parrett 750]

@Uwe Raabe are you interested in helping out with this project (since you are the dproj wizard!) - https://github.com/VSoftTechnologies/DPackGen

 

The idea is to create a definition file which can be used with the tool to generate dpk/dpr/dproj files for the supported delphi versions - something that pretty much all delphi library authors struggle with (add one file, edit 50 files to add that one file!). 

 

I have the basics working - dproj generation is somewhat iffy as I need to figure out all the differences in the various dproj versions. Also need to flesh it out with more properties etc in the definition file. 

 

[Uwe Raabe 2057]

5 hours ago, Vincent Parrett said:

are you interested in helping out with this project (since you are the dproj wizard!)

 Thanks! Although a bit busy in the moment, I will nevertheless have a look when time allows.

 

5 hours ago, Vincent Parrett said:

dproj generation is somewhat iffy as I need to figure out all the differences in the various dproj versions.

Actually I don't care for that. My (probably limited) experience is that dproj files are downward compatible at least for the recent versions. There may be problems when you go down to the lower XE levels, though.

[Vincent Parrett 750]

3 hours ago, Uwe Raabe said:

 Thanks! Although a bit busy in the moment, I will nevertheless have a look when time allows.

Yeah I understand! I really wasn't planning on working on this, but I realised after reading this thread that it was a major pain point on my OS projects. 

3 hours ago, Uwe Raabe said:

Actually I don't care for that. My (probably limited) experience is that dproj files are downward compatible at least for the recent versions. There may be problems when you go down to the lower XE levels, though.

Thanks, that's useful to know. I've only gone as far back as XE2 as that's the earliest version I have installed or support in my projects.

[Uwe Raabe 2057]

On 2/1/2023 at 9:05 AM, Vincent Parrett said:

Actually it's probably not that much work to create a package generator - the hardest part will be creating the definition schema.

I suggest another approach: Instead of a definition scheme, I would rather take a project for the latest Delphi version and derive the lower ones from that. You need that anyway (at least for some more recent version) and you neither have to invent a new format nor write a GUI for that.

 

IIRC, I gave you access to the Package Magician sources a while ago. Have a look in the CreatePackageStructure and PackageSynchronizer projects in the Berlin package folder, that make heavy use of the uPackageMagicianStructure unit. It will definitely need some work, but if I were to find a solution, that one would be my starting point.

[Angus Robertson 574]

The concept of taking the latest packages and modifying them to be acceptable to earlier versions of RAD Studio would seem rather faster and safer than generating all those packages from scratch using templates and rules.  

 

Or at least starting from minor modification of the latest packages, such as changing a version name to a macro. 

 

The tool would then strip out new XML for platforms according to hard coded rules, depending on when they were supported, and so on.  But I've no ideal how much the XML schema has changed over the years.

 

Angus

 

[Vincent Parrett 750]

34 minutes ago, Angus Robertson said:

he concept of taking the latest packages and modifying them to be acceptable to earlier versions of RAD Studio would seem rather faster and safer than generating all those packages from scratch using templates and rules.  

 

My issue with this is that it's not uncommon for the units etc to differ between platforms. If we use the latest version and just copy that then any customisations you need to make to a particular version would be lost each time you generate the packages. 

[Angus Robertson 574]

Guess I'm thinking primarily from the perspective of Windows VCL and FMX packages, don't get involved in any other platforms. 

 

Angus