Best way to replace D11 distributed Indy with latest Git Indy?

[Ian Branch 128]

Hi Team,

I fudged it with D10 but I'm not sure I did it the best way.

So what is the best way to replace/isolate the D11 installed/distributed Indy with latest Git Indy?

I would prefer NOT to delete any of the installed/distributed Indy but if I must...

 

Regards & TIA,

Ian

[Fr0sT.Brutal 900]

Just remove all built-in Indy's paths from Library and Search. To be 100% sure just rename Indy's folder adding "_" or something alike.

[Ian Branch 128]

Thank you.  I thought it would be like that.

I also thought the Git Indy would have the D11 Libraries build bits but I don't see any. 😞

 

Regards,

Ian

[Anders Melander 1795]

55 minutes ago, Fr0sT.Brutal said:

Just remove all built-in Indy's paths from Library and Search. To be 100% sure just rename Indy's folder adding "_" or something alike.

I would leave the bundled Indy, and the global library search path, be and instead add the Indy repo as a submodule to the project repo and add the local path to that submodule to the project search path.

Of course that only works as long as the design time properties of the bundled Indy and the one in the Git repo are compatible.

[Fr0sT.Brutal 900]

The even simplest option is to just rename Indy folder. But anyway one has to deal with design packages (ensure they're replaced by new ones).

1 hour ago, Anders Melander said:

I would leave the bundled Indy, and the global library search path, be and instead add the Indy repo as a submodule to the project repo and add the local path to that submodule to the project search path. 

Well, it depends on whether he intends to use built-in Indy or not

Edited October 4, 2021 by Fr0sT.Brutal

[dummzeuch 1506]

2 hours ago, Anders Melander said:

I would leave the bundled Indy, and the global library search path, be and instead add the Indy repo as a submodule to the project repo and add the local path to that submodule to the project search path.

While I agree on having the new Indy sources in the repo as svn:external/submodule, I think leaving the original library in the search path (or even on the hard disk) may be dangerous. I have spent countless hours hunting for bugs that eventually turned out where caused by compiling the wrong sources / linking the wrong dcus. This can't happen if you delete them.

[Remy Lebeau 1403]

8 hours ago, Ian Branch said:

I also thought the Git Indy would have the D11 Libraries build bits but I don't see any. 😞

Not yet, because I don't have D11 installed to generate them. But anyone is more than welcome to submit a Pull Request to contribute them.

6 hours ago, dummzeuch said:

I think leaving the original library in the search path (or even on the hard disk) may be dangerous. I have spent countless hours hunting for bugs that eventually turned out where caused by compiling the wrong sources / linking the wrong dcus. This can't happen if you delete them.

Agreed (or at least rename the folder, in case you need to recover it later).  That is usually the first thing I do when I install a new IDE version, and then have my projects point to the latest Indy repo source.

Edited October 4, 2021 by Remy Lebeau

[Ian Branch 128]

Hi Team,

As I did with D10, I want to take the 'distributed' Indy out of the equation and just use the Git Indy as it is, I believe, more up-to date.

 

Remy,

I had a try at creating he D11 versions but got lost and it failed to build. 😞

 

Ian

[corneliusdavid 214]

This is an old topic but I just ran across it while updating Delphi 11 with Indy from GitHub (will have a pull-request for D11 project files soon). While working through this, I decided to write up some instructions for replacing the default installed Indy in various versions of Delphi since I've needed to do this in several cases where I support legacy Delphi projects. I figured others might benefit from this--reading this discussion confirms that hunch.

 

My assumption going into this (after having done it in a few older versions of Delphi) is that people simply remove all traces of the installed Indy that comes with Delphi, then install Indy from GitHub into a completely separate folder and compile/install it like they would any other third-party or open source library where the binaries are put out in the "public documents" folders (e.g. C:\Users\Public\Documents\Embarcadero\Studio\22.0).

 

One of the suggestions here is to rename the Indy folder. Does that mean the Indy10 folder under Delphi's source? But that doesn't replace the compiled libraries installed in Delphi; the BPLs and DCUs are mixed in with other library files in the bin, bin64, lib\win32, lib\win64, and other platform folders of Delphi.

 

So is the idea that when you compile the fresh source from GitHub, you'll direct the binaries to the Delphi bin and lib folders? If they're installed under C:\Program Files (x86)\Embarcadero\Studio\22.0, you'll have to run Delphi as administrator to update those protected files.  Or do you install Delphi outside of the "C:\Program Files (x86)" folder?

 

I'm really just trying to understand how others do it. Personally, I let Delphi get installed into the default "Program files (x86)" folder like other Windows applications. And libraries/packages that I build go to the "public documents" folder.

 

So then, when I want to override the default installed Indy library with fresh source from GitHub, I have two choices: 

• Delete all Indy files from Delphi's bin and lib folders,
• Add the paths for the new files at the top of the Library path so Delphi will find the new ones first.

I've tried the second option and have had problems. Having multiple copies of the same BPLs and DCUs, even if they're properly ordered in the Library Path option, causes problems for me--there's always some other library that uses the wrong version of the binary and complains about it compiled with the wrong version or incompatible parameters or something. I found that deleting the files is best and works every time.  Of course, this is rather tedious because there are debug and release versions of all the Id*.dcu and Indy*.dcu and Indy*.bpl and dclIndy*.bpl for each platform--that's a lot of files to make sure you've cleaned out (which is why I'm also considering writing a "clean" app to assist with this).

 

So before I release these instructions, I just want to ask those who have replaced the default Indy with the one from GitHub: What is the process you use?

[Fr0sT.Brutal 900]

Isn't built-in Indy an optional feature that could be unchecked during install?

[Remy Lebeau 1403]

4 hours ago, Fr0sT.Brutal said:

Isn't built-in Indy an optional feature that could be unchecked during install?

No, it is not optional, it is always pre-installed.  Some day, in the future, I would like to get Indy into GetIt (https://github.com/IndySockets/Indy/issues/134) and out of the installer.

Edited July 13, 2022 by Remy Lebeau

[Ian Branch 128]

3 minutes ago, Remy Lebeau said:

No, it is not optional, it is always pre-installed.  Some day, in the future, I would like to get Indy into GetIt and out of the installer.

Hear Hear.  The sooner the better.

It is interesting that other libraries can be optionally installed.

I wonder if Delphi is using some element of Indy, thus requiring it to be installed....

 

[Remy Lebeau 1403]

4 hours ago, corneliusdavid said:

My assumption going into this (after having done it in a few older versions of Delphi) is that people simply remove all traces of the installed Indy that comes with Delphi, then install Indy from GitHub into a completely separate folder and compile/install it like they would any other third-party or open source library where the binaries are put out in the "public documents" folders (e.g. C:\Users\Public\Documents\Embarcadero\Studio\22.0).

Essentially, yes - with caveats, depending on the IDE version (mostly old ones), which are documented in Indy's install instructions.

4 hours ago, corneliusdavid said:

So is the idea that when you compile the fresh source from GitHub, you'll direct the binaries to the Delphi bin and lib folders?

I rename the pre-installed Indy folders and move the pre-installed Indy binaries to a separate folder (in case I ever need to restore them), and then I compile Indy in my own folder and install the binaries from there.  That way, I can make changes to Indy and have them show up in projects/IDE.

4 hours ago, corneliusdavid said:

If they're installed under C:\Program Files (x86)\Embarcadero\Studio\22.0, you'll have to run Delphi as administrator to update those protected files.  Or do you install Delphi outside of the "C:\Program Files (x86)" folder?

I install the IDE in the default installation path.

4 hours ago, corneliusdavid said:

So then, when I want to override the default installed Indy library with fresh source from GitHub, I have two choices: 

• Delete all Indy files from Delphi's bin and lib folders,
• Add the paths for the new files at the top of the Library path so Delphi will find the new ones first.

Basically, yes.

4 hours ago, corneliusdavid said:

I've tried the second option and have had problems. Having multiple copies of the same BPLs and DCUs, even if they're properly ordered in the Library Path option, causes problems for me--there's always some other library that uses the wrong version of the binary and complains about it compiled with the wrong version or incompatible parameters or something. I found that deleting the files is best and works every time.

Some caveats similar to that are mentioned in Indy's install instructions, but only in relation to Embarcadero's own technologies that use Indy internally.

4 hours ago, corneliusdavid said:

Of course, this is rather tedious because there are debug and release versions of all the Id*.dcu and Indy*.dcu and Indy*.bpl and dclIndy*.bpl for each platform--that's a lot of files to make sure you've cleaned out

Yup.  And you have to make sure you clean up ONLY the ones that relate to Indy, and not to someone else who might have also used the "Id" prefix (albeit rare)...

4 hours ago, corneliusdavid said:

(which is why I'm also considering writing a "clean" app to assist with this).

Cool.

[Remy Lebeau 1403]

4 minutes ago, Ian Branch said:

I wonder if Delphi is using some element of Indy, thus requiring it to be installed....

Yes, there are some Embarcadero technologies that use Indy internally, and so the installer has included Indy because of that.  At one point, the techs were linked directly to the public version of Indy that shipped with the IDE, making it virtually impossible for users to update Indy without breaking those techs.  But over the years, Embarcadero has updated their tech to use a private version of Indy instead (or their own libraries), so users can freely update the public version of Indy.  But, there have been a few glitches in that process from time to time, causing unwanted linkages to the public Indy again.  Nothing in recent years, that I recall, though.

[Vincent Parrett 753]

I have this issue every time I install Delphi. I never use the bundled version, because I have no idea what version it is (usually outdated, they are terrible at keeping third party libs up to date). I cannot look at github and figure out what commit it is and see what has changed.  

 

So after install I manually delete all Id*.dcu, Indy*.dcp etc - it's annoying and tedious (have to do it for each platform). 

[corneliusdavid 214]

53 minutes ago, Vincent Parrett said:

So after install I manually delete all Id*.dcu, Indy*.dcp etc - it's annoying and tedious (have to do it for each platform). 

Seems like this might be a ripe opportunity for an Automise script. 

[Vincent Parrett 753]

26 minutes ago, corneliusdavid said:

Seems like this might be a ripe opportunity for an Automise script. 

More like a FinalBuilder script since we are in a dev environment (although either tool can do it). 

[corneliusdavid 214]

Thanks @Remy Lebeauand @Vincent Parrettfor confirming the direction I'm taking. I'm going to work towards an automation tool of some sort for the Indy libraries.

[Vincent Parrett 753]

Just now, corneliusdavid said:

I'm going to work towards an automation tool of some sort for the Indy libraries.

Or you could help working towards an automation tool for all libraries 😉 - I haven't had much time to work on it lately (too busy working on FinalBuilder) but I'm still keen to get it going.. I use it every day with my own packages.  

 

 

[corneliusdavid 214]

I had heard of this but hadn't spent time reading about it because I didn't immediately see a use case. But now that you mention it in context with keeping Indy updated, I'm taking a second look--and will definitely look deeper. 

[Vincent Parrett 753]

2 minutes ago, corneliusdavid said:

I didn't immediately see a use case.

Do you use GetIt? Or a package manager on other dev platforms - like nuget for .net, npm for javascript, Gem for ruby, Crate for rust etc. The idea is to provide a standard way to deliver and install libraries. GetIt doesn't meet the expectations of anyone who has used a real package manager. 

 

There is quite a lot done - but still plenty more to make it production ready. 

IDE integration looks like this - installing components into the pallete isn't working yet as it requires more work on the project group support (dealing with conflicts etc). 

 

I have also made some good progress on a package server (not a public repo yet). 

 

 

I have so far not managed to get the community interersted in helping out - so progress is painfully slow. 

[corneliusdavid 214]

4 minutes ago, Vincent Parrett said:

Do you use GetIt?

Yes--quite a bit. In fact, I wrote AutoGetIt so I guess I do see a use case for automating the management of packages! But after installing Delphi and the packages, my installation needs are done. I don't completely get the need to be continually installing/uninstalling/upgrading packages all the time--or maybe that's not the real purpose of a package manager. I use a handful of components and there aren't that many changes that often--my time is spent working on projects, not package management so much. This seems like a lot of work to only occasionally save some time.

9 minutes ago, Vincent Parrett said:

Or a package manager on other dev platforms - like nuget for .net, npm for javascript, Gem for ruby, Crate for rust etc.

I've worked a little with .NET and gone through some courses on Javascript so have been exposed to nuget and npm but only used them to follow explicit directions for getting an environment up to speed. My perspective is that there are far more libraries and packages in the .NET and JS world so the need there is obvious.

 

Do you see DPM eventually being able to take a fresh Delphi install, run DPM with a list of libraries and in a few minutes, have a complete dev environment set up and ready to go? I think that's how I read it. I would like that ability and even explored that idea by installing Delphi with Chocolatey a few years ago, but there are so many different methods of installation, acknowledging copyrights, inserting license keys, etc. (like you point out in your blog), that I dropped it--the task took far more work than it was worth (to me).

 

Now with Indy, having an automated install/update process could benefit a LOT of people right away--and with it being an open-source project, it's far easier to work with.

 

I'd be happy to take this discussion off-line and look at DPM further with you.

[Vincent Parrett 753]

6 minutes ago, corneliusdavid said:

I've worked a little with .NET and gone through some courses on Javascript so have been exposed to nuget and npm but only used them to follow explicit directions for getting an environment up to speed. My perspective is that there are far more libraries and packages in the .NET and JS world so the need there is obvious.

Before nuget and npm, there were not nearly as many libraries, and those that were out therre were hard to find. It was just like Delphi.  Those tutorials you found would not have worked without the ability to restore the packages those projects referenced. 

 

1 minute ago, corneliusdavid said:

 

Do you see DPM eventually being able to take a fresh Delphi install, run DPM with a list of libraries and in a few minutes, have a complete dev environment set up and ready to go?

If you mean having every third party library you use all installed at the same time - then no. 

 

The idea is you open a project in the IDE, DPM ensures that any libraries that project (or project group) uses will be installed. Not just any version of the libraries - the specific version that the project references (this is an important feature that Getit doesn't.. get). If the package has been installed before, it would available in your local package cache and the install takes ms - if not then it will download that package and it's dependencies (another feature Getit is missing) and install them. 

 

8 minutes ago, corneliusdavid said:

Now with Indy, having an automated install/update process could benefit a LOT of people right away--and with it being an open-source project, it's far easier to work with.

Sure, but Indy is just one of many open source delphi projects out there. Currrently they all have different instructions for installing.. go and look at the issues tab on github for most of them and there will be loads of people struggling to install them - expecially if they depend on other libraries. 

 

Delphi is absolutely being held back as legacy tool because of it's lack of a real package manager (well that and it's poor quality and lack of modern language features). GetIt is a toy, a tightly controlled black box with a curated list of packages (only ones embarcadero likes). Most of those are constantly out dated.. and if you install from there you always get the latest version that they decide on - what happens when that latest version breaks your project - how do you roll back. Being able to control the version of packages used by a project is an absolute must. Add automated builds/CI into the mix -  do you really want a random version installed when you run a production build? 

 

Anyway, I'll shut up now.. I just get frustrated when I see or read about yet another way to install a single library 😞

 

[corneliusdavid 214]

34 minutes ago, Vincent Parrett said:

The idea is you open a project in the IDE, DPM ensures that any libraries that project (or project group) uses will be installed. Not just any version of the libraries - the specific version that the project references

Ah! I see! That would be nice. I have a couple of legacy projects I support in Delphi XE that I have to keep in separate virtual machines because of conflicting components--DPM would solve that. (Oh wait--DPM is for XE2 and above...)

41 minutes ago, Vincent Parrett said:

I just get frustrated when I see or read about yet another way to install a single library

Got it! I'll look to see how Indy can fit in with DPM. 

[Vincent Parrett 753]

1 minute ago, corneliusdavid said:

DPM would solve that. (Oh wait--DPM is for XE2 and above...)

Yeah there are just too many issues/barriers with XE that made it too hard to support - for all my libraries now XE2 is the minimum I will support.