The Delphi Parser - FIBPlus, BDE, legacy Migration

[Jasonjac2 0]

Hi All,

 

I have an old system:

•  2 x EXEs (D7)
• 330K combined lines of code
• BDE
• FibPlus
• IBX
• JVCL
• JCL
• TMS
• SMExport (very small amount)
• Custom Components (very small part)
• CNWizards – logging – (very small part)

I want to end up with on DAC (FireDAC unless anyone talks me out of it) and D10.4

 

I have invested in "The Delphi Parser".  The author is being very helpful, but ultimately for FIBPlus, I am going to get my hands dirty and modify the script to teach it what to convert into what.  I was wondering if anyone else has travelled this path or is about to.  Things I was interested in.  How you did it, what gotchas you got etc.

 

I like the idea of a "brighter than Grep" parser that understands the structure of a Delphi App. 

[Fr0sT.Brutal 900]

I didn't understood a thing

[Jasonjac2 0]

Hi,

Sorry Fr0sT.Brutal I didn't explain the background very well.  I was hoping that someone had done some work on migrating FibPlus code to a supported data access platform and at the same time was sharing my thoughts on migrating my 300K+ lines of code from D7 to D10.4.

 

See (*) notes at the bottom.

Update: While I have a hiccup with The Delphi Parser (1), I thought I would try and split the problem into:

• JVCL / JCL - Just get that installed (2)
• CnWiazrds - Just get that installed (3)
• BDE & IBX (as I am hoping the Delphi Parser will sort almost all of that for me)
• FibPlus - Hoping to use the Delphi Parser, but need to know what needs translating
• Everything else

I took the simplest part of the system that I could get to compile in isolation and all the dependent units (basically the logging into the database and the datamodule that does the auditing, but required about 4 more units), got it compiling under D7, then moved to D10.4.  Of course, without FIbPlus, it didn't compile, so I went back and forth creating Mocks by:

• 10.4 Attempt to Compile
• Take the error (undeclared class / property / method), goto the line and copy the exact line
• Go back to D7, find the line, ctrl click to the declaration, record the unit,  etc
• Go back to 10.4 and create the stubb in the correct unit.
• Repeat until compile.

This has left me with about 8 new units that mirror the declaration of the missing classes, but with no actual code.  It compiles and of course, I can't view the DFMs in the designer as there are none of the fibplus components installed in D10.4.

 

Of course, there could be lots of component properties stored in the .dfm's that doesn't exist in the code, so even though it now compiles, it won't run as the first property it finds as is streams the .dfms in is going to stop it.

 

So next up was to use GExperts component to code back in D7 to pull in the different types of FIB+ components in and copy these to some extra unit in my new 10.4.

Did this and got those to compile, so in my FibPlus mock units I have things like this:

  TpFIBDatabase = class(TFIBDatabase)
  public
    DatabaseName : String;
    Connected : boolean;
    DBParams : TStrings;
    SQLDialect : Integer;
    procedure close;
    procedure open;
  public // dfm Only
    DBName : String;
    DefaultTransaction : TFIBTransaction;
    Timeout : integer;
    DesignDBOptions : TDesignDBOptions;
    WaitForRestoreConnect : integer;

  end;

So next up:

• work out what these should map onto in FirDAC
• Then either teach Delphi Parser to convert to FireDAC OR inherit these components from the suitable FirDAC components and add in mapping code.
  • Oren of The Delphi Parser and it looks like is happy to give me a hand / work out the mapping with some examples.

 

I realise there aren't any questions in this one, but I searched and searched and couldn't find any real help with the FIBPlus migration, so these notes may be of some help to someone, sometime. 

 

I'll post another update when it compiles and actually accesses the data!  Have a good week all.

 

 

(1) Delphi Parser- Raised on Friday PM, so not a negative thing as I know the Author is very responsive, so expect to get that sorted when he is back in work

Had contact with the Author over the weekend and managed to get a conversion done, which is great.  Now I need time to see how well it managed BDE -> FIREDAC conversion.

(2) JVCL / JCL -  had a strange one with the debug library paths confusing the compiler and pointing to the older version that was installed with Tokyo - not sure if something got dragged over with the upgrade, but meant I: refreshed my memory about the order that paths are used; cleaned my paths up a bit.

(3) CnWizards - strange, the compiler directive doesn't seem to be working: 

 

 

[Fr0sT.Brutal 900]

I've no idea what Delphi Parser is but AFAIK FIBPlus are available in more or less recent version. IBX is included in Delphi RTL so almost no need to change. BDE is of course dead.

Anyway as you're migrating from D7 the most serious problem you'll likely have is Ansi=>Unicode transfer. After that all other changes will look like  piece a cake.

[Jasonjac2 0]

1 hour ago, Fr0sT.Brutal said:

I've no idea what Delphi Parser is but AFAIK FIBPlus are available in more or less recent version. IBX is included in Delphi RTL so almost no need to change. BDE is of course dead.

Anyway as you're migrating from D7 the most serious problem you'll likely have is Ansi=>Unicode transfer. After that all other changes will look like  piece a cake.

Hi,

OK, cool, "FIBPlus are available in more or less recent version." - where is this, I could only find a reference on:  https://shareappscrack.com/fibplus/?token=75016374 which worried me a bit as it looks like a crack site.

 

BDE - I actually installed the BDE a while back, but don't think the VCL components actually exist, so I agree.... completely dead 🙂

 

" Ansi=>Unicode transfer" - in practical terms, I am not sure what this is going to mean to me.

 

Another issue is the system is still on Dialect 1 of an old version of FB.  I need to work out the migration path for that also, but I thought to get it all over to latest Delphi and then worry about that.

 

So Devrace aren't operating anymore, so would you stick with FIbPlus or migrate away?  I don't see many dev's using it or a lot of recent hits regarding it, so worry that the ongoing maintenance for me might be high - but I would be keen to know if there is a group of dev's still using it.   I loved FIB & FIBPlus by the way and have excellent memories of meeting up with Serge (sp) and Marina at the FB Conference.

 

Regards & keen to get your input. Thank you for taking the time to reply.

 

"The Delphi Parser" is a 3rd party tool that has scripts for converting Delphi code, so the classic is BDE -> FireDAC, it is more than refind and it understands context and the author can tweak the system to add new types of conversion for the .pas and .dfm.   It is interesting stuff and I think much needed.  Historically it has been out of my price range, but recently there have been promotions and the author is doing a webinar with EMBT this week I think.

 

[Fr0sT.Brutal 900]

1 hour ago, Jasonjac2 said:

OK, cool, "FIBPlus are available in more or less recent version." - where is this, I could only find a reference on:  https://shareappscrack.com/fibplus/?token=75016374 which worried me a bit as it looks like a crack site.

I found this https://github.com/madorin/fibplus

1 hour ago, Jasonjac2 said:

" Ansi=>Unicode transfer" - in practical terms, I am not sure what this is going to mean to me.

Starting from D2009, string contains 2-byte widechars. Depending on code, transition could take from minutes to months (if many code relies on assumption that Char is 1-byte sized).

1 hour ago, Jasonjac2 said:

Another issue is the system is still on Dialect 1 of an old version of FB.  I need to work out the migration path for that also, but I thought to get it all over to latest Delphi and then worry about that.

There was no critical compatibility breaks in dialect 3. Again, depending on code and complexity of queries, it could take from simple structure clone and data pumping to long and hard process.

1 hour ago, Jasonjac2 said:

So Devrace aren't operating anymore, so would you stick with FIbPlus or migrate away?  I don't see many dev's using it or a lot of recent hits regarding it, so worry that the ongoing maintenance for me might be high - but I would be keen to know if there is a group of dev's still using it.   I loved FIB & FIBPlus by the way and have excellent memories of meeting up with Serge (sp) and Marina at the FB Conference.

Again it depends on how hard your code is tied to data access components. In theory, changing them is a matter of several hours. Nowadays I'd advice to use a maintained library like *DAC or Zeos

Edited August 24, 2020 by Fr0sT.Brutal

[Jasonjac2 0]

On 8/24/2020 at 1:00 PM, Fr0sT.Brutal said:

I found this https://github.com/madorin/fibplus

Starting from D2009, string contains 2-byte widechars. Depending on code, transition could take from minutes to months (if many code relies on assumption that Char is 1-byte sized).

There was no critical compatibility breaks in dialect 3. Again, depending on code and complexity of queries, it could take from simple structure clone and data pumping to long and hard process.

Again it depends on how hard your code is tied to data access components. In theory, changing them is a matter of several hours. Nowadays I'd advice to use a maintained library like *DAC or Zeos

You star - I don't know how I missed the FibPlus Repo.  See "Later section"

 

Update - I started this a couple of days ago and forgot to post: While I have a hiccup with The Delphi Parser (1), I thought I would try and split the problem into:

• JVCL / JCL - Just get that installed (2)
• CnWizards - Just get that installed (3)
• BDE & IBX (as I am hoping the Delphi Parser will sort almost all of that for me)
• FibPlus - Hoping to use the Delphi Parser, but need to know what needs translating
• Everything else

 

 

 

(1) - Raised on Friday PM, so not a negative thing as I know the Author is very responsive, so expect to get that sorted when he is back in work

(2) JVCL / JCL -  had a strange one with the debug library paths confusing the compiler and pointing to the older version that was installed with Tokyo - not sure if something got dragged over with the upgrade, but meant I: refreshed my memory about the order that paths are used; cleaned my paths up a bit.

(3) CnWizards - strange, the compiler directive doesn't seem to be working in 10.4

 

.... Some time later....... 27th August...

 

I actually installed the BDE & BDE components in D10.4 and forked the FibPlus Repo.  This isn't to continue with either in 10.4, but means I can move all migration effort to the 10.4 machine and not worry.  To this end I have brought the project up in the IDE and realised I also use : 2 components I wrote, a TVersionControl and WPTools from WPCubed.

 

Still planning on the scripted migration.  Thanks for your time and hope this might help someone else some day.

 

I'll try and do another update over the weekend.

 

Regards, JAC

 

[Zentori 0]

Sorry for necroposting this thread, but I'm interested in knowing if you finally went through automated migration from BDE/IBX and FibPlus to FireDAC. Thanks.,

[Jasonjac2 0]

"necroposting" love it!

No I didn't.  I didn't get the the tool I was looking at to work had to put the project down.  It is back on the agenda, but now moving from 7 -> 12.x!   

Are you doing the same and how are you getting on?

Another entry into the marketplace is GDK who advertise specialising in such migrations.  I had a quick chat with them, but haven't progressed it further.

 

I could really do with moving to a later version as there are lots of things I want our app to do that would leverage modern libraries / components that use modern language features.

[corneliusdavid 214]

I used (or tried to use) Delphi Parser about three years ago. I don't have FIBPlus and I was using their VCL->UniGUI migration tool, so my experience may not be as relevant but I did work with their script quite a bit, fixing several missing standard VCL controls, adding support for RaizeControls, and several other things.

 

My attempt was to take a working Delphi XE desktop (VCL) application and migrate it to UniGUI to turn it into a web app. I never got it completed and don't remember all the reasons why but I do remember being frustrated at Delphi Parser's interface, where each time you run it, you basically have to start the program over and go through several steps. It was a slow, iterative process and while suggestions for improvement were sent to the author (who seemed to appreciate them), none of them ever made it into an update.

 

Have you looked at reFind? I'm using that in a major migration from Delphi 5 and the BDE to Delphi 12 and FireDAC. Perhaps between these two tools, you'll find a solution.

[Jasonjac2 0]

@corneliusdavid potentially collaborate on this?  FIBPLus is  a bit more complicated than the BDE IMO as there are some advanced properties and also I think the developer had less defaults or whatever it is that ensures the DFM doesn't bloat with a gazilion properties that are just default values.  How far are you through?

[corneliusdavid 214]

2 hours ago, Jasonjac2 said:

How far are you through?

Are you asking about my progress with reFind? I've got a pretty long list of regular expressions but it's heavily concentrated on the specific libraries in use at the company for which I work. We have base forms and much of the script deals with commenting out code or adding TODOs to instruct the programmer on how to change a program to work with the updated libraries and base forms. But some of them might be useful as examples, like the conversion from TPageControl to TRzPageControl or TToolbar to TRzToolbar.

 

Your Delphi installation comes with ReFind and several example rule files you can use as a starting point; there's one to migrate from BDE to FireDAC which would be quite relevant to your task. I have never used FIBPlus so I wouldn't be very helpful on that end.

[Zentori 0]

@Jasonjac2, I'm trying to estimate a migration like yours, from 7 to 12.x. It involves data access only, so it should be easier than @corneliusdavid task, but also more "reusable". Part of data access is FIBplus and the other IBObjects. Collaboration options are very welcome!

[Jasonjac2 0]

I gave Marco from GDK a heads up about this thread as I think I said GDK are in the market of upgrading systems.

 

I am on another large tight deadline project for the same client as the large system (I also need to move from a very of version of Firebird and move from Dialect 1 -> 3).  My hope would be able to do this over a period of time, set up a test database, then try and migrate parts of the system that I can then compile (e.g. just include Datamodule units so I am not worrying about UI to begin with).  Our system has about 700units (50 /50 data modules / forms and frames).  Ultimately, once I get a compile and testing, I am likely to look at formalising a few of the things I have learnt over the years, so I don't carry forward dozens of ways of doing the same task.

 

If you want I am really happy to have a chat about ways of doing this and sharing ideas / scripts.

What does your system look like (size, DFM's or coded creation of components, standard CRUD mechanisms etc). 

 

Working with someone else is much more likely that I will put the time in, rather than just continue with D7 forever or until something really breaks. 

 

It is a problem when the change the customer will see if very small once initially migrated, but the effort is VERY high and risks are introduced that need a lot of testing.... and they say "why do we need to do this now".

[Frickler 11]

20 hours ago, Jasonjac2 said:

FIBPLus is  a bit more complicated than the BDE 

As you always need to acquire the newest Delphi version  in order to get bugs fixed on FireDAC, why not try a third party DAC?

DevArt IBDAC can convert from BDE, IBX and FibPlus. Never needed to convert, so I cannot say how well this works though. But you can download a trial version and try it for yourself.

[Jasonjac2 0]

3 minutes ago, Frickler said:

As you always need to acquire the newest Delphi version  in order to get bugs fixed on FireDAC, why not try a third party DAC?

DevArt IBDAC can convert from BDE, IBX and FibPlus. Never needed to convert, so I cannot say how well this works though. But you can download a trial version and try it for yourself.

I hadn't looked at this, It would be very interesting.

[Jasonjac2 0]

18 hours ago, corneliusdavid said:

Are you asking about my progress with reFind? I've got a pretty long list of regular expressions but it's heavily concentrated on the specific libraries in use at the company for which I work. We have base forms and much of the script deals with commenting out code or adding TODOs to instruct the programmer on how to change a program to work with the updated libraries and base forms. But some of them might be useful as examples, like the conversion from TPageControl to TRzPageControl or TToolbar to TRzToolbar.

 

Your Delphi installation comes with ReFind and several example rule files you can use as a starting point; there's one to migrate from BDE to FireDAC which would be quite relevant to your task. I have never used FIBPlus so I wouldn't be very helpful on that end.

Interesting about the todo's - I wrote a script to add this to every file in the project many years ago when I made an architectural change.

I was speaking generally - i.e. gotchas, considerations, progress etc.

TIA - JAC. 

 

 

 

[corneliusdavid 214]

8 hours ago, Jasonjac2 said:

I was speaking generally - i.e. gotchas, considerations, progress etc.

Some tips using refind for migration:

• Format your code first (using "formatter" command-line utility).
• The "uses" clause doesn't seem to be updated with either #unuse or #migrate (to add units) if it's indented.
• Look through the BDE-to-FireDAC scripts to learn techniques.
• Learn regular expressions really well. 
• Expressions only work on a single line at a time, so code spread over multiple lines may be left broken or may not match like you think they will.
• Create separate rules lists for the DPR vs PAS files.
• Create a script you can run over and over as refining this is an iterative process.
• Consider using Thomas Mueller's PrepBuild utility for managing project version info.

Those are what I can think of. I hope they help.

[Jasonjac2 0]

19 hours ago, corneliusdavid said:

Some tips using refind for migration:

•  

Exactly what I was after.  I m not working on it at the moment.  If anyone wanted to work on it at the same time with a similar project, I would be up for that, based on it being more likely to get done, but these hints are great and I'll put them on the list when I warm up the project.