Bad build a mystery

[Bill Meyer 337]

I am working on a large legacy project, and recently encountered an unexpected failure. We added a new form, did a full build, and ran the program in debug. It showed some activity, but returned to the IDE, as though the app had shut down, and it never reached the splash screen.

Running the executable outside the debugger, it reports an AV, then when you close, it reports runtime error 217. If EurekaLog is enabled, no elf file is produced. Some searching online shows that the error in question occurs before exception handling is set up.

Removing the newly added form, the app builds and runs as normal.

Adding a new unit, with no code, also provokes the total failure. 

 

I am at a loss to know how to discover the real problem. Any thoughts would be welcome.

[Uwe Raabe 2057]

Could it be that the new form was auto-created in the dpr when it should not?

[Bill Meyer 337]

24 minutes ago, Uwe Raabe said:

Could it be that the new form was auto-created in the dpr when it should not?

No, that was checked. But also, adding a new unit, as in File, New, Unit, and putting in it no code, also provokes the problem.

[Darian Miller 361]

Just spit balling here - but add a new unit, save the project and do a file compare of the DPR + DPROJ files... anything odd? 

 

[Bill Meyer 337]

3 minutes ago, Darian Miller said:

Just spit balling here - but add a new unit, save the project and do a file compare of the DPR + DPROJ files... anything odd? 

 

No. And also have done a comparison of the map files of good and bad. Nothing odd there, either.

[Bill Meyer 337]

We do have over 2,000 forms and units in the app.

[Darian Miller 361]

I experienced a similar problem a while ago.  From what I recall, it was a bad binary DFM which had to be isolated/corrected.  For some reason, that experience came back to me after seeing your response.  (Too many bad memories that have been suppressed!)  

 

I believe it turned out to be an ancient form (or two) that we hadn't changed in a very long time.  Unfortunately I don't remember exactly how I found it among the many hundreds of forms in the project.  I do remember searching for all binary DFMs and I believe I used the tool to auto-convert all the binary DFMs to text and it blew chunks on a bad DFM.  It was either that or I manually opened all binary DFMs until I ran into a problem opening the form within the IDE.  (It compiled fine and didn't cause errors in the app until some magic moment in time.)

 

It was a few hours of head scratching for sure...

 

Good luck!

 

[Bill Meyer 337]

13 minutes ago, Darian Miller said:

I experienced a similar problem a while ago.  From what I recall, it was a bad binary DFM which had to be isolated/corrected.  For some reason, that experience came back to me after seeing your response.  (Too many bad memories that have been suppressed!)  

A worthy notion, but a relatively short time ago I wrote my own tool to find all of the (52, at the time) binary DFMs and convert them. So no binaries present in the mix.

[Attila Kovacs 629]

the last time I got runtime error 217 on startup was an winapi call in the initialization where I queried the computer name into a (15-16?) byte lenght buffer but the computer name was longer. I can't remember the problem/solution, it was on g+.

 

btw. you could step through with ollydebug, maybe you find something

Edited July 19, 2019 by Attila Kovacs

[PeterBelow 238]

18 hours ago, Bill Meyer said:

No. And also have done a comparison of the map files of good and bad. Nothing odd there, either.

Did you check the dpr file for bad line ends (only cr or only lf)? What happend to me some time ago was that the IDE inserted a new form unit name in the DPR file smack in the middle of an existing one, which of course did not build anymore and resulted in some weird error messages (no IDE crash, though). I found out that the file had become corrupted, with  mix of linebreak styles. I never found out how that happended...

[Darian Miller 361]

22 hours ago, Bill Meyer said:

A worthy notion, but a relatively short time ago I wrote my own tool to find all of the (52, at the time) binary DFMs and convert them. So no binaries present in the mix.

 

The only thing left for me to suggest is to step through each and every initialization section until you find one that blows chunks.  I'd do a debug build (without system DCUs first) and then step through (F7) until you hit the main form display or you get an exception.

 

I love a good mystery!   Sorry, I know it's much more fun on this end...

[Bill Meyer 337]

8 hours ago, PeterBelow said:

Did you check the dpr file for bad line ends (only cr or only lf)? What happend to me some time ago was that the IDE inserted a new form unit name in the DPR file smack in the middle of an existing one, which of course did not build anymore and resulted in some weird error messages (no IDE crash, though). I found out that the file had become corrupted, with  mix of linebreak styles. I never found out how that happended...

Another good thought, but I re-saved with Notepad++ set for CR/LF endings. No change.

[Bill Meyer 337]

3 hours ago, Darian Miller said:

 

The only thing left for me to suggest is to step through each and every initialization section until you find one that blows chunks.  I'd do a debug build (without system DCUs first) and then step through (F7) until you hit the main form display or you get an exception.

 

I love a good mystery!   Sorry, I know it's much more fun on this end...

Before adding a module, the units all execute initialization with no problem. After adding a unit, execution does not reach that point. And the debugger is useless.

[Darian Miller 361]

4 hours ago, Bill Meyer said:

Before adding a module, the units all execute initialization with no problem. After adding a unit, execution does not reach that point. And the debugger is useless.

 

You could perhaps try adding more new forms/modules until it runs.  You could also open a trouble ticket with Embarcadero, but they would likely want to see your entire project.  Most maintainers of huge projects shudder at the idea of handing over the entire thing, and there's no way for Embarcadero to repeat it otherwise.

 

I'm sure you've done a full project folder clean already.  It sure seems like a resource issue - add some extra items to the project .res or re-verify all custom resource items?

 

Paging @Marco Cantu - any ideas?

 

 

[David Heffernan 2345]

Is it possible that you have reached the upper size limit for resources? Try removing some other forms and see if the project builds. That would strengthen this hypothesis. 

 

I can't begin to understand how you end up with 2000 forms in a single executable. 

[miab 25]

And how do you delete one more and add two?

[Bill Meyer 337]

6 hours ago, David Heffernan said:

Is it possible that you have reached the upper size limit for resources? Try removing some other forms and see if the project builds. That would strengthen this hypothesis. 

 

I can't begin to understand how you end up with 2000 forms in a single executable. 

Interesting thought. How does one assess the size of resource in an entire project?

As to the how: Years of development, a complex set of requirements, and certainly a failure to consolidate where possible.

[David Heffernan 2345]

2 hours ago, Bill Meyer said:

Interesting thought. How does one assess the size of resource in an entire project?

As to the how: Years of development, a complex set of requirements, and certainly a failure to consolidate where possible.

I've no idea, never needed to try. Not even sure that there is a limit, or what it is. But if you take out some other forms and the build succeeds then that is very strong evidence. 

 

How do you even begin to test 2000 forms? I just can't get me head around this at all. 

[John Kouraklis 94]

How about if you locate a simple form in your project, add a new form, copy all the components from the old form to the new one and remove the old form. Maybe you will manage to trick the compiler.

 

Another thought...do you change the name of the form? maybe some kind of name conflict? like a form and a frame? 

 

Also, I guess it is impossible to create a new project and drag/drop/add all the forms/units?

Edited July 21, 2019 by John Kouraklis

[Bill Meyer 337]

19 hours ago, David Heffernan said:

I've no idea, never needed to try. Not even sure that there is a limit, or what it is. But if you take out some other forms and the build succeeds then that is very strong evidence. 

 

How do you even begin to test 2000 forms? I just can't get me head around this at all. 

OK, I thought you might know,as you raised the idea. I haven't worried about resources in a very long time, and will have to do some research, as well, to learn whether there is a limit. Taking out "some forms" isn't really an answer, for a number of reasons, chiefly that there is too much coupling to make this a trivial activity. Adding in a unit -- not even a form -- is enough to trigger the problem.

 

Testing is, as you would guess, problematic. And all the worse for there being too much code on forms, which is another characteristic I simply identify now as Delphi legacy style.

Edited July 22, 2019 by Bill Meyer

[David Heffernan 2345]

I wasn't suggesting that removing forms was the ultimate solution. I'm giving you debugging tips. At some point, you'll have to do some debugging. And no, I don't mean debugging with the IDE debugger. 

[Bill Meyer 337]

1 hour ago, David Heffernan said:

I wasn't suggesting that removing forms was the ultimate solution. I'm giving you debugging tips. At some point, you'll have to do some debugging. And no, I don't mean debugging with the IDE debugger. 

I understand. I have been pursuing a number of possibilities, but ultimately, someone else will make the call about how we address the problem. The mass of modules and the tight coupling in the project obviously need to be corrected, but that is a long path, and the issue, as always in a commercial enterprise, will be decided by managers whose appreciation of the factors may be less complete than one might wish.

[Bill Meyer 337]

Update: I have now completed a massive set of minor changes, affecting over 1400 units. Removed all units from uses clauses which were not needed, and demoted from interface to implementation, where possible. The other change was to add a module to manage unit initialization, rather than leaving that to Delphi. All initialization and finalization clauses have been replaced by procedures. That should leave us less susceptible to sequence issues as we continue to resolve the coupling issues. In the process, also reduced by about 25% the unit dependency cycles identified by MMX. And in my VM on an SSD, the time for a first full build after loading the IDE dropped from nearly 4 minutes to about 40 seconds. Progress.

[Sherlock 663]

There is a german saying: Kaum macht man es richtig,funktioniert es. It roughly translates to: As soon as you do it right, it works. It is all too true for most everything 😄

 

[Bill Meyer 337]

6 hours ago, Sherlock said:

There is a german saying: Kaum macht man es richtig,funktioniert es. It roughly translates to: As soon as you do it right, it works. It is all too true for most everything 😄

When you are dealing with a large legacy project, however, untangling years of ... interesting ... coding takes time.