FYI: Graph showing iOS crashes with recent Delphi versions

[Hans♫ 75]

Our iOS app made with Delphi showed an alarming increase of crashes with the latest update, that was built with Delphi 10.3. So I made this graph in "App Store Connect" that shows a similar increase of crashes when upgrading to Delphi 10.2.2. This is a bit scary... where will it end...

 

Graph shows crashes per week:

The vertical gray lines shows when we released a new update of our app.

The number of sessions per week is almost the same during the whole period, so that does not influence the graph.

Unfortunately I don't remember what version of Delphi I was using before the 10.2.2 upgrade.

[Dalija Prasnikar 1396]

Do you have any information about iOS versions used?

 

At some point (around version 11) iOS became more sensitive to touching UI from background threads. I had small test apps built with Tokyo and Rio that were happily running on iOS 9.x and were instantly killed on iOS 11.x.

[Hans♫ 75]

Yes, I have the graph by OS also, but in both situations where the the graph stepped up, it happened with the same OS. So that is not the reason.

 

[ByteJuggler 45]

This is interesting stuff.  What produces the above stats? 

 

(As background: Back in when XE5 was released we did some evaluation and based on the admittedly somewhat limited testing we did, concluded that mobile support was [at that time] not good enough to risk a project on.  I/we have generally however had the impression in recent releases (since let's just say 10.x) that support was plausibly nowadays "good enough" based on generally favourable anecdotal reports by people with production apps in the Google and Apple app stores.  We have therefore been considering revisiting and doing another trial and possibly proceeding with a mobile app development soon.  However the above puts a slightly more cautionary spin on things.  So your report is useful, thanks, and any further comments are highly appreciated.)

 

 

Edited February 7, 2019 by ByteJuggler

[Hans♫ 75]

Its just a screen dump from App Analytics in App Store. I did not make the graph, Apple did:

 

 

The graph is not a scientific proof that those Delphi versions are the cause of the leaps, but it is interesting that both leaps happened exactly at the same time as upgrading the Delphi version. 

 

Except for 10.3 Rio, I am still very pleased with Delphi as it allows us to share 99% of the codebase among the Windows, OSX and IOS versions of our app.

BTW, on iOS we are using the native iCL components and not FMX.

[Sherlock 663]

Well, as long as you didn't change your code, it's a strong hint towards deteriorating compiler quality (it could also be that iOS is getting worse, there seem to be some indicators towards that). Otherwise...well 😉

[Hans♫ 75]

Ha, ha, we only release an update when there is nothing new, nothing fixed and absolutely no change in the code 😉 

 

The graph is no proof, but its an indication.

[Sherlock 663]

Yeah, my comment was rather tongue in cheek. I do sense an overall trend there 😉

[Dalija Prasnikar 1396]

1 hour ago, Sherlock said:

Well, as long as you didn't change your code, it's a strong hint towards deteriorating compiler quality (it could also be that iOS is getting worse, there seem to be some indicators towards that). Otherwise...well 😉

Not necessarily compiler, but RTL/FMX or other frameworks.

 

Yes, there were some changes in compiler that may have influence, but such errors tend to have impact across all platforms.

 

Now, it would be interesting to figure out what exactly is the problem.

[Der schöne Günther 316]

Are there more information available about the crashes themselves? 

 

And didn't Delphi get famous by demonstrating how an unhandled access violation would just produce a messagebox and continued to work afterwards instead of bringing the whole app down?

[Sherlock 663]

Just now, Der schöne Günther said:

And didn't Delphi get famous by demonstrating how an unhandled access violation would just produce a messagebox and continued to work afterwards instead of bringing the whole app down?

When was the last time you actually witnessed that?

[Dalija Prasnikar 1396]

7 minutes ago, Der schöne Günther said:

Are there more information available about the crashes themselves? 

 

And didn't Delphi get famous by demonstrating how an unhandled access violation would just produce a messagebox and continued to work afterwards instead of bringing the whole app down?

On Windows, yes. On other platforms not so much. Android and especially iOS are just sitting and waiting for opportunity to nuke your app.

[Der schöne Günther 316]

1 hour ago, Sherlock said:

When was the last time you actually witnessed that?

It was some article that referred to a presentation from the early 90s, that's for sure. We once tried making a small application for iOS & Android, and yes, the application had so many fatal crashes with no way of debugging it that we finally gave up.

[Hans♫ 75]

I investigated all recent crash reports delivered to XCode for the app. The majority of the crashes contains a reference to a sqlite file in our app and "Code 0xdead10cc", which means "terminated by the OS because it held on to a file lock or sqlite database lock during suspension"

 

The positive news is that the user will not discover the crash because it happens when it enters suspension state anyway.

 

[Dalija Prasnikar 1396]

6 hours ago, Hans♫ said:

I investigated all recent crash reports delivered to XCode for the app. The majority of the crashes contains a reference to a sqlite file in our app and "Code 0xdead10cc", which means "terminated by the OS because it held on to a file lock or sqlite database lock during suspension"

 

The positive news is that the user will not discover the crash because it happens when it enters suspension state anyway.

 

Quick search through Quality Portal didn't reveal any reported issue similar to yours. Can you please file QP report. 

[Hans♫ 75]

Thank you for your persistence Dalija. I have a heart for Delphi too, and I somewhat regret that I posted the image, because I still have no proof if it is our code or Delphi code that fails.
With the graph I am basically bashing 10.3 RIO because I am tired of its quality.

 

Our database access uses exactly the same code on Windows, Mac and iOS. No crashes detected or reported on Windows and Mac.
But as you said, iOS is just waiting for a reason to take your app out. It is a lot more strict on everything.
 

[Dalija Prasnikar 1396]

1 hour ago, Hans♫ said:

Thank you for your persistence Dalija. I have a heart for Delphi too, and I somewhat regret that I posted the image, because I still have no proof if it is our code or Delphi code that fails.
With the graph I am basically bashing 10.3 RIO because I am tired of its quality.

 

Our database access uses exactly the same code on Windows, Mac and iOS. No crashes detected or reported on Windows and Mac.
But as you said, iOS is just waiting for a reason to take your app out. It is a lot more strict on everything.
 

I understand how you feel. Sometimes it is going two steps forward three steps backward.

 

One of the problems with cross-platform development is that not all code is the same. And different principles apply. OS architectures are different and what you can do on one platform, you cannot do on another. I am talking generally, regardless of whether offending code is yours or Embarcadero's.

Also mobile platforms change frequently and are also changing rules and behaviors. What worked today, may break tomorrow.

 

This is not related just to Delphi applications but all. I am using Xcode - Swift and Android Studio - Java for mobile development and situation is not much different. The only difference, is that when some policy or behavior changes, you immediately have appropriate tools to implement change. But your code will be broken left and right. You cannot take year old code and make simple recompile - in Swift everything is in a state of flux - language, frameworks, so every time you touch it, it is almost a rewrite. I don't even know what is the proper method for sorting list - its name changed dozen times - from sort to sorted to sort again... what is current one I have no idea - and that is just the simplest example. On Android, Java is the same, but frameworks are changing every week. If you have to support newer versions of OS, you have to use newer frameworks, that also change all the time, are buggy as hell... 

 

I know Delphi could and should be better. But, I also painfully know that grass is not greener on the other side.  What we can do, is relentlessly report issues to Embarcadero. Not all will be fixed and not all will be fixed as fast as we would like, but Embarcadero is fixing them. Especially, more critical ones.

 

[Rollo62 536]

5 hours ago, Dalija Prasnikar said:

You cannot take year old code and make simple recompile

 

@Dalija Prasnikar Good to hear that with Fmx we're not the only hopeless bunch of people out there, and all others face similar issues.
But your' reight, XCode e.g. fixes most issues by itself, and I wigh Delphi could have that state of smartness too.

 

Regarding above "old code" line, thats true.

IMHO the only reliable way to keep old code running is to separate code parts into many quite small, well testable fractions.
Much more code-modularization is needed than it would be necessary for traditional Windows development.

Its good coding practice anyway, so we have to have even more discipline maybe.

 

[ByteJuggler 45]

Relatedly and somewhat in the same vein I read an article yesterday about AirBnB and why they're now moving off of React native, basically too much churn and too many landmines, even if it works really well when it works.  

[Marcio 2]

 

On 2/7/2019 at 12:38 PM, Hans♫ said:

I investigated all recent crash reports delivered to XCode for the app. The majority of the crashes contains a reference to a sqlite file in our app and "Code 0xdead10cc", which means "terminated by the OS because it held on to a file lock or sqlite database lock during suspension"

 

The positive news is that the user will not discover the crash because it happens when it enters suspension state anyway.

 

 

 

In this crash you reported on blocking sqlite, you tried to disconnect from the base when the application enters IDLE mode, and perform the connection again, because as the base is leased exclusively, the system probably has to maintain resources for this, it might be a way to save battery power, I'm not an IOS expert, but I'm just interpreting the message.

 

Marcio

[Hans♫ 75]

Good news about iOS crashes!

I owe Embarcadero and all of you to show the updated crash report graph. The problem was an internal hard limit in librtlhelper.a on the number of OC wrappers that could be created. As my app uses the TMC iCL native components, and is a large and comprehensive app, it wraps thousands of OC objects. Each new Delphi version used more and more OC wrappers on its own, leaving less for use in the app. That's seen clearly on the previous graph I showed.

 

Marco sent an updated version of librtlhelper.a to me with a higher internal limit, and the result is seen clearly on the graph below. There are still some crashes though, which might indicate that a some of our users still reach the limit, <irony>since I can't imagine that there can be other errors in our software</irony>.

 

Edited October 25, 2019 by Hans♫

[ENord 0]

On 10/25/2019 at 11:00 AM, Hans♫ said:

Marco sent an updated version of librtlhelper.a to me with a higher internal limit, and the result is seen clearly on the graph below. There are still some crashes though, which might indicate that a some of our users still reach the limit, <irony>since I can't imagine that there can be other errors in our software</irony>.

Hi, could you send me this file as well? 

I also have a quite comprehensive application which uses TMS iCL components, and crashes are plenty (do not have any graphs available as it is distributed in-house). This is very interesting and worth a try for me. 

[Hans♫ 75]

5 minutes ago, ENord said:

Hi, could you send me this file as well? 

No problem. Send me an email, then I will send it to you: hans[at]earmaster.com

Note that the file is built for Delphi 10.3.2 so it might not work with other versions.

[ENord 0]

2 minutes ago, Hans♫ said:

No problem. Send me an email, then I will send it to you: hans[at]earmaster.com

Note that the file is built for Delphi 10.3.2 so it might not work with other versions.

Thanks, sent you a private message. 

[vfbb 285]

@Hans♫ in the iTunes you can download each user crash report file and view the call stack to identify the problem. You will need to symbolize the crash file using the dSYM file of the version of your app thats crashed.

Read about: