RawStackTraces in FastMM4 (4.992)

[Clément 148]

Hi,

 

I'm using D10.4.1.

Recently I switch to FastMM4 from github ( 4.992 ) to track down some (unexpected) leaks. With the embedded version of FastMM4 those leaks were not reported.

I manage to find all of them.. But when I close the application the below exception is raised..
I thought it was memory leak related , but even after removing all leaks, it continues to "kind of" show.

 

I google that address to see if has a meaningful magical number but I found nothing. No matter for how long I use the application, or what screens are opened/closed. Always the same address.
If I close the main form without opening any other forms, the program exits normally. if I either open one form, or all the application's forms , the below exception is raised:

 

 

---------------------------
Debugger Exception Notification
---------------------------
Project Project007.exe raised exception class $C0000005 with message 'access violation at 0x02099099: read of address 0x74be973f'.
---------------------------
Break   Continue   Help   
---------------------------

 

If I press the "Break button" then this is shown:

 

 

The call stack:

 

The call stack first line points to "FastMM_FullDebugMode.dll" which is in the same path as the executable. I'm using the precompiled version 32 bits.

The "Build configuration" is set to DEBUG.
If I run "With Debug" (F9) the error is raised.
If I run "Without Debug" (CTRL+SHFT + F9)  no error is raised.
There's no memory leak no matter what version of FastMM4 runs.

FastMM4 is set to report as follows:

 

If I uncheck RawStackTraces the exceptions no longer is raised.
Can I gracefully ignore "RawStackTraces" and leave it unchecked, or there's a bomb in my code?!

 

TIA

 

Edited February 3, 2021 by Clément

[Guest]

On 2/3/2021 at 9:34 PM, Clément said:

Can I gracefully ignore "RawStackTraces" and leave it unchecked, or there's a bomb in my code?!

To answer that we need to analyze your information first and get better understanding what is going on, only after that we can see the danger of this, if there is any !

 

On 2/3/2021 at 9:34 PM, Clément said:

I google that address to see if has a meaningful magical number but I found nothing. No matter for how long I use the application, or what screens are opened/closed. Always the same address.

Finding a bug by an address it rarely does, so let see these address to see what we can learn form them.

0x02099099 from the stack screenshot it does belong to FastMM debug dll, so it is not important to us here, the only thing we read from it, that your EXE is smaller than 30MB!

0x74be973f now this is very interesting as it is very high for 32bit and very close to the OS dll's default location (very high), and thing of it is the same every time, i think it is just luck, it can be changed after a restart, but to be accurate and for getting a confirmation for the following i will write, it might be due the FastMM configuration to allocate from top-to-down, so here if you tried to disable AlwaysAllocateTopDown , will this address start to become random on every run ?!

 

On 2/3/2021 at 9:34 PM, Clément said:

Now lets see the stack calls, Halt been called that called a finalization on some unit, the lack of the name of the unit is interesting and will require from you some effort to track, 

Anyway, the sequence of the calls showing that

1) a generic list that hold an array of record, this record does have an array of string, 

2) The location of both array's are still there, while the location of the array of strings was gone (???) at calling UStrArrayClr.

3) The record type info also gone at FinalizeRecord (..., ???)

4) Both TypeInfo of the array's are still there, judging by the address of the second one $4012BC, it is RTL supplied one, while the the first at location $692BB8 is yours.

 

From the above the exception cause is double freeing but due to its strange case on termination to happen, FastMM4 failed to catch it, but due to my lack experience with managed record (i don't have latest Delphi versions), here some guessing by me and i might be wrong, how this can happen as FastMM in full debug should be able to catch it ? also there is a very small chance that this might be a bug in managed records, as i have no idea how global generic list or/with managed records does clean up.

 

This might be happening because FastMM keep record of the free/alloc calls in separated block in parallel to its internal blocks allocations, and while the application is terminating many blocks is been freed as long as there is no indication to FastMM to keep, so in theory if the second array items (array of strings) been freed while the record still refer to an array where the locations of these record and array ($7F881358 and $7F88135C with only 4 byte away ), these address's belong to a block that is not freed yet.

 

To fix this you need first to identify the culprit array of record ( or the generic list of record), then track the items of its grandchild strings in the array, where/how are these strings been freed right without updating its array.

 

Now to answer if this is a bomb or not ? 

Yes it is, unless you are sure 100% that such free will only happen once on application termination, there will be a chance for unpredicted behaviour and exceptions that might be a show stopper.

 

Hope that helps.

[Clément 148]

On 2/7/2021 at 7:43 AM, Kas Ob. said:

To answer that we need to analyze your information first and get better understanding what is going on, only after that we can see the danger of this, if there is any !

 

Finding a bug by an address it rarely does, so let see these address to see what we can learn form them.

0x02099099 from the stack screenshot it does belong to FastMM debug dll, so it is not important to us here, the only thing we read from it, that your EXE is smaller than 30MB!

0x74be973f now this is very interesting as it is very high for 32bit and very close to the OS dll's default location (very high), and thing of it is the same every time, i think it is just luck, it can be changed after a restart, but to be accurate and for getting a confirmation for the following i will write, it might be due the FastMM configuration to allocate from top-to-down, so here if you tried to disable AlwaysAllocateTopDown , will this address start to become random on every run ?!

First of all let me thank you very much for such great explanation!

Once I unselected "AlwaysAllocateTopDown" the address changes every time, but it still a high address
access violation at 0x02089099: read of address 0x755f973f'.
access violation at 0x02099099: read of address 0x755f973f'.
access violation at 0x020d9099: read of address 0x755f973f'.

 

On 2/7/2021 at 7:43 AM, Kas Ob. said:

Now lets see the stack calls, Halt been called that called a finalization on some unit, the lack of the name of the unit is interesting and will require from you some effort to track, 

Anyway, the sequence of the calls showing that

1) a generic list that hold an array of record, this record does have an array of string, 

2) The location of both array's are still there, while the location of the array of strings was gone (???) at calling UStrArrayClr.

3) The record type info also gone at FinalizeRecord (..., ???)

4) Both TypeInfo of the array's are still there, judging by the address of the second one $4012BC, it is RTL supplied one, while the the first at location $692BB8 is yours.

 

From the above the exception cause is double freeing but due to its strange case on termination to happen, FastMM4 failed to catch it, but due to my lack experience with managed record (i don't have latest Delphi versions), here some guessing by me and i might be wrong, how this can happen as FastMM in full debug should be able to catch it ? also there is a very small chance that this might be a bug in managed records, as i have no idea how global generic list or/with managed records does clean up.

 

This might be happening because FastMM keep record of the free/alloc calls in separated block in parallel to its internal blocks allocations, and while the application is terminating many blocks is been freed as long as there is no indication to FastMM to keep, so in theory if the second array items (array of strings) been freed while the record still refer to an array where the locations of these record and array ($7F881358 and $7F88135C with only 4 byte away ), these address's belong to a block that is not freed yet.

 

To fix this you need first to identify the culprit array of record ( or the generic list of record), then track the items of its grandchild strings in the array, where/how are these strings been freed right without updating its array.

 

So far I manage to pass by all finalization sections. The RTL code that executes the finalizations is a while loop with a decreasing counter and in my case the count start at 732 and the error is raise when that counter reaches 196. At that point all my finalizations passed ( they ended by 650). When entering the correct point (entry 196) I can just follow up a few "Tobject.Free" methods and the exception is raised. No qualified named to help me locate the code. I'm using "debug dcus" and almost all my debug settings are on.

On 2/7/2021 at 7:43 AM, Kas Ob. said:

Now to answer if this is a bomb or not ? 

Yes it is, unless you are sure 100% that such free will only happen once on application termination, there will be a chance for unpredicted behaviour and exceptions that might be a show stopper.

 

Hope that helps.

That's what I was afraid of. And of course that bomb will only explode at customer's

[Guest]

10 hours ago, Clément said:

So far I manage to pass by all finalization sections. The RTL code that executes the finalizations is a while loop with a decreasing counter and in my case the count start at 732 and the error is raise when that counter reaches 196. At that point all my finalizations passed ( they ended by 650). When entering the correct point (entry 196) I can just follow up a few "Tobject.Free" methods and the exception is raised.

The right way to investigate, only you missed the shorter and more accurate way to do it.

10 hours ago, Clément said:

No qualified named to help me locate the code.

For exactly that you should focus on the the TypeInfo, which i mentioned above, re-reading what i wrote, i see i failed to point you to follow on that.

 

So what you need now is either conditional break points on FinalizeArray when the TypeInfo is $692BB8 (or other any other address been passed, you can see it on the stack dump), 

On 2/7/2021 at 12:43 PM, Kas Ob. said:

while the the first at location $692BB8 is yours.

also if break points is not your cup of tea then use DDetours to capture the TypeInfo in the question or all of them if you want, dump the calls into a file later check for the address and put an if with break point as that address should change only if you changed the source but replacing it will not change the source and the address's will still the same, not only on FInalizeArray but other RTL functions like New and InitializeArray .. you might need to go through few of them to capture the parent array, which i still can't say if it is generic array or generic list 

 

One more thing, have you tried EurekaLog ?

In case you want to give it a go, i would like to hear what report it does generate, the trial version i think should be enough for this case.

EurekaLog does the memory allocation and tracking differently from FastMM, so it might still have the allocation calls for that array when the free call in question been executed.

[Clément 148]

I managed to get the typeinfo. The addess in my code has changed, but all the others are the same. The $692BB8 became $6AC4D8.
This is what I came up with:

 

Is this correct?

 

I installed the trial version of EurekaLog, activated it, set all the trace raw stack options and nothing happens. No report is produced or dialog displayed. If I force an exception in my application it does kick in. Even the exception raised by FastMM4's RawStackTrace  option is no longer raised. Since there's a lot of options to set up, I will have to play a little more with it.  As soon as I deactivate EL, the exception raises back again iif FastMM4's RawStacktrace is set.

[Guest]

36 minutes ago, Clément said:

Is this correct?

It should be.

 

I have no Idea what is TdxAlphaColors , or why it does lead to to an array of strings, but may be colors names are initialized, any way it is irrelevant for this stage.

 

Now you have the type of the array, now you need to pinpoint the variant that hold that array, which is a list in a generic list or an array, so searching where is that type been used and tracking it is your next step.

 

Searching the internet for that type result in DevExpress related pages, so you might need some insight from their support on what generics types might have such double freeing or trimming, assuming you are not using them directly in some sort of singleton or global structure/var, if this is caused by DevExpress library then you are better with their support, or if it is related to theming then you are safe to ignore this error completely as it will always happen on exiting, but again you need their support confirmation.

[Clément 148]

Anyway thanks for your help. I will do some more tracking to be sure I'm not messing TdxAlphaColors in any way. But I'm using their skins and all of their components. So it might well be some acceptable exception. I will check with their support.

 

Thanks a lot for your help!