Mahdi Safsafi

Yes you're right ! Yes its an exception number but honestly I don't know for what its used for. The explanation you gave is very logic and I'd adopt it until its proven otherwise. I prefer this notation "if x = $0EEDFADE { cDelphiException } then" because it makes comparing to the assembly more easy.

Yep you're totally right ! Unlike software exception, hardware exception are not blessed to Delphi exception class. So the problem occurs because the expert is trying to read(dereference) class information for HE that does not have a class associated. Why it tries to do that ? ... Well that's my bad, it appeared that I reversed a branch when translating some assembly code to pascal . I really appreciate your effort ... you did great 🙂 ! Thanks ! This happens even if you don't use GExperts ... So I guess its coming from the IDE. For IsBadReadPtr, you can use VirtualQuery to check for read access. @dummzeuch Below there is a fix ! BTW Tomas, you should do a cleanup and remove all unused codes/features. I see that you're not using GetExceptionObject and yet its still being called and caused this issue. Same for sl.add(xxx). Another thing, I found that "Ignore All this Session" button does not scale properly when resizing the exception dialog. function GetExceptionObjectNew(Thread: TThread): TAddress; var I: Integer; Src: TThreadOsInfoArray; Parsed: TParsedThreadOsInfoArray; P: PByte; C: Cardinal; begin // this function should be used only with new delphi versions Result := 0; ZeroMemory(@Src, SizeOf(Src)); ZeroMemory(@Parsed, SizeOf(Parsed)); I := GetThreadOsInfo(Thread, Src); if I <> 0 then begin case I of 4, 6, 8, 7, 9, 10: Exit; // ==> end; ParseThreadOsInfo(Thread, Src, Parsed); // disasm TNativeThread::DoGetExceptionName P := @Parsed[0]; Inc(P, $A8); P := PPointer(P)^; C := PCardinal(Integer(P) + $18)^; { !!! don't optimize me !!! } if (C <> $0EEDFAE6) then begin if C = $0EEDFADE then begin Inc(P, $38); Result := PUInt64(P)^; end else if C <> $0EEDFAE4 then begin Exit; // ==> hex. end else begin Inc(P, $38); Result := PUInt64(P)^; end; end else begin C := PCardinal(Integer(P) + $34)^; if C <> 0 then begin Inc(P, $48); Result := PUInt64(P)^; end else begin C := PCardinal(Integer(P) + $30)^; if C <> 1 then begin Inc(P, $48); Result := PUInt64(P)^; end else begin Exit; end; end; end; end; end; {$ENDIF} function GetExceptionObjectLegacy(Thread: TThread): TAddress; begin Result := 0; // This function should only be used with old Delphi versions, where GetExceptionObjectNew does // not work, that is ParseThradOsInfo does not exist. FDebugEventCritSect.Enter; if FDebugEvent.Exception.ExceptionRecord.NumberParameters > 1 then begin // Param[1] = Exception object. // FDebugEvent.dwProcessId = process id. // FDebugEvent.dwThreadId = dwThreadId id. // FDebugEvent.Exception.ExceptionRecord.ExceptionAddress = exception address. // see TExceptionRecord for more info. if FDebugEvent.Exception.ExceptionRecord.ExceptionCode = $0EEDFADE { cDelphiException } then Result := FDebugEvent.Exception.ExceptionRecord.ExceptionInformation[1]; end; FDebugEventCritSect.Leave; end;

Typically both produce the same output. Stefan code is based on template. Mine is not. I found my code more readable and friendly for typing (you don't need to worry about <>). But its just a taste of manner.

There is another way without specifying explicitly the generic type param and it generates the same output. Internally slice requires dst(open array type), src(pointer type) and of course a count(integer type). It uses src to get the pointer and dst info to compute the offset meaning you can provide a fake declaration for the source: type TOpenArray = array[0 .. 0] of Byte; POpenArray = ^TOpenArray ; MergeSort(Slice(POpenArray(@values[mid])^, len - mid));

Very nice article Stefan and the way you used Slice to avoid the copy is amazing ! Speaking of bugs, I found two drawback related to using open array param: First, I already wrote on this forum a topic about unnamed type and rtti. Compiler do not generate rtti for an unnamed type ! this is easy to avoid with most types because all what you need to do is to declare the type : type x = xxx. But for open array type that's impossible because the syntax conflicts with dynamic array syntax ! So I suppose there is no official way to have a correct rtti for a function that uses open array param. Second, On my D10.3, the TArray<T> example caused an "IDE/Compiler not responding" issue that always ends up by IDE restart: I was able to isolate the line that causes the issue : MergeSort(Slice(TOpenArray<T>((@values[mid])^), len - mid)); // if I comment this line everything works great ! I tried with different build mode (debug,release) and on different project and the result is always the same

Delphi IDE uses System.Classes.TStream.WriteDescendentRes to serialize the form/components properties. Property that belongs to the class is handled by System.Classes.TWriter.WriteProperty. However internal property such ExplicitXxx is handled by System.Classes.TWriter.DefineProperty. // TPersistent.DefineProperties defines Explicit properties. and TWriter.DefineProperty encode them. procedure TControl.DefineProperties(Filer: TFiler); begin ... Filer.DefineProperty('IsControl', ReadIsControl, WriteIsControl, DoWriteIsControl); Filer.DefineProperty('ExplicitLeft', ReadExplicitLeft, WriteExplicitLeft, not (csReading in ComponentState) and DoWriteExplicit(edLeft)); Filer.DefineProperty('ExplicitTop', ReadExplicitTop, WriteExplicitTop, not (csReading in ComponentState) and DoWriteExplicit(edTop)); Filer.DefineProperty('ExplicitWidth', ReadExplicitWidth, WriteExplicitWidth, not (csReading in ComponentState) and DoWriteExplicit(edWidth)); Filer.DefineProperty('ExplicitHeight', ReadExplicitHeight, WriteExplicitHeight, not (csReading in ComponentState) and DoWriteExplicit(edHeight)); end; In a nutshell, hooking TWriter.DefineProperty will get you off EplicitXxx: procedure InterceptDefineProperty(Obj: TWriter; const Name: string; ReadData: TReaderProc; WriteData: TWriterProc; HasData: Boolean); begin // do nothing !!! or write a filter to allow certain ExplicitXxx. end; // install the hook : DefinePropertyPtr := GetProcAddress(GetModuleHandle('rtl260.bpl'), '@System@Classes@TWriter@DefineProperty$qqrx20System@UnicodeStringynpqqrp22System@Classes@TReader$vynpqqrp22System@Classes@TWriter$vo'); @TrampolineDefineProperty := InterceptCreate(DefinePropertyPtr, @InterceptDefineProperty);

For the same reason why anybody should think that your comment is constructive ! What does it add to OP ? I'm going to ask you gently, please stop trolling. We just ended up a hot argument and I said I'm sorry ... Please don't let me regret.

My guess is that the library you're using for complexes is not well optimized as it tries to emulate complexes ! Some compilers support native complex operation such CLANG/LLVM. The wonderful Julia has a native support too. Its definitely worth checking other compilers as @Kas Ob. suggested or checking other libraries.

@Arnaud Bouchez @David Heffernan I'm really sorry guys. I didn't meant to be offensive by any way. I just got some external things that subjectively influenced the way I express. @Daniel I just discarded my last post.

Mostly forgot You're wrong ! its not the example that is brilliant. its people that understand it. You're the brilliant.

@Arnaud Bouchez @David Heffernan EDIT: We reached a far point and I believe that we're not going to agree. So lets make an end to this argument.

I really don't know what you mean. https://raima.com/memory-management-allocation/

The paging is not the only player. The guys were running on a large RAM(128GB) and suffered from a potential degradation of performance : see Thrashing

Its not like you don't need to ... the fact there is none !

Yeah ! The real problem is not how to allocate the memory. But how to free it. In other word how efficiently someone will make difference about a pointer that belongs to a small block and a pointer from a large block without de-referencing it. I believe that understanding the drawbacks is really required. i.e: Someone can benefit form it by freeing a collection items in the reverse order. Just one question Stefan : Why not let the user specify the nature of the collection (small, large) ?

Far now, we only have your words ! Mention a study, benchmark or a formal statement that claims your words. With all what was pointed out about the difference between GetMemory/FreeMemory and VirtualAlloc/VirtualFree and you still think that GetMemory/FreeMemory is suitable for large data ... WOW !

Yes Stefan but that does not change the fact that FreeMemory can be a bottleneck. The example I showed demonstrates paging. While its the most one that may have a several impact, there are other players too (cache thrashing, tlb thrashing, ...).

You and Arnaud have read the article but you both didn't understand it clearly perhaps because it requires being familiar with some details. That's why I told Arnaud that he is wrong again and its not related to the malloc ! Even if someone replaces the malloc with FastMM, he will likely going to have the same result. I'll give a very simple example(I will do my best to make it understood by anyone) to explain why a VirtualFree is much better than using FreeMemory. Suppose you have allocated a bunch of large data using GetMemory. Obviously soon or later, a system paging will work and starts to swap pages from memory to disk. It happens that some of your pages(more likely the first allocated one) end-up on disk instead of memory. When time comes to a cleanup, you will call FreeMemory to free the allocated memory : // FreeMemory for large block function FreeLargeBlock(APointer: Pointer): Integer; var LPreviousLargeBlockHeader, LNextLargeBlockHeader: PLargeBlockHeader; begin {Point to the start of the large block} APointer := Pointer(PByte(APointer) - LargeBlockHeaderSize); {Get the previous and next large blocks} LPreviousLargeBlockHeader := PLargeBlockHeader(APointer).PreviousLargeBlockHeader; ... end; 1 - As you can see the function tries to de-reference the pointer. 2 - Because the page was on disk and not on memory, an interrupt (for simplification, think it just like an invisible exception) occurs at CPU level. CPU then suspends the process that tried to access the memory and sends an interrupt to the OS kernel : Hi kernel, a process "A" is trying to access an invalid memory at address "X". 3 - Kernel search for a page "B" associated with that address "X". If the page was not found, it just an AV exception. Otherwise it proceeds to step 4. 4 - Kernel moves a page "C" from memory to disk in order to make a place for the requested page "B". 5 - Kernel loads the requested page "B" from disk to memory. 6 - Kernel resumes execution of the process "A" like nothing happened. 7 - Process "A" calls VirtualFree. 8 - Kernel dis-allocates page "B". Now, if you just used VirtualAlloc/VirtualFree, de-referencing the pointer is not required and all steps from 1 to 6 aren't necessary at all and paging is not happening too !!! The important thing is that some of the above steps are heavy ... and that's why on their analyze they were taking hours to free the memory. Because a swap from/to disk/memory is happening all the time. The Reverse Memory Free On Window Server 2008 R2 Datacenter Benchmark was consuming seconds instead of hours because they were clever to avoid system paging : They were freeing page in a reverse order than it was allocated. Meaning the last allocated page is getting free first(a last allocated page will likely resides on memory and not on disk) and steps 2 - 6 may not be necessary for all transaction. Yet the paging may still happen but not important as the original code. By understanding this, anyone can quickly understand that a FreeMemory (that implies using GetMemory) is an evil for large block when paging is active. And cannot at any case competes with VirtualFree. Going from hours to seconds is really a high optimization level that worth understanding and changing many bad practices/wrong believes. A better practice would be to use OS functions for larges block: - They don't de-reference pointer ... no issue with paging. - They don't suffer from fragmentation as GetMemory does. Remember the last thing that anyone wants to see when allocating large data is to have fragmentation issue. - They are always aligned to SPS. - Thread safe : they only do one look. GetMemory/FreeMemory is looking twice ! - Provides more additional options. - If portability is an issue: a wrapper will be very useful. Its not like we have a bench of system. Mostly a Windows and Posix: function GetLargeMemory(Size: SIZE_T): Pointer; begin {$IFDEF MSWINDOWS} Result := VirtualAlloc(nil, Size, MEM_COMMIT or MEM_RESERVE, PAGE_READWRITE); {$ELSE POSIX} Result := mmap(nil, Size, PROT_READ or PROT_WRITE, MAP_ANONYMOUS or MAP_PRIVATE or MAP_FIXED, 0, 0); {$ENDIF MSWINDOWS} end; procedure GetLargeMemory(P: Pointer; Size: SIZE_T); begin {$IFDEF MSWINDOWS} VirtualFree(P, Size, MEM_RELEASE); {$ELSE POSIX} munmap(P, size); {$ENDIF MSWINDOWS} end; I hope by now, everyone here is understanding the difference between OS functions and Delphi MM functions.

Are you saying that the benchmark from the article is not enough ?

Again you're completely wrong ! Even if you change C MM with Delphi MM. There is no way by any chance that FastMM can outperform OS functions. Allocating memory perhaps may give just a little closet result (single thread) to what OS may give but freeing memory will never outperform specially if you're under an environment where system paging is actively working. Should I explain more what does that mean ? I'd be very interested if you have some benchmark that prove otherwise. What you're missing Arnaud is the following : At a software level, it may look from the first sigh that there is no difference when using C/Delphi MM against OS Api. But the fact is the memory management is a very complex thing and requires collaboration from different component(RAM, Disk, Software, OS and even CPU). Without understanding the nature/relationship between those components ... you will never understand the full story.

Yes and I will definitely give it a try under my second machine.

@Edwin Yip I just tried on D10.3 and it didn't work. However Startup layout gave me the same expectation as yours. Perhaps if someone can mention a macro plugin that supports mouse event will help you. off topic : I'm really nostalgic seeing your Gif ... I really miss the old IDE look 😥

First, I'm not sure if you're aware about that. But 4KB is not the only available page size ! while most environment supports it, there're some that are using more than 4KB (link) !!! Second, you should know better than anyone what does this mean for fragmentation. Try to convince someone that he runs on a limited environment that for each large allocation, a 4KB or whatever is not an issue ! Personally, the last thing that I want to have when allocating some Gb is dealing with unnecessary fragmentation that can be easily avoided. Third, if I tell you that some OS functions are thread safe, what will be your answer ? Spoiler: Think twice before answering because the answer you're preparing is not what I'm expecting ! You're completely wrong on this ! its not like you need an aligned memory at SPS only to do some basic stuff like changing memory protection. There're many other case. Take a look at the high-performance I/O : ReadFileScatter, WriteFileGather. Do you know that for a read/write operation between mem/disk, its better to have an alignment at SPS. i.e: sections of PE files, large files. Do you know that disk manufacture today tries to align their sector size at SPS (today we have 4Kb, 8Kb)? See File buffering. I'm just going to pretend that I didn't hear that. You knew exactly what I was referring to ! Final thing Arnaud and just to make things clear : I didn't invented the rule that says "using OS functions for large allocation is better than MM" myself. In fact, if you just did some research on the internet, you will find that many developers recommend using OS function for large data against any MM. I remember I saw a statement from Microsoft too ! There was also an interesting benchmark comparing c malloc against Linus/Windows function for allocating/freeing small/large block ... you should definitely take a look at that: https://raima.com/memory-management-allocation/ @David Heffernan I also recommend that you read the above article. It will change your mind about when to call it "Premature optimisation" and when not !

I guess there is so little developers are currently interested about assembly. Use nasm(its very well maintained) to assemble your code and then use gdb/ndisasm to get the opcodes. Also, worth checking FPC (if I remember correctly, it already supports avx).

I know that already and I agree to all what you said except the thing that says there is no benefit for using OS function : Large data tends to be aligned, calling Delphi MM will likely result to allocate one extra page for storing pointer info if the requested size is at the system granularity. Also pointer is not aligned at the system page granularity. Moreover OS functions provides many options against DMM. So if portability is not an issue ... I really don't understand why someone will not use OS functions.