GExperts supports even more laziness

[Mahdi Safsafi 225]

7 minutes ago, dummzeuch said:

There are two reasons why I didn't want to use DDetours:

1. I didn't want to add yet another 3rd party library to GExperts. Using 3rd party libraries means that with a new Delphi version I have to either update these libraries myself to support the new version or wait for the maintainer to do it. And even if that wasn't a problem, you never know when such a library drops support for older Delphi versions or maybe just adds changes without keeping older Delphi versions in mind. On top of it they always add complexity and increase executable size and memory requirements.
2. DDetours did not compile with Delphi 2007. Since that's currently my main development version this was the end of it.

I see the advantage of allowing chained hooks, but supporting Delphi 2007 is still more important to me.

Thanks for the explanation. I clearly understood you by now. 

You opened my mind, I'll try to give much priority for old Delphi versions when releasing the next version of DDetours as its obviously many 3rd party plugins that hooks the IDE intend to support older Delphi versions.

[dummzeuch 1505]

26 minutes ago, Mahdi Safsafi said:

Thanks for the explanation. I clearly understood you by now. 

You opened my mind, I'll try to give much priority for old Delphi versions when releasing the next version of DDetours as its obviously many 3rd party plugins that hooks the IDE intend to support older Delphi versions.

That would be really nice. If you need any help testing, please give me a shout.

Edited July 6, 2020 by dummzeuch

[dummzeuch 1505]

P := GetParam(Obj);
P[$A1] := 1; // resume = true.
PostDebugMessage(Obj, 1, P); // this will resume the app.

How did you come up with this @Mahdi Safsafi ?

It works fine on my GExperts development computer, but it doesn't at work, where it skips to the end of the procedure that raised the exception and stops there. So there must be something missing.

Both are the same Delphi Version, but different Windows versions (10 vs. 8.1) and (AMD Phenom II X6 1090T vs. Intel Xeon).

 

Edit: It also works fine in a XenServer virtual machine running on a (different) Intel Xeon with Windows 8.1.

And also in another XenServer VM with Windows 8.1.

 

(Which probably means that there is something really odd with my work computer.)

 

edit2: This is even weirder: It works with my usual test project (with just a raise exception.Create('bla') in a form's OnCreate event), but fails in a DUnit unit test where it would have been really useful to ignore the expected exceptions.

 

edit3: It's not the computer, it's the Delphi version. Delphi 2007 works with the test project but not with the unit test project. Delphi 10.2 works with both. At least I can now reproduce it on my GExperts development computer, not that it helps much.

 

Any input would be appreciated.

Edited July 18, 2020 by dummzeuch

[Guest]

@dummzeuch Hold on that PostDebugMessage.

 

Mahdi tried to extract it right but missed one thing, i assume he is familiar with reversing and tracking code generated by C/C++ compiler but here we are talking Delphi with no StdCall, that PostDebugMessage is 5 parameters function not 2 !

 

Let me dig more and see if i can do it, though i think Mahdi can extract it right after this note.

[Guest]

What got till now is that the parameters are at least 4 for sure.

[Mahdi Safsafi 225]

@dummzeuch 

Quote

How did you come up with this @Mahdi Safsafi ?

Go to DoShowException entry point and then go to the end of that function(on the right panel, I made some pseudo pascal code for clarification) :
https://i.ibb.co/zZd5spY/Dbg.png

 

Quote

edit3: It's not the computer, it's the Delphi version. Delphi 2007 works with the test project but not with the unit test project. Delphi 10.2 works with both. At least I can now reproduce it on my GExperts development computer, not that it helps much.

Can you give a simple DUnit test ? Basically when an exception occurs and you click continue, the RTL tries to find a handler for that exception and executes it. So stopping at the end of procedure is weird!

I'm not very familiar with D2007 but was DUnit shipped with Delphi 2007 at that time ?

I guess, either DUnit is doing something behind the scene ! Otherwise why under D2007, it worked with a simple test project and failed on DUnit project . Why it just worked fine under 10.2 for both project ? Or there is something with D2007 RTL. Can you confirm if the same behavior occurs without hooking (disabling GExpert) ?
 

Also, I suggest that you disable GExpert for DUnit on D2007(make sure there is no hook is installed as well), put a break point on DoShowException, run a dummy project that raise exception and try to follow until the function returns. See whether a call to PostDebugMessage was made or not.

 

50 minutes ago, Kas Ob. said:

@dummzeuch Hold on that PostDebugMessage.

 

Mahdi tried to extract it right but missed one thing, i assume he is familiar with reversing and tracking code generated by C/C++ compiler but here we are talking Delphi with no StdCall, that PostDebugMessage is 5 parameters function not 2 !

 

Let me dig more and see if i can do it, though i think Mahdi can extract it right after this note.

100% sure its 3 param !

[dummzeuch 1505]

6 minutes ago, Mahdi Safsafi said:

@dummzeuch 

Go to DoShowException entry point and then go to the end of that function(on the right panel, I made some pseudo pascal code for clarification) :
https://i.ibb.co/zZd5spY/Dbg.png

I see. That definitely looks like the code should be correct, unless @Kas Ob. is right and there are more parameters.

 

I dug a bit deeper with the DUnit weirdness: When I enable debug dcus the debugger stops not at the end of the procedure but on an exception handler inside the DUnit code. That makes sense of course: The debugger stops at the position where there is code to display.

 

The test case is simple:

type
  TSomeTestCase = class(TTestCase)

[...]

procedure TSomeTestCase.TestException;
begin
  raise Exception.Create('bla');
end;

(Of course that's not the original code where it happened, but it happens with this too.)

 

11 minutes ago, Mahdi Safsafi said:

I'm not very familiar with D2007 but was DUnit shipped with Delphi 2007 at that time ? 

Yes, but isn't it still shipped with the current version too? I updated to the latest version of the test framework from source forge though. But that reminds me of something I have to check: Do I use the updated framework code in Delphi 10.2 too? I'll check and come back with the result.

 

This is the code in TestFramework.pas where it stops:

procedure TTestCase.RunTest(testResult: TTestResult);
begin
  assert(assigned(FMethod), sMethodNotFound + FTestName + '" ');
  FExpectedException := nil;
  try
    try
{$IFDEF CLR}
      testResult.FMethodPtr := nil;
{$ELSE}
      CheckMethodIsNotEmpty(tMethod(FMethod).Code);
      testResult.FMethodPtr := tMethod(FMethod).Code;
{$ENDIF}
      FCheckCalled := False;
      Invoke(FMethod);
      if FFailsOnNoChecksExecuted and (not FCheckCalled) then
        Fail(sNoChecksExecuted, testResult.FMethodPtr);
      StopExpectingException;
    except
      on E: ETestFailure  do
      begin
        raise;
      end;
      on E: Exception  do // <<<< the debugger stops here
      begin
        if  not Assigned(FExpectedException) then
          raise
        else if not E.ClassType.InheritsFrom(fExpectedException) then
          FailNotEquals(fExpectedException.ClassName, E.ClassName, sExceptionUnexpected, ExceptAddr);
      end;
    end;
  finally
    FExpectedException := nil;
  end;
end;

It stops on the line with "on E: Exception do".

15 minutes ago, Mahdi Safsafi said:

Can you confirm if the same behavior occurs without hooking (disabling GExpert) ?

Yes, it definitely only happens when the Filter Exceptions expert is used. If it's disabled (which uninstalls the hooks, now using ddetours btw.) and the IDE shows its original dialog, pressing "Continue" works as expected.

[dummzeuch 1505]

1 minute ago, dummzeuch said:

Do I use the updated framework code in Delphi 10.2 too? I'll check and come back with the result.

I just checked: Yes, it's the same framework code in both versions.

[Guest]

22 minutes ago, Mahdi Safsafi said:

100% sure its 3 param !

You are right, you were looking on the way it been called and i was checking by walking it from inside.

What did confused me is those

 

The interface typecasting on local+8 , which i recognize now that it been the initialized to nil, but what really perplexing me is the push edi !

edi is not changed in the function itself, if the point is to save the param3(ecx) then why not just push it instead of loading it into edi before the try..finally block.

[Mahdi Safsafi 225]

@dummzeuch 

Ok, I downloaded and installed D2007. On Win10 its mostly unusable for me (a lot of exceptions). But I could disasm DoShowException. Here is what I find :

As you can see, unlike Rio, the field offset is 0x99 and not 0xA1. Try change the offset for D2007 and let me know if that works for you.

If it does, you may need to check all other old Delphi version too !

[Mahdi Safsafi 225]

@Kas Ob. 

Never rely on "on fly tools" ... from my experience they can give wrong information. Better is to use your hand with your mind.

BTW, what's your debugger ? Its interface looks familiar for me. Is it IDA ?

[dummzeuch 1505]

33 minutes ago, Mahdi Safsafi said:

As you can see, unlike Rio, the field offset is 0x99 and not 0xA1. Try change the offset for D2007 and let me know if that works for you.

That did it! Thanks a lot (again), for your time and effort.

34 minutes ago, Mahdi Safsafi said:

If it does, you may need to check all other old Delphi version too ! 

It's probably time to brush up on my (dis)assembly skills.

[Guest]

31 minutes ago, Mahdi Safsafi said:

BTW, what's your debugger ? Its interface looks familiar for me. Is it IDA ?

Not a debugger and not IDA, my very old idea detect it as 5 parameters !

 

That was Ghidra https://ghidra-sre.org/ , a full decompiler from NSA

 

 

[Mahdi Safsafi 225]

1 hour ago, Kas Ob. said:

Not a debugger and not IDA, my very old idea detect it as 5 parameters !

Very sorry man I thought that you relied on some tools ... shame one me.

[Mahdi Safsafi 225]

2 hours ago, dummzeuch said:

That did it! Thanks a lot (again), for your time and effort.

Happy for you !

Quote

It's probably time to brush up on my (dis)assembly skills.

In the attached file, you will find a small tool that automatically extracts offset for you (this should help you to validate all older versions). The tool is written using C as it uses my AMED  decoder.

I attached a source an a binary. Use it as follow: 

OffsetDumper win32debugide260.bpl @Win32debug@TNativeDebugger@DoShowException$qqrv

Good luck.

OffsetDumper.rar

[Guest]

13 minutes ago, Mahdi Safsafi said:

Very sorry man I thought that you relied on some tools ... shame one me.

No problem man.

 

Tracking and Decompiling when interfaces from Delphi involved is completely different beast, i need to work on that part.

[dummzeuch 1505]

1 hour ago, Mahdi Safsafi said:

Happy for you !

In the attached file, you will find a small tool that automatically extracts offset for you (this should help you to validate all older versions). The tool is written using C as it uses my AMED  decoder.

I attached a source an a binary. Use it as follow: 

OffsetDumper win32debugide260.bpl @Win32debug@TNativeDebugger@DoShowException$qqrv

Good luck.

OffsetDumper.rar

Thanks. Yet again.

Apparently it's $99 up to XE and $A1 from XE2 up.

Which is a bit odd, because the class name of the debugger changed between 2010 and XE from TWin32Debugger to TNativeDebugger so I would have expected that offset to change at the same time. But that's not the case.

 

I could not get this to work with Delphi 2006. It kind of works but throws access violations internally. I haven't tried Delphi 2005. I guess I'll not bother with these unless somebody else wants to do the debugging work.

[Mahdi Safsafi 225]

8 minutes ago, dummzeuch said:

Thanks. Yet again.

Apparently it's $99 up to XE and $A1 from XE2 up.

Which is a bit odd, because the class name of the debugger changed between 2010 and XE from TWin32Debugger to TNativeDebugger so I would have expected that offset to change at the same time. But that's not the case.

 

I could not get this to work with Delphi 2006. It kind of works but throws access violations internally. I haven't tried Delphi 2005. I guess I'll not bother with these unless somebody else wants to do the debugging work.

Just assume its 0x99 on D2006 and under... If it works than its 0x99.

If it doesn't, send me win32debugideXxx. Bpl file for D2006... I'll take a look but I don't promise you anything as I only can do disassembling. 

[dummzeuch 1505]

9 minutes ago, Mahdi Safsafi said:

Just assume its 0x99 on D2006 and under... If it works than its 0x99.

If it doesn't, send me win32debugideXxx. Bpl file for D2006... I'll take a look but I don't promise you anything as I only can do disassembling. 

I know it's $99, I had already found the code in the debugger before you sent me the tool. The problem seems to be something else, maybe it's really different parameters this time. I will send you the bpl via pm anyway, if you like to have a look.

[Mahdi Safsafi 225]

2 hours ago, dummzeuch said:

I know it's $99, I had already found the code in the debugger before you sent me the tool. The problem seems to be something else, maybe it's really different parameters this time. I will send you the bpl via pm anyway, if you like to have a look.

Sorry about that I miss understand your comment.

I did some static disasm (no debugging) for the file you send me and it appeared that offset to DbgObj.Param has changed too 0x3C instead of 0x40. and for Param.FResume its 0x99 instead of 0xA1(you know that already). Try again with (0x3C, 0x99) and tell me if it works.

[dummzeuch 1505]

14 hours ago, Mahdi Safsafi said:

Sorry about that I miss understand your comment.

I did some static disasm (no debugging) for the file you send me and it appeared that offset to DbgObj.Param has changed too 0x3C instead of 0x40. and for Param.FResume its 0x99 instead of 0xA1(you know that already). Try again with (0x3C, 0x99) and tell me if it works.

 

edit: Stupid me got the ifdef wrong:

{$IFNDEF GX_DELPHI2007_UP}
function GetParam(Obj: Pointer): Pointer;
asm
  mov eax, [eax + $40]
end;
{$ELSE}
function GetParam(Obj: Pointer): Pointer;
asm
  mov eax, [eax + $3C]
end;
{$ENDIF}

This should of course have been IFDEF rather than IFNDEF. Now it seems to work fine. I have to make some more tests.

 

This seems to work better: Only one AV before the dialog is shown, the second one after it was shown is gone. Also, if I filter the dialog, it works like in the later versions. There is still the AV before the dialog would have been shown though:

 

[4589E8C4]{bds.exe     }
 + $0[20006D23]{rtl100.bpl  } System.System.@HandleAnyException (Line 9963, "sys\system.pas" + 13) + $0
 + $1AF[77A98E3F]{ntdll.dll   } RtlInterlockedCompareExchange64 + $1AF
 + $21[77A842D1]{ntdll.dll   } KiUserExceptionDispatcher + $21
 + $5[20A89EB0]{dbkdebugide100.bpl} Debug.Debug.TDebugger.HandleDebugMessage (Line 8569, "Debug.pas" + 😎 + $5
 + $1E[20C21189]{coreide100.bpl} DebuggerMgr.DebuggerMgr.TDebuggerMgr.HandleDebugMessage (Line 1942, "DebuggerMgr.pas" + 1) + $1E
 + $D[00415A1F]{bds.exe     } AppMain.AppMain.TAppBuilder.actnDockEditWindowUpdate (Line 6345, "ui\AppMain.pas" + 6) + $D
 + $6[201406A7]{vcl100.bpl  } Controls.Controls.TWinControl.WndProc (Line 7304, "Controls.pas" + 111) + $6
 + $5[20159E7F]{vcl100.bpl  } Forms.Forms.TCustomForm.WndProc (Line 3512, "Forms.pas" + 136) + $5
 + $6[2013FDD0]{vcl100.bpl  } Controls.Controls.TWinControl.MainWndProc (Line 7073, "Controls.pas" + 3) + $6
 + $0[20040E4C]{rtl100.bpl  } Classes.Classes.StdWndProc (Line 11583, "common\Classes.pas" + 😎 + $0
 + $49[76F044B9]{user32.dll  } AddClipboardFormatListener + $49
 + $B27[76EE4FF7]{user32.dll  } CallWindowProcW + $B27
 + $229[76EE4149]{user32.dll  } DispatchMessageW + $229
 + $B[76EFE8AB]{user32.dll  } DispatchMessageA + $B

 

I'll try to find where it happens.

Edited July 19, 2020 by dummzeuch

[dummzeuch 1505]

It works now with Delphi 2006. It also kind of works with Delphi 2005, but there is a different issue that might or might not be related to this.

 

Thanks again for your help. Care to get a free life time license of GExperts? Oh wait, it's free already. 😉

 

At least I would like to add you to the list of major contributors to the project shown on the about dialog. Please contact me via PM if that's OK with you and whether I should add you as @Mahdi Safsafi or if you prefer an alias.

[Mahdi Safsafi 225]

58 minutes ago, dummzeuch said:

Thanks again for your help. Care to get a free life time license of GExperts? Oh wait, it's free already. 😉

Free and much important for me it is open source. I've a lot of respect for people that are behind great open source projects: FPC/Lazarus, JEDI, S4D, Fast/ScaleMM, HxD, ... and GExperts is no exception.

I learned a lot from open source community. The least thing I can do is to provide feedback/help as long as its not beyond my knowledge.

Thanks for your time and effort to keep GExpers shining. 

 

Quote

At least I would like to add you to the list of major contributors to the project shown on the about dialog. Please contact me via PM if that's OK with you and whether I should add you as @Mahdi Safsafi or if you prefer an alias.

This is very kindness from you. Thanks man ! 

[dummzeuch 1505]

I managed to fix the problem with Delphi 2005. It was totally unrelated: Some code when loading the code formatter configuration raised an external exception C000001D ((Illegal Instruction) for an inlined function when optimization was enabled. In the process I added the same exception handling to the editor experts that was already in place for regular experts, so now one faulty editor expert can no longer prevent all others from being loaded and initialized. Also the user gets a better error message in that case.

[dummzeuch 1505]

9 minutes ago, Mahdi Safsafi said:

This is very kindness from you. Thanks man ! 

You have definitely earned it. Without your help, I wouldn't have been able to add this functionality and track down all those bugs.