Threading question

[Vandrovnik 214]

7 minutes ago, David Schwartz said:

This is running inside a VM and I cannot install Delphi in it, so the debugger cannot be used.

What about remote debugging?

[David Schwartz 426]

11 minutes ago, Vandrovnik said:

What about remote debugging?

Hmmmm ... totally forgot about that. Never done it, actually. I'll look into it. Thanks!

[A.M. Hoornweg 144]

On 4/17/2020 at 10:39 PM, Anders Melander said:

While Application.ProcessMessages is evil it isn't the cause of the problem here.

 

Use PostMessage instead of SendMessage. SendMessage calls the message handler directly and doesn't use the message queue at all.

https://docs.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-sendmessage

 

I wonder about sendmessage being executed in the context of the calling thread.  Is that documented somewhere?  The HWnd could be in an entirely different process, so there must be some kind of marshaling in between. 

 

 

 

 

 

 

[David Heffernan 2345]

46 minutes ago, A.M. Hoornweg said:

I wonder about sendmessage being executed in the context of the calling thread.  Is that documented somewhere?  The HWnd could be in an entirely different process, so there must be some kind of marshaling in between. 

Yes it is documented in the remarks section of the SendMessage documentation.

 

Parameters are marshaled for messages known by the system.  A good example is WM_COPY.  The data that you pass to that message is marshaled to the target process, because the system knows its structure.  For user defined messages, no marshaling of parameters is performed.

[Guest]

Hi, I  had to register here again to answer this and save you time.

 

Andres example does in fact shows the problem, but assuming this will happen only with SendMessage is wrong, on contrary this happen more often with SendMessage, but not directly with the sent message itself.

 

Let me explain:

1_ First message sent by SendMessage a procedure in thread and that procedure does own the pointer to the that string and will free the memory assigned to it on exiting.

2_ SendMessage is blocking that procedure till return, so far so good, and that pointer with its memory is OK and SAFE to be used in the receiving procedure.

3_ The problem is here start, when the receiving procedure will handle the data, i mean you had data and you need to do something with it, like any normal output operation, so here is the thing, you might send the data to :

A) device with handle like files,sockets.. etc

B) will display the thing on control like Memo or ListBox ..

with the second case most likely there will be another SendMessage sent to the control, and that receiving procedure will exit releasing the thread procedure -> releasing that pointer on stack and its memory, you might still see the data OK and that by chance as MemoryManager will not free it yet (per allocation it is there and accessible)

With first one (A) the thing is even harder to find and debug because you might be using overlapped operation and those are the hardest and nastiest bugs to find !

 

After all that, @David Schwartz may i ask if simply this solve this issue or not ?

procedure StatusOut(const stat: string);
var
  AMsg:string;
begin
  AMsg := UniqueString(stat);
  LogWrite(AMsg, true, true);
  FrmMain.memStatus.Lines.Add(AMsg);
end;

 

[Guest]

I want to add that Application.ProcessMessages was there to solve this issue, by trying to empty the message queue to ensure the execution the following message(s) before exiting and releasing the thread, that is wrong approach with unpredictable result.

[Guest]

On 4/17/2020 at 8:09 PM, David Schwartz said:

Access violation at address 0040DD41 in module 'DocLink.exe'. Read of address 00000001

The easy way to interpret this is the auto reference counting had changed a nil to 1 when some code access freed/released string ( out of scope ), this happen as explained above on second message.

 

On 4/17/2020 at 8:09 PM, David Schwartz said:

I've done some research and it seems that SendMessage should be sufficient. It's ok for low volumes of traffic, but as I send more data, the threads start throwing these AVs.

I would suggest to leave SendMessage/PostMessage to simpler things that doesn't include typecasting or passing data, while using more stable and bullet proof approach to pass something like log strings, like this small class

 

type
  TLogTextContainer = class
  protected
    FLines: TStringList;
    FLock: TCriticalSection;
    procedure Lock;
    procedure Unlock;
    procedure DoFlushInternal;
  public
    constructor Create;
    destructor Destroy; override;
    procedure LogText(const AMsg: string);
    function GetLogged(AutoClear: Boolean = True): string;
    procedure LogClear;
  end;

{ TLogTextContainer }

constructor TLogTextContainer.Create;
begin
  FLock := TCriticalSection.Create;
  FLines := TStringList.Create;
end;

destructor TLogTextContainer.Destroy;
begin
  DoFlushInternal;
  FLines.Free;
  FLock.Free;

  inherited;
end;

procedure TLogTextContainer.DoFlushInternal;
begin
  FLines.Clear;
end;

procedure TLogTextContainer.Lock;
begin
  FLock.Acquire;
end;

procedure TLogTextContainer.Unlock;
begin
  FLock.Release;
end;

procedure TLogTextContainer.LogClear;
begin
  Lock;
  try
    DoFlushInternal;
  finally
    Unlock;
  end;
end;

procedure TLogTextContainer.LogText(const AMsg: string);
begin
  Lock;
  try
    FLines.Add(AMsg);
  finally
    Unlock;
  end;
end;

function TLogTextContainer.GetLogged(AutoClear: Boolean = True): string;
begin
  Lock;
  try
    Result := FLines.Text;
    if AutoClear then
      DoFlushInternal;
  finally
    Unlock;
  end;
end;

now to use that you don't need to send or post a message just call it from any thread anytime, on the UI side use a timer with small interval like 100ms or even less, to get the text and clear it, this way your application are ready to be bombarded with thousands of lines, with almost visible effect on the UI.

 

One more thing here about PostMessage, don't use it without checking the result for false !

It is way more dangerous than you thing and happens in many cases, one of the worse to hunt, debug and reproduce, simply check for false and raise something or repeat it with some logging as it did happen.

[David Heffernan 2345]

1 hour ago, Kas Ob. said:

Hi, I  had to register here again to answer this and save you time.

 

Andres example does in fact shows the problem, but assuming this will happen only with SendMessage is wrong, on contrary this happen more often with SendMessage, but not directly with the sent message itself.

 

Let me explain:

1_ First message sent by SendMessage a procedure in thread and that procedure does own the pointer to the that string and will free the memory assigned to it on exiting.

2_ SendMessage is blocking that procedure till return, so far so good, and that pointer with its memory is OK and SAFE to be used in the receiving procedure.

3_ The problem is here start, when the receiving procedure will handle the data, i mean you had data and you need to do something with it, like any normal output operation, so here is the thing, you might send the data to :

A) device with handle like files,sockets.. etc

B) will display the thing on control like Memo or ListBox ..

with the second case most likely there will be another SendMessage sent to the control, and that receiving procedure will exit releasing the thread procedure -> releasing that pointer on stack and its memory, you might still see the data OK and that by chance as MemoryManager will not free it yet (per allocation it is there and accessible)

With first one (A) the thing is even harder to find and debug because you might be using overlapped operation and those are the hardest and nastiest bugs to find !

 

After all that, @David Schwartz may i ask if simply this solve this issue or not ?

procedure StatusOut(const stat: string);
var
  AMsg:string;
begin
  AMsg := UniqueString(stat);
  LogWrite(AMsg, true, true);
  FrmMain.memStatus.Lines.Add(AMsg);
end;

 

 

This is completely wrong. 

[Lars Fosdal 1792]

A good place to start for writing solid thread code:
https://leanpub.com/omnithreadlibrary

 

[Guest]

14 minutes ago, David Heffernan said:

This is completely wrong. 

Why wrong?

StatusOut is receiving in the original code does receive a const string and does forward it to Memo.Lines.Add which in turn send a message to the end of the message queue, mean while exiting the thread SendMessage lock/wait which in turn release the string. and second message that sent to Memo does have the original pointer from the thread stack( local variable there), Right ?

Please correct me.

[Anders Melander 1782]

2 hours ago, A.M. Hoornweg said:

I wonder about sendmessage being executed in the context of the calling thread.

Please read the whole thread before commenting. I already answered that and posted the relevant section from the SendMessage documentation.

 

6 minutes ago, Lars Fosdal said:

A good place to start for writing solid thread code:
https://leanpub.com/omnithreadlibrary

How would that help when:

• The cause of the problem isn't known.
• The original author of the code doesn't have good understanding of threading issues.
• The current maintainer doesn't either.

   

[Lars Fosdal 1792]

The book gives a reasonably good understanding of threading issues.
That understanding would suggest alternate solutions that avoids the pitfalls of messaging.

[David Heffernan 2345]

5 hours ago, Kas Ob. said:

Why wrong?

StatusOut is receiving in the original code does receive a const string and does forward it to Memo.Lines.Add which in turn send a message to the end of the message queue, mean while exiting the thread SendMessage lock/wait which in turn release the string. and second message that sent to Memo does have the original pointer from the thread stack( local variable there), Right ?

Please correct me.

The code in the original post, using SendMessage is fine. It passes a pointer to the first character of the string, and the call to SendMessage doesn't return until the message has been fully processed.

[Guest]

10 minutes ago, David Heffernan said:

and the call to SendMessage doesn't return until the message has been fully processed.

That is true, but in that "fully processed" there is a call "FrmMain.memStatus.Lines.Add(stat);" which add the string to TMemo, how many SendMessage involved in that ?

Is there any guarantee that Delphi RTL ( or any other skinning library, or even 3rd party software that is hooking this control ) will not Switch to PostMessage on some stage and release the thread before that line return ? 

 

 

[David Heffernan 2345]

17 minutes ago, Kas Ob. said:

That is true, but in that "fully processed" there is a call "FrmMain.memStatus.Lines.Add(stat);" which add the string to TMemo, how many SendMessage involved in that ?

Is there any guarantee that Delphi RTL ( or any other skinning library, or even 3rd party software that is hooking this control ) will not Switch to PostMessage on some stage and release the thread before that line return ? 

 

 

You clearly don't have a robust understanding of how threads work. The outer caller of SendMessage blocks until that call returns. 

[Guest]

And what about that LogWrite(stat, true, true) is it synchronous or asynchronous ? (  guessing here it does write to a file)

To memo or a file, it will matter as that pointer is only valid before that first thread sender return from SendMessage, finish the procedure and clean up.

[Attila Kovacs 629]

41 minutes ago, David Heffernan said:

and the call to SendMessage doesn't return until the message has been fully processed

That's true. But if he goes:

var
FSomething: PChar;

procedure HandleMessage(var Msg: TMessage);
begin
  FSomething := PChar(Msg.WParam);
end;

then that was it. And depending on how he is populating the log messages, the memory manager could give him the same address for X times in the client thread. To have even more fun in debugging.

 

Edited April 20, 2020 by Attila Kovacs

[Guest]

Just now, David Heffernan said:

You clearly don't have a robust understanding of how threads work. The outer caller of SendMessage blocks until that call returns. 

That doesn't answer the question about how many messages involved in adding text to a memo, are they guaranteed to be SendMessage or will involve PostMessage.

[David Heffernan 2345]

Just now, Kas Ob. said:

That doesn't answer the question about how many messages involved in adding text to a memo, are they guaranteed to be SendMessage or will involve PostMessage.

It doesn't matter. The original call, made from the thread does not return until that message has been processed. If the code in the main thread, that processes the sent message, sends or posts messages, that's fine. 

 

I don't think that we should hijack this thread anymore. 

[David Schwartz 426]

FWIW, I'd like to think I have a better-than-average understanding of multi-threading. I spent the first 10 years of my career in the real-time machine-control world and even spent a couple of years working on a real-time multitasking kernel. The thing is, these platforms were all designed from the ground-up to do this, and they were very consistent and predictable.

 

In contrast, I find Windows to be more of a swampy mess when it comes to multi-threading. And when you layer Delphi on top of that, it only gets worse. Thankfully, I have not had to deal with it much. So I do find this mess rather confusing to deal with.

 

The fact that you guys keep arguing with each other about things that should be quite simple only points to the underlying complexity of how Windows offers up it's ugly, hydra-like threading models. It makes me feel like it's not just my own stupidity. 

 

But I do really appreciate the help. 🙂

Edited April 20, 2020 by David Schwartz

[Guest]

@David Schwartz Can you confirm if my suggestion above ( using UniqueString ) solve those AV?

[David Schwartz 426]

can anybody guess why they'd both write to the TMemo and use a separate logger to save the same data to disk instead of just calling memo.Lines.SaveToDisk( ) periodically?

 

In a sense, the memo is a queue, although it's not flushed and cleared.

 

Is there a chance the logger is the source of these AVs?

 

(I tried to set up a TStringList as a buffer instead of the memo and logger, but I couldn't get it to flush when the program terminated.)

Edited April 20, 2020 by David Schwartz

[David Schwartz 426]

35 minutes ago, Kas Ob. said:

@David Schwartz Can you confirm if my suggestion above ( using UniqueString ) solve those AV?

I'll give it a try and let you know.

[Remy Lebeau 1394]

8 hours ago, David Heffernan said:

Parameters are marshaled for messages known by the system.  A good example is WM_COPY.  The data that you pass to that message is marshaled to the target process, because the system knows its structure.  For user defined messages, no marshaling of parameters is performed.

You mean WM_COPYDATA, not WM_COPY.

[Remy Lebeau 1394]

1 hour ago, Kas Ob. said:

That is true, but in that "fully processed" there is a call "FrmMain.memStatus.Lines.Add(stat);" which add the string to TMemo, how many SendMessage involved in that ?

Is there any guarantee that Delphi RTL ( or any other skinning library, or even 3rd party software that is hooking this control ) will not Switch to PostMessage on some stage and release the thread before that line return ?

Yes, because it CAN'T switch to PostMessage().  That would grant the caller permission to free the string data before it has actually been used to update the Memo.  The Add() must be synchronous, regardless of its implementation, blocking the caller until the update is complete to ensure the integrity of the string data.