Communicating from one thread to another and the latter "talking" to a Dbgrid.

[promero 0]

Hello Dear Friends, I have a problem and I would like to hear your opinion. It's in Delphi 11.2
 

I have 2 threads, one that "prepares" emails from templates and another, which is the one that sends them.
 

The first thread, which manufactures/prepares the emails, stores them in an 'EMAILS' table in a SQLITE database, ready to be sent. It is inside a Task class. It works ok.
 

The second thread actually sends the emails recorded in the 'EMAILS' table to the recipients. It is based on the Thread class. With every email sent, it updates the "SENT" column from EMAILS table, with "true" value.
 

What I need:
 

-I need the first thread "notify" the second thread "hey, I sent you lot of new emailss, please send them": that's it: send it a signal. If the second thread don't receive that signal, let it remain paused.
 

-That the second threads, also sends an update of the sent emails to another process. This process is simply a Dbgrid showing the EMAILS table with a Column "Sent". I want that view to update 'sent' column...that is, to shows what the second thread is doing.

Some suggestions?
 

Many thanks Friends

 

[Remy Lebeau 1633]

Have a look at PostThreadMessage(). The email thread can run a mesaage loop looking for a custom message which the generator thread posts to it. The email thread can then post another message to the main UI thread when the grid should update.

 

Alternatively, the email thread can run a loop waiting on a TEvent which the generator thread signals. The email thread can then post a message to the main UI thread. Or use TThread.Queue() instead.

 

Alternatively, you can get rid of the email thread entirely. Wrap the email logic into a TTask which the generator thread creates for a batch of emails. When the TTask is done running, it can update the UI thread to display the results.

Edited February 4, 2024 by Remy Lebeau

[promero 0]

Thank you Dear Remy
 

But I have an almost existential problem.
 

My application is an ERP. Assume this scenario.
 

The Accounting process prepares Account Summaries to clients by e-mail.
 

So, that process calls my "Preparer" process, made with TTask and create the e-mails by recording them prepared in that EMAILS table of an SQLITE database.
 

When the "Preparer" process finishes, it sends the message to the SenderEmails thread (SetEvent style)  so, this thread starts sending them.
 

Now, thanks to the fact that these processes runs in the background, the user enters the Billing process. So, creates invoices to send to clients.

The Billing process calls the process "Preparer", creates the e-mails, records them...and sends the signal AGAIN to the SenderEmails thread...but this thread IS STILL BUSY SENDING the previous emails...

The SenderEmails thread has a method called "BeginDispatch", like this

procedure TSenderEmails.Dispatch;
begin
   FEvento.SetEvent;   //FEvento is a TEvent.
end;

procedure TSenderEmails.Execute;
begin
  FEvento.ResetEvent;
  while not Terminated do
  begin
    if FEvento.WaitFor(10000) = wrSignaled then
    begin
            //query the EMAILs table where SENT field is FALSE
            //send the emails to the SMTP 
            FEvento.ResetEvent; 
    end;
  end;
end;

Here comes my doubt:
 

The SenderEmails thread turns off the TEvent with ResetEvent when all e-mail are sent. 
 

But before, another process had turned it on...isn't a concurrency problem occurring here?
 

A process sent a signal, the worker thread started working upon receiving that signal, but another process signaled the thread again...and the worker thread turned off the signals when finished.
 

So, is it possible to ACCUMULATE the signals so that they are not lost if several processes tell the EmailSender thread to proceed to work?

As you can see, it's more a design issue. What I'm looking for is that all this takes up the least amount of resources possible and it always runs in background.
 

Regards

[Kas Ob. 151]

Hi,

 

5 hours ago, promero said:

So, is it possible to ACCUMULATE the signals so that they are not lost if several processes tell the EmailSender thread to proceed to work?

Of course it is possible.

 

You asked about threads and Remy gave a perfect answer to your question.

 

Now let put my thoughts here:

1) Now you are thinking right and want to do the right thing, i mean eliminate the failing point or at least minimize and if possibly implement a recovery.

2) In my opinion best threads are threads that doesn't talk to each other.

3) You have Sqlite as storage and can serve as central coordinator or a hub while serving its propose to store data/task.

 

What i suggest is to redesign you process a little differently, 

1) Lets say you generated an email ( being to recover/restore/reset password or an invoice ...), this generated in a thread (any thread), then it will be pushed into a table in Sqlite, here comes the table itself with few extra columns,

dtGenerated : will holds datatime of generation.

dtSent : will hold the time it being picked to send !

intSend : the count of send tries.

dtDelivered : hold the time when the email being delivered, and here when the SMTP finished receiving the email, we will consider this a success,

enStatus : hold the status, Error, Failed, Pending , Delivered.... not so crucial but nice to have

You can also put a an extra field to confirm the result of the sending from the SMTP, but this is an extra.

 

Now one thread generate an email, and push it to the DB, there is another (one thread) its job to perform a query for emails in the DB with no dtSent value !, or with pending or ... the way you comfortable with.

After grabbing these emails it will dispatch them to another one or more threads to perform the real SMTP sending, mean while and before dispatching them to these workers, that watcher threads will insert an ID of that email into special table called (lets say tbWorking) where that ID is key hence it is impossible to insert it twice, in Sqlite you can replace ID with simple ROWID from the email table, this is unique value, (also notice all DB servers has something similar like in Sqlite RowID is an Integer but in MYSQL it is hex or short string and there is ROWNUM i think..) 

 

If an email being sent successfully or not, then and only then the email table can be updated accordingly, and a delete should be issued to tbWorking, this tbWorking will server as lock free list that persist in case of catastrophic failure, like app crash or power cut, 

Threads will have a small and focused task to do, not generating emails, then store, them then, send them, then store the result... that is way too much work and it is also slow and will take time maximizing the risk of losing an email due to some other failure or Internet lose ....

 

Anyway, what i wrote above is not a full solution that will fit you, most likely you will need to address the main point, what could happen and how to recover from each and every step and moment ?

You need to think out side the box of generate-store-send-update and linear processing.

 

I had done a project that similar to what you you need, but it was using MySQL and worked without a problem, With MySQL ?, the emails can be generated from completely different server running php and pushed to the MySQL, my server no matter if itself generated the email or from the web server remotely being added could and can send them all.

 

Also don't forget to store logs for these operation, the logging the faster and more secure your process will be, i used two logging method with that, one on files and the other was in table in the DB, recoding the failures and successes.

 

ps the watcher thread has one job to check for newly added or the emails not in tbWorking, it has low interval like 5-10 second, and will assume nothing on it first run, so it will/must update tbWorking on start, because filled value in that table on start means there is a conflict in need to be address, and the best way is to resend unless these are times email like resetting password then drop them.

 

Hope that helps and give you insight.

[Kas Ob. 151]

Forgot the moral of all that 

 

The best way for high performance is to not push the threads (job or whatever), let them pull/poll on their own time.

 

By centralizing the email handling with watcher in intervals, you will be removing huge bottleneck in case there is many emails, while the dispatchers will be almost a sleep sending and using SMTP because it is slow, so yes you can have 64 dispatcher at the same time and will not notice the CPU usage goes above %1, while the watcher can read hundreds in one go and also this is very short and fast process.

[promero 0]

Thank you guys! Remy and Kas Ob.

One problem about desing is overthinking the future problems: these ideas unlock my problem.