Olivier EXEL.

On 2/15/2019 at 4:23 PM, Sonjli said:

Thanks Olivier. I'll try the background workers.

 

For the sentence:

Sorry, but I didn't understand... (maybe my poor english? 🙂 )

I meant to do that once only before the loop : 

otlWaitGroup := CreateTaskGroup; // Here create it

18 minutes ago, Sonjli said:

While experimenting I found that the problem is the creation of the otlWaitGroup.

I created it out of the main loop; so now I moved it inside the loop like in the code below.

I have some doubts:

Why the reference to OmniTasks stay inside the WaitGroup until it is detroyed? Are they weak references?

Why do I have to create many waitGroups instead of creating only one, and use it many times?

Thanks anyone.

	   while not GetArticoloVersato.getTabella.Eof do
               begin
                  ...
                  ATaskList := CreateTaskControlList;
                  otlWaitGroup := CreateTaskGroup; // Here create it
                  try
                     // Run all registered threads
                     FBroadcastActions.ForEach(Action<IBroadcastAction>(
                        procedure(ABroadcastAction: IBroadcastAction)
                        begin
                           ATaskList.Add(CreateTask(
                              procedure(const Task: IOmniTask)
                              begin
                                 Sleep(100); // Only for test
                                 // ABroadcastAction.Run(Task, GetMacchina);
                              end).MonitorWith(FEventMonitor).Join(otlWaitGroup).Schedule(otlPool));
                        end
                        ));

                     otlWaitGroup.WaitForAll;

                     if otlPool.CountExecuting > 0 then
                        raise Exception.Create('Oh my..., thread still running');

                     ATaskList.Clear;
                  finally
                     otlWaitGroup := nil; // Here free it
                     ATaskList := nil;
                  end;
					...
                end;

I've just experimented an equivalent scenario and had an increasing memory too. I moved for an Parallel.for loop and it has resolved the issue ; but I had some freeing for long tasks.

I noted that, in this case (parallel.for), if you have an exception not taken in charge  internaly in the thread (try ...), then the threads is killed without freeing and u may experiment not finishing the group, for what I understand.

For now, for what I need  OTL Background Workers is the best way (has Gab wrote me in a precedent flow) ; one interesting thing is ... the exception management 'cause I had some in the threads with SQL timeout (that's I have to treat anyway).

Your use of TaskGroup is once, before ce loop ; then your "join" in the loop attach the task to the group..

2 hours ago, Primož Gabrijelčič said:

Great that you got it working! 

 

Re 4000 tasks I was thinking more about - what if you set up a Parallel.BackgroundWorker with multiple workers and then schedule work units to it? You would have one always running execution engine (i.e. background worker) and instead of 4000 tasks you would have 4000 work units.

Yep, I was just reading the 3.9 section of your book, while checking DelphiPraxis posts !  😉

I think BackgroundWoker is the case I've to use,

I've tested TaskGroup option this week-end and it's a great functionnality too...

Thanks for your answer.

I was so sure of having all freeing that ... I got a doubt while reading again your OTL book, not finding "the missing point" in my implemntation of OTL.

And. Finally, I found some memory Leak from an external DLL that I did not covered. So, now, it seems that I have no more troubles. Sorry for disturbing the forum for a so "typical" issue.

To answer your question, I'm "playing" with a big database of daily evoluting datas (macro economics). So, as an example, the main first calcs needs more or less 7 days x 24 hours to do it with my hardware configuration ; the daily recalcs tooks more or less at least 1 hour ; and that with OTL use, and SQL engine and so with very various and tested (mesured) optimizations.

So OTL is the right answer for me to improve my hardware and reduce time consuming. And I have to say OTL is really a great solution to use. Thanks.

Hi,

Working on RIO delphi version. Having a tricky issue.

I'm looking for weeks for a solution to create a virtual MONITOR device on Windows 10 using the VCL (not FMX).. I'm not looking on creating Windows Desktop or so. That is.

I explain the situation. In Windows Parameters, going on view showing the monitors, you can click on DETECT. On some workstation, the system show a new monitor with the information "Monitor not detected" ; but, when you click on this new monitor, and go on "Mutliple screen option", you can select "ever, try to connect to VGA - or Screen Laptop). Then, for exemple i have two monitors, then the third monitors is usable as u can drop you application and you mouse on this "virtual monitor" for example.

My problem is double : on some systems, there is no dectection ; how to code the fact to enforce the creation of this "third" monitor like it's possible on some systems.

As you see, it is certainly a low system issue, perhaps depending on monitors drivers, ... ???

Thanks for your information.

Olivier

It's now available in Get It menu.

Hi, and thanks for any information.

On RIO, using OTL, with GlobalOmniThreadPool, on an AMD Ryzen 1700X (8 cores x 2 threads), 48 Go RAM and SSD.

I use usefull OTL like that, has described in the OTL Programming book I bought ; my program is based on the ThreadPool demo of the book :

        CreateTask(THelloWorker.Create(IA.Handle, delay_ms)).SetParameter('P1',par1).MonitorWith(OmniTED).Schedule;

I notice that "delay_ms" is set with 3.000 ms value, (or even mor, no change), and my task has a "Task.Terminate" as the last line of the procedure. And with GlobalOmniThreadPool.MaxExecuting := 45; before launching tasks loop (as a result more or less 95% of full cores used).

I launch more or less 4 000 tasks, from a compiled Delphi program (.EXE = 18 Mo).

The HelloWorker does always the same work for each 4 000 tasks, processing from 1 to 12 seconds each facing work to do.

Each task use arrays and tlist and so, create and freeing in the right way for what i controlled many times just in case...

But, so, at the end of the total works, my main process RAM use grew up of about 1 Go.

Launching some more works (4 000 taks, many times), the growing result going on...

Is there anything I missed in the OTL process ?...

Thanks

Olivier