Freeing tasks memory

[Olivier EXEL. 1]

Hi, and thanks for any information.

 

On RIO, using OTL, with GlobalOmniThreadPool, on an AMD Ryzen 1700X (8 cores x 2 threads), 48 Go RAM and SSD.

I use usefull OTL like that, has described in the OTL Programming book I bought ; my program is based on the ThreadPool demo of the book :

        CreateTask(THelloWorker.Create(IA.Handle, delay_ms)).SetParameter('P1',par1).MonitorWith(OmniTED).Schedule;

 

I notice that "delay_ms" is set with 3.000 ms value, (or even mor, no change), and my task has a "Task.Terminate" as the last line of the procedure. And with GlobalOmniThreadPool.MaxExecuting := 45; before launching tasks loop (as a result more or less 95% of full cores used).

 

I launch more or less 4 000 tasks, from a compiled Delphi program (.EXE = 18 Mo).

The HelloWorker does always the same work for each 4 000 tasks, processing from 1 to 12 seconds each facing work to do.

Each task use arrays and tlist and so, create and freeing in the right way for what i controlled many times just in case...

 

But, so, at the end of the total works, my main process RAM use grew up of about 1 Go.

Launching some more works (4 000 taks, many times), the growing result going on...

 

Is there anything I missed in the OTL process ?...

 

Thanks

 

Olivier

[Primož Gabrijelčič 223]

Create a reproducible test case and post i here.

 

BTW, why are you createing 4000 tasks? Surely, there's a better way to solve your problem.

 

[Olivier EXEL. 1]

Thanks for your answer.

 

I was so sure of having all freeing that ... I got a doubt while reading again your OTL book, not finding "the missing point" in my implemntation of OTL.

And. Finally, I found some memory Leak from an external DLL that I did not covered. So, now, it seems that I have no more troubles. Sorry for disturbing the forum for a so "typical" issue.

 

To answer your question, I'm "playing" with a big database of daily evoluting datas (macro economics). So, as an example, the main first calcs needs more or less 7 days x 24 hours to do it with my hardware configuration ; the daily recalcs tooks more or less at least 1 hour ; and that with OTL use, and SQL engine and so with very various and tested (mesured) optimizations.

So OTL is the right answer for me to improve my hardware and reduce time consuming. And I have to say OTL is really a great solution to use. Thanks.

[Primož Gabrijelčič 223]

Great that you got it working! 

 

Re 4000 tasks I was thinking more about - what if you set up a Parallel.BackgroundWorker with multiple workers and then schedule work units to it? You would have one always running execution engine (i.e. background worker) and instead of 4000 tasks you would have 4000 work units.

[Olivier EXEL. 1]

2 hours ago, Primož Gabrijelčič said:

Great that you got it working! 

 

Re 4000 tasks I was thinking more about - what if you set up a Parallel.BackgroundWorker with multiple workers and then schedule work units to it? You would have one always running execution engine (i.e. background worker) and instead of 4000 tasks you would have 4000 work units.

Yep, I was just reading the 3.9 section of your book, while checking DelphiPraxis posts !  😉

I think BackgroundWoker is the case I've to use,

I've tested TaskGroup option this week-end and it's a great functionnality too...