Python code is running in main thread (on standard) ?

[diaroeh 0]

Dear all,

 

after testing Python4Delphi which is, I think, a good bridge/library between these two worlds, 

I would like to know if it is right that on standard, the python scripts are running in the main thread

of the program, so that the GUI is blocked as long as the python script is running and that there are

also no callback (NotifyEvents) available during this time. This is the behavior that I observed.

Or is there a switch/paramter on e.g. TPythonEngine so that interprocess GUI communication is

available during python script execution?

 

If communication is only possible when the script execution is running in a seperate thread, what is

about TPythonGUIInputOutput and the interprocess communication ?

Does it mean that this class as GUI object is always executed in main  thread and therefore it is not

threadsafe?

 

Thanks for your kind answers  

 

Sincerely 

 

Dirk

[Vincent Gsell 11]

Hello

 

Pi4D is great! (Here is a "just in time" webinar ;))

 

IMHO, the use of threads is not straightforward with python.
(in python alone, it's already not great, that said ;))

 

there are demos with thread management in the Pi4D subdirectories.

 

Please, study it : for your issue, it will help.

 

That said, here is my experience in python threading scenarios applied in production:

 

- Use only 1 pythonEngine.

 

- Simple Thread: Use the provided TPythonThread class.
If we use only 1 thread so as not to "block the GUI" (as you seem to want to do), it works fine.

Be careful, if you want to use the IO components, you *must not* set PythonEngine.ThreadMode to newInterpreter.

 

- NewInterpreter disables IO (see PythonExecute)

 

- In all case, An "AllowThreading" directive must be called before and after using thread. (see demo)

 

- In multithreading (more than 2 threads), it is imperative to set PythonEngine.ThreadMode to newInterpreter (at least, according to my unit tests).

--> Consequence: no more IO. We have to manage differently via the scripts

- In Massif mutlithread : 
Under linux, as part of a cloud server running python services, I use Python by running a command line run and getting the standard output.
This saves resources but we no longer have the advantages of handling scripts via P4D at all. So I adapt my scripts.

 

Hope this helps!

[David Heffernan 2345]

Python multithreading is kinda broken because of the GIL, which tends to prevent scaling. Multiprocessing is the way. 

[pyscripter 689]

There are many good reasons (and some not so good ones) for using P4D and one of them is to get access to the libraries available for python (another one is for application scripting),  Quite a few of python libraries already make use of the available CPUs or even GPUs (e.g. Tensorflow, PyTorch, scikit-learn etc.).   There is absolutely no benefit (maybe the opposite) in using multi-threading or multi-processing with say Tensorflow stuff.  For other tasks you need to bear in mind:

• Avoid premature optimization and then identify and focus only on bottlenecks
• Tasks you implement in Delphi instead of pure python are likely to be much faster, even before you employ Delphi multi-threading.
• Use TPythonThread mainly to avoid blocking the Delphi main thread, but not as a tool to process things faster.
• You can use multi-processing as a last resort.  In the vast majority of cases you should not need to do that.
• Make sure you know what you are doing before trying to use Python threads (and multi-threading in general) .  Study Demo 33 in depth for instance.

By the way the list of languages that have the same limitations as python in not being able to exploit multiple CPUs directly via threading, includes other popular languages such as JavaScript, Ruby and R.  It is interesting that this limitation has not prevented their widespread adoption in an era in which the main increase in processing power comes from the increase in the number of cores.

Edited October 14, 2021 by pyscripter

[David Heffernan 2345]

1 hour ago, pyscripter said:

By the way the list of languages that have the same limitations as python in not being able to exploit multiple CPUs directly via threading, includes other popular languages such as JavaScript, Ruby and R.  It is interesting that this limitation has not prevented their widespread adoption in an era in which the main increase in processing power comes from the increase in the number of cores.

Certainly in the case of R it has a *nix heritage and is well supported on *nix and Windows platforms. Multiprocessing has always been the primary way to do parallel processing on *nix so it's entirely reasonable the R is single threaded and also reasonable that this has not got in the way of its success. 

[pyscripter 689]

13 minutes ago, David Heffernan said:

Certainly in the case of R it has a *nix heritage and is well supported on *nix and Windows platforms.

From R: Contributors (r-project.org)

Quote

The Windows port was developed by Guido Masarotto (for a while a member of R Core) and Brian Ripley, then Duncan Murdoch (a former member of R Core) and currently by Jeroen Ooms (base) and Uwe Ligges (packages).

Incidentally, and off-topic, did you know that Duncan Murdoch was a originally a Turbo/Borland Pascal enthusiast, with many contributions to the Pascal community?

Edited October 14, 2021 by pyscripter

[diaroeh 0]

Dear all,

 

I thank you all for your manifold answers, informations and discussions,

 

I have experienced for sometimes in the last months and I saw the problem,

that I have tried to explain, maybe not in the best matter.

 

My task is to build up a GUI-Application on existing python scripts, and therefore 

Multithreading is not a central question. Pragmatically it will be used/implemented 

if it is neccessary. 

 

I have understood the GIL system and therefore my goal is not! a multithreading  application,

but

1. Delphi GUI-Thread is not blocked

2. Interaction between python-script and Delphi application during script execution,

    this means at first

    - output from script was given to TPython(GUI)InputOutput immediatly

    - Setting a value to a variable like myVar01.value in Delphi was registered in Python-Script during execution and vice versa

 

In my tests/experience (without using) TPythonThread a script like :

-----------------------------------------------------

for i in range(1,10):
    print(i) 
    myVal01.value=i //(DelphiVar)

    time.sleep(1)

-----------------------------------------------------

generates :

- print()-output on TPython(GUI)InputOutput

- changes on myValue via notify

both first after!! execution of the whole scipt,

which means in this example 10 seconds

 

So the first problem (blocked GUI) can be solved easily by using TPythonThread.

 

But I just don't see the solution to get response/interaction from the script during execution,

my sorrow, that there also I have to wait finishing the script (e.g. for response for 10 seconds)

when it is running in a thread.

 

If this is running, everything is fine for me, independent from multithreading or not.

At this point I am asking me if I have overlooked somthing.

 

Thanks for the direct hint on TPythonThread and IO... 

and any hint that will come ...

 

Sincerely

 

Dirk

Edited October 14, 2021 by diaroeh

[diaroeh 0]

Dear all,

in addition to my last post I want to say, that the problem (no print() output during script execution) is a
behaviour, that seems to happens under FMX development, using VCL I got a normal print() as expected
every second.
@pyscripter : You know a reason why ?

Thanks in advance for your answer.

Sincerely

Dirk

Edited October 16, 2021 by diaroeh

[pyscripter 689]

Hard to tell without knowing what you are doing.   If you keep the main thread busy with your python script, your application may not repaint the controls until it gets idle.  You could force a repaint though:

 

-  Assign and event handler to OnSendUniData.  Inside that event handler do something like:

GuiInputOutput1.DisplayString(Data);
Edit1.Repaint;

The above could slow down things though.  I assume you have DelayWrites set to False.

 

 

[diaroeh 0]

Dear pyscripter :

Ok, this is what I am doing at this moment, nothing spectactular, just like examples :

1 (Main-)Form
1 TPythonEngine
1 TPythonGUIInputOutput
1 TPythonDelphiVar
3 TMemos (Python-Script(Input),GUI-IO, DelphiVar-IO(later))
1 Button to Start Script
Thats all on Components

Button-ClickEvent :
PythonEngine1.ExecuteString(Memo1.Lines);

Python-Script itself :
#-------------------
import time

for i in range(1,5) :
print(i)
time.sleep(1)
print("Done ...")
#-------------------

Running this under FireMonkey(Win64) :
Memo2(GUI-IO) is updated after 5 seconds
- DelayWrites doesn't change behaviour

Running this under VCL(Win64) :
Memo2(GUI-IO) is updated every second (as expected)

Did you mean that FMX did not repaint after pyscipt::print(), but VCL did?

Edited October 16, 2021 by diaroeh

[pyscripter 689]

A Vcl Memo is a Windows native control and different from the FMX Memo.

 

Your code keeps the main thread busy.  Sleep passes control to other threads. 

Why don't you try something along the lines of what I suggested above.

Edited October 16, 2021 by pyscripter

[diaroeh 0]

Dear pyscripter,

 

I did it as you suggested (OnSendUniData), but it seems this event handler was never called

under FMX.

I also erased IO from TPythonGUIInputOutput, no response. Now there is no output at 

any time....   ???

[pyscripter 689]

Have you set UnicodeIO to true?  Otherwise the OnSendData handler is called instead of OnSendUniData.

Edited October 16, 2021 by pyscripter

[diaroeh 0]

Yes I did and adding event handlers on both OnSendUniData and OnSendData ...

IDE is RAD Studio 11 ...

[pyscripter 689]

Here is a super simple approach. 

SimpleDemo.zip

[pyscripter 689]

And here is a less simple way running Python code in a thread and keeping the main thread free to handle printing.  It also shows how to buffer the output.

LessSimpleDemo.zip

[diaroeh 0]

Dear pyscripter,

 

thanks for your demos, I will have a look on it tomorrow.

 

Have a nice sunday evening 

 

Dirk

[diaroeh 0]

Dear pyscripter,

thanks for your Demos, I have studied and tested them.

Even just the less simple Demo was very interesting.

I never thought that a standard TMemo could be too slow,
even I never thought that I need a TCriticalSection in a normally
not critical state, also when using VCL and not FMX there has been
no problems.
But maybe it is the price for developement on different platforms?

However, thank you very much, good to know that there is not really
a problem running python4delphi under FMX and now I know that FMX
has just to mannered in a different way.


Cheers

Dirk