Help wanted: Run Python (P4D) in a Delphi thread without blocking UI

[Tom F 85]

Goal: a standalone unit that wraps Python4Delphi so Python code runs inside a Delphi 12 Windows thread, not a Python thread. The main thread must stay responsive, and the user must be able to cancel the worker thread.
 

Workload: Python opens ~30 files sequentially, about one second per file, and prints one line per file. (i.e. a low number of stdout and perhaps stderr lines).  Less than 100 lines of py code using numpy and others. I don't need any py code written, just the Delphi piece.
 

Reference: https://github.com/pyscripter/python4delphi/wiki/PythonThreads
 

Status: I have a prototype that runs, but I am not confident about thread safety or correctness, especially around a clean thread stop, engine shutdown, and memory management. I redirect stdout and stderr via TInputOutput. Stdin is not needed. During development I've been using CodeSite or OutputDebug string. Later I will need to marshal messages safely to the main thread.
 

Ask: I need someone with prior experience doing this kind of thing. Paid consulting for this small project. If you're interested, please contact me here via DM.

Me: long-time Delphi programmer. 
 

Sample code to show the path I was on:

 

procedure TForm1.InitializeEngine; // Dynamically configure paths to a 
begin
  Codesite.Clear;
  if PythonEngine1.AutoLoad then
    raise Exception.Create('Caught: So you don''t get two instances of the engine, turn off PythonEngine.AutoLoad in the Object Inspector');
  PythonEngine1.UseLastKnownVersion := False;
  PythonEngine1.DllName    := 'python312.dll';
  PythonEngine1.DllPath    := 'C:\Program Files (x86)\MyApp\Python\Python312-32';
  PythonEngine1.SetPythonHome('C:\Program Files (x86)\MyApp\Python\Python312-32');
  try
    PythonEngine1.LoadDll;
    if not PythonEngine.PythonOK  then
      CodeSite.Send( 'Caught: PythonEngine failed to load. (Probably a DDLName or DLLPath error.' );
  except
    On E: Exception do
      CodeSite.Send( 'Caught: Error in FormCreate while loading the DLL', E.Message );
  end;
end;  

procedure TForm1.btnRunClick(Sender: TObject);
var
  CodeToRun: String;
begin
  try
    CodeToRun :=  'print("hello world!")' 
    TPythonThread.Py_Begin_Allow_Threads;
    ThreadPythonExec(
      procedure   {ExecuteProc}
      begin
          GetPythonEngine.IO := PythonInputOutput1; // do these assignments go here, or before the ThreadPythonExec
          GetPythonEngine.FatalMsgDlg :=  FALSE;
          GetPythonEngine.FatalAbort := TRUE;
          GetPythonEngine.Py_FatalError := @MyFatalMessage;
          try
            GetPythonEngine.ExecString(AnsiString(CodeToRun));
          except
            on E: Exception do
              CodeSite.Send( 'Caught: ExecString Exit value', e.Message ); // I definitely need this 
          end;
      end,
      procedure   {TerminateProc }
      begin
        CodeSite.Send( 'Thread ended.' );
        TPythonThread.Py_End_Allow_Threads; // Does this go here???
      end
    );
  except
    on E: Exception do
      CodeSite.Send( 'Caught: Exception with engine when running= ' + E.Message );
  end;
end;

 

Edited August 13 by Tom F

[pyscripter 796]

From https://github.com/pyscripter/python4delphi/wiki/PythonThreads

Quote

 

Py_Begin_Allow_Threads saves the thread state and releases the GIL, whilst Py_End_Allow_Threads tries to get hold of GIL and restore the previous thread state. You should only call Py_Begin_Allow_Threads if you alread hold the GIL. Otherwise a deadlock will occur.

You should call Py_Begin_Allow_Threads in your main thread, after the python DLL was loaded, for example in your FormCreate event handler. Also you can call Py_End_Allow_Threads when you are done executing python in threads.

 

Why do the following inside the thread?

          GetPythonEngine.IO := PythonInputOutput1; // do these assignments go here, or before the ThreadPythonExec
          GetPythonEngine.FatalMsgDlg :=  FALSE;
          GetPythonEngine.FatalAbort := TRUE;
          GetPythonEngine.Py_FatalError := @MyFatalMessage;

Edited August 15 by pyscripter

[Tom F 85]

Thanks, Kiriakos.  I really appreciate your taking the time to answer my post. Thank you.

However, I decided that P4D in a thread was too difficult to use unless I had far more knowledge of threads. So, I'm now just shelling out to Python and running my .py that way.

Hooking up PythonExec was too difficult for me to get working.  And I heard from several other members of our group that they, too, had abandoned trying to use it for the same reason.

I encourage you to create a demo that is a bare-bones "hello, world" skeleton for using PythonExec, including instantiating it (perhaps not on a form, but DM or non-form unit),  launching it, receiving stdout/stderr, getting exit codes, and (particularly vexing for me) proper shutdown and cleanup.   (Like most overly optimistic programmers, I think I was probably only a handful of lines from getting all this  going with PythonExec.)

I think such a demo would be very welcome by many.  And it would ease the startup process a lot. 

Thank you again for your response. And thank you for your contributions to our community.

Tom