Help wanted: Run Python (P4D) in a Delphi thread without blocking UI

[Tom F 85]

Goal: a standalone unit that wraps Python4Delphi so Python code runs inside a Delphi 12 Windows thread, not a Python thread. The main thread must stay responsive, and the user must be able to cancel the worker thread.
 

Workload: Python opens ~30 files sequentially, about one second per file, and prints one line per file. (i.e. a low number of stdout and perhaps stderr lines).  Less than 100 lines of py code using numpy and others. I don't need any py code written, just the Delphi piece.
 

Reference: https://github.com/pyscripter/python4delphi/wiki/PythonThreads
 

Status: I have a prototype that runs, but I am not confident about thread safety or correctness, especially around a clean thread stop, engine shutdown, and memory management. I redirect stdout and stderr via TInputOutput. Stdin is not needed. During development I've been using CodeSite or OutputDebug string. Later I will need to marshal messages safely to the main thread.
 

Ask: I need someone with prior experience doing this kind of thing. Paid consulting for this small project. If you're interested, please contact me here via DM.

Me: long-time Delphi programmer. 
 

Sample code to show the path I was on:

 

procedure TForm1.InitializeEngine; // Dynamically configure paths to a 
begin
  Codesite.Clear;
  if PythonEngine1.AutoLoad then
    raise Exception.Create('Caught: So you don''t get two instances of the engine, turn off PythonEngine.AutoLoad in the Object Inspector');
  PythonEngine1.UseLastKnownVersion := False;
  PythonEngine1.DllName    := 'python312.dll';
  PythonEngine1.DllPath    := 'C:\Program Files (x86)\MyApp\Python\Python312-32';
  PythonEngine1.SetPythonHome('C:\Program Files (x86)\MyApp\Python\Python312-32');
  try
    PythonEngine1.LoadDll;
    if not PythonEngine.PythonOK  then
      CodeSite.Send( 'Caught: PythonEngine failed to load. (Probably a DDLName or DLLPath error.' );
  except
    On E: Exception do
      CodeSite.Send( 'Caught: Error in FormCreate while loading the DLL', E.Message );
  end;
end;  

procedure TForm1.btnRunClick(Sender: TObject);
var
  CodeToRun: String;
begin
  try
    CodeToRun :=  'print("hello world!")' 
    TPythonThread.Py_Begin_Allow_Threads;
    ThreadPythonExec(
      procedure   {ExecuteProc}
      begin
          GetPythonEngine.IO := PythonInputOutput1; // do these assignments go here, or before the ThreadPythonExec
          GetPythonEngine.FatalMsgDlg :=  FALSE;
          GetPythonEngine.FatalAbort := TRUE;
          GetPythonEngine.Py_FatalError := @MyFatalMessage;
          try
            GetPythonEngine.ExecString(AnsiString(CodeToRun));
          except
            on E: Exception do
              CodeSite.Send( 'Caught: ExecString Exit value', e.Message ); // I definitely need this 
          end;
      end,
      procedure   {TerminateProc }
      begin
        CodeSite.Send( 'Thread ended.' );
        TPythonThread.Py_End_Allow_Threads; // Does this go here???
      end
    );
  except
    on E: Exception do
      CodeSite.Send( 'Caught: Exception with engine when running= ' + E.Message );
  end;
end;

 

Edited 3 hours ago by Tom F