Help wanted: Run Python (P4D) in a Delphi thread without blocking UI

[Tom F 85]

Goal: a standalone unit that wraps Python4Delphi so Python code runs inside a Delphi 12 Windows thread, not a Python thread. The main thread must stay responsive, and the user must be able to cancel the worker thread.
 

Workload: Python opens ~30 files sequentially, about one second per file, and prints one line per file. (i.e. a low number of stdout and perhaps stderr lines).  Less than 100 lines of py code using numpy and others. I don't need any py code written, just the Delphi piece.
 

Reference: https://github.com/pyscripter/python4delphi/wiki/PythonThreads
 

Status: I have a prototype that runs, but I am not confident about thread safety or correctness, especially around a clean thread stop, engine shutdown, and memory management. I redirect stdout and stderr via TInputOutput. Stdin is not needed. During development I've been using CodeSite or OutputDebug string. Later I will need to marshal messages safely to the main thread.
 

Ask: I need someone with prior experience doing this kind of thing. Paid consulting for this small project. If you're interested, please contact me here via DM.

Me: long-time Delphi programmer. 
 

Sample code to show the path I was on:

 

procedure TForm1.InitializeEngine; // Dynamically configure paths to a 
begin
  Codesite.Clear;
  if PythonEngine1.AutoLoad then
    raise Exception.Create('Caught: So you don''t get two instances of the engine, turn off PythonEngine.AutoLoad in the Object Inspector');
  PythonEngine1.UseLastKnownVersion := False;
  PythonEngine1.DllName    := 'python312.dll';
  PythonEngine1.DllPath    := 'C:\Program Files (x86)\MyApp\Python\Python312-32';
  PythonEngine1.SetPythonHome('C:\Program Files (x86)\MyApp\Python\Python312-32');
  try
    PythonEngine1.LoadDll;
    if not PythonEngine.PythonOK  then
      CodeSite.Send( 'Caught: PythonEngine failed to load. (Probably a DDLName or DLLPath error.' );
  except
    On E: Exception do
      CodeSite.Send( 'Caught: Error in FormCreate while loading the DLL', E.Message );
  end;
end;  

procedure TForm1.btnRunClick(Sender: TObject);
var
  CodeToRun: String;
begin
  try
    CodeToRun :=  'print("hello world!")' 
    TPythonThread.Py_Begin_Allow_Threads;
    ThreadPythonExec(
      procedure   {ExecuteProc}
      begin
          GetPythonEngine.IO := PythonInputOutput1; // do these assignments go here, or before the ThreadPythonExec
          GetPythonEngine.FatalMsgDlg :=  FALSE;
          GetPythonEngine.FatalAbort := TRUE;
          GetPythonEngine.Py_FatalError := @MyFatalMessage;
          try
            GetPythonEngine.ExecString(AnsiString(CodeToRun));
          except
            on E: Exception do
              CodeSite.Send( 'Caught: ExecString Exit value', e.Message ); // I definitely need this 
          end;
      end,
      procedure   {TerminateProc }
      begin
        CodeSite.Send( 'Thread ended.' );
        TPythonThread.Py_End_Allow_Threads; // Does this go here???
      end
    );
  except
    on E: Exception do
      CodeSite.Send( 'Caught: Exception with engine when running= ' + E.Message );
  end;
end;

 

Edited August 13 by Tom F

[pyscripter 829]

From https://github.com/pyscripter/python4delphi/wiki/PythonThreads

Quote

 

Py_Begin_Allow_Threads saves the thread state and releases the GIL, whilst Py_End_Allow_Threads tries to get hold of GIL and restore the previous thread state. You should only call Py_Begin_Allow_Threads if you alread hold the GIL. Otherwise a deadlock will occur.

You should call Py_Begin_Allow_Threads in your main thread, after the python DLL was loaded, for example in your FormCreate event handler. Also you can call Py_End_Allow_Threads when you are done executing python in threads.

 

Why do the following inside the thread?

          GetPythonEngine.IO := PythonInputOutput1; // do these assignments go here, or before the ThreadPythonExec
          GetPythonEngine.FatalMsgDlg :=  FALSE;
          GetPythonEngine.FatalAbort := TRUE;
          GetPythonEngine.Py_FatalError := @MyFatalMessage;

Edited August 15 by pyscripter

[Tom F 85]

Thanks, Kiriakos.  I really appreciate your taking the time to answer my post. Thank you.

However, I decided that P4D in a thread was too difficult to use unless I had far more knowledge of threads. So, I'm now just shelling out to Python and running my .py that way.

Hooking up PythonExec was too difficult for me to get working.  And I heard from several other members of our group that they, too, had abandoned trying to use it for the same reason.

I encourage you to create a demo that is a bare-bones "hello, world" skeleton for using PythonExec, including instantiating it (perhaps not on a form, but DM or non-form unit),  launching it, receiving stdout/stderr, getting exit codes, and (particularly vexing for me) proper shutdown and cleanup.   (Like most overly optimistic programmers, I think I was probably only a handful of lines from getting all this  going with PythonExec.)

I think such a demo would be very welcome by many.  And it would ease the startup process a lot. 

Thank you again for your response. And thank you for your contributions to our community.

Tom

[Celebr0 3]

On 8/19/2025 at 4:17 AM, Tom F said:

Thanks, Kiriakos.  I really appreciate your taking the time to answer my post. Thank you.

However, I decided that P4D in a thread was too difficult to use unless I had far more knowledge of threads. So, I'm now just shelling out to Python and running my .py that way.

Hooking up PythonExec was too difficult for me to get working.  And I heard from several other members of our group that they, too, had abandoned trying to use it for the same reason.

I encourage you to create a demo that is a bare-bones "hello, world" skeleton for using PythonExec, including instantiating it (perhaps not on a form, but DM or non-form unit),  launching it, receiving stdout/stderr, getting exit codes, and (particularly vexing for me) proper shutdown and cleanup.   (Like most overly optimistic programmers, I think I was probably only a handful of lines from getting all this  going with PythonExec.)

I think such a demo would be very welcome by many.  And it would ease the startup process a lot. 

Thank you again for your response. And thank you for your contributions to our community.

Tom

Hi, I encountered the same problem. I need to run a ready-made .py script in several windows/threads simultaneously, while also setting launch parameters and reading input/output for further parsing. I've been struggling for 2 years now and can't do it. Probably the Python4Delphi library only works in a single thread.

I also tried to make my own simplified module for working with the library:

unit pythonmodule;

interface

uses
  System.SysUtils, System.IOUtils, System.Types, System.StrUtils, System.Classes, PythonEngine;

procedure InitPython(const Data: TSendUniDataEvent; const Path: string);
procedure ExecPythonFile(const FFile, Params:string);
procedure FreePython;

function CommandLineToArgvW(lpCmdLine: PWideChar; out pNumArgs: Integer): PPWideChar; stdcall; external 'shell32.dll';
function LocalFree(hMem: Pointer): Pointer; stdcall; external 'kernel32.dll';

var
PythonEngine: TPythonEngine;
script: string='';

implementation

procedure InitPython(const Data: TSendUniDataEvent; const Path: string);
var
pyio:TPythonInputOutput;
begin
    script:=TFile.ReadAllText(Path + '\script.py');
    pyio := TPythonInputOutput.Create(nil);
    pyio.UnicodeIO := True;
    pyio.DelayWrites := False;
    pyio.OnSendUniData  := Data;
    PythonEngine := TPythonEngine.Create(nil);
    PythonEngine.IO := pyio;
  try
    PythonEngine.UseLastKnownVersion := False;
    PythonEngine.DllPath := ExtractFilePath(ParamStr(0)) + 'python';
    PythonEngine.DllName := 'python314.dll';
    PythonEngine.AutoLoad := False;
    PythonEngine.AutoFinalize := False;
    PythonEngine.AutoUnload := False;
    PythonEngine.RedirectIO := True;
    PythonEngine.UseWindowsConsole:=False;
    PythonEngine.InitScript.Text:='import sys'#13#10+'sys.path.append("'+Path+'")';
    PythonEngine.LoadDll;
    TPythonThread.Py_Begin_Allow_Threads;
  finally
//
  end;
end;

procedure ExecPythonFile(const FFile, Params:string);
const
Arg =
'import sys, shlex' + #13#10 +
'sys.argv = ["script.py"] + shlex.split("';
var
ArgCount: integer;
ArgList: PPWideChar;
config: PyConfig;
FGILState: PyGILstate_STATE;
begin
//ArgList:=CommandLineToArgvW(PWideChar(FFile+' '+Params), ArgCount);
//FGILState := PythonEngine.PyGILState_Ensure;
try
PythonEngine.ExecString(Arg + StringReplace(Params, '"', '\"', [rfReplaceAll]) + '")');
//PythonEngine.ExecStrings(TStrings(script));
PythonEngine.ExecFile(FFile);
finally
//PythonEngine.PyGILState_Release(FGILState);
end;
end;

procedure FreePython;
begin
TPythonThread.Py_End_Allow_Threads;
FreeAndNil(PythonEngine);
end;

end.

I'm trying:

procedure TForm2.OnData(Sender: TObject; const Data: string);
begin
TPythonThread.Synchronize(nil,
procedure
begin
Form2.Memo1.Lines.Add(Data);
end);
end;

procedure TForm2.Button2Click(Sender: TObject);
var
pt:TPythonThread;
begin
home:=ExtractFilePath(ParamStr(0))+'myscript';
InitPython(OnData, home);
pt.CreateAnonymousThread(
procedure
begin
pt.ThreadExecMode:=emNewInterpreterOwnGIL;
ExecPythonFile(home + '\script.py', '-n -m -o "result.txt"');
end).Start;
end;
end;

 

Edited 1 hour ago by Celebr0