Jump to content
Soulflesh

Running Python scripts in threads

Recommended Posts

Hi,

 

I have a problem running Python scripts in threads.

The execution stucks in TPythonThread.Execute --> gilstate := PyGILState_Ensure(); and never return.

Here is my example console application.

program p4dtest;

{$APPTYPE CONSOLE}
{$R *.res}

uses
  System.SysUtils,
  System.Classes,

  PythonEngine;

type
  TMyPythonThread = class(TPythonThread)
  protected
    procedure ExecuteWithPython; override;
  public
    constructor Create;
  end;

  TMyServiceHelper = class
  public
    ThreadCount : Integer;
  public
    procedure ThreadDone(Sender: TObject);
  end;

{ TMyPythonThread }

constructor TMyPythonThread.Create;
begin
  Self.ThreadExecMode  := emNewInterpreter;
  Self.FreeOnTerminate := true;

  inherited Create(true);
end;

procedure TMyPythonThread.ExecuteWithPython;
begin
  Writeln('Test1');
  GetPythonEngine.ExecString('print("Test2")');
end;

{ TMyServiceHelper }

procedure TMyServiceHelper.ThreadDone(Sender: TObject);
begin
  Dec(ThreadCount);
end;

var
  MyPythonEngine  : TPythonEngine;
  MyPyInOut       : TMyPythonGUIInputOutput;
  MySvcHelper     : TMyServiceHelper;
  Threads         : Array of TMyPythonThread;
  i               : Integer;

begin
  try
    MyPythonEngine := TPythonEngine.Create(nil);
    MySvcHelper    := TMyServiceHelper.Create;
    try
      MyPythonEngine.InitThreads := true;
      MyPythonEngine.LoadDll;

      MySvcHelper.ThreadCount := 1;
      SetLength(Threads,1);
      for i:=0 to 0 do
      begin
        Threads[i] := TMyPythonThread.Create;
        Threads[i].OnTerminate := MySvcHelper.ThreadDone;
        Threads[i].Start;
      end;

      repeat
        CheckSynchronize;
        sleep(10);
      until MySvcHelper.ThreadCount = 0;

      Writeln('Done');
      ReadLn;
    finally
      FreeAndNil(MyPythonEngine);
      FreeAndNil(MySvcHelper);
    end;
  except
    on E: Exception do
    begin
      Writeln(E.ClassName, ': ', E.Message);
      ReadLn;
    end;
  end;
end.

 

Any ideas?

Best regards.

Share this post


Link to post

You need to call PyBeginAllowThreads in the main thread. 

Share this post


Link to post

Unfortunately I cannot find anything like "PyBeginAllowThreads" or "AllowThreads" in all sources... 🤔

Share this post


Link to post
Posted (edited)
3 hours ago, Soulflesh said:

Unfortunately I cannot find anything like "PyBeginAllowThreads" or "AllowThreads" in all sources

class procedure TPythonThread.Py_Begin_Allow_Threads;

 

You also need to call Py_End_Allow_Threads when all threads finish.

Edited by pyscripter
  • Thanks 1

Share this post


Link to post

Additional question: 😀

 

I did some performance tests and unfortunately came to the expected result that normal thread processing does not bring any significant performance benefits.

For my test I used the Fibonacci algorithm to generate a high CPU load and a low IO load. 

 

Are there ways, apart from additional processes, to improve the performance of parallel processing?

I am talking here in the context of processing a wide variety of scripts or problems that cannot be parallelized themselves.

 

Is there a possibility to use Python derivatives like CPython or IronPython without or with a disableableable GIL?

 

From another area of resource sharing of dll libraries, a viable way was to duplicate the dll's and load them multiple times.

Would this also be a possible approach here with Python and Delphi? 

 

Best regards

Share this post


Link to post
Posted (edited)

Hard to answer this question in a meaningful way, since it depends a great deal on what you are trying to do.  In general though:

Edited by pyscripter

Share this post


Link to post

My topic is more in the sense of a web service, whereby it is more a matter of the overall speed and less of that of individual processes. In my view, true parallelism is the only way here.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×