Soulflesh

Hi uefi,

have a look at Running Python scripts in threads - Python4Delphi - Delphi-PRAXiS [en] (delphipraxis.net).

Unfortunately, this still does not work in the form of true parallelism at the moment. Only the pseudo Python thread parallelism. From my point of view, there is currently only the possibility/attempt to multiply the Python dlls and load them per thread.

Bets regards

Hi FabDev,

In my estimation, this is not a good idea due to the lack of parallelisation with Python. We also use the TMS Scripter for web applications (f.e. https://help.clarc-software.de/display/CD7/WEB+VALIDATION+Arbeitsbereich). Here, however, you also reach the limits very quickly as soon as you use the generic classes in the scripter. These require a lot of performance and about 1MB per object instance. We have therefore transferred a lot to the client/browser. We were able to achieve acceptable results with this. 

Best regards

3 hours ago, hschmid67 said:

Hello,

 

I'm looking for something similar - implementing python scripts in webservice procedures. Were You able to find a solution for this and would You mind to share it?

 

Regards

Harald

Hello Harald,

Unfortunately, I have not yet found a way to implement this. 
I think at the moment the only possibility would be, as I wrote at the time, to multiply the dlls and then load them several times so that you have an independent dll environment for each thread.  

However, this would probably mean quite a lot of effort to rewrite this in the core libraries.

Many greetings

My topic is more in the sense of a web service, whereby it is more a matter of the overall speed and less of that of individual processes. In my view, true parallelism is the only way here.

Additional question: 😀

I did some performance tests and unfortunately came to the expected result that normal thread processing does not bring any significant performance benefits.

For my test I used the Fibonacci algorithm to generate a high CPU load and a low IO load. 

Are there ways, apart from additional processes, to improve the performance of parallel processing?

I am talking here in the context of processing a wide variety of scripts or problems that cannot be parallelized themselves.

Is there a possibility to use Python derivatives like CPython or IronPython without or with a disableableable GIL?

From another area of resource sharing of dll libraries, a viable way was to duplicate the dll's and load them multiple times.

Would this also be a possible approach here with Python and Delphi? 

Best regards

Thank you very much. That solves my issue!

Unfortunately I cannot find anything like "PyBeginAllowThreads" or "AllowThreads" in all sources... 🤔

Hi,

I have a problem running Python scripts in threads.

The execution stucks in TPythonThread.Execute --> gilstate := PyGILState_Ensure(); and never return.

Here is my example console application.

program p4dtest;

{$APPTYPE CONSOLE}
{$R *.res}

uses
  System.SysUtils,
  System.Classes,

  PythonEngine;

type
  TMyPythonThread = class(TPythonThread)
  protected
    procedure ExecuteWithPython; override;
  public
    constructor Create;
  end;

  TMyServiceHelper = class
  public
    ThreadCount : Integer;
  public
    procedure ThreadDone(Sender: TObject);
  end;

{ TMyPythonThread }

constructor TMyPythonThread.Create;
begin
  Self.ThreadExecMode  := emNewInterpreter;
  Self.FreeOnTerminate := true;

  inherited Create(true);
end;

procedure TMyPythonThread.ExecuteWithPython;
begin
  Writeln('Test1');
  GetPythonEngine.ExecString('print("Test2")');
end;

{ TMyServiceHelper }

procedure TMyServiceHelper.ThreadDone(Sender: TObject);
begin
  Dec(ThreadCount);
end;

var
  MyPythonEngine  : TPythonEngine;
  MyPyInOut       : TMyPythonGUIInputOutput;
  MySvcHelper     : TMyServiceHelper;
  Threads         : Array of TMyPythonThread;
  i               : Integer;

begin
  try
    MyPythonEngine := TPythonEngine.Create(nil);
    MySvcHelper    := TMyServiceHelper.Create;
    try
      MyPythonEngine.InitThreads := true;
      MyPythonEngine.LoadDll;

      MySvcHelper.ThreadCount := 1;
      SetLength(Threads,1);
      for i:=0 to 0 do
      begin
        Threads[i] := TMyPythonThread.Create;
        Threads[i].OnTerminate := MySvcHelper.ThreadDone;
        Threads[i].Start;
      end;

      repeat
        CheckSynchronize;
        sleep(10);
      until MySvcHelper.ThreadCount = 0;

      Writeln('Done');
      ReadLn;
    finally
      FreeAndNil(MyPythonEngine);
      FreeAndNil(MySvcHelper);
    end;
  except
    on E: Exception do
    begin
      Writeln(E.ClassName, ': ', E.Message);
      ReadLn;
    end;
  end;
end.

Any ideas?

Best regards.

Thank you!

Hi pyscripter,

Unfortunately a wrong character has crept into the code in the PythonEngine.pas in line 4396 for the "I":

Best regards

This raises an exception:

SetLength(Test,0);
Result := UCS4StringToUnicodeString(Test);

Your suggested code has solved the problem! 😀

Python Scripts runs now on Linux (Ubuntu 18.04.5 LTS with Python 3.9.6).

Best regards

The exception is raised in the UCS4StringToUnicodeString function.

Called by function  TPythonEngine.GetProgramName: UnicodeString;

And FProgramName is "{...}" ??!

Unfortunately now I get a "range check" exception here:

[PythonEngine.pas / Line 4219]

if Assigned(Py_SetProgramName) and (ProgramName <> '') then
    Py_SetProgramName(PWCharT(FProgramName));

My "configuration" is:

PythonEngine.UseLastKnownVersion := false;
PythonEngine.DllPath := '/usr/lib/x86_64-linux-gnu';
PythonEngine.DllName := 'libpython3.9.so.1.0';

Any ideas?

Best regards

Oh yes!! Sorry.

Hi pyscripter,

Thank you. I have additionally changed the following lines and now I'm able to compile:

199 --> PPWCharT = ^PUCS4Char;

4396 --> WL := UnicodeStringToUCS4String(TempS);

Best regards

Hi,

unfortunately it's not possible to compile PythonEngine.pas for Linux64 with Delphi 10.4.

[DCC Fehler] PythonEngine.pas(198): E2003 Undeklarierter Bezeichner: 'PUCSChar'
[DCC Fehler] PythonEngine.pas(199): E2007 Konstante oder Typenbezeichner erwartet
[DCC Fehler] PythonEngine.pas(4396): E2010 Inkompatible Typen: 'Dynamic array' und 'UCS4String'

Any ideas?

Best regards