The TPythonEngine instance is being terminated automatically

[djxandytche 0]

Hello,

 

I would like help again.

 

In a code I created here, the TPythonEngine instance is being terminated automatically.
But, I wish that didn't happen.
I tried assigning False to the TPythonEngine.AutoFinalize property, but it didn't help.

 

I created this small example to reproduce the case:

procedure TForm1.Button1Click(Sender: TObject);
var
    zMemTable: TFDMemTable;
    zPythonModule: TPythonModule;
    zPyDelphiWrapper: TPyDelphiWrapper;
    pyObj: PPyObject;

const
    ctPythonScript = 'import DBFireDac'                                    + sLineBreak +
                                                                             sLineBreak +
                     'dataset = DBFireDac.FDMemTable'                      + sLineBreak +
                                                                             sLineBreak +
                     'dataset.Append()'                                    + sLineBreak +
                     'dataset.FieldByName(''id'').AsInteger = 1'           + sLineBreak +
                     'dataset.FieldByName(''name'').AsString = ''name 1''' + sLineBreak +
                                                                             sLineBreak +
                     'dataset.Append()'                                    + sLineBreak +
                     'dataset.FieldByName(''id'').AsInteger = 2'           + sLineBreak +
                     'dataset.FieldByName(''name'').AsString = ''name 2''';
begin
    zMemTable := TFDMemTable.Create(nil);

    with zMemTable do
        try
            with FieldDefs.AddFieldDef do
                begin
                    Name := 'id';
                    DataType := ftInteger;
                end;

            with FieldDefs.AddFieldDef do
                begin
                    Name := 'name';
                    DataType := ftString;
                end;

            Open;

            BeginBatch;

            zPythonModule := TPythonModule.Create(nil);

            try
                with zPythonModule do
                    begin
                        Engine := GetPythonEngine;
                        ModuleName := 'DBFireDac';
                        Initialize;
                    end;

                zPyDelphiWrapper := TPyDelphiWrapper.Create(nil);

                try
                    with zPyDelphiWrapper do
                        begin
                            Engine := GetPythonEngine;
                            Module := zPythonModule;
                            Initialize;

                            pyObj := Wrap(zMemTable);
                        end;

                    zPythonModule.SetVar('FDMemTable', pyObj);

                    with GetPythonEngine do
                        begin
                            Py_XDecRef(pyObj);
                            ExecString(UTF8Encode(ctPythonScript));
                        end;
                finally
                    zPyDelphiWrapper.Free;
                end;
            finally
                zPythonModule.Free; { at this point the PythonEngine instance is finalized automatically: TPythonEngine.Finalize is called because ClientCount=0 }
            end;

            EndBatch;

            First;

            while not Eof do
                begin
                    ShowMessage(FieldByName('id').AsInteger.ToString + ': ' + FieldByName('name').AsString);

                    Next;
                end;
        finally
            Free;
        end;
end;

 

When I click the button for the first time, the code runs correctly.
When I click the button for the second time, the error below occurs:

 

Is there any way for the TPythonEngine instance not to be terminated automatically?

 

Note: The complete example project is attached.

 

Thanks,

 

Alexandre da Silva.

Project1.zip

[pyscripter 689]

Why create/destroy/recreate the PythonModule and PythonWrapper instead of reusing them?

To avoid the finalization of the PythonEngine keep at least one client alive.

[djxandytche 0]

Hello,

 

I'm using python4delphi on a REST backend developed with DataSnap.
Each incoming request creates instances of TPythonModule and TPyDelphiWrapper, executes the Python script and destroys the instances when the request thread ends.
Just the instance of PythonEnginand I create it once at the backend service startup and reuse it in requests.

 

Do you have any recommendations for this use case?
Maybe create any instance just to always have an active client? If so, which class instance would be most appropriate?
Or perhaps create a singleton for the TPythonModule and TPyDelphiWrapper classes? Wouldn't this cause problems with multi-threads since the DataSnap server can receive several requests at the same time?

 

Thank you again.

[pyscripter 689]

34 minutes ago, djxandytche said:

create a singleton for the TPythonModule and TPyDelphiWrapper classes

A single TPythonModule and TPyDelphiWrapper should do.

 

34 minutes ago, djxandytche said:

Wouldn't this cause problems with multi-threads since the DataSnap server can receive several requests at the same time?

Running python code in threads is quite complex.  See PythonThreads · pyscripter/python4delphi Wiki (github.com) for details.

[djxandytche 0]

Thanks for the tip.
I will create a singleton for the TPythonModule and TPyDelphiWrapper classes.

 

Thanks also for the document that talks about working with threads.
I had already implemented the same code here that exists in the TPythonThread class.
For each request that arrives at the DataSnap server I call this method at the beginning:

procedure InitializePythonThread;
begin
    if PythonOK then
        with GetPythonEngine do
            begin
                zGilState := PyGILState_Ensure;

                try
                    zGlobalThreadState := PyThreadState_Get;

                    try
                        PyThreadState_Swap(nil);

                        zThreadState := Py_NewInterpreter;

                        if not Assigned(zThreadState) then
                            raise EPythonError.Create( 'Could not create a new thread state');

                        PyThreadState_Swap(zThreadState);

                        zPythonThreadInitialized := True;
                    except
                        PyThreadState_Swap(zGlobalThreadState);
                        raise;
                    end;
                except
                    PyGILState_Release(zGilState);
                    raise;
                end;
            end;
end;

And I call this method at the end:

procedure FinalizePythonThread;
begin
    if zPythonThreadInitialized and PythonOK then
        with GetPythonEngine do
            begin
                Py_EndInterpreter(zThreadState);
                PyThreadState_Swap(zGlobalThreadState);
                PyGILState_Release(zGilState);
            end;
end;

Here is the complete code I created to create and destroy the global TPythonEngine instance on DataSnap server startup and termination. And the thread initialization and termination part as well:

unit UnitPythonUtils;

interface

uses
    Data.DB, Variants, System.SysUtils, System.DateUtils, Fe.Value, Vcl.StdCtrls;

procedure InitializePythonEngineIfNecessaryAndIfPossible;
procedure FinalizePythonEngine;

threadvar
    zPythonThreadInitialized: Boolean;

procedure InitializePythonThread;
procedure FinalizePythonThread;

implementation

uses
    PythonVersions, PythonEngine, UnitGeneralHelpers, System.JSON, UnitStreams,
    Data.FMTBcd, UnitStringManager, UnitDialogs, UnitFormPythonInstallation, Vcl.Controls;

var
    zMainThreadState: PPyThreadState;

threadvar
    zGilState: PyGILState_STATE;
    zThreadState: PPyThreadState;
    zGlobalThreadState: PPyThreadState;

procedure InitializePythonEngineIfNecessaryAndIfPossible;
var
    zPythonVersion: TPythonVersion;
begin
    // Primeiro verifico se o Python já está inicializado.
    if PythonOK then
        Exit;

    if GetLatestRegisteredPythonVersion(zPythonVersion) then
        with TPythonEngine.Create(nil) do
            try
                AutoLoad := False;
                zPythonVersion.AssignTo(ThisEx);
                LoadDLL;

                // Isso configura o engine do Python para permitir multithread no Delphi.
                zMainThreadState := PyEval_SaveThread;
            except
                // TODO: Será que precisa fazer um log do erro que der aqui??
                Free;
            end;
end;

procedure FinalizePythonEngine;
var
    zPythonEngine: TPythonEngine;
begin
    // Primeiro verifico se o Python está inicializado.
    // Pode ser que não foi possível inicializa-lo (se não estiver instalado por exemplo).
    if not PythonOK then
        Exit;

    zPythonEngine := GetPythonEngine;
    zPythonEngine.PyEval_RestoreThread(zMainThreadState); { será que precisa disso?? }
    FreeAndNil(zPythonEngine);
end;

procedure InitializePythonThread;
begin
    if PythonOK then
        with GetPythonEngine do
            begin
                zGilState := PyGILState_Ensure;

                try
                    zGlobalThreadState := PyThreadState_Get;

                    try
                        PyThreadState_Swap(nil);

                        zThreadState := Py_NewInterpreter;

                        if not Assigned(zThreadState) then
                            raise EPythonError.Create( 'Could not create a new thread state');

                        PyThreadState_Swap(zThreadState);

                        zPythonThreadInitialized := True;
                    except
                        PyThreadState_Swap(zGlobalThreadState);
                        raise;
                    end;
                except
                    PyGILState_Release(zGilState);
                    raise;
                end;
            end;
end;

procedure FinalizePythonThread;
begin
    if zPythonThreadInitialized and PythonOK then
        with GetPythonEngine do
            begin
                Py_EndInterpreter(zThreadState);
                PyThreadState_Swap(zGlobalThreadState);
                PyGILState_Release(zGilState);
            end;
end;

end.

This way, the DataSnap request thread will have the same as the TPythonThread.
So far everything has worked correctly with multi-threds.