djxandytche

Hello, First of all I'd like to congratulate you on the excellent work you've done on P4D. Secondly, I want to apologize for my bad English (I'm Brazilian and I'm using Google Translate to help). I'm studying and testing P4D and I came across a situation that I couldn't understand. The situation is a memory leak that occurs whenever the "ExecString" method is called to execute any script. To simulate it, just do this simple example: procedure TForm1.Button1Click(Sender: TObject); var i: Integer; begin for i := 1 to 1000 do GetPythonEngine.ExecString('print("hello")'); end; I'm using Delphi 11.1 and compiling for 64-bit Windows. For every 1000 scripts executed, memory consumption increases by about 0.5 MB. In software that runs 100000 scripts per day, memory consumption could increase by around 50 MB per day. I intend to use P4D in the software of the company I work for and in most cases it will exceed 100000 scripts executed per day and the executable will be running 24x7. Considering this, I believe that at some point the memory may reach its limit. Is there anything I can do to free up the memory consumed by each call to the "ExecString" method? Thanks, Alexandre da Silva.

Hello, I work on developing software that uses DataSnap. On the client I am creating this simple Python script: import pandas as pd print('done') Just to illustrate, here is the screen where I enter the script: This script is submitted to the DataSnap server to be executed in Python. Just to illustrate, this is the snippet of code that executes it: When the line "GetPythonEngine.ExecString(UTF8Encode(sFormula));" is executed, the program freezes. And it just sits there without doing anything (there is no processor consumption). You need to end the process to end execution. The problem only occurs if the line "import pandas as pd" exists. As the DataSnap server receives multiple requests (multi-thread), I am following the recommendations/rules for working with multi-threads with Python/P4D. Other scripts, including more complex ones, work normally, as long as the "import pandas as pd" line does not exist. I did the same test in Visual Studio Code and it works normally: I tried to reproduce this problem outside of my DataSnap project, but I was unable to do so. When doing this in a simple example project (even within a thread), it works normally. Therefore, I cannot send an example project so that the problem can be reproduced there. I searched the posts here for a problem similar to this, but I didn't find anything. Is it possible to help me with this?

pyscripter, I made the adjustments you suggested and now it is working correctly. Thank you for your help. Thank you for recommending Lucas Belo. Alexandre.

I'll try to explain better... Imagine you have the thread below: TPythonThread_Test1 = class(TPythonThread) protected procedure ExecuteWithPython; override; public fOutputs: TStringList; Script: string; end; procedure TPythonThread_Test1.ExecuteWithPython; var i: Integer; zPythonDelphiVar: TPythonDelphiVar; begin fOutputs := TStringList.Create; zPythonDelphiVar := TPythonDelphiVar.Create(nil); try zPythonDelphiVar.Engine := GetPythonEngine; zPythonDelphiVar.VarName := 'result'; zPythonDelphiVar.Initialize; for i := 1 to 10000 do begin GetPythonEngine.ExecString(UTF8Encode(Script)); fOutputs.Add(zPythonDelphiVar.Value); end; finally zPythonDelphiVar.Free; end; end; Now you create 10 instances of this thread and assign the following values to the "Script" property of each of them. For example: Thread 0: Script = result.value = "thread 0" Thread 1: Script = result.value = "thread 1" Thread 2: Script = result.value = "thread 2" Thread 3: Script = result.value = "thread 3" Thread 4: Script = result.value = "thread 4" Thread 5: Script = result.value = "thread 5" Thread 6: Script = result.value = "thread 6" Thread 7: Script = result.value = "thread 7" Thread 8: Script = result.value = "thread 8" Thread 9: Script = result.value = "thread 9" Thread 10: Script = result.value = "thread 10" Then you start running all instances at the same time. This will cause each instance to execute the same script 10000 times and assign the result in the "fOutputs" property. In other words, "Thread 0", for example, will add 10000 times the string "thread 0" in the "fOutputs" list. But, some executions of some threads are interfering with the execution of other threads. For example, sometimes the value "thread 2" appears in the "fOutputs" list of "Thread 5". This shouldn't happen. Here is a screenshot of the example project I created that reproduces this problem: I hope I was clear enough. Thanks, Alexandre.

pyscripter, My intention was to show something that seems to me to be a problem in p4d. And sending this example was the best way I found. Answering your questions: 1 (why do you start by running python code in threads): Because I want to use p4d in the backend of the company's system for which I work and this backend is a Delphi DataSnap REST server that runs on Apache, therefore, it receives multiple requests (multi- thread); 2 (there is no performance benefits in running python code in threads): Yes, I started to understand this in the last few weeks that I have been studying and working on integrating p4d into my system; My plan B is to create a single point to execute Python scripts in a serialized way (using Delphi's lock-unlock like TMonitor). But first I would like to try to make it work using threads. Just to give more context: My company develops a BI system (similar to the idea of Microsoft Power BI) and we intend to allow the user to create a data source based on a Python script (in addition to being able to connect to various databases) . Company website: https://weknowhealthtech.com.br/. We are from Brazil. Note: I know that p4d is free and open source and therefore no one is obliged to help me, but the company I work for would be willing to hire some type of paid help/consultancy so that we can continue with the p4d integration project into our system. Grateful, Alexandre.

pyscripter, I only read the docs from the p4d Git repository. These here: https://github.com/pyscripter/python4delphi/wiki. I hadn't read this one yet: https://docs.python.org/3/c-api/init.html#c.Py_NewInterpreterFromConfig. But, I read it now. However, I was unable to get a complete understanding. I'm a beginner at Python. Well, I'm going to forget the "emNewInterpreter" option from now on. I've done some testing now with the "emNewState" option and found a behavior that seems like a problem, but I would like to ask you if this is really a problem. It could be that it's something I did wrong in my code because I didn't read or didn't correctly understand some of the existing documentation. I found something that seems to be a memory sharing of Python script variables between threads (and I think that's a problem). I created a small example project that reproduces this case. The project is attached or can be downloaded here: https://drive.google.com/file/d/1KFqygNb0MKJODgqcYaFQ8Cny5x9tu0xi/view?usp=sharing Brief explanation of the example project: It creates 10 threads. Each thread executes the same script 10000 times. For example: thread 1 has the script "result.value = 1", thread 2 has the script "resulta.value = 2", and so on. When all threads finish executing, I check the returns from script executions. If I find returns in a thread that are from another thread, I generate a LOG line in a TMemo. Could you check and help me with this case? Note: If you think it's better to create a new topic to avoid mixing topics, let me know and I'll do it. Thanks, Alexandre. testepythonmultithread1.zip

pyscripter, got it, thanks. I found another problem (at least I think it's a problem) when using "TTask" + "SafePyEngine", but, later I report it in a new topic. Now I would like to ask for help again with the "hanging" problem when using "import pandas as pd" in a script that runs within a Delphi thread. I made a new example project, even simpler, based on the demos and managed to reproduce the "hanging" problem. The new project is attached and can be downloaded from this link: https://drive.google.com/file/d/1xdv8NDK31wao3EjXzuschispAnzdDqNs/view?usp=sharing To reproduce the problem, simply select the "emNewInterpreter" option and click the "Button1" button, as shown in the image below: If you select the "emNewState" option the problem does not occur. Could you check if there is still a problem in my code that I am not seeing/noticing? Thanks in advance. Alexandre. travamentopandas3.zip

Hi pyscripter, Thanks for the guidance. I had already read this documentation and had also seen the demos, but now I have reviewed everything again and I think I managed to improve my understanding. After reviewing everything, I adjusted the code and then resolved the problem. I believe that my code is now correct (if there is still something to be improved, I will be grateful if you let me know). The new project with the corrected code is attached or can be downloaded from the link below: https://drive.google.com/file/d/18LFBKkMv5v0Q7ZzWUb7a1CZVWh5DqmED/view?usp=sharing But, I had a question: In my case, where I am using Delphi's "TTask" class and p4d's "SafePyEngine" function, how do I set p4d's "TThreadExecMode (emNewState, emNewInterpreter, emNewInterpreterOwnGIL)"? I saw that in the p4d "TPythonThread" class it is possible to choose "TThreadExecMode". But, I couldn't figure out how to do this when I use "TTask" + "SafePyEngine". Thank you in advance for your help, Alexandre. travamentopandas2.zip

Hi limelect, Thanks for trying. Unfortunately it didn't help in my case. But, I discovered that the problem only occurs if I run the Python script within a Delphi thread and I was able to reproduce the problem in a simple example project. The complete example project is attached or can be downloaded from the link below: https://drive.google.com/file/d/1F0AbuU5TFTzP2wzdampzLi8gSPQRiTvp/view?usp=sharing To see the problem occur, just run "Project1.exe". Execution will be stuck at the line "GetPythonEngine.ExecString(UTF8Encode(sScript));". If you remove the "import pandas as pd" line from the Python script to be executed, then it works normally. Delphi version: 11.1. Python version: 3.11.4. Pandas library version: 2.0.3. p4d version: I put the source code along with the example project. If anyone can find out the reason for the crash and know how to resolve it, I would be grateful. travamentopandas.zip

Hello, I would like help again. In a code I created here, the TPythonEngine instance is being terminated automatically. But, I wish that didn't happen. I tried assigning False to the TPythonEngine.AutoFinalize property, but it didn't help. I created this small example to reproduce the case: procedure TForm1.Button1Click(Sender: TObject); var zMemTable: TFDMemTable; zPythonModule: TPythonModule; zPyDelphiWrapper: TPyDelphiWrapper; pyObj: PPyObject; const ctPythonScript = 'import DBFireDac' + sLineBreak + sLineBreak + 'dataset = DBFireDac.FDMemTable' + sLineBreak + sLineBreak + 'dataset.Append()' + sLineBreak + 'dataset.FieldByName(''id'').AsInteger = 1' + sLineBreak + 'dataset.FieldByName(''name'').AsString = ''name 1''' + sLineBreak + sLineBreak + 'dataset.Append()' + sLineBreak + 'dataset.FieldByName(''id'').AsInteger = 2' + sLineBreak + 'dataset.FieldByName(''name'').AsString = ''name 2'''; begin zMemTable := TFDMemTable.Create(nil); with zMemTable do try with FieldDefs.AddFieldDef do begin Name := 'id'; DataType := ftInteger; end; with FieldDefs.AddFieldDef do begin Name := 'name'; DataType := ftString; end; Open; BeginBatch; zPythonModule := TPythonModule.Create(nil); try with zPythonModule do begin Engine := GetPythonEngine; ModuleName := 'DBFireDac'; Initialize; end; zPyDelphiWrapper := TPyDelphiWrapper.Create(nil); try with zPyDelphiWrapper do begin Engine := GetPythonEngine; Module := zPythonModule; Initialize; pyObj := Wrap(zMemTable); end; zPythonModule.SetVar('FDMemTable', pyObj); with GetPythonEngine do begin Py_XDecRef(pyObj); ExecString(UTF8Encode(ctPythonScript)); end; finally zPyDelphiWrapper.Free; end; finally zPythonModule.Free; { at this point the PythonEngine instance is finalized automatically: TPythonEngine.Finalize is called because ClientCount=0 } end; EndBatch; First; while not Eof do begin ShowMessage(FieldByName('id').AsInteger.ToString + ': ' + FieldByName('name').AsString); Next; end; finally Free; end; end; When I click the button for the first time, the code runs correctly. When I click the button for the second time, the error below occurs: Is there any way for the TPythonEngine instance not to be terminated automatically? Note: The complete example project is attached. Thanks, Alexandre da Silva. Project1.zip

Thanks for the tip. I will create a singleton for the TPythonModule and TPyDelphiWrapper classes. Thanks also for the document that talks about working with threads. I had already implemented the same code here that exists in the TPythonThread class. For each request that arrives at the DataSnap server I call this method at the beginning: procedure InitializePythonThread; begin if PythonOK then with GetPythonEngine do begin zGilState := PyGILState_Ensure; try zGlobalThreadState := PyThreadState_Get; try PyThreadState_Swap(nil); zThreadState := Py_NewInterpreter; if not Assigned(zThreadState) then raise EPythonError.Create( 'Could not create a new thread state'); PyThreadState_Swap(zThreadState); zPythonThreadInitialized := True; except PyThreadState_Swap(zGlobalThreadState); raise; end; except PyGILState_Release(zGilState); raise; end; end; end; And I call this method at the end: procedure FinalizePythonThread; begin if zPythonThreadInitialized and PythonOK then with GetPythonEngine do begin Py_EndInterpreter(zThreadState); PyThreadState_Swap(zGlobalThreadState); PyGILState_Release(zGilState); end; end; Here is the complete code I created to create and destroy the global TPythonEngine instance on DataSnap server startup and termination. And the thread initialization and termination part as well: unit UnitPythonUtils; interface uses Data.DB, Variants, System.SysUtils, System.DateUtils, Fe.Value, Vcl.StdCtrls; procedure InitializePythonEngineIfNecessaryAndIfPossible; procedure FinalizePythonEngine; threadvar zPythonThreadInitialized: Boolean; procedure InitializePythonThread; procedure FinalizePythonThread; implementation uses PythonVersions, PythonEngine, UnitGeneralHelpers, System.JSON, UnitStreams, Data.FMTBcd, UnitStringManager, UnitDialogs, UnitFormPythonInstallation, Vcl.Controls; var zMainThreadState: PPyThreadState; threadvar zGilState: PyGILState_STATE; zThreadState: PPyThreadState; zGlobalThreadState: PPyThreadState; procedure InitializePythonEngineIfNecessaryAndIfPossible; var zPythonVersion: TPythonVersion; begin // Primeiro verifico se o Python já está inicializado. if PythonOK then Exit; if GetLatestRegisteredPythonVersion(zPythonVersion) then with TPythonEngine.Create(nil) do try AutoLoad := False; zPythonVersion.AssignTo(ThisEx); LoadDLL; // Isso configura o engine do Python para permitir multithread no Delphi. zMainThreadState := PyEval_SaveThread; except // TODO: Será que precisa fazer um log do erro que der aqui?? Free; end; end; procedure FinalizePythonEngine; var zPythonEngine: TPythonEngine; begin // Primeiro verifico se o Python está inicializado. // Pode ser que não foi possível inicializa-lo (se não estiver instalado por exemplo). if not PythonOK then Exit; zPythonEngine := GetPythonEngine; zPythonEngine.PyEval_RestoreThread(zMainThreadState); { será que precisa disso?? } FreeAndNil(zPythonEngine); end; procedure InitializePythonThread; begin if PythonOK then with GetPythonEngine do begin zGilState := PyGILState_Ensure; try zGlobalThreadState := PyThreadState_Get; try PyThreadState_Swap(nil); zThreadState := Py_NewInterpreter; if not Assigned(zThreadState) then raise EPythonError.Create( 'Could not create a new thread state'); PyThreadState_Swap(zThreadState); zPythonThreadInitialized := True; except PyThreadState_Swap(zGlobalThreadState); raise; end; except PyGILState_Release(zGilState); raise; end; end; end; procedure FinalizePythonThread; begin if zPythonThreadInitialized and PythonOK then with GetPythonEngine do begin Py_EndInterpreter(zThreadState); PyThreadState_Swap(zGlobalThreadState); PyGILState_Release(zGilState); end; end; end. This way, the DataSnap request thread will have the same as the TPythonThread. So far everything has worked correctly with multi-threds.

Hello, I'm using python4delphi on a REST backend developed with DataSnap. Each incoming request creates instances of TPythonModule and TPyDelphiWrapper, executes the Python script and destroys the instances when the request thread ends. Just the instance of PythonEnginand I create it once at the backend service startup and reuse it in requests. Do you have any recommendations for this use case? Maybe create any instance just to always have an active client? If so, which class instance would be most appropriate? Or perhaps create a singleton for the TPythonModule and TPyDelphiWrapper classes? Wouldn't this cause problems with multi-threads since the DataSnap server can receive several requests at the same time? Thank you again.

That was it! I updated, tested and now the memory leak no longer occurs. Thank you for your help!

Hi, Do you mean capturing the output via "TPythonGUIInputOutput"? If so: I'm not even using it. I've made the code as simple as possible. See below: Unit1.pas: unit Unit1; interface uses Winapi.Windows, Winapi.Messages, System.SysUtils, System.Variants, System.Classes, Vcl.Graphics, Vcl.Controls, Vcl.Forms, Vcl.Dialogs, Vcl.StdCtrls, PythonEngine; type TForm1 = class(TForm) PythonEngine1: TPythonEngine; Memo1: TMemo; Button1: TButton; Label1: TLabel; procedure Button1Click(Sender: TObject); private { Private declarations } public { Public declarations } end; var Form1: TForm1; implementation {$R *.dfm} procedure TForm1.Button1Click(Sender: TObject); var i: Integer; begin for i := 1 to 1000 do GetPythonEngine.ExecString('print("hello")'); end; end. Unit1.dfm: object Form1: TForm1 Left = 0 Top = 0 Caption = 'Form1' ClientHeight = 234 ClientWidth = 351 Color = clBtnFace Font.Charset = DEFAULT_CHARSET Font.Color = clWindowText Font.Height = -12 Font.Name = 'Segoe UI' Font.Style = [] TextHeight = 15 object Label1: TLabel Left = 8 Top = 8 Width = 73 Height = 15 Caption = 'Python script:' end object Memo1: TMemo Left = 8 Top = 29 Width = 185 Height = 89 Lines.Strings = ( 'print("hello")') TabOrder = 0 end object Button1: TButton Left = 8 Top = 124 Width = 185 Height = 25 Caption = 'Run 1000 times' TabOrder = 1 OnClick = Button1Click end object PythonEngine1: TPythonEngine Left = 112 Top = 48 end end Is there something else I could be doing wrong?