Python freezes when running a certain script

[djxandytche 0]

Hello,

 

I work on developing software that uses DataSnap.

On the client I am creating this simple Python script:

import pandas as pd
print('done')

Just to illustrate, here is the screen where I enter the script:

 

This script is submitted to the DataSnap server to be executed in Python.
Just to illustrate, this is the snippet of code that executes it:

When the line "GetPythonEngine.ExecString(UTF8Encode(sFormula));" is executed, the program freezes.
And it just sits there without doing anything (there is no processor consumption).
You need to end the process to end execution.

The problem only occurs if the line "import pandas as pd" exists.

 

As the DataSnap server receives multiple requests (multi-thread), I am following the recommendations/rules for working with multi-threads with Python/P4D.

Other scripts, including more complex ones, work normally, as long as the "import pandas as pd" line does not exist.

I did the same test in Visual Studio Code and it works normally:

 

I tried to reproduce this problem outside of my DataSnap project, but I was unable to do so.
When doing this in a simple example project (even within a thread), it works normally.
Therefore, I cannot send an example project so that the problem can be reproduced there.

I searched the posts here for a problem similar to this, but I didn't find anything.

 

Is it possible to help me with this?

[limelect 48]

I do not know if this helps but see

In my case, the behavior changed using Microsoft Visual Studio 2019 for Python

 

[djxandytche 0]

Hi limelect,
 

Thanks for trying.
Unfortunately it didn't help in my case.
 

But, I discovered that the problem only occurs if I run the Python script within a Delphi thread and I was able to reproduce the problem in a simple example project.
The complete example project is attached or can be downloaded from the link below:
https://drive.google.com/file/d/1F0AbuU5TFTzP2wzdampzLi8gSPQRiTvp/view?usp=sharing
 

To see the problem occur, just run "Project1.exe".
Execution will be stuck at the line "GetPythonEngine.ExecString(UTF8Encode(sScript));".
 

If you remove the "import pandas as pd" line from the Python script to be executed, then it works normally.
 

Delphi version: 11.1.
Python version: 3.11.4.
Pandas library version: 2.0.3.
p4d version: I put the source code along with the example project.
 

If anyone can find out the reason for the crash and know how to resolve it, I would be grateful.

travamentopandas.zip

[pyscripter 689]

@djxandytche Your code for running python code in threads is reinventing the wheel and the result is evidently incorrect.

 

Just follow the guidelines at PythonThreads · pyscripter/python4delphi Wiki (github.com) paying attention also to the preparation section.

 

Also, have a close look also to Demos 33 and 36.

Edited July 11 by pyscripter

[djxandytche 0]

Hi pyscripter,
 

Thanks for the guidance.
I had already read this documentation and had also seen the demos, but now I have reviewed everything again and I think I managed to improve my understanding.
After reviewing everything, I adjusted the code and then resolved the problem.
I believe that my code is now correct (if there is still something to be improved, I will be grateful if you let me know).
The new project with the corrected code is attached or can be downloaded from the link below:
https://drive.google.com/file/d/18LFBKkMv5v0Q7ZzWUb7a1CZVWh5DqmED/view?usp=sharing
 

But, I had a question:
In my case, where I am using Delphi's "TTask" class and p4d's "SafePyEngine" function, how do I set p4d's "TThreadExecMode (emNewState, emNewInterpreter, emNewInterpreterOwnGIL)"?
I saw that in the p4d "TPythonThread" class it is possible to choose "TThreadExecMode". But, I couldn't figure out how to do this when I use "TTask" + "SafePyEngine".
 

Thank you in advance for your help,
 

Alexandre.

travamentopandas2.zip

[pyscripter 689]

With SafePyEngine you do not get that choice.   It is equivalent to emNewState, which is by far the most common use case.

[djxandytche 0]

pyscripter, got it, thanks.
 

I found another problem (at least I think it's a problem) when using "TTask" + "SafePyEngine", but, later I report it in a new topic.
 

Now I would like to ask for help again with the "hanging" problem when using "import pandas as pd" in a script that runs within a Delphi thread.
I made a new example project, even simpler, based on the demos and managed to reproduce the "hanging" problem.
The new project is attached and can be downloaded from this link:
https://drive.google.com/file/d/1xdv8NDK31wao3EjXzuschispAnzdDqNs/view?usp=sharing

To reproduce the problem, simply select the "emNewInterpreter" option and click the "Button1" button, as shown in the image below:

If you select the "emNewState" option the problem does not occur.

Could you check if there is still a problem in my code that I am not seeing/noticing?
 

Thanks in advance.
 

Alexandre.

travamentopandas3.zip

[pyscripter 689]

1 hour ago, djxandytche said:

Could you check if there is still a problem in my code that I am not seeing/noticing?

Have you read the docs and the limitations of using subinterpreters?

 

Initialization, Finalization, and Threads — Python 3.12.4 documentation  Also the docs for the PyInterpreterConfig structure?

 

pandas and numpy probably do not support running in subinterpreters.

 

I suggest that you forget about sub-interpreters unless you know very well what you are doing.  But then you are on your own.

Edited July 12 by pyscripter

[djxandytche 0]

pyscripter,
 

I only read the docs from the p4d Git repository.
These here: https://github.com/pyscripter/python4delphi/wiki.

I hadn't read this one yet: https://docs.python.org/3/c-api/init.html#c.Py_NewInterpreterFromConfig.
But, I read it now.
However, I was unable to get a complete understanding.
I'm a beginner at Python.
 

Well, I'm going to forget the "emNewInterpreter" option from now on.
I've done some testing now with the "emNewState" option and found a behavior that seems like a problem, but I would like to ask you if this is really a problem.
It could be that it's something I did wrong in my code because I didn't read or didn't correctly understand some of the existing documentation.

I found something that seems to be a memory sharing of Python script variables between threads (and I think that's a problem).

I created a small example project that reproduces this case. The project is attached or can be downloaded here:
https://drive.google.com/file/d/1KFqygNb0MKJODgqcYaFQ8Cny5x9tu0xi/view?usp=sharing
Brief explanation of the example project:
It creates 10 threads.
Each thread executes the same script 10000 times. For example: thread 1 has the script "result.value = 1", thread 2 has the script "resulta.value = 2", and so on.
When all threads finish executing, I check the returns from script executions.
If I find returns in a thread that are from another thread, I generate a LOG line in a TMemo.

Could you check and help me with this case?
 

Note: If you think it's better to create a new topic to avoid mixing topics, let me know and I'll do it.

Thanks,

Alexandre.

testepythonmultithread1.zip

[pyscripter 689]

@djxandytche  "Here is my code, please check and find the bugs" is not a good way to get support.  Why should anyone spend one hour of their time, understanding and debugging your code?

 

You say you are beginner in Python.  Then why do you start by running python code in threads, which requires some expert knowledge?  Do you realize that there is no performance benefits in running python code in threads like this?  They will be executed sequentially, thanks to the GIL.

 

[djxandytche 0]

pyscripter,

 

My intention was to show something that seems to me to be a problem in p4d.
And sending this example was the best way I found.
 

Answering your questions:
1 (why do you start by running python code in threads): Because I want to use p4d in the backend of the company's system for which I work and this backend is a Delphi DataSnap REST server that runs on Apache, therefore, it receives multiple requests (multi- thread);
2 (there is no performance benefits in running python code in threads): Yes, I started to understand this in the last few weeks that I have been studying and working on integrating p4d into my system;
 

My plan B is to create a single point to execute Python scripts in a serialized way (using Delphi's lock-unlock like TMonitor).
But first I would like to try to make it work using threads.
 

Just to give more context: My company develops a BI system (similar to the idea of Microsoft Power BI) and we intend to allow the user to create a data source based on a Python script (in addition to being able to connect to various databases) . Company website: https://weknowhealthtech.com.br/. We are from Brazil.
 

Note: I know that p4d is free and open source and therefore no one is obliged to help me, but the company I work for would be willing to hire some type of paid help/consultancy so that we can continue with the p4d integration project into our system.
 

Grateful,

Alexandre.

[pyscripter 689]

2 hours ago, djxandytche said:

My intention was to show something that seems to me to be a problem in p4d.

You never explained what is the problem.

[djxandytche 0]

I'll try to explain better...

Imagine you have the thread below:

TPythonThread_Test1 = class(TPythonThread)
protected
    procedure ExecuteWithPython; override;
public
    fOutputs: TStringList;
    Script: string;
end;

procedure TPythonThread_Test1.ExecuteWithPython;
var
    i: Integer;
    zPythonDelphiVar: TPythonDelphiVar;
begin
    fOutputs := TStringList.Create;

    zPythonDelphiVar := TPythonDelphiVar.Create(nil);

    try
        zPythonDelphiVar.Engine := GetPythonEngine;
        zPythonDelphiVar.VarName := 'result';
        zPythonDelphiVar.Initialize;

        for i := 1 to 10000 do
            begin
                GetPythonEngine.ExecString(UTF8Encode(Script));
                fOutputs.Add(zPythonDelphiVar.Value);
            end;
    finally
        zPythonDelphiVar.Free;
    end;
end;

Now you create 10 instances of this thread and assign the following values to the "Script" property of each of them. For example:
Thread 0: Script = result.value = "thread 0"
Thread 1: Script = result.value = "thread 1"
Thread 2: Script = result.value = "thread 2"
Thread 3: Script = result.value = "thread 3"
Thread 4: Script = result.value = "thread 4"
Thread 5: Script = result.value = "thread 5"
Thread 6: Script = result.value = "thread 6"
Thread 7: Script = result.value = "thread 7"
Thread 8: Script = result.value = "thread 8"
Thread 9: Script = result.value = "thread 9"
Thread 10: Script = result.value = "thread 10"
 

Then you start running all instances at the same time.
This will cause each instance to execute the same script 10000 times and assign the result in the "fOutputs" property.
In other words, "Thread 0", for example, will add 10000 times the string "thread 0" in the "fOutputs" list.
 

But, some executions of some threads are interfering with the execution of other threads.
For example, sometimes the value "thread 2" appears in the "fOutputs" list of "Thread 5". This shouldn't happen.
 

Here is a screenshot of the example project I created that reproduces this problem:

 

I hope I was clear enough.
 

Thanks,

 

Alexandre.

Edited July 15 by djxandytche

[pyscripter 689]

You don't need to create and destroy zPythonDelphiVar in every thread.

 

Add zPythonDelphiVar to the form:

  object zPythonDelphiVar: TPythonDelphiVar
    Engine = PythonEngine1
    Module = '__main__'
    VarName = 'result'
    Left = 64
    Top = 120
  end

 

Then modify ExecuteWithPython to 

 

procedure TPythonThread_Test1.ExecuteWithPython;
var
    i: Integer;
begin
    fOutputs := TStringList.Create;

        for i := 1 to 10000 do
            begin
                GetPythonEngine.ExecString(UTF8Encode(Script));
                GetPythonEngine.CheckError;
                fOutputs.Add(Form1.zPythonDelphiVar.Value);
            end;
end;

and it should work as you would expect.

 

And since you mentioned it, a fellow Brazilian @lmbelo Lucas Belo, is a major contributor to P4D and would certainly be in a position to help.

Edited July 16 by pyscripter

[djxandytche 0]

pyscripter,
 

I made the adjustments you suggested and now it is working correctly.
Thank you for your help.

 

Thank you for recommending Lucas Belo.

 

Alexandre.