Multiple Instances of Python Engine

[psla314 3]

Hi

 

We are looking at using the Python engine in a Delphi graphing application, so we would have a different python script for each chart, with multi charts opened at once.

So in this case we would have different python scripts each with different python4delphi modules containing different input data for processing and producing different output data.

I first tried this with with each chart having a separate python4delphi engine and module and script, but I got an exception 

There is already one instance of TPythonEngine running

 

After examining the code there appears to be a global variable for the engine, not allowing more than one instance of the engine.

 

My question is what is the best way of handing multiple scripts with multiple modules simultaneously?

Is there a way to bypass the global engine variable so we can have multiple engine instances ?

Or do we need to keep a list of all the scripts/modules and shuffle them through the global engine, one at a time ?

 

Any suggestions would be appreciated.

 

Regards

Peter

 

 

[David Heffernan 2353]

What exactly is your rationale for wanting to use multiple engines? Why is it not possible to do this with a single engine? 

[pyscripter 694]

• You can only load one pythonxy.dll in the Delphi process, hence you can only have one PythonEngine in an application.
• Look at Demo33 on how to use threads and/or multiple sub-interpreters without blocking the main thread.  But also note that you cannot bypass the python famous GIL (Global Interpreter Lock), so that only one python thread can be executing at the same time.
• Under Tutorials/Webinar II look at PyVizSVG.dproj on how to generate svg files in python scripts and how to display them in Delphi,
• The only way to really use multiple cores to generate the graphs is to use external processes.  (there is the mutliprocessing unit in python, but it would be easier just to start multiple python processes from Delphi).

[Arnaud Bouchez 407]

I guess this is a python runtime limitation.

Due to the https://en.wikipedia.org/wiki/Global_interpreter_lock then the engines are not thread-safe.

 

Whereas for instance, we use SpiderMonkey (for running JavaScript) with a per-thread engine, with no problem.

[fjames 0]

There should be no reason you cant use a single engine, and then have a separate module for being imported by separate scripts.
The same engine can execute all the scripts, (albeit sequentially due to issues described above), but for your use case I don't see this being an issue.

[psla314 3]

Thanks for the advice, I'll try using multiple modules and pass through a single engine. Didn't know about the Python global interpreter lock.

[David Heffernan 2353]

Do you need multiple modules? You talked about multiple modules to hold different data. But why does the data have to live in a module? 

[psla314 3]

I think we need multiple modules because we are injecting vars into the module for each script. Each script will have different data injected. So figured it would be better to inject data once and then swap modules/script to execute the script. Note the scripts will get execute multiple times after new data is added to the injected data objects.

Edited March 19, 2021 by psla314

[psla314 3]

I have managed to get multiple modules (each with a different script) working simultaneously (run one at a time, but modules in memory at same time), but not sure its the most efficient way. 

 

I am creating multiple modules  then when the script is changed and run, I am clearing vars and methods and Initalizing the module each time , then wrapping a few delphi classes and injecting into module via module.SetVar('Source', p) and module.SetVar('Output', p).

To execute script I am locking the Engine and then using PythonModule.GetVar('__dict__') to get the local and passing that to the ExecString as local and global. (To have a clean namespace each time)

Basic code is below.

 

Question - Is there a more efficient way to doing this without having to Initialize the module each time the script changes ? And Ideally not have to wrap and insert the variables each time.

 

m_sScript.LoadFromFile(sFilePath);


m_PythonModule.ClearVars;
m_PythonModule.ClearMethods;
m_PythonModule.Initialize;

datalist := TDatalist.create;  // this is freed by python object
pSource := m_PyDelphiWrapper.Wrap(datalist, WrapDelphi.soOwned);
m_PythonModule.SetVar('Source', pSource );
m_PythonEngine.Py_DecRef(pSource);

dataout := TDatalist.create;
pDataOut := m_PyDelphiWrapper.Wrap(dataout, WrapDelphi.soOwned);
m_PythonModule.SetVar('DataOut', pDataOut );
m_PythonEngine.Py_DecRef(pDataOut);

// Run Process Method in Python SCript
m_PythonEngine.Lock;
try
   locals := m_PythonModule.GetVar('__dict__');
   m_PythonEngine.ExecString(ansistring(m_sScript.DataString), locals, locals);
   m_PythonEngine.CheckError;
   pyfunc := m_PythonEngine.FindFunction(m_PythonModule.ModuleName , 'Process');
   if assigned(pyfunc) then
   begin
      try
        m_PythonEngine.EvalFunction(pyfunc,[]);
      finally
        m_PythonEngine.Py_DecRef(pyfunc);
      end;
   end;
   m_PythonEngine.Py_DecRef(locals);
finally
   m_PythonEngine.Unlock;
end;

 

 

 

Edited March 19, 2021 by psla314

[David Heffernan 2353]

3 hours ago, psla314 said:

I think we need multiple modules because we are injecting vars into the module for each script. Each script will have different data injected. So figured it would be better to inject data once and then swap modules/script to execute the script. Note the scripts will get execute multiple times after new data is added to the injected data objects.

I'd design the code not to use global vars.