How can globals be shared amongst multiple modules?

[RSG 0]

I am using Python4Delphi, Oct2Py, and Octave to add Octave scripts to my Delphi app. I've got it all working beautifully in a test setup using one Python4Delphi "module" (here, a Python module that is imported via VarPyth.Import, as in Demo25's code after the comment "importing an external module and using it"). However, my production app will have several modules.

 

The problem I am facing now is how to get them all to work together. Note that by nature of this application, only one module will ever be "running" at any point in time; the nature of the problem is that exactly one is chosen and then used, then another chosen and used, etc.; in other words, strictly sequential usage. One approach which would normally work if these were simple Python scripts is just to load the modules then run them sequentially; however, each script must call a global instance of an Oct2Py object, which is essentially the Octave interpreter (fully analogous to the Python engine!). Since we can only have one Python engine, we can only have one Octave interpreter; besides, it can take 5-10 seconds to load the Octave interpreter, so we only want to do that once. So the real issue is how to share the single Oct2Py object among the various Python modules.

 

PythonEngine does have a GlobalVars dictionary property that appears to do what I need; if a Python script is run that loads a global variable with the Oct2Py object, then it could be provided to the other modules when they are called. However, there are no examples of this usage and I haven't been able to figure out how to do this. Is it possible? If so, how do I set up the GlobalVars property so that it contains the Oct2Py global, and then how do I provide it to the Python scripts?

 

For reference, here is some Python-only code that works; imagine each of the two functions being separate modules.

if 'octave' in globals():
    print('Oct2Py/Octave already set up - reusing')
else:
    print('Setting up Oct2Py/Octave')
    from oct2py import Oct2Py
    octave = Oct2Py()

def addMembers(fam):
    return octave.addMembers(fam)

def removeMembers(fam):
    return octave.removeMembers(fam)

 

[pyscripter 689]

I am not sure what the issue you are trying to solve is.

 

You can have the statement:

 

from oct2py import Oct2Py

 

or just

 

import oct2py

 

within different python modules or functions.

It does not result in the module being reimported.   If it has been already imported it just uses the already imported one and there is no speed penalty.

 

The same applies to the VarPyth Import statement.

Edited June 24, 2023 by pyscripter

[RSG 0]

Maybe getting closer; TPythonEngine has a GlobalVars property that isn't initialized to anything; this seems to set it:

 

GlobalEngine.GlobalVars := GlobalEngine.ArrayToPyDict([]);

But it doesn't seem to work, in that a global variable defined in one Python script is not available in another script. 

 

I've also confused things a bit in my earlier post; here are my two Python scripts. Script 1:

print('1a. p4d-oct2py-bpl/Win32/Debug/octaveTest.py')
if 'octave' in globals():
    print('1a. Oct2Py already set up - reusing')
else:
    print('1a. Setting up Oct2Py')
    from oct2py import Oct2Py
    octave = Oct2Py()

def addMax(fam):
    print('1b. Hello Max')
    return octave.addMax(fam)

Script 2:

print('2a. p4d-oct2py-bpl/Win32/Debug/emilyTest.py')
if 'octave' in globals():
    print('2a. Oct2Py already set up for Emily- reusing')
else:
    print('2a. Setting up Oct2Py for Emily')
    from oct2py import Oct2Py
    octave = Oct2Py()

def addEmily(fam):
    print('2b. Hello Emily')
    return octave.addEmily(fam)

Here is the output from Python (I use pyIOconnector to send standard output to a TMemo):

1a: p4d-oct2py-bpl/Win32/Debug/octaveTest.py
1a: Setting up Oct2Py
2a: p4d-oct2py-bpl/Win32/Debug/emilyTest.py
2a: Setting up Oct2Py for Emily
1b: Hello Max
2b: Hello Emily
1b: Hello Max

The 1a and 2a are output when I load the two modules; clearly, both modules create their own `octave` variables. When I run the functions, we see from the 1b and 2b that the def functions are called, and the global code (the code outside of the def functions) is not.

 

So what appears to be going on here is that the variable 'octave' in each module is really global to that module. The only problem I was worried about is that I thought each module was going to take the 5 second loading of Octave, but that doesn't appear to be the case. At this point, I suppose I don't need globals (in the sense of the single Python DLL) after  all! Does that make sense?

 

For what it's worth, the expensive call here is the first `octave = Oct2Py()` call; the second one returns immediately. 

[RSG 0]

Good to know, pyscripter, and thanks! I just got your reply after my last! However, it is the call to Oct2Py() that is expensive, not the import, so that doesn't help. Fortunately, as I just discovered, that call is both expensive only during the first call, and can be called by each of my modules! So it appears my "octave" Python variable can be initialized separately by each module.

 

I do have some concerns about what Octave preserves between calls to its scripts, but this is off-topic here. I'm hoping (and will be investigating) that it maintains state between calls to its scripts!

 

For completeness, I may still want to share globals across multiple Python modules and script calls, so I'm still interested in my original questions...

[pyscripter 689]

If you want to preserve globals in script execution you need to use the ExecString(s) overload with the locals, globals parameters.

Edited June 25, 2023 by pyscripter

[RSG 0]

I see that is possible, but if you look at the code, it will use the GlobalVars member if those parameters are nil, right? From PythonEngine.pas:

 

procedure TPythonEngine.ExecString(const command : AnsiString);
begin
  Py_XDecRef( Run_CommandAsObject( command, file_input ) );
end;

function TPythonEngine.Run_CommandAsString(const command : AnsiString; mode : Integer) : string;
var
  v : PPyObject;
begin
  Result := '';
  v := Run_CommandAsObject( command, mode );
  Result := PyObjectAsString( v );
  Py_XDECREF(v);
end;

function TPythonEngine.Run_CommandAsObject(const command : AnsiString; mode : Integer) : PPyObject;
begin
  Result := Run_CommandAsObjectWithDict(command, mode, nil, nil);
end;

function TPythonEngine.Run_CommandAsObjectWithDict(const command : AnsiString; mode : Integer; locals, globals : PPyObject) : PPyObject;
var
  m : PPyObject;
  _locals, _globals : PPyObject;
begin
  CheckPython;
  Result := nil;
  Traceback.Clear;
  CheckError(False);

  m := GetMainModule;
  if m = nil then
    raise EPythonError.Create('Run_CommandAsObject: can''t create __main__');

  if Assigned(locals) then
    _locals  := locals
  else if Assigned(FLocalVars) then
    _locals  := LocalVars
  else
    _locals  := PyModule_GetDict(m);

  if Assigned(globals) then
    _globals := globals
  else if Assigned(FGlobalVars) then
    _globals := GlobalVars
  else
    _globals := _locals;

  try
    Result := PyRun_String(PAnsiChar(CleanString(command)), mode, _globals, _locals);
    if Result = nil then
      CheckError(False);
  except
    if PyErr_Occurred <> nil then
      CheckError(False)
    else
      raise;
  end;
end;

 

Thinking a bit more, I'm wondering if any globals that are defined in a Python script get returned to the PythonEngine object; is that what I'm missing here? It seems that could be the issue I'm having now - thoughts?

 

Thanks for the great work, by the way!

[pyscripter 689]

Could you post a sample project that demonstrates the issue?

[RSG 0]

On 6/26/2023 at 1:45 AM, pyscripter said:

Could you post a sample project that demonstrates the issue?

I tried to recreate the problem, but the sample project works differently, and it seems my app is now working (on my machine). I don't know what I changed, or maybe my understanding has changed. I may revisit this if things change, but right now this is a mystery that I cannot recreate, simplify, or even explain correctly - not to mention understand it! - so I'm going to have to let this idle for now. Thanks regardless!