How to get Python functions running in the background (P4D)

[JGMS 2]

I want to run python functions in the background, for example like so:

MI_CopyrightAddition.Enabled := false;
TTask.Run( Procedure
begin
 PyForm.Add_Copyright_Using_Python(FotoSerie,CopyRightTeken + String(CopyRightTekst), False);
 TThread.Synchronize(nil, procedure
   begin
     MI_CopyrightAddition.Enabled := TRUE;
   end);
end);

"PyForm.Add_Copyright_Using_Python" is a correctly working Python function, running normally in the main Thread.

But in the above scheme it crashes:
"Project .... raised exception class $C0000005 with message 'c0000005 ACCESS_VIOLATION'."

Is there a way to get this working?

Many thanks for your time.

[pyscripter 782]

Everything you need to know is here.

[JGMS 2]

After studying Demo 36 I made a unit to use the concept in a VCL program to process a bunch of jpg pictures, in parallel mode. I used exactly the same code, except of course for the script.

It failed in the procedure "DestroyEngine" when freeing the engine: 'Project .... raised exception class $C0000005 with message 'c0000005 ACCESS_VIOLATION'".
I suspect that the Event.Waitfor command does not wait until the saving of the jpg's is finished. Or perhaps, file processing isn't possible at all in this way.
Anyway, I could not find a long enough sleeping time as set before destroying the PythonEngine. 

I also tried to run the program using "emNewInterpreter" instead of "emNewInterpreterOwnGIL". This does not give errors when freeing the engine, but the script neither runs;

The Pythoncode that is in the script runs perfect in PyScripter, but not in the modified Demo 36.

Here is the function that should do the job:

Function RunParallel(MapBron, MapDoel : String; InitScript : String) : Boolean;
var
  I, N: Integer;
  InTijdlijnStr : String;
begin
  Result := True;
  InArray := TDirectory.GetFiles(MapBron,'*' + ExtensieFotoBestand);
  N := 25;//length(Inarray); // limit for testing
  IF N = 0 then exit;
  try
    CreatePyEngine;
    try
      Event := TCountdownEvent.Create(N);
      // 'Subinterpreter with own GIL:
      for I := 1 to N do
      begin
        Var Doel : String := IncludeTrailingPathDelimiter(B2F(MapDoel)) + ExtractFilename(InArray[I-1]);
        Script := InitScript + 'mirror("' + B2F(InArray[I-1]) + '")';
        TPyThread.Create(emNewInterpreter); // TPyThread.Create(emNewInterpreterOwnGIL);
      end;
      Event.WaitFor;
      Event.Free;
    finally
      Sleep(3000);  // allow some time for the threads to terminate
      DestroyEngine;
    end;
  except
    on E: Exception do
    Result := False;
  end;
end;

 

I used a simple test script, as shown below. It is rerun in a loop. The final line is added in the loop, as seen in the above function.

import piexif
from PIL import Image

def mirror(input_path):
   img = Image.open(input_path)
   try:
     exif_dict = piexif.load(img.info["exif"])
   except:
     exif_OK = False
   else:
     exif_OK = True
   rot_img = img.transpose(method=Image.FLIP_LEFT_RIGHT)
   w, h = rot_img.size
   if exif_OK:
     exif_dict["0th"][piexif.ImageIFD.XResolution] = (w, 1)
     exif_dict["0th"][piexif.ImageIFD.YResolution] = (h, 1)
     try:
       exif_bytes = piexif.dump(exif_dict)
     except:
       rot_img.close()
       return
     rot_img.save(input_path, "jpeg", exif=exif_bytes, quality=95)
   else:
     rot_img.save(input_path, "jpeg", quality=95)
   rot_img.close()
   
mirror("F:/Eline/Marie/20150720 103259_NIKON.JPG")   <=== added in the loop

Would you please give me some hints on how to proceed.
Many thanks ahead.

Jan

[pyscripter 782]

I guess ExecStrings raises an exception so, the event is not signalled.

 

You can modify ExecuteWithPython to always signal:

 

procedure TPyThread.ExecuteWithPython;
begin

  try
    GetPythonEngine.ExecString(Script);

  finally
    Event.Signal;

   end;
end;

 

Don't you get the error printed when running your script?  You can also use the debugger to see if/what the exception is.

 

Note that not all modules are compatible with  PyInterpreterConfig_OWN_GIL.  I suspect PIL isn't, Read the documentation about the limitations.

 

Does you program work with emNewState?  Note that emNewInterpreter does not offer any performance advantages compared to emNewState.

 

Given that Demo 36 works, start from that and then gradually move towards what you want, until you find what fails.  The first thing that I would test is the imports.   Add:

import sys
print(sys.path)

to make sure that you are using the correct version of python with your desired imports installed.

 

Then add your imports to see whether they work.

Edited June 15 by pyscripter

[JGMS 2]

The three possible ways in creating the threads give totally different memory leakage messages when closing the application.
But none of them successfully mirrors the pictures, nor do they show error messages.

1) TPyThread.Create(emNewInterpreter) gave no memory leak messages.
2) TPyThread.Create(emNewInterpreterOwnGIL) results 17 out 25 files memory leaks.

3)TPyThread.Create(emNewInterpreterOwnGIL) 25 memory, and different to the previous one

I started off with demo 36 and got it running, but experienced the same  'c0000005 ACCESS_VIOLATION' initially, until I started playing with the sleep function.
The demo appears to need the sleep anyway, with or without the try ..finally addition, you mentioned.
I tryed to read the documentation in the link, but failed to see info about whether or not PIL is supported. I begin to believe that that is indeed the problem that I face. That would be all too pity.

[JGMS 2]

@pyscripter, I tryed OpenCV as well: it neither works, unfortunately.
The same script runs normally in PyScripter (catching it via debugging).

emNewInterpreterOwnGIL does not give the EPyImporterror memory leak message, by the way. That leaves some hope, though.

Edited June 15 by JGMS