Using Python4Delphi library in a Windows service

[Luca Pretti 0]

I am trying to implement a Windows service to run some Python scripts in background. Is that possible ?

As soon as I try to create a TPythonEngine instance with AutoLoad set to true the service hangs. 

If I create an instance with AutoLoad set to False the service starts correctly but as soon I try to call LoadDll the service hangs.

Any help will be greatly appreciated.

[pyscripter 667]

When you run as a service the registry information is not available.  You need to do something like:

procedure TService1.CreatePyEngine;
begin
  PythonEngine := TPythonEngine.Create(nil);
  PythonEngine.Name := 'PythonEngine';
  PythonEngine.DLLName := 'python313.dll';
  PythonEngine.DllPath := 'c:\pathtoyourpythonhome\';
  PythonEngine.RegVersion := '3.13';
  PythonEngine.UseLastKnownVersion := False;
  PythonEngine.FatalAbort := False;
  PythonEngine.FatalMsgDlg := False;
  PythonEngine.LoadDll;
end;

I have tested and it works.

[Luca Pretti 0]

Yes, the library is loaded correctly, but as soon as the program calls ExecString the program crashes.

In the Winows Application Event Viewer I find this error event :

 

 

 

This is the code in the ServiceExecute event handler:

 

  while not terminated do
  begin
    ServiceThread.ProcessRequests(False);
    inc(LoopsCount);
    if LoopsCount > 100 then
    begin
      PythonEngine.ExecString('2*2');
      LoopsCount := 0;
    end;
    sleep(100);
  end;

 

[Remy Lebeau 1373]

28 minutes ago, Luca Pretti said:

In the Winows Application Event Viewer I find this error event :

 

 

An AV near address 0 usually means a nil pointer is being dereferenced. Did you try debugging your service?

Quote

This is the code in the ServiceExecute event handler:

Where and how are you creating the Python Engine?

Edited yesterday at 07:54 AM by Remy Lebeau

[Luca Pretti 0]

29 minutes ago, Remy Lebeau said:

An AV near address 0 usually means a nil pointer is being dereferenced. Did you try debugging your service?

Where and how are you creating the Python Engine? 

I dropped the component on the service module and set AutoLoad to False. Then in the ServiceCreate event handler I do:

 

  PythonEngine.DLLName := 'python312.dll';
  PythonEngine.DllPath := 'C:\Users\Luca\AppData\Local\Programs\Python\Python312\';
  PythonEngine.RegVersion := '3.12';
  PythonEngine.UseLastKnownVersion := False;
  PythonEngine.FatalAbort := False;
  PythonEngine.FatalMsgDlg := False;
  PythonEngine.LoadDll;

 

[Cristian Peța 103]

I am not sure right now but I suppose ServiceCreate and ServiceExecute are not in the same thread.

Why not creating PythonEngine in ServiceExecute?

[Luca Pretti 0]

I managed to debug the service and I found that the ACCES VIOLATION is raised at line 5091 of the PhythonEngine.pas unit, "CheckError(False)".

The problematic line of code of this procedure is the first (line 6398)  : " PyException := PyErr_Occurred;"

 

[Cristian Peța 103]

8 minutes ago, Luca Pretti said:

The problematic line of code of this procedure is the first (line 6398)  : " PyException := PyErr_Occurred;"

 

PythonEngine appears to be nil. Did you checked there if Self is nil?

Edited yesterday at 09:37 AM by Cristian Peța

[Luca Pretti 0]

30 minutes ago, Cristian Peța said:

I am not sure right now but I suppose ServiceCreate and ServiceExecute are not in the same thread.

Why not creating PythonEngine in ServiceExecute?

You were right !

If I create the PythonEngine instance in ServiceExecute I do not get the access violation anymore.

Thank you very much !

[DelphiUdIT 158]

I think should be better that you create the Python engine in a START event of controller, and Free it on Destroy event.

 

Execute method is call more times during the "live" state of Service.

 

Bye

[Cristian Peța 103]

7 minutes ago, DelphiUdIT said:

I think should be better that you create the Python engine in a START event of controller, and Free it on Destroy event.

 

Execute method is call more times during the "live" state of Service.

But then it should setup Python engine to run in other thread context. I don't think it is worth.

[pyscripter 667]

What I think is best:

- Create the pythonEngine (or load the python dll) on Start

- Destroy the pythonEngine (or unload the python dll) on Stop

- Follow these guideliines to execute python code in threads.

 

Please see the attached project for an example that does that.

Sample log output after  starting and stopping the engine:

 

Logs From Background Thread: 24/10/2024 14:49:18

Python eval = 4

Logs From Background Thread: 24/10/2024 14:49:19

Python eval = 4

Logs From Background Thread: 24/10/2024 14:49:20

Python eval = 4

Logs From Background Thread: 24/10/2024 14:49:21

Python eval = 4

Service.zip

Edited 22 hours ago by pyscripter

[Luca Pretti 0]

Now the Windows service starts correctly and keeps running but when I try to execute a script I get the following error:

 

ImportError: Unable to import required dependencies:
numpy: No module named 'numpy'
pytz: No module named 'pytz'
dateutil: No module named 'dateutil'

 

The same exact code works perfectly when the application is compiled as a console.

It looks like the Python engine is not able to resolve the path to base scripts when launched by a service.

This is my registered path: PATH=C:\Program Files\Python313\Scripts\;C:\Program Files\Python313\;

[DelphiUdIT 158]

If there are DLL required for python here is the logic, from: https://learn.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-search-order

 

Quote

If a DLL has dependencies, then the system searches for the dependent DLLs as if they were loaded by using only their module names. That's true even if the first DLL was loaded by specifying a full path.

Quote

Standard search order for unpackaged apps

The standard DLL search order used by the system depends on whether or not safe DLL search mode is enabled.

Safe DLL search mode (which is enabled by default) moves the user's current folder later in the search order. To disable safe DLL search mode, create the HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\SafeDllSearchMode registry value, and set it to 0. Calling the SetDllDirectory function effectively disables safe DLL search mode (while the specified folder is in the search path), and changes the search order as described in this topic.

If safe DLL search mode is enabled, then the search order is as follows:

1. DLL Redirection.
2. API sets.
3. SxS manifest redirection.
4. Loaded-module list.
5. Known DLLs.
6. Windows 11, version 21H2 (10.0; Build 22000), and later. The package dependency graph of the process. This is the application's package plus any dependencies specified as <PackageDependency> in the <Dependencies> section of the application's package manifest. Dependencies are searched in the order they appear in the manifest.
7. The folder from which the application loaded.
8. The system folder. Use the GetSystemDirectory function to retrieve the path of this folder.
9. The 16-bit system folder. There's no function that obtains the path of this folder, but it is searched.
10. The Windows folder. Use the GetWindowsDirectory function to get the path of this folder.
11. The current folder.
12. The directories that are listed in the PATH environment variable. This doesn't include the per-application path specified by the App Paths registry key. The App Paths key isn't used when computing the DLL search path.

If safe DLL search mode is disabled, then the search order is the same except that the current folder moves from position 11 to position 8 in the sequence (immediately after step 7. The folder from which the application loaded).

 

LOL: I really didn't know about that "The 16-bit system folder. There's no function that obtains the path of this folder, but it is searched."

Edited 21 hours ago by DelphiUdIT

[pyscripter 667]

In the demo project modify the CreateEngine function as follows:

 

procedure TService1.CreatePyEngine;
begin
  PythonEngine := TPythonEngine.Create(nil);
  PythonEngine.Name := 'PythonEngine';
  PythonEngine.DLLName := 'python313.dll';
  PythonEngine.DllPath := 'c:\python\python313\';
  PythonEngine.PythonHome := PythonEngine.DLLPath;
  PythonEngine.VenvPythonExe := PythonEngine.DLLPath + 'python.exe'; // may be needed
  PythonEngine.RegVersion := '3.13';
  PythonEngine.UseLastKnownVersion := False;
  PythonEngine.FatalAbort := False;
  PythonEngine.FatalMsgDlg := False;
  PythonEngine.LoadDll;
  TPythonThread.Py_Begin_Allow_Threads;
end;

 

I think this should set the python path correctly and avoid import errors.

Edited 20 hours ago by pyscripter

[Remy Lebeau 1373]

5 hours ago, Cristian Peța said:

I am not sure right now but I suppose ServiceCreate and ServiceExecute are not in the same thread.

That is correct. The TService's DataModule is created in the process' main thread, whereas its On(Start|Execute|Stop|Shutdown) events are called in a worker thread.