Debug issue with Numpy loading

[RSG 0]

The app I am working on needs to support running Octave (GNU answer to Matlab) scripts. My approach is to use Python4Delphi to enable Python scripting in my Delphi app, which works fine. (Note: I have other uses for Python scripting in other areas of the app.) In the Python scripts, I am using Oct2Py to connect to Octave. Because Oct2Py requires NumPy and SciPy, my Python installation is recent Anaconda, which comes with those packages pre-installed. After much trial and error, with great help from here and Stack Overflow, I got it all working. But...

 

Yesterday I tried to set up the app on my client's machine and I couldn't get it to work.  The problem is shown with the following two-line Python script:

from oct2py import Oct2Py
octave = Oct2Py()

I'm not sure, but I believe the first line is the problem; if not that, then the second.  The following message is displayed:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy c-extensions failed. This error can happen for
different reasons, often due to issues with your setup.

If I run the exact same script from an Anaconda shell (i.e. from the Python shell), it works correctly, so Anaconda/Oct2Py/Octave is working correctly. When run from my Delphi app, it fails with that error. Other Python can run correctly from my Delphi app, so the basic Python4Delphi mechanism seems to be working. It is the whole Delphi/Python4Delphi/Oct2Py/Octave chain that fails, on that machine (Windows 10; mine is Windows 11 - I have not been able to find any differences of significance yet!).

 

I think the key question is why Numpy appears broken to scripts running via TPythonEngine vs. the Anaconda shell? Any ideas?

 

For reference, this is the procedure I followed when installing everything on my client's computer:

 

1. Install Anaconda (https://repo.anaconda.com/archive/Anaconda3-2022.05-Windows-x86.exe) for All Users as administrator

2. Install Octave (https://ftpmirror.gnu.org/octave/windows/octave-8.2.0-w32-installer.exe) the same way.

3. In Anaconda admin shell, install oct2py (conda install -c conda-forge oct2py)

4. Make sure PATH is set to point to Anaconda correctly

5. Add system variable OCTAVE_EXECUTABLE to point to octave-cli.exe

[pyscripter 583]

Have you read these:

• MaskFPUExceptions · pyscripter/python4delphi Wiki (github.com)
• FindingPython · pyscripter/python4delphi Wiki (github.com) with reference to Anaconda?

[RSG 0]

On 7/15/2023 at 4:42 PM, pyscripter said:

Have you read these:

• MaskFPUExceptions · pyscripter/python4delphi Wiki (github.com)
• FindingPython · pyscripter/python4delphi Wiki (github.com) with reference to Anaconda?

Thanks for the reply!

 

Yes, and I've taken them into account. As I stated above, my app works on my Windows 11 machine, but not on my client's Windows 10 machine. Obviously, there are nearly infinite differences, but I don't think most are relevant. There is one final location where I could have made a mistake (the config file where my code reads the location of the Python installation), but I don't think that is the issue; I'll know soon enough.

 

Regarding your second link, here is the most relevant text:

Quote

• If you want to use a specific unregistered version set the following properties
  • DLLName e.g. python38.dll
  • RegVersion e.g 3.8
  • Set UseLastKnownVerion property to False
  • Set the DLLPath to the path where the DLL is located
  • Set the AutoLoad property to False
  • Add the following statement to the PythonEngine OnBeforeLoad event handler, assuming that PythonEngine is the name of the component

  
    PythonEngine.SetPythonHome('your python installation directory')

  • Add the following statement to your FormCreate handler:

  
    PythonEngine.LoadDll;

All of those items have been addressed, with one exception; RegVersion is not set in my code, and seems to default to 3.11. The Anaconda installation we are using (same in both locations) is 3.9.12, so that could be a problem - and I will fix that! - but it doesn't seem to explain why it doesn't fail to work on my machine! To be clear, both installations use the same (incorrect) 3.11 for this setting. Tomorrow morning I will check the path and add this fix for RegVersion, but I don't expect it to make I difference - of course, I hope I'm wrong!

 

One little detail that could be an issue: we had installed Anaconda using admin rights to make it available to all users, which the wiki calls a "registered" version. As indicated above, I am following the "specific unregistered version" (SUV) instructions. I don't think this is the cause, as it appears to me that SUV is a superset of the "specific registered version" (SRV) approach. IIRC I had some trouble using SRV when I got started with P4D and Anaconda (I had both an existing Python version and added Anaconda), which I resolved by switching to the SUV approach. If you think using SUV is wrong with a registered version, I can revisit that.

[RSG 0]

Everything checks out but numpy still won't load. Here is the complete error message it gives me:

 

Quote

C:\ProgramData\Anaconda3\lib\site-packages\numpy\__init__.py:148: UserWarning: mkl-service package failed to import, therefore Intel(R) MKL initialization ensuring its correct out-of-the box operation under condition when Gnu OpenMP had already been loaded 
by Python process is not assured. Please install mkl-service package, see http://github.com/IntelPython/mkl-service
  from . import _distributor_init
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\site-packages\numpy\core\__init__.py", line 22, in <module>
    from . import multiarray
  File "C:\ProgramData\Anaconda3\lib\site-packages\numpy\core\multiarray.py", line 12, in <module>
    from . import overrides
  File "C:\ProgramData\Anaconda3\lib\site-packages\numpy\core\overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
ImportError: DLL load failed while importing _multiarray_umath: The specified module could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\ProgramData\Anaconda3\lib\site-packages\oct2py\__init__.py", line 27, in <module>
    from .core import Oct2Py
  File "C:\ProgramData\Anaconda3\lib\site-packages\oct2py\core.py", line 13, in <module>
    import numpy as np
  File "C:\ProgramData\Anaconda3\lib\site-packages\numpy\__init__.py", line 150, in <module>
    from . import core
  File "C:\ProgramData\Anaconda3\lib\site-packages\numpy\core\__init__.py", line 48, in <module>
    raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.9 from "C:\Program Files (x86)\RSGAssociates\ICE_v6-15-7\ice.exe"
  * The NumPy version is: "1.21.5"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: DLL load failed while importing _multiarray_umath: The specified module could not be found.

I checked the mkl-service "warning", and found it weird; when I attempted to install it, conda said it already was installed, and the warning (and the numpy error) persist.

 

The Python version is correct, though it seems weird to show my app as the "from" output. However, the only other Python on the machine is definitely different; also, if I check the version first before attempting to import from Oct2Py, I see the expected 3.9.12. 

 

Numpy version similarly checks out.

 

Any other ideas? I'm back to Google as I'm blocked on it now...