Slow loading of numpy in EmbeddedResEnvironment

[JGMS 2]

After adding a new function that uses the "Numpy" and "CV2" libraries, lots of DLLs are loaded when the function is called. See "code" below.

Initially, Delphi (or Python) complained because of missing the libraries. Then I copied the Numpy folder from the Python/3.10/site-packages of the machine to the ditto folders near the program.

Apart from slowing down loading, the function fails, although it runs perfect in PyScripter.

 

Is copying the folders Numpy and OpenCV sufficient? Or better: when compiling goes without complaining, could anything still be missing?

 

Here are some of the messages, shown here in hope for that it rings a bell in anyone of you. If so, please let me know.

Thanks ahead,

  jan

Module Load: libffi-7.dll. No Debug Info. Base Address: $00007FF93C590000. Process FotoPower.exe (11132)
Module Load: VCOMP140.DLL. No Debug Info. Base Address: $00007FF923BC0000. Process FotoPower.exe (11132)
Module Load: MSVCP140.dll. No Debug Info. Base Address: $00007FF91FA50000. Process FotoPower.exe (11132)
Module Load: _check_build.cp310-win_amd64.pyd. No Debug Info. Base Address: $00007FF92F730000. Process FotoPower.exe (11132)

onecore\com\combase\dcomrem\resolver.cxx(2414)\combase.dll!00007FF93FCA6FFA: (caller: 00007FF93FC0B9F8) ReturnHr(58) tid(2dc0) 80070005 Toegang geweigerd.



Module Load: libopenblas64__v0.3.21-gcc_10_3_0.dll. Has Debug Info. Base Address: $00007FF8E9DA0000. Process FotoPower.exe (11132)
Module Load: _multiarray_umath.cp310-win_amd64.pyd. No Debug Info. Base Address: $00007FF8F19D0000. Process FotoPower.exe (11132)
Module Load: _multiarray_tests.cp310-win_amd64.pyd. No Debug Info. Base Address: $00007FF929EA0000. Process FotoPower.exe (11132)
Module Load: _umath_linalg.cp310-win_amd64.pyd. No Debug Info. Base Address: $00007FF923BA0000. Process FotoPower.exe (11132)
Module Load: _pocketfft_internal.cp310-win_amd64.pyd. No Debug Info. Base Address: $00007FF922860000. Process FotoPower.exe (11132)
Module Load: mtrand.cp310-win_amd64.pyd. No Debug Info. Base Address: $00007FF9197E0000. Process FotoPower.exe (11132)
Module Load: bit_generator.cp310-win_amd64.pyd. No Debug Info. Base Address: $00007FF921870000. Process FotoPower.exe (11132)
Module Load: _common.cp310-win_amd64.pyd. No Debug Info. Base Address: $00007FF9210D0000. Process FotoPower.exe (11132)
Module Load: _hashlib.pyd. No Debug Info. Base Address: $00007FF9221F0000. Process FotoPower.exe (11132)
Module Load: libcrypto-1_1.dll. No Debug Info. Base Address: $00007FF8F0440000. Process FotoPower.exe (11132)
Module Load: _bounded_integers.cp310-win_amd64.pyd. No Debug Info. Base Address: $00007FF9201C0000. Process FotoPower.exe (11132)

 

Edited April 21, 2023 by JGMS
Simple test program with Numpy and CV2 gave addional error (oncore\comcombas\...)e

[SwiftExpat 65]

29 minutes ago, JGMS said:

Is copying the folders Numpy and OpenCV sufficient? Or better: when compiling goes without complaining, could anything still be missing?

I used pip to install them into my embedded dist, that will check for any dependencies.

python -m pip install numpy
python -m pip install opencv-contrib-python

 

You should be able to diff using

--List modules in embedded
python -m pip list -v
--list modules from other distribution
pip list -v

 

[JGMS 2]

33 minutes ago, SwiftExpat said:

pip list -v

Both commands result in exactly the same list, though the contents of the folders are quite different.
Can you explain how to install to an embedded list?

[SwiftExpat 65]

This covers how to set it up, after that you can run pip against your embedded dist. Note, path is important and likely why you got the same results.

 

https://dev.to/fpim/setting-up-python-s-windows-embeddable-distribution-properly-1081

 

Read through this one as well,

 

Edited April 21, 2023 by SwiftExpat
add link to other thread

[KoRiF 1]

Try this. With proper installation of the whole stack of components, this should solve your problem automatically 

[JGMS 2]

18 hours ago, KoRiF said:

Try this. With proper installation of the whole stack of components, this should solve your problem automatically 

Thanks for your tip. I downloaded and unpacked it. I opened the file "P4DDataSciencesComponentSuite.groupproj" in Delphi and did "Build all Projects" of the group.

This succeeded when for Windows- 32bit. Is there nothing more that I should do upfront to get all of these libraries available in P4D projects? 

[KoRiF 1]

You have to install every of dcl***.dpk

But first you must install foundation libraries as said there:

 

 

• PythonEnviroments allows effectively manage [embedded] Python environments 
• Lightweight-Python-Wrappers allows easily wrap any python library as Delphi component and easily use it in any Object Pascal project

Read this for more concept understanding  

[JGMS 2]

On 4/23/2023 at 1:21 AM, KoRiF said:

You have to install every of dcl***.dpk

But first you must install foundation libraries as said there:

 

 

• PythonEnviroments allows effectively manage [embedded] Python environments 
• Lightweight-Python-Wrappers allows easily wrap any python library as Delphi component and easily use it in any Object Pascal project

Read this for more concept understanding  

Thank you KoRiF,
Being very unfamiliar with installing components, it took me quit some time to get things sorted out. I have the P4D sciences libs installed as components now. Next is getting them to work. I want OpenCV + Numpy to deal with image handling smoothly. Any tips for examples?
 

 

 

Edited April 24, 2023 by JGMS
Found out in the meantime

[JGMS 2]

On 4/21/2023 at 4:41 PM, SwiftExpat said:

This covers how to set it up, after that you can run pip against your embedded dist. Note, path is important and likely why you got the same results.

 

https://dev.to/fpim/setting-up-python-s-windows-embeddable-distribution-properly-1081

 

Read through this one as well,

 

Hi SwiftExpat,

 

I created a folder with an embedded python 3.11 version in it, and installed all required libraries for my projects. Now my Delphi application(s) all have "PythonEngine1.DllPath" set to this "embedded folder".
I installed the libraries from within with the  "Scripts" subfolder using commands like ".\pip install opencv-contrib-python --no-warn-script-location".
The command "python -m pip list -v" still returns the list of the libraries in the 3.10 system, though. Likely, that is due to the system's environment path setting, in where there is (and, I bet, should be) no entry added for the embedded version. I can live with that.
Next, I compressed everthing in a zip file "Python.zip", with the intention to unpack this as "3.11" aside the executable on other machines, using Inno Setup.
I can probably manage that, but after installation on the target machine the DLLPath should first be redirected to subfolder "3.11" in the path of the executable. On this one I am stuck.
 
Do you have tips to achieve this path change? Or do you have helpful comments on the approach I described?

Edited April 26, 2023 by JGMS

[SwiftExpat 65]

Glad to see you sticking with it and it sounds like you are close. I spent a long time on this but having it stable pays off in the end.

 

Something simple like this is what I use, so just change python to 3.11

2 hours ago, JGMS said:

I can probably manage that, but after installation on the target machine the DLLPath should first be redirected to subfolder "3.11" in the path of the executable. On this one I am stuck.

TPath.Combine(TPath.GetDirectoryName(paramstr(0)), 'python');

 

Pip should resolve packages in your embedded dist, this is why you use the package with the -m option. If it does not the dist is not importing site correctly and you are going to chase errors.

Here is my working, by changing to the directory it will use the python binary from that folder first, so double check with the double -V.  After that the concern is the location of the package, it should show in your embedded.

C:\Users\Coder\AppData\Local\Programs\RunTime_ToolKit\python>python -V -V
Python 3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 bit (AMD64)]

C:\Users\Coder\AppData\Local\Programs\RunTime_ToolKit\python>python -m pip list -v
Package               Version     Location                                                                       Installer
--------------------- ----------- ------------------------------------------------------------------------------ ---------
astroid               2.5         c:\users\coder\appdata\roaming\python\python39\site-packages                   pip
bcrypt                3.2.0       c:\users\coder\appdata\local\programs\runtime_toolkit\python\lib\site-packages pip

Now for the pth file, it should look like this:

python39.zip
.

.\Scripts
.\Lib
.\Lib\site-packages
# Uncomment to run site.main() automatically
import site

The last piece you will need is msvc runtime, python does not require it but likely a module you use will

python -m pip install msvc_runtime

Now you are not done yet, because Delphi wants to load that DLL from the same directory as your exe. So as part of your install just copy the dlls from python over to your exe dir.

I did this for the same question a while back, so here it is as reference:

[shineworld 68]

15 hours ago, JGMS said:

Hi SwiftExpat,

 

I created a folder with an embedded python 3.11 version in it, and installed all required libraries for my projects. Now my Delphi application(s) all have "PythonEngine1.DllPath" set to this "embedded folder".
I installed the libraries from within with the  "Scripts" subfolder using commands like ".\pip install opencv-contrib-python --no-warn-script-location".
The command "python -m pip list -v" still returns the list of the libraries in the 3.10 system, though. Likely, that is due to the system's environment path setting, in where there is (and, I bet, should be) no entry added for the embedded version. I can live with that.
Next, I compressed everthing in a zip file "Python.zip", with the intention to unpack this as "3.11" aside the executable on other machines, using Inno Setup.
I can probably manage that, but after installation on the target machine the DLLPath should first be redirected to subfolder "3.11" in the path of the executable. On this one I am stuck.
 
Do you have tips to achieve this path change? Or do you have helpful comments on the approach I described?

Take care:

You can have a Python embed version OUT of path which works but if you have the same version of Python installed normally

your embedded version pip list will reach ALSO a cached path (from the normally installed Python version) in %APPDATA%\Python\Python<version>\site-packages.

This does not overload your native Python embed local packages, just add others to pip visibility.

[KoRiF 1]

On 4/24/2023 at 6:19 PM, JGMS said:

I want OpenCV + Numpy to deal with image handling smoothly. Any tips for examples?
 

 

 

1. You could consider direct use of OpenCV for Delphi depending on your task and supposed "ecosystem". This is live project with working examples. There is also experimental version of OOP-oriented OpenCV
2. Usage of P4D-Data-Scinces does not preclude you from using Python in a script style. If you can deal with some namespace confusion ("global" python variables and functions are actually accessible on Delphi side via the name of the corresponding modules available via BuiltinModule() and MainModule(), you can experiment with print(dir(...)) ) then you may well use some mix (and in some cases even be forced to use it) - although this may not be the best idea if done haphazardly 
3. If you are strong in C++ you can try use OpenCV in Delphi through C++Builder intermediate adapter