SystemError: initialization of cnc_vision_ext did not return an extension module

[shineworld 73]

To simplify Python + DelphiVCL + owner Delphi PYD modules for our customer I would like to use the embedded version of Python.
When the customer installs our software it installs also an embedded version of Python in the known path so do not touch customer versions or virtual environments.

So I can have a customer with a full install of Python 3.10.1 in: 
C:\Users\silverio.di.QEMSRL\AppData\Local\Programs\Python\Python310\

and our embedded Python 3.10.4 with pre-installed custom modules at:
D:\x\develop\qem\cnc_vision_1\python\

The Embedded Python version has PIP and all related modules as well as DelphiVCL, Skia, OpenCV, etc.
When I run the embedded Python version I can import delphivcl, skia, cv2, etc without issue BUT not our PYD made with Delphi.
However, the same module, called cnc_vision_ext.cp310-win_amd64.pyd, is loaded without issue with the full Python version...

Could be that the embedded python distro is missing some lib or basic module required to load it?

I've captured the CMD window reported below.

Any suggestion?
 

D:\x\develop\qem\cnc_vision_1\python>dir
 Il volume nell'unità D è Volume
 Numero di serie del volume: 48BE-3E0C

 Directory di D:\x\develop\qem\cnc_vision_1\python

04/05/2022  18:09    <DIR>          .
04/05/2022  18:09    <DIR>          ..
03/05/2022  16:58         5.347.328 cnc_vision_ext.cp310-win_amd64.pyd
22/10/2021  18:30         2.159.352 get-pip.py
04/05/2022  18:05    <DIR>          Lib
23/03/2022  23:22         3.439.512 libcrypto-1_1.dll
23/03/2022  23:22            32.792 libffi-7.dll
23/03/2022  23:22           698.784 libssl-1_1.dll
23/03/2022  23:22            32.763 LICENSE.txt
23/03/2022  23:22           194.000 pyexpat.pyd
23/03/2022  23:22           589.053 python.cat
23/03/2022  23:22            99.280 python.exe
23/03/2022  23:22            62.416 python3.dll
23/03/2022  23:22         4.445.648 python310.dll
23/03/2022  23:23         2.638.493 python310.zip
04/05/2022  18:04                79 python310._pth
23/03/2022  23:22            97.744 pythonw.exe
04/05/2022  18:06               302 requirements.txt
04/05/2022  18:06    <DIR>          Scripts
23/03/2022  23:22            26.064 select.pyd
23/03/2022  23:22         1.476.048 sqlite3.dll
23/03/2022  23:22         1.118.672 unicodedata.pyd
23/03/2022  23:22            98.224 vcruntime140.dll
23/03/2022  23:22            37.256 vcruntime140_1.dll
23/03/2022  23:22            27.088 winsound.pyd
23/03/2022  23:22            61.392 _asyncio.pyd
23/03/2022  23:22            79.824 _bz2.pyd
23/03/2022  23:22           119.760 _ctypes.pyd
23/03/2022  23:22           248.272 _decimal.pyd
23/03/2022  23:22           124.368 _elementtree.pyd
23/03/2022  23:22            60.880 _hashlib.pyd
23/03/2022  23:22           154.064 _lzma.pyd
23/03/2022  23:22            40.912 _msi.pyd
23/03/2022  23:22            30.672 _multiprocessing.pyd
23/03/2022  23:22            46.032 _overlapped.pyd
23/03/2022  23:22            27.600 _queue.pyd
23/03/2022  23:22            75.216 _socket.pyd
23/03/2022  23:22            94.672 _sqlite3.pyd
23/03/2022  23:22           156.624 _ssl.pyd
23/03/2022  23:22            21.456 _uuid.pyd
23/03/2022  23:22            40.912 _zoneinfo.pyd
04/05/2022  18:04    <DIR>          __archive__
              37 File     24.003.554 byte
               5 Directory  206.815.965.184 byte disponibili

D:\x\develop\qem\cnc_vision_1\python>C:\Users\silverio.di.QEMSRL\AppData\Local\Programs\Python\Python310\Python.exe
Python 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, 23:13:41) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import delphivcl as vcl
>>> import cnc_vision_ext as ext
>>> quit()

D:\x\develop\qem\cnc_vision_1\python>.\python.exe
Python 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, 23:13:41) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import delphivcl as vcl
>>> import cnc_vision_ext as ext
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
SystemError: initialization of cnc_vision_ext did not return an extension module
>>> quit()

D:\x\develop\qem\cnc_vision_1\python>

 

[SwiftExpat 65]

What is in python310._pth?

I think the 2nd line is relative to your exe, so it is not trustable.

python39.zip
.

.\Scripts
.\Lib
.\Lib\site-packages
# Uncomment to run site.main() automatically
import site

 

Personally I would put cnc_vision_ext.cp310-win_amd64.pyd in \Lib\site-packages\'your folder'\ and then modify the import statement.

[tomye 1]

8 hours ago, SwiftExpat said:

What is in python310._pth?

I think the 2nd line is relative to your exe, so it is not trustable.

python39.zip
.

.\Scripts
.\Lib
.\Lib\site-packages
# Uncomment to run site.main() automatically
import site

 

Personally I would put cnc_vision_ext.cp310-win_amd64.pyd in \Lib\site-packages\'your folder'\ and then modify the import statement.

 

it should be OK, my project needs cv2, onnxruntime etc. 

i just copy c:\programdata\anaconda3\lib\site-packages\cv2 ; c:\programdata\anaconda3\lib\site-packages\onnxruntime to {my project} \Libs\ 

and set the PyEngine.DLLPath to {my project}\Libs, that will be worked fine and the distribution as well

[shineworld 73]

I have a lot, pretty much everything, to learn about Python, importing and building modules and packages.
 

I had NEVER used Python or had never been interested in Python before the appearance and knowledge of Python4Delphi, some months ago.
 

Since Embarcadero made DelphiVCL and Python4Delphi available, I have thrown myself headlong into bringing code that normally runs on native Delphi programs

but standalone on Delphi modules and extensions for Python so that customers can make their own customizations via Python.

 

Now, looking at how DelphiVCL initializes the Python engine I found that I have misunderstood a lot of things.

 

Solve the problem is based on changing:
    ExtensionManager.FEngine.UseLastKnownVersion := True;
    ExtensionManager.FEngine.OpenDll('');

to
    ExtensionManager.FEngine.UseLastKnownVersion := True;
    ExtensionManager.FEngine.LoadDllInExtensionModule();

Obviously, in the end, I will collect any PYD (actually 32 Pyds) in one PyPi wheel install package so all Pyds will be in the Lib\ite-packages folder like Embarcadero does with DelphiVCL.

The resulting framework will use opencv to capture, filter, and evaluate images to find specific objects, measure sizes, etc.
At moment I don't use any form of AI to do that to remain fast and simple.

Edited May 5, 2022 by shineworld

[SwiftExpat 65]

2 hours ago, shineworld said:

I have a lot, pretty much everything, to learn

I feel the same way every time I try something different.

You already know a lot of python, now you are on to finishing details to make your app just a little better.

[SwiftExpat 65]

8 hours ago, tomye said:

it should be OK,

When you use python embedded, you have to be aware of the OS system path.  Assume that nothing is on the OS path and you will be ok.

 

I choose not to modify the path as part of my app install, and python is in a subdirectory to my exe.  This allows me to delete the python directory when I need to update.  This forces me to load everything relying only on site to find the module.

 

8 hours ago, tomye said:

i just copy c:\programdata\anaconda3\lib\site-packages\cv2 ; c:\programdata\anaconda3\lib\site-packages\onnxruntime to {my project} \Libs\ 

I install pip in the embedded distribution and let pip install the packages and dependencies. 

[shineworld 73]

A moment I've tried to create a simple framework package for Python language.

The goal is to add vision features such as finding two printer markes to rotate and scale a CAD/CAM toolpath before running in a CNC.

UI main program is a pure Delphi program.
The called external program is a Python + delphivcl + opencv + skia and more other packages.
 

 

[SwiftExpat 65]

It looks like you have a great start, what are you using for communication between the 2 processes?

 

Just curious, is the CNC a laser or mechanical machine?

[shineworld 73]

To communicate between Delphi pure UI process and Python process I've implemented an API Server, based on TCP/IP + JSON encrypted messages.

A Python package (cnc-api-client-core in PyPi) implements the same protocol (API Client).
CNC Board covers any type of CNC (plasma, laser, mill, etc, till 9-Axis in EtherCAT Drivers or 6-Axis Step/DIR Drivers).

 

With Python API Client interface I can do anything on CNC Control Software and CNC Board,
exposing CNC features to any Python UI interface.

Here an old example of Python process which use API Client and PySimpleGUI UI to interact with CNC:

Now UI was moved to DelphiFMX to be used in Windows and Linux OS.

Edited May 5, 2022 by shineworld

[tomye 1]

12 hours ago, shineworld said:

To communicate between Delphi pure UI process and Python process I've implemented an API Server, based on TCP/IP + JSON encrypted messages.

A Python package (cnc-api-client-core in PyPi) implements the same protocol (API Client).
CNC Board covers any type of CNC (plasma, laser, mill, etc, till 9-Axis in EtherCAT Drivers or 6-Axis Step/DIR Drivers).

 

With Python API Client interface I can do anything on CNC Control Software and CNC Board,
exposing CNC features to any Python UI interface.

Here an old example of Python process which use API Client and PySimpleGUI UI to interact with CNC:

Now UI was moved to DelphiFMX to be used in Windows and Linux OS.

if your requirements is only simple opencv detection recommend  the Mitov Lab Series, very strong, all native Delphi codes.

 

includes video decoder / encoder ,  machine vision etc. 

 

https://mitov.com/products/visionlab

 

 

 

[shineworld 73]

Very interesting the vision lab project.

[shineworld 73]

However, in my case, Python + OpenCV + DelphiVCL is very very very fast in a big range of cases.
Previously UI was TkInter, followed by PsSimpleGUI but very limited in UI things.

[tomye 1]

so cool ! your CNC machine is 5 Axis ?

 

[shineworld 73]

I work in a company that makes control boards for CNC and related control environments on PC, but I don't physically own a CNC 🙂

However, we also drive 5-axis CNCs and more.
 

The PC control part has always been done with Delphi and I couldn't be happier.
I have abandoned other development environments and find that as fast and productive as Delphi there is no other, at least for this sector.
 

I work in software/firmware development for embedded boards and proprietary real-time OS but in recent years most of my time is spent with my nose in Delphi pascal 🙂 
I'm starting to get old to still work with assembler and C...
 

[SwiftExpat 65]

36 minutes ago, shineworld said:

However, we also drive 5-axis CNCs and more.

I just had some time to look for examples of 9 axis and that is incredible work.

 

Your UI looks good and I think it is interesting to extend via python.

 

You should be Embarcadero's case study 🙂

[shineworld 73]

On 5/6/2022 at 6:47 PM, SwiftExpat said:

Your UI looks good and I think it is interesting to extend via python.

You should be Embarcadero's case study 🙂

Could be interesting, to have time to write something about....
Natively the Delphi project is around 10 million lines and with the introduction of expandability via Python I don't know where it will end 🙂

Fortunately, in an I7, Delphi is so fast that it takes just over 3 minutes to compile the whole project.
In another language, I could go for a quick lunch.

 

 

Edited May 11, 2022 by shineworld