shineworld

Hi all. What I love of Delphi is memory management and overall the integration of FastMM and related memory leak management. All my projects are Memory Leak Free (just thanks to reports of FastMM which permits me to create a better code). Now I'm migrating some programs to Python using DelphiVCL and I'm falling into odd memory leaks. In a Python (3.9.12) application that uses DelphiVCL (0.1.40) I've noticed a continuous memory leak in some simple operations with VCL objects like Checkbox. I've attached a very simple Python program that uses a Timer to constantly update the Enabled state of a Checkbox. In the sample, I've also added a const to enable a tracemalloc and confirm the continue grow of memory used by <Checkbox>.Enabled = True # enable/disable tracemalloc to exclude the memory impact of tracemalloc TRACEMALLOC_ENABLED = False By default, TRACEMALLOC_ENABLED is set to False to remove any impact of tracemalloc framework. If enabled, the first mouse down on the form capture the BASE snapshot of memory. Any following mouse down, report on console the comparison of current snapshot with the first. Initially, tracemalloc internals is in the top ten results, but after some time Checkbox1.Enabled = True gain the top. If I disable the interesting lines, commenting on them, any memory leak disappears: def __on_timer(self, sender): """ self.CheckBox1.Enabled = True self.CheckBox1.Enabled = True self.CheckBox1.Enabled = True self.CheckBox1.Enabled = True self.CheckBox1.Enabled = True self.CheckBox1.Enabled = True self.CheckBox1.Enabled = True self.CheckBox1.Enabled = True self.CheckBox1.Enabled = True self.CheckBox1.Enabled = True self.CheckBox1.Enabled = True self.CheckBox1.Enabled = True self.CheckBox1.Enabled = True """ I've placed many CheckBox1.Enabled to exasperate the case-test but usually, I do this for a lot of controls in update events. This is the memory usage captured with Process Explorer on running Python process with TRACEMALLOC_ENABLED = False: test.zip

Great ! Monday morning I will try!

a) YES without any code in __on_timer(...) I have no memory leaks. b) YES if I add any operations with a checkbox in __on_timer() the memory leaks begin. c) I've used a const to DISABLE trace malloc os I can exclude that memory-leaks depends by it. d) YES the graphical images posted are without the use of trace malloc but are a bird-eye view of the python windows process memory consumption. In the test, I've checked only CheckBox.Enabled, to be simple, but in the full project, I've got some memory leaks also in <vcl>.Caption = etc. The memory leaks began when I switched from PySimpleGUI to DelphiVCL UI. The project uses a lot of DelphiVCL components so, in running time, the memory leak increases a lot. A program, at first start, uses 139MB, but after two hours it reaches 700MB or memory usage and trace malloc suggests that is in DelphiVCL. In the below video, I try to show the behavior. The system is simple. An embedded board with a camera, in this case, an RPI4, captures the camera frames and implements a server service to obtain an IP-CAMERA. To simplify the case is written in Python. A PC program, with Python + DelphiVCL + OpenCV + Skia connects to IP-Camera, gets frames, and applies code to elaborate it. In the test code, I've added trace malloc with a first memory snapshot (take_snaphot) at the start. When I press the middle button on the frame view I capture a memory snapshot that I compare with the start snapshot to show the top 10 memory consumers. It is normal that no VCL objects reach temporary the top because I capture some frames and I take them for future uses but looking on the counter is low. What I see is the usage memory and allocated counter increase continuously in some DelphiVCL operations. To be honest, I don't know so wheel Python and inners, I've begun to use it some time ago to try to port some apps to Python language on customer request, I'm a Delphi developer. So I can have mistakes on how to use tracemalloc to catch memory leaks. https://www.youtube.com/watch?v=rhUniVnXgEQ

It is embarrassing but I was not able to set a multiline text in a Button Caption. Usually in Delphi is only necessary to: MyButton.Caption = 'first line' + #13#10 + 'second line' What to get same behaviour in Python + DelphiVCL ?

Thanks for the replies. I've also missed to set WordWrap to True in button properties 🙂 Happens!

As like as David Heffernan I use the embedded python version for our customers. You can install pip in the embedded version so the end-user can add new packages or update packages from whl or Pypi. The embedded way is perfect to have an isolated python in which trust to deploy software for end-user. For example, my embedded version contains right OpenCV, NumPy, delphivcl/fmx, and support packages that I will use to create the end-user python program avoiding any conflicts with different versions already installed in other customer python installations and without use conda, etc.

Actually, I've used the simplest way to send an image to Python, send it completely (header, image structure, data). This required a python cv2.imdecode to get back a NumPy array clean of container (GIF/BMP/JPG/PNG/etc). You can use GetDIB in Delphi to extract only pure image data (RGB or RGBA), packet it, and return it to python then reorganize data in NumPy without the use of cv2.imdecode. TAKE CARE ======= In your time test for ByteIO, you have an overhead of cv2.imdecode.... Try without it.

Attached to the post there is a very simple Delphi application that should help you. The demo creates a PythonEngine and adds a new Module called delphi_vcl_ext in which wraps two functions: get_loaded_image_as_bytes() # get delphi loaded image as bytes update_image_from_bytes(...) # update delphivcl Image object from bytes array with width, height & channels The program has two panels: - Left Panel is a TImage and shows the loaded image to transfer to the Python script. - Right Panel is a TImage to show the python script evaluated image. By default, the program preloads a test BMP file (640x480 so fits the left image panel). Default script: - Get Delphi loaded image using get_loaded_image_as_byte(). - Decode the image to a NumPy array. - Apply an automatic canny filter. - Send back to Delphi the resulting image to be shown in the right panel. The time to transfer images is shown in the right log panel. With Load Image, you can try other files but must be supported by the TImage component. After all, is only a demo code made for you during rest time. Take care If you call script in a Delphi Thread you can't write python sent image directly in a TImage, or in any VCL component but use rightly Thread.Synchronized method to be done in the main thread. delphi_python_001.7z

I've to move IMemento and IPersistable to JSON. I've already made that in Python and moving to Delphi should be simple. At moment I cannot do that, current project timings..... cnc_memento.py Delphi, Sydeny in my case, have a full JSON support. Here an example of element read to compare with python way.... Delphi JSON is very close to dict in python and so to related python json: function TAPITCPEngineClient.GetAxesInfo: TAPIAxesInfo; var Request: string; Response: string; JSONValue: TJSONValue; begin try if not GetActive then AbortFast; Result.Init; Request := '{"get":"axes.info"}'; Response := SendCommand(Request); if Response = '' then Exit; JSONValue := TJSONObject.ParseJSONValue(Response); try if not (JSONValue is TJSONObject) then Exit; if not JSONValue.TryGetValue<TAPIAxesArray>('["res"]["joint.position"]', Result.JointPosition.V) then AbortFast; if not JSONValue.TryGetValue<TAPIAxesArray>('["res"]["machine.position"]', Result.MachinePosition.V) then AbortFast; if not JSONValue.TryGetValue<TAPIAxesArray>('["res"]["program.position"]', Result.ProgramPosition.V) then AbortFast; if not JSONValue.TryGetValue<TAPIAxesArray>('["res"]["machine.target.position"]', Result.MachineTargetPosition.V) then AbortFast; if not JSONValue.TryGetValue<TAPIAxesArray>('["res"]["program.target.position"]', Result.ProgramTargetPosition.V) then AbortFast; if not JSONValue.TryGetValue<TAPIAxesSpeedArray>('["res"]["actual.velocity"]', Result.ActualVelocity.V) then AbortFast; if not JSONValue.TryGetValue<Cardinal>('["res"]["working.wcs"]', Result.WorkingWCS) then AbortFast; if not JSONValue.TryGetValue<TAPIAxesArray>('["res"]["working.offset"]', Result.WorkingOffset.V) then AbortFast; if not JSONValue.TryGetValue<Boolean>('["res"]["homing.done"]', Result.HomingDone) then AbortFast; if not JSONValue.TryGetValue<Cardinal>('["res"]["homing.done.mask"]', Result.HomingDoneMask) then AbortFast; Result.HasData := True; finally JSONValue.Free; end; except Result.Init; end; end; JSon is simplest than XML but more readable.

Usually, I use osMemento with BDS2006 (which is IDENTICAL to Delphi7) and Sydney. MSXML DOM is also very fast. With latest software, however, I'm migrating my settings files to JSON so I can open them in Python, where I've another implementation of Memento very close to Delphi version but for JSON.

CreateChild creates a new child empty node overwriting the eventual existent node. CreateChildSmart at first check if the child already exists and return it, maintaining contents otherwise it creates a new one. PS: I've missed a unit in git: osExceptionUtils.pas osExceptionsUtils add a new fast Abort (AbortFast) that I use often, but in source, you can replace any AbortFast with a native Abort.

If compile for Windows you can use MSXML directly to load/get/set/DOM and save. Here you can find how I use MSXML with Delphi implementing a light version of the Memento pattern: https://github.com/shineworld/memento Example of use in a system to load/save recently opened files: unit osMRUManager; interface uses osIMemento; type TMRUManager = class private FBackupPath: string; FCount: Integer; FItems: array of string; FMaxItems: Integer; private function GetItems(Index: Integer): string; procedure SetMaxItems(Value: Integer); public procedure Clear; procedure Delete(Index: Integer); function LoadFromFile(const FileName: string): Boolean; function LoadFromMemento(Memento: IMemento): Boolean; procedure Push(const Item: string); function SaveToFile(const FileName: string): Boolean; function SaveToMemento(Memento: IMemento): Boolean; procedure ValidateItems; public constructor Create; public property BackupPath: string read FBackupPath write FBackupPath; property Count: Integer read FCount; property Items[Index: Integer]: string read GetItems; property MaxItems: Integer read FMaxItems write SetMaxItems; end; implementation uses System.SysUtils, osIPersistable, osXMLMemento, osExceptionUtils; const DEF_MAX_ITEMS = 8; constructor TMRUManager.Create; begin // sets default members values FBackupPath := ''; FCount := 0; FItems := nil; FMaxItems := 0; // sets initial max items MaxItems := DEF_MAX_ITEMS; end; procedure TMRUManager.Delete(Index: Integer); var I: Integer; begin if (Index < 0) or (Index >= FMaxItems) then Exit; if (Index >= FCount) then Exit; for I := Index to FCount - 2 do FItems[I] := FItems[I + 1]; FItems[FCount - 1] := ''; Dec(FCount); end; procedure TMRUManager.Clear; begin FCount := 0; end; function TMRUManager.GetItems(Index: Integer): string; begin if (Index < 0) or (Index >= Count) then Result := '' else Result := FItems[Index]; end; function TMRUManager.LoadFromFile(const FileName: string): Boolean; var Memento: IMemento; BackupFileName: string; function GetBackupFileName: string; begin try if FBackupPath = '' then AbortFast; if not DirectoryExists(FBackupPath) then AbortFast; Result := FBackupPath + ExtractFileName(FileName); except Result := ''; end; end; function InternalLoadFromFile(const FileName: string): Boolean; begin try Memento := CreateReadRoot(FileName); if Memento = nil then AbortFast; if Memento.GetName <> 'mru_root' then AbortFast; if not LoadFromMemento(Memento) then AbortFast; Result := True; except Result := False; end; end; begin Clear; try BackupFileName := GetBackupFileName; Result := InternalLoadFromFile(FileName); if Result then begin if BackupFileName <> '' then SaveToFile(BackupFileName); Exit; end; Result := InternalLoadFromFile(BackupFileName); if not Result then AbortFast; SaveToFile(FileName); Result := True; except Clear; Result := False; end; end; function TMRUManager.LoadFromMemento(Memento: IMemento): Boolean; var I: Integer; W: TXMLString; MainNode: IMemento; FileNodes: IMementoArray; begin Clear; try MainNode := Memento.GetChild('mru'); if MainNode = nil then AbortFast; FileNodes := MainNode.GetChildren('file'); if FileNodes = nil then AbortFast; if Length(FileNodes) > MaxItems then AbortFast; FCount := Length(FileNodes); for I := 0 to FCount - 1 do begin if not FileNodes[I].GetString('name', W) then AbortFast; FItems[I] := W; end; Result := True; except Clear; Result := False; end; end; procedure TMRUManager.Push(const Item: string); var I: Integer; J: Integer; begin for I := 0 to FCount - 1 do begin if FItems[I] = Item then begin if I = 0 then Exit; for J := I downto 1 do FItems[J] := FItems[J - 1]; FItems[0] := Item; Exit; end; end; if FCount < FMaxItems then Inc(FCount); for I := FCount - 1 downto 1 do FItems[I] := FItems[I - 1]; FItems[0] := Item; end; function TMRUManager.SaveToFile(const FileName: string): Boolean; var Memento: IMemento; BackupFileName: string; function GetBackupFileName: string; begin try if FBackupPath = '' then AbortFast; if not DirectoryExists(FBackupPath) then AbortFast; Result := FBackupPath + ExtractFileName(FileName); except Result := ''; end; end; function InternalSaveToFile(const FileName: string): Boolean; begin try Memento := CreateWriteRoot('mru_root'); if not SaveToMemento(Memento) then AbortFast; if not (Memento as IPersistable).SaveToFile(FileName, nrmd_UTF8, False) then AbortFast; Result := True; except Result := False; end; end; begin try BackupFileName := GetBackupFileName; if not InternalSaveToFile(FileName) then AbortFast; if BackupFileName <> '' then InternalSaveToFile(BackupFileName); Result := True; except Result := False; end; end; function TMRUManager.SaveToMemento(Memento: IMemento): Boolean; var I: Integer; Node: IMemento; MainNode: IMemento; begin try MainNode := Memento.CreateChildSmart('mru'); for I := 0 to Count - 1 do begin Node := MainNode.CreateChild('file'); Node.PutString('name', FItems[I]); end; Result := True; except Result := False; end; end; procedure TMRUManager.SetMaxItems(Value: Integer); var I: Integer; begin if FMaxItems = Value then Exit; SetLength(FItems, Value); for I := FMaxItems to Value - 1 do FItems[I] := ''; FMaxItems := Value; if FCount > FMaxItems then FCount := FMaxItems; end; procedure TMRUManager.ValidateItems; var I: Integer; begin I := FCount - 1; while I >= 0 do begin if not FileExists(FItems[I]) then Delete(I); Dec(I); end; end; end.

To simplify Python + DelphiVCL + owner Delphi PYD modules for our customer I would like to use the embedded version of Python. When the customer installs our software it installs also an embedded version of Python in the known path so do not touch customer versions or virtual environments. So I can have a customer with a full install of Python 3.10.1 in: C:\Users\silverio.di.QEMSRL\AppData\Local\Programs\Python\Python310\ and our embedded Python 3.10.4 with pre-installed custom modules at: D:\x\develop\qem\cnc_vision_1\python\ The Embedded Python version has PIP and all related modules as well as DelphiVCL, Skia, OpenCV, etc. When I run the embedded Python version I can import delphivcl, skia, cv2, etc without issue BUT not our PYD made with Delphi. However, the same module, called cnc_vision_ext.cp310-win_amd64.pyd, is loaded without issue with the full Python version... Could be that the embedded python distro is missing some lib or basic module required to load it? I've captured the CMD window reported below. Any suggestion? D:\x\develop\qem\cnc_vision_1\python>dir Il volume nell'unità D è Volume Numero di serie del volume: 48BE-3E0C Directory di D:\x\develop\qem\cnc_vision_1\python 04/05/2022 18:09 <DIR> . 04/05/2022 18:09 <DIR> .. 03/05/2022 16:58 5.347.328 cnc_vision_ext.cp310-win_amd64.pyd 22/10/2021 18:30 2.159.352 get-pip.py 04/05/2022 18:05 <DIR> Lib 23/03/2022 23:22 3.439.512 libcrypto-1_1.dll 23/03/2022 23:22 32.792 libffi-7.dll 23/03/2022 23:22 698.784 libssl-1_1.dll 23/03/2022 23:22 32.763 LICENSE.txt 23/03/2022 23:22 194.000 pyexpat.pyd 23/03/2022 23:22 589.053 python.cat 23/03/2022 23:22 99.280 python.exe 23/03/2022 23:22 62.416 python3.dll 23/03/2022 23:22 4.445.648 python310.dll 23/03/2022 23:23 2.638.493 python310.zip 04/05/2022 18:04 79 python310._pth 23/03/2022 23:22 97.744 pythonw.exe 04/05/2022 18:06 302 requirements.txt 04/05/2022 18:06 <DIR> Scripts 23/03/2022 23:22 26.064 select.pyd 23/03/2022 23:22 1.476.048 sqlite3.dll 23/03/2022 23:22 1.118.672 unicodedata.pyd 23/03/2022 23:22 98.224 vcruntime140.dll 23/03/2022 23:22 37.256 vcruntime140_1.dll 23/03/2022 23:22 27.088 winsound.pyd 23/03/2022 23:22 61.392 _asyncio.pyd 23/03/2022 23:22 79.824 _bz2.pyd 23/03/2022 23:22 119.760 _ctypes.pyd 23/03/2022 23:22 248.272 _decimal.pyd 23/03/2022 23:22 124.368 _elementtree.pyd 23/03/2022 23:22 60.880 _hashlib.pyd 23/03/2022 23:22 154.064 _lzma.pyd 23/03/2022 23:22 40.912 _msi.pyd 23/03/2022 23:22 30.672 _multiprocessing.pyd 23/03/2022 23:22 46.032 _overlapped.pyd 23/03/2022 23:22 27.600 _queue.pyd 23/03/2022 23:22 75.216 _socket.pyd 23/03/2022 23:22 94.672 _sqlite3.pyd 23/03/2022 23:22 156.624 _ssl.pyd 23/03/2022 23:22 21.456 _uuid.pyd 23/03/2022 23:22 40.912 _zoneinfo.pyd 04/05/2022 18:04 <DIR> __archive__ 37 File 24.003.554 byte 5 Directory 206.815.965.184 byte disponibili D:\x\develop\qem\cnc_vision_1\python>C:\Users\silverio.di.QEMSRL\AppData\Local\Programs\Python\Python310\Python.exe Python 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, 23:13:41) [MSC v.1929 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import delphivcl as vcl >>> import cnc_vision_ext as ext >>> quit() D:\x\develop\qem\cnc_vision_1\python>.\python.exe Python 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, 23:13:41) [MSC v.1929 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import delphivcl as vcl >>> import cnc_vision_ext as ext Traceback (most recent call last): File "<stdin>", line 1, in <module> SystemError: initialization of cnc_vision_ext did not return an extension module >>> quit() D:\x\develop\qem\cnc_vision_1\python>

Could be interesting, to have time to write something about.... Natively the Delphi project is around 10 million lines and with the introduction of expandability via Python I don't know where it will end 🙂 Fortunately, in an I7, Delphi is so fast that it takes just over 3 minutes to compile the whole project. In another language, I could go for a quick lunch.

I work in a company that makes control boards for CNC and related control environments on PC, but I don't physically own a CNC 🙂 However, we also drive 5-axis CNCs and more. The PC control part has always been done with Delphi and I couldn't be happier. I have abandoned other development environments and find that as fast and productive as Delphi there is no other, at least for this sector. I work in software/firmware development for embedded boards and proprietary real-time OS but in recent years most of my time is spent with my nose in Delphi pascal 🙂 I'm starting to get old to still work with assembler and C...

However, in my case, Python + OpenCV + DelphiVCL is very very very fast in a big range of cases. Previously UI was TkInter, followed by PsSimpleGUI but very limited in UI things.

Very interesting the vision lab project.

To communicate between Delphi pure UI process and Python process I've implemented an API Server, based on TCP/IP + JSON encrypted messages. A Python package (cnc-api-client-core in PyPi) implements the same protocol (API Client). CNC Board covers any type of CNC (plasma, laser, mill, etc, till 9-Axis in EtherCAT Drivers or 6-Axis Step/DIR Drivers). With Python API Client interface I can do anything on CNC Control Software and CNC Board, exposing CNC features to any Python UI interface. Here an old example of Python process which use API Client and PySimpleGUI UI to interact with CNC: Now UI was moved to DelphiFMX to be used in Windows and Linux OS.

A moment I've tried to create a simple framework package for Python language. The goal is to add vision features such as finding two printer markes to rotate and scale a CAD/CAM toolpath before running in a CNC. UI main program is a pure Delphi program. The called external program is a Python + delphivcl + opencv + skia and more other packages.

I have a lot, pretty much everything, to learn about Python, importing and building modules and packages. I had NEVER used Python or had never been interested in Python before the appearance and knowledge of Python4Delphi, some months ago. Since Embarcadero made DelphiVCL and Python4Delphi available, I have thrown myself headlong into bringing code that normally runs on native Delphi programs but standalone on Delphi modules and extensions for Python so that customers can make their own customizations via Python. Now, looking at how DelphiVCL initializes the Python engine I found that I have misunderstood a lot of things. Solve the problem is based on changing: ExtensionManager.FEngine.UseLastKnownVersion := True; ExtensionManager.FEngine.OpenDll(''); to ExtensionManager.FEngine.UseLastKnownVersion := True; ExtensionManager.FEngine.LoadDllInExtensionModule(); Obviously, in the end, I will collect any PYD (actually 32 Pyds) in one PyPi wheel install package so all Pyds will be in the Lib\ite-packages folder like Embarcadero does with DelphiVCL. The resulting framework will use opencv to capture, filter, and evaluate images to find specific objects, measure sizes, etc. At moment I don't use any form of AI to do that to remain fast and simple.

Following your suggestion, I've solved using PyBytes_FromStringAndSize... very very fast!!! 1] I've created a VideoCapture as native Delphi component TCncVisionCapturePxc. This can be used for standard Delphi programs or wrapped in an expansion module pyd to be used in Python. In this class, I can't use python objects, obviously as return values but the dynamic array of bytes for a frame. 2] I wrapped the class to export in pyd adding an extra method to get fastly the frame with PyBytes_FromStringAndSize: uses ... osCncVideoCapturePxc; type TPyCncVisionVideoCapturePxc_Wrapper = class(TPyDelphiObject) private constructor Create(APythonType: TPythonType); override; constructor CreateWith(PythonType: TPythonType; args: PPyObject); override; function Repr: PPyObject; override; class function DelphiObjectClass: TClass; override; class procedure RegisterMethods(PythonType: TPythonType); override; function GrabbedFrameEx(pself, args: PPyObject): PPyObject; cdecl; end; ... { TPyCncVisionVideoCapturePxc_Wrapper } constructor TPyCncVisionVideoCapturePxc_Wrapper.Create(APythonType: TPythonType); begin inherited; DelphiObject := TCncVisionVideoCapturePxc.Create; Owned := False; end; constructor TPyCncVisionVideoCapturePxc_Wrapper.CreateWith(PythonType: TPythonType; args: PPyObject); begin inherited; //### end; class function TPyCncVisionVideoCapturePxc_Wrapper.DelphiObjectClass: TClass; begin Result := TCncVisionVideoCapturePxc; end; function TPyCncVisionVideoCapturePxc_Wrapper.Repr: PPyObject; begin Result := GetPythonEngine.PyUnicodeFromString('none to show at moment'); end; class procedure TPyCncVisionVideoCapturePxc_Wrapper.RegisterMethods(PythonType: TPythonType); begin inherited; with PythonType do begin AddMethod('grabbed_frame_ex', @TPyCncVisionVideoCapturePxc_Wrapper.GrabbedFrameEx, 'get grabbed frame as string' ); end; end; function TPyCncVisionVideoCapturePxc_Wrapper.GrabbedFrameEx(pself, args: PPyObject): PPyObject; cdecl; var S: AnsiString; FrameSize: Integer; Frame: TByteDynArray; VideoCapture: TCncVisionVideoCapturePxc; begin Adjust(@Self); VideoCapture := TCncVisionVideoCapturePxc(DelphiObject); Frame := VideoCapture.grabbed_frame; FrameSize := Length(Frame); if FrameSize = 0 then Exit(GetPythonEngine.ReturnNone); SetLength(S, FrameSize); CopyMemory(@S[1], Frame, FrameSize); Result := GetPythonEngine.PyBytes_FromStringAndSize(@S[1], FrameSize); end; Adjust(@Self) was a nightmare because I don't know about that and without the self access to DelphiObject is always invalid. Now from 80ms to transfer an image from Delphi to Python is moved to 128uS. I'm happy.

When I can I move thread things to Delphi Thread in an expansion module but this has a limited range of operations, you have to use only Delphi libraries, no Python code there. For example, I'm banging my head on a very odd problem without a solution. 1] In a Delphi extension module I get frames from a proprietary camera that can't be accessed with OpenCV VideoCapture, for a lot of reasons. 2] A Python thread gets the frame data from the Delphi extension module as a property to TByteDynArray data. TByteDynArray is converted to List by PythonWrapper and the inner system copies any byte of the array in a VariantArray which takes a big amount of time. To get back from the Delphi module 48000 image bytes take 0.080 (80ms) which is too much, slowing down a lot the FPS. A classical bottleneck..... I've tried to change the FrameData returned from TByteDynArray to AnsiString and data movement is very very fast (0.001ms) but AnsiString with Frame Data is encoded to Unicode before sending it back to python and str.encode(frame_data) in Python is not applicable.... So my question? What is the best way to get an array of bytes from the Delphi extension module (pyd) back to python? Have I to enter in the HELL of array pointers shared between Delphi to Python ? Delphi extension module is able to capture till 60fps, but the bottleneck of frame retrieve from PYD to Python fall down my FPS to 6FPS...

I'm not a Win guru but Windows has owner rules on how much time is left to a process and related task. Usually, it does not permit to use of more than 40/50% of the time to a process/thread depending on many factors. GIL Lock system however limits a lot of threading performances in Python, is not permit a true Threading programming as like as in other languages . When possibile I move python math code in delphi extension module native threads or I use Cython to improve moudule performances. To improve a little the performances of time.Sleep() and timers and so of GIL I've tried to change the precision timer of Windows with: from os_utils import os_improve_timings ... .. . def main(): """Main entry point.""" ... .. . # MAIN entry point when module is called if __name__ == '__main__': # improve os timings and run main with os_improve_timings(): main() os_utils.py

Do you have tried the tutorials from Webinar II? Perhaps they cover your question. https://github.com/pyscripter/python4delphi/tree/master/Tutorials/Webinar II

There are many ways to "define" what Pythonxxx.DLL will be used by PythonEngine used in the Delphi application or in a PYD extension module made with Delphi. You can set "MyEngine.UseLaskKnownVersion := True" so the Engine search for 3.10, then 3.9, ... till the oldest version is supported. You can also "fix" a version leaving it to PythonEngine to search for the path. This is done thanks to fact that Python "stores" any installed Python version path in Windows Registry. {** * TAKE CARE * ========= * PYD modules need to be build for a specific Python version using below defines. * The defines MUST be mutually exclusive. * If not defined the "Use Last Known Version" will be used. * **} {$DEFINE USE_PYTHON_39} {.DEFINE USE_PYTHON_310} {$IF Defined(USE_PYTHON_39) and Defined(USE_PYTHON_310)} {$Message Fatal 'Only one Python version can be set.'} {$IFEND} { module import functions } function PyInit_cnc_vision_ext: PPyObject; begin Result := nil; try ExtensionManager.FEngine := TPythonEngine.Create(nil); ExtensionManager.FEngine.AutoFinalize := False; {$IF Defined(USE_PYTHON_39)} ExtensionManager.FEngine.DLLName := 'python39.dll'; ExtensionManager.FEngine.UseLastKnownVersion := False; ExtensionManager.FEngine.OpenDll(ExtensionManager.FEngine.DLLName); {$ELSEIF Defined(USE_PYTHON_310)} ExtensionManager.FEngine.DLLName := 'python310.dll'; ExtensionManager.FEngine.UseLastKnownVersion := False; ExtensionManager.FEngine.OpenDll(ExtensionManager.FEngine.DLLName); {$ELSE} ExtensionManager.FEngine.UseLastKnownVersion := True; ExtensionManager.FEngine.OpenDll(''); {$IFEND} ExtensionManager.FModule := TPythonModule.Create(nil); ExtensionManager.FModule.Engine := ExtensionManager.FEngine; ExtensionManager.FModule.ModuleName := 'cnc_vision_ext'; ExtensionManager.FWrapper := TPyDelphiWrapper.Create(nil); ExtensionManager.FWrapper.Engine := ExtensionManager.FEngine; ExtensionManager.FWrapper.Module := ExtensionManager.FModule; ExtensionManager.FModule.Initialize; ExtensionManager.FWrapper.OnInitialization := ExtensionManager.WrapperInitializationEvent; ExtensionManager.FWrapper.Initialize; Result := ExtensionManager.FModule.Module; except end; end; This is a part of my extension module in which I can define if the module is built for 3.9, 3.10, or looking for the last known version. OpenDLL internally searches in Windows Registry where the requested python engine path is. Take a look at file: PythonVersions.pas GetRegisteredPythonVersion() function there is also a good description of system. https://github.com/pyscripter/python4delphi/wiki/FindingPython I don't know how works with Anaconda envs or embedded python distros.