Celebr0

Hi, for some reason, I'm experiencing massive memory leaks when reading a file and parallelizing by lines procedure ReadF(const fname: string); const BUFSIZE = 1024 * 18; var Line:string; List: TStringList; Reader: TStreamReader; Buf: array [0..BUFSIZE] of char; begin List := TStringList.Create; Reader := TStreamReader.Create(fname, TEncoding.Default, True, 4096); while not Reader.EndOfStream do begin Reader.ReadBlock(@Buf, 0, BUFSIZE); List.Text := Buf; parallel.&For(0, List.Count - 1).Execute( procedure(value: integer) begin Line:=List[value]; end); end; List.Free; Reader.Close; FreeAndNil(Reader); end;

Hi, where and to whom have I been disrespectful, arrogant, or insulting? Why are you blatantly lying about it ?

No, it looks like you don't understand multithreaded development at all if you're suggesting using critical sections. If you put critical sections and threads have to wait for each other every time, what's the point of using threads then?

How much are you betting ?

I checked and solved this issue, and yes, that is indeed the case: Memory Leak: procedure ReadF(const fname: string); const BUFSIZE = 1024 * 18; var Line:string; List: TStringList; Reader: TStreamReader; Buf: array [0..BUFSIZE] of char; begin List := TStringList.Create; Reader := TStreamReader.Create(fname, TEncoding.Default, True, 4096); while not Reader.EndOfStream do begin Reader.ReadBlock(@Buf, 0, BUFSIZE); List.Text := Buf; parallel.ForEach(0, List.Count - 1).Execute( procedure(value: integer) begin Line:=List[value]; end); end; List.Free; Reader.Close; FreeAndNil(Reader); end; No memory Leak: procedure ReadF(const fname: string); const BUFSIZE = 1024 * 18; var Line:string; List: TStringList; Reader: TStreamReader; Buf: array [0..BUFSIZE] of char; begin List := TStringList.Create; Reader := TStreamReader.Create(fname, TEncoding.Default, True, 4096); while not Reader.EndOfStream do begin Reader.ReadBlock(@Buf, 0, BUFSIZE); List.Text := Buf; parallel.ForEach(0, List.Count - 1).NoWait.Execute( procedure(value: integer) begin Line:=List[value]; end); end; List.Free; Reader.Close; FreeAndNil(Reader); end;

I think I figured it out when I was writing my own module for parallelizing the for loop, this is what I encountered: 1. There will be memory leaks: procedure ParallelFor(const AStart, AEnd: Integer; AProc: TProc<Integer>); var i, ThreadCount, RangeStart: Integer; begin ThreadCount:=GetCPUCount; RangeStart:=AStart-1; for i := 0 to ThreadCount - 1 do WaitForSingleObject(TThread.CreateAnonymousThread( procedure begin while RangeStart < AEnd do begin AProc(InterlockedIncrement(RangeStart)); end; end).Create(False).Handle, INFINITE); end; 2. There will be no memory leaks here: procedure ParallelFor(const AStart, AEnd: Integer; AProc: TProc<Integer>); var i, ThreadCount, RangeStart: Integer; begin ThreadCount:=GetCPUCount; RangeStart:=AStart-1; for i := 0 to ThreadCount - 1 do TThread.CreateAnonymousThread( procedure begin while RangeStart < AEnd do begin AProc(InterlockedIncrement(RangeStart)); end; end).Create(False); end; It looks like OtlParallel.For is written based on the same principle as in the first case !

Delphi XE2 I wrote my own module for parallelizing threads using the same code as here, and there are no memory leaks at all!

Working with memory, since it is fast, is always absolutely thread-safe. However, working with form1 for example Memo1, ListBox, and so on is not thread-safe. If you are implying that I should wrap Line := List[value]; in a critical section, it makes absolutely no difference—memory leaks still remain.

Who told you such nonsense? The variable will just be overwritten!

Hello, after the last update 5 days ago, python4delphi crashes immediately after launch: Delphi XE2

Hello, the problem has not yet been resolved. I downloaded the newest version of Python4Delphi from GitHub and Python 3.13.0, but it still crashes

Hello, I just started studying the OmniThreadLibrary library, but I can’t figure out how to parallelize this Regex procedure ? procedure pars(const Regex: TRegEx; const str: string); var Match: TMatch; begin Match := Regex.Match(str); //Parallel.ForEach(???).Execute( ??????????????? while Match.Success do begin //Match.Value; Match := Match.NextMatch; end; end;

That even Regex cannot be accelerated in any way through OmniLibrary ?

I guess I found something CRASH in AssignPyFlags(Config); from here: var i : Integer; Config: PyConfig; Status: PyStatus; ErrMsg: string; begin if Assigned(gPythonEngine) then raise Exception.Create('There is already one instance of TPythonEngine running' ); gPythonEngine := Self; FIORedirected := False; if FInExtensionModule then FInitialized := True else begin // Fills Config with zeros and then sets some default values if pfIsolated in FPyFlags then PyConfig_InitIsolatedConfig(Config) else PyConfig_InitPythonConfig(Config); try AssignPyFlags(Config); // CRASH HERE // Set programname and pythonhome if available if FProgramName <> '' then PyConfig_SetString(Config, PPWcharT(PByte(@Config) + ConfigOffests[MinorVersion, TConfigFields.program_name]), PWCharT(StringToWCharTString(FProgramName))); if FPythonHome <> '' then PyConfig_SetString(Config, PPWcharT(PByte(@Config) + ConfigOffests[MinorVersion, TConfigFields.home]), PWCharT(StringToWCharTString(FPythonHome))); // Set venv executable if available if FVenvPythonExe <> '' then PyConfig_SetString(Config, PPWcharT(PByte(@Config) + ConfigOffests[MinorVersion, TConfigFields.executable]), PWCharT(StringToWCharTString(FVenvPythonExe))); // Set program arguments (sys.argv) SetProgramArgs(Config); // PythonPath SetPythonPath(Config);

I don’t understand how and where initialization is performed, you should know this

Hello debugger brought me here: procedure TPythonEngine.AfterLoad; begin inherited; Initialize; end; If I comment out Initialize; then there is no crash

I can’t debug it because it’s crashing here please fix it somehow or try a virtual machine with windows 7 + delphi XE2

I was wondering if this error could be related to the fact that I am using Python under Windows 7 https://github.com/adang1345/PythonWin7 ? Although this version https://github.com/pyscripter/python4delphi/commit/46cd7a66dd9a249b3bc6bac9c2be611f2bde2069 python4delphi works successfully with this !

Yes, I was really mistaken and the error is not where I previously thought the error was with LoadDll: procedure CreatePyEngine; begin PythonEngine := TPythonEngine.Create(nil); PythonEngine.LoadDll; // CRASH HERE //TPythonThread.Py_Begin_Allow_Threads; end;

No, the problem is not in demo36, but in pythonengine.pas itself, I tried my ready-made projects, which worked and are now working with previous versions of python4delphi. The problem most likely manifested itself after editing: Paths := string(FPythonPath).Split([PathSep]); on Paths := SplitString(string(FPythonPath), PathSep); When previous versions did not work because of this syntax, I myself tried to fix it on SplitString and immediately such crashes appeared last work version from Delphi XE2 https://github.com/pyscripter/python4delphi/commit/46cd7a66dd9a249b3bc6bac9c2be611f2bde2069 UPD: Most likely I think that there is an incorrect conversion here: PyWideStringList_Append(PWSL, PWCharT(StringToWCharTString(Paths))); Please study this demo TStringDynArray is not friendly with such transformations, the output turns out to be crooked program Project2; {$APPTYPE CONSOLE} uses System.SysUtils, System.StrUtils, System.Types; var s:string; ar:TStringDynArray; begin s:='python4 delphi'; ar:=splitstring(s,' '); writeln(PansiChar(ar)); writeln(PChar(ar)); writeln(PansiChar(ansistring(ar))); readln; end.

1. https://github.com/pyscripter/python4delphi 2. Python version 3.12.1 3. Demo36

Hello, I am faced with a problem that I need to run one .py script in several instances, but I can’t achieve this through python4delphi ( I'm trying to do it like this, but since the script has been running for some time, the program just crashes: program ParallelPython; {$APPTYPE CONSOLE} {$R *.res} uses System.SysUtils, System.Diagnostics, System.Variants, System.SyncObjs, PythonEngine; var PythonEngine: TPythonEngine; type TPyThread = class(TPythonThread) protected procedure ExecuteWithPython; override; public constructor Create(createsuspended:boolean); end; procedure CreatePyEngine; begin PythonEngine := TPythonEngine.Create(nil); PythonEngine.LoadDll; TPythonThread.Py_Begin_Allow_Threads; end; procedure DestroyEngine; begin TPythonThread.Py_End_Allow_Threads; PythonEngine.Free; end; procedure TPyThread.ExecuteWithPython; const Arg = 'import sys'+#13#10+ 'def run_python_script(args_str):'+#13#10+ ' args_list = args_str.split()'+#13#10+ ' sys.argv = [sys.argv[0]] + args_list'; begin inherited; while true do begin PythonEngine.ExecString(Arg); PythonEngine.ExecFile(extractfilepath(paramstr(0))+'script.py'); end; end; constructor TPyThread.Create(createsuspended:boolean); begin inherited Create(CreateSuspended); ThreadExecMode := emNewInterpreterOwnGIL; FreeOnTerminate := True; end; var I: Integer; begin try CreatePyEngine; for I := 1 to 10 do TPyThread.Create(False); finally //DestroyEngine; end; ReadLn; end. And it probably also crashes because the script is trying to be executed in one interpreter (

If I run 10 instances of my own program, then all 10 instances work, and if I create and then destroy pythonengine, it will write an error that the python .dll is already in use ?

Hello, I ran into a problem that input/output does not work through the OnReceiveUniData/OnReceiveData event and others, but it works through TPythonGUIInputOutput.Output unit Unit2; interface uses Winapi.Windows, Winapi.Messages, System.SysUtils, System.Variants, System.Classes, Vcl.Graphics, Vcl.Controls, Vcl.Forms, Vcl.Dialogs, Vcl.StdCtrls, PythonEngine, PythonGUIInputOutput; type TForm2 = class(TForm) Button1: TButton; Button2: TButton; Memo1: TMemo; procedure Button1Click(Sender: TObject); procedure Button2Click(Sender: TObject); procedure OnReceiveUniData(Sender: TObject; var Data: string); procedure CreatePyEngine; private { Private declarations } public { Public declarations } end; var Form2: TForm2; PythonEngine: TPythonEngine; working: boolean = True; implementation {$R *.dfm} procedure TForm2.OnReceiveUniData(Sender: TObject; var Data: string); begin Memo1.Lines.Add(Data); //No Memo1.Lines.Add('2'); //No end; procedure TForm2.CreatePyEngine; var pyio:TPythonGUIInputOutput; begin pyio := TPythonGUIInputOutput.Create(nil); pyio.UnicodeIO := True; pyio.DelayWrites := True; pyio.OnReceiveUniData := OnReceiveUniData; //pyio.Output:=Memo1; //------- WORK PythonEngine := TPythonEngine.Create(nil); PythonEngine.IO := pyio; PythonEngine.RedirectIO := True; PythonEngine.LoadDll; TPythonThread.Py_Begin_Allow_Threads; end; procedure TForm2.Button1Click(Sender: TObject); const Script ='print("Hello")'; begin CreatePyEngine; // TThread.CreateAnonymousThread( procedure var Py: IPyEngineAndGIL; begin Py := SafePyEngine; while working do PythonEngine.ExecString(Script); end).Start; end; procedure TForm2.Button2Click(Sender: TObject); begin working:=False; end; end.

Big Thanks This Is Work: procedure TForm2.OnData(Sender: TObject; const Data: string); begin Memo1.Lines.Add(Data); end; procedure TForm2.CreatePyEngine; var pyio:TPythonInputOutput; begin pyio := TPythonInputOutput.Create(nil); pyio.UnicodeIO := True; pyio.DelayWrites := False; pyio.OnSendUniData := OnData; PythonEngine := TPythonEngine.Create(nil); PythonEngine.IO := pyio; PythonEngine.RedirectIO := True; PythonEngine.LoadDll; TPythonThread.Py_Begin_Allow_Threads; end;