Jump to content

pyscripter

Members
 View Profile  See their activity

  • Content Count

    1003

  • Joined

  • Last visited

  • Days Won

    65

Posts posted by pyscripter

  2. Posted · Edited by pyscripter

    There are many components/libraries available for running processes and capturing their output.  But, I got frustrated with their design and functionality, mostly for the following reasons:

    • Fixation with and premature conversion to strings. Processes produce and consume bytes.
    • Blocking reading of process output, resulting to inefficiencies (tight loops with Sleep, or separate threads for reading the output or providing input to the process)
    • Incomplete features and/or over-bloated

     

    So, I have made my own pascal-process single unit library.

     

    Main features:

    • Asynchronous reading of process output
    • Separate stdout and stderr reading which can optionally be merged
    • Ability to consume output as it is produced or else let it accumulate and read the final result
    • Ability to provide input to the running process before or while the process is running.
    • Ability to terminate the running process.
    • Synchronous and asynchronous execution of processes.
    • Interfaced-based facilitating memory management.
    • MIT licence

     

    Usage:

    You do not need to install the library. Just download or clone the repo and add the source subdirectory to the Library path.  Then add PascalProcess to your uses clause.

     

    If you just want to get the output of a process you can use the class functions of TPProcess.

      

      TPProcess = class(TInterfacedObject, IPProcess)
    class function Execute(const ACommandLine: string;
      const ACurrentDir: string = ''): TBytes; overload;
    class procedure Execute(const ACommandLine: string;
      const ACurrentDir: string; out Output, ErrOutput: TBytes) overload;
  end;

    This is an example: 

    var
  Output: TBytes;
begin
  Output := TPProcess.Execute('cmd /c echo Hi');
  Writeln(TEncoding.ANSI.GetString(Output));
end;

    For more demanding cases you can use the IPProcess interface.

     

    Example: 

    type
  TUtils = class
    class procedure OnRead(Sender: TObject; const Bytes: TBytes);
  end;

class procedure TUtils.OnRead(Sender: TObject; const Bytes: TBytes);
begin
  Writeln(TEncoding.ANSI.GetString(Bytes));
end;

procedure Test2;
// Processes ouput as it gets produced
// The main thread terminates the process
var
  Process: IPProcess;
begin
  Process := TPProcess.Create('cmd /c dir c:\ /s');
  Process.OnRead := TUtils.OnRead;
  WriteLn('Press Enter to start the process. Press Enter again to terminate');
  ReadLn;
  Process.Execute;
  ReadLn;
  Process.Terminate;
end;

    See here the definition of IPProcess.

     

    Limitations:

    Currently the library is Windows only.  The intention is to support other platforms (help wanted).

    • Like 5
    • Thanks 2

  3. Posted · Edited by pyscripter

    For now, there are very serious limitations you need to consider when using emNewInterpreterOwnGIL  to achieve true parallelism.  See for instance the limitations in What’s new in Python 3.14 — Python 3.14.0b3 documentation.   

     

    Even the innocent looking print command is not thread-safe.  And almost nothing can be share between the interpreters.  Also there are limitations about what you can import, and P4D modules are not yet considered safe for use with emNewInterpreterOwnGIL.  

     

    This is changing with the forthcoming python 3.14.  It will include a new module concurrent.interpreters that exposes the interpreters with their own GIL to pure python code.  There will also be an addition of ways to communicate between the interpreters using queues.  See PEP 734.   So for example you could store your output to such a queue and print it when everything is finished.  

     

    But all the above is cutting edge and unless you really need it you should avoid it,  If you decide to use it then make sure you fully understand the limitations and implications.

     

    I will try to make P4D modules compatible with emNewInterpreterOwnGIL and that would at least give you the option to say add output to a Delphi string list (or something similar) protected with a global lock on the Delphi side.

     

    Note that all the above are not related to the free-threading version of Python, which in itself is another story altogether.

     

     

    • Like 2

  5. Posted · Edited by pyscripter

    These two tests appear to be identical.

     

    Parsing them leads to a call 

     

    result := TJSONNumber.Create(str);

     

    where str = '0.1.2'

     

    Apparently this succeeds in Delphi 10.4 but correctly fails in later versions.  This is easily fixed by using TryStrToFloat before creating the TJSONNumber. 

    @dummzeuch I have committed a potential fix for the above10.4 failed invalid tests.  Could you please try again.

    • Like 1

  6. Posted · Edited by pyscripter

    17 minutes ago, dummzeuch said:

    That's TStringList.Contains which doesn't find. And that apparently was introduced in Delphi 12.

    I have now replaced that call to Contains.  Tested with Delphi 11. 

    ✖  comment\after-literal-no-ws.toml: 'inf' is not a valid floating point value
✖  float\inf-and-nan.toml: 'nan' is not a valid floating point value
✖  spec-1.0.0\float-2.toml: 'inf' is not a valid floating point value

Completed: 205, Succeeded: 202, Failed: 3

Completed: 529, Succeeded: 529, Failed: 0
✓ All tests passed!

     

    I think I can live with that.  Apparently StrToFloat was extended to cope with inf, nan, and -inf in Delphi 12.  In Delphi 12 you can serialize such special floating point values.

     

    Could you please test again with Delphi 10.4  The parsing might work, but the serializer had many bugs in 10.4.

  7. Posted

    Backward compatibility is going to be an issue at least for the serializer part.  Right now I am trying to fix Delphi 11 compatibility.

    It should be possible to make the parsing staff made compatible with earlier versions of Delphi, as long as TJSONObject exists.

     

     

  11. Posted

    3 minutes ago, dummzeuch said:

    Is that the expected result?

    Here: 

    Completed: 205, Succeeded: 205, Failed: 0
✓ All tests passed!


Completed: 529, Succeeded: 529, Failed: 0
✓ All tests passed!

     

    I can guess the issue is with the TFormatSettings in the conversion to float.   I wlll fix it.

  14. Posted · Edited by pyscripter

    TOML is a "config file format for humans", that has gained a lot of traction in the python and rust communities among others.   It is basically the INI file format on steroids.   It compares quite well to alternative formats such as JSON, YAML and XML.   Compared to JSON is way more readable and compact.

     

    Since I could not find any Delphi library for processing TOML, I have created my own: toml-delphi.  

     

    Features:

    • TOML v1.0.0 compliant.
    • Passes all 734 (valid/invalid) official validation tests.
    • Fast. Single stream tokenizer and lexer that doesn't use regex.
    • Converts TOML documents to Delphi's RTL's TJSONObject, thus allowing for easy traversal, manipulation and query of the generated documents.
    • Includes TTOMLWriter for converting TJSONObjects back to TOML.
    • Provides for easy (de)serialization of Delphi objects and records from/to TOML.

     

    This is the interface of the main unit: 

      TJSONObjectHelper = class helper for TJSONObject
    function ToTOML(MultilineStrings: Boolean = False; Indent: Integer = 4): string;
    procedure StreamTOML(Stream: TStream; MultilineStrings: Boolean = False; Indent: Integer = 4);
    procedure SaveTOMLtoFile(const FileName: string; MultilineStrings: Boolean = False; Indent: Integer = 4);
    class function FromTOML(const Contents: string): TJSONObject; overload;
    class function FromTOML(Contents: TBytes): TJSONObject; overload;
    class function FromTOML(Stream: TStream): TJSONObject; overload;
    class function FromTOMLFile(const FileName: string): TJSONObject;
  end;

  ETOMLSerializer = class(Exception);

  TTOMLSerializer = class
    class function Serialize<T>(const AValue: T): string; overload;
    class function Deserialize<T>(const ATOML: string): T; overload;
  end;

    Example usage:

     

    You can convert TOML source to TJSONObject using one of the FromTOML functions. For example to parse a TOML file you use: 

    var JsonObject := TJSONObject.FromTOMLFile(FileName);

//or for parsing a TOML string:

var JsonObject := TJSONObject.FromTOML(TOMLstring);

     

    To convert a TJSONObject to TOML you use one of the methods ToTOML, StreamTOML or SaveTOMLToFile. For example: 

    TOMLString := JsonObject.ToTOML;

// or

JsonObject.SaveTOMLToFile(FileName);

     

    Example serialization: 

    type
  TTestRec = record
    IntValue: Integer;
    FloatValue: double;
    StringValue: string;
    DateValue: TDateTime;
    ArrayValue: TArray<string>;
 end;

 procedure TestSerializer;
 var
   Rec: TTestRec;
   TOMLString: string;
 begin
   Rec.IntValue := 123;
   Rec.FloatValue := 3.14;
   Rec.StringValue := 'abc';
   Rec.DateValue := Now;
   Rec.ArrayValue := ['A', 'B', 'C'];

   Writeln('Serialized record:');
   WriteLn('==================');
   TOMLString := TTOMLSerializer.Serialize(Rec);
   Writeln(TOMLString);
   Writeln('Record deserialized and serialized again:');
   Writeln('=========================================');
   Rec := TTOMLSerializer.Deserialize<TTestRec>(TOMLString);
   TOMLString := TTOMLSerializer.Serialize(Rec);
   Writeln(TOMLString);
 end;

    Output: 

    Serialized record:
==================
IntValue = 123
FloatValue = 3.14
StringValue = "abc"
DateValue = "2025-06-18T05:37:02.110+03:00"
ArrayValue = [
    "A",
    "B",
    "C"
]

Record deserialized and serialized again:
=========================================
IntValue = 123
FloatValue = 3.14
StringValue = "abc"
DateValue = "2025-06-18T05:37:02.110+03:00"
ArrayValue = [
    "A",
    "B",
    "C"
]

     

    I hope you find it useful.

     

    • Like 9
    • Thanks 6

  15. Posted · Edited by pyscripter

    You can do something similar in python.   You can use sys.exit(n) which raises the SystemExit exception.   n can be a number or a string or anything else.    But if you call sys.exit(n)  with different integer values then you can do the following: 

    try
  PythonEngine.ExecString(...);
except
   on E: EPySystemExit do
     begin
        case IntToStr(e.EValue):
           1:
           2: 
         end;
     end;
end;

     

  16. Posted

    PythonEngine1.ExecStrings(lines)

     

    will raise in exception (a subclass of EPyException) if an error occurs.   If you redirecting the python output using for instance TPythonGUIInputOutput then error information is printed and you will be able to see where the error occurred including a traceback.   You can also use the Traceback object to extract information about the error.  If you want to handle the Exception you can use

      

    try
  PythonEngine1.ExecStrings(lines);
except
  on E: EPyException
    // Do whatever you want. You can use the PythonEngine1.Traceback to get information about what went wrong
end

  17. Posted · Edited by pyscripter

    I guess ExecStrings raises an exception so, the event is not signalled.

     

    You can modify ExecuteWithPython to always signal:

     

    procedure TPyThread.ExecuteWithPython;
    begin

      try
        GetPythonEngine.ExecString(Script);

      finally
        Event.Signal;

       end;
    end;

     

    Don't you get the error printed when running your script?  You can also use the debugger to see if/what the exception is.

     

    Note that not all modules are compatible with  PyInterpreterConfig_OWN_GIL.  I suspect PIL isn't, Read the documentation about the limitations.

     

    Does you program work with emNewState?  Note that emNewInterpreter does not offer any performance advantages compared to emNewState.

     

    Given that Demo 36 works, start from that and then gradually move towards what you want, until you find what fails.  The first thing that I would test is the imports.   Add: 

    import sys
print(sys.path)

    to make sure that you are using the correct version of python with your desired imports installed.

     

    Then add your imports to see whether they work.

  19. Posted · Edited by pyscripter

    1 hour ago, david berneda said:

    Oops, you mean plugin units, or demos?

    In Installing R and Python - TeeBI Documentation you say that

    Quote

    The unit containing Python4Delphi classes is BI.Plugins.Python.Engine.pas

    I cannot find that unit.

     

    Also you say that:

    Quote

     

    TeeBI includes several units that implement "algorithms" (statistical and machine-learning) using the R Language and Python Scikit-Learn runtimes.

    These units are located in the Sources\Algorithms folder.

     

    That folder does not exist.

  22. Posted · Edited by pyscripter

    I would like to share the following in case you encounter the same issue.

     

    Class and Record Helpers (Delphi) - RAD Studio states that:

    Quote

    You can define and associate multiple helpers with a single type. However, only zero or one helper applies in any specific location in source code. The helper defined in the nearest scope will apply. Class or record helper scope is determined in the normal Delphi fashion (for example, right to left in the unit's uses clause).

     

    Actually, this is not entirely correct.  Consider the following:

     

    Unit HelperUnit1.pas: 

    TObjectHelper1 = class helper for TObject
  procedure Test;
end;

     

    Unit2 HelperUnit2.pas: 

    TObjectHelper2 = class helper for TObject
  procedure Test;
end;

    Unit SupportClasses,pas: 

    uses
   HelperUnit1;
type
  TMyClass: class
  end;  

    Unit MainUnit.pas 

    interface
implementation
uses
  SupportClasses, HelperUnit2;
begin
  var MyClass:= TMyClass.Create;
  MyClass.Test;
end;

    MyClass.Test will use the HelperUnit1.TObjectHelper1.Test implementation even if HelperUnit1 is not even in scope, let alone being "in nearest scope".

     

    So it appears that if a class helper is in scope where a class is defined, it is used unconditionally in all units of a project.  If not, then what it is stated in the documentation applies.

    • Like 2

  23. Posted · Edited by pyscripter

    See python4delphi/Demos/Demo36 at master · pyscripter/python4delphi for an example of using the buffer protocol to read/write numpy arrays in Delphi using the buffer protocol.   

     

    If you do not care about speed watch the video tutorials and the tutorial demos that show you how to create numpy arrays from delphi and pass them to python and back.  In a similar way you can work with dataframes directly.

  24. Posted · Edited by pyscripter

    29 minutes ago, aehimself said:

    Does TJSonSerializer work with TObjectList and TObjectDictionary already? As far as I remember this is why I started to (de)serialize objects manually back around Delphi 10.4 but would love to automate things finally 🙂

    It includes a number of converters to handle generic collections, but I think you have to add them manually:

     

    image.thumb.png.227259b88e9cefe181f6b54c5856f9b4.png

  25. Posted · Edited by pyscripter

    Since you asked for Serializer benchmakrs:

     

    paolo-rossi/delphi-neon: JSON Serialization library for Delphi includes a benchmark against the Rest.Json serializer and it beats it hand down:

     

    image.thumb.png.d8d3caa2c25afcda0c606f7c4951a2f5.png

     

    I have replaced the Rest.Json serializer with the System.Json.Serializers TJSONSerializer.  Here are the results:

     

    image.thumb.png.ab71e4cbcea5cb822d8bb9d3b33699cc.png

     

    So now TJsonSerializer beats Neon hands down.

     

    TJsonSerializer looks good but it has some rough edges.   To run the benchmarks I had to add a converter that handles Enumerated values as strings instead of the default integers: 

    type
  TEnumStringConverter = class(TJsonConverter)
  public
    function CanConvert(ATypeInf: PTypeInfo): Boolean; override;
    function ReadJson(const AReader: TJsonReader; ATypeInf: PTypeInfo; const AExistingValue: TValue;
      const ASerializer: TJsonSerializer): TValue; override;
    procedure WriteJson(const AWriter: TJsonWriter; const AValue: TValue;
      const ASerializer: TJsonSerializer); override;
  end;

{ TEnumStringConverter }

function TEnumStringConverter.CanConvert(ATypeInf: PTypeInfo): Boolean;
begin
  // This converter can handle any type that is an enumeration
  Result := (ATypeInf.Kind = TTypeKind.tkEnumeration) and
    (ATypeInf <> TypeInfo(Boolean));
end;

function TEnumStringConverter.ReadJson(const AReader: TJsonReader; ATypeInf:
    PTypeInfo; const AExistingValue: TValue; const ASerializer:
    TJsonSerializer): TValue;
var
  LIntValue: Integer;
begin
  LIntValue := System.TypInfo.GetEnumValue(ATypeInf, AReader.Value.AsString);

  if LIntValue = -1 then // GetEnumValue returns -1 if the name is not found
    raise EJsonSerializationException.CreateFmt('Invalid string value "%s" for enumeration "%s".', [AExistingValue.AsString, ATypeInf.Name]);

  // Create a TValue of the specific enum type using its ordinal value.
  Result := TValue.FromOrdinal(ATypeInf, LIntValue);
end;

procedure TEnumStringConverter.WriteJson(const AWriter: TJsonWriter; const
    AValue: TValue; const ASerializer: TJsonSerializer);
begin
    AWriter.WriteValue(System.TypInfo.GetEnumName(AValue.TypeInfo, AValue.AsOrdinal));
end;

    See also:  

     

    Bummer: System.Json.Converters already includes TJsonEnumNameConverter that does the job.

     

     

×