pascal-process: A new library for running processes and redirecting their output.

[pyscripter 796]

There are many components/libraries available for running processes and capturing their output.  But, I got frustrated with their design and functionality, mostly for the following reasons:

• Fixation with and premature conversion to strings. Processes produce and consume bytes.
• Blocking reading of process output, resulting to inefficiencies (tight loops with Sleep, or separate threads for reading the output or providing input to the process)
• Incomplete features and/or over-bloated

 

So, I have made my own pascal-process single unit library.

 

Main features:

• Asynchronous reading of process output
• Separate stdout and stderr reading which can optionally be merged
• Ability to consume output as it is produced or else let it accumulate and read the final result
• Ability to provide input to the running process before or while the process is running.
• Ability to terminate the running process.
• Synchronous and asynchronous execution of processes.
• Interfaced-based facilitating memory management.
• MIT licence

 

Usage:

You do not need to install the library. Just download or clone the repo and add the source subdirectory to the Library path.  Then add PascalProcess to your uses clause.

 

If you just want to get the output of a process you can use the class functions of TPProcess.

 

  TPProcess = class(TInterfacedObject, IPProcess)
    class function Execute(const ACommandLine: string;
      const ACurrentDir: string = ''): TBytes; overload;
    class procedure Execute(const ACommandLine: string;
      const ACurrentDir: string; out Output, ErrOutput: TBytes) overload;
  end;

This is an example:

var
  Output: TBytes;
begin
  Output := TPProcess.Execute('cmd /c echo Hi');
  Writeln(TEncoding.ANSI.GetString(Output));
end;

For more demanding cases you can use the IPProcess interface.

 

Example:

type
  TUtils = class
    class procedure OnRead(Sender: TObject; const Bytes: TBytes);
  end;

class procedure TUtils.OnRead(Sender: TObject; const Bytes: TBytes);
begin
  Writeln(TEncoding.ANSI.GetString(Bytes));
end;

procedure Test2;
// Processes ouput as it gets produced
// The main thread terminates the process
var
  Process: IPProcess;
begin
  Process := TPProcess.Create('cmd /c dir c:\ /s');
  Process.OnRead := TUtils.OnRead;
  WriteLn('Press Enter to start the process. Press Enter again to terminate');
  ReadLn;
  Process.Execute;
  ReadLn;
  Process.Terminate;
end;

See here the definition of IPProcess.

 

Limitations:

Currently the library is Windows only.  The intention is to support other platforms (help wanted).

Edited June 28 by pyscripter

[Kas Ob. 151]

@pyscripter , nice !

 

Removing "([])" from https://github.com/pyscripter/pascal-process/blob/main/Source/PascalProcess.pas#L454-L455 will make it compile and run on XE8

[pyscripter 796]

1 hour ago, Kas Ob. said:

will make it compile and run on XE8

Done. Thanks.

Edited June 29 by pyscripter

[pyscripter 796]

pascal-process is now multi-platform!

So now there is a Delphi library to match fpc's TProcess.

 

Desktop POSIX platforms (LINUX and MACOS) are now supported in addition to Windows.   I have tested with LINUX.  Since, I do not have a MACOS system available,  I would appreciate some help in testing with MACOS.

Edited July 6 by pyscripter

[Dave Nottage 624]

1 hour ago, pyscripter said:

I would appreciate some help in testing with MACOS.

I tested it on macOS using this command:

"/Applications/PAServer-23.3.app/Contents/MacOS/iosinstall" -q -c -t 5 -W

Which ran as expected. This is the command PAServer issues when it wants to detect what devices are visible to the Mac.

 

One thing to note: in the docs on your repo, it suggests using this in the OnRead (of which the parameters in the example do not match the actual source):

TEncoding.ANSI.GetString(Bytes)

Which might work elsewhere, but on macOS this works:

TEncoding.Default.GetString(Bytes)

 

[pyscripter 796]

@Dave Nottage Thanks for testing. 

 

I have updated the Readme with the corrected parameter list for OnRead.

Regarding the conversion to string, the whole point of the library is that you get bytes.  It is up to the user to use the bytes whichever way is appropriate.  The console encoding may be not be Delphi's default encoding.   In Linux it is most likely UTF8.   On Windows one might use the GetConsoleOutputCP API to get the code page for the output.   However even that is not reliable, For instance, on Windows, the scp command expects input and produces output in UTF8 encoding irrespective of the console encoding. The use of ANSI encoding on the project page was just an example. 

 

I agree thought that for the example TEncoding.Default is a better choice.  So I have updated that as well.

 

Edited July 6 by pyscripter

[Kas Ob. 151]

@pyscripter Hi, 

 

Are you up for suggestions ? even if they are not so small touching the core approach with overlapped operation (on Windows) ? 

[pyscripter 796]

@Kas Ob.  This is an open-source project.  You are welcome to create PRs, open issues, make suggestions etc.

[Kas Ob. 151]

49 minutes ago, pyscripter said:

You are welcome to create PRs, open issues, make suggestions etc.

In case i don't remember my Github password and don't use it ?! 

 

Well, if you are OK with suggestion here then please take what you see fit, if not just ignore.

 

Thoughts on the implementation;

1) Overlapped is well, we all know, but they do exist to provide specific usage, they allow converting simple IO operation from right-now synchronous or asynchronous to something fire now and poll later, if you need to block and wait on one operation then you lost the need for using overlapped in the first place, as example you can issue Read over socket or file, either synchronous or asynchronous, but with overlapped you can issue 10 or 100 read file then poll the status later, or even block on 64 of them with one thread and get notified when such slow operation finished, also one great advantage is you can unify multiple and different operation with one thread, now to the point, if you are using overlapped IO operation then block with WaitForSingleObject or WaitForSingleObjectEx then you are missing the advantage of overlapped operation, you could use block read and it will behave the same, 

suggestion;

Don't use ReadCompletionRoutine, not here, it is overkill.

Issue ReadFile once not any more the whole unit could have one read inside a loop, read then use WaitForMultipleObjects or WaitForMultipleObjectsEx if you wish, because here we can monitor not just the read operation but an event to signal thread exit and the handle if the child process, all in one loop simple, short, efficient and straight to the point.

 

2) Terminating the child and its spawns, TerminateProcessTree will do most of the time unless one of the child had tweaked its security and privilege's, now it will stay there, there is better solution than terminating the child tree or sending signals, it is https://learn.microsoft.com/en-us/windows/win32/procthread/job-objects

 

Using Job objects is way easy than it look, and because i have many versions of using Pipes named or unnamed, i picked one and DM you, will not paste it here, it is vetted and used but not in your thread and your code, it is for you look and cherry pick what you like form it.

 

 

ps;

I use this command to measure time for console command 

PipeIPC.SendData('cmd /v:on /c "set start=!time! && dir j:\android /s && set end=!time! && echo Start time: !start! && echo End time: !end!"'#13#10);

the unit i sent is fully bidirectional, and this command will show the time that a command like "dir j:\android /s" takes, one problem here is for this to work we need to spawn another console with /v:on as i couldn't make "setlocal enabledelayedexpansion" work, anyway , that command will spawn another console and execute the dir command, and what is interesting is that the time reported is different from the real console ran from Windows Explorer, your TProcess and my unit execute that command 

From standalone console the result of that dir on my fastest drive j: 

and using my unit which was almost the same as yours, (your last changes refuse to compile as AttachConsole is not declared in older Delphinos)

 

And that is due the flush the console that executing the command had to flush, Delphi console application generally are slower due the excessive flush, but in this case and because the child is sending so many data, Delphi is flushing once per 4k, and that is really nice effect.

[Kas Ob. 151]

Well the forum is not allowing me to send more text in private, but here the one last important thing to use when "  FProcess.SyncEvents := True;"

 

Hope you are OK with suggestion of such style !

[pyscripter 796]

@Kas Ob. There are many ways to skin a cat.  The approach adopted in pascal-process uses asynchronous (overlapped) IO with  alertable wait.  So there are no tight loops with Sleep, or additional threads for reading.  I think it is as efficient as it can get.    

 

I compared your code to pascal-process by using: 'cmd /v:on /c "set start=!time! && dir C:\Windows /s && set end=!time! && echo Start time: !start! && echo End time: !end!"'#13#10

 

Your code:

 

Start time: 14:31:08.09
End time: 14:31:22.64

about 14 secs

 

pascal-process

 

Start time: 14:35:54.74
End time: 14:36:04.58

about 9 secs.

 

To make sure disk caching did not play a role, I run your code again, with similar results:

 

Start time: 15:01:58.56
End time: 15:02:14.17

About 15 secs.

 

Of course one should not read much in a single benchmark.    But, I do not see any reason your code is faster or otherwise better than the one used in pascal-process.

 

Regarding AttachConsole do you happen to know, in which Delphi version it was first declared?

 

Edited July 7 by pyscripter

[Kas Ob. 151]

32 minutes ago, pyscripter said:

But, I do not see any reason your code is faster or otherwise better than the one used in pascal-process.

I didn't say mine faster, all what i pointed to is centralizing the 3 important handles in one loop, one thread, short and direct code, and that is it, although it is very strange it is performing slower, as there is no locking at all.

 

Anyway, glad you had the look and thank you !

[pyscripter 796]

1 minute ago, Kas Ob. said:

centralizing the 3 important handles in one loop

Although, it does not look like it, the alertable wait does the same.

[Kas Ob. 151]

14 minutes ago, pyscripter said:

Although, it does not look like it, the alertable wait does the same.

I very well familiar with the alertable wait and APC, i simply have different point view of them, they are fast and nice but also dangerous namely where there is a chance for an exception to raise, 

 

See, Delphi RTL can raise exception in so many places starting form the Memory Manager to most functionality, one exception raise and things can go very wrong, and most dangerously if any parameter passed or utilized where stack allocated.

 

Here a very nice article about something very similar, and remember this happen with Unity and that code was shipped with millions if not billions within applications and games running by end users;

https://unity.com/blog/engine-platform/debugging-memory-debugging-memory-corruption-who-wrote-2-into-my-stack-who-the-hell

[Kas Ob. 151]

Here a suggestion and highly important;

Make sure all call backs from the system 100% safe against exception, you can't raise an exception or allow the RTL to raise one, make sure to encapsulate the code with try..except and no re-raise.

[Vincent Parrett 852]

This is interesting - I have my own process library that uses IOCP for reading stdout/stderr and I thought it was fast (it's a lot faster than my old library that uses threads)  - but in a quick test your library has pretty much identical performance. That makes me consider switching to overlapped io and avoiding the extra complexity with IOCP.

 

My approach is similar to yours, my library does not convert stdout to strings by default. It was inspired by this library, which has a nice abstraction over the IO side of things. 

 

https://github.com/Tyrrrz/CliWrap

 

Not sure if I will end up publishing my lib, I have become rather disallusioned with the whole open source side of things - but that's another story. 

[Kas Ob. 151]

@pyscripter i don't have your hardware or anything even close to it, so when you show a command like "dir C:\Windows /s" takes less than 20 seconds, well this amazing, 

 

 Can you confirm if both libraries on the same reading buffer length, your i think yours at 16kb by default, while mine is left fixed by a constant at 4kb, could that have such huge difference ? in theory it might as "dir C:\Windows /s" do takes minutes on my machine, meaning the output is huge and again the flush operations could causing this difference.

[pyscripter 796]

1 minute ago, Kas Ob. said:

 Can you confirm if both libraries on the same reading buffer length,

I set the pascal-process buffer to 4 *1024 for the comparison.

[Vincent Parrett 852]

@Kas Ob. not sure what hardware he is using but on my 5900X ssd system I am seeing 16-17s for both his library and mine. 

[Kas Ob. 151]

1 minute ago, Vincent Parrett said:

@Kas Ob. not sure what hardware he is using but on my 5900X ssd system I am seeing 16-17s for both his library and mine. 

I just sent you my unit.

[Vincent Parrett 852]

I'm also using a 4K buffer, but in my testing a larger buffer size makes no discernible difference to performance. 

[Kas Ob. 151]

Mine showed difference in timing with when used 16kb.

[Vincent Parrett 852]

2 minutes ago, Kas Ob. said:

I just sent you my unit.

Interesting, similar to my old library (except that I handle stderr) - I will have to study it further as I have been considering using job objects since seeing a talk on them at DelphiSummit (I have looked at them in the past but never implemented them). 

 

[Kas Ob. 151]

2 minutes ago, Vincent Parrett said:

Interesting, similar to my old library (except that I handle stderr) - I will have to study it further as I have been considering using job objects since seeing a talk on them at DelphiSummit (I have looked at them in the past but never implemented them). 

 

This version doesn't, i have others, this one i use to capture debug from different tools, parse the output then run command command, it is bidirectional, i use it for building applications instead of batch files, so it should be able to handle InnoSetup and WinLicense ..etc