Best Practice Question: Bidirectional EXE-to-EXE communication

[Alexander Halser 26]

I've taken over a legacy project from a friend, that needs a structural redesign. The application basically does file analysis and modification. It is currently a GUI app with everything built into the main executable. I would like to split that and am thinking of multiple EXEs. One that resembles the GUI, a console application that executes one particular task. On top of that, it needs a Windows service to execute scheduled tasks automatically, if no user is logged in.

 

What is the best practice to accomplish this?

Bi-directional communication is a must and could be done with custom messages and console output.

Any other thoughts? Plugin system via DLL? Runtime packages?

 

My thoughts about the general design so far:

• EXE1 (the GUI app)
• EXE2 (the non-GUI Windows service)
• EXE3 (the console app that executes tasks, can be used stand-alone in console mode and must run in multiple instances)

 

EXE1 and EXE2 would both call the console EXE3. However, the tasks being executed can be quite long (from minutes to hours), so bi-directional communication is required. And while a task is being executed, it is not feasible to simply kill the thread in order to stop it. EXE3 must complete or roll back the detail, it's currently working on.

 

Furthermore, EXEs should run without installation (except for the Windows service, that is). If EXE3 was, for instance, a COM server, this wouldn't work, would it? We could of course build everything into every EXE and separate the functionality in the code design. But I don't like the idea in general and multi-threaded execution is easier to handle with separate processes - in particular if the GUI app and the Windows service run at the same time, but must not execute the same task twice.

[Ruslan 5]

For communication between apps/processes I used to use Redis Message Queue to send some signals between them or to send data. But I didn't use that way in Delphi yet (planning to).

I agree with splitting GUI from the working code, for that I've chose MVVM pattern (ChatGPT gave me some examples but Grijji has a nice framework).

[ertank 27]

What you are looking for is "Inter Process Communication" aka IPC. For more details you can look here

I used Cromis for named pipes. There is built in encryption support in that library, if you may need such functionality.

[David Heffernan 2345]

FWIW there is registration free COM. Not saying that's what you need, just that you don't need registration to use COM. 

[tinyBigGAMES 58]

CreateFileMapping, shared memory between processes.
Creating Named Shared Memory - Win32 apps | Microsoft Learn

 

[FPiette 383]

You need "Inter Process Communication" (IPC) between your different executable.

I personally use sockets to communicate between exe because the exact same socket and code can also be used when the exe are running on different networked computers! This means you can choose with no code change between local computing or distributed computing.

Using sockets, as any other IPC, you have to design messages that will be exchanged between your executable. You can use your own design or use XML or JSON or whatever you like most.

There are several "socket" libraries fro Delphi. I'm using "Internet Component Suite" (ICS) which also supports SSL/TLS if you are concerned with communication security.

 

[Der schöne Günther 316]

There's tons of different IPC mechanisms. What you need depends on your requirements.

 

If you already have the "logic" part in a separate console application, why don't use its stdin/stdout (which are already pipes)?

Edited March 7, 2023 by Der schöne Günther

[DelphiUdIT 176]

7 hours ago, Alexander Halser said:

Furthermore, EXEs should run without installation (except for the Windows service, that is). If EXE3 was, for instance, a COM server, this wouldn't work, would it? We could of course build everything into every EXE and separate the functionality in the code design. But I don't like the idea in general and multi-threaded execution is easier to handle with separate processes - in particular if the GUI app and the Windows service run at the same time, but must not execute the same task twice.

1) It depends on how the COM Server is built: from Windows 7 onwards any application that works as a service cannot have ANY REFERENCE to graphic objects of any type or kind nor access to the graphic surface (obviously not even as an indirect reference).
The only exception is to show a "message" window, even before logging in with a specific API.
A service cannot launch a "normal" COM, ie which refers to graphical parts, but it can dialogue with a COM object already "running". I dont' know if the last implementation something is changed (is a lot of years that i don't develop COM object)

As @david-heffernan mentioned, the COMs can be used even without registration using the sXs technology, basically a Windows Manifest describes it (but it can be used only and exclusively if the COM is built as "inprocess" and this is not compatible with a windows service). Now, the references I had for using this technology (which doesn't use .NET components) are no longer available.

2) Preventing multiple instances of the same application is simple to implement. And if you want to prevent multiple unwanted parallel "tasks" from being executed, for example The service might pause (almost) when it detects the GUI executable (after it has done all its operations). However, the interaction mechanism between a Windows service and an EXE application is relatively simple.

3) For multiprocess communication I always use TCP communication (Indy components like TIdTcpClient / Server) and sometimes (actually very often in industrial applications) I use UDP in BROADCAST.

[David Heffernan 2345]

3 hours ago, tinyBigGAMES said:

CreateFileMapping, shared memory between processes.
Creating Named Shared Memory - Win32 apps | Microsoft Learn

 

Ugh, this has to be the worst possible solution

[Angus Robertson 574]

I used shared memory 20 years ago, but there is no queuing as such. 

 

So my later applications use sockets, specifically the ICS IP Log Streaming Component that handles all the connection and retries if lost (when the service stops) with minimal application code and events, with my own simple ASCII protocol on top for different messages.  Being TCP, queuing is automatic, buffer sizes permitting.  ICS has a sample that illustrates this.

 

Angus

 

[David Champion 48]

I have used the RemObjects  Remoting SDK. This also has a degree of language and platform independence.

 

The Remoting SDK has bidirectional TCP communications when using the called 'Super TCP Channel'.

Remote procedure calls can be made asynchronously, includes client side proxy generation, security features and event notifications.

 

 

 

Edited March 7, 2023 by David Champion
edit: name SuperSocket --> Super TCP Channel

[Alexander Halser 26]

Thank you for all your answers! I have used memory mapped files in the past, which works just fine in many situations, so this is definitely among the options. I'm also going to explore pipes, which I am not very familiar. 

 

Quote

"logic" part in a separate console application, why don't use its stdin/stdout (which are already pipes)?

Interesting idea! Console output is of course the most simple solution, but I did not think of console input.

I like this idea because of its simplicity.

 

Because back in my mind (not yet on the map) is the idea to make the execution part platform-independent from the beginning. If that requires a completely separate GUI part for MacOS/Linux (no plans for iOS/Android here) or can be done in one FMX app, is another story.

Edited March 7, 2023 by Alexander Halser

[Der schöne Günther 316]

2 hours ago, Alexander Halser said:

Console output is of course the most simple solution, but I did not think of console input.

I like this idea because of its simplicity.

I recently did it to extend a Delphi application. The extension was written in C++ and is a regular console application. The Delphi "master" app launches it, sends it commands via stdin and gets result via stdout. It was super easy to test in isolation with a clients machinery because the console app can also be launched and used by a regular ... console window.

 

Would definitely do so again. Not sure if it's a feasible solution if you need to rapidly exchange gargantuan amounts of data. For that, memory mapped files or sockets are probably a better approach.

Edited March 7, 2023 by Der schöne Günther

[tinyBigGAMES 58]

5 hours ago, David Heffernan said:

Ugh, this has to be the worst possible solution

lol, I will assume that you never used CreateFileMapping? So, with just three calls with no other dependencies, you can setup IPC between EXEs. With just two calls you can allocate virtual memory into the address space of your EXE or those same calls with different parameters, you can map a large file into your EXE address space and access it as it was contiguous memory. The power of CreateFileMapping at your disposal. Why does it need to overlay complicated?

[David Heffernan 2345]

13 minutes ago, tinyBigGAMES said:

lol, I will assume that you never used CreateFileMapping? So, with just three calls with no other dependencies, you can setup IPC between EXEs. With just two calls you can allocate virtual memory into the address space of your EXE or those same calls with different parameters, you can map a large file into your EXE address space and access it as it was contiguous memory. The power of CreateFileMapping at your disposal. Why does it need to overlay complicated?

I've used memory mapping for sure. But it sucks for this application which wants messages. Important to use the right tool for each job.

[Roger Cigol 103]

I have been following this forum and am interested that no-one has mentioned the Delphi "App Tethering" components which I thought were specifically designed to help make this task straightforward. Am I incorrect or does AppTethering not work well? (I hope I am not going to distract the forum question from the main question by asking this here)

[programmerdelphi2k 237]

at end... AppTethering use TCP UDP protocol behind scenes

http://www.malcolmgroves.com/blog/?p=1854

 

Edit: the beggin: http://www.malcolmgroves.com/blog/?p=1842

Edited March 7, 2023 by programmerdelphi2k

[Roger Cigol 103]

7 minutes ago, programmerdelphi2k said:

at end... AppTethering use TCP UDP protocol behind scenes

http://www.malcolmgroves.com/blog/?p=1854

Edited 4 minutes ago by programmerdelphi2k

I realised that -  but that seems to me to be a good reason to use it. Or is the problem that it ONLY uses UDP so there is no guarantee that all data gets through all the time ?

[programmerdelphi2k 237]

AppTethering is understood just publish info, basically. Malcom know it better than me

Quote

The receiver opens a TCP connection back to the sender and requests the value, which then comes back via TCP.

This makes sense, given a Stream Resource could potentially be quite large. While UDP has the advantage for low latency, TCP is much better suited to large transfers.

Edited March 7, 2023 by programmerdelphi2k

[Der schöne Günther 316]

Is there a public documentation how one has to implement "AppTethering" to be compatible? Otherwise, it's another vendor-lockin, and you're stuck with Delphi/C++ Builder for implementing both sides.

[programmerdelphi2k 237]

maybe better place would be Embarcadero... Malcolm Grooves blog is good for basic understanding... (above)

https://docwiki.embarcadero.com/RADStudio/Alexandria/en/Using_App_Tethering

https://docwiki.embarcadero.com/RADStudio/Alexandria/en/Connecting_to_Remote_Applications_Using_App_Tethering

 

note: behind scenes = Indy

Edited March 7, 2023 by programmerdelphi2k

[DelphiUdIT 176]

As mentioned in a previous post, I use classic (raw) UDP communication because Thethering is not normally supported in the industrial environment.
To give an example of UDP performance and reliability (in this context, it may not be appropriate for other applications), I explain an application developed and in operation with 20 plants in operation for 5 years (some for 5 years, some for 1 year).

The software runs on a PC with 3 x 1Gbit network cards and a USB / GigaEthernet adapter.

There are 6 to 18 devices connected to the local network and they use only UDP communications.
The average traffic on the various PC network cards is 150 Mbit to 750 Mbit per card.

The PC software to trigger the devices (basically cameras) transmits between 500 thousand and one million UDP packets (for synchronization only) per day (10 / 12 hours of continuous work 7 days a week). If even one packet is lost the trigger chain stops, the plant stops, a log is updated and an email is sent to me to report this.
In all installations, in all these years it has never happened that a UDP packet has been lost.

Of course, all the hardware has been designed to ensure maximum reliability, starting from the power supply and network components such as switches.

Creating a reliable network using UDP is possible in certain scenarios.

Bye

[tinyBigGAMES 58]

Quick and dirty example using CreateFileMapping

 

 

IPCTest.zip

Edited March 8, 2023 by tinyBigGAMES

[David Heffernan 2345]

3 hours ago, tinyBigGAMES said:

Quick and dirty example using CreateFileMapping

 

IPCTest.zip

Polling on a timer. Nice. As I said. The worst possible solution to this problem. 

[Anders Melander 1783]

IMO the restrictions you've put on the implementation seem a bit arbitrary and are getting in the way of the solution.

I would definitely go with an out-of-process COM server; It's a piece of cake to implement, it's natively supported by Delphi and you'll get IPC and threading thrown in for free. You could even have the service host it - or have it as a separate component that is used simultaneously from three modules.

 

So why is it a problem that the exe needs to run once (to take care of the COM registration) before the client can access it? If you really can't register the COM server before the modules are run, then just have the modules do it themselves when they are run. No "installation" is necessary when you have complete control of all the components of the system.