What is the correct approach to "phone home"?

[Der schöne Günther 316]

Consider the following scenario:

1. A full-screen Win32 application (Delphi, mostly) which must run 24 / 7.
2. The computers are located inside a factory. They generally have internet access, but might have pesky company firewalls.
3. Since the computers need to run 24/7, they never get Windows updates. We still have to support the earliest versions of Windows 7, for a few years to come

 

Requirement:

• When the user actively requests it,
• gather some data, zip it, and "send it to our technical support department"
• This is what I call "phone home": I want the running application to send data (a single ZIP file) to our companies support department.
• There are a few thousand installations worldwide. They are not authenticated against anything, they're running local Windows accounts.

 

So far, this has been done by using Delphi's Indy library to send an email to an old Exchange server of ours, by SMTP. I had taken the email approach because it will end up directly in the inbox of our support department. The downside of this method is that

1. More and more companies are blocking outgoing SMTP traffic. They probably suspect spam. Some security solutions will even fake our servers reply and indicate success although the email never got sent.
2. Our company wants to shut down our old in-house email server

 

I am now looking for an alternative approach. Here is what I can think of:

 

(1) Set up an FTP server, upload files there

Advantages:

• The FTP server is under our control and will be running as long as we want to
• Our application can properly report upload progress

Disadvantages:

• No experience with how likely this will be blocked by company firewalls
• Our service department will still have to be notified that something arrived at all. Either by email, or other means.

 

(2) Post stuff to our SharePoint / Microsoft Teams

Advantages:

• Posting text messages from the "outside world" (meaning: no authentication) to a channel in the support team works like a charm

Disadvantages:

• When Microsoft breaks this API, it won't be working anymore
• I have found absolutely no way of uploading files to our SharePoint server anonymously. APIs like "Microsoft Graph" seem to be centered around authenticated ActiveDirectory/Azure users, and do not allow public access

 

(3) Setup an own web server and POST data to this

Advantages:

• It can probably be automated so that after successfully POSTing a file, an email or some other kind of notification gets sent to our department

Disadvantages:

• No developer in our company knows anything about how to properly do web stuff (PHP?).

 

(4) Setup a new, simple email server and continue as always

Advantages:

• Existing software will continue to work

Disadvantages:

• Lots of companies will block outgoing SMTP traffic and it just won't work

 

(5) Drop "phone home" entirely

Query the operator to download the ZIP file to a USB drive, go elsewhere, and send it to us

Advantages:

• None

Disadvantages:

• Too much hassle so many operators simply won't bother
• In order to access the USB port, operator will have to unlock and open an electrical cabinet which is, in some countries, limited to special personnel.

 

 

What are your thoughts? Did I miss an obvious option?

Thanks for taking the time to read.

[Vandrovnik 214]

I vote for 3) - I am using this approach too.

[sakura 45]

Number 3, and creating a simple HTTP web server is no magic work. You may do that with Apache/PHP, IIS/ASP.NET, or even Delphi and internet network protocols. For each way, there are simple solutions.

[Rollo62 536]

1 hour ago, Vandrovnik said:

I vote for 3) - I am using this approach too.

 

45 minutes ago, sakura said:

Number 3, and creating a simple HTTP web server is no magic work. You may do that with Apache/PHP, IIS/ASP.NET, or even Delphi and internet network protocols. For each way, there are simple solutions.

Me too

And even a simple, virtual website provider with PHP will do, at very affordable cost.

[Angus Robertson 574]

Agree HTTP POST is the safest upload method, but I'd make sure the client can have a proxy server specified for cases where default internet access is blocked.  And maybe offer your clients that proxy server as being easier to monitor than the factory floor systems.  

 

Angus

 

[Fr0sT.Brutal 900]

HTTP(S) POST is the best when firewalls/proxies come to the scene. FTP is nightmare for gateways. Making upload server for this primitive task is easy, I even believe only NGinx will be enough without any coding (not sure though).

[aehimself 396]

#3.

You might also want to look into WebDav. That way you simply can copy the file without any additional code (what you have to maintain) on the webserver.

[Guest]

Utilizing HTTP Post is the best out there, because:

1) HTTP is faster than FTP and SMTP, like a lot faster.

2) HTTP/HTTPS is the most firewall friendly, one outgoing port and you are good to go.

 

Here i want to recommend to use your built own server, there is many approaches to achieve that and here few points

1) Use your stand alone server to receive these POST requests, or/and make sure you are using TLS secure connection, and here you can stick to the HTTPS with POST or use you are free to internally use your own TCP protocol, if we can call uploading a file (sending data in one way) a protocol.

2) Use Indy or ICS, RTC or what you see fit, it really doesn't matter.

3) I am/was using RTC for many years now to upload many sorts of files, most frequent files were EurekaLog reports, have a look here at that demo for uploading file https://rtc.teppi.net/realthinclient-sdk-lesson-3-sending-small-files-from-a-folder/

4) RTC simplify multithreading beyond imagining, also it give you the ability to build any server as stand alone EXE, Windows Service or ISAPI for IIS.

5) Once you had your HTTP(s) server receiving the files then if needed go ahead and send them using SMTP, which i do for EL reports after parsing so i have nice emails with many details in the mail body and nice detailed subject, of course along the attached report.

6) In few cases with few of my clients, they wanted to received the reports on their server (and their emails) which wasn't IIS, so i wrote php script to forward the reports to their emails, the script is 30 lines including brackets lines!, google PHPMailer or any other mailing method for php along how to receive a file with php, it will make you smile.

7) HTTP (preferably HTTPS) is OK with CloudFlare like services, so no effect in switching to such extra layer of protection in the future.

 

A little off topic:

I use hMailServer for many years and it is just more than great, portable folder, fast and secure, can't be more breath of fresh air to use and configure, i witnessed a server with weak account (email) password been hacked and being used to send spam, hMailServer managed to send 48-60 million spam email per day while the back log was 220 million before i been called to stop it and figure what it was, someone created account for testing with one char password and forgot to delete the account ! 

 

[aehimself 396]

Off: leaving my like because of hMailServer 🙂

[Guest]

I made a mistake with the lesson/demo for RTC, the one i want to point is in the Examples folders with the library called ClientUpload and it is short and can't be wrong implemented

implementation

{$R *.dfm}

procedure TForm1.btnConnectClick(Sender: TObject);
  begin
  with RtcHttpClient1 do
    begin
    if not isConnected then
      begin
      ServerAddr:=eServerAddr.Text;
      ServerPort:=eServerPort.Text;
      Connect;
      end
    else
      Disconnect;
    end;
  end;

procedure TForm1.RtcHttpClient1Connect(Sender: TRtcConnection);
  begin
  btnConnect.Caption:='Disconnect';
  end;

procedure TForm1.RtcHttpClient1Disconnect(Sender: TRtcConnection);
  begin
  btnConnect.Caption:='Connect';
  btnPutFile.Caption:='Upload';
  end;

procedure TForm1.btnPutFileClick(Sender: TObject);
  begin
  btnPutFile.Caption:='Clicked ...';
  with RtcDataRequest1 do
    begin
    // File Name on Server (need to URL_encode all Query parameters)
    Request.Query['file'] := URL_Encode(Utf8Encode(eRequestFileName.Text));
    // Local File Name
    Request.Info.asText['file'] := eLocalFileName.Text;
    Post;
    end;
  end;

procedure TForm1.RtcDataRequest1BeginRequest(Sender: TRtcConnection);
  begin
  btnPutFile.Caption:='Sending ...';
  with TRtcDataClient(Sender) do
    begin
    Request.Method:='PUT';
    Request.FileName:='/UPLOAD';
    Request.Host:=ServerAddr;
    Request.ContentLength:=File_Size(Request.Info.asText['file']);
    WriteHeader;
    end;
  end;

procedure TForm1.btnOpenClick(Sender: TObject);
  begin
  if OpenDialog1.Execute then
    begin
    eLocalFileName.Text:=OpenDialog1.FileName;
    eRequestFileName.Text:=ExtractFileName(eLocalFileName.Text);
    end;
  end;

procedure TForm1.RtcDataRequest1DataOut(Sender: TRtcConnection);
  begin
  with Sender as TRtcDataClient do
    begin
    pInfo.Caption:='Sending: '+
       IntToStr(Request.ContentOut)+'/'+
       IntToStr(Request.ContentLength)+' ['+
       IntToStr(round(Request.ContentOut/Request.ContentLength*100))+'%]';
    end;
  end;

procedure TForm1.RtcDataRequest1DataSent(Sender: TRtcConnection);
  var
    bSize:int64;
  begin
  with TRtcDataClient(Sender) do
    begin
    if Request.ContentLength>Request.ContentOut then
      begin
      bSize:=Request.ContentLength-Request.ContentOut;
      if bSize>64000 then bSize:=64000;
      Write(Read_File(Request.Info.asText['file'], Request.ContentOut, bSize));
      end;
    end;
  end;

procedure TForm1.RtcDataRequest1DataReceived(Sender: TRtcConnection);
  begin
  with TRtcDataClient(Sender) do
    begin
    { We do not expect a long response,
      so we can wait for the response to be Done before reading it. }
    if Response.Done then
      begin
      { We can use "Read" here to get the complete content sent from the Server
        if we expect the Server to send a short content. If the Server is expected
        to send a long response, you should use Read before Response.Done and
        write the read content to a file as it arrives to avoid flooding your memory. }
      btnPutFile.Caption:='Done, Status = '+
          IntToStr(Response.StatusCode)+' '+Response.StatusText;
      end;
    end;
  end;

end.

 

[Der schöne Günther 316]

Thank you all very much for the replies so far 👍

 

I hope I can convince the management to host a native server here inside the company and make it accessible with a constantly open port. Just like the Exchange server they want to shut down.

 

With a server in house, we could have full control about what is going on and could basically run any application there. I didn't really think about that. I had my mind on that external server that hosts our web page. I think it can only do PHP.

[Guest]

Off: leaving my like because of RTC

[Guest]

11 minutes ago, Der schöne Günther said:

With a server in house, we could have full control about what is going on and could basically run any application there. I didn't really think about that. I had my mind on that external server that hosts our web page. I think it can only do PHP.

Many of my clients were incapable of hosting their inhouse servers, and many were afraid of dedicated hosting servers, but after trying the hosted dedicated ones they liked them very much, in case you you want amuse the idea then i would recommend an OVH server or its little sisters 

https://www.ovh.ie/

https://www.soyoustart.com/en/

https://www.kimsufi.com/en/

 

Anyone will do !, and trust me even the smallest one will manage all of your emails and uploading and downloading files, unless you need hardcore data traffic and process then stick to OVH monsters.