Update framework question

[aehimself 396]

Hello,

 

I have a pretty basic update mechanism in my applications which gets the job done - but it has it's limitations. It would be nice to have "dev" and "stable" channels, messages to be shown to the users (e.g. if there's a yet unfixed bug with a workaround), going back to a previous version, delta updates only, etc.

 

So I started to rework the thing as it is but I am simply stuck on the design stage... I can not agree with myself on how it should be done properly. My base theory is one file in one update archive, the application determines what it needs to update and download only those archives to minimize network traffic.

- Put a static JSON with all version information, this gets refreshed each time a version is deployed? This is how the current system works: easy to implement but...

- Just put the ready archive with the new version in the folder and have a service explore the changes and rebuild the static JSON accordingly? This option sounds the best to avoid any lingering files / entries (e.g. archive is placed but changed were not placed in the DB or vice versa) but where the changelog is coming from in this case?

- Store everything in a database and use a PHP script to query and assemble the reply JSON? My PHP knowledge is really limited, so I'd prefer not to have this option. Although, a dynamic list is crucial to minimize traffic (why to download version information for different products, or changelogs for versions below the current one?)

So my question is... are there any readily available update platforms for Delphi (server and client side too) which I can simply implement in my applications and forget about this matter?

I'm also open to suggestions on how the thing should work, on all possible levels:

- Backend. Is a database really needed or overkill?

- Static vs dynamic update definitions. Dynamic is better from many perspectives but does it really worth the extra effort?

- Protocol. Should I really stick to HTTP, or is there a better / create my own?

- How the information is translated and sent, including how the application should know if a new file is added to the distribution?

 

Cheers!

[SwiftExpat 65]

19 minutes ago, aehimself said:

minimize network traffic.

What is the concern here, number of requests per hour or amount of data transferred? Size might dictate solution, but basic server load balancing would be an option.  Do you have a bandwidth limit at the hosting provider?

 

23 minutes ago, aehimself said:

Protocol. Should I really stick to HTTP, or is there a better / create my own?

HTTPS for easy proxy support. Also if you use a HEAD request you can get the file info from the server with 0 bytes transferred, supports your minimize traffic requirement.

[SwiftExpat 65]

One other item you should analyze is what size JSON is actually transferred by the server.  Most web servers can zip that file before transfer, so size on disk does not necessarily equal size transferred.

Would you be able to post some sizes so people can comment with a little more perspective?

[aehimself 396]

9 minutes ago, SwiftExpat said:

What is the concern here, number of requests per hour or amount of data transferred?

Data. Amount of requests can easily be controlled from the client (check for updates every application start / x days).

 

11 minutes ago, SwiftExpat said:

Do you have a bandwidth limit at the hosting provider?

I do, but it's so high it's not a concern now / for a couple of years for sure. The reason I want to do it like this is because it is the way how it should be done. If I'm doing everything from scratch I should think of the future too... have the least amount of data to be transferred / processed.

 

16 minutes ago, SwiftExpat said:

HTTPS for easy proxy support. Also if you use a HEAD request you can get the file info from the server with 0 bytes transferred, supports your minimize traffic requirement.

Proxy is a really good catch, I did not even think about it! Can you please point me to a direction where I can learn more about how HTTP HEAD might return file info? I only found last modified date and content size which could be used to see if the static JSON changed; it won't tell me though if the project has new files / versions which I am interested in. 

[aehimself 396]

7 minutes ago, SwiftExpat said:

Would you be able to post some sizes so people can comment with a little more perspective?

For the time being they are negligible. The update JSON which holds all version information is 60k, zipped to 20k; but it contains only 200-ish versions from two projects up until now. The archives containing the latest version of projects are ranging between 5 and 20MB, but these are only downloaded if a newer version was found in the JSON.

Edited April 27, 2022 by aehimself

[SwiftExpat 65]

14 minutes ago, aehimself said:

it won't tell me though if the project has new files / versions which I am interested in. 

I use a delta file to give me that information, so it is always a 2 step approach. It helps drive some of my update logic, you would have to judge the savings for your case.

 

As a bench mark, because you are already doing most of what this would do for you,

Commerically a new product from TMS, I have not tried it, only read the docs:
https://www.tmssoftware.com/site/tmsfncapptools.asp

Edited April 27, 2022 by SwiftExpat
Benchmark comment

[aehimself 396]

13 minutes ago, SwiftExpat said:

I use a delta file to give me that information, so it is always a 2 step approach.

This idea popped up in my mind too - first to query only the project, server would reply with only the latest version of each file. If a new file or version is detected, the client could make a second request, sending the current version of each file in the reply. This way the second answer would only contain the changelog since said version - thus further reducing data to transfer and process.

This could be one in one step too, actually, if the project query includes the known local file versions.

 

I don't know why I want to minimize the amount of queries though as during the actual update - according to my new plan - one request will be performed for each file as they will be stored separately.

[Arnaud Bouchez 407]

I would stick with a static JSON resource, if it is 20KB of data once zipped.

 

Don't use HEAD for it.

With a simple GET, and proper E-Tag caching, it would let the HTTP server return 304 on GET if not modified: just a single request, only returning the data when it changed.
All will stay at HTTP server level, so it would be simple and effective.

Edited April 27, 2022 by Arnaud Bouchez
added link

[SwiftExpat 65]

5 minutes ago, aehimself said:

I don't know why I want to minimize the amount of queries though as during the actual update

The tradeoff is between retries for failed / interrupted transfers. Probably better to add a robust retry mechanism, failures will happen even with a good connection.

[aehimself 396]

2 hours ago, Arnaud Bouchez said:

With a simple GET, and proper E-Tag caching, it would let the HTTP server return 304 on GET if not modified: just a single request, only returning the data when it changed.
All will stay at HTTP server level, so it would be simple and effective.

Wow, I didn't know about this!

 

Without E-Tag caching:

 

With E-Tag caching:

 

Code is as easy as follows if someone is wondering:

Var
 hr: IHTTPResponse;
 head: TNameValuePair;
 headers: TArray<TNameValuePair>;
begin
 SetLength(headers, 1);
 headers[0].Name := 'If-None-Match';
 headers[0].Value := ETAG;
 hr := NetHttpClient1.Get('https://localintra.net/test.zip', nil, headers);
 For head In hr.Headers Do
  Memo1.Lines.Add(head.Name + ' -> ' + head.Value);
 Memo1.Lines.Add('');
 Memo1.Lines.Add(hr.StatusCode.ToString + ' ' + hr.StatusText + ', ' + hr.ContentLength.ToString + ' bytes received');

As a quick and dirty optimization I can even add it to my current update mechanism...

 

Nice, thank you!

[Fr0sT.Brutal 900]

I use changelog file as both versions catalog and info for user to read, it has simple format like

 

3.2.1

+ add stuff

! fix stuff

...

 

Update checker requests the file, parses it, checks if latest version differs from the current binary version and suggests to update displaying all the changes from the current version to the latest one. I also have beta versions:

 

b865

+ some untested changes

 

where 865 is build number that is compared with that in current binary.

I don't think I need something more advanced - the changelogs are so small I don't bother with caching and traffic.

 

Depending on the server settings, you could put the versions into their personal folders so that directory listing will give you the list of all newer versions and their paths; then you can loop through all newer versions getting their changelogs (if each version has its own one) or just retrieve one common file.

 

Proxy support is really something to be considered; however, you can rely on system-wide settings if you don't want to implement custom editors for these options.

Edited April 28, 2022 by Fr0sT.Brutal

[aehimself 396]

So I made a basic sketch of my updating mechanism, freely available for anyone to check: https://github.com/aehimself/AEFramework

 

The only things you'll need are:

- AE.Updater.Updater

- AE.Misc.FileUtils

- AE.Updater.UpdateFile

- AE.Application.Settings

- AE.Misc.ByteUtils

 

At the moment it is using the System.Zip2 unit but can be reverted easily to Delphi's built in one by changing it to System.Zip in AE.Updater.Updater. It was built and tested on Delphi 11, relies heavily on generics and other built-in components. File versioning is strictly Windows-only... for the time being I found no better way to determine file data than in AE.Misc.FileUtils. That could use a refactor, but as it works for the time being I didn't bother.

 

To test, have any type of web server ready and decide where you want to put your repository. Let's say our repository will be https://dev.lan/updates, locally available at D:\WWW_root\devlan\updates. I'll make the references accordingly.

- Create a TAEUpdateFile instance and add a test product:

updatefile := TAEUpdateFile.Create;
Try
 var fname = ParamStr(0);
 updatefile.Product[FileProduct(fname)].URL := 'myproduct';
 var pfile := updatefile.Product[FileProduct(fname)].ProjectFile[ExtractFileName(fname)];
 pfile.LocalFileName := fname;
 var ver = FileVersion(fname).VersionNumber;
 var fver = pfile.Version[ver];
 fver.ArchiveFileName := ChangeFileExt(ExtractFileName(fname), Format('_%s.zip', [FileVersionToString(ver)]));
 fver.Changelog := 'Improved some stuff' + sLineBreak +
   'Broke lots of things I don''t yet know about';
 fver.DeploymentDate := 1; // Use your favorite UNIX timestamping method, just don't leave it on 0. 0 means undeployed and will not be considered when checking for updates

 var ms := TMemoryStream.Create;
 Try
  updatefile.SaveToStream(ms);
  ms.SaveToFile('D:\WWW_root\devlan\updates\update.dat');
 Finally
  ms.Free;
 End;
Finally
 updatefile.Free;
End;

Deploying the actual update file is manual for the time being, just zip your .exe, rename it to "Project1_1.0.0.0.zip" (or whatever the original .EXE name and version number is) and copy it to D:\WWW_root\devlan\updates\myproduct. Basically right next to the update file there will be a bunch of folders (one for each product) and inside this folder there will be tons of .zip files, one for each version of each file. Later on this can be used to downgrade as long as the .zip is still available.

 

Updating is a lot easier:

Var
 upd: TAEUpdater;
 s, fname: String;
 ver: UInt64;
Begin
 upd := TAEUpdater.Create(nil);
 Try
  upd.UpdateFileURL := 'https://dev.lan/updates/updates.dat';
  upd.UpdateFileEtag := _etag; // string var on form to minimize web traffic
  upd.CheckForUpdates;
  _etag := upd.UpdateFileEtag;

  s := '';
  For fname In upd.UpdateableFiles Do
   Begin
    s := s + fname + sLineBreak;
    For ver In upd.UpdateableFileVersions[fname] Do
     s := s + FileVersionToString(ver) + sLineBreak + upd.FileVersionChangelog[fname, ver] + sLineBreak + sLineBreak;
    upd.Update(fname);
   End;

  If Not s.IsEmpty Then ShowMessage(s);
 Finally
  FreeAndNil(upd);
 End;

At the start of your application call TAEUpdater.Cleanup to remove the old version of files - if any.

 

Todo: Error checking and handling... empty product url will probably result in 404 (unless https://dev.lan/updates//file.zip is a valid URL - didn't check). Files in subfolders aren't yet supported, all will be placed right next to the executable. Files without version information are not yet supported. Hash checking to be implemented, messages to be added, plus a basic demo app to manipulate the update file... in the long run I might replace generics and allow a custom way to download the files so instead of TNetHTTPClient ICS or Indy can be used according to the users taste. Yeah, this is only a skeleton for the time being but it seems to work.

 

Any suggestion is greatly appreciated!

[SwiftExpat 65]

This is just a followup for reference to staying at the HTTP level for checking server file details.  It is related to this thread but applies more to large files or binaries.
I was exploring this and went a slightly different direction using another HTTP header, If-Modified-Since.

 

Working implementation can be found here, I am using in my Deputy expert update code.

https://github.com/SwiftExpat/Deputy/blob/main/Source/SE.UpdateManager.pas

 

Add your header, the supplied value had to be a valid date for my server was the only catch, so I defaulted it with a constant.

const
	dt_lastmod_default = 'Fri, 01 Apr 2010 23:15:56 GMT';
	hdr_ifmodmatch = 'If-Modified-Since';
...
  FHTTPRequest.CustomHeaders[hdr_ifmodmatch] := LastModified; // date must be a valid value;
  FHTTPRequest.Get(URL);
...

 

Handling the response is easy, just switch on the status code.

if AResponse.StatusCode = 200 then
  begin
    LastModified := AResponse.HeaderValue[hdr_lastmodified];
  end
else if AResponse.StatusCode = 304 then
  begin
    RefreshDts := now;
    LogMessage('File not modified Http result = ' + AResponse.StatusCode.ToString);
  end
else
    LogMessage('URL Cache Http result = ' + AResponse.StatusCode.ToString);