aehimself

Hello,

We just met a strange issue and was wondering if anyone can explain why it is happening. The below code throws a stack overflow, as the compiler doesn't make a difference between TDateTime and Integer and keeps calling the first method:

program Project1;

{$APPTYPE CONSOLE}

{$R *.res}

uses
  System.SysUtils,
  System.DateUtils;

function GetDay(dt: TDateTime): string; Overload;
begin
  Result := GetDay(DayOfTheWeek(dt));
end;

function GetDay(i: Integer): string; Overload;
const LDays: array[1..7] of string = ('H', 'K', 'S', 'C', 'P', 'S', 'V');
begin
  Result := LDays[I];
end;

begin
  WriteLn(GetDay(Today));
end.

It works perfectly if you turn it to a dummy class and publish these methods as class functions:

Type
  x = Class
    class function GetDay(dt: TDateTime): string; Overload;
    class function GetDay(i: Integer): string; Overload;
  End;

It also works if you push these two methods in a separate unit with proper Interface section:

unit Unit1;

interface

function GetDay(dt: TDateTime): string; overload;
function GetDay(i: Integer): string;  overload;

implementation

I guess it'll have something to do on how overloads are interpreted in the implementation area...? Delphi 10.4, 10.4.2 and 11 are producing the same symptom.

So I made a basic sketch of my updating mechanism, freely available for anyone to check: https://github.com/aehimself/AEFramework

The only things you'll need are:

- AE.Updater.Updater

- AE.Misc.FileUtils

- AE.Updater.UpdateFile

- AE.Application.Settings

- AE.Misc.ByteUtils

At the moment it is using the System.Zip2 unit but can be reverted easily to Delphi's built in one by changing it to System.Zip in AE.Updater.Updater. It was built and tested on Delphi 11, relies heavily on generics and other built-in components. File versioning is strictly Windows-only... for the time being I found no better way to determine file data than in AE.Misc.FileUtils. That could use a refactor, but as it works for the time being I didn't bother.

To test, have any type of web server ready and decide where you want to put your repository. Let's say our repository will be https://dev.lan/updates, locally available at D:\WWW_root\devlan\updates. I'll make the references accordingly.

- Create a TAEUpdateFile instance and add a test product:

updatefile := TAEUpdateFile.Create;
Try
 var fname = ParamStr(0);
 updatefile.Product[FileProduct(fname)].URL := 'myproduct';
 var pfile := updatefile.Product[FileProduct(fname)].ProjectFile[ExtractFileName(fname)];
 pfile.LocalFileName := fname;
 var ver = FileVersion(fname).VersionNumber;
 var fver = pfile.Version[ver];
 fver.ArchiveFileName := ChangeFileExt(ExtractFileName(fname), Format('_%s.zip', [FileVersionToString(ver)]));
 fver.Changelog := 'Improved some stuff' + sLineBreak +
   'Broke lots of things I don''t yet know about';
 fver.DeploymentDate := 1; // Use your favorite UNIX timestamping method, just don't leave it on 0. 0 means undeployed and will not be considered when checking for updates

 var ms := TMemoryStream.Create;
 Try
  updatefile.SaveToStream(ms);
  ms.SaveToFile('D:\WWW_root\devlan\updates\update.dat');
 Finally
  ms.Free;
 End;
Finally
 updatefile.Free;
End;

Deploying the actual update file is manual for the time being, just zip your .exe, rename it to "Project1_1.0.0.0.zip" (or whatever the original .EXE name and version number is) and copy it to D:\WWW_root\devlan\updates\myproduct. Basically right next to the update file there will be a bunch of folders (one for each product) and inside this folder there will be tons of .zip files, one for each version of each file. Later on this can be used to downgrade as long as the .zip is still available.

Updating is a lot easier:

Var
 upd: TAEUpdater;
 s, fname: String;
 ver: UInt64;
Begin
 upd := TAEUpdater.Create(nil);
 Try
  upd.UpdateFileURL := 'https://dev.lan/updates/updates.dat';
  upd.UpdateFileEtag := _etag; // string var on form to minimize web traffic
  upd.CheckForUpdates;
  _etag := upd.UpdateFileEtag;

  s := '';
  For fname In upd.UpdateableFiles Do
   Begin
    s := s + fname + sLineBreak;
    For ver In upd.UpdateableFileVersions[fname] Do
     s := s + FileVersionToString(ver) + sLineBreak + upd.FileVersionChangelog[fname, ver] + sLineBreak + sLineBreak;
    upd.Update(fname);
   End;

  If Not s.IsEmpty Then ShowMessage(s);
 Finally
  FreeAndNil(upd);
 End;

At the start of your application call TAEUpdater.Cleanup to remove the old version of files - if any.

Todo: Error checking and handling... empty product url will probably result in 404 (unless https://dev.lan/updates//file.zip is a valid URL - didn't check). Files in subfolders aren't yet supported, all will be placed right next to the executable. Files without version information are not yet supported. Hash checking to be implemented, messages to be added, plus a basic demo app to manipulate the update file... in the long run I might replace generics and allow a custom way to download the files so instead of TNetHTTPClient ICS or Indy can be used according to the users taste. Yeah, this is only a skeleton for the time being but it seems to work.

Any suggestion is greatly appreciated!

2 hours ago, Arnaud Bouchez said:

With a simple GET, and proper E-Tag caching, it would let the HTTP server return 304 on GET if not modified: just a single request, only returning the data when it changed.
All will stay at HTTP server level, so it would be simple and effective.

Wow, I didn't know about this!

Without E-Tag caching:

With E-Tag caching:

Code is as easy as follows if someone is wondering:

Var
 hr: IHTTPResponse;
 head: TNameValuePair;
 headers: TArray<TNameValuePair>;
begin
 SetLength(headers, 1);
 headers[0].Name := 'If-None-Match';
 headers[0].Value := ETAG;
 hr := NetHttpClient1.Get('https://localintra.net/test.zip', nil, headers);
 For head In hr.Headers Do
  Memo1.Lines.Add(head.Name + ' -> ' + head.Value);
 Memo1.Lines.Add('');
 Memo1.Lines.Add(hr.StatusCode.ToString + ' ' + hr.StatusText + ', ' + hr.ContentLength.ToString + ' bytes received');

As a quick and dirty optimization I can even add it to my current update mechanism...

Nice, thank you!

13 minutes ago, SwiftExpat said:

I use a delta file to give me that information, so it is always a 2 step approach.

This idea popped up in my mind too - first to query only the project, server would reply with only the latest version of each file. If a new file or version is detected, the client could make a second request, sending the current version of each file in the reply. This way the second answer would only contain the changelog since said version - thus further reducing data to transfer and process.

This could be one in one step too, actually, if the project query includes the known local file versions.

I don't know why I want to minimize the amount of queries though as during the actual update - according to my new plan - one request will be performed for each file as they will be stored separately.

7 minutes ago, SwiftExpat said:

Would you be able to post some sizes so people can comment with a little more perspective?

For the time being they are negligible. The update JSON which holds all version information is 60k, zipped to 20k; but it contains only 200-ish versions from two projects up until now. The archives containing the latest version of projects are ranging between 5 and 20MB, but these are only downloaded if a newer version was found in the JSON.

9 minutes ago, SwiftExpat said:

What is the concern here, number of requests per hour or amount of data transferred?

Data. Amount of requests can easily be controlled from the client (check for updates every application start / x days).

11 minutes ago, SwiftExpat said:

Do you have a bandwidth limit at the hosting provider?

I do, but it's so high it's not a concern now / for a couple of years for sure. The reason I want to do it like this is because it is the way how it should be done. If I'm doing everything from scratch I should think of the future too... have the least amount of data to be transferred / processed.

16 minutes ago, SwiftExpat said:

HTTPS for easy proxy support. Also if you use a HEAD request you can get the file info from the server with 0 bytes transferred, supports your minimize traffic requirement.

Proxy is a really good catch, I did not even think about it! Can you please point me to a direction where I can learn more about how HTTP HEAD might return file info? I only found last modified date and content size which could be used to see if the static JSON changed; it won't tell me though if the project has new files / versions which I am interested in. 

Hello,

I have a pretty basic update mechanism in my applications which gets the job done - but it has it's limitations. It would be nice to have "dev" and "stable" channels, messages to be shown to the users (e.g. if there's a yet unfixed bug with a workaround), going back to a previous version, delta updates only, etc.

So I started to rework the thing as it is but I am simply stuck on the design stage... I can not agree with myself on how it should be done properly. My base theory is one file in one update archive, the application determines what it needs to update and download only those archives to minimize network traffic.

- Put a static JSON with all version information, this gets refreshed each time a version is deployed? This is how the current system works: easy to implement but...

- Just put the ready archive with the new version in the folder and have a service explore the changes and rebuild the static JSON accordingly? This option sounds the best to avoid any lingering files / entries (e.g. archive is placed but changed were not placed in the DB or vice versa) but where the changelog is coming from in this case?

- Store everything in a database and use a PHP script to query and assemble the reply JSON? My PHP knowledge is really limited, so I'd prefer not to have this option. Although, a dynamic list is crucial to minimize traffic (why to download version information for different products, or changelogs for versions below the current one?)

So my question is... are there any readily available update platforms for Delphi (server and client side too) which I can simply implement in my applications and forget about this matter?

I'm also open to suggestions on how the thing should work, on all possible levels:

- Backend. Is a database really needed or overkill?

- Static vs dynamic update definitions. Dynamic is better from many perspectives but does it really worth the extra effort?

- Protocol. Should I really stick to HTTP, or is there a better / create my own?

- How the information is translated and sent, including how the application should know if a new file is added to the distribution?

Cheers!

In some occasions the inaccuracy of Delphi's TTimer cased serious headaches so I did some digging and found an old post on how to turn a thread into a much more accurate timer. While I analyzed the code to find out how it works I made some comments and also took the liberty to change it here and there... and wrapped the whole thing in a TComponent which easily can be installed in the IDE and dropped on almost anything.

The TThreadedTimer's OnTimer event runs in the VCL thread so any UI element can be manipulated from there. It is also a drop-in replacement for TTimer, meaning you change your DFM and PAS and it should work exactly the same way.

My version delays enabling the thread so it won't spin freely until it has an event handler, properly resets the timing sequence if the Timer is set to Disabled and then Enabled again and also - in theory - the OnTimer event will no longer fire during destruction. Being a Windows-only guy it relies on WinApi, but I guess it can be made cross-platform by using Delphi's own TEvent class... as it yields zero benefits I didn't care to look into it.

As the original idea wasn't mine the right thing to do is to release this version under the do-whatever-you-want license. Feel free to use, point out possible issues or modify it to fit your needs.

God bless open source 🙂

uThreadedTimer.7z

18 minutes ago, Tom F said:

You reported these on EMB's Jira?

No, as

1, TZipFile is clearly abandoned (ZIP64 was introduced over 20 years ago)

2, There is a workaround for most of these

3, I could not wait until Delphi 25 Llanfair­pwllgwyngyll­gogery­chwyrn­drobwll­llan­tysilio­gogo­goch when it will be officially fixed 🙂

Anyway. let's not hijack the topic.

1 hour ago, corneliusdavid said:

I have only used it briefly here and there. What are the limitations you've run into?

- Inability to remove a file from the archive

- Does not support ZIP64 and LZMA (which can be solved by Zip2)

- Inability to read data from "corrupted" archives, like 7-zip can

- If the ZIP file was created by C# (don't know which lib) TZipFile sometimes says that the archive is not corrupt, but there are no files in it. With 7-Zip I can see and extract the file without issues

Completely offtopic.

On 3/29/2022 at 1:01 PM, Angus Robertson said:

[...] and not migrating from something like VCLZip.

I'm using an upped version of Delphi's TZipFile and always had enough of it's limitations so I went on and searched for VCLZip. Never seen a project this old before 😮

On 3/24/2022 at 4:56 PM, Remy Lebeau said:

Offhand, I would guess that a nil pointer is being accessed.

This is really interesting, since it's in VCL260.bpl...? Whenever I made these mistakes the AV always pointed to the executable (or .dll), rarely (or never) to a Delphi internal.

In case it's true, how exactly this can happen?

I'm not saying it is the case at you but when my 24/7 application was closing unexpectedly it was always

1, Memory corruption

2, Improper thread object access / synchronization

3, In some cases memory / handle leak

I found that if the code gets to a really messed up state no exceptions are raised when something happens (no more AVs), the program simply halts. My guess is somewhere in the stack... but I'm far away from this proficiency to be able to tell.

I hope you have a test environment. Launch your application and set up performance monitors for memory and handle usages. I guess on a test machine only 1 serial port will be used, so give it 5x - 7x time, then extract your log and create a graph. If a value steadily increases - that CAN be your issue.

For memory corruption, doublecheck your Move-s. I even wrote an own move procedure which in debug builds raises exceptions if I'm overflowing an area (I'm mostly using TBytes - that's how I know the limits). PChar reservations can cause this too - always reserve the memory for (Length(String) + 1) * SizeOf(Char).

Thread things can be the hardest to find out, especially if they happen randomly. As performance is usually not crucial I tend to use a synchronized access to everything, that's why I have a generic TAtomicValue<T> class, which is handling the locking and unlocking for me. I'm using this instead of every simple value in the thread, plus inside every class which the thread allows external access to.

Detecting memory or handle leaks is the worst. Build a test app and run each and every method 1.000.000 times. Whenever said value is increasing, something will be wrong there (also take into account that it CAN be inside a component, not necessarily your code). Or, simply purchase DeLeaker (or something similar). When I joined my current company it became a must-have; after I realized the memory management of our codebase was as messed up as the first applications I wrote. Took me more than a year to get rid of them (p.s.: I'm still wondering how noone recognized this flaw before...)

"Happy" hunting. I wish you'll have your "ohgoddammit" moment soon.

Seems the editor adds an extra line break to the end of the translation 😞

Yes, exactly 🙂 Didn't see the button there. Thanks!

@Anders Melander Any chance for adding multi-line support in translations? I have a couple of button captions like

Do something
Shift + F9

If not needed, I wouldn't add the key combinations from code.

24 minutes ago, Remy Lebeau said:

You can't send window messages to a non-windowed class.  Well, unless you are calling TObject.Dispatch() directly, that is.  Are you?

Why are you not posting (not sending) the message to an actual window?

Well.. technically I am. I have one allocated with AllocateHwnd. I guess in this case all messages are sent there, even if it's filtered.

30 minutes ago, Remy Lebeau said:

144 is in the range of system-reserved message IDs, however there is no known system message with an ID of 144.  Message ID 2 is WM_DESTROY, though.

Yes, I managed to find 2, but there was nothing on 144 though which I could find.

Hello,

I have a TObject descendant which has a message handler defined like:

 TProgressContent = Class
 strict private
  Procedure ProcessWindowMessage(Var Msg: TMessage); Message UM_WORKERENDED;
 End;

This message gets sent when a worker thread finished so I extract some data and free it up. I started to get some strange results and some application freezes, and it quickly turned out that lots of other messages arrive in this handler. I have some 144's and 2's entering up here.

The solution is easy - I just made an if Msg.Message = UM_WORKERENDED inside the handler and now all the issues are solved but isn't explicitly defining the message should already filter the rest out?

Also, out of curiosity, what is a window message with the ID of 144? Couldn't find anything on Google so far.

Cheers!

On 2/13/2022 at 12:21 PM, PeterBelow said:

I had a similar problem with Delphi Alexandria, McAfee would detect a freshly build Win32 debug EXE as virus

Issue is not Delphi Alexandria but that piece of ga... wonderful McAffee.

I remember removing viruses by hand from multiple servers because the freshly updated McAfee failed to recognize a ~5 yo worm. We knew it worked because it always quarantined our patching tool.

At work ESET, at home M$ Defender (AntiMalware, Security... idk how it's called nowadays) works just fine. The only reason we excluded Delphi executables is to speed up the building process on dev PCs.

It seems I found and fixed the problem. For a quick recap: Using UTF16 character encoding I often experienced ORA-01406 fetched column value was truncated errors and when this went away by manually increasing the buffer size text data got malformed.

As it turns out these two were independent errors and for the first my workaround is actually the solution.

Text corruption was a bit trickier. There is a setting which controls how LOB data is cached and internally this chooses what kind of resultset is being created. The difference between the two of them was how the columns were defined: the one working properly checked if the codepage is set to Unicode and if yes it changed String types to UnicodeString.

Moving the same block to the other resutset fixed the corruption.

15 hours ago, Uwe Raabe said:

The OS thread scheduler will take care of running it on whatever core is available.

I might be wrong here now as I believe this was on D7, but I think I met the same issue as @DavidJr. once.

I lost my key to my own password database so I put together a simple brute-force program to try and open it with all possible combinations, using as many TThread descendants as physical CPU cores (which was 4 in my laptop back then). During the first test run it was immediately clear that something is off, as the total amount of CPU used was 25%; I couldn't max out the CPU.

I seem to remember having some tries with BeginThread but I ended up running separate, single-threaded processes to utilize the rig's full protentional I managed to recover my passwords on (we had an unused 32 core Dell server which I could "borrow" for this task).

P.s.: It seems yes, it was solved one day. Using the following test I can increase the total CPU usage count all the way up just by adding more instances to the list:

program Project1;

{$APPTYPE CONSOLE}

{$R *.res}

uses
  System.SysUtils, System.Classes, System.Generics.Collections;

Type
 TMyThread = Class(TThread)
 protected
  Procedure Execute; Override;
 End;

Var
 threads: TObjectList<TMyThread>;
 a: Integer;

Procedure TMyThread.Execute;
Var
 a, b: Integer;
Begin
 b := 13;
 For a := Integer.MinValue To Integer.MaxValue Do
  b := b XOr a;
 a := b;
End;

begin
 threads := TObjectList<TMyThread>.Create(True);
 Try
  threads.Add(TMyThread.Create(False));
  {...}

  For a := 0 To threads.Count - 1 Do
   threads[a].WaitFor;
 Finally
  FreeAndNil(threads);
 End;
end.

My laptop has only 1 CPU though, it's possible that the problem still exists, just on multiple CPUs?

@Attila Kovacs you are right. I extracted the data behind RawByteString, which was 195,129,108,108,97,110,100,195,179,32,108,97,107,99,195,173,109. This is indeed UTF8, and it just gave me an idea on where to look...

@Dany Marmur The setting I'm talking about controls whether downloaded LOB data is cached or not. Inside the component this causes a different type of resultset created. On 3 PCs this setting was off (default), on one I turned it on. This is why one machine acted differently, as it was using a different type of resultset (the one which is faulty).

1 hour ago, Fr0sT.Brutal said:

and the issue is...?

...that the journey didn't end just yet 🙂

What I found is that the two things I thought are connected aren't. The original "column value is truncated" Oracle issue is a simple case of not enough buffer allocated and my workaround (forcing to use a larger buffer) is the solution.

However, this has nothing to do with the corruption issue; which I just found out is NOT memory corruption but wrongly transformed character array:

Here, the method's return value is PAnsiChar, fRawTemp is a RawByteString:

RawByteString is correct, but in the moment it's being casted to a PAnsiChar it is immediately messed up:

This then is stored in the resultset, so the dataset and the components display this mess.

...so a little update, the issue is in the component 🙂

The behavior difference between the 4 PCs are caused by a setting in my application which is changing a property on the connection, which is causing the corruption. This setting by default is off and wasn't changed on 3 PCs.

Took me a while to figure this out, as I had to import half of my main program bit-by-bit to the simple test app for the issue to finally occur.

At least it also made it clear what I thought is one issue is actually two; and fortunately both have a workaround.

4 minutes ago, Der schöne Günther said:

How do you actually detect that something has now gone wrong? Some component (like a DBGrid) displays a TDataSet that contains gobbledegook? Or is there more to detect it?

Messed up text in components. This is why I explicitly mentioned that it's not a display issue, data in the dataset is garbage too. I guess this never worked on my side, for a long time I simply got a "Fetched value truncated" error from Oracle until I found out changing a property gets rid of that (see the prev, thread mentioned in the linked one). Data is still corrupted, though.

7 minutes ago, Der schöne Günther said:

How do you know it's memory corruption (like writing out of bounds), and not generally using a stale pointer? Have you tried compiling the application with a memory manager like full FastMM and enabled runtime checks?

I don't know, but based on the initial Oracle error I guess it is going to be a memory corruption. I already checked MadExcept's buffer under / overrun protection, did not fire off.

6 minutes ago, Attila Kovacs said:

Why is it relevant what's in the dataset after an error in the client lib?

After increasing the initial_buffer_size, Oracle doesn't throw an error anymore.

7 minutes ago, Attila Kovacs said:

Could you do some selects where the actual data is the half of the maximum data etc.. to see the threshold for the error?

Yes, I did that, but it's not consistent. I have a feeling that it is going to be a sweetspot, when a specific amount of character data was returned the error appeared. Simply removing 3 random text columns already returned all data, formatted correctly afaik.

9 minutes ago, Attila Kovacs said:

Btw. are the actual RDBMS's also the same?  (You and your collegue)

Failed to mention this, thanks! Yes, we are connecting to the exact same DB.

12 minutes ago, Attila Kovacs said:

One side, you were mentioning VARCHAR somewhere and the other side you are setting the clientcodepage to UTF16. I don't know oracle but this is interesting.

Maybe zeos calculates for varchar and server delivers in nvarchar?

This is something I can not tell. I don't have access to the DB create scripts, PL/SQL says all the textual fields are VARCHARs only.

Unfortunately my knowledge in private life of RDBMSs is very limited, I consider myself a user only, so I don't know what is (should be) the difference between normal varchar and nvarchar.