Jump to content

aehimself

Members
 View Profile  See their activity

  • Content Count

    1090

  • Joined

  • Last visited

  • Days Won

    23

Posts posted by aehimself

  1. Posted · Edited by aehimself

    I wanted to do this for a really long time so I started to extract the core improvements of Delphi's TDBGrid into a separate component. This includes:

    - New public BeginUpdate / EndUpdate methods, which can disconnect the dataset and keep the previous image on the component. This is useful if you are doing opens / posts / anything in a background thread but don't want to show emptiness until

    - Automatic and manual fitting of columns which considers the column and the content width but won't let a column be wider than half it's size

    - Vertical scrollbar now works properly, not only 3 positions and is not visible when not needed

    - Grid properly handles mouse wheel scrolling

    - Content is shown as the scrollbar is dragged (not only updating when the mouse is released)

    - If there is no connected dataset or it is not active, the two empty cells won't be shown

    - Every second row has a slightly different background, out-of-focus selection is now drawn with a separate shade of grey, so you can see that the grid is not in focus. This considers VCL styles.

    - If TitleClick or TitleHotTrack is enabled, the cursor changes to a hand instead of the pointers

     

    It might get more updates later on when I see what code can be generally used from my heavily customized one.

     

    Feel free to grab it / check how things were done. Tried to put comments everywhere.

    • Like 2

  4. Posted

    Something like this: 

    Var
  pbar: TProgressBar;
  lbl: TLabel;
  a: Integer;
Begin
  TProgressBarInStatusBar.CreateIn(StatusBar1, pbar, lbl);
  
  For a := 0 To 1000 Do
  Begin
    pbar.Position := a Div 10;
    lbl.Caption := 'Working ' + a.ToString + '...';
    Application.ProcessMessages; // Don't do this. It's just pseudocode.
    Sleep(500);
  End;
End;

     

  5. Posted

    I create my progress bars in the first panel of the status bar, with a label on it to show some meaningful information on the progress... like "30 of 999 items processed". Yes, it won't work if you resize the panel, needs some adjustments if you want to create it in the 3rd, but this is the code I use: 

    Unit uProgressBarInStatusBar;

Interface

Uses Vcl.ComCtrls, Vcl.StdCtrls;

Type
 TProgressBarInStatusBar = Class
 public
  Class Procedure CreateIn(Const inStatusBarPanel: TStatusPanel; Var outProgressBar: TProgressBar; Var outLabel: TLabel);
 End;

Implementation

Uses Vcl.Controls, System.Classes;

Class Procedure TProgressBarInStatusBar.CreateIn(Const inStatusBarPanel: TStatusPanel; Var outProgressBar: TProgressBar; Var outLabel: TLabel);
Var
 statusbar: TStatusBar;
Begin
 statusbar := inStatusBarPanel.Collection.Owner As TStatusBar;

 outProgressBar := TProgressBar.Create(statusbar);
 outProgressBar.Parent := statusbar;
 outProgressBar.Top := 2;
 outProgressBar.Left := 1;
 outProgressBar.Width := inStatusBarPanel.Width - 3;
 outProgressBar.Height := statusbar.ClientHeight - 3;

 outLabel := TLabel.Create(outProgressBar);
 outLabel.Parent := outProgressBar;
 outLabel.Align := alClient;
 outLabel.AutoSize := False;
 outLabel.Alignment := taCenter;
End;

End.

    The small sacrifice of having it in the first panel makes up to it with no custom drawings / hacks needed at all. And it looks good enough:

     

    image.png.a052b18560be4c685eedc70179a3f2ae.png

  9. Posted

    Hello,

     

    We have an application built on Delphi 10.4.1 / 10.4.2 which is communicating with a server using the Indy TidHttp component. It works perfectly, but there is one particular call when the result can arrive in 1-1,5 hours... and this is where things get strange. The request is sent with idHttp.Post (using a stream as an outgoing and an incoming data buffer) and if the reply arrives in 30 minutes, all is fine. Somewhere between 30 minutes and one hour, the underlying WinSock .select never returns. Data is sent out, received by the server, processed and the data is sent out - but never received by the client. Using WireShark it can be seen that the moment the server sends the reply, the client issues a TCP retransmission... maybe it thinks that the data was lost but was unable to use the channel while waiting for data? Then the reply arrives, channel gets free and it sends the retransmission but discards the data received? These are just guesses, I'm not very familiar with this low-level functionality of WinSock.

    Oh, one more thing... this issue is NOT present if the server and the client is on the same machine; connecting to localhost makes a difference.

     

    Stack trace where the application stops is as follows: 

    :772729dc ntdll.ZwWaitForSingleObject + 0xc
:74417555 ; C:\WINDOWS\SysWOW64\mswsock.dll
:751c5f1e WS2_32.select + 0xce
IdStackWindows.TIdSocketListWindows.FDSelect(???,???,nil,???)
IdStackWindows.TIdSocketListWindows.SelectRead(-2)
IdSocketHandle.TIdSocketHandle.Select(???)
IdSocketHandle.CheckIsReadable(???)
IdSocketHandle.TIdSocketHandle.Readable(-2)
IdIOHandlerStack.TIdIOHandlerStack.Readable(???)
IdIOHandler.TIdIOHandler.ReadFromSource(True,-2,False)
IdIOHandler.TIdIOHandler.ReadLn(#$A,-1,16384,TIdASCIIEncoding($1A138AD4) as IIdTextEncoding)
IdIOHandler.TIdIOHandler.ReadLn(nil)
IdHTTP.TIdCustomHTTP.InternalReadLn
IdHTTP.TIdCustomHTTP.DoRequest(???,'http://10.0.2.53:12345/BIN',$2E6D75A0,$2E6D73A0,(...))
IdHTTP.TIdCustomHTTP.Post('http://10.0.2.53:12345/BIN',$14E0F58,$2E6D73A0)

    Before you say anything, I know this is a bad design. We shouldn't wait on long lasting operations but to poll for it on HTTP. What I'd like to know is what happens and why it happens so I can get an insight if patching the mess worth it or just jump straight to refactoring.

     

    Cheers!
     

  10. Posted

    Hello,

     

    We just met a strange issue and was wondering if anyone can explain why it is happening. The below code throws a stack overflow, as the compiler doesn't make a difference between TDateTime and Integer and keeps calling the first method:

      

    program Project1;

{$APPTYPE CONSOLE}

{$R *.res}

uses
  System.SysUtils,
  System.DateUtils;

function GetDay(dt: TDateTime): string; Overload;
begin
  Result := GetDay(DayOfTheWeek(dt));
end;

function GetDay(i: Integer): string; Overload;
const LDays: array[1..7] of string = ('H', 'K', 'S', 'C', 'P', 'S', 'V');
begin
  Result := LDays[I];
end;

begin
  WriteLn(GetDay(Today));
end.

    It works perfectly if you turn it to a dummy class and publish these methods as class functions: 

    Type
  x = Class
    class function GetDay(dt: TDateTime): string; Overload;
    class function GetDay(i: Integer): string; Overload;
  End;

    It also works if you push these two methods in a separate unit with proper Interface section: 

    unit Unit1;

interface

function GetDay(dt: TDateTime): string; overload;
function GetDay(i: Integer): string;  overload;

implementation

    I guess it'll have something to do on how overloads are interpreted in the implementation area...? Delphi 10.4, 10.4.2 and 11 are producing the same symptom.

  11. Posted

    So I made a basic sketch of my updating mechanism, freely available for anyone to check: https://github.com/aehimself/AEFramework

     

    The only things you'll need are:

    - AE.Updater.Updater

    - AE.Misc.FileUtils

    - AE.Updater.UpdateFile

    - AE.Application.Settings

    - AE.Misc.ByteUtils

     

    At the moment it is using the System.Zip2 unit but can be reverted easily to Delphi's built in one by changing it to System.Zip in AE.Updater.Updater. It was built and tested on Delphi 11, relies heavily on generics and other built-in components. File versioning is strictly Windows-only... for the time being I found no better way to determine file data than in AE.Misc.FileUtils. That could use a refactor, but as it works for the time being I didn't bother.

     

    To test, have any type of web server ready and decide where you want to put your repository. Let's say our repository will be https://dev.lan/updates, locally available at D:\WWW_root\devlan\updates. I'll make the references accordingly.

    - Create a TAEUpdateFile instance and add a test product: 

    updatefile := TAEUpdateFile.Create;
Try
 var fname = ParamStr(0);
 updatefile.Product[FileProduct(fname)].URL := 'myproduct';
 var pfile := updatefile.Product[FileProduct(fname)].ProjectFile[ExtractFileName(fname)];
 pfile.LocalFileName := fname;
 var ver = FileVersion(fname).VersionNumber;
 var fver = pfile.Version[ver];
 fver.ArchiveFileName := ChangeFileExt(ExtractFileName(fname), Format('_%s.zip', [FileVersionToString(ver)]));
 fver.Changelog := 'Improved some stuff' + sLineBreak +
   'Broke lots of things I don''t yet know about';
 fver.DeploymentDate := 1; // Use your favorite UNIX timestamping method, just don't leave it on 0. 0 means undeployed and will not be considered when checking for updates

 var ms := TMemoryStream.Create;
 Try
  updatefile.SaveToStream(ms);
  ms.SaveToFile('D:\WWW_root\devlan\updates\update.dat');
 Finally
  ms.Free;
 End;
Finally
 updatefile.Free;
End;

    Deploying the actual update file is manual for the time being, just zip your .exe, rename it to "Project1_1.0.0.0.zip" (or whatever the original .EXE name and version number is) and copy it to D:\WWW_root\devlan\updates\myproduct. Basically right next to the update file there will be a bunch of folders (one for each product) and inside this folder there will be tons of .zip files, one for each version of each file. Later on this can be used to downgrade as long as the .zip is still available.

     

    Updating is a lot easier: 

    Var
 upd: TAEUpdater;
 s, fname: String;
 ver: UInt64;
Begin
 upd := TAEUpdater.Create(nil);
 Try
  upd.UpdateFileURL := 'https://dev.lan/updates/updates.dat';
  upd.UpdateFileEtag := _etag; // string var on form to minimize web traffic
  upd.CheckForUpdates;
  _etag := upd.UpdateFileEtag;

  s := '';
  For fname In upd.UpdateableFiles Do
   Begin
    s := s + fname + sLineBreak;
    For ver In upd.UpdateableFileVersions[fname] Do
     s := s + FileVersionToString(ver) + sLineBreak + upd.FileVersionChangelog[fname, ver] + sLineBreak + sLineBreak;
    upd.Update(fname);
   End;

  If Not s.IsEmpty Then ShowMessage(s);
 Finally
  FreeAndNil(upd);
 End;

    At the start of your application call TAEUpdater.Cleanup to remove the old version of files - if any.

     

    Todo: Error checking and handling... empty product url will probably result in 404 (unless https://dev.lan/updates//file.zip is a valid URL - didn't check). Files in subfolders aren't yet supported, all will be placed right next to the executable. Files without version information are not yet supported. Hash checking to be implemented, messages to be added, plus a basic demo app to manipulate the update file... in the long run I might replace generics and allow a custom way to download the files so instead of TNetHTTPClient ICS or Indy can be used according to the users taste. Yeah, this is only a skeleton for the time being but it seems to work.

     

    Any suggestion is greatly appreciated!

  12. Posted

    2 hours ago, Arnaud Bouchez said:

    With a simple GET, and proper E-Tag caching, it would let the HTTP server return 304 on GET if not modified: just a single request, only returning the data when it changed.
    All will stay at HTTP server level, so it would be simple and effective.

    Wow, I didn't know about this!

     

    Without E-Tag caching:

    image.png.874b2339e849ec02f6ded629dd44c75a.png

     

    With E-Tag caching:

    image.png.17863dcae688494a67ac0b406bf422d9.png

     

    Code is as easy as follows if someone is wondering: 

    Var
 hr: IHTTPResponse;
 head: TNameValuePair;
 headers: TArray<TNameValuePair>;
begin
 SetLength(headers, 1);
 headers[0].Name := 'If-None-Match';
 headers[0].Value := ETAG;
 hr := NetHttpClient1.Get('https://localintra.net/test.zip', nil, headers);
 For head In hr.Headers Do
  Memo1.Lines.Add(head.Name + ' -> ' + head.Value);
 Memo1.Lines.Add('');
 Memo1.Lines.Add(hr.StatusCode.ToString + ' ' + hr.StatusText + ', ' + hr.ContentLength.ToString + ' bytes received');

    As a quick and dirty optimization I can even add it to my current update mechanism...

     

    Nice, thank you!

  13. Posted

    13 minutes ago, SwiftExpat said:

    I use a delta file to give me that information, so it is always a 2 step approach.

    This idea popped up in my mind too - first to query only the project, server would reply with only the latest version of each file. If a new file or version is detected, the client could make a second request, sending the current version of each file in the reply. This way the second answer would only contain the changelog since said version - thus further reducing data to transfer and process.

    This could be one in one step too, actually, if the project query includes the known local file versions.

     

    I don't know why I want to minimize the amount of queries though as during the actual update - according to my new plan - one request will be performed for each file as they will be stored separately.

  14. Posted · Edited by aehimself

    7 minutes ago, SwiftExpat said:

    Would you be able to post some sizes so people can comment with a little more perspective?

    For the time being they are negligible. The update JSON which holds all version information is 60k, zipped to 20k; but it contains only 200-ish versions from two projects up until now. The archives containing the latest version of projects are ranging between 5 and 20MB, but these are only downloaded if a newer version was found in the JSON.

  15. Posted

    9 minutes ago, SwiftExpat said:

    What is the concern here, number of requests per hour or amount of data transferred?

    Data. Amount of requests can easily be controlled from the client (check for updates every application start / x days).

     

    11 minutes ago, SwiftExpat said:

    Do you have a bandwidth limit at the hosting provider?

    I do, but it's so high it's not a concern now / for a couple of years for sure. The reason I want to do it like this is because it is the way how it should be done. If I'm doing everything from scratch I should think of the future too... have the least amount of data to be transferred / processed.

     

    16 minutes ago, SwiftExpat said:

    HTTPS for easy proxy support. Also if you use a HEAD request you can get the file info from the server with 0 bytes transferred, supports your minimize traffic requirement.

    Proxy is a really good catch, I did not even think about it! Can you please point me to a direction where I can learn more about how HTTP HEAD might return file info? I only found last modified date and content size which could be used to see if the static JSON changed; it won't tell me though if the project has new files / versions which I am interested in. 

  16. Posted

    Hello,

     

    I have a pretty basic update mechanism in my applications which gets the job done - but it has it's limitations. It would be nice to have "dev" and "stable" channels, messages to be shown to the users (e.g. if there's a yet unfixed bug with a workaround), going back to a previous version, delta updates only, etc.

     

    So I started to rework the thing as it is but I am simply stuck on the design stage... I can not agree with myself on how it should be done properly. My base theory is one file in one update archive, the application determines what it needs to update and download only those archives to minimize network traffic.

    - Put a static JSON with all version information, this gets refreshed each time a version is deployed? This is how the current system works: easy to implement but...

    - Just put the ready archive with the new version in the folder and have a service explore the changes and rebuild the static JSON accordingly? This option sounds the best to avoid any lingering files / entries (e.g. archive is placed but changed were not placed in the DB or vice versa) but where the changelog is coming from in this case?

    - Store everything in a database and use a PHP script to query and assemble the reply JSON? My PHP knowledge is really limited, so I'd prefer not to have this option. Although, a dynamic list is crucial to minimize traffic (why to download version information for different products, or changelogs for versions below the current one?)


    So my question is... are there any readily available update platforms for Delphi (server and client side too) which I can simply implement in my applications and forget about this matter?

    I'm also open to suggestions on how the thing should work, on all possible levels:

    - Backend. Is a database really needed or overkill?

    - Static vs dynamic update definitions. Dynamic is better from many perspectives but does it really worth the extra effort?

    - Protocol. Should I really stick to HTTP, or is there a better / create my own?

    - How the information is translated and sent, including how the application should know if a new file is added to the distribution?

     

    Cheers!

  17. Posted

    In some occasions the inaccuracy of Delphi's TTimer cased serious headaches so I did some digging and found an old post on how to turn a thread into a much more accurate timer. While I analyzed the code to find out how it works I made some comments and also took the liberty to change it here and there... and wrapped the whole thing in a TComponent which easily can be installed in the IDE and dropped on almost anything.

    The TThreadedTimer's OnTimer event runs in the VCL thread so any UI element can be manipulated from there. It is also a drop-in replacement for TTimer, meaning you change your DFM and PAS and it should work exactly the same way.

     

    My version delays enabling the thread so it won't spin freely until it has an event handler, properly resets the timing sequence if the Timer is set to Disabled and then Enabled again and also - in theory - the OnTimer event will no longer fire during destruction. Being a Windows-only guy it relies on WinApi, but I guess it can be made cross-platform by using Delphi's own TEvent class... as it yields zero benefits I didn't care to look into it.

     

    As the original idea wasn't mine the right thing to do is to release this version under the do-whatever-you-want license. Feel free to use, point out possible issues or modify it to fit your needs.

     

    God bless open source 🙂

    uThreadedTimer.7z

    • Like 2
    • Thanks 4

  18. Posted

    18 minutes ago, Tom F said:

    You reported these on EMB's Jira?

    No, as

    1, TZipFile is clearly abandoned (ZIP64 was introduced over 20 years ago)

    2, There is a workaround for most of these

    3, I could not wait until Delphi 25 Llanfair­pwllgwyngyll­gogery­chwyrn­drobwll­llan­tysilio­gogo­goch when it will be officially fixed 🙂

     

    Anyway. let's not hijack the topic.

    • Like 1

  19. Posted

    1 hour ago, corneliusdavid said:

    I have only used it briefly here and there. What are the limitations you've run into?

    - Inability to remove a file from the archive

    - Does not support ZIP64 and LZMA (which can be solved by Zip2)

    - Inability to read data from "corrupted" archives, like 7-zip can

    - If the ZIP file was created by C# (don't know which lib) TZipFile sometimes says that the archive is not corrupt, but there are no files in it. With 7-Zip I can see and extract the file without issues

  20. Posted

    Completely offtopic.

     

    On 3/29/2022 at 1:01 PM, Angus Robertson said:

    [...] and not migrating from something like VCLZip.

     

    I'm using an upped version of Delphi's TZipFile and always had enough of it's limitations so I went on and searched for VCLZip. Never seen a project this old before 😮

     

    image.png.f0b93105e8ec00b0a97bf2881f422943.png

    • Haha 1

  21. Posted

    On 3/24/2022 at 4:56 PM, Remy Lebeau said:

    Offhand, I would guess that a nil pointer is being accessed.

    This is really interesting, since it's in VCL260.bpl...? Whenever I made these mistakes the AV always pointed to the executable (or .dll), rarely (or never) to a Delphi internal.

    In case it's true, how exactly this can happen?

  22. Posted

    I'm not saying it is the case at you but when my 24/7 application was closing unexpectedly it was always

    1, Memory corruption

    2, Improper thread object access / synchronization

    3, In some cases memory / handle leak

     

    I found that if the code gets to a really messed up state no exceptions are raised when something happens (no more AVs), the program simply halts. My guess is somewhere in the stack... but I'm far away from this proficiency to be able to tell.

    I hope you have a test environment. Launch your application and set up performance monitors for memory and handle usages. I guess on a test machine only 1 serial port will be used, so give it 5x - 7x time, then extract your log and create a graph. If a value steadily increases - that CAN be your issue.

     

    For memory corruption, doublecheck your Move-s. I even wrote an own move procedure which in debug builds raises exceptions if I'm overflowing an area (I'm mostly using TBytes - that's how I know the limits). PChar reservations can cause this too - always reserve the memory for (Length(String) + 1) * SizeOf(Char).

    Thread things can be the hardest to find out, especially if they happen randomly. As performance is usually not crucial I tend to use a synchronized access to everything, that's why I have a generic TAtomicValue<T> class, which is handling the locking and unlocking for me. I'm using this instead of every simple value in the thread, plus inside every class which the thread allows external access to.

    Detecting memory or handle leaks is the worst. Build a test app and run each and every method 1.000.000 times. Whenever said value is increasing, something will be wrong there (also take into account that it CAN be inside a component, not necessarily your code). Or, simply purchase DeLeaker (or something similar). When I joined my current company it became a must-have; after I realized the memory management of our codebase was as messed up as the first applications I wrote. Took me more than a year to get rid of them (p.s.: I'm still wondering how noone recognized this flaw before...)

     

    "Happy" hunting. I wish you'll have your "ohgoddammit" moment soon.

×