How to solve System Out of Resources in TScrollBox

[araujoarthur 0]

Hello! I am writing a log viewer using Delphi VCL and AlmediaDev's StyleControls, and my approach to show a styled list of log entries was to add a TScrollBox (TscScrollBox actually, using AlmediaDev StyleControls, which extends the original), and dynamically fill it with frames (one frame per log entry) as the following image shows:

 

 

The Frame itself looks like this:

 

And has this component tree:

 

The frames have variable width according to the window width, but the height is always fixed.

 

This approach works for smaller log files (The above screenshot was taken with 217 records) but fails with "Out of System Resources" and "Canvas Does Not Allow Drawing" with bigger files (my failed test file had 3000 entries). Although I understand why is this happening  (or atleast I believe it's because the windows won't let me draw 3.000 controls) I don't know what approach I could take to either improve the rendering to reduce resource usage or change the way it renders frames without affecting the scroll list size (otherwise I could just "remove" the frames out of sight and insert the ones that are on sight) and entry UI style (As I believe I would have to if changed for a TVirtualListView as suggested in topic/3661-how-to-detect-when-control-is-scrolled-into-view/).

Can anyone give me any direction on what should I do here? Right now it's really more a matter of "WHAT should I do" than "HOW do I do that". 

For quick reference, I'm also adding the code for the TEntryListItemTemplate and the code that creates and feed it into the scrollbox. It's also fully available here.

 

TEntryListItemTemplate

type
  TEntryListItemTemplate = class(TFrame)
    svgLogIcon: TSkSvg;
    labelDateTime: TscGPLabel;
    scGPPanel1: TscGPPanel;
    labelDescription: TscGPLabel;
    scGPPanel2: TscGPPanel;
    scGPPanel3: TscGPPanel;
    scGPGlyphContainerButton2: TscGPGlyphContainerButton;
    svgEyeIcon: TSkSvg;
  private
    { Private declarations }
    FAssociatedRecord: TLogEntry;
    FAssociatedControlIndex: Integer; // TODO: When one is removed all indexes must above must change (needs a routine)
    procedure SetupWarningIcon();
    procedure SetupErrorIcon();
    procedure SetupInfoIcon();
  public
    { Public declarations }
    constructor Create(AOwner: TComponent; ARecord: TLogEntry; AControlIndexInList: Integer); reintroduce;
  end;

const
  {
    The Following Icons are part of Google's Material Core Icon Pack
  }
  ERROR_ICON = '''
   ...
  ''';

  INFO_ICON = '''
  ...
  ''';

  WARNING_ICON = '''
  ...
  ''';

implementation

{$R *.dfm}

{ TEntryListItemTemplate }

constructor TEntryListItemTemplate.Create(AOwner: TComponent; ARecord: TLogEntry; AControlIndexInList: Integer);
begin
  inherited Create(AOwner);
  FAssociatedRecord := ARecord;
  FAssociatedControlIndex := AControlIndexInList;
  case FAssociatedRecord.Severity of
    lsUNKNOWN: SetupInfoIcon();
    lsUNIMPORTANT: SetupInfoIcon();
    lsREQUESTRECEIVED: SetupInfoIcon();
    lsINFORMATION: SetupInfoIcon();
    lsWARNING: SetupWarningIcon();
    lsERROR: SetupErrorIcon();
  end;

  labelDescription.Caption := Copy(FAssociatedRecord.Message, 0, 150);
  labelDateTime.Caption := DateTimeToStr(FAssociatedRecord.Date);

end;

procedure TEntryListItemTemplate.SetupErrorIcon;
begin
  svgLogIcon.Svg.OverrideColor := $FFFF2D2D;
  svgLogIcon.Svg.Source := ERROR_ICON;
end;

procedure TEntryListItemTemplate.SetupInfoIcon;
begin
  svgLogIcon.Svg.OverrideColor := $FF2D4DFF;
  svgLogIcon.Svg.Source := INFO_ICON;
end;

procedure TEntryListItemTemplate.SetupWarningIcon;
begin
  svgLogIcon.Svg.OverrideColor := $FFFFAB2D;
  svgLogIcon.Svg.Source := WARNING_ICON;
end;

 

Creation and Insertion in TscScrollBox:

procedure TfrmMain.Open1Click(Sender: TObject);
var
  OpenedFile: TLogFile;
begin
  OpenedFile := ActionUtils.OpenFile;
  if OpenedFile <> nil then
  begin
    gStateHolder.CurrentFile := OpenedFile;
    gStateHolder.HasOpenFile := True;

    var Progress: Extended := 0;
    var Increment: Extended := 100/Length(gStateHolder.CurrentFile.FLogEntries);
    ProgressBarInc(0);

    {
      I decided to make invisible while I add the entries so the it doesn't have to draw every single entry on each insert.
      By setting it to Visible := False before the loop and True after the loop I got it to render all at once, which
      **Improved significantly the Load time for 217 records**. Still fails for 3.000
    }
    contentScrollBox.Visible := False;
    for var I := Length(gStateHolder.CurrentFile.FLogEntries) - 1 downto 0 do
    begin
      var rec := gStateHolder.CurrentFile.FLogEntries[I];
      var frame := TEntryListItemTemplate.Create(contentScrollBox, rec, contentScrollBox.ControlCount);
      frame.Name := TypeUtils.GenerateFrameName();
      contentScrollBox.InsertControl(frame);
      Progress := Progress + Increment;
      ProgressBarInc(Round(Progress));
    end;

    contentScrollBox.Visible := True;  // System out of Resources. How to deal with?

  end;
end;

Thank you for your time!

 

[stijnsanders 37]

If you have a potentially really long list, and you create components in memory (and with GUI elements), but only have a few of them at once on screen, that's a lot of wasted resources, so the 'out of resources' error you get may be actually correct.

 

If I recall correctly, what things like VirtualTreeList do, and you can do this yourself with TListView by switching property Style to dlOwnerDrawFixed or lbOwnerDrawVariable and using the OnDrawItem event, is actually just draw the information of the items when they are on screen. By setting the numer of items, the scrollbars show as if there are that many items, but it is in fact a kind of illusion.

 

I wish I had a nice example to show you, but I only have this one I once wrote for a complete code-comparing tool. I hope you can find more examples if you look for them on the net.

Edited Thursday at 09:52 PM by stijnsanders

[Vincent Parrett 763]

Using a TScrollbox with thousands of child controls is a terrible idea, apart from the memory overhead the performance would not be greate. As others have said, using a virtual list control is much better solution. 

 

This is what I use for these scenarios - you do have to wite the paint code yourself but it's very lightweight

 

https://github.com/VSoftTechnologies/VSoft.VirtualListView - supports XE2 - D12 and vcl themes - although I have only tested themes within an IDE plugin

 

There is a demo app and you can also see it in use here 

 

https://github.com/DelphiPackageManager/DPM/blob/master/Source/IDE/EditorView/DPM.IDE.EditorViewFrame.pas

 

[Dmitry Arefiev 106]

... or TControlList

[Remy Lebeau 1436]

For my custom log viewer, I used a standard TListView in virtual mode with owner-drawn items, and it handles millions of items (GB-sized log files) just fine with low overhead. The hardest part was implementing a caching mechanism for the on-screen items, as I also filter and search items so don't want to keep the whole file in memory at one time.

Edited Friday at 03:36 PM by Remy Lebeau

[ToddFrankson 3]

I use an In memory table (TFDMemtable) for somethings I am working on, assigning only the records that will be visible on the screen at any given time.  So if I will see 20 -30 items, I reuse the controls, just change the data in them as someone "scrolls" .  It's an illusion, but visually it seems extremely fast, and low memory overhead due to only a set number of visual components 

[araujoarthur 0]

On 12/19/2024 at 6:51 PM, stijnsanders said:

If you have a potentially really long list, and you create components in memory (and with GUI elements), but only have a few of them at once on screen, that's a lot of wasted resources, so the 'out of resources' error you get may be actually correct.

 

If I recall correctly, what things like VirtualTreeList do, and you can do this yourself with TListView by switching property Style to dlOwnerDrawFixed or lbOwnerDrawVariable and using the OnDrawItem event, is actually just draw the information of the items when they are on screen. By setting the numer of items, the scrollbars show as if there are that many items, but it is in fact a kind of illusion.

 

I wish I had a nice example to show you, but I only have this one I once wrote for a complete code-comparing tool. I hope you can find more examples if you look for them on the net.

Hello! Although I have been able to add controls to the TListView I wasn't able to make it scrollable. I have set both OwnerData and OwnerDrawn to true. I couldn't find a property Style that can hold dlOwnerDrawFixed or lbOwnerDrawVariable.

 

On 12/20/2024 at 12:32 PM, Remy Lebeau said:

For my custom log viewer, I used a standard TListView in virtual mode with owner-drawn items, and it handles millions of items (GB-sized log files) just fine with low overhead. The hardest part was implementing a caching mechanism for the on-screen items, as I also filter and search items so don't want to keep the whole file in memory at one time.

Once I can make it scrollable I'd like to implement caching. You do so by storing a chunk of the file and the visible ones in a separate (from the visual control) data structure?

[araujoarthur 0]

UPDATE: By setting its viewstyle to vsReport I got a scroll bar that isn't functional (in the sense that it doesn't move between controls even if all have been drawn). Is it related to the fact that I'm inserting controls via TListView.InsertControl() procedure? Or do I have to write the scroll procedure myself?

Edit: I guess I'm also failing to understand what is a TListView operating in virtual mode.

Edited yesterday at 12:16 AM by araujoarthur

[Remy Lebeau 1436]

5 hours ago, araujoarthur said:

I wasn't able to make it scrollable. I have set both OwnerData and OwnerDrawn to true.

Set TListView.OwnerData=true, set TListView.Items.Count to the number of items you want to display, and use the TListView.OnData event to provide details for each item on demand.  The scrollbar will be handled for you.

5 hours ago, araujoarthur said:

I couldn't find a property Style that can hold dlOwnerDrawFixed or lbOwnerDrawVariable.

Because those are TListBox styles, not TListView styles.  In this situation, set TListView.ViewStyle=vsReport, and add some columns to the TListView.Columns.  Then, in the TListView.OnData event, you can provide data for the 1st column using TListItem.Caption, and data for the subsequent columns using TListItem.SubItems.

Quote

Once I can make it scrollable I'd like to implement caching.

Use the TListView.OnData... events for that purpose.

Quote

You do so by storing a chunk of the file and the visible ones in a separate (from the visual control) data structure?

Yes.

1 hour ago, araujoarthur said:

UPDATE: By setting its viewstyle to vsReport I got a scroll bar that isn't functional

Probably because you didn't populate the TListView with data yet (see above).

1 hour ago, araujoarthur said:

(in the sense that it doesn't move between controls even if all have been drawn). Is it related to the fact that I'm inserting controls via TListView.InsertControl() procedure?

Yes. You should not be putting any child controls on to the TListView at all.  It is not designed for that.  Let it handle the items for you.  You just worry about supplying the items with data.  And using the TListView.On...Draw... events if you want to custom-draw the items.

1 hour ago, araujoarthur said:

Or do I have to write the scroll procedure myself?

No.

1 hour ago, araujoarthur said:

Edit: I guess I'm also failing to understand what is a TListView operating in virtual mode.

It is just a wrapper for the native behavior provided by Microsoft, see:

How to Use Virtual List-View Controls

Edited yesterday at 01:01 AM by Remy Lebeau

[araujoarthur 0]

13 hours ago, Remy Lebeau said:

Set TListView.OwnerData=true, set TListView.Items.Count to the number of items you want to display, and use the TListView.OnData event to provide details for each item on demand.  The scrollbar will be handled for you.

Because those are TListBox styles, not TListView styles.  In this situation, set TListView.ViewStyle=vsReport, and add some columns to the TListView.Columns.  Then, in the TListView.OnData event, you can provide data for the 1st column using TListItem.Caption, and data for the subsequent columns using TListItem.SubItems.

Use the TListView.OnData... events for that purpose.

Yes.

Probably because you didn't populate the TListView with data yet (see above).

Yes. You should not be putting any child controls on to the TListView at all.  It is not designed for that.  Let it handle the items for you.  You just worry about supplying the items with data.  And using the TListView.On...Draw... events if you want to custom-draw the items.

No.

It is just a wrapper for the native behavior provided by Microsoft, see:

How to Use Virtual List-View Controls

So if I understand this right, I cannot use my frames as "entries" in TListView, I must use the TListView columns to actually draw the layout I want. Following from this understanding, how would I add a button (like the eye button on the right) to this custom drawn list item (TListItems don't accept controls afaik)

[Remy Lebeau 1436]

2 hours ago, araujoarthur said:

So if I understand this right, I cannot use my frames as "entries" in TListView

Correct.  If you want to use your Frames, then I would suggest using a plain TPanel, put your visible Frames on it, and then use a separate TScrollBar to "scroll" through your data, updating the Frames accordingly.

 

If you want to use a virtual TListView, you will have to custom-draw anything beyond simple text. You can try using the TListItem.SubItemImages property or TListView.OnGetSubItemImage event, but I'm not sure if those work in virtual mode or not.  I have never used them.  I always custom-draw my own images.

2 hours ago, araujoarthur said:

Following from this understanding, how would I add a button (like the eye button on the right) to this custom drawn list item (TListItems don't accept controls afaik)

You can't add an actual button control to a TListView item.  You would have to custom-draw an image of a button instead, and then handle the TListView.OnMouse(Down|Up) events to know where the user is clicking within the TListView. You can use the TListView.GetItemAt() and TListView.GetHitTestInfoAt() methods to know which item is underneath the mouse, and in some cases even which portion of the item.  However, in the case of your eye button, that is not granular enough, so I would suggest sending an LVM_SUBITEMHITTEST message to the TListView window, which will tell you the item and column indexes that are at a given coordinate.  And if needed, there is also the LVM_GETSUBITEMRECT message which gets the display coordinates of a given column within an item.

Edited yesterday at 05:23 PM by Remy Lebeau

[araujoarthur 0]

UPDATE

 

The current state of the application is shown in the picture:

 

It's not quite my objective. Still I've learned a lot during the process. I'm still going to implement the Icons (instead of text) in severity and Actions.

Also, I wonder if there's a way to change height of the TListItem without using the imagelist trick and if there's a way to let the user select and copy text drawn with  TextOut().

 

 

[Remy Lebeau 1436]

3 hours ago, araujoarthur said:

Also, I wonder if there's a way to change height of the TListItem without using the imagelist trick

There are three options for influencing the height of TListView items:

 

1. Assign an ImageList of desired size.

2. Assign a Font of sufficient size.

3. Owner-draw the TListView and handle the WM_MEASUREITEM notification in the parent window (or subclass the TListView to handle CN_MEASUREITEM). 

3 hours ago, araujoarthur said:

if there's a way to let the user select and copy text drawn with  TextOut().

No, there is not.