Clean cache memory after generete fast report

[Zazhir 0]

I have a button that generate multiples reports acccording to a qry (the same report, but differets data).

I'am trying to clean temporary memory after generete the report. All the calls to fastReport is inside a loop while (while qry not .Eof - generate the report).

 

The problem is that sometimes, can be like more than 2000 thousands reports, that's gonna be saved in a specfic folder (pdf type).

When i open the task maneger, is possible to see that my delphi applicaton, increase the use of memory a lot.

 

Is there a code to clean the memory after the report is generated?

How can i clean this memory after the reports is genereted?

 

//start while

	//do some stuf
	
    //call for the report
    RelatorioFastReport.frxPDFExport1.FileName := qryPesq.FieldByName('IMOV_ID_CODPREF').asString+'.pdf';
    RelatorioFastReport.frxPDFExport1.DefaultPath := frmSIG.dlgFilePDF.FileName;
    RelatorioFastReport.frxPDFExport1.ShowDialog := False;
    RelatorioFastReport.frxPDFExport1.ShowProgress := False;
    RelatorioFastReport.frxPDFExport1.OverwritePrompt := False;

    RelatorioFastReport.frxAumImovel.LoadFromFile(path+'RelAumImovel.fr3');
    RelatorioFastReport.frxAumImovel.PrepareReport();
    RelatorioFastReport.frxAumImovel.Export(RelatorioFastReport.frxPDFExport1);

    RelatorioFastReport.frxAumImovel := nil;
    RelatorioFastReport.Close;
    qryPesq.Next;

	//try to clean memory, without luck
    if Win32Platform = VER_PLATFORM_WIN32_NT then
    SetProcessWorkingSetSize(GetCurrentProcess, $FFFFFFFF, $FFFFFFFF);

    RelatorioFastReport.frxAumImovel.Clear;
//end while 

   The reports are indeed being generate, but if a have a lot of then, this error apper in my delphi applicaation https://www.technewstoday.com/not-enough-memory-resources-are-available-to-process-this-command/. How can i fix it?

Edited November 30, 2022 by Zazhir

[aehimself 388]

1 hour ago, Zazhir said:

 

    SetProcessWorkingSetSize(GetCurrentProcess, $FFFFFFFF, $FFFFFFFF);

 

When I was hired this was the first thing I removed from our legacy application.

This command only pushes most of the applications data in memory to the swap file, drastically slowing down your application when it tries to access those. It won't decrease your applications memory usage, only relocates it to the hard disk. Avoid it, it does more harm than good.

 

Btw, at us it was in a Timer's OnTimer event, ran once about every minute...

 

As for the memory usage, use a tool like DeLeaker. Put a breakpoint in your loop and make a snapshot each time. When you compare those snapshots, you'll see the objects in memory and the callstack which created said objects. It'll point you in the generic direction of what you should .Close or .Free,

 

WITH THAT SAID

 

Using Task manager to monitor your application usage is vaguely incorrect. It includes memory areas what your code already released but the OS did not take back yet due to memory allocation speed optimization as far as I know.

[programmerdelphi2k 235]

First of all, I think you must be adopting the wrong way to generate your batch reports!
Of course, if the FastReport component acts as a command that can be executed without needing to wait for its return, then it might mistakenly give you the impression that you can run multiple FastReports in sequence! As if you a "multi-thread" software!

 

I'm not discussing here whether FastReport is multi-threading or not! OK! Just to try to make a comparison with the report above! (run multiple (2000) batch report);

 

On the other hand, running 2000 tasks in an unsecured and uncontrolled manner is, to say the least, a wrong thing to do!

 

But anyway, you're already doing it. So, try to check the component properties to work with the memory in use, because FastReport provides some options!

A report and its data can be cached both in memory (to increase speed) and in a file on disk (to reduce RAM usage). There are several types of caching in Fast Report:

• “TfrxReport.EngineOptions.UseFileCache” - when True the whole text and objects of a built report are saved in a temporary file on disk; “TfrxReport.EngineOptions.MaxMemoSize” sets how many MB are reserved for the report in RAM
• “TfrxReport.PreviewOptions.PagesInCache” - the number of pages which can be kept in cached memory, which greatly increases preview speed, but uses a lot of memory (especially if there are pictures in the report)
• “TfrxReport.PreviewOptions.PictureCacheInFile” - when True all pictures in a built report are saved in a temporary file on disk, which greatly reduces memory use in reports that have a large number of pictures; but it reduces the speed

[Zazhir 0]

I've find out that this options inside the fast report component, but without luck to get this working as I want to be. 

 

EngineOptions.MaxMemSize;
EngineOptions.UseFileCache := true;

PreviewOptions.PagesInCache := true;

I need use this settup configuration, having in mind that I want do reduces the use of RAM memory... 

Have more ideas to reduces the use of it? please let me know...

 

 

[programmerdelphi2k 235]

Well, memory consumption is closely linked to the content of each report, ie the amount of components used and the when of information generated by them.
So if you carry or generate a lot of information on each report, say 1 page, 100 pages, etc ... This will be decisive for your final size in memory (RAM or disc).

Another factor is the number of consecutive executions and also present in memory while previous reports have not yet been discharged from memory.

 

It would be necessary, perhaps, to determine the maximum memory volume egixide by the highest consumption report, for example. And take this level as a midpoint in the execution of the other reports.

But, of course, sharing the 2000 reports into smaller portions of execution could also help.

 

How do you perform this batch of executions? What is your code in the software?

[Zazhir 0]

while not qryPesq.Eof do
      begin
        try
          //Scope no Lote e Imóvel que foi selecionado.
          for i := 0 to frmSIG.SIG.Items.Count - 1 do
          begin
            if not (TGIS_LayerAbstract(frmSIG.SIG.Items[i]) is TGIS_LayerPixel) then
              TGIS_LAYERVECTOR(frmSIG.SIG.Items[i]).DeselectAll;
          end;

          geo_layer_vector:= TGIS_LayerVector( frmsig.SIG.Get('Imovel.legt') );
          if not Assigned(geo_layer_vector) then
          begin
            showMessage('O layer Imóvel não está aberto. Operação cancelada.');
            Exit;
          end;
          geo_layer_vector.Scope := 'IMOV_ID_CODPREF = ' +#39+ qryPesq.FieldByName('IMOV_ID_CODPREF').asString+#39;

          geo_layer_vector:= TGIS_LayerVector( frmsig.SIG.Get('Piscina.legt') );
          if not Assigned(geo_layer_vector) then
          begin
            showMessage('O layer Piscina não está aberto. Operação cancelada.');
            Exit;
          end;
          geo_layer_vector.Scope := 'PISC_ID_CODPREF = ' +#39+ qryPesq.FieldByName('IMOV_ID_CODPREF').asString+#39;
          frmSIG.SIG.Refresh;

          geo_layer_vector:= TGIS_LayerVector( frmsig.SIG.Get('Lote.legt') );
          if not Assigned(geo_layer_vector) then
          begin
            showMessage('O layer Lote não está aberto. Operação cancelada.');
            Exit;
          end;
          geo_layer_vector.Scope := 'LOTE_ID_INSCRICAO = ' +#39+ qryPesq.FieldByName('IMOV_ID_CODPREF').asString+#39;
          frmSIG.SIG.Refresh;

          case geo_layer_vector.DefaultShapeType of
            gisShapeTypePolygon:
              begin
                qryUID := TADOQuery.create(self);
                qryUID.connection := frmSIG.ADOConnection1;

                qryUID.Close;
                qryUID.SQL.text := 'Select GID from Lote_FEA where LOTE_ID_INSCRICAO = '+#39+qryPesq.FieldByName('IMOV_ID_CODPREF').AsString+#39;
                qryUID.Open;

                geo_shape_Polygon :=  TGIS_ShapePolygon(geo_layer_vector.GetShape(qryUID.FieldByName('GID').AsInteger));
                if geo_shape_Polygon <> nil then
                begin
                  geo_shape_Polygon :=
                    TGIS_ShapePolygon(geo_layer_vector.GetShape(qryUID.FieldByName('GID').AsInteger)
                    .MakeEditable);
                  geo_shape := geo_shape_Polygon;

                  coordenadas.XMax := geo_shape.Extent.XMax + 10;
                  coordenadas.YMax := geo_shape.Extent.YMax + 10;
                  coordenadas.XMin := geo_shape.Extent.XMin - 10;
                  coordenadas.YMin := geo_shape.Extent.YMin - 10;
                end;
              end;
          end;

          //Centraliza a tela para tirar a foto do Croqui
          frmSIG.SIG.VisibleExtent := coordenadas;
          frmSIG.SIG.Center := geo_shape.Centroid;

          //Cria uma pasta chamada "Croqui" dentro da pasta de fotos do Imóvel e exporta.
          if not DirectoryExists(frmSIG.fotoImovel+'\Croqui') then
            MkDir(frmSIG.fotoImovel+'\Croqui');

          geo_extend.XMax := frmSIG.SIG.VisibleExtent.XMax + 10;
          geo_extend.YMax := frmSIG.SIG.VisibleExtent.YMax + 10;
          geo_extend.XMin := frmSIG.SIG.VisibleExtent.XMin - 10;
          geo_extend.YMin := frmSIG.SIG.VisibleExtent.YMin - 10;

          frmSig.SIG.ExportToImage(frmSIG.fotoImovel+'\Croqui\'+qryPesq.FieldByName('IMOV_ID_CODPREF').asString+'.jpg',geo_extend,1024,768,90,0,96);


          //Limpa o Scope de Lote e Imóvel para voltar o mapa normal
          geo_layer_vector:= TGIS_LayerVector( frmsig.SIG.Get('Imovel.legt') );
          geo_layer_vector.Scope := '';

          geo_layer_vector:= TGIS_LayerVector( frmsig.SIG.Get('Lote.legt') );
          geo_layer_vector.Scope := '';
          frmSIG.SIG.Refresh;

          if not Assigned(RelatorioFastReport) then
            RelatorioFastReport := TRelatorioFastReport.Create(self);

          path := ExtractFilePath(Application.ExeName);
          RelatorioFastReport.qryAumImovel.Close;
          RelatorioFastReport.qryAumImovel.SQL.Text := 'Select Lote_FEA.*, Imovel_FEA.*,(Imovel_FEA.IMOV_NR_AREA_CONSTRUIDA_PREF * 1.1) as DezPorCento '+
                                                       'from Imovel_FEA inner join Lote_FEA '+
                                                       'on Imovel_FEA.IMOV_ID_CODPREF = Lote_FEA.LOTE_ID_INSCRICAO '+
                                                       'where Imovel_FEA.IMOV_ID_CODPREF = '+#39+qryPesq.FieldByName('IMOV_ID_CODPREF').asString+#39+' and IMOV_TX_TIPO_IMOVEL = '+#39+'PRINCIPAL'+#39;
          RelatorioFastReport.qryAumImovel.Open;


          RelatorioFastReport.frxPDFExport1.FileName := qryPesq.FieldByName('IMOV_ID_CODPREF').asString+'.pdf';
          RelatorioFastReport.frxPDFExport1.DefaultPath := frmSIG.dlgFilePDF.FileName;
          RelatorioFastReport.frxPDFExport1.ShowDialog := False;
          RelatorioFastReport.frxPDFExport1.ShowProgress := False;
          RelatorioFastReport.frxPDFExport1.OverwritePrompt := False;

          RelatorioFastReport.frxAumImovel.LoadFromFile(path+'RelAumImovel.fr3');
          RelatorioFastReport.frxAumImovel.PrepareReport();
          RelatorioFastReport.frxAumImovel.Export(RelatorioFastReport.frxPDFExport1);

          RelatorioFastReport.frxAumImovel := nil;
          RelatorioFastReport.Close;
          qryPesq.Next;



         //deleta todos os arquivos com extensão de desenho
         try
            filesDirectory :=  frmSIG.fotoImovel+'\Croqui';
            FindFirst(filesDirectory + '\*.tab', faAnyFile, SearchRec);
            repeat
               DeleteFile(filesDirectory + '\' + SearchRec.name);
            until FindNext(SearchRec) <> 0;
         finally
            FindClose(SearchRec);
         end;
         //deleta todos os arquivos com extensão de desenho
         try
            filesDirectory :=  frmSIG.fotoImovel+'\Croqui';
            FindFirst(filesDirectory + '\*.JGW', faAnyFile, SearchRec);
            repeat
               DeleteFile(filesDirectory + '\' + SearchRec.name);
            until FindNext(SearchRec) <> 0;
         finally
            FindClose(SearchRec);
         end;
         //deleta todos os arquivos com extensão de desenho
         try
            filesDirectory :=  frmSIG.fotoImovel+'\Croqui';
            FindFirst(filesDirectory + '\*.jpg.prj', faAnyFile, SearchRec);
            repeat
               DeleteFile(filesDirectory + '\' + SearchRec.name);
            until FindNext(SearchRec) <> 0;
         finally
            FindClose(SearchRec);
         end;
        except
          on E: Exception do
          begin
            ShowMessage('Algo deu errado na geração de notificação!' + #13 + E.Message);
            abort
          end;
        end;
      end;
        //maximiza a tela apos gerar todos...
        frmSIG.WindowState := wsMaximized;

So, as i explain before, the thing is inside my loop.

 

I've found that deleting some files that are generete to be send to inside the report, decrease the use of memory, after generete the .jpg and send to the report, i delete some others files that are genereted in the same time.

 

Other thing is that I minimezed the window, read about that doing it can avoid increase the use of ram. 

 

After set this improvements, are generating more than the usual reports, guess I'am in the right way to get this done. 

 

[Lajos Juhász 238]

It's hard to verify code this large. One thing you're for every step create a query:

 

gisShapeTypePolygon:
              begin
                qryUID := TADOQuery.create(self);

This is not going to be a definite leak as it has self (most probably the form as the owner). The query will remain open until the owner is destroyed.

[Stano 139]

I can't understand the code 
But is it necessary to create a new TQuery every time? I can't judge. In such cases, I have only one TQuery and I only change its SQL.Text.

[programmerdelphi2k 235]

@Zazhir

 

first: YOUR CODE IS, at least: CONFUSED! .. bad!

 

let's start:

• divide it in "small parts":  each part, do some!!! "single responsability"
  • create a procedure for each phase, then, you can work it separately
• Do not mix Query responsibilities with decision-making responsibilities on data to be used! :<
  • you use:
    • Query;
    • gisOBJECTxxxx ... Exporting to file.... etc...
    • Files procedure  .. FInd, Close, etc....
    • etc...
  • a complete Chaos!

 

if you need "get" some info before generate your "QUERY", AND after this, gerenate your FASTREPORT, then, do it:

• first "get the info necessary to create your SQL text"
  • do all avaliations necessary here
  • dont put all in a "loop" like your, because if any part needs a long time to process, the report will go wait it for always....
• now, create your SQL Text, as expected
• if all ok above, then you can use your file *.FR3 FastReport to feed it.

IF YOU BE USING FireDAC components or not, you can use "PARAMS" from this components to feed it, and many others tasks... then, many code can be "take out"

 

--------- try implement this idea... im using "IF" just for show how would be ... ok?

 

procedure MyDeselectAllInFrmSIG;
begin
	for i := 0 to frmSIG.SIG.Items.Count - 1 do
        begin
			if not (TGIS_LayerAbstract(frmSIG.SIG.Items[i]) is TGIS_LayerPixel) then
            TGIS_LAYERVECTOR(frmSIG.SIG.Items[i]).DeselectAll;
        end;
end;

function MyComponentPropertyItsOK( AComp:TGIS_LayerVector; AYourMsgError:string; const AFrmSig: TYourFrmSigType; const ALegtFileName:string; out AErrMsg:string ):boolean;
begin
	result := false;
	//
	if not(AComp is TGIS_LayerVector) or (AComp = nil) then
	begin
		AErrMsg := 'AComp is not "TGIS_LayerVector" or is nil');
		//
		exit;
	end;
	//
	if not(AFrmSig is TYourFrmSigType) or (AFrmSig = nil) then
	begin
		AErrMsg := 'AFrmSig is not "TYourFrmSigType" or is nil');
		//
		exit;
	end;
	//
	if (ALegtFileName='') then
		AErrMsg := 'ALegtFileName is empty');
		//
		exit;
	end;
	//
    try
		AComp:= TGIS_LayerVector( AFrmSig.SIG.Get(ALegtFileName) ) then
		//
		if Assinged(AComp) then
        	result := true;
        else
            AErrMsg := AYourMsgError;
	except
		on E:Exception do
			AErrMsg := E.Message;
	end;
end;
		  

procedure YourProcedureXXXXX;		  
var
	MyErrMsg:string;
	MyGeoLayerVectorScope:string;
	LCompOkForUsage:boolean;
	MyQueryUID:TADOQuery;
begin
...
	//
    MyDeselectAllInFrmSIG;
    //
	MyQueryUID:= TADOQuery.Create(nil); // create just one time!!!! OUT OF LOOPING...
	MyQueryUID.Connection := xxxxx;
    //
    // THIS would can be "into" your looping, but I think that is wrong... because your looping is very big (2000 iterations...)
	try
		MyErrMsg:='';
		MyGeoLayerVectorScope:='';
        //
        // using this way for show how would be... of course, you can use another approach (right way)
		if MyComponentPropertyItsOK( geo_layer_vector,'Imove not for use', AFrmSig, 'Imovel.legt', MyErrMsg) then
			MyGeoLayerVectorScope :=  'IMOV_ID_CODPREF = ' +#39+ qryPesq.FieldByName('IMOV_ID_CODPREF').asString+#39;
		else if MyComponentPropertyItsOK( geo_layer_vector,'Piscine not for use', AFrmSig, 'Piscina.legt', MyErrMsg) then
			MyGeoLayerVectorScope := 'PISC_ID_CODPREF = ' +#39+ qryPesq.FieldByName('IMOV_ID_CODPREF').asString+#39
		else if MyComponentPropertyItsOK( geo_layer_vector,'Lote not for use', AFrmSig, 'Lote.legt', MyErrMsg) then
			MyGeoLayerVectorScope := 'LOTE_ID_INSCRICAO = ' +#39+ qryPesq.FieldByName('IMOV_ID_CODPREF').asString+#39;		
		//
		if (MyGeoLayerVectorScope='') then
		begin
			ShowYourErrorMessage( MyErrMsg OR AYourMsgError ); // showing Dialog into your "Looping" will go stop all flow...
			//
			exit;
		end;
		//
		geo_layer_vector.Scope :=  MyGeoLayerVectorScope;
...
		MyQueryUID.Close; // close to reusage!
		//
		MyQueryUID.SQL.Text := Format('Select GID from TableX where %s',[MyGeoLayerVectorScope]);
	...
	// now, you FastReport here... then you need to know that any problem in this task will can crash the rest of "loop" query...
	//
	...  what to do from forward
	///
	///
	finally
		MyQueryUID.Free;
	end;
end;
	

 

Edited December 1, 2022 by programmerdelphi2k

[programmerdelphi2k 235]

just for my curiosity:

--- your original code , works in real world?  (of course... no talking about your out-of-memory)

Edited December 1, 2022 by programmerdelphi2k

[Zazhir 0]

10 minutes ago, programmerdelphi2k said:

just for my curiosity:

--- your original code , works in real world?  (of course... no talking about your out-of-memory)

Yes, this code is for generate a multiples reports of house increase area with the picture of it geolocated. The code works, but generate like 500 reports than gives a error of memory space (not HD memory). 

[programmerdelphi2k 235]

reflection:

• Imagine multiplying 500x all objects generated in just one report, and you'll have an idea of how much memory your app needs to run!

•  

  Note: it doesn't matter if your computer has a lot more memory than that! Your app will only use a limited amount of memory to keep track of the objects it creates or uses!

[Zazhir 0]

This is the error that i got, It is in portuguese because is my birth language

[programmerdelphi2k 235]

of course, there are many object on memory in this moment! (many FastReports object or any other that you use)

 

my tip:

1. try start with max 100 on loop...
2. use the MS TaskManager to see the memory consume
3. go on until your "2000" to see the chaos!

look, while you dont "re-factor" your code, this situation always will be your rotine!

• keep in mind that if you put all the tasks inside the Query.EOF "loop"... you will be replaying all the same occurrences until you reach the last record in the query!

• So if you access 100 objects in the life of your loop, then you will have 100x the memory used by the first one, plus the amount of memory used by FastReport to generate the PDFs on disk, etc...

Edited December 1, 2022 by programmerdelphi2k