TJsonTextWriter out of memory

[jhoward1801 0]

I am creating a large amount of json data using TJsonTextWriter using Delphi 10.4. When I get to around 100,000 objects being loaded into the array, and each object has around 25 values, I am getting an out of memory error.

 

I also tried the other JSON framework using the JSONObject and I get the same error.

 

What are the size and or object limits using  Readers and Writers JSON Framework: TJSonWriter?

 

Thanks

[Lars Fosdal 1792]

Well, Delphi 32-bit stops at 2Gb.  If you need more memory, I suggest a 64-bit app.

 

[David Heffernan 2345]

Use a SAX style parser rather than DOM style parser is sometimes an option

[KodeZwerg 54]

Since you did not told for what purpose you need to have everything stored to memory, another approach can be to use a database and only get/put data that you actual need for current task.

[Uwe Raabe 2057]

Please show your code - otherwise we all can just guess.

[Fr0sT.Brutal 900]

Pseudocode

 

fileStream.WriteLn('{');

fileStream.WriteLn('"arrOfObjects": [');

while GetObject(obj) do

begin

  JSONWriter.SaveToStream(fileStream, obj);

  fileStream.WriteLn(','); // Note - avoid writing comma after the last record, I'm too lazy to do this here

end;

fileStream.WriteLn(']');

fileStream.WriteLn('}');

Edited May 13, 2021 by Fr0sT.Brutal

[jhoward1801 0]

Thank you. I was hitting the 2GB limit for 32bit applications and it was a combination of some cached memory for some large SQL calls and so my SQL writer was pushing me over the threshold so I was able to do some better house cleaning and was able to get it to work. Thanks for the responses.

[Der schöne Günther 316]

In that case, I would probably resort to writing them to a string "by hand" and flushing the string to disk every few megabytes (or using a TBufferedFileStream).

[Josep 8]

Hello,

 

I Use a combination of TStreanWriter with TJsonTextWriter. 

I write an array of more than 1.000.000 objects with no memory errors, during all process it uses the same 15 mb o RAM.

 

Sample:

procedure WriteJSONStream(const FileName: string);
var
  StreamWriter: TStreamWriter;
begin
  StreamWriter := TStreamWriter.Create(FileName);
  try
    WriteStream(StreamWriter);
  finally
    StreamWriter.Free;
  end;
end;

procedure WriteStream(TextWriter: TTextWriter);
var
  ArrayBuilder: TJSONArrayBuilder;
  Writer: TJsonTextWriter;
  Obj: TJSONCollectionBuilder.TPairs;
  Arr: TJSONCollectionBuilder.TElements;
begin
  // ExeuteQuery
  FQuery.Open;
  // --

  Writer := TJsonTextWriter.Create(TextWriter);
  try
    ArrayBuilder := TJSONArrayBuilder.Create(Writer);
    try
      Arr := TJSONArrayBuilder(Builder).BeginArray;

      while not FQuery.eof do
      begin
        Obj := Arr.BeginObject;
        with Obj do
        begin
          Add('Field1', FQueryField1.asString);
          Add('Field2', FQueryField2.asString);
          // ...
        end;
        Obj.EndObject;

        FQuery.Next;
      end;
      Arr.EndArray;
    finally
      ArrayBuilder.Free;
    end;
  finally
    Writer.Free;
  end;
end;