Jump to content
jhoward1801

TJsonTextWriter out of memory

Recommended Posts

I am creating a large amount of json data using TJsonTextWriter using Delphi 10.4. When I get to around 100,000 objects being loaded into the array, and each object has around 25 values, I am getting an out of memory error.

 

I also tried the other JSON framework using the JSONObject and I get the same error.

 

What are the size and or object limits using  Readers and Writers JSON Framework: TJSonWriter?

 

Thanks

Share this post


Link to post

Well, Delphi 32-bit stops at 2Gb.  If you need more memory, I suggest a 64-bit app.

 

Share this post


Link to post

Since you did not told for what purpose you need to have everything stored to memory, another approach can be to use a database and only get/put data that you actual need for current task.

Share this post


Link to post

Pseudocode

 

fileStream.WriteLn('{');

fileStream.WriteLn('"arrOfObjects": [');

while GetObject(obj) do

begin

  JSONWriter.SaveToStream(fileStream, obj);

  fileStream.WriteLn(','); // Note - avoid writing comma after the last record, I'm too lazy to do this here

end;

fileStream.WriteLn(']');

fileStream.WriteLn('}');

Edited by Fr0sT.Brutal

Share this post


Link to post

Thank you. I was hitting the 2GB limit for 32bit applications and it was a combination of some cached memory for some large SQL calls and so my SQL writer was pushing me over the threshold so I was able to do some better house cleaning and was able to get it to work. Thanks for the responses.

Share this post


Link to post

Hello,

 

I Use a combination of TStreanWriter with TJsonTextWriter. 

I write an array of more than 1.000.000 objects with no memory errors, during all process it uses the same 15 mb o RAM.

 

Sample:

procedure WriteJSONStream(const FileName: string);
var
  StreamWriter: TStreamWriter;
begin
  StreamWriter := TStreamWriter.Create(FileName);
  try
    WriteStream(StreamWriter);
  finally
    StreamWriter.Free;
  end;
end;

procedure WriteStream(TextWriter: TTextWriter);
var
  ArrayBuilder: TJSONArrayBuilder;
  Writer: TJsonTextWriter;
  Obj: TJSONCollectionBuilder.TPairs;
  Arr: TJSONCollectionBuilder.TElements;
begin
  // ExeuteQuery
  FQuery.Open;
  // --

  Writer := TJsonTextWriter.Create(TextWriter);
  try
    ArrayBuilder := TJSONArrayBuilder.Create(Writer);
    try
      Arr := TJSONArrayBuilder(Builder).BeginArray;

      while not FQuery.eof do
      begin
        Obj := Arr.BeginObject;
        with Obj do
        begin
          Add('Field1', FQueryField1.asString);
          Add('Field2', FQueryField2.asString);
          // ...
        end;
        Obj.EndObject;

        FQuery.Next;
      end;
      Arr.EndArray;
    finally
      ArrayBuilder.Free;
    end;
  finally
    Writer.Free;
  end;
end;

 

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×