JSON benchmarks

[pyscripter 842]

hydrobyte/TestJSON: A simple project to test JSON libraries with Delphi and C++Builder. presents JSON library benchmarks comparing a large number of alternatives.

 

One thing that strikes me, is that the System.JSON rtl library is doing relatively well compared to the competition, both in terms of performance and in terms of JSON validation.   With the exception of Find, is very competitive in all other areas.

 

I have seen many claims that System.JSON is very slow and that the xyz library is so many times faster.   Do these benchmarks suck (like most benchmarks)?   Or is it the case that System.JSON is not that bad?  What is your experience?

[Tommi Prami 159]

13 hours ago, pyscripter said:

hydrobyte/TestJSON: A simple project to test JSON libraries with Delphi and C++Builder. presents JSON library benchmarks comparing a large number of alternatives.

Way cool! 

Is there test/comparison is Objects and/or Record serialization supported and speed etc?? Did not check code, because bit busy...

I've used Grijjy in one of my hobby projects for serializing objects and/or records. Can't remember which they all are, poossibly both)

Edited May 16 by Tommi Prami

[pyscripter 842]

1 hour ago, Tommi Prami said:

Is there test/comparison is Objects and/or Record serialization supported and speed etc??

No. Most of the alternatives do not support serialization.  Grizzy and Superobject do.   Delphi offers a couple of ways.   But serialization is not  necessarily dependent on JSON parsing.   For example NEON is using System.JSON.   

Edited May 16 by pyscripter

[Stefan Glienke 2187]

The tests ignore some subtle differences in the implementations. While some implementations ensure that you don't create multiple pairs with the same name the System.JSON AddPair method simply ignores that.

 

You can test that by adding a fLib.Add(SKey, SVal); after the loop in TTestSpeedRun.DoSubTestGen - some libraries will report 50000 items, while some report 50001 - System.JSON for example.

The fact that JDO uses a linear search here is a bit disappointing and the reason it completely falls off the cliff on the generate test.

[Vincent Gsell 14]

16 hours ago, pyscripter said:

I have seen many claims that System.JSON is very slow and that the xyz library is so many times faster.   Do these benchmarks suck (like most benchmarks)?   Or is it the case that System.JSON is not that bad?  What is your experience?

Hi, 

For me, certains of the fastest JSON  libraries that I practiced is just not present (among others, Mormot, pasjson, etc.).
 

41 minutes ago, Stefan Glienke said:

The tests ignore some subtle differences in the implementations. While some implementations ensure that you don't create multiple pairs with the same name the System.JSON AddPair method simply ignores that.

Indeed ! And validation test could be usefull in this kind of bench ! 

 

It is astonishing that many libs does not pass certain basic "proper Json format" validation ! 

• An empty object can be represented by {}
• Arrays are encapsulated within opening and closing square brackets
• An empty array can be represented by []
• Each member should have a unique key within an object structure

[Der schöne Günther 340]

As far as I know, nowhere does it specify that keys must be unique.

[Stefan Glienke 2187]

5 minutes ago, Der schöne Günther said:

As far as I know, nowhere does it specify that keys must be unique.

No, but according to RFC 8259, they *should* be unique.

 

I just pointed out that by supporting that feature, there needs to be done more than simply adding pairs without any lookup.

In fact, it looks like some implementations don't even allow adding duplicate names.

Edited May 16 by Stefan Glienke

[Lars Fosdal 1946]

IMO, if you need multiple values for a field, you should use a list element.

Duplicate key/value pairs in JSON just doesn't make sense.
https://JsonLint.com does not like them either.

[pyscripter 842]

Since you asked for Serializer benchmakrs:

 

paolo-rossi/delphi-neon: JSON Serialization library for Delphi includes a benchmark against the Rest.Json serializer and it beats it hand down:

 

 

I have replaced the Rest.Json serializer with the System.Json.Serializers TJSONSerializer.  Here are the results:

 

 

So now TJsonSerializer beats Neon hands down.

 

TJsonSerializer looks good but it has some rough edges.   To run the benchmarks I had to add a converter that handles Enumerated values as strings instead of the default integers:

type
  TEnumStringConverter = class(TJsonConverter)
  public
    function CanConvert(ATypeInf: PTypeInfo): Boolean; override;
    function ReadJson(const AReader: TJsonReader; ATypeInf: PTypeInfo; const AExistingValue: TValue;
      const ASerializer: TJsonSerializer): TValue; override;
    procedure WriteJson(const AWriter: TJsonWriter; const AValue: TValue;
      const ASerializer: TJsonSerializer); override;
  end;

{ TEnumStringConverter }

function TEnumStringConverter.CanConvert(ATypeInf: PTypeInfo): Boolean;
begin
  // This converter can handle any type that is an enumeration
  Result := (ATypeInf.Kind = TTypeKind.tkEnumeration) and
    (ATypeInf <> TypeInfo(Boolean));
end;

function TEnumStringConverter.ReadJson(const AReader: TJsonReader; ATypeInf:
    PTypeInfo; const AExistingValue: TValue; const ASerializer:
    TJsonSerializer): TValue;
var
  LIntValue: Integer;
begin
  LIntValue := System.TypInfo.GetEnumValue(ATypeInf, AReader.Value.AsString);

  if LIntValue = -1 then // GetEnumValue returns -1 if the name is not found
    raise EJsonSerializationException.CreateFmt('Invalid string value "%s" for enumeration "%s".', [AExistingValue.AsString, ATypeInf.Name]);

  // Create a TValue of the specific enum type using its ordinal value.
  Result := TValue.FromOrdinal(ATypeInf, LIntValue);
end;

procedure TEnumStringConverter.WriteJson(const AWriter: TJsonWriter; const
    AValue: TValue; const ASerializer: TJsonSerializer);
begin
    AWriter.WriteValue(System.TypInfo.GetEnumName(AValue.TypeInfo, AValue.AsOrdinal));
end;

See also:  

 

Bummer: System.Json.Converters already includes TJsonEnumNameConverter that does the job.

 

 

Edited May 16 by pyscripter

[Stefan Glienke 2187]

FWIW, such benchmarks should also run in parallel because especially string-related parsing is usually full of heap allocations that can suffer highly in multithreading.

 

I am also sure that @Arnaud Bouchez could join the game and stomp all these libraries into the ground performancewise with mormot 

Edited May 16 by Stefan Glienke

[Arnaud Bouchez 413]

To be fair, the tests do not seem very realistic to me.

 

And using any non-standard library for a core process like JSON should be not only with Delphi, but also with FPC.
We are on the 21st century, and modern object pascal is not just Delphi. ;)

[Anders Melander 2121]

14 minutes ago, Arnaud Bouchez said:

We are on the 21st century, and modern object pascal is not just Delphi. 😉

[jbg 243]

On 5/16/2025 at 10:28 AM, Stefan Glienke said:

The fact that JDO uses a linear search here is a bit disappointing and the reason it completely falls off the cliff on the generate test.

Well, as of today, it uses a binary search and has a much stricter parser.

[MarceloHByte 2]

On 5/16/2025 at 6:15 AM, Vincent Gsell said:

Hi, 

For me, certains of the fastest JSON  libraries that I practiced is just not present (among others, Mormot, pasjson, etc.).
 

Indeed ! And validation test could be usefull in this kind of bench ! 

 

It is astonishing that many libs does not pass certain basic "proper Json format" validation ! 

Hi,

 

I've added mORMot version 2 and pasjson is on its way.

 

https://github.com/hydrobyte/TestJSON

Marcelo.

[pyscripter 842]

@MarceloHByte Have you also updated JSONDataObjects?   There was a significant update after the first post in this topic.

[Vincent Parrett 893]

This is interesting, I tried it with VSoft.YAML - whilst it can parse JSON, there are some differences in what json and yaml permit.

YAML doesn't allow tabs, so it failed the load file test. I ended up adding a JSONMode to the parser options and made some changes to handle some differences between json and yaml.

[     0/   0]: Starting test
[     0/   0]:   Test   : Validation
[     0/   0]:   Library: VSoft.YAML
[     0/   0]: 
[     0/   0]: FAIL = fail01.json = \x is not a valid escape character = [Exception] = Invalid escape sequence in JSON: \x is not supported at line 1, column 30
[     0/   0]: FAIL = fail02.json = Objects require colon between name/value = [Exception] = Expected Colon but got Value at line 1, column 18
[     0/   0]: FAIL = fail03.json = Objects do not have comma separators = [Exception] = Expected Colon but got Comma at line 1, column 26
[     0/   0]: FAIL = fail04.json = Arrays don't have colon separators = [Exception] = Invalid syntax: object key-value pairs are not allowed in JSON arrays at line 1, column 2
[     0/   0]: FAIL = fail05.json = Truth is not a valid boolean value = [Exception] = Invalid literal value in JSON: "truth". JSON only supports true, false, null, numbers, and quoted strings. at line 1, column 20
[     0/   0]: FAIL = fail06.json = Strings need double quotes, not single quotes = [Exception] = Single-quoted strings are not valid in JSON at line 1, column 2
[     0/   0]: FAIL = fail07.json = Line break in a string value is not valid = [Exception] = Literal line breaks are not allowed in JSON strings. Use \n for newlines. at line 1, column 7
[     0/   0]: FAIL = fail08.json = Escaped line break char is still not valid = [Exception] = Line continuation (backslash followed by newline) is not valid in JSON at line 1, column 7
[     0/   0]: FAIL = fail09.json = Unclosed array = [Exception] = Unexpected end of file in flow sequence (missing "]") at line 1, column 18
[     0/   0]: FAIL = fail10.json = Numbers require exponent if 'e' is there = [Exception] = Invalid number format in JSON: exponent must have at least one digit after e/E at line 1, column 4
[     0/   0]: FAIL = fail11.json = Only 1 sign char can precede the value = [Exception] = Invalid number format in JSON: exponent must have at least one digit after e/E at line 1, column 5
[     0/   0]: FAIL = fail12.json = Commas cannot close objects = [Exception] = Unexpected end of file in flow mapping (expected key) at line 1, column 41
[    16/  16]: FAIL = fail13.json = Brackets must be matching = [Exception] = Expected "," or "]" in flow sequence at line 1, column 12
[    16/   0]: FAIL = fail14.json = Double quotes must be escaped = [Exception] = Expected "," or "]" in flow sequence at line 1, column 18
[    16/   0]: FAIL = fail15.json = Key string must be quoted = [Exception] = Object keys must be quoted strings in JSON at line 1, column 2
[    16/   0]: FAIL = fail16.json = Arrays must not have comma after last value = [Exception] = Missing array element: trailing comma is not allowed in JSON at line 1, column 16
[    16/   0]: FAIL = fail17.json = Arrays must have values between commas = [Exception] = Missing array element: array cannot start with comma in JSON at line 1, column 5
[    16/   0]: FAIL = fail18.json = Nothing but whitespace can follow the root value = [Exception] = Unexpected comma after closing bracket at line 1, column 26
[    16/   0]: FAIL = fail19.json = Each opening bracket must be closed = [Exception] = Unexpected closing bracket "]" - no matching opening bracket at line 1, column 16
[    32/  16]: FAIL = fail20.json = Extra comma after object = [Exception] = Expected key in flow mapping at line 1, column 22
[    32/   0]: FAIL = fail21.json = Numbers cannot have leading 0s = [Exception] = Numbers with leading zeros are not valid in JSON at line 1, column 41
[    32/   0]: FAIL = fail22.json = Numbers can't be hex encoded = [Exception] = Hexadecimal numbers are not valid in JSON at line 1, column 28
[    32/   0]: FAIL = fail23.json = Decimal numbers need a digit before the dot = [Exception] = Decimal numbers must have a leading digit before the dot in JSON at line 1, column 1
[    32/   0]: PASS = pass01.json = General large array testing valid values
[    32/   0]: PASS = pass02.json = Heavily nested array
[    32/   0]: PASS = pass03.json = Nested object
[    32/   0]: PASS = pass04.json = Simple string value
[    47/  15]: PASS = pass05.json = Unicode character string
[    47/   0]: PASS = pass06.json = From https://json.org/example.html
[    47/   0]: Test finished

Whilst VSoft.YAML will not win any races with performance (that was never my focus, adherence to the specs was) - it doesn't do badly.

[   781/   0]: Speed Statistics for System.JSON
[   781/   0]: Average of 5 repetitions (in ms)
[   781/   0]:   Generate:     6.40
[   781/   0]:   Save    :     9.40
[   781/   0]:   Clear   :     6.00
[   781/   0]:   Load    :    25.20
[   797/  16]:   Find    :    62.40
[   797/   0]:   Parse   :    22.00
[   797/   0]:   Total   :   156.20
[   797/   0]: 
[   797/   0]: All done: 419.01 kiB

[  4922/   0]: Speed Statistics for VSoft.YAML
[  4922/   0]: Average of 5 repetitions (in ms)
[  4922/   0]:   Generate:    21.80
[  4922/   0]:   Save    :   315.80
[  4922/   0]:   Clear   :    25.00
[  4922/   0]:   Load    :   328.00
[  4938/  16]:   Find    :     3.00
[  4938/   0]:   Parse   :   256.40
[  4938/   0]:   Total   :   984.40
[  4938/   0]: 
[  4938/   0]: All done: 421.61 kiB

I suspect the poor save performance is due to differences in the the tests and the library.

 

If I change the Save test from

  TYAML.WriteToJSONFile(FYAML, aFileName);

to
  TFile.WriteAllText(aFileName, TYAML.WriteToJSONString(FYAML));

 

Then we see

[  3547/   0]: Speed Statistics for VSoft.YAML
[  3563/  16]: Average of 5 repetitions (in ms)
[  3563/   0]:   Generate:    22.20
[  3563/   0]:   Save    :    37.40
[  3563/   0]:   Clear   :    21.40
[  3563/   0]:   Load    :   331.20
[  3563/   0]:   Find    :     0.00
[  3563/   0]:   Parse   :   256.40
[  3563/   0]:   Total   :   709.40
[  3563/   0]: 
[  3563/   0]: All done: 421.57 kiB

So this was useful for improving json compatibility, but I don't think I am going to be chasing performance improvements unless there are some easy wins - it's fast enough.

Edited 7 hours ago by Vincent Parrett
Found Load was silently failing

[Vincent Parrett 893]

On 5/16/2025 at 6:28 PM, Stefan Glienke said:

The tests ignore some subtle differences in the implementations.

Absolutely correct. In particular, file loading/saving, handling. System.JSON has no Load from file or Stream as far as I can tell - so you are loading the entire file in memory.   I have questions over encoding handling with some of the others too. The Parse test needs to be split up (its really "to json string -> parse json string").

 

I made some performance improvements to VSoft.YAML - but the IO side of things needs more work - I have no idea why it's so slow (it uses buffers). 

[Alexander Sviridenkov 370]

IMHO sample generated in this test is far from real applications. Test data contains single object with lot of unique key/values, while in most real cases there will be a lot of objects having limited number of same keys.

J've made a short test using such file (comes from real REST server)  - 100 times parsing 400Kb file, for System.JSON, Mormot and HtJSON  unit included in HTML Library, here are  results:

 

System.JSON 812ms 258mb memory 

Mormot 266ms, 153Mb memory

HtJson 234ms, 58Mb memory